ImpetusModule Lead
Jan. 2016 - Apr. 2019Indore, Madhya Pradesh, IndiaProject:- American Express (APCommons, Denali & SupplierIQ) •Created batch workflow pipeline from scratch using Oozie to perform various Hive, Java, Shell action etc. for APCommons.
•Created SupplierIQ application from ground up using Spark SQL, Kafka.
•Implemented Matching overriding mechanism using Spark SQL for SupplierIQ.
•Optimised Spark SQL Job to process around 1TB data for Denali.
•Optimization helped in generation of 15x more reports daily for Denali.
•Participated in design discussions around all the applications and involved in development of the same.
•Provided Training and guidance to new members joining the team around various applications.
•Worked on improving data quality, reliability & efficiency of the individual components & the complete system.