Capital OneSoftware Engineer
Oct. 2022 - Nov. 2023Remote● Directed the design and implementation of scalable data pipelines for an AWS-based Data Lake, utilising advanced tools such as Scala, Apache Spark, AWS Lambda, and Docker, which resulted in significant enhancements in data acquisition and processing.
● Leveraged AWS Step Functions and Spark to optimise data transformation processes, resulting in improved performance and increased reliability.
● Identified and remedied inefficiencies in Dockerfile structures, which led to a considerable reduction in image size and optimised AWS deployment time.
● Played a crucial role in driving technical improvements and innovation through the application of sophisticated software engineering principles and methodologies.
● Exemplified exceptional leadership skills, guiding and mentoring junior team members, fostering a collaborative work environment, and enhancing overall team productivity.
● Collaborated with cross-functional teams to implement monitoring and alerting systems for data pipeline reliability, reducing incident response time by 40%