MVR IT LIMITEDAzure Data engineer
Nov. 2021United KingdomProject: POCKET GEEK / CONNECTED LIVING
As a Data Engineer, I have been involved in designing, developing and deploying data pipelines and data solutions for our clients. Some of my day-to-day activities include:
• Creating Pipelines in Azure Data Factory (ADF) using Linked Services/Datasets/Pipeline to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and backward.
• Developing Spark applications using Spark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing and transforming the data to uncover insights into customer usage patterns.
• Analyzing the requirements and developing the technical components of the project, including the ETL (Extract, Transform, and Load) strategy to populate the Data Warehouse from various source systems to Azure Data Lake using a combination of Azure Data Factory, Spark, and Spark-SQL.
• Performing Data Ingestion to Adls gen2 from different web sources using API's and web scraping.
• Creating pipelines in ADF using Linked Services, Datasets, and Pipelines to extract data from different sources.
• Creating build and release for production release using Visual Studio Team Services.
• Writing unit test conditions to ensure the quality of the code.
• Migrating the existing Informatica Jobs to Azure Data Bricks by using Python (Spark), Azure functions.
• Developing BI reports based on a wide range of data sets using BI development tools such as SSIS, SSAS, SSRS, Tableau, etc., to ensure reports deliver accurate and meaningful information to our stakeholders.