MyDataCompassData Engineer
Jan. 2022 - Jul. 2023London, England, United Kingdom• Automating the use of Azure Data Factory to extract daily, weekly, and monthly data from transactional database and inserting into the Datawarehouse for consumption by the analytics team. • Development and monitoring of ETL pipelines for analytical purposes using ADF, Azure Databricks, HDInsight, Azure Log Analytics Etc. • Spearheading the building and maintenance of client’s data warehouses and data lakes on Azure SQL databases and Azure Synapse Analytics workspace (for data quality checks, data modeling and data standardization). • Monitoring the overall performance and stability of data pipelines. • Pioneered the use of SQL scripts which includes stored procedures, views, functions and CTEs to build business intelligence solutions. • Configuration of Azure Key Vaults for credentials life cycle management and security. • Migration of data pipelines from dev environment to test and production using Azure devOps while following best software engineering standard practices. • Implementation of the use of Azure blob storage (data lake) for staging and archiving data. • Collaborating with the Analytics team and other Engineers to understand client environments, organizational reporting requirements and proposing the configurations that meets these requirements. • Documentation and Implementation of deployment instructions in line with operating procedures.