Keep in touch with meI'm using Intch to connect with new people. Use this link to open chat with me via Intch app
Work Background
Avenue CodeSenior Data Engineer
Feb. 2023Rio de Janeiro, BrasilDigital Transformation Project: migration of SSIS packages from on-premise infrastructure to Azure for an international client (Canada). Phase 1: Lift & Shift
- Successfully migrated SSIS packages to Azure, using Azure Data Factory for orchestrations, Azure SQL for data storage, and Databricks for scalable data processing.
Phase 2: Refactoring
- Refactored codebase from C# to Python, enhancing performance and maintainability.
- Developed an extensive Python library within the Azure DevOps repository to standardize and streamline processes.
- Implemented scheduled jobs in Databricks, automating workflows and ensuring timely data processing.
- Designed CI/CD pipelines in Azure DevOps, incorporating unit tests and integration tests to ensure that updates to the main branch do not disrupt the process. Data Handling & Integration:
- Managed diverse data sources and formats including CSV, TXT, JSON, Parquet, and XML, ensuring seamless integration across the data pipeline.
- Integrated data from a variety of systems, including APIs, SFTP, and direct database queries, maintaining a high standard of data accuracy and availability. Data Platform Creation:
- Architected and implemented a data platform following the Medallion Strategy, utilizing the Bronze, Silver, and Gold layers to manage data lifecycle and quality.
- Bronze Layer: Ingested raw data from multiple sources in its original format, ensuring data integrity.
- Silver Layer: Conducted data cleaning and transformation to produce a curated dataset, ready for analytical processing.
- Gold Layer: Created business logic-driven BI tables (DIM & FACT), ensuring data was ready for reporting and analytics.
- Established QA workflows to maintain data quality across all layers, ensuring accurate and reliable data.
Ame DigitalTech Lead Analytics Engineer
Jul. 2021 - Feb. 2023Rio de Janeiro, BrasilContinued Excellence in Core Responsibilities:
- Maintained high standards in executing all previous roles, ensuring the continued success and efficiency of data processes and projects. Team Coordination & Leadership:
- Led and coordinated data analysts across multiple squads, including Loyalty, Finance, Anti-Fraud, Compliance, and Marketing. Ensured the adoption of best practices across teams, using agile methodologies such as Kanban, sprints, and sprint retrospectives to effectively manage and guide the analysts. Creation of Data Academy:
- Established a company-wide Data Academy, offering online courses in key areas such as Excel, SQL, Power BI, and Python. This initiative aimed to spread knowledge and enhance data literacy across the organization, empowering employees to leverage data in their respective roles.
Ame DigitalSenior Analytics Engineer
Mar. 2020 - Jul. 2021Rio de Janeiro, BrasilETL Development & Data Integration:
- Developed and maintained diverse ETL processes, facilitating seamless data movement across various platforms: - Transferred files from FTP servers. - Integrated with Eventhub Kafka for real-time data streaming. - Sent data to Eventhub and API endpoints. - Managed data exchanges via Azure DataShare. - Accessed and processed files from EC2 and Blob Storage. - Queried data from Google BigQuery and AWS MySQL. - Implemented a Change Data Capture (CDC) process to handle updated transactions from a MySQL & DynamoDB based wallet system. Gold Layer Expansion:
- Developed new tables in the gold layer to accommodate new products within the wallet. Some of these tables were integrated into an OLAP cube to enhance analytical capabilities. Data Life Cycle Management:
- Implemented a data life cycle strategy to optimize storage costs, ensuring that data older than a specified period was automatically moved to cold storage, significantly reducing storage expenses. Cost Optimization Initiatives:
- Led a project to reduce operational costs by optimizing queries and tables, partitioning large tables, and implementing resource tagging to track and manage job costs effectively. Data Governance & Management:
- Established data governance protocols, including data access management flows, and utilized Unity Catalog to catalog and organize data assets, ensuring data integrity and compliance.
Ame DigitalSenior Data Analyst
Jan. 2020 - Mar. 2020Rio de Janeiro, BrasilSenior Data Analyst - Technology Department Transition to Technology Department:
- Due to my excellent performance in the marketing department, I was invited by the Technology Department to join the Data team. This opportunity aligned perfectly with my career goals and allowed me to further develop my expertise in data analysis and engineering. Gold Layer Development:
- In my new role, I was responsible for creating the company's gold layer, beginning with two critical tables: transactional and customer. These tables significantly reduced the complexity of data access by eliminating the need for complex joins and became the foundation for most of our BI reports. I also ensured the availability of extensive documentation to support these efforts. Self-Service BI Implementation:
- Lead the development of a self-service BI solution using Azure Analysis Services. By integrating the transactional and customer datasets, I created a robust OLAP resource with DAX-based measurements, empowering company collaborators to connect and analyze official datasets through Excel or Power BI.
Ame DigitalSenior Business Intelligence Analyst
Nov. 2019 - Jan. 2020Rio de Janeiro e Região, BrasilSenior Business Analyst - Marketing Department BI Reporting & Automation:
- Developed comprehensive BI reports that highlighted key performance indicators (KPIs) for the company, driving strategic decision-making.
- Automated repetitive tasks that marketing analysts previously spent significant time on, increasing efficiency and enabling the team to focus on more impactful activities. Data-Driven Insights:
- Worked closely with commercial and marketing analysts to extract actionable insights from data, directly contributing to the enhancement of promotional strategies. Data Platform Development:
- Developed the first version of the company's gold layer. Leveraging my close collaboration with business analysts, I designed a flat table that addressed multiple business challenges by reducing the need for complex joins, simplifying data access and analysis. First Black Friday Campaign:
- During my first-ever Black Friday, I created dashboards to monitor goal achievement, year-over-year growth, and projections. These dashboards were crucial in providing real-time insights and driving timely decisions throughout the campaign.
OiSenior Business Intelligence Analyst
Apr. 2019 - Oct. 2019Rio de Janeiro e Região, BrasilETL Development:
- Created ETL processes using scripts and procedures in Microsoft SQL Server, integrating various data sources such as .csv, .xlsx, .txt, and .accdb files. Process Improvement:
- Enhanced existing scripts to optimize data processing and improve overall efficiency. Data Analysis & Visualization:
- Analyzed data, created clusters, and developed interactive and management dashboards using Power BI to provide actionable insights. Web Crawling Automation:
- Developed web crawlers in Python (using selenium and pandas libraries) to automate website data extraction, perform ETL processes, and update the database with captured information. Excel Automation:
- Developed Excel Macros (VBA) to consolidate spreadsheets, perform automatic checks, and connect to databases for query export, streamlining data management tasks.
EYTrainee
Jul. 2018 - Apr. 2019Rio de Janeiro e Região, BrasilParticipated in consulting projects in the following industries:
- Oil & Gas – Digital Transformation
- Telecommunications – Reconciliation of balances in judicial processes Process Management
- Mapped process flows to improve efficiency and clarity in operations. ETL Development:
- Created ETL processes using scripts and procedures in Microsoft SQL Server, integrating various data sources such as .csv, .xlsx, .txt, and .accdb files. Excel Automation:
- Developed Excel Macros (VBA) to consolidate spreadsheets, perform automatic checks, and connect to databases for query export.
Intch is a Professional Networking App for the Future of WorkSee more people like Rafael on Intch
300k+ people
130+ countries
AI matching
Get Intch if you'd like to
453430 people
17
IT Project Manager @ Freelancer
26
Mobile Engineer @ Raiô Benefícios
15
.Net Fullstack Developer/Angular Developer @ Webfis Processamento de Dados Ltda
47999 people
15
.Net Fullstack Developer/Angular Developer @ Webfis Processamento de Dados Ltda
15
Socio @ Domti
17
Dev PHP, Vue