Senior Data Engineer / ETL Architect@Banco Bradesco
Verified
Data Engineer
As a data engineer, I am leading the migration of processes executed in on-premises environments using Hive to the cloud, utilizing Databricks as the main tool. This migration involves not only replicating but also improving the orchestration processes currently executed in IBM's IWS, a tool that no longer meets our needs.
Another important aspect of my work is migrating the data we send to FICO to MongoDB. We are creating unstructured data succinctly, allowing FICO to consume our JSON-based datasets in the best possible manner.
Tools Used:
Databricks
Hive
DevOps
Athena
Teradata
SAS
Main Languages:
Spark
Python
SQL
Within Databricks, we mainly use:
Workflow with jobclusters for each task group, aiming to optimize resources and processing time.More...