Data Engineer

The Data Engineer plays a critical role within the Enterprise Data & AI Technology organization. This organization drives data enabled decision making, AI innovation, and technology modernization across the organization.

What is the Role

• You will design, build, and maintain scalable, resilient data pipelines to support the ingestion, processing, and distribution of large datasets.
• You will leverage SaaS platforms and tools to develop, configure, and automate data workflows, improving efficiency across the data engineering lifecycle.
• You will partner with stakeholders and product managers to gather requirements, design ingestion patterns, and onboard new data sources into the cloud data platform.
• You will implement observability capabilities and continuously optimize application services and pipeline performance.
• You will perform data integrity validation to ensure accuracy, reliability, and consistency of data.

Experience:

• Experience building data pipelines, and composable cloud-based data platforms (Azure), using Azure Data Factory, ADLS.
• Experience working with Lakehouse platforms: Databricks, Unity Catalog, Auto Loader
• Experience with data engineering, programming, ETL and ELT processes for data extraction and processing
• Experience working with structured, semi-structured, and unstructured data.
• Strong programming and scripting skills (SQL, Python, Java, Scala)
• Strong knowledge of cloud infrastructure and microservices architecture
• Experience in creating technical design documentations, both logical and physical view
• Experience collaborating and working with DevOps, Scrum, and Product Teams
• Demonstrated team player with strong communication skills and a track record of successful delivery of product development.
• Solid communication skills – ability to translate complex technical problems to business language
• Experience in leading and driving technical discussions
• Expert at problem solving and debugging

Technical Skills

• 4+ years of experience working with Data Warehouse / Data Platforms
• 4+ years of experience creating ELT / ETL data pipelines, working with structured, semi-structured, and unstructured data
• 4+ years of experience working on continuous integrations and continuous deployment pipeline (CI/CD) and working with source control systems such as Github, Bitbucket, and Terraform
• Must have 4+ years of experience in Cloud: Azure
• Must have 4+ years of experience in Lakehouse: Databricks