The Data Engineer plays a critical role within the Enterprise Data & AI Technology organization. This organization drives data enabled decision making, AI innovation, and technology modernization across the organization.
What is the Role
- You will design, build, and maintain scalable, resilient data pipelines to support the ingestion, processing, and distribution of large datasets.
- You will leverage SaaS platforms and tools to develop, configure, and automate data workflows, improving efficiency across the data engineering lifecycle.
- You will partner with stakeholders and product managers to gather requirements, design ingestion patterns, and onboard new data sources into the cloud data platform.
- You will implement observability capabilities and continuously optimize application services and pipeline performance.
- You will perform data integrity validation to ensure accuracy, reliability, and consistency of data.
Experience:
- Experience building data pipelines, and composable cloud-based data platforms (Azure), using Azure Data Factory, ADLS.
- Experience working with Lakehouse platforms: Databricks, Unity Catalog, Auto Loader
- Experience with data engineering, programming, ETL and ELT processes for data extraction and processing
- Experience working with structured, semi-structured, and unstructured data.
- Strong programming and scripting skills (SQL, Python, Java, Scala)
- Strong knowledge of cloud infrastructure and microservices architecture
- Experience in creating technical design documentations, both logical and physical view
- Experience collaborating and working with DevOps, Scrum, and Product Teams
- Demonstrated team player with strong communication skills and a track record of successful delivery of product development.
- Solid communication skills – ability to translate complex technical problems to business language
- Experience in leading and driving technical discussions
- Expert at problem solving and debugging
Technical Skills
- 4+ years of experience working with Data Warehouse / Data Platforms
- 4+ years of experience creating ELT / ETL data pipelines, working with structured, semi-structured, and unstructured data
- 4+ years of experience working on continuous integrations and continuous deployment pipeline (CI/CD) and working with source control systems such as Github, Bitbucket, and Terraform
- Must have 4+ years of experience in Cloud: Azure
- Must have 4+ years of experience in Lakehouse: Databricks
