Data Engineer | Data & AI Technology
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
The Data Engineer plays a critical role within the Enterprise Data & AI Technology organization. This organization drives data enabled decision making, AI innovation, and technology modernization across the Bank.
What is the Role
- You will design, build, and maintain scalable, resilient data pipelines to support the ingestion, processing, and distribution of large datasets.
- You will leverage SaaS platforms and tools to develop, configure, and automate data workflows, improving efficiency across the data engineering lifecycle.
- You will partner with stakeholders and product managers to gather requirements, design ingestion patterns, and onboard new data sources into the cloud data platform.
- You will implement observability capabilities and continuously optimize application services and pipeline performance.
- You will perform data integrity validation to ensure accuracy, reliability, and consistency of data.
Experience:
- Experience building data pipelines, and composable cloud-based data platforms (Azure), using Azure Data Factory, ADLS.
- Experience working with Lakehouse platforms: Databricks, Unity Catalog, Auto Loader
- Experience with data engineering, programming, ETL and ELT processes for data extraction and processing
- Experience working with structured, semi-structured, and unstructured data.
- Strong programming and scripting skills (SQL, Python, Java, Scala)
- Strong knowledge of cloud infrastructure and microservices architecture
- Experience in creating technical design documentations, both logical and physical view
- Experience collaborating and working with DevOps, Scrum, and Product Teams
- Demonstrated team player with strong communication skills and a track record of successful delivery of product development.
- Solid communication skills – ability to translate complex technical problems to business language
- Experience in leading and driving technical discussions
- Expert at problem solving and debugging
Technical Skills
- 4+ years of experience working with Data Warehouse / Data Platforms
- 4+ years of experience creating ELT / ETL data pipelines, working with structured, semi-structured, and unstructured data
- 4+ years of experience working on continuous integrations and continuous deployment pipeline (CI/CD) and working with source control systems such as Github, Bitbucket, and Terraform
- Must have 4+ years of experience in Cloud: Azure
- Must have 4+ years of experience in Lakehouse: Databricks
More about this job
Responsibilities
Design, build, and maintain scalable data pipelines and cloud data workflows, including onboarding data sources and supporting ingestion, processing, and distribution. Improve pipeline performance and observability, validate data integrity, and collaborate with stakeholders and product teams on requirements and technical designs.
Requirements
Requires at least four years of experience with Azure, Databricks, data platforms, ETL/ELT pipelines, and CI/CD and source control systems. Candidates should have strong programming skills, experience with varied data types and cloud architecture, and the ability to lead technical discussions, troubleshoot problems, and communicate effectively with technical and business teams.
Skills
- Data Pipeline Engineering
- Azure
- Azure Data Factory
- Azure Data Lake Storage
- Databricks
- Unity Catalog
- Auto Loader
- SQL
- Python
- Java
- Scala
- ETL/ELT
- CI/CD
- Terraform
- Data Validation
- Microservices Architecture
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Engineering
Keywords
- Enterprise Data
- Artificial Intelligence
- Data Engineering
- Data Pipelines
- Azure
- Azure Data Factory
- Azure Data Lake Storage
- Databricks
- Unity Catalog
- Auto Loader
- Lakehouse
- Cloud Data Platform
- ETL
- ELT
- Structured Data
- Semi-Structured Data
- Unstructured Data
- SQL
- Python
- Java
- Scala
- Microservices Architecture
- Data Warehousing
- CI/CD
- GitHub
- Bitbucket
- Terraform
- Data Observability
- Data Integrity Validation
- Scrum