Data Integration Lead
- Canada
Applications open in a new tab.
Job details
- Posting language
- English
- Posted
Original job description
Real-time data pipelines using Azure Databricks, PySpark, Kafka, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS), enabling reliable ingestion and transformation of 10M+ records daily. • Developed and optimized SQL and PySpark transformations within medallion-style ETL workflows, improving processing efficiency, maintainability, and data quality for downstream analytics. • Identified requirement gaps and data quality issues stemming from upstream data sources and solutioned root-cause fixes by implementing validation rules, refining transformation logic, and collaborating with stakeholders to improve downstream data reliability.
More about this job
Responsibilities
The Data Integration Lead will design and maintain real-time data pipelines using Azure technologies to process large volumes of daily records. They will also optimize ETL workflows and implement validation rules to ensure high data quality and reliability for downstream analytics.
Requirements
The role requires extensive experience in building and optimizing data pipelines using PySpark, SQL, and Azure services. Candidates must be proficient in identifying data quality issues and collaborating with stakeholders to implement effective root-cause solutions.
Skills
- Azure Databricks
- PySpark
- Kafka
- Azure Data Factory
- Azure Data Lake Storage
- SQL
- ETL
- Data Pipelines
- Data Ingestion
- Data Transformation
- Data Quality
- Validation Rules
- Medallion Architecture
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Engineering
Keywords
- Data Integration
- Azure Databricks
- PySpark
- Kafka
- Azure Data Factory
- Azure Data Lake Storage
- SQL
- ETL
- Medallion Architecture
- Data Pipelines
- Data Quality
- Data Ingestion
- Data Transformation
- Validation Rules
- Analytics
- Cloud Computing