Azure Data Engineer
- Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF).
- Develop data transformation and processing solutions using Azure Databricks and PySpark.
- Build and manage data storage solutions using Azure Data Lake Storage (ADLS Gen2).
- Write complex SQL queries, stored procedures, and optimize query performance.
- Implement incremental data loads, CDC, and batch processing.
- Work with Delta Lake for data management, including MERGE, UPDATE, and DELETE operations.
- Develop data pipelines following Medallion Architecture (Bronze, Silver, Gold).
- Implement data integration from multiple sources such as databases, APIs, and flat files.
- Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
- Implement security using Azure Key Vault, Managed Identity, and RBAC.
- Work on CI/CD deployments using Azure DevOps and Git.
- Collaborate with business analysts, data architects, and other technical teams.
Preferred Skills
- Hands-on experience with Azure Data Factory (ADF).
- Strong knowledge of Azure Databricks and PySpark.
- Proficiency in Python programming and SQL.
- Experience with Azure Data Lake Storage Gen2.
- Knowledge of Azure Synapse Analytics.
- Experience with Delta Lake and Lakehouse Architecture.
- Understanding of ETL/ELT concepts and data warehousing.
- Experience with incremental loading and Change Data Capture (CDC).
- Knowledge of Azure DevOps, Git, and CI/CD pipelines.
- Understanding of data modeling and data quality concepts.
- Good problem-solving and analytical skills.
- Excellent communication and collaboration skills.
More about this job
Responsibilities
Design, build, and maintain scalable Azure data pipelines and storage solutions, using ADF, Databricks, PySpark, SQL, and Delta Lake. Integrate data from multiple sources, optimize pipeline performance and reliability, implement security and CI/CD practices, and collaborate with business and technical teams.
Requirements
The posting seeks hands-on experience with Azure Data Factory, Databricks, PySpark, Python, SQL, and Azure Data Lake Storage Gen2, along with knowledge of data warehousing, Lakehouse Architecture, and data quality. Familiarity with incremental loading, CDC, Azure DevOps, Git, and CI/CD is also preferred, as are strong problem-solving, analytical, communication, and collaboration skills.
Skills
- Azure Data Factory
- Azure Databricks
- PySpark
- Python
- SQL
- Azure Data Lake Storage Gen2
- Azure Synapse Analytics
- Delta Lake
- Lakehouse Architecture
- ETL/ELT
- Incremental Loading
- Change Data Capture
- Azure DevOps
- Git
- CI/CD
- Data Modeling
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- Azure Data Factory
- Azure Databricks
- PySpark
- Python
- SQL
- Azure Data Lake Storage Gen2
- Azure Synapse Analytics
- Delta Lake
- Lakehouse Architecture
- ETL/ELT
- Data Warehousing
- Incremental Data Loads
- Change Data Capture
- Batch Processing
- Medallion Architecture
- Bronze, Silver, Gold
- Data Integration
- APIs
- Flat Files
- Data Modeling
- Data Quality
- Azure Key Vault
- Managed Identity
- RBAC
- Azure DevOps
- Git
- CI/CD
- Stored Procedures
- Query Optimization
- Data Pipelines