Azure Data Enginee
- Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Microsoft Azure services.
- Build and manage ETL/ELT workflows for data extraction, transformation, and loading.
- Develop data integration solutions using Azure Data Factory (ADF).
- Work with Azure Databricks to process and transform large datasets.
- Develop and optimize SQL queries, stored procedures, and data models.
- Implement data processing solutions using Python, PySpark, and Spark SQL.
- Work with Azure Data Lake Storage (ADLS Gen2) for data storage and management.
- Design and maintain data warehouses and data lake solutions.
- Implement data quality checks, validation, and monitoring processes.
- Optimize data pipelines for performance, scalability, and cost efficiency.
- Implement data security, access controls, and governance.
- Troubleshoot and resolve data pipeline and integration issues.
- Collaborate with data analysts, data scientists, and business stakeholders.
- Deploy and manage data solutions using Azure DevOps and CI/CD pipelines.
Preferred Skills
- Strong knowledge of SQL and database concepts.
- Hands-on experience with Microsoft Azure cloud services.
- Experience with Azure Data Factory (ADF).
- Knowledge of Azure Databricks and Apache Spark.
- Strong programming knowledge in Python and PySpark.
- Experience with Azure Data Lake Storage (ADLS Gen2).
- Knowledge of ETL/ELT processes and data pipeline development.
- Experience with Azure Synapse Analytics.
- Knowledge of data warehousing and dimensional modeling.
- Experience with Spark SQL and Delta Lake.
- Understanding of batch and real-time data processing.
- Knowledge of data integration and data transformation techniques.
More about this job
Responsibilities
Design, build, and maintain scalable Azure data pipelines, ETL/ELT workflows, data integration solutions, and data lake and warehouse systems. Optimize and secure data processing, ensure data quality, troubleshoot issues, and deploy solutions through Azure DevOps and CI/CD pipelines in collaboration with stakeholders.
Requirements
The posting seeks strong SQL and programming skills in Python and PySpark, along with hands-on experience with Azure services, Azure Data Factory, Databricks, Spark, and ADLS Gen2. Familiarity with data warehousing, dimensional modeling, Synapse Analytics, Delta Lake, and batch and real-time processing is also preferred.
Skills
- SQL
- Microsoft Azure
- Azure Data Factory
- Azure Databricks
- Apache Spark
- Python
- PySpark
- Azure Data Lake Storage Gen2
- ETL/ELT
- Data Pipeline Development
- Azure Synapse Analytics
- Data Warehousing
- Dimensional Modeling
- Spark SQL
- Delta Lake
- Azure DevOps
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Engineering
Keywords
- Microsoft Azure
- Azure Data Factory
- Azure Databricks
- Apache Spark
- SQL
- Python
- PySpark
- Spark SQL
- Azure Data Lake Storage Gen2
- Azure Synapse Analytics
- Delta Lake
- ETL/ELT
- Data Pipelines
- Data Integration
- Data Transformation
- Data Warehousing
- Dimensional Modeling
- Batch Processing
- Real-Time Processing
- Data Quality
- Data Validation
- Data Monitoring
- Data Security
- Access Controls
- Data Governance
- Azure DevOps
- CI/CD
- Cost Optimization
- Stored Procedures
- Data Models