Senior Data Engineer
- Toronto, ON, Canada
- Greater Toronto Area, Canada, Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Apply by
- Posting language
- English
- Posted
Original job description
Role : Azure Data Engineer with Databricks Expertise
Location : Ottawa/Toronto (Hybrid)
Contract: 06+ Months
Job Description:
Key Responsibilities:
1. Data Pipeline Development:
o Build and maintain scalable ETL/ELT pipelines using Databricks.
o Leverage PySpark/Spark and SQL to transform and process large datasets.
o Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems.
2. Collaboration & Analysis:
o Work Closely with multiple teams to prepare data for dashboard and BI Tools.
o Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
3. Performance & Optimization:
o Optimize Databricks workloads for cost efficiency and performance.
o Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
- Governance & Security:
o Implement and manage data security, access controls and governance standards using Unity Catalog.
o Ensure compliance with organizational and regulatory data policies.
- Deployment:
o Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments.
o Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.
More about this job
Responsibilities
Develop and maintain scalable data pipelines using Databricks, PySpark/Spark, and SQL, integrating data from Azure storage and other systems for analytics and BI. Optimize and monitor workloads, implement governance and security controls with Unity Catalog, and deploy Databricks artifacts across environments using Databricks Asset Bundles.
Requirements
The role calls for expertise in Azure data engineering and Databricks, including PySpark/Spark, SQL, pipeline development, and data integration. It also requires collaboration with cross-functional teams and experience with workload optimization, data governance, security, deployment, and version control.
Skills
- Databricks
- Azure
- PySpark
- Apache Spark
- SQL
- ETL/ELT Pipelines
- Data Integration
- Data Transformation
- Azure Blob Storage
- Azure Data Lake Storage
- Unity Catalog
- Data Governance
- Data Security
- Performance Optimization
- Databricks Asset Bundles
- Version Control
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- Azure Data Engineering
- Databricks
- PySpark
- Apache Spark
- SQL
- ETL/ELT
- Data Pipelines
- Azure Blob Storage
- Azure Data Lake Storage
- Relational Systems
- Non-Relational Systems
- Business Intelligence
- Dashboards
- Workload Optimization
- Cost Efficiency
- Pipeline Monitoring
- Troubleshooting
- Unity Catalog
- Data Governance
- Access Controls
- Data Security
- Regulatory Compliance
- Databricks Asset Bundles
- Deployment
- Notebooks
- Version Control
- Cross-Functional Collaboration