Senior Databricks Data Engineer (Azure)

Position: Senior Databricks Data Engineer

Location: Hybrid- Toronto, ON

Job type: Full-time/Permanent hiring

We are seeking an experienced Senior Databricks Data Engineer with 8–10 years of industry experience to architect, build, and optimize enterprise data pipelines on the Databricks Lakehouse platform. You will spearhead end-to-end ETL/ELT development across batch and streaming workloads, implement medallion architectures, and enforce fine-grained data governance using Unity Catalog. Working in an agile environment, you will automate deployment workflows using modern DevOps practices and ensure high performance and reliability across production data ecosystems.

Core Responsibilities

• ETL & Streaming Pipeline Development: Design, build, and performance-tune scalable batch and real-time streaming pipelines using PySpark, Spark SQL, and Delta Lake.
• Lakehouse & Medallion Architecture: Architect and implement multi-layered medallion architectures (Bronze, Silver, Gold) utilizing Delta Live Tables (DLT) for declarative data transformations.
• Workflow Orchestration & Automation: Build, schedule, and monitor production Databricks jobs and end-to-end data workflows using Azure Data Factory (ADF) and Databricks Workflows.
• Data Governance & Security: Configure and manage fine-grained access controls, data lineage, metadata management, and security compliance via Unity Catalog.
• CI/CD & DataOps Engineering: Maintain version-controlled repositories in Git and build automated deployment pipelines utilizing Azure DevOps, GitHub Actions, or Databricks Asset Bundles (DABs).

Required Technical Skills

• Experience: 8–10 years of overall IT/data engineering experience, with deep hands-on expertise delivering production workloads on the Azure Databricks Lakehouse platform.
• Core Big Data Stack: Advanced proficiency in PySpark, Spark SQL, Delta Lake, and Delta Live Tables (DLT), including cluster sizing and query optimization.
• Orchestration & Cloud Ecosystem: Strong practical experience integrating Azure Data Factory (ADF), Databricks Workflows, and Azure storage services (ADLS Gen2).
• Governance & Compliance: Proven track record setting up and administering Unity Catalog for enterprise access control and metadata governance.
• DevOps & Automation: Solid hands-on experience with Git, CI/CD pipelines (Azure DevOps / GitHub Actions), and Databricks Asset Bundles.