Azure Databricks Architect
- Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
We are seeking an experienced and visionary Azure Databricks Architect to lead the design, implementation, and scaling of our enterprise data and AI platform. In this role, you will define the core architecture for our Lakehouse environment, champion governance standards using Databricks Unity Catalog, and spearhead end-to-end performance tuning and cost optimization initiatives. You will work closely with data engineering teams, cloud infrastructure teams, security officers, and business stakeholders to deliver a robust, secure, and high-performance data ecosystem.
Key Responsibilities
- Lakehouse Architecture & Design
- Design and implement enterprise-grade Lakehouse architectures using Azure Databricks and Delta Lake, following Medallion Architecture (Bronze, Silver, Gold layers).
- Architect secure, scalable, and highly available cloud data pipelines supporting batch and real-time streaming workloads.
- Integrate Azure Databricks seamlessly with the broader Azure ecosystem (e.g., Azure Data Lake Storage Gen2, Azure Data Factory, Microsoft Entra ID, and Azure Monitor).
- Data Governance & Security (Unity Catalog)
- Design, deploy, and manage Databricks Unity Catalog for centralized metadata management, data discovery, and lineage tracking across multi-workspace environments.
- Implement fine-grained access controls (RBAC/ABAC), row-level security (RLS), and column-level masking (CLM) to protect sensitive assets (PII/PHI).
- Establish auditing procedures, data sharing frameworks, and compliance guardrails using Unity Catalog audit logs.
- Performance Tuning & Cost Optimization
- Drive Databricks and Spark optimization strategies, including managing shuffle partitions, resolving data skew, tuning cache, and applying Adaptive Query Execution (AQE).
- Implement layout strategies such as Z-ordering, data compaction (OPTIMIZE), and vacuum scheduling to curtail I/O overhead and boost query performance.
- Optimize cluster configurations, autoscaling parameters, spot instances, and serverless compute pools to maximize cost efficiency without compromising SLAs
More about this job
Responsibilities
Design and scale an enterprise Azure Databricks Lakehouse platform, including secure batch and streaming pipelines integrated with Azure services. Lead Unity Catalog governance and security, and optimize Spark performance, cluster configurations, and platform costs.
Requirements
The role calls for mid-to-senior-level expertise in Azure Databricks, Lakehouse architecture, Delta Lake, Unity Catalog, and the Azure ecosystem. It also requires the ability to implement data governance and security controls and tune Spark workloads and cloud resources for performance and cost efficiency.
Skills
- Azure Databricks
- Lakehouse Architecture
- Delta Lake
- Medallion Architecture
- Databricks Unity Catalog
- Data Governance
- Data Security
- RBAC/ABAC
- Row-Level Security
- Column-Level Masking
- Apache Spark Optimization
- Adaptive Query Execution
- Z-Ordering
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Cost Optimization
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Management & Leadership
- Consulting
Keywords
- Azure Databricks
- Lakehouse
- Delta Lake
- Medallion Architecture
- Bronze Layer
- Silver Layer
- Gold Layer
- Batch Processing
- Real-Time Streaming
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Microsoft Entra ID
- Azure Monitor
- Databricks Unity Catalog
- Metadata Management
- Data Discovery
- Data Lineage
- RBAC
- ABAC
- Row-Level Security
- Column-Level Masking
- PII
- PHI
- Audit Logs
- Apache Spark
- Shuffle Partitions
- Data Skew
- Adaptive Query Execution
- Z-Ordering
- OPTIMIZE