Azure Databricks Platform engineer
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
Job Title: Azure Databricks Platform engineer
Location: Toronto, ON Hybrid
Job Type: Full-Time Permanent
- We are looking for an experienced Azure Databricks Platform Engineer to design, implement, administer, automate, and support enterprise-scale Azure Databricks environments.
- The engineer will work closely with data engineering, cloud infrastructure, security, networking, DevOps, and application teams to provide a secure, scalable, governed, reliable, and cost-efficient Databricks platform.
- Design, configure, administer, and support Azure Databricks workspaces across development, testing/UAT, and production environments.
- Manage Databricks compute, cluster policies, jobs, workflows, user access, monitoring, and cost controls.
- Implement and administer Unity Catalog, including catalogs, schemas, permissions, storage credentials, and external locations.
- Integrate Azure Databricks with ADLS Gen2, Azure Data Factory, Azure SQL, Azure Key Vault, and managed identities. Implement secure authentication and authorization using Microsoft Entra ID, Azure RBAC, managed identities, and Key Vault.
- Work with Azure networking concepts including VNets, NSGs, private endpoints, DNS, routing, VPN/ExpressRoute, and hub-and-spoke connectivity.
- Build automation using Python, PowerShell, Bash, Terraform/Bicep and CI/CD tooling to reduce manual platform activities.
- Infrastructure-as-code and scripting are also highlighted in the related internal Databricks requirement.
- Support CI/CD and release management using Azure DevOps, GitHub Actions, or equivalent tooling.
- Establish platform monitoring and observability using Azure Monitor, Log Analytics, Application Insights, Grafana, Prometheus, Datadog, Dynatrace, or New Relic as applicable.
- Monitor platform availability, performance, capacity, and operational health.
- Troubleshoot complex Databricks, Azure infrastructure, storage, identity, networking, and connectivity issues. Implement platform-level security, governance, access controls, and operational standards.
More about this job
Responsibilities
Design, administer, automate, and support enterprise Azure Databricks environments, including workspaces, compute, jobs, Unity Catalog, security, governance, and cost controls. Integrate Databricks with Azure services, build infrastructure automation and CI/CD processes, monitor platform health, and troubleshoot complex infrastructure and connectivity issues.
Requirements
The role calls for experience operating enterprise-scale Azure Databricks platforms and working across data engineering, cloud infrastructure, security, networking, DevOps, and application teams. Relevant capabilities include Azure identity and networking, infrastructure as code and scripting, CI/CD, observability, platform security, and troubleshooting; no specific education requirement is stated.
Skills
- Azure Databricks
- Unity Catalog
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Azure SQL
- Azure Key Vault
- Microsoft Entra ID
- Azure RBAC
- Azure Networking
- Python
- PowerShell
- Bash
- Terraform
- Bicep
- CI/CD
- Azure DevOps
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Data & Analytics
- Software
- Engineering
- Security & Safety
Keywords
- Azure Databricks
- Databricks Workspaces
- Unity Catalog
- Azure Data Lake Storage Gen2
- Azure Data Factory
- Azure SQL
- Azure Key Vault
- Microsoft Entra ID
- Azure RBAC
- Managed Identities
- Azure Virtual Networks
- Network Security Groups
- Private Endpoints
- DNS
- VPN
- ExpressRoute
- Hub-And-Spoke Connectivity
- Python
- PowerShell
- Bash
- Terraform
- Bicep
- Infrastructure As Code
- CI/CD
- Azure DevOps
- GitHub Actions
- Azure Monitor
- Log Analytics
- Application Insights
- Grafana