Azure Databricks Platform engineer
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
- We are looking for an experienced Azure Databricks Platform Engineer to design, implement, administer, automate, and support enterprise-scale Azure Databricks environments.
- The engineer will work closely with data engineering, cloud infrastructure, security, networking, DevOps, and application teams to provide a secure, scalable, governed, reliable, and cost-efficient Databricks platform.
- Design, configure, administer, and support Azure Databricks workspaces across development, testing/UAT, and production environments.
- Manage Databricks compute, cluster policies, jobs, workflows, user access, monitoring, and cost controls.
- Implement and administer Unity Catalog, including catalogs, schemas, permissions, storage credentials, and external locations.
- Integrate Azure Databricks with ADLS Gen2, Azure Data Factory, Azure SQL, Azure Key Vault, and managed identities. Implement secure authentication and authorization using Microsoft Entra ID, Azure RBAC, managed identities, and Key Vault.
- Work with Azure networking concepts including VNets, NSGs, private endpoints, DNS, routing, VPN/ExpressRoute, and hub-and-spoke connectivity.
- Build automation using Python, PowerShell, Bash, Terraform/Bicep and CI/CD tooling to reduce manual platform activities.
- Infrastructure-as-code and scripting are also highlighted in the related internal Databricks requirement.
- Support CI/CD and release management using Azure DevOps, GitHub Actions, or equivalent tooling.
- Establish platform monitoring and observability using Azure Monitor, Log Analytics, Application Insights, Grafana, Prometheus, Datadog, Dynatrace, or New Relic as applicable.
- Monitor platform availability, performance, capacity, and operational health.
- Troubleshoot complex Databricks, Azure infrastructure, storage, identity, networking, and connectivity issues. Implement platform-level security, governance, access controls, and operational standards.
More about this job
Responsibilities
Design, administer, automate, and support secure, scalable Azure Databricks workspaces and platform services across development, testing, and production environments. Manage governance, access, compute, networking, integrations, monitoring, cost controls, and operational troubleshooting in collaboration with cross-functional teams.
Requirements
The role calls for an experienced Azure Databricks platform engineer with expertise in platform administration, Unity Catalog, Azure services, security, identity, and networking. It also requires automation and infrastructure-as-code skills, CI/CD experience, and the ability to monitor and troubleshoot complex platform issues.
Skills
- Unity Catalog
- Azure Cloud Infrastructure
- Infrastructure as Code
- Python
- PowerShell
- Bash
- Terraform
- Bicep
- CI/CD
- Azure DevOps
- GitHub Actions
- Azure Networking
- Identity and Access Management
- Platform Monitoring and Observability
- Troubleshooting
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Software
- Data & Analytics
- Engineering
- Security & Safety
Keywords
- Azure Databricks
- Databricks Workspaces
- Unity Catalog
- ADLS Gen2
- Azure Data Factory
- Azure SQL
- Azure Key Vault
- Microsoft Entra ID
- Azure RBAC
- Managed Identities
- Azure VNets
- Network Security Groups
- Private Endpoints
- DNS
- VPN
- ExpressRoute
- Hub-and-Spoke Connectivity
- Python
- PowerShell
- Bash
- Terraform
- Bicep
- CI/CD
- Azure DevOps
- GitHub Actions
- Azure Monitor
- Log Analytics
- Application Insights
- Grafana
- Prometheus