Job Title: Principal Azure Databricks Architect
Location: Toronto, ON (hybrid)
Duration: long term contract
Rate: CAD110-CAD120/hr
Role Overview
We are seeking an experienced Principal Azure Databricks Architect to design, implement, and optimize enterprise-scale data platforms within the Azure ecosystem. The ideal candidate will have deep expertise in Azure Databricks, enterprise data lakes, Lakehouse architecture, data engineering, cloud cost optimization, and modern CI/CD practices.
This role requires strong hands-on coding skills, architectural leadership, and the ability to work independently with business and technical stakeholders to deliver scalable, secure, and cost-effective data solutions.
Key Responsibilities
- Design, architect, and implement enterprise-scale data lakes and Lakehouse solutions on Azure.
- Lead the setup and optimization of Azure Databricks and Azure Data Lake Storage environments.
- Develop scalable data pipelines using Azure Data Factory, Databricks, Python, PySpark, and SQL.
- Design robust data models and integration frameworks for structured and unstructured data.
- Write complex, production-ready code and troubleshoot application, pipeline, and performance issues.
- Establish CI/CD processes using Jenkins and GitHub.
- Implement monitoring and observability solutions using Dynatrace.
- Provide architectural guidance around Lakehouse design, Databricks environments, governance, security, and performance.
- Drive Databricks and cloud FinOps initiatives to optimize platform usage and costs.
- Work with relational and NoSQL databases to support enterprise data integration requirements.
- Collaborate with stakeholders to understand business needs and independently deliver robust technical solutions.
- Provide technical leadership, establish engineering best practices, and guide development teams.
- Evaluate emerging technologies related to AI agents, skills, and Model Context Protocol, also known as MCP.
Required Qualifications
- Extensive experience designing and implementing enterprise data lakes and Lakehouse architectures.
- Strong hands-on expertise in:
- Azure Databricks
- Azure Data Factory
- Azure Data Lake Storage
- Python and PySpark
- Advanced SQL
- Jenkins and GitHub
- Dynatrace or similar observability tools
- Strong knowledge of Databricks FinOps, cloud cost optimization, and platform performance.
- Experience working with relational and NoSQL databases.
- Strong programming experience in Python, PySpark, or Java.
- Ability to develop complex code and independently diagnose and resolve technical issues.
- Excellent analytical and problem-solving capabilities.
- Strong stakeholder management and communication skills.
- Ability to translate business requirements into scalable technical architecture.
Nice-to-Have Skills
- Experience developing dashboards and reports using Power BI.
- Exposure to one or more BI tools, such as Cognos, Power BI, Sigma, or Strategy.
- Experience building AI agents and related AI capabilities.
- Understanding of Model Context Protocol, or MCP.
- Additional hands-on experience with Dynatrace.
- Functional experience in retail banking, capital markets, or securities processing.
