Senior Data Engineer

Senior Data Engineer

Toronto, CA (Onsite)

Contract

Senior Data Engineer - Databricks, Microsoft Fabric:

We are seeking a highly skilled Data Engineer with strong expertise in Databricks and modern cloud data platforms. The ideal candidate will design, develop, and optimize scalable data pipelines, data lakehouse solutions, and analytics platforms using Azure Databricks. Secondary experience in Microsoft Fabric and Databricks Genie is highly desirable to support self-service analytics, conversational AI, and enterprise reporting initiatives.

## Key Responsibilities

### Databricks Data Engineering (Primary)

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.

• Build and manage Medallion Architecture (Bronze, Silver, Gold) for enterprise data platforms.

• Ingest data from various structured and unstructured sources including APIs, databases, files, ERP, CRM, and cloud storage.

• Optimize Spark jobs, cluster configurations, and data processing workloads for performance and cost efficiency.

• Develop reusable frameworks, libraries, and components for ingestion, transformation, and data quality validation.

• Implement CI/CD pipelines using Azure DevOps or Git-based deployment methodologies.

• Monitor, troubleshoot, and resolve production data pipeline issues.

• Collaborate with business analysts, architects, and stakeholders to translate business requirements into technical solutions.

### Microsoft Fabric (Secondary)

• Develop and maintain data solutions using Microsoft Fabric components including Data Factory, Lakehouse, Warehouse, Notebooks, and Data Pipelines.

• Support data integration and modernization initiatives from legacy platforms to Fabric.

• Enable Power BI semantic models and reporting solutions leveraging Fabric architecture.

• Implement governance, security, and data management best practices within Fabric.

### Databricks Genie (Secondary)

• Support implementation and adoption of Databricks Genie for conversational analytics and natural language querying.

• Collaborate with business users to define semantic layers and business-friendly data models.

• Enable self-service analytics through AI-powered data exploration capabilities.

• Assist in developing governance and controls for AI-driven data access.

## Required Technical Skills

• Azure Databricks

• PySpark

• Spark SQL

• Delta Lake

• Python

• SQL

• Azure Data Lake Storage (ADLS)

• Azure Data Factory (ADF)

• Git / Azure DevOps

• Data Modeling

• ETL / ELT Development

• Performance Tuning & Optimization

• Data Warehousing Concepts

## Preferred Skills

• Microsoft Fabric

• Fabric Lakehouse & Warehouse

• Power BI

• Databricks Genie

• Unity Catalog

• Databricks Workflows

• REST APIs

• Data Governance & Security

• AI/GenAI-enabled Analytics Solutions

## Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or related field.

• 7+ years of experience in Data Engineering.

• 4+ years of hands-on experience with Azure Databricks.

• Experience delivering enterprise cloud data platforms.

• Databricks certification (Associate/Professional) is a strong advantage.