Senior/Lead Data Engineer
Duration: Long Term
Location: Ottawa, ON K2J 0P7
Role Summary
Our client is seeking an experienced, hands-on Senior Data Engineer to help build and scale the enterprise data foundation supporting Analytics, AI, and self-service decision-making across the company.
This role requires more than just moving data between systems. The successful candidate will act as a data product architect, transforming complex, messy operational data into trusted, reusable, business-ready data products. These products will power enterprise analytics, Tableau reporting, Databricks Genie / Genie Code, machine learning, and emerging GenAI use cases across Manufacturing, Supply Chain, Sales, Finance, and HR.
You will play a pivotal role in shifting the organization away from one-off, siloed pipelines toward a highly governed, centralized single-source-of-truth platform built entirely on Databricks and AWS.
Key Responsibilities:
1. Data Engineering & Lakehouse Development
- Design, develop, test, and maintain scalable ETL/ELT pipelines using Databricks, Spark, SQL, Python, and AWS.
- Build and maintain lakehouse solutions leveraging Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
- Integrate structured and semi-structured data from enterprise applications, manufacturing systems, databases, APIs, and cloud operational sources.
- Optimize Databricks workloads, SQL queries, Delta tables, compute utilization, and end-to-end pipeline performance.
- Implement orchestration, monitoring, error handling, data reconciliation, and automated recovery for production data pipelines.
2. Enterprise Data Modeling & Data Products
- Design scalable enterprise data models (dimensional modeling, star schemas, fact/dimension structures) supporting cross-functional analytics and standardized KPIs.
- Transform raw operational data into curated, business-ready data products that can be reused across dashboards, analytics, AI, and ML applications.
- Collaborate closely with business and analytics teams to translate complex business definitions into technically robust data models.
3. Analytics AI & Self-Service Enablement
- Build AI-ready and Genie-ready data products that allow business users to securely analyze enterprise data using Databricks Genie and Genie Code.
- Structure data, metadata, relationships, business definitions, and semantic/contextual layers to maximize the accuracy of conversational AI analytics.
- Enable engineers, analysts, and business users to perform governed self-service analysis without creating uncontrolled copies of data.
- Partner with data science teams to prepare feature-ready datasets for predictive analytics, machine learning, and MLOps workloads.
4. Data Governance, Quality & DataOps
- Implement data governance, cataloging, lineage, metadata management, and role-based access controls using Databricks Unity Catalog.
- Build automated data-quality checks for completeness, accuracy, consistency, freshness, and cross-system reconciliation.
- Apply Git-based development, code reviews, testing, version control, and automated CI/CD processes for Databricks assets.
- Focus heavily on developing reusable engineering frameworks and templates rather than building hard-coded, one-off pipelines.
