Principal Databricks Data Engineer

Role - Principal Databricks Data Engineer

Location - Toronto ,ON

overall data engineering experience8 years across enterprise Data Warehouse Data Lake platforms5 years with Databricks Spark at scale Role Summary The Principal Databricks Data Engineer is a senior technical leader responsible for modernizing large scale finance and risk data platforms from legacy Cloudera ecosystems to cloud native Databricks lakehouse architectures. This role demands deep hands on expertise in data warehousing| data lakes| finance risk data models| and semantic consumption layers| with strong experience supporting regulatory| management reporting| and analytics use cases. The individual will serve as a hands on architect and technical authority| leading platform modernization while partnering closely with Finance| Risk| Analytics| and Governance stakeholders .Key Responsibilities Cloudera Databricks Modernization Lead modernization of legacy Cloudera platforms (CDH CDP| Hive| HBase| Impala| Spark) into Databricks Lakehouse. Redesign ingestion| transformation| and consumption patterns from HDFS centric architectures to cloud object storage and Delta Lake. Refactor legacy Hive Impala logic into PySpark Spark SQL based ELT pipelines .Ensure data parity| reconciliation| and audit integrity during platform migration .Enterprise Data Warehouse Data Lake Architecture Design and govern enterprise Data Warehouse and Data Lake Lakehouse architectures .Implement layered architectures spanning o Raw Landing zones o Curated conformed layer so Semantic consumption layers Modernize traditional EDW patterns into domain aligned| scalable lakehouse designs. Finance Risk Data Modeling Support implementation of finance and risk data models| including o General Ledger Sub ledger data o Accounting events and financial hierarchies o Risk exposure| liquidity| credit| and market risk models Enable aggregation| drill down| and drill back from reports to transaction level data. Support regulatory reporting| management reporting| and analytics use cases. Semantic Consumption Layers Build and manage semantic consumption layers to ensure consistent business logic across o BI and reporting toolsoFinance Risk analytics o Self service analytics platforms Define metrics| dimensions| hierarchies| and KPIs aligned to finance and risk definitions .Implement semantic models using o Databricks SQL o Delta tables o dbt or equivalent transformation frameworks Databricks Engineering Optimization Engineer large scale pipelines using PySpark| Spark SQL| and Delta Lake .Implement medallion architecture (Bronze Silver Gold) aligned to business domains. Optimize Databricks workloads for cost| performance| and reliability (Z ORDER| OPTIMIZE| caching| cluster policies).Data Governance| Quality Lineage Implement data quality frameworks| reconciliation co