Data Architect (Databricks)

Position: Data Architect (Databricks & Medallion)

Location: Hybrid- Toronto, ON

Job type: Full-time/Permanent hiring

The Senior Data Architect will design the end-to-end data platform for APEX, ingesting call center telephony and quality data from Genesys A3S Snowflake and Verint WFM feeds, and delivering it through a Bronze/Silver/Gold (Medallion) architecture into a Kimball-style reporting layer. This role requires deep hands-on expertise in dimensional modeling, Databricks/Delta Lake, Spark SQL, and API-driven ingestion to deliver reliable, production-grade analytics for sales and telephony reporting.

Roles & Responsibilities

• Medallion Architecture: Design Bronze/Silver/Gold data flow for Genesys A3S Snowflake and Verint sources.
• Ingestion Strategy: Define ingestion cadence and watermarking strategy per source, based on true data freshness.
• Star Schema Design: Design Gold-layer star schema (facts, dimensions) to support sales and telephony reporting.
• Data Integrity & Idempotency: Define data integrity checks, batch control, idempotency, and failure-recovery patterns for each ingestion pipeline.
• Architecture Documentation: Document architecture decisions and trade-offs for both engineering and business audiences.
• Parallel-Run & Decommission: Partner with the existing CCARS pipeline team during the parallel-run and decommission phases.
• Dimensional Modeling: Lead Kimball star schema, fact/dimension design, grain definition, and SCD Type I/II modeling.
• Databricks Optimization: Tune Databricks workloads for cost and performance at scale (cluster sizing, job clusters vs. all-purpose, Photon).
• SQL Expertise: Write and review production-grade transformation logic in both T-SQL (SQL Server) and Spark SQL.
• Data Integrity Enforcement: Enforce referential integrity, deduplication, reconciliation, validation/audit checks, and late-arriving data handling.
• API Integration at Scale: Consume and design around REST APIs at scale (pagination, rate limiting, async job patterns, per-entity fan-out APIs).
• Multi-Source Ingestion: Ingest from external Snowflake sources, vendor exports (flat files, SFTP, APIs), normalizing non-standardized formats during landing.
• High-Volume Pipeline Design: Design pipelines handling millions of records/day, with backpressure, retry, and failure isolation.
• Orchestration & Scheduling: Work with Azure Data Factory, Databricks Workflows, and job scheduling (AutoSys or similar).
• Azure Ecosystem: Work with Azure ecosystem tools (Databricks, ADF, SQL Managed Instance, Key Vault) and Python/PySpark for pipeline/transformation logic.

Required Technical Competencies

• Experience: 10+ years designing data warehouse and lakehouse solutions, with at least 5+ years of hands-on Databricks/Delta Lake and Snowflake/API-driven ingestion experience.
• Dimensional Modeling (Primary): Deep, hands-on Kimball star schema, fact/dimension design, grain definition, SCD Type I/II expertise.
• Medallion Architecture (Primary): Proven experience designing Bronze/Silver/Gold Lakehouse architectures.
• Databricks & Delta Lake (Primary): Strong hands-on expertise with Delta tables, ACID transactions, Z-ordering, optimize/vacuum, and workload tuning for cost/performance at scale.
• SQL (Primary): Expert-level T-SQL and Spark SQL - complex joins, window functions, query tuning, indexing, partitioning strategy.
• Data Integrity: Strong track record enforcing referential integrity, deduplication, reconciliation, validation/audit checks, and late-arriving data handling.
• Databases (In and Out): Deep hands-on knowledge of relational databases (SQL Server preferred), understanding OLTP vs. OLAP design differences.
• API Integration: Experience consuming and designing around REST APIs at scale (pagination, rate limiting, async job patterns).
• Multi-Source Ingestion: Experience integrating Genesys Snowflake, vendor exports (Verint-style flat files, SFTP, APIs), with schema drift tolerance.
• High-Volume Pipelines: Track record designing pipelines handling millions of records/day, with backpressure, retry, and failure isolation design.
• Programming & Tooling: Python or PySpark for pipeline/transformation logic; familiarity with Azure ecosystem (Databricks, ADF, SQL Managed Instance, Key Vault).
• Orchestration: Experience with Azure Data Factory, Databricks Workflows, or equivalent; job scheduling (AutoSys or similar a plus).
• Preferred Domain: Prior experience with Genesys Cloud or Verint data models, contact center/telephony domain knowledge, and parallel-run migration strategies.