Closed listing
Senior Data Engineer (Databricks)
- Toronto, ON, Canada
Contract
This opening has closed.
The original listing is kept below for reference.
Job details
- Job type
- Contract
- Closing date
- Posting language
- English
- Posted
Similar jobs hiring now
Search current openingsOriginal job description
Archived listing. The details below describe a past opening.
Title: Senior Data Engineer (Databricks)
Location: Toronto, ON - Hybrid
Experience: 8+ years
Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
- Build and optimize data solutions using Databricks, PySpark, and Spark SQL.
- Implement data ingestion frameworks from various source systems including databases, APIs, files, and streaming platforms.
- Develop and maintain data lakehouse architectures leveraging Databricks best practices.
- Design and implement Medallion Architecture (Bronze, Silver, Gold layers) for enterprise data platforms.
- Collaborate with business stakeholders, data scientists, architects, and application teams to understand data requirements.
- Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
- Implement data quality, governance, security, and monitoring frameworks.
- Support AI/ML initiatives by preparing and engineering datasets for model development and deployment.
- Develop CI/CD pipelines and automate deployment processes for data engineering workloads.
- Participate in architecture reviews and establish data engineering standards and best practices.
- Mentor junior engineers and provide technical leadership across projects.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, or related field.
- 8+ years of experience in Data Engineering and Data Platform development.
- Strong hands-on experience with Databricks and Apache Spark.
- Proficiency in Python, PySpark, and SQL.
- Experience with cloud platforms such as:
o Microsoft Azure (preferred)
o AWS
o Google Cloud Platform
- Experience with Delta Lake, Unity Catalog, and Databricks workflows.
- Strong understanding of Data Lake, Data Warehouse, and Lakehouse architectures.
- Experience working with large-scale distributed data processing systems.
- Knowledge of data modeling, partitioning, indexing, and query optimization techniques.
- Experience with Git, CI/CD pipelines, and DevOps practices.
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional Certification.
- Experience with Azure Data Factory, Synapse Analytics, or equivalent cloud-native services.
- Experience with streaming technologies such as Kafka, Event Hubs, or Spark Streaming.
- Exposure to AI/ML platforms, MLOps, and Generative AI solutions.
- Experience working in Insurance, Financial Services, or Enterprise Digital Transformation programs.
More about this job
Responsibilities
Design, build, and optimize scalable data pipelines and lakehouse solutions using Databricks, Spark, and cloud technologies, including implementing Medallion Architecture and data quality, governance, security, and monitoring frameworks. Collaborate with stakeholders and AI/ML teams, automate deployments, establish engineering standards, and mentor junior engineers.
Requirements
Requires a bachelor's or master's degree in a relevant field, at least 8 years of data engineering experience, and strong hands-on skills in Databricks, Apache Spark, Python, PySpark, and SQL. Candidates should also have cloud platform, Delta Lake, Unity Catalog, distributed processing, data architecture, optimization, Git, and CI/CD experience; Databricks certification and streaming or AI/ML exposure are preferred.
Skills
- Databricks
- Apache Spark
- Python
- PySpark
- SQL
- Delta Lake
- Unity Catalog
- ETL/ELT Pipelines
- Lakehouse Architecture
- Medallion Architecture
- Data Modeling
- Cloud Platforms
- CI/CD
- Data Governance
- Streaming Data
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Engineering
Keywords
- Databricks
- Apache Spark
- PySpark
- Python
- SQL
- ETL/ELT
- Data Ingestion
- Data Lake
- Data Warehouse
- Lakehouse Architecture
- Medallion Architecture
- Delta Lake
- Unity Catalog
- Databricks Workflows
- Microsoft Azure
- AWS
- Google Cloud Platform
- Azure Data Factory
- Azure Synapse Analytics
- Kafka
- Event Hubs
- Spark Streaming
- Data Modeling
- Data Governance
- CI/CD
- DevOps
- MLOps
- Generative AI
- Databricks Certified Data Engineer
- Financial Services