Lead Data Engineer
- Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Posting language
- English
- Posted
Original job description
Lead Data Engineer
Contract
Ontario
PySpark & Optimization
Role Summary
We are looking for a hands-on Lead Data Engineer to support a proof of concept in Azure Databricks.
The role focuses on converting complex SQL stored procedures into optimized Python and PySpark code. The objective is to demonstrate that the platform can process 100,000 simulation years in minutes rather than hours, while maintaining accurate results.
Key Responsibilities
- Refactor complex T-SQL logic, including pricing rules, layer applications, and reinstatements, into Python and PySpark.
- Replace row-by-row processing with efficient, vectorized DataFrame operations.
- Develop Azure Data Factory pipelines to ingest loss data from Touchstone/RMS into ADLS Gen2.
- Optimize Spark jobs through effective partitioning, memory management, and skew handling.
- Validate the new solution against existing SQL results to ensure accuracy.
- Contribute to performance testing and troubleshooting.
Required Skills
- Advanced Python skills, including Pandas and PySpark.
- Strong T-SQL experience and the ability to analyze complex stored procedures.
- Hands-on experience with Azure Databricks, Delta Lake, and Azure Data Factory.
- Strong knowledge of distributed computing and Spark optimization.
- Experience with partitioning, sharding, broadcast variables, and data-skew management.
- Familiarity with CI/CD and version control for data engineering projects.
• Desirable Skills
- Experience with Monte Carlo simulations or stochastic modelling.
- Knowledge of Parquet and Avro file formats.
- Experience analyzing SQL Server execution plans.
- Background in insurance, risk analytics, or simulation-based modelling.
More about this job
Responsibilities
Refactor complex T-SQL procedures into optimized Python and PySpark, using vectorized DataFrame operations and tuning Spark performance. Build Azure Data Factory pipelines to ingest loss data, validate results against existing SQL outputs, and support performance testing and troubleshooting.
Requirements
Requires advanced Python, Pandas, and PySpark skills; strong T-SQL experience; and hands-on experience with Azure Databricks, Delta Lake, and Azure Data Factory. Candidates should understand distributed computing and Spark optimization, including partitioning, sharding, broadcast variables, and data-skew management; simulation, insurance, and risk analytics experience is desirable.
Skills
- Python
- Pandas
- PySpark
- T-SQL
- Azure Databricks
- Delta Lake
- Azure Data Factory
- Spark Optimization
- Distributed Computing
- Data Partitioning
- Sharding
- Broadcast Variables
- Data Skew Management
- CI/CD
- Version Control
- Monte Carlo Simulation
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Engineering
Keywords
- Python
- Pandas
- PySpark
- T-SQL
- Azure Databricks
- Delta Lake
- Azure Data Factory
- ADLS Gen2
- Touchstone
- RMS
- Spark
- Distributed Computing
- DataFrame Operations
- Spark Optimization
- Partitioning
- Sharding
- Broadcast Variables
- Data Skew Management
- CI/CD
- Version Control
- Monte Carlo Simulation
- Stochastic Modelling
- Parquet
- Avro
- SQL Server Execution Plans
- Insurance
- Risk Analytics
- Simulation-Based Modelling
- Performance Testing