Senior Data Engineer – Capital Markets (Databricks)
Location: Mississauga, Ontario, Canada
Work Model: Hybrid
Experience: 8+ Years
Role Overview
We are seeking a highly skilled Senior Data Engineer with extensive hands-on experience in Databricks, Python/PySpark, SQL, and Enterprise Data Engineering to join our Capital Markets technology team.
The ideal candidate will have strong expertise in designing, developing, and optimizing large-scale data engineering solutions using the Databricks Lakehouse platform. This role requires a hands-on professional capable of building scalable data pipelines, implementing complex data transformations, developing high-performance SQL queries, and processing large volumes of financial data.
The successful candidate will work closely with business analysts, trading teams, risk management professionals, and technology stakeholders to deliver enterprise data solutions supporting trading operations, market data, regulatory reporting, risk management, and other Capital Markets functions.
Candidates with strong Databricks development experience, advanced Python/PySpark and SQL skills, and a background in Capital Markets or Investment Banking will be highly preferred.
Key Responsibilities
- Design, develop, implement, and maintain scalable data engineering solutions using Databricks, Python, PySpark, and SQL.
- Build and optimize enterprise-scale data pipelines and data processing frameworks using the Databricks Lakehouse platform.
- Develop complex data transformation workflows using PySpark, Spark SQL, Python, and advanced SQL programming.
- Design and implement ETL/ELT pipelines to ingest, process, transform, and integrate large volumes of financial and transactional data.
- Develop and maintain scalable data warehouse and lakehouse architectures to support Capital Markets applications and analytical requirements.
- Work extensively with Databricks notebooks, workflows, jobs, and distributed data processing capabilities.
- Implement data processing solutions using Delta Lake, including efficient data ingestion, transformation, storage, and optimization.
- Develop complex SQL queries, stored procedures, views, analytical functions, and data transformation logic.
- Collaborate with business analysts, traders, risk management teams, and other stakeholders to understand business requirements and translate them into technical solutions.
- Design and implement data models supporting financial reporting, business intelligence, analytics, and regulatory compliance.
- Ensure data quality, integrity, accuracy, consistency, security, and governance across enterprise data platforms.
- Identify and resolve performance bottlenecks by optimizing Spark workloads, SQL queries, data pipelines, and distributed processing operations.
- Participate in technical architecture discussions and recommend best practices for Databricks implementation, data engineering, and cloud adoption.
- Support production deployments, troubleshoot data pipeline failures, and manage ongoing enhancements to enterprise data platforms.
- Develop and maintain technical documentation, data mapping specifications, data lineage, and operational support procedures.
- Work with cross-functional development, infrastructure, architecture, and business teams to ensure successful delivery of data engineering initiatives.
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline.
- 8+ years of experience in Data Engineering, Data Warehousing, Big Data Development, or related technology roles.
- Strong hands-on experience working with Databricks in enterprise data engineering environments.
- Advanced programming experience in Python and PySpark.
- Strong SQL development experience, including complex queries, data transformations, and performance optimization.
- Experience designing and implementing enterprise-scale ETL/ELT pipelines and data processing solutions.
- Strong understanding of data warehousing, data modeling, distributed processing, and data integration concepts.
- Experience working within Capital Markets, Investment Banking, or financial services technology environments.
- Understanding of financial products, market data, reference data, trading systems, risk management, securities, fixed income, equities, or derivatives.
- Experience working in Agile development and delivery environments.
- Strong analytical, problem-solving, communication, and collaboration skills.
Must-Have Technical Skills
1. Databricks – Strong Hands-on Experience
- Extensive hands-on experience with the Databricks platform for enterprise data engineering and large-scale data processing.
- Strong knowledge of Databricks architecture, workspace, clusters, notebooks, and compute environments.
- Experience developing and maintaining data engineering workflows using Databricks notebooks and jobs.
- Proficiency in implementing ETL/ELT pipelines using Databricks.
- Strong understanding of Apache Spark and distributed data processing.
- Experience working with Delta Lake, Delta Tables, and Lakehouse architecture.
- Knowledge of data ingestion, transformation, incremental processing, and data optimization techniques.
- Experience optimizing Databricks workloads, Spark jobs, cluster configurations, and processing performance.
- Understanding of data governance, access management, and security within Databricks environments.
- Experience integrating Databricks with cloud storage, databases, and enterprise data platforms.
2. Python / PySpark
- Expert-level hands-on experience developing data engineering solutions using Python.
- Strong proficiency in PySpark for large-scale distributed data processing and transformation.
- Experience developing complex data ingestion, transformation, and processing pipelines.
- Strong understanding of Apache Spark architecture, DataFrames, Spark SQL, and distributed computing.
- Experience implementing data validation, reconciliation, and data quality frameworks using Python.
- Knowledge of API integration, automated data extraction, and workflow orchestration.
- Ability to troubleshoot and optimize Python and PySpark applications.
- Experience handling large and complex datasets in enterprise environments.
3. SQL Development
- Expert-level SQL development experience with strong proficiency in writing, maintaining, and optimizing complex queries.
- Extensive experience with complex joins, analytical functions, window functions, subqueries, stored procedures, and views.
- Strong understanding of query execution plans, indexing strategies, and performance tuning.
- Hands-on experience with Spark SQL and SQL-based data transformation.
- Experience performing data reconciliation, validation, and optimization of high-volume financial datasets.
- Ability to troubleshoot complex queries and improve database and data processing performance.
4. Data Warehousing & Data Modeling
- Extensive experience designing, developing, and implementing enterprise-scale Data Warehouse solutions.
- Strong understanding of relational databases, dimensional modeling, and enterprise data architecture.
- Hands-on experience with Star Schema, dimensional modeling, and data mart architecture.
- Experience designing scalable data models to support reporting, analytics, and financial data processing.
- Strong knowledge of data integration, historical data management, and data lifecycle processes.
- Understanding of modern Lakehouse and enterprise data warehouse architectures.
- Experience implementing data optimization techniques for large-scale analytical workloads.
5. ETL / ELT & Data Integration
- Extensive experience designing, developing, and maintaining enterprise ETL/ELT frameworks.
- Strong understanding of data extraction, transformation, loading, and integration processes.
- Experience integrating structured and unstructured data from multiple enterprise sources.
- Knowledge of batch processing, incremental loading, data synchronization, and automated processing techniques.
- Ability to develop reusable and scalable data integration frameworks.
- Experience monitoring pipelines, troubleshooting processing failures, and improving ETL/ELT performance.
- Strong understanding of data mapping, transformation rules, and data validation procedures.
6. Cloud & Enterprise Data Platforms
- Experience working with cloud-based data engineering and enterprise data platforms.
- Understanding of cloud architecture, data storage, processing, and integration methodologies.
- Experience supporting enterprise data migration, modernization, and cloud adoption initiatives.
- Knowledge of data security, access controls, governance, and compliance requirements.
- Experience integrating Databricks with cloud-based storage and enterprise databases.
Capital Markets Domain Knowledge
- Previous experience working within Capital Markets, Investment Banking, or financial services technology environments.
- Understanding of financial instruments, including equities, fixed income, derivatives, and securities.
- Knowledge of trading systems, market data, reference data, and risk management processes.
- Understanding of front-office, middle-office, and back-office operations.
- Experience supporting financial data processing, regulatory reporting, risk analytics, and trading-related data platforms.
- Ability to collaborate with business and technology stakeholders to translate complex financial requirements into scalable data engineering solutions.
Preferred Qualifications
- Previous experience supporting Capital Markets technology teams within investment banks or major financial institutions.
- Experience implementing Databricks-based Lakehouse solutions in financial services environments.
- Familiarity with Delta Lake, Delta Tables, Unity Catalog, and Databricks Workflows.
- Experience handling large-scale financial datasets and complex data integration requirements.
- Knowledge of enterprise data governance, data lineage, and regulatory data management.
- Strong communication and stakeholder management skills.
