Lead Data Engineer

Lead, Data Engineer

Full-time

Hybrid:** 4 days onsite/week, Monday–Thursday

The Lead, Data Engineering will join the Software Engineering team that leads the delivery of data and application solutions that enable Global Investments (GI) and Corporate teams to analyze opportunities, support investment decision-making, optimize portfolio insights, and strengthen enterprise management capabilities. Reporting to the Manager, Software Engineering, the Lead, Data Engineering works across investment teams including Capital Markets, Infrastructure, Private Equity, Private Capital and Ventures, and Corporate, in close collaboration with Finance, Risk & Compliance teams. As a key member of the Software Engineering team, the Lead, Data Engineering will be hands-on across the full data solution delivery lifecycle, leading the design, development, deployment, and operation of scalable data solutions. This role requires the ability to build trusted relationships with business and technology stakeholders, understand business priorities, define solution architectures, and deliver reliable data products that enable analytics, AI-driven capabilities, and operational excellence. The individual will collaborate closely with Power BI developers, Data Scientists, Investment Professionals, Architects, and Platform Teams, providing technical leadership and integration expertise across one or more application teams.

Responsibilities

• Translate business requirements into scalable, secure, and reliable data solutions.
• Lead the design, development, deployment, and operational support of data platforms, pipelines, and products using modern cloud and engineering practices.
• Define and execute testing strategies, including unit, integration, performance, data quality, and automated regression testing.
• Source, transform, and deliver structured and unstructured data assets for analytics, reporting, operational, and AI-enabled use cases.
• Develop and maintain ELT/ETL pipelines that achieve high levels of reliability, scalability, performance, and security.
• Optimize data pipelines, data models, and platform components for performance, cost efficiency, and operational excellence.
• Implement and maintain CI/CD practices, monitoring, observability, auditing, and operational controls to support production-grade solutions.
• Provision and manage Azure resources, ensuring appropriate security, governance, and cost management.
• Troubleshoot complex technical issues and drive continuous improvement across data solutions and platforms.

Qualifications

• Post-secondary education in Computer Science, Software Engineering, Information Technology, or a related discipline.

Required Skills

• 5+ years delivering complex data engineering and analytics solutions using Microsoft Azure. Experience in the financial services or investment industry is preferred.
• Strong understanding of data engineering, cloud, and software engineering best practices, with experience developing and implementing operational procedures, standards, and documentation.
• Consistent track record of building deep technical relationships. Experience in aligning expectations across various partners.
• Strong analytical and troubleshooting skills, with the ability to trace data lineage, investigate complex workflows, and resolve technical issues with minimal direction.
• Motivated and keen to work in a collaborative environment with a focus on team success, knowledge sharing, continuous improvement, and delivering business value.
• Enterprise-scale experience designing, building, and supporting cloud-based data platforms, data products, and scalable ELT/ETL solutions on Microsoft Azure.
• Strong experience with data architecture, data modeling, data warehousing, and dimensional modeling methodologies such as Kimball and Data Vault 2.0.
• Experience with Snowflake.
• Strong hands-on experience with Python, PySpark, and SQL.
• Deep expertise and Subject Matter Expertise (SME) in Snowflake and dbt, including data modeling, transformation, testing, deployment and performance optimization.
• Experience with data orchestration tools such as Prefect, Dagster, Airflow, ADF, or Synapse.
• Experience designing and developing Azure Functions and Azure Logic Apps for data integration, automation, and event-driven solutions.
• Experience designing data solutions that support AI-enabled use cases, semantic search, or unstructured data processing.
• Proven experience implementing and maintaining CI/CD pipelines using GitHub Actions.
• Experience translating business requirements into scalable technical solutions and collaborating effectively with business and technology stakeholders.
• Experience working in the financial services, asset management, pension, investment management, or similar data-intensive industries.

Preferred Skills

• Experience with Azure Databricks.
• Experience with Microsoft Fabric.
• Experience with Power BI report development, semantic models, and dashboard design.
• Experience with containerization and container orchestration technologies.
• Experience with Infrastructure-as-Code (IaC) and cloud automation practices.