GCP Data Engineering Lead

Job Summary

We are seeking a highly experienced GCP Data Engineering Lead to provide senior technical leadership and delivery management for enterprise-scale cloud data engineering initiatives. The ideal candidate will combine deep hands-on expertise in Google Cloud Platform data pipeline architecture with strong Agile delivery and quality assurance leadership.

This role will be responsible for leading end-to-end pipeline architecture, QA strategy, sprint execution, technical governance, and offshore team coordination across high-volume data ingestion and analytics environments.

Key Responsibilities

Technical Leadership & Architecture

• Lead pipeline architecture design for Netscout and Sandvine ingestion pipelines.

• Design scalable data ingestion and transformation workflows using:

• GCS landing zones

• Google Cloud Dataflow

• Google BigQuery

• Bronze/Silver/Gold Medallion architecture layers

• Define Medallion Architecture data flow standards and transition rules.

• Lead data reconciliation and parity validation strategies across enterprise data platforms.

• Provide hands-on support for complex transformation logic and pipeline optimization.

QA & Data Quality Management

• Define and own end-to-end QA strategy for data pipelines including:

• Test coverage

• Quality gates

• Defect management

• Governance validation

• Architect reusable test templates for batch and streaming pipelines.

• Define data quality specifications for each Medallion layer.

• Review and assure quality of all pipeline and QA deliverables.

Delivery & Team Management

• Lead Agile sprint planning, sprint reviews, and retrospectives.

• Maintain RAID logs (Risks, Assumptions, Issues, Dependencies) and drive mitigation activities.

• Produce weekly delivery status reports and maintain project trackers.

• Coordinate with client engineering teams on environments, access, and technical decisions.

• Manage offshore Data Engineers and Observability Engineers including task allocation and quality oversight.

• Drive mentoring, knowledge transfer, and delivery excellence initiatives.

Required Qualifications

• Advanced experience with Google Cloud Platform data engineering services including:

• Google BigQuery

• Google Cloud Dataflow / Apache Beam

• Pub/Sub

• GCS

• Strong hands-on development expertise in Python and SQL.

• Experience designing QA frameworks for cloud data pipelines using tools such as Great Expectations or dbt tests.

• Strong understanding of Medallion Architecture and enterprise data processing patterns.

• Experience leading Agile delivery processes and technical project governance.

• Strong stakeholder communication and team leadership skills.

Preferred Qualifications

• Advanced Google Cloud Platform certifications.

• Telecom, telemetry, or high-volume streaming data experience.

• Experience with observability and data governance frameworks.

• Prior experience managing offshore delivery teams.

• Strong mentoring and knowledge transfer capabilities.

Key Skills & Competencies

• GCP Data Engineering

• BigQuery & Dataflow

• Apache Beam & Pub/Sub

• Medallion Architecture

• Data Pipeline QA & Governance

• Python & SQL Development

• Agile Delivery Management

• Data Quality & Reconciliation

• Team Leadership & Stakeholder Management