We are hiring for a Lead Data Engineer. The role focuses on production support and management of data engineering pipelines, including data ingestion, processing, monitoring, and incident management.
A strong fit will have 8+ years of production data engineering experience with GCP, CDC pipelines, Java, and data processing technologies.
What you bring
- 8+ years of experience supporting and managing data engineering pipelines in production environments.
- Strong experience with Change Data Capture using Debezium and file-based ingestion using CSV.
- Hands-on experience with GCP Pub/Sub, Dataflow (Apache Beam), and Datastream.
- Experience working with Cloud SQL (PostgreSQL) environments.
- Experience managing staging and curated data layers for downstream consumption.
- Strong understanding of production monitoring, alerting, and incident management processes.
- Experience with API-based integrations within data pipelines.
- Strong Java programming experience for data processing and support activities.
What you'll do
- Provide end-to-end production support for data ingestion and processing pipelines.
- Monitor and manage CDC pipelines and batch ingestion processes for multiple source systems.
- Handle incidents, perform root cause analysis, and implement preventive fixes.
- Work with GCP Pub/Sub topics, subscriptions, DLQs, and alerting mechanisms.
- Support staging and transformed data processing layers.
- Perform production deployments and coordinate testing across lower environments.
- Troubleshoot and resolve data, performance, and pipeline failures within SLAs.
- Serve as technical lead for AMS support and coordinate day-to-day operations.
- Own incident communication, escalation, and stakeholder updates.
- Maintain operational runbooks and provide guidance to L2/L3 support teams.
- Identify recurring issues and drive continuous improvements for system stability and reliability.
