Cloud Data Engineer (GCP and AWS)
Our Challenge
We are seeking a Cloud Data Engineer to design, develop, and maintain scalable data solutions on AWS and Google Cloud Platform. The successful candidate will build reliable data ingestion and processing pipelines, automate workflows, and support efficient deployment through modern DevOps practices.
Responsibilities:
· Develop cloud data solutions using AWS Athena, Amazon S3, AWS Glue Data Catalog, and Google BigQuery.
· Design and implement API-based data ingestion and integration processes.
· Consume REST APIs, extract data, and transform it for analytical use.
· Develop batch, near-real-time, and change data capture (CDC) ingestion patterns.
· Build and maintain ETL/ELT data pipelines.
· Use Apache Airflow for workflow scheduling and pipeline orchestration.
· Manage pipeline dependencies, monitoring, alerts, error handling, and recovery.
· Use GitHub for source control and collaborative development.
· Develop and maintain CI/CD pipelines and release automation.
· Support deployment processes and promote consistent DevOps practices across data engineering initiatives.
Skills:
· 6+ years experience Developing cloud data solutions using AWS Athena, Amazon S3, AWS Glue Data Catalog, and Google BigQuery.
· Design and implement API-based data ingestion and integration processes.
· Consume REST APIs, extract data, and transform it for analytical use.
· Develop batch, near-real-time, and change data capture (CDC) ingestion patterns.
· Build and maintain ETL/ELT data pipelines.
· Use Apache Airflow for workflow scheduling and pipeline orchestration.
· Manage pipeline dependencies, monitoring, alerts, error handling, and recovery.
· Use GitHub for source control and collaborative development.
· Develop and maintain CI/CD pipelines and release automation.
· Support deployment processes and promote consistent DevOps practices across data engineering initiatives.
