Data Engineer
- Halifax, NS, Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Apply by
- Posting language
- English
- Posted
Original job description
Key Requirements:We are seeking a skilled Data Engineer to design, deploy, and manage robust data pipelines across hybrid cloud environments. The ideal candidate will combine expertise in Python, AWS services, and enterprise cloud data platforms to build scalable, secure solutions for structured and unstructured data at enterprise scale.
Key Responsibilities:
Design and implement complex ETL/ELT workflows for diverse data sources both on-prem and Cloud platform (AWS)
Build and optimize data lake architectures on Cloudera Data Platform (CDP)
Develop data models in Snowflake cloud warehouse and CDP environments
Implement CI/CD pipelines using Helios for data infrastructure automation
Collaborate with analysts to enable advanced analytics on data lake assets
Ensure security/governance compliance for sensitive data at scale
Technical Expertise Required:
Python: Advanced scripting for data processing, API integration, and automation
AWS : Amazon S3, AWS Glue
Containerization: Docker image management; orchestration via Kubernetes/OCP4
Data Platforms:
Cloudera Data Platform (CDP) for Hadoop/Spark ecosystems
Snowflake cloud data warehouse architecture
Storage Solutions: Data lake design (S3/ADLS), structured/unstructured data optimization
Pipeline Tools: Airflow, Kafka, Helios CI/CD
Version Control: GitHub collaboration
Cloud Services: AWS/Azure storage integration with CDP/Snowflake
Soft Skills:
Problem-solv
ing for hybrid cloud data challenges
Cross-functional collaboration with data scientists and analysts
Clear communication of complex technical solutions
Nice-to-Have Qualifications:
Certifications: Cloudera CCP, SnowPro, Red Hat OpenShift
Experience with streaming platforms (Flink, Kafka Streams)
MLOps patterns for model deployment
More about this job
Responsibilities
Design, implement, and optimize ETL/ELT pipelines and data lake architectures across on-premises and hybrid cloud environments, using AWS, Cloudera Data Platform, and Snowflake. Automate data infrastructure with CI/CD, support analytics collaboration, and ensure security and governance for sensitive data.
Requirements
The role requires advanced Python skills and experience with AWS services, Cloudera Data Platform, Snowflake, and enterprise-scale data pipelines. Candidates should also have experience with containerization and orchestration, CI/CD and pipeline tools, cloud storage integration, and collaboration across technical teams.
Skills
- Python
- AWS
- Amazon S3
- AWS Glue
- Cloudera Data Platform
- Snowflake
- Docker
- Kubernetes
- OpenShift
- Airflow
- Kafka
- Helios
- CI/CD
- GitHub
- ETL/ELT
- Data Lake Architecture
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- Python
- AWS
- Amazon S3
- AWS Glue
- Cloudera Data Platform
- Hadoop
- Spark
- Snowflake
- Docker
- Kubernetes
- OpenShift
- Airflow
- Kafka
- Helios
- CI/CD
- GitHub
- ETL/ELT
- Data Pipelines
- Data Lakes
- Data Modeling
- Hybrid Cloud
- Azure
- ADLS
- Structured Data
- Unstructured Data
- Data Governance
- Flink
- Kafka Streams
- MLOps