Data Engineer
- Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
Key Responsibilities
- Design, develop, and maintain scalable data pipelines.
- Build and manage ETL/ELT workflows.
- Extract, transform, and load data from multiple sources.
- Develop data processing solutions using Python, SQL, and PySpark.
- Work with structured and unstructured data.
- Design and maintain data warehouses and data lakes.
- Develop automated workflows using Apache Airflow.
- Implement real-time data processing using Apache Kafka.
- Integrate data from APIs, databases, and external systems.
- Ensure data quality, accuracy, and consistency.
- Monitor, troubleshoot, and optimize data pipelines.
- Implement data validation, governance, and security.
- Collaborate with Data Scientists, Data Analysts, and BI teams.
Preferred Skills
- Strong knowledge of Python, SQL, and PySpark.
- Experience with Apache Spark and Hadoop.
- Knowledge of ETL/ELT processes.
- Hands-on experience with Apache Airflow.
- Knowledge of Apache Kafka and real-time data processing.
- Experience with MySQL, PostgreSQL, and MongoDB.
- Knowledge of Snowflake, Databricks, Redshift, or BigQuery.
- Experience with AWS, Azure, or GCP cloud platforms.
- Knowledge of AWS Glue and Azure Data Factory.
- Understanding of Data Warehousing and Data Modelling.
- Experience with Data Lakes and Lakehouse architecture.
- Knowledge of Git, GitHub, and CI/CD pipelines.
- Familiarity with Docker and Kubernetes.
- Knowledge of Linux and Shell scripting.
- Strong problem-solving and analytical skills.
More about this job
Responsibilities
Design, build, and maintain scalable data pipelines and ETL/ELT workflows that integrate data from multiple sources, including APIs, databases, and external systems. Develop and optimize batch and real-time data processing, data warehouses, and data lakes while ensuring data quality, governance, and security.
Requirements
The posting prefers strong skills in Python, SQL, and PySpark, along with experience in data processing frameworks, ETL/ELT, workflow orchestration, and real-time streaming. It also lists experience with databases, cloud platforms, data warehousing and lakehouse architectures, CI/CD, containers, Linux, and shell scripting, as well as strong analytical and problem-solving skills.
Skills
- Python
- SQL
- PySpark
- Apache Spark
- Hadoop
- ETL/ELT
- Apache Airflow
- Apache Kafka
- MySQL
- PostgreSQL
- MongoDB
- Snowflake
- Databricks
- Cloud Platforms
- Data Warehousing
- Data Modeling
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- Data Engineering
- Data Pipelines
- ETL/ELT
- Python
- SQL
- PySpark
- Apache Spark
- Hadoop
- Structured Data
- Unstructured Data
- Data Warehousing
- Data Lakes
- Lakehouse Architecture
- Apache Airflow
- Apache Kafka
- Real-Time Data Processing
- APIs
- MySQL
- PostgreSQL
- MongoDB
- Snowflake
- Databricks
- Amazon Redshift
- Google BigQuery
- AWS
- Azure
- Google Cloud Platform
- AWS Glue
- Azure Data Factory
- Data Modeling