Data Engineer
- Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
Key Responsibilities
- Design, develop, and maintain scalable data pipelines for batch and streaming data.
- Extract, transform, and load data from multiple sources using ETL/ELT processes.
- Develop and optimize SQL queries for data processing and analysis.
- Build data integration workflows using Python and relevant data engineering tools.
- Design and maintain data warehouses, data lakes, and data models.
- Ensure data quality, accuracy, consistency, and reliability.
- Monitor pipeline performance and troubleshoot data processing issues.
- Collaborate with Data Analysts, Data Scientists, and software development teams.
- Implement data validation, monitoring, logging, and error-handling mechanisms.
- Optimize data storage, processing performance, and infrastructure costs.
- Document data pipelines, workflows, and technical processes.
- Follow data security, access control, and governance practices.
Required Skills
- Strong knowledge of SQL, including joins, subqueries, CTEs, and window functions.
- Programming knowledge in Python.
- Understanding of ETL/ELT pipelines and data integration concepts.
- Knowledge of relational databases such as MySQL, PostgreSQL, or SQL Server.
- Understanding of data warehousing, data modeling, and database design.
- Familiarity with file formats such as CSV, JSON, and Parquet.
- Knowledge of data quality checks and pipeline troubleshooting.
- Understanding of distributed data processing concepts.
Preferred Skills
- Experience with Apache Spark or PySpark.
- Familiarity with Apache Airflow or other workflow orchestration tools.
- Knowledge of cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
- Experience with cloud data warehouses such as Snowflake, BigQuery, or Amazon Redshift.
- Familiarity with Apache Kafka for streaming data.
- Knowledge of Databricks and Delta Lake.
- Understanding of Docker, Git, and CI/CD pipelines.
- Familiarity with data governance, metadata management, and data security.
More about this job
Responsibilities
Design, build, and maintain scalable batch and streaming data pipelines, integration workflows, warehouses, lakes, and data models. Ensure data quality and security, monitor and troubleshoot pipeline performance, optimize processing and costs, document technical workflows, and collaborate with analytics and software teams.
Requirements
Requires strong SQL and Python skills, knowledge of ETL/ELT, relational databases, data warehousing, data modeling, common file formats, data quality, and distributed processing. Preferred qualifications include experience with Spark, Airflow, cloud platforms and warehouses, Kafka, Databricks, Delta Lake, Docker, Git, CI/CD, and data governance.
Skills
- SQL
- Python
- ETL/ELT
- Data Pipelines
- Data Integration
- Relational Databases
- Data Warehousing
- Data Modeling
- Data Quality
- Distributed Data Processing
- Apache Spark
- Apache Airflow
- Cloud Platforms
- Cloud Data Warehouses
- Apache Kafka
- Databricks
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- SQL
- Python
- ETL/ELT
- Batch Processing
- Streaming Data
- Data Pipelines
- Data Integration
- Data Warehouses
- Data Lakes
- Data Modeling
- MySQL
- PostgreSQL
- SQL Server
- CSV
- JSON
- Parquet
- Distributed Data Processing
- Apache Spark
- PySpark
- Apache Airflow
- AWS
- Microsoft Azure
- Google Cloud Platform
- Snowflake
- BigQuery
- Amazon Redshift
- Apache Kafka
- Databricks
- Delta Lake
- Docker