Big Data Engineer (PySpark / Spark / Kafka)
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
Work Mode: Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities- Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
- Build and maintain batch and real-time data processing solutions.
- Develop and support Kafka-based streaming applications and event-driven architectures.
- Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
- Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
- Implement data ingestion solutions from databases, APIs, files, and streaming sources.
- Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
- Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
- Support platform upgrades, deployments, testing, certification, and production releases.
- Ensure data quality, governance, security, and operational excellence across data platforms.
- PySpark
- Apache Spark (Spark SQL, DataFrames)
- Apache Kafka
- Hadoop Ecosystem (HDFS, Hive, YARN)
- Apache NiFi
- SQL
- Python
- Spark Streaming
- Airflow / Oozie
- Hive
- Scala
- Jenkins, Bitbucket, Git
- JIRA, Confluence
- Cloud Platforms (GCP/AWS/Azure)
- Data Warehousing Concepts and Dimensional Modeling
More about this job
Responsibilities
Design, develop, and optimize scalable batch and real-time data pipelines and Kafka-based streaming applications using PySpark, Spark, Hadoop, and Apache NiFi. Build and maintain ETL/ELT workflows, troubleshoot and improve pipeline performance, and support data quality, governance, security, deployments, and production releases.
Requirements
The role requires hands-on expertise in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, Apache NiFi, SQL, and Python. It also calls for experience with large-scale data ingestion and transformation, batch and real-time processing, and collaboration on enterprise data platforms; preferred skills include streaming, workflow orchestration, cloud platforms, and data warehousing.
Skills
- PySpark
- Apache Spark
- Spark SQL
- DataFrames
- Apache Kafka
- Hadoop
- HDFS
- Hive
- YARN
- Apache NiFi
- SQL
- Python
- Spark Streaming
- Airflow
- Oozie
- Scala
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Engineering
Keywords
- PySpark
- Apache Spark
- Spark SQL
- DataFrames
- Apache Kafka
- Hadoop
- HDFS
- Hive
- YARN
- Apache NiFi
- SQL
- Python
- Spark Streaming
- Airflow
- Oozie
- Scala
- Jenkins
- Bitbucket
- Git
- JIRA
- Confluence
- GCP
- AWS
- Azure
- ETL/ELT
- Data Ingestion
- Real-Time Processing
- Data Warehousing
- Dimensional Modeling
- Data Governance