Data Engineer
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
- Build Kafka-based streaming and event-driven solutions.
- Write complex SQL queries for data transformation and validation.
- Implement data ingestion from databases, APIs, files, and streaming sources.
- Monitor performance, troubleshoot issues, and ensure data quality, security, and governance.
- Support deployments, testing, upgrades, and production releases.
- PySpark and Apache Spark (Spark SQL, DataFrames)
- Apache Kafka
- Hadoop Ecosystem (HDFS, Hive, YARN)
- Apache NiFi
- SQL
More about this job
Responsibilities
Build Kafka-based streaming and event-driven solutions, write complex SQL queries for data transformation and validation, and implement ingestion from databases, APIs, files, and streaming sources. Monitor performance, troubleshoot issues, ensure data quality, security, and governance, and support deployments, testing, upgrades, and production releases.
Requirements
The posting lists PySpark, Apache Spark (including Spark SQL and DataFrames), Apache Kafka, the Hadoop ecosystem (HDFS, Hive, and YARN), Apache NiFi, and SQL as mandatory skills. No education requirement or specific years of experience are stated; the role is identified as entry level.
Skills
- PySpark
- Apache Spark
- Spark SQL
- DataFrames
- Apache Kafka
- Hadoop
- HDFS
- Hive
- YARN
- Apache NiFi
- SQL
- Data Ingestion
- Data Transformation
- Data Validation
- Data Quality
- Data Governance
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Consulting
Keywords
- PySpark
- Apache Spark
- Spark SQL
- DataFrames
- Apache Kafka
- Hadoop Ecosystem
- HDFS
- Hive
- YARN
- Apache NiFi
- SQL
- Streaming
- Event-Driven Solutions
- Data Ingestion
- Databases
- APIs
- Files
- Data Transformation
- Data Validation
- Performance Monitoring
- Troubleshooting
- Data Quality
- Data Security
- Data Governance
- Deployments
- Testing
- Upgrades
- Production Releases
- IT Services
- IT Consulting