Big Data Developer
- Mississauga, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
Develop and maintain scalable data pipelines using PySpark and Spark SQL for processing large datasets efficiently. Write clean, reusable, and optimized code in Python for data manipulation, analysis, and automation tasks. Design and implement ETL workflows to extract, transform, and load data from various structured and unstructured sources. Collaborate with data engineers, analysts, and stakeholders to understand data requirements and deliver solutions. Optimize Spark jobs for performance tuning, resource utilization, and minimizing execution time. Work with distributed computing frameworks to process and analyze big data in a cloud or on-premises environment. Develop and maintain unit tests to ensure the accuracy and reliability of data pipelines and transformations. Utilize Spark SQL for querying and managing large datasets stored in distributed systems like Hadoop or cloud storage. Monitor and troubleshoot data pipeline issues, ensuring reliability and timely delivery of data. Stay updated with the latest advancements in PySpark, Spark SQL, and big data technologies to improve existing systems.
More about this job
Responsibilities
Develop, maintain, and optimize scalable data pipelines and ETL workflows using PySpark, Spark SQL, and Python to process large datasets from structured and unstructured sources. Collaborate with engineers, analysts, and stakeholders to meet data requirements, test and troubleshoot pipelines, and ensure reliable, timely data delivery.
Requirements
The role calls for practical experience with PySpark, Spark SQL, Python, ETL workflows, and distributed data processing in cloud or on-premises environments. It also requires the ability to optimize Spark jobs, develop unit tests, and monitor and troubleshoot data pipelines; no formal education requirement is stated.
Skills
- PySpark
- Spark SQL
- Python
- Data Pipelines
- ETL
- Data Transformation
- Big Data Processing
- Distributed Computing
- Performance Tuning
- Unit Testing
- Data Analysis
- Data Automation
- Hadoop
- Cloud Storage
- Troubleshooting
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- PySpark
- Spark SQL
- Python
- Big Data
- Data Pipelines
- ETL
- Structured Data
- Unstructured Data
- Data Manipulation
- Data Analysis
- Automation
- Distributed Computing
- Spark Job Optimization
- Performance Tuning
- Resource Utilization
- Unit Testing
- Hadoop
- Cloud Storage
- Cloud Computing
- On-Premises Environments
- Data Engineering
- Pipeline Monitoring
- Troubleshooting
- Data Reliability
- IT Services
- IT Consulting