Big Data Developer
- Mississauga, ON, Canada
Salary : CAD 120K + Benefits · Full-time
Applications open in a new tab.
Job details
- Pay
- Salary : CAD 120K + Benefits
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
Role : Big Data Engineer (PySpark & Python)
Location: Mississauga, ON, Canada
Job Type: FTE
Salary : CAD 120K + Benefits
Job Summary
We are looking for an experienced Big Data Engineer with strong expertise in PySpark and Python to design, develop, and optimize scalable data processing solutions. The ideal candidate will have experience working with large datasets, building robust ETL/ELT pipelines, and supporting data integration and transformation initiatives in a cloud or big data environment.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using PySpark and Python.
Build and optimize ETL/ELT workflows to process large volumes of structured and unstructured data.
Perform data cleansing, transformation, aggregation, and validation.
Develop efficient Spark jobs and troubleshoot performance and scalability issues.
Work with SQL, relational databases, and data warehouses to support analytics and reporting.
Implement data quality checks, monitoring, and error-handling mechanisms.
Collaborate with data engineers, architects, analysts, and business stakeholders to deliver data solutions.
Follow best practices for coding standards, documentation, testing, and data security.
Support cloud-based big data platforms and CI/CD processes where applicable.
Required Skills
Strong hands-on experience with Python and PySpark.
Solid understanding of Apache Spark, distributed computing, and big data processing.
Proficiency in SQL and experience with complex queries.
Experience developing ETL/ELT pipelines and data integration solutions.
Knowledge of data modeling, data warehousing, and data lake concepts.
Experience with performance tuning, debugging, and data quality management.
Familiarity with Git, unit testing, and Agile development methodologies.
Preferred Qualifications
Experience with cloud platforms such as AWS, Azure, or GCP.
Exposure to Databricks, Delta Lake, Apache Kafka, or Hadoop.
Experience with orchestration tools such as Airflow or Azure Data Factory.
Knowledge of CI/CD pipelines and automated deployment practices.
Education & Experience
Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
5+ years of experience in data engineering or big data development, with strong hands-on expertise in PySpark and Python.
More about this job
Responsibilities
Design, develop, and optimize scalable data pipelines and ETL/ELT workflows using PySpark and Python to process large structured and unstructured datasets. Perform data transformation and validation, troubleshoot Spark performance, implement data quality and monitoring controls, and collaborate with technical and business stakeholders.
Requirements
Requires a bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, and 5+ years of data engineering or big data development experience. Candidates should have strong hands-on Python and PySpark expertise, knowledge of Spark and distributed computing, proficiency in SQL, and experience with data pipelines, data warehousing, performance tuning, and data quality.
Skills
- Python
- PySpark
- Apache Spark
- Distributed Computing
- SQL
- ETL/ELT Pipelines
- Data Integration
- Data Modeling
- Data Warehousing
- Data Lakes
- Performance Tuning
- Data Quality Management
- Git
- Unit Testing
- Agile Development
- Cloud Platforms
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- Big Data Engineering
- Python
- PySpark
- Apache Spark
- Distributed Computing
- SQL
- ETL/ELT
- Data Pipelines
- Data Integration
- Data Transformation
- Data Cleansing
- Data Validation
- Data Quality
- Data Modeling
- Data Warehousing
- Data Lakes
- Performance Tuning
- Git
- Unit Testing
- Agile
- AWS
- Azure
- GCP
- Databricks
- Delta Lake
- Apache Kafka
- Hadoop
- Airflow
- Azure Data Factory
- CI/CD