Role: Python Developer – Data Engineering
Location: Toronto, ON (4 days hybrid)
Fulltime
Responsibilities:
- Develop and optimize data manipulation and transformation workflows using pandas and Polars.
- Design, build, and maintain efficient data processing pipelines for large-scale datasets.
- Optimize Python code and data workflows for performance, scalability, and memory efficiency.
- Develop containerized applications using Docker and deploy scalable services using Kubernetes.
- Build and maintain data pipelines integrated with ClickHouse or similar columnar databases for analytical workloads.
- Design and implement event-driven architectures using NATS for asynchronous data processing and messaging.
- Develop distributed computing solutions using Dask to process datasets that exceed single-machine memory limitations.
- Write and maintain comprehensive unit and integration tests using pytest.
- Maintain high code quality, test coverage, reliability, and performance standards.
- Use Git for version control and follow established branching, code review, and collaboration practices.
- Collaborate with data engineers, software developers, DevOps teams, and other stakeholders to deliver reliable data solutions.
- Troubleshoot production issues and support the continuous improvement of data engineering processes.
Requirements:
- Bachelor’s degree in computer science or Diploma in IT with equivalent experience.
Python and Data Processing
- Advanced proficiency in Python.
- Strong experience with pandas and Polars for data manipulation, transformation, and analysis.
- Experience working with large datasets and optimizing code for performance and memory usage.
- Strong understanding of Python development best practices, object-oriented programming, and clean code principles.
Containerization and Orchestration
- Hands-on experience building and managing container images using Docker.
- Experience creating and maintaining multi-container applications.
- Working knowledge of Kubernetes for container orchestration, deployment, scaling, and service management.
Data Infrastructure
- Experience with ClickHouse or a similar columnar database.
- Understanding of OLAP workloads, analytical queries, data modeling, and query optimization.
- Experience designing data pipelines for analytical and reporting use cases.
Messaging and Streaming
- Familiarity with NATS.io and message-driven architectures.
- Understanding of asynchronous processing, event-driven systems, messaging patterns, and distributed workflows.
Testing and Quality Assurance
- Proficiency with pytest.
- Experience writing unit tests, integration tests, and regression tests.
- Understanding of test coverage, mocking, test automation, and quality assurance practices.
Distributed Computing
- Experience with Dask for parallel processing and out-of-core computation.
- Understanding of distributed data processing and techniques for handling datasets that exceed available system memory.
Version Control
- Strong command of Git, including branching strategies, pull requests, code reviews, merging, and conflict resolution.
- Experience collaborating through shared code repositories and following software development best practices.
