Lead Data Engineer
- Vancouver, BC, Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Apply by
- Posting language
- English
- Posted
Original job description
Key Responsibilities
- Lead the design and development of scalable batch and streaming data pipelines using Java, Spark, and Kafka
- Architect data platforms covering ingestion, transformation, storage, and serving layers
- Build and maintain data services and APIs using Spring Boot
- Define data models, partitioning strategies, and schema management for data lake and warehouse environments
- Own code quality through code reviews, design reviews, and engineering best practices
- Mentor and coach data engineers, and support their technical growth
- Optimize pipeline performance, cost, and JVM/Spark tuning
- Implement data quality checks, monitoring, alerting, and observability
- Drive CI/CD, automated testing, and infrastructure-as-code practices
- Lead root-cause analysis for production data incidents
- Ensure data security, governance, and privacy compliance (including PIPEDA requirements)
- Contribute to estimation, sprint planning, and delivery in an Agile environment
More about this job
Responsibilities
Lead the design, development, and optimization of scalable batch and streaming data pipelines, data platforms, and data services using Java, Spark, Kafka, and Spring Boot. Guide engineering quality and team growth while driving data reliability, security, governance, compliance, and Agile delivery.
Requirements
The role calls for strong expertise in Java, Spark, Kafka, Spring Boot, data architecture, and production pipeline optimization. It also involves technical leadership, mentoring, code and design reviews, observability, CI/CD, infrastructure as code, and compliance with PIPEDA requirements.
Skills
- Java
- Apache Spark
- Apache Kafka
- Spring Boot
- Data Pipeline Design
- Data Platform Architecture
- Data Modeling
- Data Lake and Warehouse Management
- JVM and Spark Tuning
- Data Quality
- Monitoring and Observability
- CI/CD
- Automated Testing
- Infrastructure as Code
- Agile Delivery
- Technical Mentoring
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
- Management & Leadership
Keywords
- Java
- Apache Spark
- Apache Kafka
- Spring Boot
- Batch Data Pipelines
- Streaming Data Pipelines
- Data Ingestion
- Data Transformation
- Data Storage
- Data Serving
- Data Services
- APIs
- Data Modeling
- Partitioning Strategies
- Schema Management
- Data Lakes
- Data Warehouses
- Code Reviews
- Design Reviews
- JVM Tuning
- Data Quality Checks
- Monitoring
- Alerting
- Observability
- CI/CD
- Automated Testing
- Infrastructure as Code
- Data Security
- Data Governance
- PIPEDA