Lead Software Engineer
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Experience
- Years of Experience: 7-10 Years
- Posting language
- English
- Posted
Original job description
Location: PCS CA~TORONTO
Years of Experience: 7-10 Years
Job Summary: We are seeking a highly skilled Databricks Consultant with strong hands-on experience in Databricks and Apache Spark (PySpark/Scala). The ideal candidate will have a solid background in SQL, data transformation techniques, and cloud platforms (Azure/AWS/GCP), with a particular emphasis on Azure. This role involves building and optimizing ETL pipelines, ensuring data governance, and implementing best practices in data modeling and performance tuning.
Responsibilities
- Design and develop ETL pipelines using Azure Databricks and Delta Lake, including batch processing (autoloader) and Spark structured streaming.
- Implement data transformation techniques and ensure data quality throughout the pipeline.
- Work with data warehousing concepts and develop slowly changing dimensions (SCD1 and SCD2) on dimension tables.
- Build and maintain change data capture (CDC) pipelines.
- Utilize Lakehouse federation to create foreign catalogs for accessing external data sources.
- Optimize Spark jobs through effective data partitioning and liquid clustering.
- Collaborate with cross-functional teams to ensure data governance and security practices are adhered to.
- Strong hands-on experience with Databricks and Apache Spark (PySpark/Scala).
- Proficient in SQL and data transformation techniques.
- Experience with ETL tools and data pipeline development.
- Strong Azure cloud background.
- Understanding of data warehousing concepts.
- Strong problem-solving and analytical skills.
- Hands-on experience with Unity Catalog.
- Exposure to CI/CD pipelines and DevOps practices.
- Familiarity with data governance and security practices.
- Experience in building and optimizing data models in Spark.
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- 7-10 years of relevant experience in data engineering or backend development.
More about this job
Responsibilities
Design, build, and optimize ETL and streaming pipelines using Azure Databricks, Delta Lake, and Apache Spark, including CDC, data quality, and slowly changing dimensions. Apply data governance and security practices, use Lakehouse federation and Unity Catalog, and improve Spark performance through partitioning and liquid clustering.
Requirements
Requires 7–10 years of relevant data engineering or backend development experience, strong hands-on Databricks and Apache Spark skills, proficiency in SQL and data transformation, and a strong Azure background. A bachelor's degree in Computer Science, Information Technology, or a related field is required; CI/CD and DevOps experience are preferred.
Skills
- Databricks
- Apache Spark
- PySpark
- Scala
- SQL
- Azure
- ETL Pipeline Development
- Data Transformation
- Data Warehousing
- Unity Catalog
- Delta Lake
- Spark Structured Streaming
- Change Data Capture
- Data Governance
- Data Modeling
- Performance Tuning
Visa sponsorship
Not detected in the job text
Categories
- Software
- Data & Analytics
- Technology
- Consulting
Keywords
- Databricks
- Apache Spark
- PySpark
- Scala
- SQL
- Azure
- AWS
- GCP
- ETL
- Data Pipelines
- Delta Lake
- Autoloader
- Spark Structured Streaming
- Data Transformation
- Data Quality
- Data Warehousing
- Slowly Changing Dimensions
- SCD1
- SCD2
- Change Data Capture
- Lakehouse Federation
- Foreign Catalogs
- Data Partitioning
- Liquid Clustering
- Unity Catalog
- CI/CD
- DevOps
- Data Governance
- Data Security
- Data Modeling