ETL QA

Role Overview

We are looking for an ETL QA Engineer with strong experience in Databricks and PySpark, combined with hands-on exposure to AI-assisted development and prompt engineering. The ideal candidate will be responsible for validating large-scale data pipelines, ensuring data quality, and leveraging tools like GitHub Copilot, GitHub Actions/Runners, and AI agents to improve testing efficiency and coverage. You will also contribute to building and validating AI-powered knowledge bases.

Required Skills & Experience

• Core Technical Skills
  
• • Strong hands-on experience in Databricks (notebooks, jobs, clusters, Delta Lake, SQL).
    
  • Expert-level PySpark skills, including transformations, actions, window functions, and optimization techniques.
    
  • Solid understanding of ETL/ELT concepts, data warehousing, and data modeling (star/snowflake schemas, dimension/fact tables).
    
  • Proven experience in data validation, data quality checks, and reconciliation techniques.
    
  • Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CD pipelines.
    
• AI & Prompt Engineering
  
• • Experience crafting effective prompts for large language models (LLMs) for coding, testing, and documentation.
    
  • Exposure to agent-based architectures or tools (e.g., workflow/agent frameworks that orchestrate multi-step AI tasks).
    
  • Familiarity with AI-based tools for code analysis, test generation, or knowledge management.
    
• Knowledge Base & Documentation
  
• • Experience building AI-driven knowledge bases or documentation systems (e.g., using vector search, embeddings, or LLM-based retrieval).
    
  • Strong documentation skills, with the ability to translate complex data/QA concepts into clear knowledge articles.
    
• Testing & QA Practices
  
• • Strong understanding of QA methodologies: test planning, test design, defect management, and regression testing.
    
  • Experience with automated testing frameworks (Python-based testing libraries such as pytest, unittest, or similar).
    
  • Familiarity with data quality tools/practices (e.g., validation rules, thresholds, anomaly detection).
    

Key Attributes

• Strong analytical and problem-solving skills with a detail-oriented mindset.
  
• Passion for data quality, automation, and continuous improvement.
  
• Ability to work in an agile, fast-paced environment and collaborate across multiple teams like Business and operations team and AD team.
  
• Curiosity and openness to adopting new AI tools and practices for QA.