Sr. Data Test Engineer (PySpark , ETL)

Position Name – Sr. Data Test Engineer

Type of hiring – Fulltime

candidates along with these questionnaires:

1. How do you validate data between source and target using SQL?
2. If source has 10,000 records and target has 9,800, how will you find missing records?
3. What kind of PySpark code have you written for data validation?
4. How do you check duplicate or null values in PySpark?
5. Have you automated any ETL Validation using Python or PySpark? What did you build?
6. What exactly have you done in Databricks?
7. What is your role in Azure data factory pipeline?
8. What data quality (DQ) checks do you usually perform?
9. How do you validate source-to-target mapping?
10. In your last project, what was one data issue you found and how did you debug it?

Job Description:

We are seeking a Senior Data Engineer with strong Quality Engineering (SDET) experience. The role is primarily focused on building and validating data solutions using Databricks, Python, and Azure cloud technologies, while also owning automated testing, end-to-end validation, and quality assurance across data pipelines, APIs, and backend systems.

Key Responsibilities & Requirements:

• 5+ Years of experience in Data Engineering, SDET, or Quality Engineering.
• Strong hands-on experience with Databricks and PySpark.
• Develop and validate data pipelines and ETL/ELT processes.
• Implement data quality and data validation practices.
• Perform REST API test automation using Karate or similar tools.
• Strong expertise in end-to-end testing.
• Experience with Microsoft Azure.
• Knowledge of Kafka and asynchronous backend testing.
• Experience with Databricks testing and data validation frameworks.

Nice to Have:

• Docker and Kubernetes.
• E-commerce domain experience.
• Git, GitHub, and Jira.