Senior Software Quality Engineer – AI
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks. The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery. The position is full-time and onsite in Toronto, ON.
Key Requirements
- 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation.
- 5+ years of programming experience with Java, Python, Groovy, or similar languages.
- Strong experience building and maintaining test automation frameworks.
- Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured.
- Experience with API, database, integration, regression, and microservices testing.
- Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter.
- Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications.
- Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias, and response consistency.
- Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing.
- Strong analytical, problem-solving, communication, and documentation skills.
Key Responsibilities
- Develop and execute comprehensive quality strategies, test plans, test cases, and test data.
- Design, develop, and maintain scalable test automation frameworks.
- Create automated tests for AI/ML and Generative AI features.
- Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency.
- Test APIs, microservices, databases, integrations, data pipelines, and third-party services.
- Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage.
- Integrate automated testing and AI quality checks into CI/CD pipelines.
- Identify, document, prioritize, and track defects through resolution.
- Analyze test coverage, quality metrics, AI evaluation results, and production feedback.
- Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.
More about this job
Responsibilities
Develop and execute comprehensive quality strategies and scalable automation frameworks for AI-enabled software products. Evaluate AI responses for accuracy and reliability while integrating quality checks into CI/CD pipelines.
Requirements
Requires over 5 years of experience in Quality Engineering and programming with Java or Python. Must have hands-on experience testing AI/ML and Generative AI applications along with proficiency in automation tools like Selenium or Playwright.
Skills
- Quality Engineering
- Test Automation
- Java
- Python
- Groovy
- Selenium
- Playwright
- Cucumber
- RestAssured
- API Testing
- Microservices Testing
- AI/ML Testing
- Generative AI
- LLM Evaluation
- CI/CD
- Agile
Visa sponsorship
Not detected in the job text
Categories
- Software
- Technology
- Data & Analytics
- Engineering
Keywords
- AI/ML
- Generative AI
- LLM
- Prompt Evaluation
- Hallucination Detection
- Toxicity
- Bias
- JUnit
- TestNG
- ReadyAPI
- qTest
- JMeter
- BDD
- TDD
- ATDD
- Risk-based Testing
- Data Pipelines
- Microservices
- Regression Testing
- Integration Testing
- Data Infrastructure
- Analytics