AI Quality Engineer
- Toronto, ON, Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Apply by
- Posting language
- English
- Posted
Original job description
Position: AI Quality Engineer
Location: Toronto, ON (2-4 days hybrid)
Contract: Long Term
Job details:
Define and execute the evaluation approach for generative AI solutions, proving that models, agents, and AI-assisted workflows are accurate, safe, reliable, performant, and fit for production controls.
Requires:
10+ years in quality engineering and at least 2 years evaluating AI/ML systems, with strong Python, test automation, API and integration testing, prompt and response test-set design, groundedness/relevance and hallucination assessment, regression, bias/safety, performance, penetration and non-functional testing.
Candidates should be comfortable with GitHub/GitHub Actions, GitHub Copilot, Claude Code, Copilot Studio, Foundry or LLM Gateway patterns, LangSmith, AppInsights, Log Analytics and Dynatrace, and must produce auditable evidence, defect analytics and go/no-go recommendations for architecture, privacy, cyber and risk reviewers.
More about this job
Responsibilities
Define and execute evaluation approaches for generative AI models, agents, and AI-assisted workflows to verify accuracy, safety, reliability, performance, and production readiness. Produce auditable evidence and defect analytics, and provide go/no-go recommendations to architecture, privacy, cybersecurity, and risk reviewers.
Requirements
Requires 10+ years of quality engineering experience and at least 2 years evaluating AI/ML systems, with strong Python, test automation, API, and integration testing skills. Candidates should have experience with AI-specific evaluation and testing, relevant development and monitoring tools, and producing evidence and recommendations for reviewers.
Skills
- Python
- Quality Engineering
- AI/ML Evaluation
- Test Automation
- API Testing
- Integration Testing
- Groundedness and Relevance Assessment
- Hallucination Assessment
- Regression Testing
- Bias and Safety Testing
- Performance Testing
- Penetration Testing
- Non-Functional Testing
- GitHub Actions
- Observability and Monitoring
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Software
- Engineering
- Data & Analytics
- Security & Safety
Keywords
- Generative AI
- AI/ML Systems
- Quality Engineering
- Python
- Test Automation
- API Testing
- Integration Testing
- Prompt Testing
- Response Test Sets
- Groundedness
- Relevance Assessment
- Hallucination Assessment
- Regression Testing
- Bias Testing
- Safety Testing
- Performance Testing
- Penetration Testing
- Non-Functional Testing
- GitHub
- GitHub Actions
- GitHub Copilot
- Claude Code
- Copilot Studio
- Foundry
- LLM Gateway
- LangSmith
- Application Insights
- Log Analytics
- Dynatrace
- Defect Analytics