AI Evaluator/ Quality Engineer
- Toronto, ON, Canada
$100,000 to $110,000 CAD / year · Full-time
Applications open in a new tab.
Job details
- Pay
- $100,000 to $110,000 CAD / year
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
Avanciers is a premier IT Staffing/Consulting organization and we are currently recruiting for a contract role for one of our premier client in Canada for AI Evaluator/ Quality Engineer.
Role: AI Evaluator/ Quality Engineer
Location: Toronto, ON
Duration: Full Time
Compensation Range: $100,000 - $110,000 per year
This is an active job and application deadline is October 30, 2026
(This posting may close earlier if a suitable candidate is found)
Job Description:
- Define and execute the evaluation approach for generative-AI solutions, proving that models,
- agents and AI-assisted workflows are accurate, safe, reliable, performant and fit for production under controls.
- Requires 10+ years in quality engineering and at least 2 years evaluating AI/ML systems, with strong Python,
- test automation, API and integration testing, prompt and response test-set design,
- groundedness/relevance and hallucination assessment, regression, bias/safety, performance,
- penetration and non-functional testing; candidates should be comfortable with GitHub/GitHub
- Actions, GitHub Copilot, Claude Code, Copilot Studio, Foundry or LLM Gateway patterns, LangSmith,
- AppInsights, Log Analytics and Dynatrace, and must produce auditable evidence,
- defect analytics and go/no-go recommendations for architecture, privacy, cyber and risk reviewers.
Artificial Intelligence (AI) Disclosure
- We use AI-enabled tools to assist in screening applications based on job-related criteria.
Our Commitment to Diversity and Inclusion
Avanciers Incorporated is an equal opportunity employer committed to creating an inclusive and accessible workplace. We welcome applications from all qualified individuals regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, or any other protected characteristic. We are committed to providing accessible employment practices that comply with the Accessibility for Ontarians with Disabilities Act (AODA).
Important Information for Applicants
- Only candidates selected for an interview will be contacted
- Proof of qualifications may be requested
- Background checks may be conducted as part of the hiring process
- Avanciers Incorporated does not use Canadian Social Insurance Numbers (SIN) during the application process
Avanciers Incorporated thanks all applicants for their interest; however, only those selected for an interview will be contacted.
Avanciers Incorporated
Follow us on LinkedIn
More about this job
Responsibilities
Define and execute evaluation strategies for generative AI models, agents, and AI-assisted workflows to verify accuracy, safety, reliability, performance, and production readiness. Produce auditable evidence and defect analytics, and provide go/no-go recommendations to architecture, privacy, cybersecurity, and risk reviewers.
Requirements
Requires 10+ years of quality engineering experience and at least 2 years evaluating AI/ML systems, with strong Python, test automation, API, and integration testing skills. Candidates should be familiar with the listed AI development, evaluation, and observability tools and with prompt testing, groundedness, hallucination, regression, bias, safety, performance, penetration, and non-functional testing.
Skills
- Quality Engineering
- AI/ML Evaluation
- Python
- Test Automation
- API Testing
- Integration Testing
- Groundedness and Relevance Assessment
- Hallucination Assessment
- Regression Testing
- Bias and Safety Testing
- Performance Testing
- Penetration Testing
- Non-Functional Testing
- Defect Analytics
- Auditable Evidence and Go/No-Go Recommendations
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Software
- Engineering
- Data & Analytics
- Consulting
Keywords
- Generative AI
- AI/ML Systems
- Quality Engineering
- Python
- Test Automation
- API Testing
- Integration Testing
- Prompt and Response Test Sets
- Groundedness
- Relevance Assessment
- Hallucination Assessment
- Regression Testing
- Bias Testing
- Safety Testing
- Performance Testing
- Penetration Testing
- Non-Functional Testing
- GitHub
- GitHub Actions
- GitHub Copilot
- Claude Code
- Copilot Studio
- Foundry
- LLM Gateway
- LangSmith
- Application Insights
- Log Analytics
- Dynatrace
- Defect Analytics
- Production Readiness