Find work in Canada.

A simpler way to find your next job.

Jobs in Canada

Email me new jobs Save this search

Added 1 month ago
PC
Salesforce Technical Manager
Pwc Canada
Toronto, ON · Vancouver, BC
- Full-time
Experience with: AI/ML model integration • Python or Java • SQL/database design • AWS/Azure cloud platforms • Omnistudio (FlexCards, DataRaptors) • Industries CPQ • Mulesoft APIs • Platform…
Added 1 month ago
Transformation Engineer - Software Development, AI & Quality Engineering
Fitch Group
Toronto, ON
Fitch Group is seeking a hands-on Transformation Engineer with strong expertise in software development, AI-enabled engineering, and modern quality practices. This role will focus on…
Added 2 months ago
BP
Data Scientist
BNP Paribas
Montreal, QC
- Full-time
…and engineering excellence and are excited to work directly with our internal clients to help them reach their AI ambitions, this is the perfect job for you! You’ll join a stimulating work…
Added 2 months ago
I
Lead Applied AI and Data Scientist | IFS Copperleaf
IFS
Vancouver, BC
- Full-time
IFS is a billion-dollar revenue company with 7000+ employees on all continents. Our leading AI technology is the backbone of our award-winning enterprise software solutions, enabling our…
Added 2 months ago
TR
AI Engineer, Product Analytics
Thomson Reuters
Toronto, ON · Canada, Toronto, Ontario
- Full-time
Proficiency in Python and strong SQL, with a solid grounding in programming concepts, design patterns, SDLC principles, and unit testing.
Added 2 months ago
A(
Machine Learning Resident - Client: Fillip Fleet
Amii (Alberta Machine Intelligence Institute)
Edmonton, AB
- Full-time
This is a paid Residency that will be undertaken over a twelve-month period with the potential to be hired by our client, Fillip Fleet, afterwards (note: at the discretion of the client).…
Added 2 months ago
A
Applied Scientist, Tax Engine
Amazon
Vancouver, BC
- Full-time
…transactional distributed systems in the world? How about working with customers and peers from the entire range of Amazon's business and Science on complex real world problems, backed with…
Added 2 months ago
SQ
AI Engineer (GPU & LLM Optimisation) – Canada / UAE / Remote
SuperQ Quantum Computing
Remote, Canada
- $90,000 to $120,000 CAD / year
- Full-time
- Remote
Cross-functional Deployment: Collaborate with backend and platform engineering teams to deploy highly scalable, low-latency AI endpoints within our Kubernetes-based infrastructure.
Added 2 months ago
LT
Software Engineer
Larus Technologies
Ottawa, ON
- Full-time
5+ years of direct experience in design, development, deployment, and verification of software, with at least 2 years in Java and 2 years in python;
Added 2 months ago
CM
Machine Learning Engineer
Clarius Mobile Health
Toronto, ON
- Temporary
Mastery in Python development with Unix/Linux
Added 2 months ago
IA
Machine Learning Engineer
Invision AI
Remote, Canada
- Full-time
- Remote
…an applied engineering position. We are looking for someone with a track record of building, shipping, and maintaining production ML…
Added 2 months ago
AM
AI and Automation Software Engineer
Advanced Micro Devices, Inc
Vancouver, BC
- Full-time
Proficiency with one or more of the following languages: Python, C, C++ programming, JavaScript.
Added 2 months ago
Principal Data and Applied Scientist
SAP
Montreal, QC
Strong Python and SQL skills with production-grade development practices; experience with ML libraries such as PyTorch, TensorFlow, or scikit-learn.
Added 2 months ago
Physical AI Engineer
EY
Toronto, ON
Strong Python software engineering skills with hands-on experience developing AI/ML, robotics, automation, or system-integration solutions.
Added 2 months ago
R
Senior Principal Software Engineer, Feature Store
Redis
Remote, Canada
- Full-time
- Remote
…status online today, you’re welcome.) At Redis, you’ll work with the fastest, simplest technology in the business - whether you’re building it, telling its story, or selling it to our…
Added 2 months ago
TA
Knowledge Graph Analyst
Tiger Analytics Inc.
Toronto, ON
- Full-time
Additional Graph Methods: Hands-on experience with Python (NetworkX), RDF/SPARQL, or SQL Graph databases is a plus.
Added 2 months ago
Principal Data and Applied Scientist
SAP
Montreal, QC
Strong Python and SQL skills with production-grade development practices; experience with ML libraries such as PyTorch, TensorFlow, or scikit-learn.
Added 2 months ago
PN
Senior Full-Stack Developer
Projekt Network
Remote, Canada
- $110,000 to $140,000 CAD / year
- Full-time
- Remote
We're looking for a Senior Full-Stack Developer who can move confidently across all three of our surfaces - the Django API, the Next.js web app, and the React Native mobile apps.
Added 2 months ago
HC
Co-op Engineer - Machine Learning and AI Infrastructure
Huawei Canada
Markham, ON
- Full-time
Strong programming skills, preferably in Python. Solid software engineering fundamentals, including debugging, testing, Git, and working with existing codebases.
Added 2 months ago
N
Staff DevOps Engineer
Nexxa.AI
Remote, Canada
- Full-time
- Remote
…physical operations - not just software. You'll partner closely with AI, data, and product engineering teams to make sure the systems they build can actually run in production, safely and…
Added 2 months ago
N
QA Engineer (AI Systems)
Nexxa.AI
Remote, Canada
- Full-time
- Remote
You'll work closely with ML engineers, backend engineers, and Forward Deployed Engineers to define what "good" looks like for an agent operating in high-stakes industrial settings, then…
Added 2 months ago
N
Backend AI Engineer
Nexxa.AI
Remote, Canada
- Full-time
- Remote
Strong proficiency in TypeScript/Node.js (our primary backend language), with strong API and microservice design skills. Working proficiency in Python is a plus for ML/model integration…
Added 2 months ago
H(
AI Enablement Business Partner
HOOPP (Healthcare of Ontario Pension Plan)
Toronto, ON · CA ON Toronto
- $103,000 to $153,000 CAD / year
- Full-time
Working technical fluency with at least one of: Power Platform (Power Automate, Power Apps, Copilot Studio), API integration with LLM providers, or scripting/light development (Python,…
Added 1 month ago
RB
Senior AI Engineer
Royal Bank of Canada
Toronto, ON
- Full-time
…maturity of our engineering organization. You will collaborate with cross-functional teams to drive AI adoption and innovation across the…
Added 3 months ago
L
Software Engineer
Larus
Ottawa, ON
- Full-time
5+ years of direct experience in design, development, deployment, and verification of software, with at least 2 years in Java and 2 years in python;

Nexxa.AI

QA Engineer (AI Systems)

Remote, Canada

Remote

Full-time

Apply About employer

Applications open in a new tab.

Job details

Job type: Full-time
Workplace: Remote
Posting language: English
Posted: Aug 27, 2026

Original job description

Nexxa is building the best AI systems for heavy industries - enabling machines, systems and operations to think, decide and act autonomously across manufacturing, large-scale infrastructure, logistics and legacy environments.

Our mission is to translate deep technical breakthroughs into operational reality, solving some of the hardest systems-level problems in industry.

Role Overview

We're looking for a Lead / Senior / Staff QA Engineer to own quality for Nexxa's AI agent systems - products that plan, call tools, and take multi-step actions autonomously in industrial environments. This isn't traditional UI testing: you'll be designing evaluation frameworks for non-deterministic, tool-using systems, building golden datasets, catching regressions in reasoning quality, and stress-testing agent behavior under adversarial and real-world edge-case conditions.

You'll work closely with ML engineers, backend engineers, and Forward Deployed Engineers to define what "good" looks like for an agent operating in high-stakes industrial settings, then build the infrastructure and processes to measure it continuously.

Key Responsibilities

Design and build evaluation harnesses and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence.
Develop golden datasets and labeled test sets, including edge cases, ambiguous inputs, and adversarial prompts specific to industrial and operational contexts.
Define and track quality metrics beyond simple accuracy - groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations.
Build automated pipelines that run evals on every model, prompt, or tool-integration change, and integrate them into CI/CD.
Conduct structured red-teaming and adversarial testing (prompt injection, jailbreaks, tool misuse, unsafe actions) in partnership with security teams.
Test agent behavior across the full action loop - planning, tool selection, tool execution, error recovery, and final output - not just the final response.
Investigate and triage failures where the root cause could be the model, the prompt, the tool/API, or the orchestration logic.
Partner with ML and backend engineers to translate eval failures into actionable, reproducible bug reports.
Establish quality bars and sign-off criteria for new agent capabilities before they reach customer environments.
Mentor other engineers on testing strategies specific to probabilistic, LLM-driven systems.
Advocate for testability and observability in agent architecture from day one.

Qualifications

5+ years in QA/SDET roles, with demonstrated ownership of test strategy for complex systems.
Hands-on experience testing LLM-based products, chatbots, or AI agents - you understand why traditional deterministic test assertions break down for generative systems.
Practical experience with eval frameworks or tooling (e.g., promptfoo, DeepEval, RAGAS, LangSmith) or a track record of building your own.
Strong scripting/programming ability (Python preferred) to build test automation, data pipelines, and eval tooling.
Understanding of how LLM agents work: prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks.
Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift over time.
Familiarity with LLM-specific failure modes: hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism.
Comfortable operating in ambiguity - defining what "correct" means for a task when there's no single right answer.
Strong written communication skills for turning fuzzy quality signals into clear, actionable findings for engineering and product stakeholders.

Preferred

Experience with human-in-the-loop evaluation workflows (labeling pipelines, inter-rater reliability, rubric design).
Background in ML/data science sufficient to read model evals and statistical significance.
Experience red-teaming or doing adversarial/security testing on ML systems.
Familiarity with observability/tracing tools for LLM applications (e.g., LangSmith, Arize, Langfuse, Weights & Biases).
Experience testing AI systems in industrial, IoT, or operational technology (OT) environments.
Prior experience setting up eval infrastructure from scratch at a startup or fast-moving team.
What We're Looking For A QA engineer who wants to define what quality means for autonomous, real-world AI systems.
Someone who can build rigorous evaluation infrastructure for problems that don't have a single right answer.
A systems thinker who enjoys turning ambiguous agent behavior into measurable, trustworthy signals.
A strong collaborator who partners well with ML engineers, backend engineers, and Forward Deployed teams.

Why Join Nexxa.AI?

Innovative Environment: Play a critical role in transforming heavy industries through groundbreaking AI and automation technologies.

Collaborative Culture: Be part of a team that values innovation, discipline, and continuous improvement.

Professional Growth: Benefit from significant opportunities for career development and advancement.

Competitive Compensation: Enjoy a comprehensive salary and equity package reflective of your expertise and contributions.

If you're passionate about AI quality and eager to help define what trustworthy autonomous systems look like in heavy industry, we'd love to connect.

Apply About employer

More about this job

Responsibilities

Design and build evaluation harnesses and regression suites for LLM-based agents to ensure reasoning quality and tool-call correctness. Develop automated pipelines and conduct structured red-teaming to stress-test agent behavior in high-stakes industrial environments.

Requirements

Requires 5+ years of experience in QA/SDET roles with hands-on experience testing LLM-based products or AI agents. Strong programming skills in Python and familiarity with evaluation frameworks like promptfoo, DeepEval, or LangSmith are essential.

Skills

QA Engineering
LLM
Python
Test automation
Evaluation frameworks
Prompt engineering
Red-teaming
Adversarial testing
Data pipelines
CI/CD
Tool calling
RAG
Observability
System testing
Debugging
Mentorship

Benefits

Competitive compensation
Equity package
Professional growth opportunities

Visa sponsorship

Not detected in the job text

Keywords

QA Engineer
AI Systems
LLM
Python
Promptfoo
DeepEval
RAGAS
LangSmith
CI/CD
Red-teaming
Adversarial testing
Hallucination
Tool calling
Orchestration
Data pipelines
Industrial automation
IoT
Observability
Tracing
Arize
Langfuse
Weights & Biases
Regression testing
Test strategy
Software engineering
Machine learning
System architecture
Quality assurance
Edge cases
Autonomous systems

Report this job