Find work in Canada.

A simpler way to find your next job.

Jobs in Canada

25+ jobs

Email me new jobs Save this search

Added 9 mins ago
C
Principal Technical Consultant, Google Cloud (GCP)
CWX
Vancouver, BC
- Full-time
Conduct regular reviews of project deliverables to ensure adherence to architectural standards, GCP best practices, and security/compliance guidelines.
Added 9 mins ago
LN
Aerospace System Safety Engineer
LGM North America
Montreal, QC
- Full-time
Added 9 mins ago
M
Delivery Project Lead- Cloud Devops Engg
Mphasis
Calgary, AB
- Full-time
…DevOps practices. The role requires expertise in microservices architecture, Kafka messaging systems, and Apigee for API management. The Delivery Project Lead will be responsible for…
Added 9 mins ago
HB
Sr. DevOps Engineer
Haventree Bank
Toronto, ON
- Full-time
Haventree Bank is a Schedule I Canadian bank dedicated to advancing financial security and upward mobility for Canadians underserved by traditional financial services. Through our…
Added 35 mins ago
UG
Power BI Report Developer
UNB Geodesy & Geomatics Engineering
Fredericton, NB
- Full-time
…advances of tomorrow. The Technology Unit is responsible for the architecture, reliability, and security of the University's core technology platforms. Within the Technical Operations Unit,…
Added 35 mins ago
D
Devsecops Engineer
Deel
Canada
- Full-time
We are currently seeking a DevSecOps Engineer to join our team. If you have a passion for automation, security, and infrastructure, we invite you to apply for this exciting opportunity. As…
Added 36 mins ago
AW
Portfolio Architect
Agilus Work Solutions
Calgary, AB
- Contract
…Resilience to lead the development of enterprise-wide resilience architecture within a complex, highly regulated environment. This strategic role is responsible for defining resilience…
Added 37 mins ago
A
Forward Deployed AI Engineer
Accenture
Calgary, AB · Vancouver, BC · Ottawa, ON · Toronto, ON
- Full-time
Experience defining enterprise AI architecture across identity, data, security, governance, and multi-system integrations
Added 37 mins ago
A
Forward Deployed AI Engineer - Senior Manager
Accenture
Calgary, AB · Vancouver, BC · Ottawa, ON · Toronto, ON
- Full-time
Define enterprise AI architecture standards and governance frameworks across the client organization, spanning identity, data, security, multi-platform integration, and AI lifecycle…
Added 37 mins ago
C
Senior Java Developer Solution Architect - Vice President
Citi
Mississauga, ON
- Full-time
…design and delivery of mission‑critical systems. You will own architecture decisions, drive engineering excellence, mentor senior engineers, and partner with product, security, and platform…
Added 52 mins ago
Software Developer
PCL Construction
Edmonton, AB
Follow established development standards and identify practical opportunities to improve team processes, quality, security, and reliability.
Added 52 mins ago
SL
Director Application, Data & Integration
Smythe LLP
Vancouver, BC
- Full-time
Embed security-by-design into application architecture and engineering practices.
Added 52 mins ago
D
Expert Engineer, Security - Central Technology
Demonware
Vancouver, BC
- Other
The Game Security team’s priorities can change quickly based on the threat landscape. The following areas of ownership will act as a baseline. Therefore, this role includes, but is not…
Added 1 hour ago
HT
Python Systems Engineer
High Tech Genesis
Toronto, ON
- Contract
…the ground up, ongoing development and maintenance, system architecture, and delivering secure, scalable, high-performance…
Added 1 hour ago
PC
Lead Data Consultant, Manager
PwC Canada
Toronto, ON
- Full-time
Evaluate technology and architecture options, articulate trade-offs, and provide pragmatic recommendations based on scalability, performance, security, maintainability and cost.
Added 1 hour ago
AC
Senior Manager, Security Advisory Services
Aviva Canada
Markham, ON
- Full-time
Minimum 10 years’ of progressive experience in cybersecurity risk management, vendor assessments, and application security design & architecture
Added 1 hour ago
SL
Director Application, Data & Integration
Smythe LLP
Vancouver, BC
- Full-time
Embed security-by-design into application architecture and engineering practices.
Added 1 hour ago
CT
SECURITY ARCHITECT
Chrome Technologies
Toronto, ON
- Full-time
The Security Architect will be part of a team who's role is to assure enterprise security architecture with a focus on the review and authorship of Architecture Decision Records (ADRs) and…
Added 1 hour ago
P
Software Engineer
PCMI
Canada
- Full-time
…Canada), Europe (Poland), and Asia (India), we're modernizing our architecture, embedding AI across our products, and scaling to meet growing demand. We move fast, we build with purpose,…
Added 1 hour ago
SS
Senior Software Developer, AI Solutions
System Soft Technologies
Quebec, QC
- Full-time
Steer agents with software-architecture direction and validate their output - hallucinated APIs, risky dependencies, security regressions.
Added 1 hour ago
L
Senior BI Platform Operations Engineer
Luxoft
Toronto, ON
- Contract
· Understand and troubleshoot security and AWS integrations, including IAM-based access and permissions.
Added 1 hour ago
M
Spécialiste infonuagique et DevOps / Cloud & DevOps Specialist
Montrium
Montreal, QC
- Full-time
…configurations, les déploiements, les procédures, les dossiers d’architecture, les dossiers de changement, les guides de reprise, les accès, la surveillance, les sauvegardes et la gestion…
Added 1 hour ago
KG
Java Backend & Integration Engineer (Quarkus & XML/XSLT)
K&K Global Talent Solutions INC.
ON · Ontario, Canada
- Full-time
…transformation pipelines, message-queue middleware, and microservices architecture. You will be responsible for designing and optimizing high-performance backend systems using Java and…
Added 1 hour ago
LH
Applied AI Developer III (AI Laboratory)
LGI Healthcare Solutions
Montreal, QC
- Full-time
…areas: AI application development and model integration, data architecture and RAG systems, and/or AI deployment and operations…
Added 1 hour ago
I
Sterling integrator administration Consultant
Infosys
Mississauga, ON
- Full-time
…level functionality like load balancer, Ips, firewall, cloud…

Nexxa.AI

Senior Software Development Engineer in Test (SDET), QA - AI Systems

Remote, Canada

Remote

Full-time

Apply About employer

Applications open in a new tab.

Job details

Job type: Full-time
Workplace: Remote
Posting language: English
Posted: Oct 19, 2026

Original job description

About Nexxa

Nexxa is building the best AI systems for heavy industries, enabling machines, systems and operations to think, decide and act autonomously across manufacturing, large-scale infrastructure, logistics and legacy environments.

Our mission is to translate deep technical breakthroughs into operational reality, solving some of the hardest systems-level problems in industry.

Role Overview

We’re looking for a Senior / Staff SDET to own quality for Nexxa’s AI agent systems, combining deep test automation with hands-on manual and exploratory testing. Our agents plan, call tools, and take multi-step actions autonomously in industrial environments. You’ll write production-grade code to build evaluation harnesses, simulation environments, and CI/CD quality gates, and you’ll also get hands-on with the agents yourself: probing behavior, reviewing traces, and validating releases where automation alone can’t judge whether the agent did the right thing.

You’ll work closely with ML engineers, backend engineers, and Forward Deployed Engineers to define what “good” looks like for an agent operating in high-stakes industrial settings, then build and run the systems and processes that measure it continuously.

Key Responsibilities

Automation and Infrastructure

Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence.
Build automated pipelines that run evals and test suites on every model, prompt, or tool-integration change, and integrate them into CI/CD as enforceable quality gates.
Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs so agents can be tested safely, repeatably, and at scale.
Implement API, contract, integration, and end-to-end tests for the services, tools, and orchestration layers that agents depend on.
Develop performance, load, and reliability tests for agent workflows, including failure injection and resilience testing.

Manual and Exploratory Testing

Perform hands-on exploratory testing of agent behavior, including new capabilities, ambiguous or underspecified requests, long multi-turn sessions, and unusual operational scenarios that scripted tests don’t anticipate.
Manually review agent traces and outputs to judge reasoning quality, appropriateness of actions, and safety, and use those findings to improve automated checks.
Execute manual regression and release verification passes in staging and customer-like environments before new agent capabilities ship.
Conduct manual red-teaming sessions (prompt injection, jailbreaks, tool misuse, unsafe actions) in partnership with security teams, then convert successful attacks into automated test cases.
Support user acceptance testing and pilot rollouts with Forward Deployed Engineers, capturing real-world failures and feeding them back into test suites.
Write clear test plans, test charters, and bug reports, and maintain manual test documentation where automation isn’t yet practical or cost-effective.

Datasets, Metrics, and Quality Strategy

Create and manage golden datasets and labeled test sets, including edge cases, ambiguous inputs, and adversarial prompts specific to industrial and operational contexts, with tooling for versioning and drift detection.
Design rubrics and run human-in-the-loop evaluations where automated scoring isn’t reliable enough.
Define and instrument quality metrics beyond simple accuracy (groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations) and build dashboards to track them over time.
Test agent behavior across the full action loop (planning, tool selection, tool execution, error recovery, and final output), not just the final response.
Investigate and triage failures where the root cause could be the model, the prompt, the tool/API, or the orchestration logic, and turn them into reproducible, actionable bug reports.
Establish quality bars, release criteria, and sign-off processes for new agent capabilities, blending automated gates with manual verification.
Decide what to automate and what to keep manual, and continually shift repeatable manual work into automation.
Mentor engineers on testing strategies and test-driven practices for probabilistic, LLM-driven systems.
Advocate for testability, observability, and tracing in agent architecture from day one.

Qualifications

5+ years as an SDET, QA engineer, or software engineer with a strong testing focus, with demonstrated ownership of test strategy and automation for complex distributed systems.
Proven experience in both manual and automated testing, including exploratory testing, test planning, and writing high-quality bug reports.
Strong programming skills (Python preferred) with a software engineering mindset: clean, maintainable, well-reviewed code for frameworks, data pipelines, and tooling.
Experience building test frameworks and infrastructure from the ground up, not just writing test cases within an existing framework.
Hands-on experience with CI/CD systems (e.g., GitHub Actions, GitLab CI, Jenkins) and containerized test environments (Docker, Kubernetes).
Experience testing APIs and microservices, including contract, integration, and end-to-end testing.
Hands-on experience testing LLM-based products, chatbots, or AI agents, and an understanding of why traditional deterministic assertions break down for generative systems.
Practical experience with eval frameworks or tooling (e.g., promptfoo, DeepEval, RAGAS, LangSmith) or a track record of building your own.
Understanding of how LLM agents work: prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks.
Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift over time.
Familiarity with LLM-specific failure modes: hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism.
Comfort operating in ambiguity, defining what “correct” means for a task when there’s no single right answer.
Strong written communication skills for turning fuzzy quality signals into clear, actionable findings for engineering and product stakeholders.

Preferred

Experience with human-in-the-loop evaluation workflows (labeling pipelines, inter-rater reliability, rubric design).
Experience with performance and load testing tools (e.g., k6, Locust) and chaos or fault-injection testing.
Background in ML/data science sufficient to read model evals and statistical significance.
Experience red-teaming or doing adversarial/security testing on ML systems.
Familiarity with observability and tracing tools for LLM applications (e.g., LangSmith, Arize, Langfuse, Weights & Biases) and general observability stacks (OpenTelemetry, Grafana, Datadog).
Experience testing systems in industrial, IoT, or operational technology (OT) environments, including industrial protocols (OPC UA, MQTT, Modbus) or simulation and hardware-in-the-loop setups.
Experience supporting UAT, pilots, or customer-facing validation of complex products.
Experience setting up test and eval infrastructure from scratch at a startup or fast-moving team.

What We’re Looking For

An engineer who is equally comfortable writing a test framework and sitting down to break an agent by hand.
Someone who can build rigorous evaluation systems for problems that don’t have a single right answer, and who knows when human judgment is the right tool.
A systems thinker who enjoys turning ambiguous agent behavior into measurable, trustworthy signals.
A strong collaborator who partners well with ML engineers, backend engineers, and Forward Deployed teams.

Why Join Nexxa.AI?

Innovative Environment: Play a critical role in transforming heavy industries through groundbreaking AI and automation technologies.
Collaborative Culture: Be part of a team that values innovation, discipline, and continuous improvement.
Professional Growth: Benefit from significant opportunities for career development and advancement.
Competitive Compensation: Enjoy a comprehensive salary and equity package reflective of your expertise and contributions.

If you’re passionate about AI quality and eager to help define what trustworthy autonomous systems look like in heavy industry, we’d love to connect.

Apply About employer

More about this job

Responsibilities

You will design and maintain automated test frameworks and evaluation harnesses for LLM-based AI agents in industrial environments. Additionally, you will perform manual exploratory testing, red-teaming, and define quality metrics to ensure the reliability and safety of autonomous systems.

Requirements

The role requires 5+ years of experience in SDET or software engineering with a strong focus on testing complex distributed systems. Candidates must possess strong Python programming skills and practical experience with CI/CD pipelines, containerization, and testing generative AI or LLM-based products.

Skills

Python
Test automation
LLM evaluation
CI/CD
Exploratory testing
API testing
Distributed systems
Docker
Kubernetes
Prompt engineering
Performance testing
Data pipelines
Software engineering
Quality assurance
Debugging
System architecture

Benefits

Competitive salary
Equity package

Remote locations

Toronto, Canada

Remote locations (standardized)

Toronto, Ontario, Canada

Visa sponsorship

Not detected in the job text

Keywords

SDET
AI Systems
LLM
Python
Test Automation
CI/CD
GitHub Actions
Docker
Kubernetes
API Testing
Exploratory Testing
Red-teaming
Prompt Injection
RAG
LangSmith
DeepEval
Promptfoo
Distributed Systems
Quality Assurance
Industrial Automation
Observability
OpenTelemetry
Data Pipelines
Regression Testing
Performance Testing
Chaos Engineering
Software Engineering
Machine Learning
Evaluation Harnesses
Autonomous Systems

Report this job