Find work in Canada.

605,493 jobs across our Canadian network. How we count

Jobs in Canada

25+ jobs

Email me new jobs Save this search

Added 11 mins ago
OT
Analyste en Modélisation et Simulation d’Électronique de Puissance / Power Electronics Modeling and Simulation Analyst
OPAL-RT TECHNOLOGIES
Montreal, QC
- Full-time
Leader mondial dans la diversité des applications, nos outils sont utilisés dans une multitude de secteurs tels que l'automobile, l'aérospatiale, l’énergie, le transport, la fabrication…
Added 11 mins ago
FH
Project Leader, Clinical Operations
Fraser Health
Surrey, BC
- Full-time
Fraser Health is seeking a Project Leader to drive the planning, design, and implementation of innovative clinical programs that improve patient care and service delivery.
Added 26 mins ago
OH
Senior Specialist, Technical-1
Ontario Health | Santé Ontario
Toronto, ON
- Full-time
Added 40 mins ago
CH
Manager, Logistics and Distribution
CMH Heli-Skiing & Summer Adventures
AB · Alberta, Canada
- Full-time
…experiences, and our vision is to be recognized as the world leader of guided mountain experiences - by our guests, our employees, our partners, and our communities. We're all drawn to CMH…
Added 41 mins ago
LD
Senior Electrical Engineer Power Electronics
Leonardo DRS
Bedford, NS
- Full-time
DRS Pivotal Power, Inc. is located in Bedford, Nova Scotia. DRS Pivotal Power is a leader in the design, development and manufacture of high reliability power conversion equipment for…
Added 41 mins ago
OT
Analyste en modélisation et simulation électrique / Electrical Modeling and Simulation Analyst
OPAL-RT TECHNOLOGIES
Montreal, QC
- Full-time
Leader mondial dans la diversité des applications, nos outils sont utilisés dans une multitude de secteurs tels que l'automobile, l'aérospatiale, l’énergie, le transport, la fabrication…
Added 41 mins ago
CR
Culinary Manager
COGIR Real Estate
Oakville, ON
- Full-time
Embrace the role of Culinary Manager and become a values-driven leader adept at managing our culinary and dining department. Your flair for creativity and culinary expertise will come to…
Added 41 mins ago
T
Personal Banking Associate - O'Brien, Ville Saint Laurent (Part-time, 18.75h/week)
TD
Montreal, QC
- Part-time
Help build a positive and equitable work environment by promoting team effectiveness, participating in personal performance development and sharing industry knowledge to achieve common…
Added 41 mins ago
T
Personal Banking Associate Trainee (Full time)- St-Sauveur
TD
Saint-Sauveur, QC
- Full-time
Help build a positive and equitable work environment by promoting team effectiveness, participating in personal performance development and sharing industry knowledge to achieve common…
Added 41 mins ago
T
Applied Machine Learning Scientist I
TD
Toronto, ON · Montreal, QC
- Full-time
We're looking for a highly motivated Applied Machine Learning Scientist I to join our TDI AI/ML team. This role is primarily focused on Generative AI, including LLM-based and agentic…
Added 41 mins ago
T
Senior Counsel
TD
Toronto, ON
- Full-time
Collaborating with talented colleagues and partners in a variety of legal specializations, members of the legal team have the chance to develop their skills every day. From mentorship and…
Added 41 mins ago
T
Personal Banking Associate Trainee - Mandarin or Cantonese Language Skills
TD
Richmond, BC
- Full-time
Help build a positive and equitable work environment by promoting team effectiveness, participating in personal performance development and sharing industry knowledge to achieve common…
Added 1 hour ago
FH
Clinical Nurse Specialist (CNS) - Pediatric Population
Fraser Health
BC · Fraser Health, British Columbia
- Full-time
…support quality improvement initiatives, and contribute to the development of nursing excellence across the health…
Added 1 hour ago
Agentic Evaluation & Testing Specialist, Kitchener, ON
Siemens
Kitchener, ON
We are seeking a System Testing and Agent Evaluation Specialist to join Siemens Digital Industries Operations Software at our AI hub in Canada. You make change verifiable in a product that…
Added 1 hour ago
T
Personal Banking Associate - Robert-Bourassa Branch, Montreal
TD
Montreal, QC
- Full-time
Help build a positive and equitable work environment by promoting team effectiveness, participating in personal performance development and sharing industry knowledge to achieve common…
Added 1 hour ago
T
Manager, Customer Experience
TD
Thunder Bay, ON
- Full-time
· Mentor a team by showcasing exceptional experiences in every customer interaction and achieving operational excellence, and personal development objectives.
Added 1 hour ago
R
Project Engineering Manager
Refresco
Mississauga, ON
- Full-time
Development of regional budgets, writing projects, managing suppliers and execute schedules.
Added 1 hour ago
DO
Sr. Industrial Designer
Design Observer
Calgary, AB
- Full-time
DIRTT is a global leader in industrialized construction. Our system of physical products and digital tools empowers organizations - alongside construction and design leaders - to build…
Added 1 hour ago
C
Senior Application Security Developer
Clio
Calgary, AB · Vancouver, BC · Toronto, ON
- Full-time
…Vulnerability Management Program within the Application Security Team. Vulnerability Management is established as a standing program at Clio, with a clear mission: Vuln0 is the goal, and…
Added 1 hour ago
AG
Director, Growth & Innovation Operations
Altus Group
Toronto, ON
- Full-time
…have the breadth and complexity to continue developing as a leader while building processes and capabilities that support the business over…
Added 1 hour ago
OT
Sr. Software Developer (Bilingual)
Open Text Corporation
Ottawa, ON · Richmond Hill, ON
- Other
We are looking for a Senior Backend Software Developer to join our development team at OpenText. Our team builds and evolves a public, multi-tenant Software-as-a-Service (SaaS) cloud…
Added 2 hours ago
AS
Facilities Maintenance - Future Opportunity
ATCO Structures
Cookstown, ON
- Full-time
Always there. Anywhere. That’s us! A team committed to delivering inspired solutions for a better world. We care for our communities and each other, and we are committed to showing up for…
Added 2 hours ago
R
AI Solutions Lead, Enterprise Applications (T & I) (Telework/Hybrid)
Radio-Canada
Montreal, QC
- Full-time
Reporting directly into the Platform team, your primary mandate is to build, scale, and leverage the platform for AI-Augmented development. You are not just building individual…
Added 2 hours ago
L
Senior Technical Program Manager - Engineering Chief of Staff
lululemon
Vancouver, BC
- Full-time
…organization’s operating rhythm, lead the programs that cross team and organizational boundaries, and lead the program management team that gets our largest launches out the…
Added 2 hours ago
S
Platform Engineer
SAP
Vancouver, BC
- Full-time
…known for leadership in enterprise resource planning (ERP) software, SAP has evolved to become a market leader in end-to-end business application software and related services for database,…

Siemens

Agentic Evaluation & Testing Specialist, Kitchener, ON

Kitchener, ON, Canada

Apply About employer

Applications open in a new tab.

Job details

Posting language: English
Posted: Nov 5, 2026

Original job description

Change the future with us.

We are looking for dedicated and talented people who solve ever-changing challenges, customer needs, and questions from colleagues with clever concepts and creativity. We embrace change and work with curious minds re-inventing the future of work. Join us and let us focus together on what’s truly important: making lives better with new ideas and the latest technology around the world.

About the role: System Testing and Agent Evaluation Specialist

We are seeking a System Testing and Agent Evaluation Specialist to join Siemens Digital Industries Operations Software at our AI hub in Canada. You make change verifiable in a product that is built AI-first: much of our code, tests, and documentation are produced by coding agents under the direction of engineers. This means that you will be working with the whole team on the challenge of validating the system quality.

You build the test infrastructure, the contract tests between components developed at our global sites, and the evaluation methods for agent behavior that cannot be judged by a simple pass or fail. You work with the industrial experts in Europe who define what a correct outcome looks like on a shop floor, and with the developer platform team that runs your tests at scale. This is a software engineering role for someone who thinks in terms of evidence and problem solving, not a manual testing role.

The role covers two separate workstreams with one underlying discipline. First, you are responsible for the verification of the product itself, covering classical software testing of a distributed, multi-site system. Second, you are responsible for building an evaluation workflow into the product that allows it to test and verify its own agent-generated output: new skills, prompts, and agent configurations are assessed for correctness, safety, performance, and cost in a controlled environment before a human decides whether to activate them. The first workstream is how we build the product; the second is a product feature our customers rely on.

About Siemens Digital Industries Operations Software: It is a suite of integrated digital solutions to engineer, simulate and operate manufacturing operations across industries. Operations Software includes our product portfolio in manufacturing engineering (COMOS, Tecnomatix, NX CAM, ...) & production operations (Win CC, Scada, PCS 7 / neo, Opcenter, Insights Hub, ...). The portfolio helps organizations across many different industries to improve operational efficiency, reduce costs, and enhance decision-making by providing real-time visibility and control across manufacturing and business processes.

Formula 1 cars, ships, mobile phones, space exploration vehicles, food & beverages, pharmaceuticals and many of the other objects we see in our daily lives are being conceived and produced using our software.

We're redefining the industrial world using data and AI to make production: Agile, Adaptive, Autonomous. Eliminate productivity boundaries for today’s and tomorrow’s production operation. Join our mission to build the most comprehensive next generation AI-native Industrial Operations Software portfolio as the new operation system for industry in the age of AI. With us you have the chance to now apply AI to where it really matters: In the physical world of an industrial production and engineering of real things.

As part of Siemens Operations Software (OPS-SW), the Technology, Data Fabric & Agentic AI organization provides the technology foundation that enables the next generation of industrial software. Its portfolio includes AI Production & Engineering Systems, Data Fabric, Agentic AI frameworks, connectivity platforms, semantic data models, cloud-native architectures, and industrial AI services that power autonomous, scalable, and intelligent operations across the OPS-SW portfolio.

What will you do?

Build automated test infrastructure for a product developed across multiple sites, covering cross-component contract tests and integration and end-to-end tests against deployed systems.
Design evaluation methods for agent behavior that a pass/fail assertion cannot capture: task completion, permission compliance, error handling, latency, and cost, measured over repeated runs and compared against the active version.
Build the assessment workflow into the product itself, so that agent-generated skills, prompts, and configurations are evaluated in isolated environments before a human decides whether to activate them.
Integrate the security checks defined by product security into that workflow, so that every artifact is assessed for correctness and containment in one run with two separate verdicts.
Keep test expectations independent of the implementation, so that generated code and generated tests cannot confirm the same wrong assumption, and reject agent-generated tests that prove nothing.
Make results usable by the product's own improvement loop as well as by human approvers, without letting anything bypass approval.
Work AI-first: use coding agents for test development and failure analysis, and own the judgment about what the results mean.

What you will bring?

Degree in computer science, software engineering, or a related field, or equivalent experience.
4+ years in software engineering with a focus on test automation, test infrastructure, or verification of distributed systems; you write production-quality code.
Experience designing test strategies for systems with asynchronous behavior, external dependencies, and non-reproducible components.
Practical experience with LLM-based applications and an understanding of why conventional assertions are not enough to evaluate them.
Experience building evaluation pipelines or quality gates that run as part of a product or platform, not only in a developer's CI.
Familiarity with contract testing, API schema validation, and CI/CD integration of test suites.
Solid Python; comfortable with containerized test environments and Kubernetes-based execution.
The instinct to ask "how would we know if this were wrong" before asking "does it pass".

What sets you apart?

Experience with agent evaluation frameworks and AI observability tooling (Langfuse, LangSmith, Promptfoo, or comparable).
Experience with property-based testing, metamorphic testing, or statistical test methods for nondeterministic systems.
Background in safety-relevant or regulated software where test evidence is an audit artifact and release decisions are documented.
Experience with simulation-based testing or hardware-in-the-loop setups.
Exposure to industrial systems (MES, SCADA, OPC UA) and an understanding of what a wrong action costs on a shop floor.
Experience designing systems that evaluate other AI systems, including LLM-as-judge setups and their known failure modes.

Comfortable working with distributed teams across time zones and communicating asynchronously by default.

Salary is commensurate with experience, and ranges between $ 100,000 CAD - $ 150,000 CAD, excluding bonus and benefits. In addition to base salary, this role includes eligibility for an annual discretionary bonus of 10%.

Why you’ll love working for Siemens!

Join our Digital World

At Siemens Digital Industries Operations Software, flexibility is how we work – hybrid, built on trust and autonomy. We build technology that shapes the real world. You'll grow through real projects, strong technical peers, and global mobility, backed by the scale and benefits of an industrial operations software leader. We're committed to equality and inclusion, and we hire based on merit, skills, and impact. Bring your curiosity and creativity and help us shape tomorrow!

Our Benefits & Rewards

The actual compensation offered is based on the successful candidate's job-related skills, experience, and relevant education/training. Siemens offers a wide range of health and wellness benefits to employees

Our Commitment to Equity and Inclusion

We value equal opportunities and welcome applications from people with disabilities. At Siemens, we believe people who've had real experiences dealing with being different will excel as leaders. Let's foster a culture of creativity and innovation. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

Siemens is proud to be an eight-time award winner of Canada’s Top 100 Employers, Canada’s Greenest Employers 2025 and Canada’s Top Employers for Young People 2025.

About us.

We share our ideas and champion the people behind them.

Siemens Canada is a leading technology company focused on industry, infrastructure, mobility and healthcare. The company’s purpose is to create technology with purpose, transforming the everyday, for everyone, since 1912. By combining the real and the digital worlds, Siemens empowers its customers to accelerate their digital and sustainability transformations, making factories more efficient, cities more liveable, and transportation more sustainable. Siemens also owns a majority stake in the publicly listed company Siemens Healthineers, a leading global medical technology provider pioneering breakthroughs in healthcare. For everyone. Everywhere. Sustainably.

In fiscal 2024, which ended September 30, 2024, Siemens Canada had revenues of approx. $2.2 billion CAD. The company has approximately 4,400 employees from coast-to-coast and 37 office and production facilities across Canada.

To learn more about Siemens Canada, visit our website at www.siemens.ca

While we appreciate all applications we receive, we advise that only candidates under consideration will be contacted.

#OneOpsSW

Siemens is committed to creating a diverse environment and is proud to be an equal opportunity employer. Upon request, Siemens Canada will provide reasonable accommodation for disabilities to support participation of candidates in all aspects of the recruitment process. All qualified applicants will receive consideration for employment.

By submitting personal information to Siemens Canada Limited or its affiliates, service providers and agents, you consent to our collection, use and disclosure of such information for the purposes described in our Privacy Notice available at www.siemens.ca.

Siemens s’engage à créer un environnement diversifié et est fière d’être un employeur souscrivant au principe de l’égalité d’accès à l’emploi. Sur demande, Siemens Canada prendra des mesures d’accommodement raisonnables pour les personnes handicapées, dans le but de soutenir la participation des candidats dans tous les aspects du processus de recrutement. Tous les candidats qualifiés seront pris en considération pour ce poste.

En transmettant des renseignements personnels à Siemens Canada limitée ou à ses sociétés affiliées, à ses fournisseurs de services ou à ses agents, vous nous autorisez à recueillir, à utiliser et à divulguer ces renseignements aux fins prévues dans notre Déclaration de protection de la confidentialité, que vous pouvez consulter au www.siemens.ca.

Job ID

528626

Posted since

05-Nov-2026

Organization

Digital Industries

Field of work

Research & Development

Company

Siemens Canada Limited

Experience level

Experienced Professional

Job type

Full-time

Work mode

Hybrid (Remote/Office)

Employment type

Permanent

Location(s)

Kitchener - Ontario - Canada

Apply About employer

More about this job

Responsibilities

You will build automated test infrastructure for distributed systems and design evaluation methods for agent-generated outputs. Additionally, you will integrate assessment workflows into the product to ensure correctness, safety, and performance before human approval.

Requirements

Candidates must have a degree in computer science or software engineering and at least 4 years of experience in test automation or verification of distributed systems. Proficiency in Python, containerized environments, and experience with LLM-based applications are required.

Skills

Software engineering
Test automation
Distributed systems
Python
Kubernetes
LLM-based applications
Contract testing
API schema validation
CI/CD
Agent evaluation frameworks
AI observability
Property-based testing
Metamorphic testing
Industrial systems
Simulation-based testing

Benefits

Health and wellness benefits
Annual discretionary bonus
Global mobility opportunities

Visa sponsorship

Not detected in the job text

Keywords

Agentic AI
System Testing
Test Automation
Python
Kubernetes
LLM
CI/CD
Contract Testing
Langfuse
LangSmith
Promptfoo
Distributed Systems
Industrial Software
MES
SCADA
OPC UA
Software Engineering
Quality Assurance
Data Fabric
Cloud-native

Report this job