Jobs in Canada

Group jobs by employerEmail me new jobs Save this search

Added 29 days ago
CO
Principal Associate, AI Engineer
Capital One
Toronto, ON
- Full-time
Design, develop, test, deploy, and support AI software components including foundation model training, large language model inference, similarity search, guardrails, model evaluation,…
Added 29 days ago
CO
Manager, Applied Researcher
Capital One
Toronto, ON
- Full-time
Partner with a cross-functional team of data scientists, software engineers, machine learning engineers and product managers to deliver AI-powered products that change how customers…
Added 29 days ago
SL
Sr. Analytics and Automation Engineer
Sun Life
Toronto, ON · Waterloo, ON · Sun Life Toronto One York
- Full-time
…Successful Candidate will be working on several automation code development primarily for network services, with initial focus on micro/macro segmentation. This person will also play a…
Added 29 days ago
VI
AI Evaluation Scientist - Learning & Cognition
Vector Institute
Toronto, ON
- $125,800 to $157,300 CAD / year
- Full-time
Strong verbal and written communication skills and the ability to work effectively with ML specialists, software developers, scientists, instructors, and subject-matter experts; and,
Added 29 days ago
RB
Senior Security Specialist - Cloud (Global Security)
Royal Bank of Canada
Toronto, ON
- Full-time
Strong technical understanding in a variety of hardware and software platforms (Linux & Windows)
Added 29 days ago
C
Software Developer, Hybrid, Ottawa, Canada
Cisco
Ottawa, ON
- Full-time
We are seeking a highly skilled Software Developer to deliver features needed for integration between Cisco Cloud Control and Cisco Catalyst Center. In this role, you will design and lead…
Added 29 days ago
E
Manager, Software Engineering
Emburse
Toronto, ON
- Full-time
4–6 years of distributed software development experience, with 1–3 years of direct line management or player-coach management experience
Added 29 days ago
E
Manager, Software Engineering
Emburse
Toronto, ON
- Full-time
4–6 years of distributed software development experience, with 1–3 years of direct line management or player-coach management experience
Added 29 days ago
A
Principal Applied AI/ML Developer
Autodesk
Toronto, ON · AMER - Canada - Ontario - Toronto - University Ave
- Full-time
Familiarity with AutoCAD, Civil 3D, or other AEC/design software (as a user or developer via their APIs/SDKs).
Added 29 days ago
S
Systems Engineering Specialist
SpaceBridge
Montreal, QC
- Full-time
We are seeking a System Engineering specialist to support system-level design, requirements definition, and - critically - integration activities across R&D development projects.
Added 29 days ago
HA
Senior Product Manager
Hridayananda Advisory
Toronto, ON
- $220,000 CAD / year
- Full-time
One of our clients is looking for a Senior Product Manager to lead the development and growth of a new AI-powered document intelligence product. This is an exciting opportunity for a…
Added 29 days ago
V
Physical AI Research Scientist / Chercheur(-euse) scientifique - IA physique
Vention
Montreal, QC
- Full-time
…their operations in record time with the only hardware and software AI-powered platform built for the factory floor. Our technology powers over 25,000 machines across 4,000 factories across…
Added 29 days ago
B
QNX Everywhere Embedded Software Developer II
BlackBerry
Ottawa, ON
- Full-time
Are you passionate about systems software, open-source technologies, and solving complex technical challenges? Do you enjoy working at the intersection of operating systems, developer…
Added 29 days ago
BQ
QNX Everywhere Embedded Software Developer II
BlackBerry QNX
Ottawa, ON
- Full-time
Are you passionate about systems software, open-source technologies, and solving complex technical challenges? Do you enjoy working at the intersection of operating systems, developer…
Added 29 days ago
E
Full-Stack Engineer (Backend-Leaning) - CoreMusic
ElevenLabs
Remote, Canada
- Full-time
- Remote
Learning & development: ElevenLabs proactively supports professional development through an annual discretionary stipend.
Added 29 days ago
I
Principal Technology Architect – AI Transformation Programs
Infosys
Toronto, ON
- Full-time
…apply technical proficiency across different stages of the software development life cycle including requirements elicitation; this opportunity is to be part of a learning culture, where…
Added 29 days ago
GV
Junior Data Scientist
Great Value Hiring
Canada
- Contract
Added 29 days ago
GD
Junior GUI Design - Human Factors Engineering
General Dynamics Missions System International
Calgary, AB
- Full-time
…Interface (GUI) firmware on military embedded hardware devices or software designs for complex applications using the Scaled Agile Framework (SAFe) development…
Added 29 days ago
GD
Junior GUI Design - Human Factors Engineering
General Dynamics Missions System International
Ottawa, ON · Calgary, Alberta
- Full-time
…Interface (GUI) firmware on military embedded hardware devices or software designs for complex applications using the Scaled Agile Framework (SAFe) development…
Added 29 days ago
ES
Senior AI/ML Delivery Manager
EPAM Systems
Toronto, ON
- Full-time
Based in the Greater Toronto Area; able to work on-site 4 days per week
Added 29 days ago
MC
Senior Security Engineer
MatchBox Consulting Group
Vancouver, BC
- Full-time
Secure the software development lifecycle by integrating and managing scanning and repository tools.
Added 29 days ago
R|
Programmer
RI-MUHC | Research Institute of the MUHC | #rimuhc
Montreal, QC
- Full-time
Department / Research Program: CVIS/IDIGH (MLOps & AI Development)
Added 29 days ago
GD
Co-op Winter 2027 - Software Engineering (TacCIS Solutions) -12 Months
General Dynamics Missions System International
Calgary, AB
- Internship
Currently enrolled in a Bachelor’s degree program in Computer Science, Software Engineering, Electrical Engineering, or Computer Engineering.
Added 29 days ago
GD
Co-op Winter 2027 - Systems Integration, Verification, and Validation - 8-12 months
General Dynamics Missions System International
Calgary, AB
- Internship
Collaborate closely with engineers, software developers, and multidisciplinary teams across the organization.
Added 29 days ago
GD
Co-op Winter 2027 - Mobile Domain System Engineering - 12 - 18 Months
General Dynamics Missions System International
Calgary, AB
- Internship
…captured and documented, and also support various SW and FW development teams in the overall product life cycle to ensure the implementation and testing is in accordance with the…

Vector Institute

AI Evaluation Scientist - Learning & Cognition

Toronto, ON, Canada

$125,800 to $157,300 CAD / year · Full-time

Apply on LinkedIn About employer

Applications open in a new tab.

Job details

Pay: $125,800 to $157,300 CAD / year
Job type: Full-time
Experience: 2+ years experience working with agentic systems with a focus on contemporary agentic evaluation science;
Posting language: English
Posted: Oct 13, 2026

Original job description

AI Evaluation Scientist, Learning and Cognition

Position Summary

We are seeking an AI Evaluation Scientist, Learning and Cognition to design and implement evaluation methods for AI learning agents to be used in the post-secondary education sector.

This Scientist role focuses on establishing criteria for high-quality learning agents and how they should be evaluated. You will translate educational goals into valid constructs, rubrics, test sets, automated evaluators, and release criteria. You will work across conformance testing, LLM-as-judge methods, synthetic learner simulation, adversarial testing, human calibration, and trace analysis of real-world use.

This role requires a combination of investigating how people learn and reason with practical experience in evaluating agentic AI systems. You will be the team's primary source of learning and cognition expertise, working closely with faculty researchers in the field of human-computer interaction and with peers building agents and evaluation harnesses.

Key Responsibilities

Translate educational requirements into valid, testable constructs and rubrics, such as explanation quality, reasoning depth, mastery checking, logical consistency, and resistance to sycophancy;
Distinguish whether an agent conforms to its intended design from whether its behaviour is likely to support learning, and design evaluations for each;
Set evidence-based release criteria for new agents and agent changes;
Design evaluation pipelines for multi-turn agent trajectories, not only single responses;
Build and validate LLM-as-judge evaluators, applying current best practice from agentic evaluation research;
Create libraries of synthetic learner personas grounded in the learning sciences, representing different prior knowledge, common misconceptions, help-seeking behaviours, and interaction patterns;
Run adversarial testing and red teaming to identify inappropriate agent behaviours against safety and learning criteria;
Build regression suites that detect behavioural changes after updates to models, prompts, tools, source material, or agent specifications;
Design and run annotation studies with teachers and students from our user base, including rater recruitment, rubric training, and quality control;
Measure inter-rater reliability and human–machine agreement and decide which judgments each rater group can validly make;
Investigate evaluator failure modes and bias, including model-family bias, position bias, verbosity bias, and evaluator sycophancy;
Communicate results clearly to the engineering team, instructors, partners, and leadership; and,
Other related duties as assigned from time to time.

KEY SUCCESS MEASURES

A reusable evaluation harness is in use across at least two kinds of learning agents;
Synthetic learner and adversarial probe libraries consistently surface meaningful agent failures;
Automated evaluators show demonstrated, documented agreement with teacher and expert ratings;
There is a clear, agreed split between evaluation tasks that are automated and those that need human review;
Regression suites reliably catch behavioural changes before release;
Results are reproducible, auditable, and tied to specific agent and evaluator versions;
Evaluation findings have led to measurable improvements in agent behaviour; and,
Partner pilots are using your agent-behaviour measures in their study designs.

PROFILE OF THE IDEAL CANDIDATE

PhD or Master’s degree with relevant experience in cognitive science, psychology, learning science, human-computer interaction, or computer science/ML with substantial research in human learning or cognition;
2+ years experience working with agentic systems with a focus on contemporary agentic evaluation science;
Familiarity with learning science research on misconceptions, formative assessment, feedback, and productive struggle;
Experience with evaluation and observability tools such as Inspect, Langfuse, or Braintrust;
Experience building and evaluating AI agents, including fluency with agent tracing and application of state-of-the-art ML and LLM tooling;
Proficient in Python for research work, including building evaluation pipelines, analyzing data, and working with LLM APIs and evaluation tooling;
Experience with statistical, LLM-as-judge methods, agent trajectory evaluation, adversarial testing, red teaming, privacy and safety evaluations, and the creation of synthetic data sets to support evaluation of agentic systems;
Experience designing evaluation datasets, benchmarks, annotation studies, or controlled experiments;
Strong understanding of quantitative and qualitative evaluation methods, experimental design, statistical comparison, and inter-rater reliability;
Ability to translate qualitative behavioural requirements into precise, measurable evaluation criteria;
Strong verbal and written communication skills and the ability to work effectively with ML specialists, software developers, scientists, instructors, and subject-matter experts; and,
A passion for learning and continuous growth.

TOTAL REWARDS: The expected salary for this position will be $125,800 - $157,300 per year, plus benefits if applicable. The final salary offer will reflect the successful candidate's experience, skills, and qualifications, in alignment with the Vector Institute's Compensation Policy and may differ from above.

The Vector Institute’s Total Rewards approach extends beyond traditional compensation and benefits. Full-time employees are eligible for a comprehensive suite of supports that recognize and value employees, including vacation time, floater days, GRRSP, a Health Spending Account, a Summer Hours program, and flexible work arrangements.

POSITION STATUS: This posting is for a new vacancy.

USE OF ARTIFICIAL INTELLIGENCE : Vector may use both internal and external third party AI-based tools to assist in the screening of applications for this posting. Any data collected will be used solely for recruitment purposes and handled in accordance with Vector’s External Privacy Policy and Use of AI-Based Tools in Recruitment and Selection Policy .

INCLUSION AND EQUAL OPPORTUNITY EMPLOYMENT: Vector believes AI powers possibility by advancing cutting-edge research and translating it into real-world impact through collaboration with research, industry, and government. Vector is committed to fostering a diverse and inclusive culture that reflects its values.

The Vector Institute welcomes applications from all qualified candidates, including those who are Indigenous, 2SLGBTQIA+, racialized persons/visible minorities, women, and people with disabilities.

If you require an accommodation at any stage of the recruitment or selection process, please contact hr@vectorinstitute.ai. The Vector Institute team will be happy to work with you to ensure your experience is as inclusive and accessible as possible.

JOIN OUR COMMUNITY: Check out the Vector Institute’s Careers Page to explore open opportunities at Vector and Follow Vector on X , LinkedIn , and Bluesky to stay connected with the latest developments in Ontario's AI ecosystem and the Vector Institute.

Apply on LinkedIn About employer

More about this job

Responsibilities

Design and implement evaluation frameworks for AI learning agents, translating educational goals into measurable constructs, rubrics, automated evaluators, test sets, and release criteria. Build and validate evaluation pipelines, synthetic learner and adversarial test libraries, regression suites, and human annotation studies, then communicate findings to improve agent behavior.

Requirements

Requires a master's or doctoral degree in a relevant field, or computer science/ML with substantial research in human learning or cognition, and at least two years of experience with agentic systems and contemporary evaluation methods. Candidates should bring learning science knowledge, Python and LLM tooling proficiency, experience with statistical and human evaluation, and strong communication skills.

Skills

AI Agent Evaluation
Learning Science
Cognitive Science
Python
Evaluation Pipeline Development
LLM-as-Judge Methods
Agent Trajectory Evaluation
Statistical Analysis
Experimental Design
Inter-Rater Reliability
Annotation Studies
Adversarial Testing
Red Teaming
Synthetic Learner Simulation
Evaluation Dataset and Benchmark Design
Agent Tracing

Benefits

Vacation Time
Floater Days
Health Spending Account
Summer Hours Program
Flexible Work Arrangements

Visa sponsorship

Not detected in the job text

Keywords

AI Evaluation
Agentic Systems
Learning Agents
Learning Science
Cognitive Science
Human-Computer Interaction
Python
LLM APIs
LLM-as-Judge
Inspect
Langfuse
Braintrust
Evaluation Harnesses
Agent Tracing
Multi-Turn Trajectory Evaluation
Synthetic Learner Personas
Adversarial Testing
Red Teaming
Regression Testing
Annotation Studies
Inter-Rater Reliability
Human-Machine Agreement
Experimental Design
Statistical Comparison
Evaluation Datasets
Benchmarks
Privacy Evaluation
Safety Evaluation
Formative Assessment
Misconceptions

Report this job

Work requirements

Jobs in Canada