Find work in Canada.

334,090 jobs across our Canadian network. How we count

Jobs in Canada

25+ jobs

Email me new jobs Save this search

Added 7 mins ago
Building Services Supervisor 1
Minnesota Direct Care and Treatment
Saint-Paul, NB · St.-Paul, NB
Full-time
Administer quality control programs, conduct regular inspections, and report results to the Building Services Manager.
Added 8 mins ago
Branch Administrator
Jacknife Oilfield Services
Bonnyville, AB
Full-time
Reporting to the Operations Manager, this position is responsible for overseeing the branch administrative support staff and effectively organizing their activities in support of the truck…
Added 8 mins ago
Talent & Culture Business Partner
Haventree Bank
Toronto, ON
Full-time
Reporting to the Senior Manager, Talent and Culture, the Talent and Culture Business Partner serve as a trusted partner to the business, leaders and employees providing hands-on support…
Added 8 mins ago
Field Nurse Case Manager
Bayshore
Mississauga, ON
…individual differences are embraced and celebrated, through programs, resources and policies that support each individual's purpose, potential and…
Added 8 mins ago
Registered Practical Nurse
Bayshore
Toronto, ON
…individual differences are embraced and celebrated, through programs, resources and policies that support each individuals purpose, potential and…
Added 8 mins ago
Branch Administrator
Trimac Transportation
Bonnyville, AB
Full-time
Reporting to the Operations Manager, this position is responsible for overseeing the branch administrative support staff and effectively organizing their activities in support of the truck…
Added 19 mins ago
Banking Advisor
National Bank of Canada
Sturgeon Falls, ON
Full-time
…availability. This position reports to the Customer service Branch Manager or the Branch Manager. Every day, you will work with a team of a experts specialized in investment, credit,…
Added 22 mins ago
Wealth Associate
National Bank of Canada
London, ON
Full-time
…400,000 clients in about 100 branches. Reporting to the regional manager, you will team up with a wealth advisor to offer an exceptional client experience. Collaboration and partnership…
Added 22 mins ago
Talent Partner, Campus
SOTI
Mississauga, ON
Full-time
Reporting to the Senior Manager, Talent Partners, the Talent Partner, Campus will attract, engage, and hire top talent from universities and colleges across Canada and globally. In this…
Added 22 mins ago
50% Registered Early Childhood Educator (RECE) - Regular - Glen Ogilvie PS- CORE/PM (2:45pm - 6:15pm)
Ottawa-Carleton District School Board (OCDSB)
Ottawa, ON
Part-time
…OCDSB. Questions about this collection should be directed to the Manager, Human Resources, 133 Greenbank Road, Nepean, Ontario K2H…
Added 22 mins ago
Wealth Associate
National Bank of Canada
Ottawa, ON
Full-time
…400,000 clients in about 100 branches. Reporting to the regional manager, you will team up with a wealth advisor to offer an exceptional client experience. Collaboration and partnership…
Added 22 mins ago
50% Registered Early Childhood Educator (RECE) - Regular - Robert Hopkins - PM EDP (2:15pm - 5:45pm)
Ottawa-Carleton District School Board (OCDSB)
Ottawa, ON
Part-time
…OCDSB. Questions about this collection should be directed to the Manager, Human Resources, 133 Greenbank Road, Nepean, Ontario K2H…
Added 22 mins ago
Senior Financial Analyst
Deloitte
Toronto, ON
Full-time
Reporting to the Finance Manager, this role will lead all aspects of Financial Planning and Analysis for our most profitable service. You will work alongside the Firm’s Executive leaders to…
Added 22 mins ago
26AE04 - Lead Risk and Insurance Advisor - Guelph Board Office
Upper Grand District School Board
Guelph, ON
Full-time
Reporting to the Manager of Procurement, the Lead Risk and Insurance Advisor leads the development, implementation, and continuous improvement of the Board's risk management and insurance…
Added 22 mins ago
Copy - ECE - BASP - PM Wraparound LTO - Central
Halton District School Board
Milton, ON
Full-time
Applications submitted directly to the Human Resources Department or the School will not be considered or processed.
Added 22 mins ago
Senior Leaves Management Specialist
Arc'teryx Equipment
North Vancouver, BC
Full-time
You have a minimum of 4 years relevant experience in human resources, leave and absence management or disability programs, ideally with exposure to US state programs and regulations.
Added 22 mins ago
Communications & Brand Manager
Flynn Group of Companies
Mississauga, ON
$110,000 to $130,000 CAD / year
Full-time
Bachelor's degree or post-graduate degree in Communications, Public Relations, Marketing, Journalism, Business, Human Resources, or a related field
Added 22 mins ago
Senior Manager, People Operations
Photonic Inc.
Coquitlam, BC
$140,800 to $193,600 CAD / year
Full-time
We're seeking an experienced and ambitious Senior Manager, People Operations to help shape the future of how we work. Reporting directly to the VP, Human Resources, this role offers a…
Added 23 mins ago
Certified Diabetes Educator, Napanee
Kingston Community Health Centres
Greater Napanee, ON · Napanee, Ontario
Contract
…the development, implementation, monitoring and evaluation of programs and services for the community, individuals and families that are affected by diabetes. The RN/RD CDE works closely…
Added 23 mins ago
Post-Production Assistant/Assistant Editor
Canadian Film Centre
Toronto, ON
$612 CAD / week
Contract
Reports to: Manager, Post-Production & Technical
Added 23 mins ago
ScotiaMcLeod Regional Manager Administration - Montreal, Qc
Scotiabank
Montreal, QC
…creating a high-performance environment and implementing a human resources people strategy that attracts, retains, develops and motivates your team by fostering an inclusive work…
Added 38 mins ago
PT Custodian- Southside Aquatic Centre (1 Vacancy)
City of Woodstock
Woodstock, ON
Part-time
…A conflict of interest must be declared to the Director of Human Resources as outlined in the City of Woodstock Nepotism Policy, HR-015 as soon as it has been…
Added 38 mins ago
Office Manager
J.D. Irving, Limited
Concord, ON
Full-time
Midland Transport is currently seeking an Office Manager in Concord, ON. The Office Manager will oversee the administrative needs of the Terminal ensuring all other departments are…
Added 38 mins ago
Managing Director, Head of SLC Compensation Delivery - SLC Management
SLC Management
Toronto, ON · Sun Life Toronto One York
Full-time
10+ years of experience and a university education in Business, Human Resources or a related field. An MBA or MIR is an asset.
Added 38 mins ago
Managing Director, Head of SLC Compensation Delivery - SLC Management
Sun Life
Toronto, ON · Sun Life Toronto One York
Full-time
10+ years of experience and a university education in Business, Human Resources or a related field. An MBA or MIR is an asset.

First National Financial LP

AI Quality and Evaluation Lead

Toronto, ON, Canada
East York, ON, Canada

Job type: Full-time
Apply by: Oct 22, 2026
Posting language: English
Posted: Sep 22, 2026

Apply About employer

Applications open in a new tab.

Original job description

We are hiring a AI Quality and Evaluation Lead, Application Development!

Reporting To

Assistant Vice President IT, Application Development

Full-Time/Part- Time

Full-time

Posting Date

September 22, 2026

Closing Date

October 20, 2026

Hours Of Work

8:30 a.m. – 5:00 p.m.

Grade

Office Location:

14.4

Toronto

Great location! Steps away from the main public transit station

What We Offer

Highly competitive compensation package which includes, base salary, bonus, benefits, and career advancement opportunities!

Eligibility for benefits is dependent on the terms of employment

The Opportunity

The AI Quality and Evaluation Lead is accountable for the evaluation strategy, quality standards, release gates, guardrails, and continuous monitoring used across First National AI Factory solutions.

This role defines what good enough to ship means for AI enabled mortgage workflows, and converts business policy, risk appetite, user expectations, and production evidence into measurable acceptance criteria. Working within the AI Platform Squad, leads hands on evaluation for residential underwriting use cases with the mandate to evolve a scalable quality capability as the AI Factory grows, including reusable test assets, automated continuous evaluation, consistent scorecards, and clear accountability between product squads and shared quality teams.

How You Will Contribute

Own and continuously evolve the AI Factory quality and evaluation strategy, including risk tiers, standard metrics, acceptance thresholds, representative test suites, release evidence, monitoring expectations, and continuous improvement practices across product squads.
Define use case specific quality measures with the Product Manager, the Residential Underwriting Business SME, the AI Engineering Lead, QA Engineers, Risk, Compliance, and Security, covering policy adherence, factual grounding, completeness, precision, recall, false positives, false negatives, confidence, explainability, user impact, and appropriate escalation.
Own the methodology for evaluation harnesses, golden datasets, adversarial tests, regression suites, human review protocols, and automated scoring workflows across document intelligence, retrieval, summarization, agent tool use, deterministic calculations, and multi step workflows, with the AI Platform Squad building and operating the underlying evaluation infrastructure.
Establish the quality and evaluation gates within First National production readiness standards, jointly with the AI Engineering Lead who owns engineering and operational readiness, including UAT and parallel run evidence, minimum performance thresholds, guardrail validation, privacy and access control checks, rollback criteria, kill switch readiness, runbooks, and approval records.
In collaboration with Information Security conduct failure mode testing for prompt injection, data leakage, unsupported claims, policy deviations, tool misuse, unsafe write back, inappropriate automation, model degradation, and other AI specific operational risks.
Monitor production quality using evaluation results, model and prompt versions, drift, latency, user feedback, exception rates, override patterns, incidents, and token or platform cost, and define the triggers for investigation, rollback, retraining, prompt changes, or workflow redesign.
Create defect taxonomies and root cause practices that distinguish model, retrieval, prompt, data, rules, integration, user experience, and process issues, and work with product and engineering teams to prioritize durable fixes.
Evolve the function from hands on evaluation of the first use cases to a federated operating model, with reusable scorecards, automated continuous evaluation, trained squad level QA practices, oversight of AI Quality Analysts as volume grows, independent model and vendor benchmarking, and audit ready quality evidence.

The Experience You Need

Bachelor’s degree in computer science, engineering, data science, statistics, mathematics, information systems or a related discipline.
7 plus years of progressive experience in quality engineering, software testing, model validation, machine learning, data science, risk analytics, AI governance, or related technology disciplines.
Experience designing evaluation frameworks and release criteria for production AI or generative AI systems, including RAG, document intelligence, agents, model and prompt lifecycle management, observability, and human in the loop controls.
Strong analytical and technical skills with Python, SQL, test automation, data analysis, statistical methods, and evaluation tooling such as MLflow, Ragas, DeepEval, or comparable frameworks.
Working knowledge of accuracy and retrieval metrics, confidence scoring, sampling, experiment design, regression testing, drift monitoring, adversarial testing, red teaming, guardrails, privacy, security, and operational risk controls.
Ability to translate business policy and risk appetite into measurable acceptance criteria, and to explain quality, risk, and trade off implications clearly to technical, business, control, audit, and senior stakeholders.
Experience building or scaling a quality capability through reusable standards, automated test assets, continuous monitoring, governance evidence, coaching, and leadership of analysts or cross functional quality communities, including exposure to model risk, credit risk, or independent validation environments.
Experience in financial services, mortgage lending, lending operations, servicing, broker channels, third party partnerships, or regulated technology environments is preferred.

Relationships

External: Engages implementation partners, cloud providers, data and AI platform providers, model providers, and specialist vendors on evaluation methods, independent benchmarking, quality evidence, and knowledge transfer to First National.

Internal: Works within the AI Platform Squad while maintaining assessment independence from the delivery teams whose work it gates.

Working Environment And Physical Demands Analysis

Office environment
Periods of high volume with tight timelines
Long periods of stationary position/sitting
Prolonged periods of repetitive movement (i.e. using a keyboard and mouse)
Long periods of time in viewing a computer screen
Multi-tasking may include speaking to customers on a telephone call while looking up information on a computer program.

Why join First National?

Competitive Compensation
Comprehensive benefits program (i.e., Health Spending Account, Maternity and Parental Leave Top Up)
Extensive training programs to set our employees up for success
Modern office environment conducive to collaboration
Supportive teamwork culture
Opportunities to give back to the communities and work through events focused on a variety of charities
Ongoing social events throughout the year

The Team You’ll Join

Founded in 1988, First National is one of Canada’s largest non-bank lenders. We provide residential mortgages exclusively through the mortgage broker channel and we are Canada’s largest commercial mortgage lender.

First National has been consistently recognized as a great place to work and we are proud that our employee engagement feedback is higher than our industry partners.

We would like to thank all applications for their interest in this existing vacancy, but only candidates selected for an interview will be contacted.

Artificial Intelligence is not used in our recruitment or hiring process for this role.

First National is proud to be an equal opportunity employer and is committed to diversity and inclusion regardless of race, color, religion, national origin, age, gender identity, physical or mental disability, sexual orientation and any other category protected by law.

First National supports requests for accommodation from applicants with disabilities; please contact Human Resources at accessibility@firstnational.ca should you need an accommodation at any point in the recruitment process.

#FNLOON

Apply About employer

More about this job

Responsibilities

The AI Quality and Evaluation Lead is responsible for defining the quality strategy, evaluation standards, and release guardrails for AI-enabled mortgage workflows. This role involves hands-on evaluation, establishing automated testing frameworks, and collaborating with cross-functional teams to ensure production readiness and operational risk management.

Requirements

Candidates must hold a bachelor's degree in a technical field and possess over 7 years of experience in quality engineering, model validation, or AI governance. Strong technical proficiency in Python, SQL, and AI evaluation frameworks is required, along with the ability to translate business policy into measurable quality metrics.

Skills

AI evaluation
Quality engineering
Python
SQL
Test automation
Data analysis
Statistical methods
MLflow
Ragas
DeepEval
Model validation
Risk analytics
AI governance
Generative AI
RAG
Prompt engineering

Benefits

Bonus
Health spending account
Maternity and parental leave top up
Training programs
Career advancement opportunities

Visa sponsorship

Not detected in the job text

Keywords

AI Factory
Quality Engineering
Model Validation
Generative AI
RAG
Python
SQL
MLflow
Ragas
DeepEval
Test Automation
Data Analysis
Risk Analytics
AI Governance
Prompt Engineering
Document Intelligence
Adversarial Testing
Red Teaming
Mortgage Lending
Financial Services
Compliance
Security
Observability
Human-in-the-loop
Regression Testing
Drift Monitoring
Operational Risk
Underwriting
Software Testing
Benchmarking

Report this job