Find work in Canada.

622,537 jobs across our Canadian network. How we count

Jobs in Canada

Email me new jobs Save this search

Added 17 days ago
AI
Senior Forward Deployed AI Engineer (Python, Data & GenAI)
Astra-North Infoteck Inc. ~ Conquering today’s challenges, achieving tomorrow’s vision!
Toronto, ON
- Full-time
Profile unfamiliar data using Python, pandas, PySpark, and SQL to assess coverage, quality, volumes, distributions, edge cases, and feasibility.
Added 17 days ago
TC
Technical Architect
Tata Consultancy Services
Toronto, ON
- $90,000 to $100,000 CAD / year
- Full-time
We are seeking a Technical Architect / Senior Developer – RPA & Automation to design, develop, maintain, and enhance enterprise automation solutions. The role will focus on UiPath,…
Added 17 days ago
PT
Senior Data Engineer – Capital Markets
Precision Technologies
Mississauga, ON
- Contract
We are looking for an experienced Senior Data Engineer with strong Capital Markets, SQL, Data Warehousing, and Snowflake expertise to join our technology team in Mississauga.
Added 17 days ago
T
Market Risk Manager, Trading Market Risk Stress Testing & Scenario Design (5111)
TD
Toronto, ON
- Full-time
…risks are transparent to Risk Management, Business Leaders, and Senior…
Added 17 days ago
RT
Business Intelligence Specialist - Senior
Russell Tobin
Toronto, ON
- $85 to $95 CAD / hour
- Contract
Oracle, Python, AWS Glue Python Shell, and AWS DMS experience, including complex SQL, source-data analysis, extraction, transformation, incremental loading, validation, reconciliation, and…
Added 17 days ago
RT
Senior Software Engineer
Russell Tobin
Toronto, ON
- $75 to $95 CAD / hour
- Contract
Microsoft Visual Studio (.NET) MVC, SQL Server (database creation and management) in Azure
Added 17 days ago
T
Senior Software Developer – .NET, Python, AWS & Azure
ThoughtStorm
Toronto, ON
- Contract
We are hiring a Senior Software Developer for Public sector client to support its Drug Formulary and Genomics platforms.
Added 17 days ago
T
Business Specialist
ThoughtStorm
ON · Ontario, Canada
- Contract
The Agency requires a Senior Business Intelligence Specialist with strong Oracle experience to support the Ontario Clinical Report Exchange (OCRE) project. The resource will develop the…
Added 17 days ago
LC
Senior Business Data Analyst
Loblaw Companies Limited
Brampton, ON
- Full-time
We are looking for a talented and motivated Senior Business Data Analyst to join the Finance Insights and Technology Engagement (FITE) team. You will be working with various teams and…
Added 17 days ago
KC
Senior Consultant, Microsoft Fabric Data Engineer, Data & AI
KPMG Canada
Calgary, AB
- Full-time
Strong proficiency in SQL and solid understanding of modern data modeling principles and data warehousing concepts.
Added 17 days ago
Associate Director, Compliance
Purpose Unlimited
Toronto, ON
Prepare clear, decision-useful compliance reporting for senior management and governance forums.
Added 17 days ago
Software Developer Sr (AI Platform & Applications)
Dayforce
Canada
We are looking for a Senior Software Developer to join the team building AI-powered capabilities within the Dayforce platform. This role is ideal for an experienced software engineer who…
Added 17 days ago
C
Senior Developer (T & I) (Telework/Hybrid)
CBC/Radio-Canada
Montreal, QC
- Full-time
…team is currently seeking a creative, qualified and talented developer. Based in Montreal as part of a cross-functional team, you will focus primarily on integrating and developing…
Added 17 days ago
W
Senior Data Analyst, Operations
Waabi
Toronto, ON
- Full-time
Strong SQL skills and experience working with relational datasets, plus experience with Python or R for analysis, pandas preferred.
Added 17 days ago
R
Senior Developer (T & I) (Telework/Hybrid)
Radio-Canada
Montreal, QC
- Full-time
…team is currently seeking a creative, qualified and talented developer. Based in Montreal as part of a cross-functional team, you will focus primarily on integrating and developing…
Added 17 days ago
AG
Application Developer III
Allegis Global Solutions
Toronto, ON
- $70 to $73 CAD / hour
- Contract
We are seeking a highly skilled and motivated senior application developer to join a dynamic Data & Digital team within an Engineering, Innovation & Architecture organization. This is a…
Added 17 days ago
RB
Senior Data Scientist
Royal Bank of Canada
Toronto, ON
- Full-time
Proficiency in programming languages (Python, R, SQL) and machine learning frameworks (e.g., Scikit-learn, TensorFlow, PyTorch).
Added 17 days ago
Senior Software Engineer II, Decisioning Studio Integrations Team
Braze
Toronto, ON
We're looking for a Senior Software Engineer II to join the Decisioning Studio Integrations team building BrazeAI Decisioning Studio™. Decisioning Studio is a reinforcement learning…
Added 17 days ago
FN
AI Quality and Evaluation Lead
First National Financial LP
Toronto, ON · East York, ON
- Full-time
Strong analytical and technical skills with Python, SQL, test automation, data analysis, statistical methods, and evaluation tooling such as MLflow, Ragas, DeepEval, or comparable…
Added 17 days ago
PF
Senior Data Engineer - Data Governance & DataOps
Pattison Food Group
Langley (District), BC · Langley Office - Langley BC
- Full-time
Strong SQL and Python; hands on experience with Spark (PySpark) and Databricks Jobs/Workflows.
Added 17 days ago
B
Senior Software Engineer II, Decisioning Studio Integrations Team
Braze
Toronto, ON
- Full-time
We're looking for a Senior Software Engineer II to join the Decisioning Studio Integrations team building BrazeAI Decisioning Studio™. Decisioning Studio is a reinforcement learning…
Added 17 days ago
HV
Optical Supply Chain Analyst
Hoya Vision Care
Mississauga, ON
- Full-time
Working closely with the Senior Supply Chain Analyst and cross-functional business teams, this role combines operational supply chain knowledge with strong analytical and business analysis…
Added 17 days ago
PC
Digital Operation - Senior Associate
Pwc Canada
Toronto, ON
- Full-time
Tools may include Python, R, and SQL; optimization solvers; simulation libraries; and enterprise or cloud data platforms.
Added 17 days ago
R
Senior Data Scientist
RBC
Toronto, ON
- Full-time
Proficiency in programming languages (Python, R, SQL) and machine learning frameworks (e.g., Scikit-learn, TensorFlow, PyTorch).
Added 17 days ago
T
Senior Treasury Analyst
TD
Toronto, ON
- Full-time
Experience with programming for extract transform load (ETL) operations and data analysis (including performance optimization) using languages such as, but not limited to, Python, SQL and R.

First National Financial LP

AI Quality and Evaluation Lead

Toronto, ON, Canada
East York, ON, Canada

Full-time

Apply About employer

Applications open in a new tab.

Job details

Job type: Full-time
Apply by: Nov 20, 2026
Posting language: English
Posted: Oct 21, 2026

Original job description

We are hiring a AI Quality and Evaluation Lead, Application Development!

Reporting To

Assistant Vice President IT, Application Development

Full-Time/Part- Time

Full-time

Posting Date

October 21, 2026

Closing Date

November 23, 2026

Hours Of Work

8:30 a.m. – 5:00 p.m.

Grade

Office Location:

14.4

Toronto

Great location! Steps away from the main public transit station

What We Offer

Highly competitive compensation package which includes, base salary, bonus, benefits, and career advancement opportunities!

Eligibility for benefits is dependent on the terms of employment

The Opportunity

The AI Quality and Evaluation Lead is accountable for the evaluation strategy, quality standards, release gates, guardrails, and continuous monitoring used across First National AI Factory solutions.

This role defines what good enough to ship means for AI enabled mortgage workflows, and converts business policy, risk appetite, user expectations, and production evidence into measurable acceptance criteria. Working within the AI Platform Squad, leads hands on evaluation for residential underwriting use cases with the mandate to evolve a scalable quality capability as the AI Factory grows, including reusable test assets, automated continuous evaluation, consistent scorecards, and clear accountability between product squads and shared quality teams.

How You Will Contribute

Own and continuously evolve the AI Factory quality and evaluation strategy, including risk tiers, standard metrics, acceptance thresholds, representative test suites, release evidence, monitoring expectations, and continuous improvement practices across product squads.
Define use case specific quality measures with the Product Manager, the Residential Underwriting Business SME, the AI Engineering Lead, QA Engineers, Risk, Compliance, and Security, covering policy adherence, factual grounding, completeness, precision, recall, false positives, false negatives, confidence, explainability, user impact, and appropriate escalation.
Own the methodology for evaluation harnesses, golden datasets, adversarial tests, regression suites, human review protocols, and automated scoring workflows across document intelligence, retrieval, summarization, agent tool use, deterministic calculations, and multi step workflows, with the AI Platform Squad building and operating the underlying evaluation infrastructure.
Establish the quality and evaluation gates within First National production readiness standards, jointly with the AI Engineering Lead who owns engineering and operational readiness, including UAT and parallel run evidence, minimum performance thresholds, guardrail validation, privacy and access control checks, rollback criteria, kill switch readiness, runbooks, and approval records.
In collaboration with Information Security conduct failure mode testing for prompt injection, data leakage, unsupported claims, policy deviations, tool misuse, unsafe write back, inappropriate automation, model degradation, and other AI specific operational risks.
Monitor production quality using evaluation results, model and prompt versions, drift, latency, user feedback, exception rates, override patterns, incidents, and token or platform cost, and define the triggers for investigation, rollback, retraining, prompt changes, or workflow redesign.
Create defect taxonomies and root cause practices that distinguish model, retrieval, prompt, data, rules, integration, user experience, and process issues, and work with product and engineering teams to prioritize durable fixes.
Evolve the function from hands on evaluation of the first use cases to a federated operating model, with reusable scorecards, automated continuous evaluation, trained squad level QA practices, oversight of AI Quality Analysts as volume grows, independent model and vendor benchmarking, and audit ready quality evidence.

The Experience You Need

Bachelor’s degree in computer science, engineering, data science, statistics, mathematics, information systems or a related discipline.
7 plus years of progressive experience in quality engineering, software testing, model validation, machine learning, data science, risk analytics, AI governance, or related technology disciplines.
Experience designing evaluation frameworks and release criteria for production AI or generative AI systems, including RAG, document intelligence, agents, model and prompt lifecycle management, observability, and human in the loop controls.
Strong analytical and technical skills with Python, SQL, test automation, data analysis, statistical methods, and evaluation tooling such as MLflow, Ragas, DeepEval, or comparable frameworks.
Working knowledge of accuracy and retrieval metrics, confidence scoring, sampling, experiment design, regression testing, drift monitoring, adversarial testing, red teaming, guardrails, privacy, security, and operational risk controls.
Ability to translate business policy and risk appetite into measurable acceptance criteria, and to explain quality, risk, and trade off implications clearly to technical, business, control, audit, and senior stakeholders.
Experience building or scaling a quality capability through reusable standards, automated test assets, continuous monitoring, governance evidence, coaching, and leadership of analysts or cross functional quality communities, including exposure to model risk, credit risk, or independent validation environments.
Experience in financial services, mortgage lending, lending operations, servicing, broker channels, third party partnerships, or regulated technology environments is preferred.

Relationships

External: Engages implementation partners, cloud providers, data and AI platform providers, model providers, and specialist vendors on evaluation methods, independent benchmarking, quality evidence, and knowledge transfer to First National.

Internal: Works within the AI Platform Squad while maintaining assessment independence from the delivery teams whose work it gates.

Working Environment And Physical Demands Analysis

Office environment
Periods of high volume with tight timelines
Long periods of stationary position/sitting
Prolonged periods of repetitive movement (i.e. using a keyboard and mouse)
Long periods of time in viewing a computer screen
Multi-tasking may include speaking to customers on a telephone call while looking up information on a computer program.

Why join First National?

Competitive Compensation
Comprehensive benefits program (i.e., Health Spending Account, Maternity and Parental Leave Top Up)
Extensive training programs to set our employees up for success
Modern office environment conducive to collaboration
Supportive teamwork culture
Opportunities to give back to the communities and work through events focused on a variety of charities
Ongoing social events throughout the year

The Team You’ll Join

Founded in 1988, First National is one of Canada’s largest non-bank lenders. We provide residential mortgages exclusively through the mortgage broker channel and we are Canada’s largest commercial mortgage lender.

First National has been consistently recognized as a great place to work and we are proud that our employee engagement feedback is higher than our industry partners.

We would like to thank all applications for their interest in this existing vacancy, but only candidates selected for an interview will be contacted.

Artificial Intelligence is not used in our recruitment or hiring process for this role.

First National is proud to be an equal opportunity employer and is committed to diversity and inclusion regardless of race, color, religion, national origin, age, gender identity, physical or mental disability, sexual orientation and any other category protected by law.

First National supports requests for accommodation from applicants with disabilities; please contact Human Resources at accessibility@firstnational.ca should you need an accommodation at any point in the recruitment process.

#FNLOON

Apply About employer

More about this job

Responsibilities

Own the AI Factory’s evaluation strategy, quality standards, release gates, and ongoing monitoring, translating business policies and risk appetite into measurable acceptance criteria for AI-enabled mortgage workflows. Lead evaluation methods and production quality oversight, including test suites, guardrails, failure analysis, release evidence, and scalable quality practices across product squads.

Requirements

Requires a bachelor’s degree in a relevant technical or quantitative discipline and at least seven years of progressive experience in quality engineering, AI, machine learning, data science, risk, or related fields. Candidates should have production AI evaluation experience, strong Python, SQL, testing, and statistical skills, and the ability to translate business and risk requirements into measurable standards; financial services experience is preferred.

Skills

AI Evaluation Strategy
Quality Engineering
Python
SQL
Test Automation
Generative AI
Retrieval-Augmented Generation
Document Intelligence
Model Validation
Statistical Analysis
Evaluation Frameworks
Adversarial Testing
Drift Monitoring
Risk Management
AI Governance
Stakeholder Communication

Benefits

Bonus
Health Spending Account
Maternity and Parental Leave Top-Up
Training Programs
Career Advancement Opportunities

Visa sponsorship

Not detected in the job text

Keywords

AI Factory
Generative AI
Retrieval-Augmented Generation
Document Intelligence
AI Agents
Python
SQL
MLflow
Ragas
DeepEval
Evaluation Harnesses
Golden Datasets
Adversarial Testing
Regression Testing
Human-in-the-Loop Controls
Continuous Evaluation
Model Validation
Prompt Lifecycle Management
Observability
Drift Monitoring
Guardrails
Red Teaming
Statistical Methods
Experiment Design
Mortgage Lending
Residential Underwriting
Model Risk
Credit Risk
Privacy and Access Controls
Release Readiness

Report this job