Jobs in Canada

25+ jobs

Email me new jobs Save this search

Added 10 mins ago
CI
Project Manager
Cambium Inc.
Peterborough, ON
- Full-time
Coordinate field programs, laboratory services, data interpretation, technical assessments, and preparation of annual and regulatory reports.
Added 10 mins ago
A
Senior Machine Learning Engineer
Autodesk
Toronto, ON · AMER - Canada - Ontario - Toronto - University Ave
- Full-time
We are looking for an exceptional Machine Learning Engineer to design, build, operationalize, and scale production-grade AI/ML and Agentic AI systems at Autodesk For Go-to-market…
Added 24 mins ago
AI
Full Stack Engineer
Apptoza Inc.
Toronto, ON
- Contract
You will be responsible for working closely with AI data scientists to support data ingestion and feature engineering, establishing industry best practices, maintaining and managing…
Added 24 mins ago
RC
SRE/ Platform Engineer
Realtek Consulting LLC
Toronto, ON
- Contract
Seeking a hands-on engineer to drive the reliability automation and modernization of the enterprise JAMS workload automation platform.
Added 24 mins ago
KG
Enterprise Security Architect - Senior Security Engineer
K&K Global Talent Solutions INC.
Toronto, ON
- Full-time
Role : Enterprise Security Architect - Senior Security Engineer
Added 24 mins ago
D
Senior Product Engineer II - Backend
Docebo
Toronto, ON
- $123,400 to $164,500 CAD / year
- Full-time
As a Senior Product Engineer II, you bring deep expertise in backend engineering alongside strong working competency across the broader stack. You own complex features and technical…
Added 26 mins ago
Forward Deployed Engineer, Application & Infrastructure - Technology Strategy & Transformation
EY
Toronto, ON
Added 26 mins ago
Forward Deployed Engineer, AI - Technology Strategy & Transformation
EY
Toronto, ON
Integrate hosted and open-weight models into enterprise applications, business processes, data sources, APIs, and development workflows.
Added 26 mins ago
H
Senior Electrical Engineer
HDR
Toronto, ON · Ottawa, ON · Richmond Hill, ON · Victoria, BC · Vancouver, BC · Calgary, AB
- Full-time
…where people work, connect, and thrive. From high-performance data centers driving the future of AI to dynamic commercial environments, your work will directly support technological…
Added 26 mins ago
H
Senior Mechanical Engineer
HDR
Calgary, AB
- Full-time
…where people work, connect, and thrive. From high-performance data centers driving the future of AI to dynamic commercial environments, your work will directly support technological…
Added 26 mins ago
H
Senior Project Manager
HDR
Calgary, AB
- Full-time
…where people work, connect, and thrive. From high-performance data centers driving the future of AI to dynamic commercial environments, your work will directly support technological…
Added 26 mins ago
GM
Manufacturing Controls Engineer
General Motors
St. Catharines, ON
- Full-time
…to troubleshoot issues. Increasingly, you will also leverage data and analytics to improve safety, quality, uptime, and…
Added 26 mins ago
Workday Integration Engineer
Asana
Vancouver, BC
…ecosystem - primarily focused on Workday - enabling smooth data flow, system reliability, and automation across our global HR tools. In this role, you will partner closely with HR, Finance,…
Added 38 mins ago
Software Developer
AppDirect
Toronto, ON
You are a strong Software Engineer that is not satisfied in meeting the status quo. You are always digging deeper into your code and solutions while collaborating with your team to create a…
Added 38 mins ago
R
Software Engineer
RBC
Toronto, ON
- Full-time
Solid understanding of relational databases and data modeling
Added 39 mins ago
KC
Senior Embedded Software Designer
Kepler Communications Inc.
Toronto, ON
- Full-time
Our mission is to provide real-time on-orbit connectivity for critical missions, enabling a new era of data-driven intelligence and innovation. With 33 satellites launched to date, Kepler…
Added 39 mins ago
CE
Network Engineer
Capgemini Engineering
Toronto, ON
- Full-time
This is a Hybrid role located in Mississauga, ON and it requires frequent travel to client’s data centers across Canada.
Added 51 mins ago
Platform Engineer - AI Engineering
DRW
Montreal, QC
We are looking for a Platform Engineer to help build and operationalize our AI tooling ecosystem. Working closely with the Team Lead and Senior/Staff SWE, you will be the hands-on…
Added 54 mins ago
M
Senior Full Stack Java Engineer - Workforce Management (WFM)
Miratech
Canada
- Full-time
We are looking for a Senior Full Stack Engineer to join the Workforce Management product stream and build enterprise-scale solution supporting forecasting, capacity planning, scheduling,…
Added 54 mins ago
V
Embedded Software Engineer
V2Soft
Windsor, ON
- Contract
Job Title: Powertrain / Propulsion Controls Software Engineer – AUTOSAR / C++
Added 1 hour ago
Senior Product Solutions Analyst
Scotiabank
Toronto, ON
…business and cross-functional teams to spark new ideas, re-engineer processes, identify opportunities, and shape AI-powered…
Added 1 hour ago
A
Senior Sales Engineer
Ataccama
Toronto, ON
- Full-time
As a Senior Sales Engineer, you will work closely with Account Executives to drive sales engineering activities across the entire deal lifecycle. This role requires adaptability, deep…
Added 1 hour ago
BS
Technical Lead
Brainhunter Systems Ltd
Toronto, ON
- Contract
8+ years building and operating complex enterprise digital products, including experience as a Technical Lead, Staff Engineer or senior full-stack developer.
Added 1 hour ago
Q
Cypress Automation Engineer
Qualitest
Canada
- Full-time
Formerly Qualitest, QualityAI is seeking a highly motivated Cypress Automation Engineer to join our Quality Engineering team. In this role, you will be responsible for designing,…
Added 1 hour ago
M
Senior Programmable Voice Engineer (CX Platform)
Miratech
Canada
- Full-time
We are looking for a Senior Programmable Voice Engineer to develop programmable voice capabilities, call control services, media processing, and API-driven communications infrastructure…

HCLTech

LLM Evaluation Engineer

Canada

Full-time

Apply About employer

Applications open in a new tab.

Job details

Job type: Full-time
Posting language: English
Posted: Oct 7, 2026

Original job description

Overall – 8+ | AI Exp – 3+

Hands-on proficiency with libraries like Ragas, DeepEval, TruLens, Promptfoo, or OpenAI Evals to measure:
Context Precision & Context Recall (retrieval quality from vector stores).
Faithfulness / Groundedness (checking against hallucination in bounded context).
Answer Relevance and Toxicity / Guardrail Compliance.
Analyzing evaluation datasets, confusion matrices, similarity score distributions, and benchmark trends.
Computing classification metrics (Precision, Recall, F1, Cosine Similarity thresholds) for retrieval systems.
Azure OpenAI SDK / OpenAI API and LangChain / LlamaIndex for building automated LLM-as-a-judge scoring scripts and synthetic test dataset generation.
PyTest / Unittest: Writing automated unit/regression tests for prompt templates and model version upgrades.
Writing complex SQL queries to extract, curate, and version golden test datasets from relational databases (e.g., PostgreSQL, Azure SQL).
Querying vector stores and telemetry databases (e.g., pgvector, Splunk/log tables) to evaluate real-world candidate/user query retrieval accuracy and fallback rates.
Handling structured benchmark datasets and prompt configurations using JSON / JSONL, YAML, and CSV.

Apply About employer

More about this job

Responsibilities

Build automated evaluation and scoring workflows for LLMs, retrieval systems, and prompt configurations, including synthetic test dataset generation and regression testing. Analyze benchmark datasets, retrieval performance, and production telemetry to assess answer quality, hallucinations, guardrail compliance, and fallback rates.

Requirements

Requires 8+ years of overall experience and 3+ years of AI experience, with hands-on proficiency in LLM evaluation frameworks and APIs, orchestration libraries, and automated testing tools. The role also requires experience with evaluation metrics, complex SQL queries, vector stores, telemetry data, and structured benchmark formats such as JSON, YAML, and CSV.

Skills

LLM Evaluation
Ragas
DeepEval
TruLens
Promptfoo
OpenAI Evals
Retrieval Quality Evaluation
Faithfulness and Groundedness Evaluation
Classification Metrics
Azure OpenAI SDK
OpenAI API
LangChain
LlamaIndex
PyTest
SQL
Vector Stores

Visa sponsorship

Not detected in the job text

Keywords

LLM Evaluation
Ragas
DeepEval
TruLens
Promptfoo
OpenAI Evals
Context Precision
Context Recall
Faithfulness
Groundedness
Answer Relevance
Toxicity
Guardrail Compliance
Confusion Matrices
Precision
Recall
F1 Score
Cosine Similarity
Azure OpenAI SDK
OpenAI API
LangChain
LlamaIndex
LLM-as-a-Judge
Synthetic Test Data
PyTest
Unittest
SQL
PostgreSQL
Azure SQL
pgvector

Report this job