Jobs in Canada

25+ jobs

Email me new jobs Save this search

Added 2 mins ago
CI
Lead Engineer, Active Equities, Public Markets Technology
CPP Investments | Investissements RPC
Toronto, ON
- Full-time
As Lead Software Engineer, Active Equities Technology, you will provide technical leadership in designing, developing, deploying, and supporting technology solutions that enable the Active…
Added 2 mins ago
CI
Principal Engineer, Risk and TFM Technologies
CPP Investments | Investissements RPC
Toronto, ON
- Full-time
As Principal Engineer, Risk & Total Fund Management Technology, you will define architectural direction and elevate engineering excellence across critical investment technology platforms…
Added 2 mins ago
EE
Engineer, Energy Services
Envari Energy Solutions
Hawthorne, NS · Ottawa, ON
- Full-time
Under the general direction of a Supervisor in Energy Services, the Energy Services Engineer is responsible for providing technical and project management support for Envari’s growing…
Added 2 mins ago
EE
Electrical Engineer
Envari Energy Solutions
Hawthorne, NS · Ottawa, ON
- Full-time
The Engineer provides internal and customer facing strategic and tactical advice related to energy infrastructure products and services with a focus on innovation and customer value. The…
Added 2 mins ago
Principal Offensive Security
Varicent
Toronto, ON
We're looking for a Principal Offensive Security engineer to be the senior technical authority for our offensive security practice across applications, cloud environments, enterprise…
Added 2 mins ago
Software Development Engineer II, WW Prime Tech
Amazon
Vancouver, BC
…would actually be worth to them. We want a Software Development Engineer who enjoys large-scale distributed systems and personalization work, where the outcome is measured in members…
Added 2 mins ago
AG
Field Engineer in Training - Electrical
Artemis Gold - Blackwater Project
BC
- Temporary
· Demonstrate effective safety leadership by addressing and enforcing BWG safety standards, procedures, and safety systems.
Added 2 mins ago
Senior Software Development Engineer, AWS Eventbridge
Amazon
Vancouver, BC
We're looking for a senior engineer who can own the design and delivery of large, complex systems and raise the technical bar across the team. As a Senior Software Development Engineer,…
Added 2 mins ago
TT
Structural Vibration Engineer
Thornton Tomasetti
Mississauga, ON
- Full-time
Thornton Tomasetti is seeking a Senior Engineer to join our Mississauga-based Acoustics, Noise & Vibration team. This role is primarily focused on structural vibration assessment, design,…
Added 13 mins ago
HS
Senior Integration Engineer (Boomi/Workato)
HeartCentrix Solutions
ON · Ontario, Canada
- Contract
We are seeking a Senior Integration Engineer with deep expertise in Boomi and Workato, specializing in payroll and HR system integrations. This role will focus on designing, developing, and…
Added 13 mins ago
BS
Application Delivery Network Engineer
Brainhunter Systems Ltd
Canada
- Contract
Job-Role/Title: Senior Network Migration & Application Connectivity Engineer (Technology Transformation Program - Applications Network Engineer)
Added 13 mins ago
BC
Director, Data Centres and New Large Loads - Power Market
BBA Consultants
Montreal, QC
- Full-time
…to us for our expertise in grid interconnections and power systems. Going forward, BBA will take a deliberate and proactive approach to this market by establishing a dedicated data centre…
Added 13 mins ago
R&
Staff Engineer / AI Builder
Robots & Pencils
Canada
- Full-time
We’re looking for a Staff AI Builder to lead the design and delivery of AI/ML systems. This role is ideal for an experienced engineer who thrives on architectural decisions, can confidently…
Added 13 mins ago
OS
Senior Forward Deployed Engineer (FDE)
Open Systems Technologies
Toronto, ON
- Contract
We are looking for Senior Forward Deployed Engineer - Spread Products (Financing & Securitization). These are Hybrid contract to Hire roles in Toronto,DT. If you have the relevant skills…
Added 16 mins ago
BC
Director, Data Centres and New Large Loads - Power Market
BBA Consultants
Toronto, ON
- Full-time
…to us for our expertise in grid interconnections and power systems. Going forward, BBA will take a deliberate and proactive approach to this market by establishing a dedicated data centre…
Added 16 mins ago
A
Cloud Software Engineer
Actalent
Ottawa, ON
- $88 to $111 CAD / hour
- Contract
Experience working with telemetry, automotive diagnostics, connected-vehicle data, manufacturing information, or engineering lifecycle systems.
Added 16 mins ago
BC
Director, Data Centres and New Large Loads - Power Market
BBA Consultants
Quebec, QC
- Full-time
…to us for our expertise in grid interconnections and power systems. Going forward, BBA will take a deliberate and proactive approach to this market by establishing a dedicated data centre…
Added 16 mins ago
BC
Director, Data Centres and New Large Loads - Power Market
BBA Consultants
Vancouver, BC
- Full-time
…to us for our expertise in grid interconnections and power systems. Going forward, BBA will take a deliberate and proactive approach to this market by establishing a dedicated data centre…
Added 16 mins ago
P
Solution Architect / Lead EBO Engineer – BAS (SABAS003)
Potenco
Toronto, ON · Greater Toronto Area, Canada
- Full-time
Solution Architect / Lead EBO Engineer – Building Automation Systems
Added 16 mins ago
Y
Full Stack Engineer
Yochana
Montreal, QC
- Full-time
…across the full stack. You'll work on both frontend and backend systems, collaborating closely with product, QA, and DevOps to deliver reliable, scalable…
Added 16 mins ago
AC
Power Supply & Traction Power Engineer
Amaris Consulting | Part of Mantu
Toronto, ON · Greater Toronto Area, Canada
- Full-time
…power generation - with hands-on expertise in Traction Power Systems (IDD, ACDD) and Power…
Added 16 mins ago
QW
Python Selenium AWS Automation Engineer
Quantum World Technologies Inc.
Toronto, ON
- Contract
We are looking for an experienced Python Automation Engineer with strong expertise in Selenium, AWS, and test automation to join our team in Toronto. The ideal candidate will have 7+ years…
Added 16 mins ago
JW
Software Engineer II, Platform
Jobs Web3
Toronto, ON
- Full-time
As a SE II Engineer, Platform you will play a critical role helping to advance Ripple's production infrastructure and release pipelines. Infrastructure Engineers at Ripple are a hybrid of…
Added 16 mins ago
D
AI Engineer – Agentic AI
DataPattern
Mississauga, ON
- Contract
We are looking for an experienced AI Engineer with strong expertise in Agentic AI, Generative AI, Microsoft AI technologies, RAG, and AI agent development.
Added 16 mins ago
JW
Software Engineer, Web3 Backend
Jobs Web3
Toronto, ON
- Full-time
As a Software Engineer, you’ll build and own backend services, infrastructure, and data systems that support Robinhood’s next-generation crypto, web3, and wallet offerings. You’ll develop…

HCLTech

LLM Evaluation Engineer

Canada

Full-time

Apply About employer

Applications open in a new tab.

Job details

Job type: Full-time
Posting language: English
Posted: Oct 7, 2026

Original job description

Overall – 8+ | AI Exp – 3+

Hands-on proficiency with libraries like Ragas, DeepEval, TruLens, Promptfoo, or OpenAI Evals to measure:
Context Precision & Context Recall (retrieval quality from vector stores).
Faithfulness / Groundedness (checking against hallucination in bounded context).
Answer Relevance and Toxicity / Guardrail Compliance.
Analyzing evaluation datasets, confusion matrices, similarity score distributions, and benchmark trends.
Computing classification metrics (Precision, Recall, F1, Cosine Similarity thresholds) for retrieval systems.
Azure OpenAI SDK / OpenAI API and LangChain / LlamaIndex for building automated LLM-as-a-judge scoring scripts and synthetic test dataset generation.
PyTest / Unittest: Writing automated unit/regression tests for prompt templates and model version upgrades.
Writing complex SQL queries to extract, curate, and version golden test datasets from relational databases (e.g., PostgreSQL, Azure SQL).
Querying vector stores and telemetry databases (e.g., pgvector, Splunk/log tables) to evaluate real-world candidate/user query retrieval accuracy and fallback rates.
Handling structured benchmark datasets and prompt configurations using JSON / JSONL, YAML, and CSV.

Apply About employer

More about this job

Responsibilities

Build automated evaluation and scoring workflows for LLMs, retrieval systems, and prompt configurations, including synthetic test dataset generation and regression testing. Analyze benchmark datasets, retrieval performance, and production telemetry to assess answer quality, hallucinations, guardrail compliance, and fallback rates.

Requirements

Requires 8+ years of overall experience and 3+ years of AI experience, with hands-on proficiency in LLM evaluation frameworks and APIs, orchestration libraries, and automated testing tools. The role also requires experience with evaluation metrics, complex SQL queries, vector stores, telemetry data, and structured benchmark formats such as JSON, YAML, and CSV.

Skills

LLM Evaluation
Ragas
DeepEval
TruLens
Promptfoo
OpenAI Evals
Retrieval Quality Evaluation
Faithfulness and Groundedness Evaluation
Classification Metrics
Azure OpenAI SDK
OpenAI API
LangChain
LlamaIndex
PyTest
SQL
Vector Stores

Visa sponsorship

Not detected in the job text

Keywords

LLM Evaluation
Ragas
DeepEval
TruLens
Promptfoo
OpenAI Evals
Context Precision
Context Recall
Faithfulness
Groundedness
Answer Relevance
Toxicity
Guardrail Compliance
Confusion Matrices
Precision
Recall
F1 Score
Cosine Similarity
Azure OpenAI SDK
OpenAI API
LangChain
LlamaIndex
LLM-as-a-Judge
Synthetic Test Data
PyTest
Unittest
SQL
PostgreSQL
Azure SQL
pgvector

Report this job