LLM Evaluation Engineer
- Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Posting language
- English
- Posted
Original job description
Overall – 8+ | AI Exp – 3+
- Hands-on proficiency with libraries like Ragas, DeepEval, TruLens, Promptfoo, or OpenAI Evals to measure:
- Context Precision & Context Recall (retrieval quality from vector stores).
- Faithfulness / Groundedness (checking against hallucination in bounded context).
- Answer Relevance and Toxicity / Guardrail Compliance.
- Analyzing evaluation datasets, confusion matrices, similarity score distributions, and benchmark trends.
- Computing classification metrics (Precision, Recall, F1, Cosine Similarity thresholds) for retrieval systems.
- Azure OpenAI SDK / OpenAI API and LangChain / LlamaIndex for building automated LLM-as-a-judge scoring scripts and synthetic test dataset generation.
- PyTest / Unittest: Writing automated unit/regression tests for prompt templates and model version upgrades.
- Writing complex SQL queries to extract, curate, and version golden test datasets from relational databases (e.g., PostgreSQL, Azure SQL).
- Querying vector stores and telemetry databases (e.g., pgvector, Splunk/log tables) to evaluate real-world candidate/user query retrieval accuracy and fallback rates.
- Handling structured benchmark datasets and prompt configurations using JSON / JSONL, YAML, and CSV.
More about this job
Responsibilities
Build automated evaluation and scoring workflows for LLMs, retrieval systems, and prompt configurations, including synthetic test dataset generation and regression testing. Analyze benchmark datasets, retrieval performance, and production telemetry to assess answer quality, hallucinations, guardrail compliance, and fallback rates.
Requirements
Requires 8+ years of overall experience and 3+ years of AI experience, with hands-on proficiency in LLM evaluation frameworks and APIs, orchestration libraries, and automated testing tools. The role also requires experience with evaluation metrics, complex SQL queries, vector stores, telemetry data, and structured benchmark formats such as JSON, YAML, and CSV.
Skills
- LLM Evaluation
- Ragas
- DeepEval
- TruLens
- Promptfoo
- OpenAI Evals
- Retrieval Quality Evaluation
- Faithfulness and Groundedness Evaluation
- Classification Metrics
- Azure OpenAI SDK
- OpenAI API
- LangChain
- LlamaIndex
- PyTest
- SQL
- Vector Stores
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Software
- Data & Analytics
- Engineering
Keywords
- LLM Evaluation
- Ragas
- DeepEval
- TruLens
- Promptfoo
- OpenAI Evals
- Context Precision
- Context Recall
- Faithfulness
- Groundedness
- Answer Relevance
- Toxicity
- Guardrail Compliance
- Confusion Matrices
- Precision
- Recall
- F1 Score
- Cosine Similarity
- Azure OpenAI SDK
- OpenAI API
- LangChain
- LlamaIndex
- LLM-as-a-Judge
- Synthetic Test Data
- PyTest
- Unittest
- SQL
- PostgreSQL
- Azure SQL
- pgvector