Jobs in Canada

Save this search

Added 6 days ago
GenAI Test Engineer
J&M Group
Halifax, NS
Full-time
Python-based API and test automation, including PyTest or Robot Framework
Added 6 days ago
Data Engineer
SPECTRAFORCE
Toronto, ON
Contract
Build solutions using Python, SAS, and SQL for data processing and automation.
Added 6 days ago
Data Platform Engineer
Apexon
Toronto, ON
Full-time
We're looking for a Senior Backend Platform Engineer with strong Python development experience and a solid background in building cloud-native applications, microservices, and REST APIs.…
Added 6 days ago
Full Stack Developer
Capco
Toronto, ON
We are looking for an experienced Senior Full-Stack Python Software Engineer to design and build high-quality digital solutions using Python web frameworks and modern front-end technologies.
Added 6 days ago
Quality Assurance Automation Engineer
Hays
Halifax, NS
Contract
We are seeking a Senior GenAI & Test Engineer to drive quality assurance for AI-powered applications, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) solutions,…
Added 6 days ago
Full Stack Engineer
Stelvio Inc.
Saskatoon, SK
Full-time
Knowledge of SQL and/or NoSQL database technologies.
Added 7 days ago
Data Engineer
BALLY’S INTRALOT SA
Toronto, ON
Full-time
Utilize Python and SQL for data analysis, backend development, and data manipulation.
Added 12 days ago
Intermediate Machine Learning Engineer
Aviva Canada
Markham, ON
Full-time
We are looking for a talented and motivated Intermediate Machine Learning Engineer to join our AI/ML Operations & Platform team. The ideal candidate will have hands-on experience in…
Added 12 days ago
Data Engineer Co-op / Intern (Winter 2027)
TD
Toronto, ON
Full-time
Familiarity with at least one of the following technologies: Java, Scala, Python, Pyspark, or SQL
Added 12 days ago
Data Engineer
J&M Group
Toronto, ON
Full-time
An experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should possess hands-on…
Added 12 days ago
Big Data/ Python Data Engineer (Hadoop Ecosystem)
Q1 Technologies, Inc.
Toronto, ON
$60 to $70 CAD / hour
Contract
An experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should possess hands-on…
Added 13 days ago
Lead Engineer - Data
ZS
Toronto, ON
Full-time
Proficiency in SQL, Python, or other relevant programming languages.
Added 13 days ago
Data Engineer
CDAI
Toronto, ON
Full-time
Design, build, and maintain scalable ETL/ELT data pipelines using Python, SQL, Airflow, Spark that hosted in AWS environment.
Added 13 days ago
Data Engineer
Elisa
Engineer, BC
Full-time
Our technology stack includes AWS, Databricks, Python, SQL, Spark, Delta Lake/Iceberg, GitHub Actions, and modern observability tooling.
Added 16 days ago
Business Intelligence Engineer II, Talent Acquisition Analytics
Amazon
Vancouver, BC
Full-time
As a Business Intelligence Engineer, this is your opportunity to be at the intersection of data, analysis, hiring, and strategy that will make it possible for our teams to fulfill that…
Added 16 days ago
Associate Data Engineer
Punt
Vancouver, BC
$70,000 to $90,000 CAD / year
Full-time
Above all, we value strong SQL fundamentals, working Python proficiency, and the instinct to double-check your output for correctness before someone else has to.
Added 16 days ago
Staff AI Engineer
Robots & Pencils
Canada
Full-time
Build and maintain backend services using Python and Node.js, including serverless architectures (e.g., AWS Lambda, API Gateway)
Added 16 days ago
Senior Software Engineer
PointClickCare
Mississauga, ON
$142,000 to $158,000 CAD / year
Full-time
At PointClickCare our mission is simple: to help providers deliver exceptional care. And that starts with our people. As a leading health tech company that’s founder-led and privately held,…
Added 16 days ago
ML Ops Engineer
Ampstek
Concord, ON
Contract
Strong proficiency in Python, SQL, and ML libraries (e.g., scikit-learn, XGBoost, TensorFlow, PyTorch).
Added 12 days ago
Lead Data Scientist- Canada
Very LLC
Remote, Canada
$100 CAD / hour
Contract · Remote
Core: Python at a production engineering level, PyTorch, the SciPy stack, Git and GitHub Actions, agentic AI development (MCP servers, LLM APIs, typed tool design)
Added 17 days ago
Senior Full Stack Gen AI Engineer/Technical Engagement Manager
Straive
Calgary, AB
Full-time
We are seeking a Senior Full Stack AI Engineer who can combine strong hands-on software engineering with technical engagement and project delivery leadership. The successful candidate will…
Added 12 days ago
Intermediate / Senior Test Engineer
Netskrt Systems Inc.
BC
$90,000 to $120,000 CAD / year
Full-time
Netskrt is looking for a talented, results-driven Intermediate/Senior Test Engineer to join the Netskrt team. We are a highly motivated team dedicated to delivering products and services…
Added 12 days ago
Software Engineer (Full Stack)
Synpulse
Toronto, ON
We seek a skilled and experienced Software Engineer (mid-level) to join our dynamic team. You will play a pivotal role in the design, development, and implementation of software solutions…
Added 13 days ago
Intermediate Machine Learning Engineer
Aviva
Markham, ON
We are looking for a talented and motivated Intermediate Machine Learning Engineer to join our AI/ML Operations & Platform team. The ideal candidate will have hands-on experience in…
Added 17 days ago
Cloud Engineer
HCLTech
Canada
Full-time
We are looking for an experienced Cloud Engineer to join our Cloud & Platform Engineering team. In this hands-on role, you will design, build, and operate the enterprise Azure platform and…

J&M Group

GenAI Test Engineer

Halifax, NS, Canada

Job type: Full-time
Apply by: Mar 13, 2027
Posting language: English
Posted: Sep 14, 2026

Apply About employer

Applications open in a new tab.

Original job description

GenAI Testing: LLM, RAG, Agentic AI, API and Test Automation

What are the top 3 skills required for this role?

GenAI, LLM, RAG and agentic AI testing
Python-based API and test automation, including PyTest or Robot Framework
LLM evaluation, prompt-injection and hallucination testing, CI/CD and observability

Additional Information

Team size, direct reports, key deliverables, unique selling points, additional qualifications, team culture etc. Key Responsibilities

Develop and execute comprehensive test strategies for GenAI applications, including LLM-powered solutions, RAG pipelines, chatbots, copilots and agentic AI workflows.

Validate functional and non-functional requirements across prompts, model responses, APIs, user interfaces, integrations and end-to-end business workflows.

Create reusable evaluation datasets, test cases and scoring rubrics for relevance, groundedness, factual accuracy, completeness, coherence, instruction adherence and response consistency.

Test RAG solutions by evaluating retrieval quality, context precision and recall, source attribution, answer faithfulness and resistance to unsupported or hallucinated responses.

Validate agentic AI behaviour, including tool selection, function calling, reasoning flows, multi-step task completion, memory handling, fallback paths and failure recovery.

Perform adversarial, security and safety testing for prompt injection, jailbreaks, sensitive-data exposure, toxic or prohibited content, authorization bypass and insecure tool use.

Conduct performance, scalability and reliability testing for response latency, throughput, token usage, cost, concurrency, rate limits and degradation under load.

Build automated regression suites in Python and integrate GenAI evaluations, API tests and quality gates into CI/CD pipelines.

Monitor production quality through traces, evaluation metrics, user feedback and drift indicators; identify defects, analyse root causes and support remediation.

Collaborate with product, development, data science, security and business teams to define acceptance criteria, test coverage and release-readiness standards.

Required Qualifications And Skills

Strong experience in software quality engineering, test strategy, test design, defect management and automation for enterprise applications.

Hands-on knowledge of GenAI concepts, LLM behaviour, prompt engineering, embeddings, vector databases, RAG architectures and agentic AI systems.

Proficiency in Python and experience with automation frameworks such as PyTest or Robot Framework, together with API testing tools and libraries.

Experience evaluating LLM outputs using deterministic checks, semantic similarity, model-based evaluation and human-review workflows.

Practical experience testing hallucinations, bias, robustness, non-deterministic behaviour, prompt injection, guardrails and content-safety controls.

Working knowledge of GenAI evaluation and observability tools such as Ragas, DeepEval, Promptfoo, LangSmith, Arize Phoenix or equivalent platforms.

Strong SQL skills and experience validating source data, document ingestion, chunking, metadata, embeddings and retrieval results.

Experience with GitHub, GitHub Actions or comparable CI/CD tools; familiarity with Docker, Kubernetes, AWS or Azure is desirable.

Ability to define measurable acceptance thresholds, analyse evaluation results and communicate quality risks clearly to technical and business stakeholders.

Strong analytical, investigative and problem-solving skills, with attention to responsible AI, privacy, security and regulatory expectations.

Preferred Qualifications

Bachelor s or Master s degree in Computer Science, Software Engineering, Data Science or a related field, with at least five years of experience in quality engineering, test automation, data testing, AI/ML testing or a related discipline.

Experience testing applications in banking or another regulated environment, including model governance, audit evidence and responsible-AI controls, is an asset.

Apply About employer

More about this job

Responsibilities

Develop and execute comprehensive test strategies for GenAI applications, including LLM-powered solutions, RAG pipelines, and agentic AI workflows. Build automated regression suites in Python and integrate quality gates into CI/CD pipelines to monitor production quality.

Requirements

Requires strong experience in software quality engineering and hands-on knowledge of GenAI concepts, Python automation frameworks, and LLM evaluation tools. A degree in Computer Science or a related field with at least five years of experience is preferred.

Skills

GenAI Testing
LLM Evaluation
RAG
Agentic AI
Python
PyTest
Robot Framework
API Testing
Prompt Engineering
CI/CD
SQL
Vector Databases
Adversarial Testing
Observability Tools
GitHub Actions
Docker

Visa sponsorship

Not detected in the job text

Keywords

GenAI
LLM
RAG
Agentic AI
Python
PyTest
Robot Framework
API Testing
Prompt Injection
Hallucination Testing
CI/CD
Observability
Ragas
DeepEval
Promptfoo
LangSmith
Arize Phoenix
SQL
Vector Databases
GitHub Actions
Docker
Kubernetes
AWS
Azure
Semantic Similarity
Model-based Evaluation
Responsible AI
Banking
Model Governance
Software Quality Engineering

Report this job