Closed listing
AI Tester
- Toronto, ON, Canada
- Greater Toronto Area, Canada, Canada
Contract
This opening has closed.
The original listing is kept below for reference.
Job details
- Job type
- Contract
- Closing date
- Experience
- 2 - 3 years of Experience in AI Tester.
- Posting language
- English
- Posted
Similar jobs hiring now
Search current openingsOriginal job description
Archived listing. The details below describe a past opening.
We have Immediate requirement with our Direct Client as AI Tester @ Toronto ON (Hybrid - 4 Days/Week). if you are interested, Please share your updated resume to nakka.prasad@irissoftware.com
Title: AI Tester
Location: Toronto ON (Hybrid - 3 Days/Week)
Description:
- 2 - 3 years of Experience in AI Tester.
GenAI test specialist to bring deep expertise in testing LLM-powered features. You'll own how we evaluate AI quality, working within the existing test frameworks and CI/CD pipelines rather than building them from scratch.
- What You'll DoDefine the GenAI testing strategy and quality metrics (accuracy, groundedness, hallucination, safety)
- Agentic AI Framework, swarm testing, Agent orchestration – Claude/Devin
- Build LLM evaluation suites, including LLM-as-a-judge, and plug them into our existing automation framework
- Create and maintain golden datasets and adversarial prompt sets
- Test RAG pipelines for retrieval quality, context relevance, and answer faithfulness
- Validate AI agents and tool-calling flows across multi-step scenarios
- Lead red-teaming: prompt injection, jailbreaks, bias, toxicity, PII leakage
- Track model and prompt regressions across releases and share findings with ML and product teams
- Upskill the wider QA team on GenAI testing practices
- What You Bring3+ years in software testing or QA automation, with working Python skills
- Hands-on experience testing LLM or GenAI applications
- Strong grasp of LLM concepts: prompts, embeddings, context windows, temperature, RAG
- Experience with eval tools such as DeepEval, Ragas, Promptfoo, LangSmith, or TruLens
- Comfort with API testing and working inside an existing CI/CD setup
- Nice to HaveLangChain / LlamaIndex and vector database experience
- Knowledge of OWASP Top 10 for LLM Applications or AI red-teaming
- ISTQB CT-AI or similar certification
Satya Prasad Nakka
Iris Software
200 Metroplex Drive, Suite #300
Edison, NJ 08817
Mobile: 732-426-9380
www.irissoftware.com
More about this job
Responsibilities
Define GenAI testing strategies and quality metrics, build LLM evaluation suites, and maintain golden datasets and adversarial prompts within existing automation and CI/CD frameworks. Test RAG pipelines, AI agents, and tool-calling flows; lead red-teaming and regression tracking; and help upskill the QA team.
Requirements
The role requires 3 or more years in software testing or QA automation, working Python skills, and hands-on experience testing LLM or GenAI applications. Candidates should understand LLM concepts such as prompts, embeddings, context windows, temperature, and RAG, and have experience with evaluation tools, API testing, and existing CI/CD setups.
Skills
- GenAI Testing
- LLM Evaluation
- Python
- QA Automation
- RAG Testing
- API Testing
- Prompt Engineering
- Agentic AI
- CI/CD
- Red-Teaming
- Test Strategy
- Golden Datasets
- Adversarial Testing
- AI Safety
- Regression Testing
Visa sponsorship
Not detected in the job text
Hiring contact
Satya Prasad Nakka
Hiring contact email
nakka.prasad@irissoftware.com
Categories
- Technology
- Software
- Data & Analytics
- Finance & Accounting
Keywords
- GenAI
- LLM
- Python
- DeepEval
- Ragas
- Promptfoo
- LangSmith
- TruLens
- LangChain
- LlamaIndex
- Vector Databases
- RAG
- LLM-as-a-Judge
- Agentic AI
- Agent Orchestration
- Claude
- Devin
- CI/CD
- API Testing
- Golden Datasets
- Adversarial Prompts
- Prompt Injection
- Jailbreaking
- AI Red-Teaming
- OWASP Top 10 for LLM Applications
- ISTQB CT-AI
- Embeddings
- Context Windows
- Hallucination
- Groundedness