Closed listing
QE / SDET – AI Chatbot & GenAI Validation
- ON, Canada
- Ontario, Canada, Canada
Full-time
This opening has closed.
The original listing is kept below for reference.
Job details
- Job type
- Full-time
- Closing date
- Experience
- Core QA Foundation:5+ years of formal automated software testing experience (SDET).
- Posting language
- English
- Posted
Similar jobs hiring now
Search current openingsOriginal job description
Archived listing. The details below describe a past opening.
Position: QE / SDET – AI Chatbot & GenAI Validation
Location: Onsite- Toronto, ON
Job type: Full-time/Permanent hiring
We are seeking an experienced Quality Engineer (QE) / SDET specializing in AI Chatbot Validation to lead the quality assurance, accuracy testing, and response analysis of our conversational AI platforms [INDEX]. In this role, you will be responsible for evaluating chatbot responses for structural completeness, catching AI hallucination vectors, and executing robust automation regression test suites across APIs and user interfaces [INDEX]. You will work at the intersection of core software testing and advanced prompt engineering to ensure a flawless, reliable user experience.
Key Responsibilities
- GenAI & Chatbot Validation: Evaluate and validate conversational AI outputs for contextual accuracy, semantic relevance, formatting bugs, and deep domain knowledge gaps.
- Hallucination & Bias Defection: Identify, document, and triage complex AI failures including hallucinations, guardrail bypasses, incorrect logic paths, and inconsistent formatting.
- Automation Engineering: Develop, scale, and maintain automated UI and API regression test scripts utilizing modern Python automation stacks (e.g., Playwright, Selenium, PyTest).
- Conversation Analysis: Perform detailed evaluation of multi-turn user conversation transcripts to recommend fine-tuning adjustments and prompt optimization metrics.
- Cross-Functional Collaboration: Partner with prompt engineers, AI developers, product owners, and content curators to isolate root causes of bad responses and track defect resolutions through Agile frameworks.
Required Technical Skill Set
- Core QA Foundation:5+ years of formal automated software testing experience (SDET).
- AI Ecosystem Experience: Direct background validating LLMs, GenAI models, virtual assistants, or intelligent conversational agents.
- Prompt Engineering Awareness: Strong conceptual understanding of prompt optimization, system instructions, and how prompt variations impact chatbot generation logic.
- Automation Tooling: Hands-on automation scripting experience using Python combined with PyTest, Playwright, Selenium, or Postman (for API testing).
- Agile Ecosystem: Everyday usage of JIRA, Confluence, GitHub, and continuous integration pipelines
More about this job
Responsibilities
Validate conversational AI responses for accuracy, relevance, formatting, hallucinations, bias, and guardrail failures, and analyze multi-turn conversations to identify quality issues. Build and maintain automated UI and API regression tests, collaborate with AI and product teams to resolve defects, and recommend prompt and fine-tuning improvements.
Requirements
Requires at least five years of formal automated software testing experience and direct experience validating LLMs, generative AI models, virtual assistants, or conversational agents. Candidates should have Python automation experience with tools such as PyTest, Playwright, Selenium, or Postman, understand prompt engineering concepts, and use JIRA, Confluence, GitHub, and CI pipelines.
Skills
- Automated Software Testing
- SDET
- AI Chatbot Validation
- LLM Validation
- Generative AI Testing
- Python
- PyTest
- Playwright
- Selenium
- Postman
- API Testing
- UI Regression Testing
- Prompt Engineering
- Conversation Analysis
- JIRA
- Agile
Visa sponsorship
Not detected in the job text
Categories
- Software
- Technology
- Engineering
Keywords
- Quality Engineering
- Automated Software Testing
- AI Chatbot Validation
- Generative AI
- Large Language Models
- Virtual Assistants
- Conversational AI
- Hallucination Detection
- Bias Detection
- Guardrails
- Prompt Engineering
- Prompt Optimization
- Python
- PyTest
- Playwright
- Selenium
- Postman
- API Testing
- UI Testing
- Regression Testing
- Conversation Analysis
- Fine-Tuning
- JIRA
- Confluence
- GitHub
- Continuous Integration
- Agile