Software Development Engineer in Test – GenAI / LLM
- Toronto, ON, Canada
Full-time
Applications open in a new tab.
Job details
- Job type
- Full-time
- Apply by
- Posting language
- English
- Posted
Original job description
Role Description
Work Arrangement: 1 day onsite per week
Experience Required: 6–8 years
- Validate chatbot responses for accuracy, relevance, completeness, and consistency.
- Create and execute test scenarios, test cases, and regression test suites.
- Identify hallucinations, incorrect responses, formatting issues, and knowledge gaps.
- Perform chatbot conversation analysis and recommend quality improvements.
- Develop and maintain test automation for chatbot validation where possible.
- Work closely with developers, AI engineers, and content teams to resolve defects.
- Track, document, and report defects, test results, and quality metrics.
- Support UAT, release validation, and production quality monitoring.
- 5 years of strong QA SDET test automation experience.
- Experience testing AI, LLM, GenAI, chatbots, virtual assistants, or conversational platforms.
- Knowledge of prompt engineering concepts.
- Experience with Jira, Confluence, GitHub, and Agile delivery.
- Experience with API testing and automation frameworks.
- Strong analytical and troubleshooting skills.
- Experience with Python, Selenium, Playwright, PyTest, or similar tools.
- Strong communication and documentation skills.
- QA SDET and test automation.
- AI, LLM, GenAI, chatbots, and conversational platforms.
- Prompt engineering concepts.
- API testing and automation frameworks.
- Python, Selenium, Playwright, or PyTest.
- Jira, Confluence, GitHub, and Agile delivery.
- Analytical and troubleshooting skills.
- Communication and documentation skills.
- Digital: Python
- Digital: Chatbots / Conversational Agents
- Generative AI – Quality Engineering
More about this job
Responsibilities
Validate chatbot responses for accuracy, relevance, completeness, and consistency; create and execute test scenarios, cases, and regression suites; and identify hallucinations, defects, formatting issues, and knowledge gaps. Develop chatbot test automation, collaborate with development, AI, and content teams to resolve issues, report quality metrics, and support UAT, release validation, and production monitoring.
Requirements
Requires 6–8 years of experience, including strong QA SDET and test automation experience, plus experience testing AI, LLM, GenAI, chatbot, or conversational platforms. Candidates should know prompt engineering, API testing, Python and relevant automation tools, Jira, Confluence, GitHub, and Agile delivery, and have strong analytical, troubleshooting, communication, and documentation skills.
Skills
- Quality Assurance
- Test Automation
- Generative AI
- Large Language Models
- Chatbot Testing
- Prompt Engineering
- API Testing
- Python
- Selenium
- Playwright
- PyTest
- Jira
- Confluence
- GitHub
- Agile Delivery
Visa sponsorship
Not detected in the job text
Categories
- Software
- Technology
- Engineering
Keywords
- Quality Assurance
- Test Automation
- Generative AI
- Large Language Models
- Chatbots
- Virtual Assistants
- Conversational Platforms
- Prompt Engineering
- API Testing
- Python
- Selenium
- Playwright
- PyTest
- Jira
- Confluence
- GitHub
- Agile Delivery
- Regression Testing
- Test Scenarios
- Test Cases
- Chatbot Conversation Analysis
- Hallucination Detection
- User Acceptance Testing
- Release Validation
- Production Quality Monitoring
- Defect Tracking
- Quality Metrics