Sr Software Quality Engineer (AI Focused) : Toronto ON// FTP

Role: Sr Software Quality Engineer (AI Focused)

Location: Toronto, ON (Onsite)

Experience: 8+ years

Type: Full Time Permanent

Roles and Responsibilities:

• Partner with DevOps, Development, Product, and QE teams to design and execute exploratory, automated, and AI-focused tests that ensure product quality.

• Define quality strategies, test plans, test cases, and test data for functional, integration, system, regression, API, database, and acceptance testing.

• Design, develop, and maintain scalable test automation frameworks using Java, Python, or Groovy.

• Build automated tests for AI-enabled features, including prompt behavior, response accuracy, hallucination risks, safety, reliability, latency, and regression coverage.

• Validate AI integrations, APIs, microservices, data pipelines, model interactions, and third-party services.

• Establish testing approaches for both deterministic and non-deterministic AI behavior, including evaluation datasets, expected behavior, quality thresholds, and repeatability.

• Integrate automated testing and AI quality checks into CI/CD pipelines in partnership with DevOps teams.

• Use tools and frameworks such as Selenium, Cucumber/Gherkin, Playwright, RestAssured, JUnit, TestNG, ReadyAPI, and JMeter.

• Identify, document, prioritize, and track defects through resolution.

• Analyze test coverage, quality trends, AI evaluation results, and production feedback to identify gaps and drive continuous improvement.

• Collaborate with cross-functional Agile teams to deliver reliable, secure, and user-focused product features.

• Communicate test results, risks, quality metrics, and release recommendations clearly to technical and non-technical stakeholders.

Required Skills:

• 5+ years of relevant experience in Quality Engineering, Quality Assurance, test automation, or software development.

• 5+ years of programming experience with Java, Python, Groovy, or a comparable language.

• 5+ years of experience building and maintaining test automation frameworks.

• Hands-on experience with Selenium, Cucumber/Gherkin, Playwright, RestAssured, or similar automation technologies.

• Experience testing functional and non-functional requirements, APIs, integrations, databases, microservices, and regression suites.

• Experience with testing frameworks and tools such as JUnit, TestNG, qTest, ReadyAPI, and JMeter.

• Understanding of TDD, ATDD, BDD, continuous testing, and risk-based testing practices.

• Experience testing AI or machine learning-powered applications, APIs, conversational systems, or generative AI features.

• Understanding of AI quality concepts such as prompt evaluation, grounding, accuracy, relevance, toxicity, bias, hallucination detection, and response consistency.

• Ability to create effective test data and evaluation datasets for AI-driven functionality.

• Strong analytical, problem-solving, communication, and technical documentation skills.

• Ability to work effectively in a fast-paced, cross-functional Agile environment.

Nice to Have:

• Experience with AI evaluation frameworks, LLM testing, prompt engineering, retrieval-augmented generation, embeddings, or model observability.

• Experience with Git, Bitbucket, Jenkins, CI/CD pipelines, Artifactory, SonarQube, Jira, Confluence, AWS, Kubernetes, Maven, or Gradle.

• Experience testing cloud-native applications and distributed microservice architectures.

• Experience with security, performance, accessibility, or reliability testing.

• Experience developing automation scripts and reusable testing utilities.

• Experience with Tableau or other analytics and visualization platforms.

• CSTE certification or an equivalent quality engineering certification.

• Experience working directly with DevOps teams to implement automated quality gates and release controls.