Senior QA Specialist / QA Lead (AI Testing Focus)

Client: City of Toronto (Technology Services)

Job Title: Systems Testing/QA Specialist – Senior (AI Testing)

Location: Hybrid: 3 Days onsite | Metro Hall, 55 John Street

Engagement: Contract Full-Time

Contract Duration: 130.00 Days (Potential Extension)

Job Description:

Must Have:

• Minimum of 3 years of solid experience in AI-assisted testing with hands-on experience in developing AI-powered QA agents using agentic AI, LLM technologies and chatbot applications to create and execute test cases.
• Minimum of 10+ years of experience performing testing for large-scale applications, including on-premise web solutions, cloud-based integrations, and mobile platforms.
• Extensive hands-on experience with all layers of testing: UI, Integration, Regression, Performance, AI-enabled systems, and User Acceptance Testing - across web-based, mobile, and cloud platforms.
• Proven ability to execute AI testing activities utilizing specialized tools such as DeepEval, Applitools, Promptfoo, SaolaAI, Reflect AI, Agentic AI, Copilot Studio Kit, etc.

NOTE: HM is looking for strong AI testing experience.

Public Sector Experience: Preferred

References are required

Responsibilities:

• The Quality Assurance Specialist (QA lead) is responsible for performing application testing tasks.
• The Quality Assurance activities including Functional Testing, Automation Testing, and various AI testing as described below.
• The Quality Assurance Specialist (QA lead) will participate in planning and estimating the activities required to test the new solutions and enhanced applications. He/She will participate in the development of Master Test Plans, Test
• Cases, Test Scripts, Test Completion Reports, etc. while providing advice to business users for the development of Test Cases for User Acceptance Test.
• Executing the tests as required, performing issue tracking and resolution.
• The Quality Assurance Specialist (QA Lead) plays a critical role in ensuring the delivery of high-quality software solutions by designing, executing, and managing testing activities throughout the software development lifecycle.
• The QA Specialist will work collaboratively with cross-functional teams to validate system functionality, performance, and usability, and ensure alignment with business requirements and quality standards.
• Quality Assurance Leadership
• Lead testing efforts for assigned projects, including planning, coordination, execution and oversight of QA activities.
• Assess risks to quality based on verification and validation results and provide recommendations for mitigation.
• Implement and maintain quality processes that support development and production change management.
• Test Design & Planning
• Design comprehensive manual and automated testing processes, including test plans, test cases, and test scripts.
• Develop functional, system, regression, integration, load, and performance test scenarios based on project requirements.
• Collaborate with development leads and business analysts to design and create test data to support test execution.
• Test Execution & Management
• Perform various types of testing manually and/or using automation tools, including:
• Functional Testing
• Regression Testing
• Integration Testing
• Usability/Accessibility Testing
• Performance Testing
• Perform the following types of AI testing activities:
• Functional AI Testing
• Response accuracy
• Hallucination testing
• Prompt testing
• AI-Security and privacy testing (prompt injection)
• AI-Safety Testing (inappropriate behaviours)
• Execute test cycles, monitor progress, and manage defect resolution and issue tracking.
• Maintain detailed records of test results and discrepancies using designated QA tools.
• Coordination & Collaboration
• Work closely with project managers, implementation team, system administrators, architects, vendors (as needed) and database administrators to ensure testing activities are aligned with project timelines and technical requirements.
• Lead and assist business users in conducting User Acceptance Testing (UAT), providing guidance and support as needed.
• Reporting & Communication
• Manage the full-cycle process for documenting and communicating test discrepancies and defects to appropriate stakeholders.
• Provide regular updates on testing progress, issues, and outcomes to project teams and management
• Develop and build a Test Completion Report summarizing all testing activities and results for Gating purposes

Qualifications:

• Post-secondary degree in Computer Science, Electronic Engineering, or a related discipline.
• Minimum of 10+ years of experience performing testing for large-scale applications, including on-premise web solutions, cloud-based integrations, and mobile platforms.
• Minimum of 3 years of solid experience in AI-assisted testing with hands-on experience in developing AI-powered QA agents using agentic AI, LLM technologies and chatbot applications to create and execute test cases.
• Experience using AI tools to increase testing coverage, negative testing, regression testing with knowledge of AI-assisted automation for CI/CD pipelines.
• Experience leading and coordinating testing activities in collaboration with project with project teams and a strong understanding of the various delivery methodologies such as: SDLC - Waterfall, Agile (Scrum), Hybrid, and DevOps.
• Certification in Software Quality Engineering or an equivalent credential is considered a strong asset.
• Excellent analytical skills in business processes, information systems, information management, and web UI design and accessibility.
• Strong written and verbal communication skills, with the ability to collaborate effectively across multidisciplinary teams.

Technical Capabilities:

• Extensive hands-on experience with all layers of testing: UI, Integration, Regression, Performance, AI-enabled systems, and User Acceptance Testing - across web-based, mobile, and cloud platforms.
• Proven ability to execute AI testing activities utilizing specialized tools such as DeepEval, Applitools, Promptfoo, SaolaAI, Reflect AI, Agentic AI, Copilot Studio Kit, etc.
• Skilled in implementing automated testing using tools such as Micro Focus UFT, TOSCA, Selenium, Cucumber, as well as conducting performance testing with JMeter and OpenText Load Runner.
• As well as experience with platforms such as: ServiceNow, Salesforce, SAP, or equivalent are considered an asset
• Strong experience in executing test cycles, analyzing results, and managing the full defect lifecycle using tools like OpenText ALM, OpenText Octane, JIRA, Visual Studio, Confluence, etc.
• Demonstrated capability in analyzing business requirements and user stories to design test scenarios, develop test cases and scripts, prepare test data, and assess results for quality assurance.