Quality Assurance Automation Engineer

Role: GenAI Test Engineer

Location: Halifax

Type: Contract (Hybrid)

Role Overview:

We are seeking a Senior GenAI & Test Engineer to drive quality assurance for AI-powered applications, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) solutions, chatbots, copilots, and agentic AI systems. The role focuses on designing comprehensive testing strategies, automating AI validation processes, and ensuring reliability, security, scalability, and responsible AI compliance in production environments.

Key Responsibilities

• Develop and execute test strategies for GenAI applications, including LLMs, RAG pipelines, chatbots, copilots, and agentic AI workflows.
• Validate functional and non-functional requirements across prompts, model outputs, APIs, integrations, user interfaces, and end-to-end business processes.
• Create and maintain evaluation datasets, test cases, and scoring frameworks to assess relevance, accuracy, groundedness, consistency, completeness, and instruction adherence.
• Evaluate RAG systems for retrieval quality, context relevance, source attribution, answer faithfulness, and hallucination resistance.
• Test agentic AI behavior, including tool selection, function calling, reasoning workflows, memory management, task completion, and failure recovery.
• Conduct security, adversarial, and safety testing for prompt injection, jailbreak attempts, sensitive data exposure, content safety, and authorization controls.
• Perform performance, scalability, reliability, and load testing, including response latency, throughput, concurrency, token consumption, and cost optimization.
• Build automated regression and API testing frameworks using Python and integrate testing into CI/CD pipelines.
• Monitor production quality through evaluation metrics, observability platforms, user feedback, and drift analysis.
• Collaborate with product, engineering, data science, security, and business teams to define quality standards, acceptance criteria, and release readiness.

Required Skills

• Strong experience with GenAI, LLMs, RAG architectures, and Agentic AI testing.
• Proficiency in Python-based test automation, including PyTest, Robot Framework, API testing frameworks, and automation best practices.
• Experience with LLM evaluation, prompt engineering validation, hallucination testing, bias assessment, robustness testing, and prompt-injection security testing.
• Hands-on experience with GenAI evaluation and observability tools such as Ragas, DeepEval, Promptfoo, LangSmith, Arize Phoenix, or similar platforms.
• Knowledge of embeddings, vector databases, document ingestion, chunking strategies, metadata validation, and retrieval testing.
• Strong understanding of software quality engineering, test strategy, defect management, test design, and automation for enterprise applications.
• Proficiency in SQL and data validation techniques.
• Experience with GitHub, GitHub Actions, CI/CD pipelines, and familiarity with Docker, Kubernetes, AWS, and Azure.
• Ability to define measurable quality metrics, evaluate AI system performance, and communicate risks effectively to technical and business stakeholders.
• Strong analytical, troubleshooting, problem-solving, and communication skills with a focus on responsible AI, security, privacy, and compliance.

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science, or a related field.
• 5+ years of experience in Quality Engineering, Test Automation, Data Testing, AI/ML Testing, or related disciplines.
• Experience testing AI applications within regulated industries such as banking, financial services, healthcare, or other compliance-driven environments.
• Knowledge of model governance, audit requirements, and responsible AI controls is highly desirable.