AI QA & Reliability Specialist

Voxira builds enterprise-grade healthcare AI solutions that transform how healthcare organizations interact with their patients. Our offerings include NoteGen, an ambient AI medical scribe, and our core conversational AI platform that automates patient interactions, integrates with leading EMRs, and helps clinics improve access to care while reducing administrative burden. Security, compliance, reliability, and AI innovation are at the core of everything we build.

About the Role

Ensure Voxira's AI agents perform accurately and reliably in production through structured testing, behavioral validation, and technical support - and serve as backup coverage for client accounts when needed.

Responsibilities

• Validate AI agent behavior against approved client-specific flows, logic, and expected outcomes.
• Perform regression and release testing across product updates.
• Conduct ongoing production audits to confirm AI agents continue to accurately follow trained flows and behaviors, flagging drift or regressions.
• Analyze prompt and model behavior to identify root causes of inaccuracies and recommend corrections.
• Provide Level 1 and Level 2 technical support.
• Investigate and troubleshoot production issues.
• Develop reusable QA methodologies and test assets.
• Collaborate with Engineering on product quality and system optimization.
• Serve as backup point of contact for client accounts during the Implementation Specialist's absence, maintaining continuity without assuming ongoing account ownership.
• Write automated test scripts and validation code (Python or JavaScript/TypeScript) to support regression and release testing.
• Collaborate on establishing validation protocols to confirm AI outputs - including NoteGen's generated summaries - accurately reflect each client's SOPs and are factually accurate, in compliance with health data security standards (e.g., TGV - Québec's MSSS security and confidentiality certification for health-sector software - and HIPAA).

Qualifications

• 5+ years in QA, software testing, or technical support.
• Experience with SaaS products.
• Analytical troubleshooting skills; comfort with structured/methodical testing approaches.
• Strong proficiency in Node, Python, or JavaScript/TypeScript; experience building or maintaining automated test suites.
• API or AI experience is desirable.
• Experience evaluating Large Language Models (LLMs): factual accuracy testing, SOP-alignment assessment, bias/fairness detection, and semantic stability testing. Familiarity with 'LLM-as-a-judge' evaluation frameworks or custom benchmarks.
• Ability to diagnose issues across unstructured data (logs, conversation transcripts) and use data visualization for error analysis.
• Outcome-driven mindset: focused on prioritizing business value, reliability, and user trust over process adherence alone.

Technologies

Node, Python, JavaScript/TypeScript, Langfuse, LangChain, AWS, Docker, GitHub Actions, CloudWatch, Jira, Confluence.

Why join Voxira?

Build technology that measurably improves healthcare. Work on cutting-edge AI projects, collaborate directly with the leadership team, help shape our product roadmap, and contribute to the future of healthcare AI - all in a hybrid, Montreal-based team that moves fast and ships real solutions to real clinics.