Senior Quality Engineer (AI) in Canada Creek, Nova Scotia at Jobgether

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Quality Engineer (AI) based in Canada.

This role offers the opportunity to define the future of quality engineering for advanced AI-powered systems. You will lead the validation strategy for technologies such as Large Language Models, Retrieval-Augmented Generation applications, and intelligent agent workflows. Working at the intersection of quality engineering, AI, software development, and data governance, you will help ensure AI solutions are accurate, reliable, and production-ready. The position combines technical depth, innovation, and strategic ownership in a remote environment. You will design frameworks, automation pipelines, and monitoring approaches that improve the safety and performance of next-generation AI products. This is an ideal opportunity for an experienced quality engineer who wants to shape how organizations build and validate intelligent systems.

As a Senior Quality Engineer (AI), you will own the development and execution of AI quality strategies, creating scalable validation approaches for complex and non-deterministic systems. You will collaborate with engineering, AI/ML, and data teams to ensure AI-powered solutions meet high standards of accuracy, reliability, and safety.

• Define and lead AI validation strategies for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, and AI agent workflows.
• Build and maintain automated validation pipelines integrated into CI/CD processes.
• Design evaluation frameworks measuring correctness, relevance, groundedness, consistency, hallucination rates, and overall AI output quality.
• Develop prompt regression testing approaches to identify quality degradation after model or prompt changes.
• Implement semantic validation techniques that evaluate meaning, context, and relevance beyond traditional software testing methods.
• Establish production monitoring strategies and define acceptable quality thresholds for AI-generated outputs.
• Partner with engineering teams to validate AI-generated code, automated test cases, and AI-assisted workflows.
• Introduce AI guardrails, confidence scoring mechanisms, and validation checkpoints to improve safety and reliability.
• Build quality metrics and KPIs to measure defect reduction, output consistency, stability, and user confidence.
• Contribute to the continuous improvement of AI development and validation practices across teams.

The ideal candidate is a technically skilled quality engineering professional with hands-on experience testing and validating AI-driven systems. You have strong expertise in automation, AI evaluation methods, and software quality practices, with the ability to collaborate effectively across technical teams.

• 7+ years of experience in Quality Engineering, Software Development, or a related technical discipline.
• Recent hands-on experience working with AI/ML systems, Large Language Models, or intelligent applications.
• Experience designing validation or evaluation frameworks for probabilistic and non-deterministic systems.
• Strong understanding of RAG architectures, agent-based workflows, and LLM evaluation concepts, including hallucination detection, relevance, groundedness, and consistency.
• Hands-on experience with LLMs and prompt engineering, including prompt design, optimization, and regression validation.
• Experience using AI-assisted development platforms or agentic coding tools such as Claude Code, Claude SDK, or similar technologies.
• Proficiency with Node.js for building automation tools, validation frameworks, and engineering utilities.
• Experience working with MongoDB for managing evaluation data, validation results, or AI output logs.
• Experience integrating automated testing and validation processes into CI/CD pipelines.
• Knowledge of AI quality metrics, confidence scoring, production monitoring, and output consistency measurement.
• Strong understanding of data validation principles, including schema validation, business rules, and data integrity.
• Excellent collaboration and communication skills with experience working across quality, engineering, AI/ML, and governance teams.
• Familiarity with prompt regression testing tools, AI governance, fairness practices, UI/API automation tools, or multi-agent systems is considered an asset.

• Fully remote work opportunity available in Canada.
• Flexible working hours designed to support work-life balance.
• Competitive compensation package with stock option opportunities.
• Benefits package exceeding standard requirements.
• Opportunity to work on innovative AI solutions shaping the future of technology.
• Collaborative environment with experienced professionals across engineering and AI disciplines.
• Access to mentorship, learning opportunities, and professional development support.
• Opportunity to contribute to meaningful projects involving advanced AI systems and emerging technologies.