Senior Software Engineer, AI in Canada Creek, Nova Scotia at Jobgether

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Software Engineer, AI in Canada.

This role sits at the core of a modern AI engineering organization focused on building, evaluating, and scaling production-grade LLM systems. You will help define how AI performance is measured, improved, and trusted across real-world use cases, shaping both foundational infrastructure and applied agent systems. Working closely with engineering, product, and AI leadership teams, you will design robust evaluation frameworks, improve agent architectures, and ensure reliability in production AI workflows. The environment is highly technical and collaborative, emphasizing strong ownership, thoughtful experimentation, and data-driven decision-making. You will contribute directly to systems that power LLM-driven experiences, including RAG pipelines, agent orchestration, and observability tools. This role is ideal for engineers who thrive in ambiguity, enjoy building from first principles, and are passionate about advancing the quality and reliability of applied AI systems at scale.

• Design and implement end-to-end AI evaluation infrastructure, including offline evaluations, production tracing systems, and human-in-the-loop feedback loops across multiple AI use cases.
• Define, track, and operationalize key AI performance metrics such as task completion rates, hallucination detection, response quality, and user engagement signals.
• Build and maintain evaluation datasets, automated test harnesses, and scoring pipelines to detect regressions and improve model reliability before production release.
• Architect and implement reusable AI agent systems, including multi-turn workflows, LLM-based DAGs, recommendation engines, and standardized orchestration patterns.
• Develop and scale retrieval-augmented generation (RAG) systems, including vector database management and retrieval quality optimization.
• Evaluate and guide build-versus-buy decisions across LLM providers, frameworks, and AI tooling based on performance, cost, and scalability considerations.
• Contribute to production AI systems with a strong focus on reliability, observability, and system performance at scale.
• Own projects end-to-end, from scoping and design through execution and delivery in collaboration with cross-functional teams.
• Partner with engineering leadership to define AI evaluation strategy and architectural direction across agent systems.
• Improve engineering standards through code reviews, documentation, technical discussions, and mentorship of peers.

• 5+ years of professional software engineering experience, with significant exposure to production machine learning or AI systems.
• Strong hands-on experience building LLM-based systems, including prompt engineering, RAG pipelines, agent orchestration, and evaluation methodologies.
• Proven experience operating agentic AI systems in production, including multi-step workflows and multi-agent architectures.
• Deep understanding of AI evaluation principles, including experiment design, metric selection, and avoiding misleading or vanity metrics.
• Strong Python engineering skills with emphasis on production-quality, maintainable, and testable code.
• Experience working with vector databases (e.g., Pinecone or similar) and designing retrieval systems.
• Familiarity with AI observability and debugging tools such as LangSmith or equivalent platforms.
• Experience working with cloud infrastructure (AWS or GCP), including serverless and event-driven architectures.
• Solid understanding of data analysis and statistical experimentation methods.
• Ability to communicate complex technical concepts clearly across technical and non-technical audiences.
• Experience with LangGraph or similar agent orchestration frameworks is highly desirable.
• Exposure to CI/CD practices, MLOps tooling, and model deployment workflows is a plus.
• Background in RLHF, model fine-tuning, or academic research in AI/ML is considered an advantage.
• Strong product mindset with a bias toward ownership, clarity, and pragmatic execution in ambiguous environments.

• Competitive annual compensation of CAD $160,000–$180,000, plus eligibility for stock options.
• Comprehensive health coverage including medical, dental, life, AD&D, and disability insurance.
• Flexible remote work setup with stipend support for home office equipment and setup.
• Paid time off, parental leave, holidays, and wellness support programs.
• Retirement savings plan and financial planning resources.
• Learning and development budget to support continuous skill growth.
• Access to wellness apps and employee support programs.
• Opportunity to work on cutting-edge AI systems powering real-world production use cases.
• Inclusive and collaborative engineering culture focused on innovation and technical excellence.