Senior Manager, AI Performance & Operations in United States at Jobgether
Explore Related Opportunities
Job Description
This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Manager, AI Performance & Operations in United States.
This role sits at the center of production AI reliability, ensuring that high-impact agentic workflows consistently meet performance, safety, and business expectations at scale. You will own the end-to-end health of live AI systems, spanning voice and text-based experiences that directly shape customer and user interactions. Acting as the operational backbone for production AI, you will monitor system behavior, detect degradation early, and lead structured responses to incidents and regressions. You will also define how performance is measured, building scalable evaluation systems that combine human review, automated scoring, and LLM-based assessment. Partnering closely with engineering, product, and operations teams, you will turn ambiguous system behavior into clear, actionable insights. This is a high-ownership role where technical depth, operational rigor, and cross-functional leadership converge to ensure AI systems continuously improve in real-world environments.
In this role, you will be responsible for ensuring the stability, quality, and continuous improvement of production AI workflows across the organization. You will own performance monitoring, incident response, and optimization cycles, while also shaping how new AI systems are evaluated before and after launch.
- Own the post-launch performance and operational health of high-volume AI workflows, ensuring quality, safety, and business KPIs are consistently met
- Maintain a portfolio-level view of all active workflows, tracking maturity, risk areas, failure modes, and ongoing improvement actions
- Design and oversee production monitoring dashboards, evaluating key indicators such as containment, escalation rates, and user impact signals
- Lead diagnosis and resolution of AI incidents and regressions, identifying whether issues stem from models, prompts, retrieval, or orchestration layers
- Drive continuous improvement cycles, including experimentation, rollout/rollback decisions, and optimization of prompts, tools, and model configurations
- Build scalable evaluation frameworks using golden datasets, human review processes, and LLM-as-judge methodologies
- Support early-stage workflow design by defining evaluation criteria, instrumentation standards, and safe launch requirements
- Communicate system performance, risks, and resolutions clearly to executive and cross-functional stakeholders
The ideal candidate brings strong hands-on experience in applied AI systems and production environments, with the ability to both diagnose technical issues and drive operational execution. You are comfortable working in ambiguity, owning outcomes, and translating complex system behavior into actionable improvements.
- 7+ years of experience in applied AI, ML operations, product analytics, or related technical program/product roles
- Strong hands-on experience with LLMs, agentic systems, RAG pipelines, or production ML workflows
- Proven ability to design and operate evaluation systems, including scoring rubrics, regression testing, and sampling frameworks
- Deep diagnostic skills with the ability to isolate failures across model behavior, retrieval systems, tooling, and workflow logic
- Experience running structured experimentation and optimization cycles, including prompt tuning and system iteration
- Strong cross-functional communication skills, with the ability to influence technical and non-technical stakeholders
- High ownership mindset with demonstrated experience driving outcomes in high-stakes or production-critical environments
- Experience in regulated or high-reliability domains (e.g., healthcare, contact centers, or service operations) is a plus
- Competitive salary range of $110,000–$172,000/year, based on experience and location
- Employer-sponsored medical, dental, and vision coverage with low or no premiums
- Generous paid time off and parental leave program
- Monthly $100 stipend for mobile or internet expenses
- Stock options for all employees
- Bonus eligibility for most roles (excluding Director and above) and commission eligibility for sales roles
- 401(k) retirement savings program
- Flexible, mission-driven work environment focused on improving healthcare outcomes
- Additional total rewards and wellness benefits designed to support long-term employee well-being