Technical Program Manager, AI Evaluation Specialist at Jobgether – United States

Technical Program Manager, AI Evaluation Specialist

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Technical Program Manager, AI Evaluation Specialist in United States.This role is perfect for a detail-oriented and analytical professional passionate about ensuring the safety, reliability, and quality of AI systems in operational settings. The Technical Program Manager, AI Evaluation Specialist, will lead the human-in-the-loop evaluation process, measuring AI model performance, identifying errors, and recommending improvements to enhance accuracy and trust. You will collaborate closely with data, operations, and model teams to standardize evaluation protocols, analyze patterns, and ensure AI outputs meet organizational standards. This position combines technical rigor with project management, offering the chance to influence model governance at scale. You will track metrics, maintain documentation, and help implement insights into operational workflows. The role requires strong problem-solving, communication, and organizational skills, and operates in a collaborative, fast-paced environment.Accountabilities:

• Own the human-in-the-loop evaluation process for AI models supporting operations, ensuring consistent and accurate assessments.
• Conduct recurring sampling and detailed reviews to assess model accuracy, consistency, and failure modes.
• Score, tag, and document instances where AI systems misclassify, hallucinate, or generate incomplete outputs.
• Maintain rubrics, guidelines, and documentation to ensure evaluator alignment and scoring consistency.
• Investigate error patterns and root causes, translating insights into actionable recommendations for model owners and partner teams.
• Track and report evaluation metrics, such as accuracy, recall, coverage, and error types, and integrate findings into dashboards and workflows.
• Support scaling of governance processes and strengthen model-health standards across operations.

Requirements:

• 35+ years of experience in QA, evaluation, operational analytics, human-in-the-loop programs, or model monitoring.
• Experience reviewing unstructured text and applying rubrics or scorecards for qualitative and quantitative assessment.
• Understanding of AI applications in operations, including classification, summarization, categorization, and automation.
• Strong analytical skills with the ability to identify patterns, edge cases, and failure modes.
• Familiarity with QA frameworks or content-review workflows.
• Experience with SQL, Looker, or Snowflake is a plus.
• Exceptional attention to detail and consistency in work.
• Clear communication and documentation skills.
• Passion for ensuring AI systems are safe, fair, and reliable.
• COPC or Lean Six Sigma experience is a plus.

Benefits:

• Competitive base salary starting at $103,680$144,000 annually, plus potential bonus and equity opportunities.
• Comprehensive medical, dental, vision, life, and disability insurance.
• 401(k) retirement plan with company match.
• Flexible vacation and paid time off policies.
• Paid parental leave for birthing and non-birthing parents.
• Wellness stipends and support for family planning services.
• Opportunities for both in-person and virtual team engagement and professional development activities.
• Remote-first work environment with occasional on-site collaboration as needed.

Why Apply Through Jobgether?We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.We appreciate your interest and wish you the best! Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1