Program Manager, Human Evaluations (Recommendations) in United States at Jobgether

• Program Strategy and End-to-End Delivery
  • Lead human evaluation and data operations initiatives from initial scoping and planning through execution, quality validation, and final delivery.
  • Partner with Product Management to translate recommendation strategies and roadmap priorities into evaluation designs, success metrics, project milestones, and timelines.
  • Coordinate cross-functional stakeholders across Product, Machine Learning, Data Science, UX, Engineering, and external vendor teams to ensure alignment and successful execution.
  • Track project progress, manage dependencies, anticipate risks, and proactively resolve bottlenecks to maintain delivery schedules and quality standards.
  • Ensure evaluation programs support continuous improvements in recommendation relevance, diversity, consistency, and overall member experience.
• Evaluation Frameworks and Quality Assurance
  • Develop clear, scalable evaluation guidelines, scoring rubrics, and assessment protocols that support consistent human judgment across recommendation use cases.
  • Design evaluation approaches for both objective criteria and subjective, taste-based judgments, accounting for rater variability and potential bias.
  • Oversee rater onboarding, training, calibration, and quality assurance workflows to improve consistency and reliability.
  • Pilot and refine evaluation tasks, instructions, and scoring methodologies to improve clarity, feedback quality, and inter-rater reliability (IRR).
  • Collaborate with product and technical teams to establish shared definitions of recommendation quality and translate evaluation findings into actionable recommendations.
• Vendor and Operational Management
  • Manage external human annotation and evaluation vendors, overseeing capacity planning, service-level agreements (SLAs), performance monitoring, and issue escalation.
  • Optimize operational metrics, including cost per task, throughput, turnaround time, and time required to achieve acceptable inter-rater reliability.
  • Monitor vendor performance and ensure evaluation activities meet established standards for accuracy, consistency, efficiency, and timely delivery.
  • Identify opportunities to improve workflows, strengthen quality controls, and scale evaluation operations while managing costs effectively.
  • Synthesize operational feedback and evaluation results into clear recommendations for continuous process improvement.

• Relevant Experience: At least 4 years of experience in human evaluation, data collection, data labeling, annotation operations, or related program management within machine learning environments.
• Program Management: Demonstrated success leading complex, cross-functional programs from planning through delivery, with strong accountability for timelines, quality, and outcomes.
• Vendor and Team Management: Experience managing external annotation vendors, human raters, or data-labeling teams, including performance monitoring, capacity planning, quality management, and cost optimization.
• Evaluation Design: Strong understanding of evaluation methodologies, scoring rubrics, annotation guidelines, and quality control systems for human-generated data.
• Operational Excellence: Proven ability to improve annotation program efficiency, including throughput, turnaround time, cost per task, and inter-rater reliability.
• Content Knowledge: Working familiarity with film and television content, including genres, formats, talent, and the ability to assess whether groups of titles are coherent and appropriate together.
• Technical Tools: Experience working with human labeling or annotation platforms and coordinating evaluation workflows in technology-driven environments.
• Analytical and Communication Skills: Excellent written and verbal communication skills, with the ability to synthesize complex feedback, communicate findings clearly, and recommend practical improvements.
• Judgment and Quality: Understanding of how subjectivity and bias can affect taste-based assessments, with the ability to design evaluation processes that encourage consistency and reliable results.
• Organizational Skills: Strong attention to detail, prioritization, and problem-solving abilities, with the capacity to manage multiple workstreams while maintaining a broader strategic perspective.
• Collaboration and Adaptability: Ability to build credibility with product managers, data scientists, and machine learning engineers, navigate ambiguity, and turn evolving requirements into actionable plans.

Preferred Qualifications

• Experience with personalization, recommendation systems, search, or ranking evaluation.
• Familiarity with how offline human evaluations complement online experimentation, including A/B testing and engagement metrics.
• Experience evaluating recommendation lists or entire page-level experiences rather than assessing individual items in isolation.
• Experience collaborating with Product Managers and Data Scientists to define metrics and establish evaluation criteria.

• Competitive compensation: Annual salary range of $230,000–$360,000, depending on location, experience, skills, and relevant market benchmarks.
• Flexible compensation structure: The opportunity to choose the annual mix of salary and stock options. The compensation structure does not include bonuses.
• Comprehensive healthcare: Health insurance plans and mental health support.
• Retirement benefits: 401(k) retirement plan with employer matching.
• Equity opportunities: Access to a stock option program.
• Financial protection: Disability coverage, health savings accounts (HSAs), flexible spending accounts (FSAs), and life and serious injury benefits.
• Family support: Family-forming benefits and paid leave programs.
• Flexible time off: Full-time salaried employees have access to flexible paid time off.
• Paid leave: Additional leave programs to support employees through different life circumstances.
• Remote work: A remote position available to eligible candidates based in the United States.
• Meaningful impact: The opportunity to help improve personalized content discovery through rigorous human evaluation and innovative machine learning technologies.
• Collaborative environment: Work alongside multidisciplinary teams focused on product innovation, data-driven decision-making, and continuous improvement.