Senior Technical Product Manager - AI Agents, Evals & Reliability in New York at Jobgether

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Technical Product Manager - AI Agents, Evals & Reliability based in United States.

This is a deeply technical product leadership role focused on turning advanced AI capabilities into reliable, user-facing experiences.
You will shape systems designed to support long-running workflows, persistent context, multi-step reasoning, and real-world task completion.
Working closely with ML, backend, mobile, and product teams, you will translate technical possibilities into clear product requirements and system decisions.
A major focus will be building robust evaluation frameworks that measure quality, reliability, and user impact across AI systems.
You’ll make high-leverage trade-offs involving quality, latency, cost, reliability, and user experience in a rapidly evolving environment.
The role offers significant ownership within a high-talent-density, hands-on team where decisions move quickly and execution matters.
Your work will directly influence how AI products behave, earn user trust, and deliver practical value at global scale.

• Define end-to-end requirements for AI-powered systems, connecting model capabilities and technical constraints to user needs and measurable product outcomes.
• Translate model performance, data limitations, evaluation results, and user feedback into clear product and system decisions.
• Partner closely with ML, backend, mobile, and other engineering teams on architecture, system design, evaluation, iteration, and product delivery.
• Establish and continuously improve evaluation frameworks spanning offline metrics, online experiments, human feedback, and real-world user outcomes.
• Make informed trade-offs across product quality, reliability, latency, cost, speed of execution, and overall user experience.
• Drive execution through clear product specifications, disciplined prioritization, strong technical judgment, and effective cross-functional alignment.
• Own product quality end-to-end, with particular emphasis on correctness, predictability, reliability, failure handling, and user trust.
• Identify weaknesses in AI behavior, including hallucinations, bias, model limitations, drift, and other sources of unreliable performance, and drive solutions to address them.
• Establish strong feedback loops that enable teams to learn quickly from production behavior and continuously improve AI-powered workflows.
• Ensure new AI capabilities are shipped efficiently and responsibly while maintaining high standards for quality and reliability.
• Help align product strategy, roadmaps, and priorities with user needs, AI capabilities, and broader business objectives.
• Bring structure to ambiguous problems and independently drive complex initiatives from definition through execution and measurable outcomes.

• Strong foundation in computer science fundamentals, including algorithms, data structures, and system design, with the ability to engage deeply in technical discussions.
• Solid understanding of machine learning fundamentals and how modern AI systems behave in production environments.
• Significant experience owning complex, technically sophisticated products from strategy through execution.
• Hands-on experience with AI-powered products, particularly LLM-based systems, model evaluation, prompt or pipeline iteration, and AI feedback loops.
• Strong intuition for AI limitations and failure modes, including hallucinations, bias, model drift, and non-deterministic behavior.
• Ability to read, review, and discuss technical design documents and confidently collaborate with senior engineers and ML specialists.
• Demonstrated ability to make sound product and technical decisions in ambiguous, rapidly changing environments.
• Strong judgment and prioritization skills, with the ability to balance ambitious product goals against technical, operational, and commercial realities.
• Excellent communication and collaboration skills, with the ability to create alignment across highly technical, cross-functional teams.
• A hands-on, independent working style and the ability to bring structure to complex problems while maintaining a high pace of execution.
• Experience working in high-talent-density or small, fast-moving teams where individuals are expected to take significant ownership.
• Experience shipping AI-heavy consumer products is strongly preferred.
• An engineering background or experience as a highly technical product manager is a plus.
• Experience defining evaluation metrics for machine learning systems, developing AI UX patterns, or working in zero-to-one product environments is highly valued.

• Opportunity to shape AI products designed to deliver practical value to billions of potential users.
• High-ownership role within a small, high-talent-density, hands-on team.
• Significant influence over AI product strategy, system behavior, evaluation, and reliability.
• Close collaboration with experienced ML, backend, mobile, and product engineering professionals.
• Fast-paced environment emphasizing rapid iteration, thoughtful experimentation, and high-quality execution.
• Opportunity to work on challenging problems involving AI agents, long-running workflows, evaluation, reliability, and real-world task completion.
• Direct impact on product quality, user trust, and the practical application of advanced AI technologies.
• Efficient interview process consisting of approximately 3–4 interviews for candidates who advance, conducted virtually and/or onsite.
• Prompt decision-making and a collaborative environment focused on meaningful outcomes.