Senior Software Engineer - AI Interaction Evaluator in United States at Jobgether

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Software Engineer - AI Interaction Evaluator in United States.

This is a highly specialized evaluation role focused on assessing the quality, reasoning, and engineering judgment of modern AI coding agents. Rather than writing production software, you will analyze how advanced AI systems behave when solving real engineering problems and determine whether their outputs reflect strong developer intuition. The work sits at the intersection of software engineering expertise and AI evaluation, requiring sharp technical instincts and the ability to judge “engineering taste” beyond syntax correctness. You will help define what high-quality AI-assisted development looks like in practice by critically reviewing model interactions and identifying strengths, weaknesses, and failure patterns. Operating in a fast-moving, experimental environment, your feedback will directly influence how AI coding tools are refined and improved. This role is ideal for senior engineers who enjoy deep technical judgment work and shaping next-generation developer experiences.

• Evaluate AI-generated coding interactions end-to-end, assessing usefulness, correctness at a conceptual level, and alignment with strong engineering reasoning.
• Analyze whether AI explanations, preambles, and reasoning reflect high-quality developer thinking or introduce confusion or weak logic.
• Distinguish between varying levels of output quality and provide structured, opinionated assessments of model performance.
• Identify what makes interactions effective or ineffective, including clarity, trustworthiness, and engineering coherence.
• Provide detailed qualitative feedback on AI behavior, highlighting what worked, what failed, and where reasoning feels misleading or incomplete.
• Contribute to defining evaluation standards for high-quality AI-assisted development workflows.
• Help shape benchmarks for what “good engineering judgment” looks like in AI coding tools.

You bring deep software engineering experience and strong technical intuition, enabling you to evaluate code quality and system behavior without relying solely on execution or line-by-line verification. You are comfortable making subjective but structured judgments and articulating why an engineering solution feels right or wrong. You have hands-on experience with modern programming workflows and AI-assisted development tools, and you understand what high-quality engineering output looks like in practice.

• Staff-level or Principal-level software engineering experience (or equivalent senior expertise)
• Strong proficiency in at least one core language: TypeScript/JavaScript or Python
• Experience using modern AI coding tools such as Codex, Claude Code, or Cursor
• Deep familiarity with AI-assisted development workflows and developer tooling
• Ability to evaluate code and system behavior at a conceptual level without full execution
• Strong engineering judgment and ability to define what “good” looks like in ambiguous scenarios
• Clear, direct communication style with ability to provide structured, opinionated feedback
• Experience mentoring engineers or defining engineering best practices (nice to have)
• Exposure to prompt engineering or AI evaluation workflows (nice to have)

• Competitive hourly compensation: $100–$200/hour
• Flexible part-time engagement (approximately 10–20 hours/week)
• Remote work with high autonomy and async-friendly collaboration
• Opportunity to influence the development of next-generation AI coding tools
• Short-term contract with potential for extension
• Work directly at the frontier of AI-assisted software engineering evaluation
• Fast onboarding process with immediate start availability