AI Tester in Los Angeles, California at OmegaHires

Job Title: AI Tester
Location: Los Angeles, CA (Onsite/Hybrid)
Duration: 12 Months

We are seeking a detail-oriented AI Tester to ensure AI-driven products perform accurately, safely, and reliably across real-world scenarios. This role will partner closely with Product, Engineering, and Subject Matter Experts (SMEs) to validate model behavior, assess data quality, and identify trust-related risks throughout the AI lifecycle.

• Evaluate AI/ML and GenAI outputs for accuracy, consistency, edge cases, and failure modes
• Validate training and inference data quality, identifying risks such as bias and data leakage
• Design, implement, and execute model evaluation and regression testing strategies
• Maintain and expand Golden Datasets for regression testing across model updates, prompt changes, and releases
• Test AI behavior within end-to-end workflows and system integrations
• Support Responsible AI initiatives, including fairness, explainability, and compliance validation
• Monitor model performance post-release, identifying drift, regressions, and user feedback trends
• Develop and enhance test automation coverage for AI systems
• Track and report quality metrics such as accuracy, precision, and recall, highlighting gaps, errors, and hallucinations
• Clearly communicate testing outcomes, risks, and recommendations to stakeholders

• Strong experience in QA/test planning, exploratory testing, and UAT
• Solid understanding of AI/ML fundamentals and evaluation metrics
• Ability to analyze data and reason about model behavior and outputs
• Experience testing GenAI prompts and responses (preferred)
• Strong domain understanding and user-centric mindset
• Excellent documentation and communication skills

• 3–7+ years of experience in QA, testing, or quality engineering roles
• Experience working with AI/ML or data-driven products preferred
• Certifications such as ISTQB, AI/Data certifications, or Responsible AI training are a plus