Member of Technical Staff, Model Evaluation in San Mateo, California at Inception
Employment Type: Full-Time
Inception
San Mateo, California, United States
Posted on
Explore Related Opportunities
Software and Web Developers, Programmers, and Testers jobs near me in CaliforniaJobs near me in CaliforniaSoftware and Web Developers, Programmers, and Testers jobs
Job Description
The Role
We seek experienced engineers and scientists to develop the evaluation metrics and systems that drive frontier LLM performance. You'll design the frameworks that tell us whether our models are improving and ensure they perform reliably at scale in production.
Key Responsibilities
- Design, develop, and maintain robust evaluation frameworks and benchmarks for measuring LLM performance across diverse tasks and domains.
- Define and implement quantitative metrics that capture model quality, safety, reliability, and regression detection.
- Build scalable, automated evaluation pipelines that integrate into model training and deployment workflows.
- Conduct rigorous statistical analysis of model outputs to identify failure modes, biases, and performance gaps.
- Partner with product and customer-facing teams to translate real-world use cases into meaningful evaluation criteria.
Qualifications
- BS/MS/PhD in Computer Science, Machine Learning, Statistics, or a related field (or equivalent experience).
- At least 2 years of experience in ML evaluation, applied ML research, or a related engineering role.
- Strong understanding of LLM fundamentals (autoregressive generation, instruction tuning, RLHF, in-context learning, decoding strategies).
- Proficiency in Python and ML frameworks such as PyTorch.
- Experience designing and implementing evaluation metrics and benchmarks for generative models.
- Solid foundation in statistics, experimental design, and hypothesis testing.
- Experience with version control (Git) and containerization (Docker).
- Excellent communication skills with the ability to distill complex evaluation results into actionable insights.
Preferred Skills
- Experience with human-in-the-loop evaluation systems (Likert-scale annotation, pairwise preference ranking, red-teaming).
- Familiarity with LLM safety and alignment evaluation (toxicity, hallucination detection, factual grounding).
- Knowledge of existing benchmark suites (MMLU, HumanEval, HELM, BIG-Bench) and their limitations.
- Experience building evaluation infrastructure at scale using cloud platforms (AWS, GCP, Azure).
- Familiarity with MLOps practices and CI/CD pipelines for model validation.
- Experience with data engineering, large-scale data labeling, or synthetic data generation for evaluation purposes.
Scan to Apply
Just scan this QR code to apply from your phone.
Job Location
San Mateo, California, United States
Frequently asked questions about this position
Similar Jobs In San Mateo, California
Senior Space Systems Engineer- San Jose
Mission Systems
San Jose, California
Senior Software Engineer - Infrastructure
Fivetran
Oakland, California
New
ServiceNow Engineer
Bright Vision Technologies
Cupertino, California
Director of Software Engineering
Fervo Energy Company
Oakland, California
New
PeopleSoft Interface Developer
Bright Vision Technologies
Cupertino, California
Continue to apply
Enter your email to continue. You’ll be redirected to the employer’s application.By clicking Continue, you understand and agree to JobTarget's Terms of Use and Privacy Policy.