AI Evaluation Researcher

Quick Facts

• Role: Researcher, AI Evaluations (Senior individual contributor)
• Focus: AI evaluation research, quality measurement, and validation

Description

You will own evaluation research for priority AI workstreams—defining quality criteria, designing studies, analyzing results, and recommending actions to improve models and products. You will build rubrics, evaluation datasets, benchmarks, and quality thresholds aligned with user needs and expert design judgment, and validate automated graders against human ratings using methods such as LLM-as-judge. Your findings will guide model selection, improvements, product priorities, and release decisions.

Responsibilities

• Own evaluation research across priority AI research workstreams
• Define quality criteria, design studies, analyze results, and recommend actions
• Build rubrics, representative evaluation datasets, benchmarks, and quality thresholds
• Integrate validated evaluation methods into repeatable evaluations and regression tests with AI Research and engineering teams
• Explain quality, failure patterns, and uncertainty; recommend actions for model and product decisions
• Validate automated graders against human ratings, including LLM-as-judge; identify bias, gaps, and uncertainty
• Connect user research insights with model development; share reusable methods and mentor peers

Requirements

• 7+ years of experience in research, applied research, or related field
• Proven track record independently leading complex evaluation projects with measurable impact
• Hands-on experience evaluating AI products and translating findings into model and product improvements
• Expertise designing human evaluations, scoring rubrics, datasets, and benchmarks (including inter-rater agreement)
• Strong qualitative and quantitative research skills, including analyzing uncertainty and drawing actionable insights
• Ability to validate automated graders against human ratings
• Excellent problem-solving and stakeholder management skills to navigate ambiguity and balance rigor and speed

Benefits

• Competitive compensation with equity
• Health, dental, and vision coverage
• Retirement benefits with company contributions
• Parental leave and reproductive or family planning support
• Mental health and wellness benefits
• Paid time off (including paid sick leave and holidays) per applicable law
• Paid flexible PTO for eligible exempt employees
• Additional potential benefits such as recharge days and reimbursements (cell phone, home internet)
• Sales incentive or annual bonus plans for eligible roles