AI Measurement Scientist (AI Evaluation Platform)
Apple
Seattle, WA • 11m ago • On-site
Mid
On-site
142,300 - 263,300 USD/yr
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Potential salary for AI Measurement Scientist (AI Evaluation Platform) role
Improve your resume now
$142,300 – $263,300/year
Quick Facts
Role: AI Measurement Scientist (AI Evaluation Platform)
Focus: Scientific evaluation methods for AI systems (validity, reliability, calibration, uncertainty, and real-world outcome alignment)
Description
Build a scientific foundation for evaluating AI systems across Apple by creating evidence that teams can act on—what to measure, whether a change helped, why failures occur, and whether systems are ready to ship. You will apply state-of-the-art measurement methods, validity frameworks, experimental design, and statistical rigor so that evaluator outputs (e.g., helpfulness/safety) are scientifically valid and trustworthy across diverse contexts. You will also develop methods to validate evaluation tools and productionize them into scalable evaluation infrastructure.
Responsibilities
Design validity frameworks for AI evaluation covering automated metrics, LLM-as-judge systems, and human evaluation protocols.
Develop measurement-science methods to assess benchmark quality, including frameworks such as IRT.
Create calibration and bias detection systems for automated evaluators to ensure interpretable, consistent, and unbiased LLM-as-judge scores.
Build statistical tools for sample-size planning, version comparisons, uncertainty quantification, error-rate control, and data visualization.
Link offline evaluation to real-world outcomes via predictive testing, randomized audits, and defensible ship-threshold methods.
Establish measurement standards for evaluator transfer and generalization across domains, languages, and contexts.
Validate novel evaluation methods with ML researchers (pattern discovery, failure analysis with uncertainty, representative synthetic data).
Work with platform engineers to productionize methods into evaluation infrastructure with self-service validity checking, reliability testing, and interpretable outputs.
Publish research at top measurement and ML venues and collaborate across disciplines to advance both science and production practices.
Requirements
PhD in Computer Science or Machine Learning with a measurement/evaluation focus, psychometrics/educational measurement/quantitative psychology/statistics/biostatistics/econometrics, or equivalent research/work experience.
Deep expertise in benchmark/test modeling (e.g., IRT, latent variable/response models) and construct validation and/or experimental design and causal inference.
Strong statistical foundation: experimental design, power analysis, sampling theory, and uncertainty quantification.
Demonstrated track record designing and validating measurement instruments or experiments (publications or applied work).
Proficiency in Python (preferred) or R for statistical analysis and psychometric modeling.
Strong working knowledge of generative AI technology.
Excellent communication skills for explaining complex measurement concepts to engineers, ML researchers, and non-technical stakeholders.
Benefits
Contribute to AI evaluation methods that are scientifically valid, productionized at scale, and used across an organization.
Collaborate with interdisciplinary teams bridging ML science and measurement science.
Research impact through publication at top measurement and ML conferences and translation into production evaluation tooling.
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See moreMachine Learning Engineer, AI Model Evaluation & Quality
Apple
Senior
Technology
California, CA · On-site
$142k - $263k/yr
4 days ago
Senior Research Manager, AI Evaluation Science
Apple
Senior
Technology
Seattle, WA · On-site
$226k - $338k/yr
11m ago
Senior Data Scientist, Evaluation
Apple
Senior
Technology
Cupertino, CA · On-site
$185k - $325k/yr
3 days ago

Senior Machine Learning Engineer, Evaluation & Reliability
Bask Health
Senior
Technology
Remote
N/A
3h ago
AI Operations & Applied AI Research Developer
Apple
Intern
Technology
Cupertino, CA · On-site
$129k - $195k/yr
10 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Software Engineer, AI Quality & Evaluation
Bask Health
Mid
Technology
Remote
N/A
3h ago

Senior Applied Scientist, Product Policy Research & Measurement
OpenAI
Senior
Technology
Hybrid
$347k - $385k/yr
1 days ago

AI Evaluation Researcher
Figma
Senior
Technology
San Francisco, CA · On-site
$169k - $250k/yr
7 days ago

Senior Director, Technical Product Management — AI Quality & Evaluation
Salesforce
Senior
Technology
Hybrid
$218k - $365k/yr
9 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights