AI Measurement Scientist (AI Evaluation Platform)

Apple

Seattle, WA • 11m ago • On-site

Mid

On-site

142,300 - 263,300 USD/yr

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Potential salary for AI Measurement Scientist (AI Evaluation Platform) role

Improve your resume now

$142,300 – $263,300/year

Quick Facts

  • Role: AI Measurement Scientist (AI Evaluation Platform)

  • Focus: Scientific evaluation methods for AI systems (validity, reliability, calibration, uncertainty, and real-world outcome alignment)

Description

Build a scientific foundation for evaluating AI systems across Apple by creating evidence that teams can act on—what to measure, whether a change helped, why failures occur, and whether systems are ready to ship. You will apply state-of-the-art measurement methods, validity frameworks, experimental design, and statistical rigor so that evaluator outputs (e.g., helpfulness/safety) are scientifically valid and trustworthy across diverse contexts. You will also develop methods to validate evaluation tools and productionize them into scalable evaluation infrastructure.

Responsibilities

  • Design validity frameworks for AI evaluation covering automated metrics, LLM-as-judge systems, and human evaluation protocols.

  • Develop measurement-science methods to assess benchmark quality, including frameworks such as IRT.

  • Create calibration and bias detection systems for automated evaluators to ensure interpretable, consistent, and unbiased LLM-as-judge scores.

  • Build statistical tools for sample-size planning, version comparisons, uncertainty quantification, error-rate control, and data visualization.

  • Link offline evaluation to real-world outcomes via predictive testing, randomized audits, and defensible ship-threshold methods.

  • Establish measurement standards for evaluator transfer and generalization across domains, languages, and contexts.

  • Validate novel evaluation methods with ML researchers (pattern discovery, failure analysis with uncertainty, representative synthetic data).

  • Work with platform engineers to productionize methods into evaluation infrastructure with self-service validity checking, reliability testing, and interpretable outputs.

  • Publish research at top measurement and ML venues and collaborate across disciplines to advance both science and production practices.

Requirements

  • PhD in Computer Science or Machine Learning with a measurement/evaluation focus, psychometrics/educational measurement/quantitative psychology/statistics/biostatistics/econometrics, or equivalent research/work experience.

  • Deep expertise in benchmark/test modeling (e.g., IRT, latent variable/response models) and construct validation and/or experimental design and causal inference.

  • Strong statistical foundation: experimental design, power analysis, sampling theory, and uncertainty quantification.

  • Demonstrated track record designing and validating measurement instruments or experiments (publications or applied work).

  • Proficiency in Python (preferred) or R for statistical analysis and psychometric modeling.

  • Strong working knowledge of generative AI technology.

  • Excellent communication skills for explaining complex measurement concepts to engineers, ML researchers, and non-technical stakeholders.

Benefits

  • Contribute to AI evaluation methods that are scientifically valid, productionized at scale, and used across an organization.

  • Collaborate with interdisciplinary teams bridging ML science and measurement science.

  • Research impact through publication at top measurement and ML conferences and translation into production evaluation tooling.

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report