Principal Applied Scientist - Copilot Quality

Quick Facts

• Role: Principal Applied Scientist (Copilot Quality)

Description

You will define and drive the strategy and technical roadmap for Copilot quality, building systems that reliably identify success and failures and translate customer signals into product improvements. The job focuses on creating trusted offline/online evaluation, diagnosing root causes of failure modes, and ensuring improvements reach customers safely and reliably through experiments, quality gates, and release criteria.

Responsibilities

• Build trusted measures of quality
  • Develop online and offline metrics, representative evaluation datasets, and labeling approaches using human and AI assistance.
  • Create benchmarks grounded in real customer tasks.
  • Validate that metrics are statistically sound, sensitive to meaningful changes, and predictive of customer outcomes.
• Turn signals into improvements
  • Identify high-impact failure modes across Copilot experiences and determine root causes using data, evaluations, and customer feedback.
  • Improve quality using approaches such as post-training, retrieval and ranking, prompting, orchestration, and product changes.
  • Design experiments to distinguish promising offline results from improvements customers actually experience.
• Experimentation and Release Quality
  • Establish rigorous A/B experimentation, quality gates, regression protection, and rollout criteria linking offline gains to customer impact (satisfaction, task success, engagement, retention, latency, reliability, safety, and cost).
  • Create operating mechanisms that rapidly resolve quality tradeoffs, dependencies, and release risks.

Requirements

• Bachelor’s degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field plus 8+ years related experience; or Master’s plus 6+ years; or Doctorate plus 5+ years (or equivalent experience).
• Significant experience building, evaluating, or improving production ML or AI systems.
• Demonstrated expertise in at least one relevant area: large-model evaluation, deep learning, post-training, reward modeling, reinforcement learning, retrieval, or ranking.
• Experience designing rigorous metrics or experiments and using results to guide technical and product decisions.

Benefits

• Base pay range across the U.S.: USD $165,600 - $296,400 per year (varies by location; SF Bay Area and NYC metro ranges differ).
• Eligible for benefits and other compensation (details provided by the employer).