Senior Manager, Evaluation and Applied AI Quality (LLM Autograders)

Quick Facts

• Role: Hands-on Senior Manager leading an evaluation function for a conversational AI product
• Focus: Automated grading, evaluation methods, datasets, and quality signals before and after launch

Description

Lead a team of applied scientists and machine learning engineers to design and run evaluations for a conversational experience that understands requests, retrieves and ranks the right results, generates grounded and accurate responses, and executes actions correctly via external integrations. Own specific evaluation components—methods, datasets, and automated grading systems—that convert product behavior into quality signals for engineering and executive leadership. Provide technical direction for LLM-based autograders and rubrics, alongside logging, instrumentation, and online experimentation to translate findings into measurable quality gains and release decision inputs.

Responsibilities

• Design and run evaluation across quality layers: request interpretation, retrieval/ranking, response composition, and action execution
• Set technical direction for LLM-based autograders and rubrics (rubric design, evaluation set development, validation, calibration)
• Build roadmaps for evaluation components and translate strategy into concrete methods and deliverables
• Pair evaluation work with logging and instrumentation and online experimentation
• Review designs and experiments, mentor scientists and engineers, and recruit/grow the team
• Partner with engineering, applied machine learning, quality assurance, and program management
• Present quality status, risks, and trade-offs to senior leadership; define explicit release quality gates

Requirements

• 6+ years of experience in machine learning, data science, or machine learning evaluation
• 3+ years of direct people management of scientists or engineers
• Proven track record of hiring, developing, and leading applied scientists/data scientists/ML engineers
• Hands-on experience building and deploying LLM-based autograders and rubrics (rubric design, evaluation set development, validation, calibration)
• Experience evaluating conversational AI, search, recommendation, or ranking systems using offline metrics and online experimentation
• Strong command of statistics and experimental design, including A/B testing and interpreting noisy metrics
• Exceptional written and verbal communication skills with proven influence over engineers and senior leaders

Benefits

• Hands-on leadership role shaping what “good” and “shippable” mean as the product and quality bar evolve
• Ownership of key evaluation components that drive model and product improvements and feed explicit release quality gates