Senior Machine Learning Platform Engineer (Evaluation)

Quick Facts

• Role: Senior Machine Learning Platform Engineer (Evaluation)

• Focus: Evaluation systems for foundation models and AI agents

Description

Build systems that evaluate and improve foundation models and agents by connecting failure analysis to measurable quality gains. You will design benchmarks, evaluators, simulation environments, and prompt/context optimization pipelines, then turn findings into datasets and reward signals used for post-training. The work closes the loop from observation to improvement across AI products used by millions of users.

Responsibilities

• Design and build evaluation infrastructure for agents and foundation models

• Set technical direction for the evaluation system and align partner teams

• Develop LLM judges and reward models calibrated against human feedback

• Build prompt and context optimization pipelines

• Create and integrate simulation environments for agent evaluation and trajectory-based data generation

• Collaborate with product and research teams to identify, prioritize, and address quality gaps

• Build shared foundations such as run lineage, a run registry, and MCP interfaces for client teams’ agents

Requirements

• Strong background in machine learning and distributed systems

• End-to-end experience solving ambiguous problems (define the problem and metric, build the solution, and show measured impact)

• Experience leading technical work across multiple codebases, teams, and organizations

• Experience building and maintaining ML infrastructure for evaluation, training, or deployment

• 6+ years of professional software engineering experience, preferably in machine learning or a related field

• Bachelor’s or Master’s degree in Computer Science or a related field, or equivalent practical experience

Benefits

• Work on systems that connect evaluation to model/agent improvements using offline benchmarks, device-in-the-loop simulation, and on-device observation

• Influence quality across a broad range of AI products, with rapid feedback loops from finding failures to fixing them