Staff Machine Learning Engineer (Agent Evaluation Platform)

Quick Facts

• Role: Staff Machine Learning Engineer focused on an Agent Evaluation Platform

• Focus: Applied ML systems for judging, calibration, and improving agent behavior without training the base model

Description

You will build the loop that turns real-world agent failures into reproducible measurements and permanent tests, improving the agent harness (instructions, tool descriptions, routing, knowledge, and memory) rather than retraining the model.

Responsibilities include designing judge systems with confidence-based routing to humans, calibrating against human-labeled trajectories, and preventing correlated blind spots and offline↔online divergence.

You will also create synthetic scenarios and infrastructure to verify proposed fixes before they ship, keeping improvements safely scoped across many customer organizations.

Responsibilities

• Create evaluation/judging components: shared judges with per-item rubrics, deterministic validators for checkable world state, and LLM judges for fuzzy aspects

• Produce scoring with confidence and route uncertain judgements to humans

• Run a calibration loop against human-labeled trajectories (own calibrated artifacts)

• Guard against correlated blind spots (user simulator and judge as LLMs)

• Detect and explain offline vs online divergence; re-seed suites from new production failures to avoid silent overfitting

• Build self-improving agent harness infrastructure without changing the model

• Surface real-use failures (including those that appear successful) and attribute them to the correct harness component

• Generate synthetic data and seeded environments from real failures or forward-looking tasks with quality controls

• Optimize and validate proposed harness changes via simulation before shipping, separating proposing and judging components

• Convert fixed failures into permanent evaluation tests to grow the suite over time

• Ensure safe deployment across customers with org scoping and admin approval for org-specific changes

• Operate evaluation infrastructure that runs reproducibly at scale across hundreds of customer organizations

Requirements

• 5+ years in applied ML or ML-adjacent engineering with systems you shipped, operated, and others relied on

• Experience translating subjective human judgement into measurable evaluation validated against human labels

• Improved multi-component systems through measurement (pipelines, retrieval, prompts, policies, routing), not only by changing model weights

• Strong applied ML fundamentals; comfort evaluating LLMs as components you prompt and evaluate

• Strong Python and production engineering for reproducibility (versioned data, deterministic replay, recorded inputs)

• Ability to communicate clearly about complex evaluation meaning and assumptions

• High ownership and bias toward shipping quickly

• Comfort with ambiguity and the judgement to know when a measurement is sufficient to act on

• Experience in at least 3 relevant areas: LLM-as-judge with calibration; human annotation/rubric/quality; experimentation rigor; failure analysis at scale; replay/simulation; synthetic evaluation data; prompt/harness optimization; search/recsys/online eval; high-volume trace/event pipelines

Benefits

• Base pay range: $176,100 - $308,200 plus equity (when applicable) and variable/incentive compensation and benefits

• Health plans, flexible spending accounts, 401(k) with match, ESPP, matching donations, flexible time away, and family leave programs