Senior Machine Learning Engineer, Agent Evaluation Platform

Quick Facts

Senior Machine Learning Engineer focused on an Agent Evaluation Platform.

Description

You will build the loop that improves AI agents based on failures observed in real enterprise use—without training the underlying model. The work combines deterministic validators for checkable system state with LLM judges for fuzzy judgments, including confidence scoring and calibration against human-labeled trajectories. You will also create synthetic environments and infrastructure to run evals reproducibly at scale across many customer organizations.

Responsibilities

• Design and calibrate a shared judge system using per-item rubrics configured next to datasets
• Split evaluation into deterministic validators (world-state checks) and LLM judges (policy/appropriateness/efficiency)
• Produce confidence-aware scoring that routes uncertain outcomes to humans
• Run a calibration loop against human-labeled trajectories to set the ceiling for judge quality
• Detect and guard against correlated blind spots in LLM-based user simulators and judges
• Handle offline↔online divergence by analyzing disagreements and re-seeding eval suites from new production failures
• Build an agent harness that improves from its own use: generate synthetic scenarios, attribute failures to harness components, and propose component optimizations against simulation
• Verify proposed changes on cases not used for tuning, detect regressions, and keep “propose” separate from “judge”
• Turn fixed failures into permanent tests so the eval suite grows over time
• Ensure org-specific scoping and admin approval for organization-specific learnings

Requirements

• 2+ years of applied ML/data science/ML-adjacent engineering with a record of shipped work
• Ability to turn subjective human judgment into reliable measurement signals others can act on
• Strong applied ML fundamentals; comfortable evaluating LLMs as configurable components (prompt/evaluate/fine-tune)
• Strong Python skills and discipline to ship production-grade code
• Clear communication about complex evaluation methodology and meaning of metrics
• High ownership and a startup-like bias toward shipping
• Comfort with ambiguity and judgment about when measurement is good enough to act
• Experience in at least 3 of: LLM-as-judge or automated eval design/calibration; human annotation programs (rubrics/label quality/throughput); search/recsys/online experimentation evaluation (offline-online divergence, rater agreement, golden-set staleness); fine-tuning & evaluating small models (SFT/preference tuning/distillation); reward modeling/RLHF/RLAIF/process reward models; agent trajectory analysis and step-level fault attribution; prompt engineering as a versioned, tested, measured discipline

Benefits

• Base pay range: $161,300–$274,200 plus equity (when applicable) and variable/incentive compensation and benefits
• Health plans (including flexible spending accounts)
• 401(k) with company match and ESPP
• Matching donations, flexible time away, and family leave programs