ML Inference Engineer (Terminal-Bench Task Designer)

Quick Facts

• Task-based compensation model; paid per task
• Designs reproducible, command-line solvable Terminal-Bench challenges for AI agents
• Each task includes an environment, a working reference solution, and automated correctness checks

Description

You will design expert-level Terminal-Bench tasks that translate real, paid ML inference work into end-to-end challenges an AI agent must solve. Each task is packaged with Docker and Git, includes a realistic serving environment (e.g., streaming), and provides a reference solution plus automated checks to verify correctness and catch silent failure modes.

Responsibilities

• Design expert-level Terminal-Bench tasks derived from real ML inference/serving work
• Define the files, systems, and environments each task runs on (including model serving with streaming)
• Write and verify a working reference solution for every task (sized for ~60–75 minutes of expert effort)
• Build automated checks that objectively prove correctness, including numerical drift and metric bugs
• Package tasks with Docker and Git and document systems, tools, data, and rules for reviewer validation
• Iterate on tasks using reviewer feedback and agent test runs to keep difficulty expert-level, fair, and objectively gradable

Requirements

• 3+ years hands-on ML inference or serving work (not only studying/selling/managing) or a PhD with active research in the subfield
• Must have done this work within the last 3 years; can name the systems, tools, data, and rules used
• Trained, served, or evaluated models at real scale (multi-GPU or production serving)
• Hands-on with PyTorch or JAX internals and serving stacks such as vLLM, SGLang, or TensorRT-LLM
• Can explain how ML results go wrong without errors (numerical drift, metric bugs) and how to catch them
• Technical fluency in Python and the command line; able to work with CSV/JSON, APIs, and databases
• Available to start by early November 2026 and complete tasks estimated at ~60–75 minutes each

Benefits

• Paid per task; task vetting instead of a traditional interview
• Reviewer feedback on submissions and support for iteration
• Work delivered task-by-task with objective grading via automated checks