Software Engineer

Quick Facts

• Role: Software Engineer

• Focus: Building infrastructure for evaluating coding agents and language models

Description

You will design and ship end-to-end features across data models, APIs, background workers, and the UI used to run and assess evaluations. You will improve agent/model evaluation by running trials in sandboxed environments, scoring results, and turning large numbers of runs into trustworthy outcomes. You will also enhance benchmark task quality checks using verifiers and rubric/LLM-as-a-judge grading, while keeping the system fast, reliable, and safe as usage grows.

Responsibilities

• Design and ship features end to end: data model, API, background workers, and UI

• Run and improve evaluation of agents/models at scale (sandboxed trials, scoring, result aggregation)

• Build quality-check pipelines for benchmark tasks using verifiers and grading approaches (rubrics and LLM-as-a-judge)

• Integrate new models/agents/benchmarks so the approach generalizes

• Ensure performance and reliability at scale (efficient queries, safe migrations, concurrency)

• Review pull requests and help keep the codebase consistent

Requirements

• 5+ years building and operating production backend systems with ownership from design through on-call

• Bachelor’s degree or higher in Software Engineering, Computer Science, or related field

• Strong Python and experience with Django (or similar), PostgreSQL, and a task queue such as Celery

• Hands-on LLM evaluation experience beyond simply calling a model API

• Ability to assess task quality and detect ambiguity, gameable verifiers, leaked answers, and misleading passes

• Good engineering tradeoffs and reasoning about what not to build

• Ownership of loosely defined requests through migrations, tests, and edge cases

• Quick adoption of team conventions (naming, structure, tests, error handling)

• Comfort with TypeScript and React for UI-visible changes

• Clear technical writing for PR descriptions and design notes

Benefits

• Work at the frontier of AI to help leading AI labs improve expert datasets, RL environments, and benchmarks

• Contribute to leading-edge AI research and showcase work at conferences such as ICLR, ICML, and NeurIPS

• Apply frontier AI innovation to real enterprise business challenges

• Work at startup speed with strong ownership and impact