Software Engineer, Applied AI Evals

Quick Facts

• Role: Software Engineer (Applied AI Evals)

Description

Build the Evals product that helps labs and enterprises run large-scale model evaluations, turning eval runs into scorecards, failure reports, and benchmark dataset views. Ship APIs, data models, and UI so customers can compare models and make trust decisions. Own APEX benchmark launches by running frontier models, publishing defensible results to a public leaderboard, and explaining why models fail.

Responsibilities

• Build APIs, data models, and UI to run evals, compare models, and share scores

• Run APEX for new frontier models and publish results to the public leaderboard

• Create failure analysis that links patterns to the exact steps that went wrong

• Build dataset views, registries, and versioning to manage benchmarks

• Develop onboarding and self-serve capabilities (roles, key management, usage views)

• Collaborate with customers in pilots to shape the product roadmap

Requirements

• Bachelor’s degree in Computer Science or related field

• Strong full-stack skills with Python, Go, or TypeScript and a frontend framework such as React

• Track record of owning product features end to end (idea to production)

• Strong product sense centered on user needs

• Experience building data-heavy products (data models, APIs, and views over large volumes)

• Excellent written and verbal communication

• Comfort with ambiguity and iterative shipping

Benefits

• Bi-annual performance bonus structure

• Generous equity grant vested over 4 years

• Up to $15k relocation bonus

• $10K housing bonus (if living within 0.5 miles of the office)

• $1.5K monthly stipend for meals

• Free Equinox membership

• $200 monthly laundry reimbursement

• $200 monthly personal wellness reimbursement

• Health, Dental, Vision insurance