Applied AI Engineer

Quick Facts

• Role: Applied AI Engineer

Description

You will be the first person owning how good an AI agent is. The role focuses on building and taking over the evaluation suite that measures quality, running an improvement loop (hill-climb) to debug failures, and validating fixes with regression gates. You will judge both task success and conversation quality, test new models, and help decide when fine-tuning is worth exploring.

Responsibilities

• Build and own the eval suite: scenarios, rubrics, graders, benchmarks, and quality gates for every change
• Run the hill-climb loop: find failure points, form hypotheses, adjust skills/prompts, rerun the suite, and prove fixes without breaking other behavior
• Evaluate conversation quality in addition to task success (understanding the customer, saying the right thing, helping)
• Test new models and separate model changes from changes in prompts, tools, and skills
• Identify repeated failures that local fixes won’t solve and propose restructuring of skills or the agent
• Develop the approach to fine-tuning: when it’s worth it, what data is needed, and how to prioritize against better skills or models

Requirements

• Experience building and running evals for an LLM product in production, including writing graders and verifying LLM judge performance against human ratings
• Ability to catch regressions before users are affected
• Experience with multi-step, tool-using agents and knowledge of how they fail
• Strong coding ability for graders, harness code, and experiments; able to read traces independently
• Experimental discipline: run repeatedly, account for variance, and distinguish real gains from noise
• Ability to assess conversation quality from transcripts even when the task is completed
• Comfort setting your own agenda and deciding what to measure first
• Comfortable working in English

Benefits

• Own a whole quality-measurement area from day one
• Work on an agent trusted by real business owners with real customer conversations
• See impact in production within weeks
• Growth opportunity to move further into training/fine-tuning and help build the team