AI Software Evaluation Engineer (Siri AI Quality)

Quick Facts

• Role: AI Software Evaluation Engineer (Siri AI Quality)

Description

Build the systems that evaluate Siri’s AI experiences and turn results into action. Work across evaluation data insights, coverage analysis, and agentic scenario generation, triage, and auto-fix to catch quality issues early and automate repetitive evaluation operations. Partner with engineering, product, evaluator, persona, and data-platform teams to define quality standards and deliver evaluation systems that guide Siri’s development.

Responsibilities

• Evaluation data insights: investigate why test scenarios regress, identify recurring issues, and convert failure/bug data into root-cause findings and recommendations.

• Coverage analysis: identify gaps across app intents, features, and product areas; collaborate with domain owners to close gaps; build and report on dataset quality, issue trends, and coverage.

• Agentic scenario generation, triage, and auto-fix: design agentic workflows that propose, ground, validate, triage, and nominate new evaluation scenarios with minimal manual effort; extend automated screening, triage, and fix tooling to new domains and device platforms.

Requirements

• 5+ years of industry experience in software engineering, quality engineering, data engineering, or ML engineering

• 2+ years of hands-on experience building LLM-based applications or agents (tool use, multi-step workflows, prompt engineering)

• Experience designing and running AI/LLM evaluation (evaluation datasets, metrics and rubrics, LLM-as-judge or automated evaluators, and failure analysis)

• Strong data analytics skills with SQL and Python; experience turning large datasets into insights and dashboards

• Strong programming skills in Python; experience with automation and CI/CD

• Strong communication and collaboration skills; ability to influence cross-functional partners

Benefits

Help deliver high-quality AI experiences to millions of users by owning end-to-end Siri pre-ship evaluation datasets and the synthetic personas that power them.