Director of Evaluation and Quality Assurance (Services Special Projects)

Quick Facts

• Role: Director of Evaluation and Quality Assurance (Services Special Projects)
• Focus: 0-to-1 build and leadership of an end-to-end evaluation and QA function for a multi-year engineering initiative

Description

You will build and lead evaluation and quality assurance across every layer of the product—from correct triggering for the right user requests, to relevance and ranking, to transaction correctness including integrations with external partner APIs and financial systems. You will define quality strategy and implementation using automated grading (LLMs/rubrics) and human evaluation programs, and wire continuous evaluation into always-on pipelines with explicit release pass/fail quality gates. As the first hire, you will set strategy and technical direction with minimal existing tooling or process, while building the team, operating rhythm, and production decision-making.

Responsibilities

• Design scalable metrics and automated grading systems (autograders) for frequently changing, large datasets; decide where to combine automation and human evaluation.
• Integrate evaluation outputs into release processes as explicit quality gates.
• Prevent quality regressions across the full product surface, including user-facing correctness and backend transaction integrity.
• Partner with engineering and applied ML to define “quality” and “done” before the product’s definitions fully settle.
• Create structure in an ambiguous environment by defining process, tooling, and org design.
• Make investment decisions for limited evaluation resources to maximize risk reduction.

Requirements

• 12+ years of experience in data science, machine learning evaluation, QA, or test engineering in a technical product environment, including leading multi-level QA/evaluation orgs through managers.
• Proven experience building an evaluation/QA function from the ground up (metrics, process, tooling, org structure).
• Extensive experience with LLM-based autograders and rubrics (rubric design, evaluation set development, validation, calibration, and human-model alignment).
• Deep expertise in human evaluation methodology, logging, and A/B testing for consumer-facing products at scale.
• Experience evaluating conversational AI, recommendation/ranking, or search systems; ideally agentic or tool-use systems that take real-world actions.
• BS in Computer Science, Engineering, or related technical field, or equivalent experience.
• Experience with QA for third-party API integrations.
• Experience with logging infrastructure and instrumentation to measure AI product quality.
• Excellent written and verbal communication; ability to present quality status, risks, and trade-offs to senior executive leadership.

Benefits

Not specified in the provided posting.