AI Evaluation Engineering Manager

Quick Facts

AI Evaluation Engineering Manager role within the AI Engineering team, focused on evaluating and validating advanced generative AI solutions.

Description

Lead a team dedicated to evaluating and validating advanced generative AI capabilities, including intelligent agents and digital assistants. Drive software quality and engineering best practices to deliver robust, reliable AI products, and scale evaluation processes across enterprise initiatives.

Responsibilities

• Provide technical support and clear direction to keep the team aligned with company goals and capable of evaluating advanced AI projects.
• Oversee performance of individuals and the team, fostering a learning culture by identifying and addressing training and development needs.
• Own AI engineering product quality to ensure solutions are robust, reliable, and meet high standards.
• Establish and supervise the implementation of engineering best practices to ensure consistent, excellent development processes.

Requirements

• Proven experience leading software quality, test automation, validation, and AI evaluation teams.
• Experience delivering enterprise-scale software and GenAI solutions across hybrid cloud and embedded/edge environments.
• Deep expertise in GenAI architecture and system design (LLMs/SLMs, multimodal, speech, real-time), including prompt engineering, agentic systems, tool use/orchestration, and RAG.
• Hands-on experience with AI development and evaluation ecosystems and libraries.
• Deep understanding of AI evaluation methodologies for non-deterministic systems, including metric design, RAG quality assessment, output reliability and safety evaluation, A/B testing, and human-in-the-loop validation.
• Expertise in AI observability and monitoring: tracing, telemetry, evaluation data collection, root-cause analysis, and evaluation automation (synthetic data, scenario expansion, LLM-assisted evaluation).

Benefits

• Medical, dental, and vision benefits
• Paid time off plan (vacation, holidays, volunteer)
• 401(k) savings plans
• Health Savings Account (HSA)
• Flexible Spending Accounts (FSAs)
• Health Lifestyle Programs
• Employee Assistance Program
• Voluntary benefits and employee discounts
• Career development
• Incentive bonus
• Disability benefits
• Life insurance
• Parental leave and adoption benefits
• Tuition reimbursement

Additional Information

• Visa sponsorship is not available for this position.
• Employment is conditioned upon successful completion of a drug screen.