Machine Learning Engineer, AI Model Evaluation & Quality

Quick Facts

• Role: Machine Learning Engineer focused on evaluation and quality for AI features in creativity applications

• Focus: Defining how to measure quality for ship/no-ship decisions

Description

Own the evaluation and quality of AI features across Creativity Apps. Collaborate with ML engineering, infrastructure, product, and design teams to translate models and use cases into evaluation methodology, testing strategy, and quality metrics aligned with what creators actually experience.

Responsibilities

• Define evaluation strategy and metrics for ML models, LLMs, and AI features tailored to the model, product, and use cases

• Curate and create evaluation datasets, measure and improve dataset quality, and run iterative improvement loops so datasets stay representative

• Design, build, maintain, and optimize evaluation infrastructure at scale with reproducible runs, tracked baselines, and comparisons between model versions

• Apply ML, LLMs, and agentic workflows to evaluation, including automated judging, test generation, and analysis

• Drive end-to-end evaluation and quality from model checkpoints to integrated, shipping features; triage issues and drive them to resolution

• Report quality clearly by converting evaluation results into ship/no-ship risk and recommendations for technical and non-technical audiences

• Partner with cross-functional teams and make educated, critical judgments in ambiguous situations

Requirements

• 5+ years of industry experience in machine learning or software engineering focused on ML/AI model evaluation and quality

• Experience with evaluation methodology for ML models, LLMs, and AI products/features

• Strong understanding of the ML development cycle from data and training through evaluation, deployment, and monitoring

• Solid machine learning fundamentals and LLM knowledge, including agentic workflows

• Track record applying LLMs or agentic workflows to ML evaluation and quality

• Experience in data curation, dataset creation, and dataset quality with iterative improvement workflows

• Experience designing, implementing, maintaining, or optimizing ML evaluation infrastructure at scale

• Experience designing evaluation metrics based on the model, the product, and its use cases

• Experience driving end-to-end evaluation and quality for AI products

• Fluency in quality reporting to both technical and non-technical audiences

• Strong communication skills and experience working with cross-functional partners

• Proficient in Python and/or Swift or comparable languages

• Bachelor’s degree in Computer Science, Data Science, Machine Learning, or a related field, or equivalent experience

Preferred Qualifications

• Knowledge of the software development life cycle and software quality assurance methodologies

• Experience with objective image and video quality evaluation and metrics

• Knowledge of color science or subjective image/video quality review experience

• Passion for creative applications; experience as a creator (photo, video, music, or design) is a plus

• Excitement about learning new technology and applying it to practical problems

• Familiarity with iOS and macOS