Machine Learning Engineer, AI/ML Model Quality

Machine Learning Engineer, AI/ML Model Quality

Quick Facts

• Focus: ML and GenAI model quality, data validation, and production monitoring
• Scope: Wallet, Payments, and Commerce
• Collaboration: Data Engineering, Privacy, and Legal

Description

Own the health of the data ecosystem that powers ML and generative AI features. Build validation frameworks, synthetic data generation models, and monitoring pipelines that keep training and evaluation datasets trustworthy—so models are trained, evaluated, and deployed on data teams can trust.

Responsibilities

• Curate, analyze, and maintain gold-standard ground-truth datasets for continuous evaluation and validation across ML and GenAI
• Audit training data for systemic bias and fairness gaps before deployment; implement ongoing checks for bias caused by data drift
• Define and analyze telemetry across GenAI workflows (e.g., output coherence, latency, task completion rates, regression patterns)
• Design privacy-preserving mechanisms to detect production model degradation
• Define and own ML observability metrics (performance, output distributions, training-serving skew, silent degradation, feature drift) and turn production signals into actionable insights
• Build observability dashboards and reporting workflows for real-time model health visibility
• Identify degradation patterns and domain-specific failure modes in GenAI via telemetry analysis and recommend fixes for model/data teams
• Define, track, and report data quality metrics (completeness, accuracy, timeliness, validity)

Requirements

• Bachelor’s degree with strong hands-on experience in ML/AI model quality; M.S. or Ph.D. strongly preferred in ML/CS/Data Science/Statistics/Math/Engineering or related quantitative fields
• 3+ years experience defining and tracking ML model health metrics in production (monitoring, feature drift detection, observability instrumentation)
• 3+ years experience in ML, data science, or related analytics focused on data quality, model evaluation, or ML observability in production
• Proficiency in Python (Pandas, NumPy, Scikit-learn), PyTorch, and SQL for analysis, metric creation, and validation
• Experience with large-scale dataset analysis using distributed computing (PySpark/Spark/distributed SQL)
• Solid statistical knowledge (hypothesis testing, distribution analysis, data drift detection, statistical process control)
• Experience with GenAI/LLM systems, including quality failure modes, evaluation approaches, and telemetry instrumentation
• Strong communication skills for actionable insights to engineering and non-technical stakeholders

Benefits

• Opportunity to be at the center of ML/AI quality and help ensure high-quality data for models used by hundreds of millions of users
• Work across conventional ML and latest generative AI systems, with ownership of observability and telemetry that drive decisions