Machine Learning Engineer, AI/ML Model Quality

Quick Facts

• Role: Machine Learning Engineer, AI/ML Model Quality
• Focus: ML/GenAI data quality, model validation, and production monitoring (observability)

Description

Build and maintain intelligent systems that define and enforce exceptional data quality for machine learning across Wallet, Payments, and Commerce. You will create validation frameworks, synthetic data generation models, and monitoring pipelines, ensuring models are trained, evaluated, and deployed on data the team can trust. Your work centers on ML observability metrics and telemetry analysis to detect drift, bias, and production degradation.

Responsibilities

• Curate, analyze, and maintain gold-standard ground-truth datasets for continuous model evaluation across ML and GenAI systems
• Audit training data for systemic bias and fairness gaps prior to deployment and add ongoing checks for drift-induced bias
• Define and analyze telemetry across GenAI workflows (e.g., output coherence, latency, task completion rates, regressions)
• Design privacy-preserving mechanisms to detect production model degradation
• Define and own ML observability metrics (model performance, output distributions, training-serving skew, silent degradation, feature drift)
• Build observability dashboards and reporting workflows for real-time model health visibility
• Identify degradation patterns and domain-specific failure modes in GenAI via telemetry analysis and recommend actions for model and data teams
• Define, track, and report key data quality metrics (completeness, accuracy, timeliness, validity)

Benefits

• At the center of ML and AI quality, owning data/telemetry-driven health of training and validation datasets at scale
• Work at the intersection of statistical rigor and production systems with Data Engineering, Privacy, and Legal teams
• High impact supporting ML features for hundreds of millions of users

Requirements

• 3+ years defining and tracking ML model health metrics in production (monitoring, drift detection, observability)
• 3+ years in ML/data science/analytics with a strong focus on data quality, model evaluation, or ML observability in production
• Proficiency in Python (Pandas, NumPy, Scikit-learn), PyTorch, and SQL
• Experience analyzing large-scale datasets with distributed computing (PySpark/Spark/distributed SQL)
• Solid statistical foundations (hypothesis testing, distribution analysis, drift detection, statistical process control)
• Experience with GenAI/LLM systems, including quality failure modes, evaluation approaches, and telemetry instrumentation
• Strong communication skills for engineering and non-technical stakeholders
• Bachelor’s degree with exceptional hands-on experience; M.S./Ph.D. preferred in a quantitative field