Machine Learning Engineer, AI/ML Model Quality

Quick Facts

• Role: Machine Learning Engineer, AI/ML Model Quality

Description

Own the data and model health behind ML and GenAI features for Wallet, Payments, and Commerce. Build validation frameworks, telemetry and observability metrics, and synthetic data generation and monitoring pipelines to ensure models are trained, evaluated, and deployed on trustworthy data.

Responsibilities

• Curate, analyze, and maintain gold-standard ground-truth datasets for continuous model evaluation (ML and GenAI).

• Audit training data for systemic bias and fairness gaps; monitor for bias introduced by data drift.

• Define and analyze telemetry across GenAI workflows (e.g., output coherence, latency, task completion, regression).

• Design privacy-preserving mechanisms to detect model degradation in production.

• Define and own ML observability metrics (performance, output distributions, training-serving skew, silent degradation, feature drift).

• Build observability dashboards and reporting workflows for real-time model health visibility.

• Identify degradation patterns and domain-specific failure modes using telemetry analysis; recommend actions for model and data teams.

• Define, track, and report core data quality metrics (completeness, accuracy, timeliness, validity).

Requirements

• Bachelor’s degree with exceptional hands-on ML/AI model quality experience; M.S./Ph.D. preferred in ML, CS, Data Science, Statistics, Mathematics, Engineering, or a related quantitative field.

• 3+ years defining and tracking ML model health metrics in production, including monitoring, feature drift detection, and observability instrumentation.

• 3+ years in ML/data science/analytical roles with a strong focus on data quality, model evaluation, or ML observability in production.

• Proficiency in Python (Pandas, NumPy, Scikit-learn), PyTorch, and SQL.

• Experience with distributed computing for large-scale data (PySpark/Spark/distributed SQL).

• Strong statistical understanding: hypothesis testing, distribution analysis, data drift detection, statistical process control.

• Experience with GenAI/LLM systems: quality failure modes, output evaluation, and telemetry instrumentation.

• Strong communication to translate model/data health risks into actionable insights for engineering and non-technical stakeholders.

Benefits

• Influence AI/ML quality at the foundation of ML features used by hundreds of millions of users.

• Work at the intersection of machine learning, statistical rigor, and production systems, collaborating with Data Engineering, Privacy, and Legal teams.

• Central ownership of ML/AI quality through observability, telemetry analytics, and privacy-preserving degradation detection.