Principal Machine Learning Engineer

Quick Facts

Role: Principal Machine Learning Engineer (Principal / Staff+, Individual Contributor) on a Personalization & Decisioning Platform.

Focus: Hands-on, backend-heavy engineering to design and build production systems for large-scale ML decisioning (one-to-one offers, content, and channel decisions).

Pay range: $144,200.00 – $288,400.00 base (plus eligible bonus/commission/short-term incentive and equity target).

Description

Join a personalization and decisioning platform team to build and operate a unified, large-scale machine learning platform that supports reliable one-to-one offer, content, and channel decisions across millions of customers. As the most senior engineer, you will set technical direction for how ML systems are designed, built, tested, deployed, and operated at scale. You will partner with data engineering, MLOps, product, and platform teams to ensure models are maintainable, observable, testable, and safe to change.

Responsibilities

• Architect and build production-grade backend services (APIs, microservices) and batch/streaming pipelines with strict latency, throughput, and reliability requirements.
• Design data contracts and training-to-serving interfaces with training-serving parity and point-in-time correctness.
• Own versioned, queryable feature stores and point-in-time lookup APIs to eliminate feature leakage and support training and real-time inference from one code path.
• Design and optimize distributed computing workflows for containerized batch jobs, parallel training, and optimization solves under tight operational SLAs.
• Set and enforce engineering standards: modular design, dependency management, testing, CI/CD, and documentation; add contract tests, reproducible environments, infrastructure-as-code, and rollback-safe deployments.
• Drive observability and monitoring for software and model health (service-level metrics, drift, calibration) with automated alerting.
• Lead design and code reviews, prevent and resolve technical debt, and raise the engineering bar via mentorship and coaching.
• Build and productionize causal inference/incrementality models, reinforcement-learning-based policies, and mixed-integer optimization for constrained allocation problems; support experimentation and counterfactual validation.

Requirements

• 7+ years of progressively complex software, ML, and data science/engineering experience, including 5+ years building and operating production backend systems or data-intensive platforms at scale.
• 5+ years building, deploying, and owning ML systems in production with strong software engineering best practices.
• Expert-level Python and SQL; deep understanding of software design patterns, API design, and distributed systems fundamentals.
• Experience architecting scalable data-intensive backend systems (microservices, REST APIs, batch/streaming pipelines) in Azure, AWS, or Google Cloud Platform.
• Hands-on containerization and orchestration experience (Docker, Kubernetes) and ML/data pipeline orchestration tools (Kubeflow, Airflow, or similar).
• Demonstrated technical leadership via direction setting, running design/code reviews, and mentoring engineers.
• Proven ability to anticipate risk, prevent production issues, and communicate tradeoffs to both engineers and business stakeholders.

Benefits

• Medical, dental, and vision coverage.
• Paid time off.
• Retirement savings options.
• Wellness programs.
• Comprehensive benefits package for eligible colleagues and their families.