Staff Machine Learning Engineer (Training Services)

Quick Facts

Staff-level Machine Learning Engineer role focused on model training services (pre-training through post-training and distillation) for foundation, recommendation, embedding, speech, and vision models.

Description

Own and improve end-to-end model training services within Apple Cloud AI, with a focus on training architectures and algorithms. Translate deep knowledge of training and model behavior into faster, more efficient, and higher-quality models using the same compute.

Responsibilities

• Accelerate post-training, including supervised fine-tuning, LoRA/QLoRA, preference optimization (DPO), and teacher-student distillation (including multi-adapter and efficient in-loop teacher inference).
• Optimize training performance by eliminating bottlenecks across computation, memory, communication, and data loading using profilers, execution traces, and benchmarks.
• Advance distributed training by designing/optimizing data, tensor, pipeline, context, and expert parallelism (including MoE routing, load balancing, communication collectives, and compute-communication overlap).
• Improve model execution with efficient attention implementations, sequence/memory optimizations, and compiler or custom accelerator kernel optimizations.
• Improve convergence and stability via mixed-precision tuning, quantization-aware training, optimizer configuration, and parameter/optimizer-state sharding with model-quality validation.
• Optimize generation-in-the-loop training, including generation efficiency, weight synchronization, and coordinated training/inference.
• Make large runs reliable with fast checkpointing/resume, fault tolerance, and early detection of hangs or instability.
• Drive technical direction with ML researchers, converting one-off fixes into reusable optimizations and contributing to upstream training frameworks.
• Design training interfaces: build the APIs and abstractions for end-to-end managed workflows and low-level primitives (forward/backward passes, optimizer steps, sampling).

Requirements

• 8+ years of software engineering or ML systems experience.
• Several years of hands-on distributed training on hundreds of accelerators or more.
• Strong understanding of modern model architectures, especially transformers, including compute and memory implications.
• Hands-on experience optimizing distributed training performance (compute, memory, communication) using PyTorch Distributed, Megatron-Core, JAX/XLA, or equivalent.
• Strong programming skills in Python plus C++, CUDA, Triton, or equivalent performance-oriented technologies.
• Demonstrated ability to diagnose complex performance problems, implement optimizations, and validate improvements.
• Proven record of leading technical work across teams and communicating complex tradeoffs clearly.
• BS/MS/PhD in Computer Science, Machine Learning, or related field, or equivalent practical experience.

Benefits

Not specified in the provided posting.