Engineering Manager, Private Cloud Compute Inference Routing

Apple

Seattle, WA • 17m ago • On-site

Senior

On-site

225,600 - 381,600 USD/yr

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Potential salary for Engineering Manager, Private Cloud Compute Inference Routing role

Improve your resume now

$225,600 – $381,600/year

Quick Facts

  • Role: Engineering Manager for Private Cloud Compute (PCC) inference routing

Description

You will lead the design and delivery of PCC inference routing and request scheduling that sends every inference request to the right compute across a global, multi-region fleet—maintaining low latency, high availability, and PCC privacy guarantees. The work includes evolving routing and scheduling strategies and building inference AI gateway capabilities that authenticate, meter, prioritize, and stream responses while preserving end-to-end encryption. In your first year, you will grow the organization, set a technical roadmap, and measurably improve routing latency, availability, and fleet utilization.

Responsibilities

  • Recruit, develop, and lead engineers and engineering managers for PCC inference routing and request scheduling
  • Set technical strategy with senior engineers for architecture and trade-offs in global traffic management, capacity-aware placement, load balancing, and failover
  • Translate organizational objectives into actionable roadmaps, execution plans, and measurable results
  • Oversee adaptive routing, admission control, prioritization, and overload management to maintain responsiveness and fairly share capacity across competing workloads
  • Evolve inference AI gateway capabilities (authentication, token-aware rate limiting and quotas, model/version routing, fallback and retries, streaming response delivery) using request metadata without exposing request content
  • Collaborate with model serving and infrastructure leaders on prefill/decode disaggregation and cache locality optimizations within PCC’s privacy model
  • Develop and scale operations with observability, SLOs, testing, safe deployment, and incident management in partnership with SRE
  • Improve efficiency via benchmarking and capacity-aware scheduling using queueing delay, latency, throughput, and utilization
  • Provide transparent updates on team health, technical risks, and strategic trade-offs, turning complex issues into decisions
  • Partner across Siri AI, Apple Intelligence, infrastructure, security, and SRE teams to deliver integrated routing capabilities

Requirements

  • Extensive software engineering experience leading teams building and operating large-scale distributed systems in production
  • Proven ability to recruit, develop, and retain high-performing engineers and leaders
  • Experience delivering API gateways and large-scale request routing/load balancing/scheduling/traffic management systems
  • Expertise in LLM inference serving, including request routing, KV-cache placement/reuse, prefill/decode disaggregation, and latency/throughput/accelerator trade-offs
  • Track record shipping reliable high-availability systems, including incident management and on-call/reliability practices
  • Strong technical depth and server development background in Java, C/C++, Go, or Swift
  • Ability to drive alignment and execution across teams in complex, matrixed environments
  • Bachelor’s degree in an engineering discipline or equivalent professional experience

Benefits

Opportunity to build technically grounded, large-scale systems with broad impact on Siri AI and Apple Intelligence performance, reliability, and inference capacity utilization.

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report