Engineering Manager, Private Cloud Compute Inference Routing
Apple
Seattle, WA • 17m ago • On-site
Senior
On-site
225,600 - 381,600 USD/yr
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Potential salary for Engineering Manager, Private Cloud Compute Inference Routing role
Improve your resume now
$225,600 – $381,600/year
Quick Facts
- Role: Engineering Manager for Private Cloud Compute (PCC) inference routing
Description
You will lead the design and delivery of PCC inference routing and request scheduling that sends every inference request to the right compute across a global, multi-region fleet—maintaining low latency, high availability, and PCC privacy guarantees. The work includes evolving routing and scheduling strategies and building inference AI gateway capabilities that authenticate, meter, prioritize, and stream responses while preserving end-to-end encryption. In your first year, you will grow the organization, set a technical roadmap, and measurably improve routing latency, availability, and fleet utilization.
Responsibilities
- Recruit, develop, and lead engineers and engineering managers for PCC inference routing and request scheduling
- Set technical strategy with senior engineers for architecture and trade-offs in global traffic management, capacity-aware placement, load balancing, and failover
- Translate organizational objectives into actionable roadmaps, execution plans, and measurable results
- Oversee adaptive routing, admission control, prioritization, and overload management to maintain responsiveness and fairly share capacity across competing workloads
- Evolve inference AI gateway capabilities (authentication, token-aware rate limiting and quotas, model/version routing, fallback and retries, streaming response delivery) using request metadata without exposing request content
- Collaborate with model serving and infrastructure leaders on prefill/decode disaggregation and cache locality optimizations within PCC’s privacy model
- Develop and scale operations with observability, SLOs, testing, safe deployment, and incident management in partnership with SRE
- Improve efficiency via benchmarking and capacity-aware scheduling using queueing delay, latency, throughput, and utilization
- Provide transparent updates on team health, technical risks, and strategic trade-offs, turning complex issues into decisions
- Partner across Siri AI, Apple Intelligence, infrastructure, security, and SRE teams to deliver integrated routing capabilities
Requirements
- Extensive software engineering experience leading teams building and operating large-scale distributed systems in production
- Proven ability to recruit, develop, and retain high-performing engineers and leaders
- Experience delivering API gateways and large-scale request routing/load balancing/scheduling/traffic management systems
- Expertise in LLM inference serving, including request routing, KV-cache placement/reuse, prefill/decode disaggregation, and latency/throughput/accelerator trade-offs
- Track record shipping reliable high-availability systems, including incident management and on-call/reliability practices
- Strong technical depth and server development background in Java, C/C++, Go, or Swift
- Ability to drive alignment and execution across teams in complex, matrixed environments
- Bachelor’s degree in an engineering discipline or equivalent professional experience
Benefits
Opportunity to build technically grounded, large-scale systems with broad impact on Siri AI and Apple Intelligence performance, reliability, and inference capacity utilization.
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
Engineering Manager, Ranking Platform
Senior
Technology
Remote
$230k - $322k/yr
5 days ago

Principal Machine Learning Engineer (Agentic AI for Cloud Security)
PaloAlto Networks
Senior
Technology
CA, United States of America · On-site
$163k - $264k/yr
5 days ago

Advanced AI Workloads Performance and Scalability Engineer
AMD (Advanced Micro Devices)
Senior
Technology
San Jose, CA · On-site
$210k - $210k/yr
11 days ago

Principal Machine Learning Engineer (AI)
Next Step Systems
Senior
Technology
Remote
$170k - $200k/yr
1 days ago

Senior Staff Software Engineer, AI/ML Inference Engineering
ServiceNow, Inc.
Senior
Technology
San Diego, CA · On-site
$181k - $317k/yr
11 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Principal Solution Architect, Platform Engineering
FIS
Senior
Technology
FL · On-site
N/A
2 days ago

Senior Staff Software Engineer, AI/ML Inference
ServiceNow, Inc.
Senior
Technology
Kirkland, WA, United States of America · On-site
$201k - $352k/yr
11 days ago

Senior Software Engineer, Compute Scheduling
ServiceNow
Senior
Technology
Santa Clara, CA · Hybrid
$143k - $243k/yr
4 days ago
Senior LLM Inference Quality Engineer
Apple
Senior
Technology
Seattle, WA · On-site
$175k - $309k/yr
5 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights