LLM Inference Quality Engineer
Apple
Seattle, WA • 20m ago • On-site
Senior
On-site
142,300 - 214,300 USD/yr
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Potential salary for LLM Inference Quality Engineer role
Improve your resume now
$142,300 – $214,300/year
Quick Facts
- Senior AI systems engineering role focused on LLM inference quality in production at data center scale.
Description
Own the correctness, consistency, and stability of LLM inference for every model, prompt, serving stack, and infrastructure change before it reaches users. Lead qualification systems and automation frameworks for output correctness, including evaluation for agentic workflows and tool-using models. Build AI-assisted QA tooling and define quality bars for “correct” and “stable” inference despite generative nondeterminism.
Responsibilities
- Design and develop qualification frameworks validating output correctness, consistency, and stability across model, weight, prompt, runtime, and infrastructure changes.
- Build the LLM inference output-correctness automation framework for large-scale deployment and platform validation.
- Architect end-to-end testing and qualification for distributed ML serving (unit, integration, regression, end-to-end).
- Create validation harnesses using mocking, trace replay, and synthetic traffic generation.
- Gate releases in CI with before/after evals for changes to models, weights, prompts, tool definitions, or runtimes.
- Create prompt-level and scenario-based regression suites and track quality across versions over time.
- Validate tool-use, function-calling, and multi-turn agentic workflows for correct invocation, argument formation, task completion, and coherence; detect wrong/empty/silently degraded outputs.
- Develop safety, refusal, and policy-compliance coverage to ensure guardrails and alignment behavior holds across releases.
- Probe AI risks including prompt injection, data leakage across users/permissions, hallucinations/ungrounded output, and cost/latency regressions.
- Build adversarial and robustness tests for long-context, ambiguous, and underrepresented inputs.
- Qualify inference optimizations (quantization, KV-cache, speculative decoding, batching) via output parity against full-precision references.
- Validate output equivalence across serving backends and accelerators.
- Run functional, performance, fault-injection, and reliability testing; benchmark quality vs latency/throughput trade-offs.
- Develop AI/agentic QA tooling to improve test-case generation, defect detection, flaky-test analysis, and triage automation.
- Define metrics, logging pipelines, distributed tracing, and observability tooling for correctness, quality, drift, latency, and system health.
- Use Splunk and internal debugging tools to root-cause failures in LLM serving systems.
- Perform regression/performance/integration testing across platforms, including iOS and macOS, for on-device and server-backed AI features.
- Develop production AI infrastructure software in Python, C++, and/or Swift with modularity, scalability, testability, and performance principles.
Requirements
- 1+ year experience with hardware and/or software development lifecycle processes.
- 1+ year experience in one or more compiled languages (C, C++, Objective-C/Swift).
- Proficiency in one or more scripting languages (Python, Go, or JavaScript).
- Experience using AI/LLMs to improve testing (test-case generation, defect detection, or reducing manual QA effort).
- Familiarity with LLM serving and inference optimization (batching, KV-cache, quantization, speculative decoding, distributed/multi-accelerator serving) and GPU-accelerated software.
- Experience with CI/CD, Kubernetes, containers, and cloud-native systems; integrate automated evals into release pipelines.
- Experience with ML/LLMs and common frameworks (PyTorch, Hugging Face Transformers, or comparable stacks).
- Experience with observability and distributed tracing for ML systems (Splunk, Prometheus, Grafana, LangSmith, or similar) and monitoring for model drift.
- B.S., M.S., or Ph.D. in Computer Engineering, Electrical Engineering, Computer Science, or equivalent experience.
Benefits
- Ship frontier AI quickly while maintaining rigorous correctness, safety, and user trust through automated qualification.
- Hands-on work across ML systems engineering, distributed inference, and quality engineering at data center scale.
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See moreSenior LLM Inference Quality Engineer
Apple
Senior
Technology
Seattle, WA · On-site
$175k - $309k/yr
5 days ago
Senior Applied AI Engineer
Flyps
Senior
Technology
Poznań, Pl-30, Poland · Remote
312k zł - 396k zł/yr
14 days ago

Senior Machine Learning Engineer, Evaluation & Reliability
Bask Health
Senior
Technology
Remote
N/A
2 days ago
Senior Machine Learning Engineer
Apple
Senior
Technology
Cupertino, CA · On-site
$185k - $325k/yr
8 days ago
Senior Machine Learning Engineer, Conversational AI (Agent & Inference Services)
Apple
Senior
Technology
Cupertino, CA · On-site
$185k - $325k/yr
1 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Senior AI Engineer
Motion Recruitment Partners, LLC
Senior
Technology
Boston, MA, United States of America · On-site
$200k - $225k/yr
7h ago

Senior Machine Learning Engineer, Agent Evaluation Platform
ServiceNow
Senior
Technology
Santa Clara, CA · On-site
$161k - $274k/yr
4 days ago
Senior AI Engineer
Sealed Air Corporation
Senior
Technology
Charlotte, NC · On-site
$93k - $161k/yr
12 days ago

Senior Software Engineer, AI
Thomson Reuters U.S. Inc.
Senior
Technology
Eagan, MN · On-site
$118k - $220k/yr
3 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights