Senior LLM Inference Quality Engineer

Quick Facts

• Role: Senior LLM Inference Quality Engineer

• Focus: Qualification of LLM inference for correctness, consistency, and stability in production

• Scope: Data-center scale distributed ML serving, evaluation automation, and release gating

Description

Own the qualification of an LLM inference pipeline in production, building automation frameworks, evaluation methodology, and qualification strategy that validate every model, prompt, serving stack, and infrastructure change before release. Lead end-to-end test and qualification for distributed ML serving systems, ensuring outputs remain correct, consistent, and stable across model/weights/prompts/runtimes, quantization, and accelerator hardware changes. You will also build AI-assisted QA tooling and observability to detect regressions, drift, and failure modes in inherently non-deterministic, probabilistic generative outputs.

Responsibilities

• Design and develop qualification frameworks for LLM inference at data-center scale

• Build LLM inference output-correctness automation frameworks for deployment and platform validation

• Architect end-to-end test and qualification strategy (unit, integration, regression, end-to-end)

• Create validation harnesses independent of upstream dependencies (mocking, trace replay, synthetic traffic)

• Gate releases in CI with relevant evals and before/after comparison

• Build prompt-level and scenario-based regression suites; track quality over time

• Validate tool-use, function-calling, and multi-turn agentic workflows for correctness and coherence

• Add safety, refusal, and policy-compliance coverage across releases

• Probe risks: prompt injection, data leakage, hallucinations/ungrounded output, and cost/latency regressions

• Build adversarial/robustness tests (long-context, ambiguous, underrepresented inputs)

• Qualify inference optimizations (quantization, KV-cache, speculative decoding, batching) against full-precision references

• Validate output equivalence across serving backends and accelerators

• Perform functional, performance, fault-injection, and reliability testing; benchmark quality vs latency/throughput

• Develop AI-powered/agentic QA tooling for smarter test generation and defect triage

• Define metrics, logging, distributed tracing, and observability for correctness, quality, drift, latency, and health

• Use Splunk/internal tools to root-cause failures in complex LLM serving systems

• Conduct regression, performance, and integration testing across platforms (including iOS/macOS) for on-device and server-backed features

• Build production AI infrastructure software in Python, C++, and/or Swift with modularity and performance principles

Requirements

• 5+ years experience with hardware and/or software development lifecycle processes

• 5+ years experience in one or more compiled languages (C, C++, Objective-C/Swift)

• Proficiency in one or more scripting languages (Python, Go, JavaScript)

• Experience using AI/LLMs to improve testing (test-case generation, defect detection, reducing manual QA)

• Familiarity with LLM serving/inference optimization (batching, KV-cache, quantization, speculative decoding, distributed/multi-accelerator serving)

• Experience with CI/CD, Kubernetes, containers, and cloud-native systems; integrate automated evals into release pipelines

• Experience with ML/LLMs and common frameworks (PyTorch, Hugging Face Transformers, or comparable)

• Experience with observability and distributed tracing for ML systems (Splunk, Prometheus, Grafana, LangSmith, or similar) and monitoring for model drift

• B.S., M.S., or Ph.D. in relevant fields or equivalent experience

Benefits

Benefits were not specified in the provided posting.