Senior Staff Software Engineer, AI/ML Inference

Quick Facts

• Role: Senior Staff Software Engineer for AI/ML Inference Engineering

• Focus: Architecture, optimization, and operational excellence of an AI inference platform

Description

Lead the end-to-end architecture and operational excellence of an AI inference platform across multiple backends on Kubernetes. You will own the inference lifecycle from onboarding cutting-edge models to deploying optimized inference engines in production, and you will architect an intelligent gateway layer using Envoy and ext_proc for access control, observability, and advanced request routing. The role emphasizes pragmatic engineering with measurable performance impact.

Responsibilities

• Own the end-to-end AI inference lifecycle: model onboarding through production deployment

• Architect and optimize inference engines across multiple backends on Kubernetes

• Build an intelligent gateway layer using Envoy and ext_proc for access control, observability, and request routing

• Apply deep expertise in GPU compute, LLM APIs, inference frameworks, containerization, orchestration, and gateway architecture

Requirements

• 8+ years of backend software engineering experience

• 4+ years designing and operating distributed systems

• Bachelor’s degree in computer science, software engineering, or closely related technical field (or equivalent practical experience)

• Expert proficiency in Go or Python, OOP, design patterns, and time/space-efficient algorithms

• Model-serving and runtime experience such as vLLM, TensorRT-LLM, or Triton Inference Server

• Exposure to LLM APIs or ML-serving systems (or ability to ramp up quickly)

• Expertise in writing efficient, modular, quality object-oriented code

• Knowledge of core AI/ML techniques and algorithms

• Knowledge of unit testing, profiling, and code tuning

Benefits

• Health plans, including flexible spending accounts

• 401(k) Plan with company match, ESPP, matching donations

• Flexible time away plan and family leave programs

• Base pay guideline: $201,300 - $352,300 plus equity (when applicable) and variable/incentive compensation (location-dependent)