ML Systems Engineer (Model Serving & Runtime)

Quick Facts

• Role: ML Systems Engineer (Model Serving & Runtime)

• Focus: End-to-end local inference stack for Windows and macOS desktop apps

Description

Own the end-to-end local inference stack to make models run fast and reliably on Windows (x64/ARM64) and macOS (Apple Silicon). You select and integrate inference engines (MLX, llama.cpp, ONNX Runtime, TensorRT-LLM), decide how models are quantized and packaged, and build the runtime connection to the Tauri frontend. You collaborate with AI Research and ship the runtime that makes the pipeline work.

Responsibilities

• Model Serving Layer: Build and maintain the inference runtime for Windows and macOS desktop apps

• Engine Integration: Integrate MLX (Apple), llama.cpp (cross-platform), ONNX Runtime, TensorRT-LLM, and vLLM; evaluate emerging engines

• Hardware-Aware Dispatch: macOS via Metal/MPS; Windows via CUDA/DirectML/Vulkan; CPU fallback with optimal threading

• Model Packaging & Conversion: Own GGUF, MLX weights, Safetensors, and Core ML pipelines; implement automated conversion and validation

• Quantization & Optimization: Implement GPTQ, AWQ, GGML, and EXL2 pipelines; profile latency/memory/battery across device tiers

• Download & Update Pipeline: Sidecar binary delivery, checksum verification, staged rollouts, and delta updates for model weights

• Cross-Compilation CI/CD: Multi-arch builds (rustup targets, Xcode, MSVC, clang), device-farm testing, and regression detection

• Compatibility Matrices: Document supported models × OS × hardware; create migration guides for new model releases

• AI Research Collaboration: Quantization strategy, kernel selection/kernel fusion, speculative decoding, and attention optimization

• Debug Platform Issues: Metal shader compilation, CUDA driver regressions, ARM64 Windows quirks, and memory mapping edge cases

Requirements

• 3+ years shipping native desktop applications on Windows and macOS

• Strong proficiency in Rust (Tauri backend), C++ (llama.cpp), and Swift/Obj-C (MLX)

• Deep understanding of inference pipelines: tokenization, KV cache, sampling, speculative decoding, attention kernels

• Hands-on experience with MLX, llama.cpp, or equivalent local inference engines

• Experience with cross-compilation toolchains and multi-arch CI/CD (e.g., GitHub Actions, self-hosted runners)

• Fluency with GPU compute APIs: Metal/MPS, CUDA, DirectML, Vulkan

• Fluency with model formats: GGUF, Safetensors, ONNX, Core ML, MLX arrays

• Systems programming experience: memory mapping, shared libraries, dynamic loading, zero-copy buffers

Benefits

• Equity + revenue share