GPU & ML Infrastructure Engineer

Quick Facts

• Role: GPU & ML Infrastructure Engineer

• Focus: End-to-end GPU/compute health data generation using repeatable workloads and multi-source telemetry

Description

You will own the data generation side of a GPU/ML hardware monitoring effort: porting test procedures to new GPU and edge node types, automating deployment and runs, collecting consistent telemetry, and delivering documented datasets to storage. Responsibilities include building a controlled benchmark and stress suite—especially LLM inference and training workloads (Llama-family, quantized and full precision)—with strict run-to-run reproducibility. You will also implement logger correctness, data-quality checks, and validation to ensure the modeling team receives reliable data.

Responsibilities

• Port the test procedure to new hardware and document what changes and why

• Build and maintain the benchmark/workload suite (synthetic kernels and LLM workloads, plus new targets as they expand)

• Ensure logger correctness and consistency (sampling rates, timestamps, consistent field names, aligned clocks across sources)

• Automate repeatable, unattended deployment, execution schedule, collection, and teardown

• Implement automated data-quality checks (missing/irregular samples, clock mismatches, workload/telemetry misalignment)

• Own the storage handoff with a consistent, documented dataset layout and full run metadata

• Document each hardware target

Requirements

• Experience deploying LLM inference and training workloads on GPUs, including quantized models

• Ability to diagnose sensor and sampling problems in time-series hardware data

• Ability to make GPU workloads reproducible run-to-run by controlling nondeterminism

• Strong understanding of Linux systems, GPU driver stacks, process orchestration, scheduling, and timing behavior

• Experience collecting hardware telemetry programmatically via in-band (NVML, DCGM) and out-of-band (BMC/IPMI/Redfish)

Benefits

• Full-time role with 40 hours per week

• Opportunity to build the data-generation system that the modeling team depends on

• Work with a fleet spanning multiple NVIDIA datacenter GPU generations and embedded/edge platforms