Senior Software Engineer - Cluster Networking

Quick Facts

• Role: Senior Software Engineer focused on Cluster Networking

• Scope: Technical ownership of networking for GPU superclusters used for large-scale AI training

Description

You will be the technical owner of how GPU clusters communicate internally and externally, including the Kubernetes networking architecture, CNI data plane, overlay mesh, and inter-cluster connectivity across regions and cloud providers. Responsibilities include designing and scaling overlay networks (CNI/mesh/VPN topologies and gateways), operating L7 gateways/load balancers/tunnels, eliminating networking scale ceilings, and building scale-test environments to prevent regressions. You will also diagnose complex distributed network issues and work with cloud and infrastructure providers to support new cluster bring-ups.

Responsibilities

• Own and evolve Kubernetes networking architecture for GPU clusters at multi-thousand-node scale

• Design, operate, and scale overlay network components: CNI, mesh, VPN topologies (Tailscale, WireGuard), and gateways connecting control and data planes

• Design, operate, and scale L7 gateways/load balancers/tunnels (Envoy, Cloudflare)

• Identify and resolve scale ceilings such as packet loss under load, control-plane saturation, IP address management exhaustion, and high-node failure modes

• Build scale-test environments and validation suites to catch networking regressions before production training runs

• Debug ambiguous cross-stack issues where symptoms in Slurm/training trace back to routing, mark collisions, stale routes, or saturated tunnels

• Partner with cloud and neocloud providers on network topology, requirements, and capabilities

• Provide senior technical judgment and depth in cluster networking

Requirements

• BS/MS in Computer Science, Electrical Engineering, or related field, or equivalent experience

• 6+ years of professional experience in systems, network, or infrastructure software engineering

• Deep command of Kubernetes networking architecture and CNI standards; production experience operating Calico strongly preferred

• Proficiency designing and maintaining modern mesh and VPN networking topologies (Tailscale, WireGuard, or equivalent)

• Strong Linux networking fundamentals: routing, netfilter and iptables/nftables, packet marking, network namespaces, and interactions with container runtimes

• Demonstrated ability to debug distributed network problems at scale using packet capture and tracing to find root cause

• Proficiency in Go, Python, C, or a comparable systems language

• Clear written and verbal communication; effective collaboration across time zones

Benefits

• Equity

• Benefits package

• Base salary range depends on location and experience (Level 4: 184,000–287,500 USD; Level 5: 224,000–356,500 USD)