Cloud Services Software Engineer

Quick Facts

• Role: Cloud Services Software Engineer (distributed backend)
• Focus: Scalable, resilient infrastructure for AI training workloads
• Key areas: distributed systems, ML, high-performance computing, Kubernetes-based microservices

Description

Joining the DGX Cloud team means contributing to the infrastructure that powers AI research. This role develops and optimizes AI infrastructure services for peak performance and high resiliency, including backend (micro-)services orchestrated by Kubernetes for large-scale training workflows and APIs that integrate with resiliency stacks. You will also build job submission abstractions that handle infrastructure failures, manage long-running job lifecycles, and support modular deployment to on-prem AI clusters.

Responsibilities

• Develop solutions at the intersection of machine learning, distributed systems, and high-performance computing
• Design, develop, and optimize (micro-)services orchestrated by Kubernetes for large-scale AI training workflows on training supercomputers at major CSPs
• Co-design and implement APIs integrating services with resiliency stacks (telemetry, break/fix automation, checkpoint and execution)
• Craft a submission abstraction to simplify long-running training job submission, including failure handling, auto-restarts, and user advisories
• Build modular services deployable on-premises AI clusters using NVIDIA hardware and cloud services

Requirements

• Bachelor’s degree or higher in Computer Science or a related technical field (or equivalent experience)
• 5+ years of hands-on backend development experience, preferably with Python, Go, C/C++, or similar high-performance languages
• Proven experience building and scaling large-scale distributed systems
• Experience with cloud computing platforms (AWS, Azure, GCP)
• Experience with container technologies (Docker, Kubernetes)
• Experience with HPC/AI platforms such as Slurm