ML Workload Platform Software Engineer

Quick Facts

• Own a Kubernetes-based platform for scheduling and running ML training, evaluation, and reinforcement-learning workloads.
• Compensated 24x7 on-call rotation.

Description

You will build and operate the ML workload platform that manages the full job lifecycle—capacity allocation, failure handling, and researcher visibility—on Kubernetes. The role includes designing workload networking and runtime isolation, as well as working with storage for checkpoints and training artifacts. You will troubleshoot production issues and improve maintainability through automation, infrastructure-as-code, and observability.

Responsibilities

• Own the platform that schedules and runs training, evaluation, and reinforcement-learning jobs on Kubernetes.
• Define capacity handling and failure behavior, and ensure researchers can determine whether issues are platform-related.
• Build networking and runtime isolation, including agent and sandbox environments.
• Partner with data infrastructure on checkpoint and training-artifact storage.
• Troubleshoot production failures and performance problems, leaving the system easier to operate than you found it.
• Build automation, infrastructure-as-code, and observability.

Requirements

• Production experience running a job platform, batch system, or Kubernetes controllers.
• Strong distributed systems judgment (retries, partial failure, and safe automation boundaries).
• Experience with ML training workflows and related infrastructure such as RL sandboxes, container networking (CNI), or checkpoint storage at scale.
• Direct GPU-cluster experience is a plus.
• Proficiency writing production software in Python or Go.
• Preference for extending solid open-source or existing systems rather than rebuilding.
• Experience collaborating with researchers/ML engineers or cross-functional teams with different goals.
• Ability to identify bottlenecks, propose fixes, and drive improvements in a fast-moving environment.

Benefits

• Weekly lunch stipend.
• Full health and dental benefits, including a separate mental health budget.
• RRSP matching, 401K, and pension scheme.
• 100% parental leave top-up for up to 6 months (either parent).
• Annual enrichment benefits and an education/learning stipend.
• 6 weeks of paid vacation (30 working days).
• $500 home office stipend and support for travel to other offices plus an annual company offsite.