Senior Software Engineer (Managed Data Transfer Control Plane)

Quick Facts

• Role: Senior Software Engineer, Managed Data Transfer (MDT)

Description

Crusoe’s Managed Data Transfer (MDT) service moves petabyte-scale training data from cloud sources (starting with AWS S3) into Crusoe storage using a single API call. This role builds the MDT control plane—job API, scheduler, and job state—so customer transfers are coordinated, resumable, and safe to retry or cancel. You’ll shape a provider-neutral API and help take the service from early commits to a preview launch with design-partner customers.

Responsibilities

• Design and build the data-transfer API (create, list, get, cancel) with a provider-neutral source/destination model

• Implement the scheduler and job state machine that shards bucket listings into work units, places them on regional worker pools, and tracks progress, retries, and resume behavior

• Ensure correctness at scale with end-to-end checksum verification, safe retry/cancel behavior, and clear errors for permission/destination issues

• Implement secure cross-cloud identity with short-lived least-privilege credentials (AWS role assumption first, other clouds next), integrated with IAM and audit logging

• Ship customer surfaces including the CLI and, with partners, Terraform and Console experiences for creating and monitoring transfers

• Instrument the service with metrics, logs, and alerts; write runbooks; participate in the team’s on-call rotation

• Collaborate with Storage, Network, IAM, and Cloud Console teams, and partner with Product and Solutions Architects on early onboarding

Requirements

• 5+ years building cloud infrastructure or platform services, including taking a service from zero to general availability

• Production experience designing and shipping backend services and APIs (ideally in Go) using gRPC or REST, relational databases, and Kubernetes

• Distributed systems judgment for long-running, failure-prone work (idempotency, retries with backoff, leases, checkpointing, partial failure)

• Ability to design public APIs that customers automate against and that can evolve without breaking changes

• Security mindset: authentication, authorization, least privilege, and careful handling of customer credentials

• Operational ownership: run production services, participate in on-call, build dashboards and alerts

• Strong collaboration and writing skills for cross-team design docs

• Bachelor’s degree in Computer Science/Engineering (or equivalent professional experience)

Bonus Points

• Experience with large-scale data movement tools (S3 multipart uploads, rclone, s5cmd, AWS DataSync, Google Storage Transfer Service)

• Cross-cloud identity experience (AWS IAM/STS, GCP service-account impersonation, Azure workload identity)

• Experience with storage systems (object storage, NFS, parallel file systems) or high-throughput networking

• Job orchestration experience (Temporal, Argo, Kubernetes operators)

• Experience building on GPU or AI infrastructure

Benefits

• Competitive compensation and equity packages, including Restricted Stock Units

• Paid time off, paid holidays, and leave of absence programs

• Comprehensive health, dental, and vision insurance

• Employer contributions to an HSA account

• Paid parental leave

• Paid life insurance, short-term and long-term disability

• Professional development and tuition reimbursement

• Mental health and wellness support

• Commuter benefits, cell phone stipend, and daily meals allowance

• 401(k) retirement plan with company match up to 4% of salary

• Volunteer time off

• Global travel insurance and emergency assistance

• Additional location-specific perks and programs