November 19, 2024

Staff Performance Optimization Engineer, AI Infrastructure

Senior • On-site

$140,000 - $348,000/yr

Palo Alto, CA

What to Expect
As a Staff Performance Optimization Engineer, you will play a crucial role in enhancing the efficiency and scalability of Tesla’s AI infrastructure. Your primary focus will be on accelerating the development and deployment of machine learning models by optimizing the performance of our training systems. This involves collaborating closely with cross-functional teams, such as data scientists, ML Engineers, and Hardware Dpecialists, to identify and resolve performance bottlenecks. You will design and implement sophisticated instrumentation tools to monitor and troubleshoot complex GPU-CPU interactions, ensuring that our AI models are not only accurate but can be delivered as quickly as possible. Additionally, you will develop and refine data-driven strategies to optimize existing software pipelines, contributing directly to the rapid iteration and deployment of cutting-edge AI technologies.


With your deep expertise in GPU programming, low-level software development, and performance analysis, you will help drive Tesla's AI initiatives forward, making significant contributions to our autonomous driving and robotics projects. Expect a dynamic, fast-paced environment where innovation and collaboration are paramount, and every improvement you make has a direct impact on advancing some of the most ambitious technological goals in the industry.

What You’ll Do
  • Work with a wide variety of teams at Tesla to accelerate time-to-market for new ML models
  • Design, implement, and deploy low-overhead instrumentation methods for troubleshooting performance issues
  • Analyze collected telemetry, identifying bottlenecks and designing practical solutions to overcome those bottlenecks
  • Develop data-driven performance improvements to existing software pipelines
  • Working with each team to validate these performance improvements and incorporate them into production training runs
  • Help make Tesla's ambitious AI-related products and services a reality
What You’ll Bring
  • A deep understanding of the internals of GPU-based training and inferencing workloads, especially handoffs of data and computation between host CPUs and GPUs
  • Real-world knowledge of supporting languages and libraries used in large-scale AI training runs (CUDA/ZLUDA, OpenCL, PyTorch, Tensorflow, GPUDirect and other RDMA-enabling services)
  • Experience developing and tuning low-level software using languages like C, x86 assembly, and Rust
  • Practical experiences using different performance analysis techniques (profiling, tracing, simulative analysis) and when each should be applied
  • Excellent spoken and written communication skills, including the ability to concisely communicate data-driven root causes of performance issues and how they can be remedied
  • An irrational love for high-performance computing and extracting the maximum number of productive FLOPs from modern AI-oriented architectures
Compensation and Benefits
Benefits

Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:

  • Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deduction
  • Family-building, fertility, adoption and surrogacy benefits
  • Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution
  • Company Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSA
  • Healthcare and Dependent Care Flexible Spending Accounts (FSA)
  • LGBTQ+ care concierge services
  • 401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits
  • Company paid Basic Life, AD&D, short-term and long-term disability insurance
  • Employee Assistance Program
  • Sick and Vacation time (Flex time for salary positions), and Paid Holidays
  • Back-up childcare and parenting support resources
  • Voluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance
  • Weight Loss and Tobacco Cessation Programs
  • Tesla Babies program
  • Commuter benefits
  • Employee discounts and perks program

    Expected Compensation

    $140,000 - $348,000/annual salary + cash and stock awards + benefits

    Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.