Software Engineer II

We are the AI Frameworks team at Microsoft, part of the CoreAI organization, and we are at the forefront of driving innovation in large-scale AI. Our mission is to enable state-of-the-art large language model (LLM) training and inference through deep optimization across the full software and hardware stack. We work across PyTorch, ONNX, and other open AI frameworks, pushing the boundaries of performance, scalability, and efficiency on a wide range of hardware accelerators.

This position is part of a highly specialized sub-team focused on building the end-to-end software stack for Microsoft’s first-party (1P) AI accelerators. Our work is foundational to Microsoft’s long-term AI infrastructure strategy, powering next-generation models and services across Azure and Microsoft products. We collaborate closely with hardware architects, compiler teams, and model experts to co-design software solutions that unlock the full potential of custom silicon. The work is deeply technical and impactful—advancing AI systems used at global scale.

If you are passionate about AI systems, low-level performance optimization, and solving hard cross-discipline engineering problems, we invite you to join us and help shape the future of AI at Microsoft.

Required Qualifications:

• Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, or Python
  • OR equivalent experience
• Experience with modern large language model (LLM) serving technologies or distributed inference concepts.
• Strong problem-solving ability and willingness to work in a collaborative team environment.

Other Requirements: 

Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: 

• Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Preferred Qualifications:

• Exposure to deep learning frameworks such as PyTorch, TensorFlow, or ONNX.
• Exposure to GPU programming (CUDA) or other accelerator software stacks.
• Familiarity with performance analysis, debugging, or profiling tools.
• Coursework, internship, or project experience related to AI infrastructure, compilers, or distributed systems.

Software Engineering IC3 - The typical base pay range for this role across the U.S. is USD $100,600 - $199,000 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $131,400 - $215,400 per year.

Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay

Microsoft posts positions for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

• Contribute to the design and development of components in the AI software stack for Microsoft’s 1P accelerators.
• Implement features and optimizations under the guidance of senior engineers.
• Collaborate with hardware, compiler, and framework teams to enable efficient execution of LLM workloads.
• Debug issues, analyze performance gaps, and propose targeted improvements.
• Participate in design discussions, code reviews, and maintain high-quality software practices.
• Learn and stay up to date on emerging AI frameworks and accelerator technologies.