Principal AI Software Engineer

Quick Facts

• Role: Principal AI Software Engineer (AI + systems software)

• Focus: LLM inference performance and memory-TCO reduction through hardware/software co-design

Description

Lead end-to-end system software prototyping to build proof-of-concepts for evaluating hardware/software co-designed capabilities that reduce memory TCO (e.g., tiering/pooling and overcommit) for Azure compute deployments. Characterize and optimize KV cache intensive LLM inference workloads across GPU HBM, host DRAM, CXL memory expansion/pooling, SSD, and emerging memory tiers, then quantify latency, throughput, token efficiency, and utilization. Build evaluation frameworks and performance models, and analyze end-to-end data movement across GPU, CPU, storage, and networking to identify optimization opportunities.

Responsibilities

• Prototype full system software to develop proof-of-concepts for memory TCO reduction

• Perform workload characterization and correlation to identify system optimization opportunities

• Collaborate with workload experts to engineer TCO-optimized solutions for Azure compute fleets

• Influence hardware architecture and industry alignment for multi-year horizons using data-driven recommendations

• Characterize and optimize LLM inference, with focus on KV cache capacity, placement, migration, and utilization

• Develop proof-of-concepts and evaluation frameworks for memory-tiering architectures for AI inference (e.g., CXL pooled memory, memory expansion, SSD-backed cache tiers)

• Run studies for agentic, multi-turn, coding, reasoning, and long-context workloads to measure memory consumption and system utilization

• Analyze end-to-end data movement across GPU/CPU/storage/network and optimize GPU Direct Storage (GDS) and GPU Direct RDMA (GDR)

• Build software prototypes, framework extensions, and instrumentation for KV cache offload/prefetch/migration/compression/deduplication/overcommit

• Develop performance models and simulation frameworks to predict impact at large deployment scale

Requirements

• Bachelor’s degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in C/C++/C#/Java/JavaScript/Python (or equivalent)

• Ability to meet Microsoft, customer, and/or government security screening requirements, including passing the Microsoft Cloud Background Check and renewing every two years

Benefits

• Competitive U.S. base pay range (typical): USD $142,800–$274,800 per year; higher ranges for specific locations (e.g., Bay Area, NYC metro)

• Additional benefits and compensation may be available (per Microsoft careers pay information)