Principal Hardware Engineer

Quick Facts

• Role: Principal Hardware Engineer

Description

Lead and implement end-to-end agentic/automated debug solutions for at-scale datacenter fleet health systems. Collaborate across hardware, firmware, and software teams to drive end-to-end root-cause analysis, define the rules and evidence agents must follow, and convert fixes into reusable lessons, runbooks, tests, or automated checks. Own agent quality and reliability, including safe execution during incident response and recovery.

Responsibilities

• Lead development and implementation of agentic/automated debug solutions for datacenter fleet health systems.
• Drive end-to-end root-cause analysis across SW, blade components, rack, and fleet; define agent operating rules; manage handoffs and work closure.
• Lead agentic defect-reduction workflows for novel faults crossing SW and multiple hardware layers; ensure every fix becomes reusable learning.
• Resolve complex problems by setting goals, specs, and evidence bars to reduce late-detected errors.
• Develop at-scale debug methodologies and build automation to improve debug efficiency for mission-critical services.
• Ensure fleet health agent quality and reliability; run persona swarms safely and set runtime limits for incident response and recovery.

Requirements

• Master’s degree in Electrical Engineering, Computer Engineering, Mechanical Engineering, or related + 7+ years technical engineering experience, or Bachelor’s degree + 8+ years technical engineering experience, or equivalent experience.
• Ability to meet security screening requirements; pass Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
• Preferred: 10+ years technical leadership owning measurable reliability/quality/capacity/recovery outcomes at production scale using reliability engineering methods.
• Preferred: Deep expertise in one server/datacenter infrastructure layer (CPU/SoC architecture, memory subsystem, storage, networking, virtualization, firmware, cloud-native OS, or fleet telemetry/data systems) and understanding of failure propagation.
• Preferred: Proven ability to lead deep dives for hard failures without a known playbook (testing hypotheses, preserving evidence, root cause across layers, permanent fix/runbook/test/automated check).
• Preferred: Proven ability to set intent, limits, and evidence bars for automated/AI-driven systems, including rollback, an audit trail, and named accountable ownership.
• Preferred: Experience with AI agents in production operations, including governing agent actions and curating memory.

Benefits

• Typical base pay range (U.S.): USD $142,800–$274,800 per year (different location-specific ranges for the San Francisco Bay area and New York City metro).
• May be eligible for benefits and other compensation depending on role.