Principal Software Engineer, Observability

Quick Facts

• Role: Hands-on Principal Software Engineer in Observability
• Focus: AI-driven telemetry, detection, diagnosis, and remediation

Description

The Observability team builds the telemetry, analytics, and diagnostic platform that helps teams understand the health of workloads and services, detect problems early, and resolve them quickly. In this role, you will architect and build distributed infrastructure for ingesting, storing, and querying logs, metrics, traces, and events at very high volume, and ship observability products to customers. You will also develop AI-driven agentic workflows that correlate signals, explain likely root causes, and recommend or execute remediation steps.

Responsibilities

• Architect and build distributed infrastructure
• Design scalable pipelines and services for telemetry ingestion, storage, and querying with predictable cost and low query latency
• Ship customer-facing observability capabilities including telemetry collection, alerting, dashboards, and diagnostics
• Build AI-driven detection and diagnosis with rigorous evaluation and high-quality telemetry
• Make AI integral to software design, implementation, testing, and operations; establish AI reliability, security, and evaluation standards
• Translate ambiguous needs into practical designs, milestones, and open-standards interfaces such as OpenTelemetry
• Write and review production code, diagnose complex failures, own operational outcomes, and mentor engineers

Requirements

• 12+ years of software engineering experience with technical leadership across multiple teams
• Deep distributed-systems experience (concurrency, consistency, caching, failure recovery, storage and query efficiency, performance)
• Strong CS fundamentals and hands-on proficiency in Java, Go, C++, or a comparable backend language
• Cloud experience with AWS, Azure, or Google Cloud
• AI-first engineering practice and demonstrated ability to improve execution, problem-solving, and quality using AI
• Experience shipping products to external customers, including adoption and reliability after launch
• Strong written communication, low-ego collaboration, ownership, and experience mentoring senior engineers
• Degree in Computer Science or related field, or equivalent practical experience

Benefits

• Build a platform where AI agents and engineers detect, diagnose, and resolve issues together
• Opportunity to lead architecture through adoption and raise the engineering bar across teams
• Work on systems that improve customer and developer workflows at large scale