Principal Solution & Production Reliability Architect

Quick Facts

• Role: Principal Solution & Production Reliability Architect
• Experience: 18+ years
• Location/Work model: Foster City, California — Onsite
• Work authorization: Any Visa

Description

Lead the design, development, and deployment of platform backend services and scheduled cron jobs, with a focus on production reliability supported by traces and metrics. Build event-driven systems using Kafka, and deliver safe workload rollouts on Kubernetes using feature-flag-based canary cohorts and service mesh traffic management. Own DevOps reliability practices including Harness-based continuous verification, Honeycomb and Prometheus/Grafana observability, automated rollback and deployment verification, and robust testing and synthetic test data with PII masking.

Requirements

• 18+ years of experience
• Experience designing, developing, and deploying platform backend services and cron jobs with tracing and metrics
• Experience in event-driven application design using Kafka
• Experience deploying workloads in Kubernetes using feature-flag-based canary cohorts and managing traffic with service mesh (North-South and East-West)
• Experience implementing DevOps with Harness, including continuous verification and tracing/metrics with Honeycomb and Prometheus/Grafana
• Experience implementing automated rollback, deployment verification, and service dependency graph identification
• Strong understanding of AWS EKS and related services
• Good understanding of load, stress, endurance, and automation testing using Kubernetes and Playwright
• Experience generating synthetic test cases and test data, including PII masking
• Hands-on experience in Python and Kotlin (Java)

Hard Skills & Technologies

• Python
• Kotlin
• Java
• Kafka
• Kubernetes
• Feature flag
• Canary
• Service mesh
• Harness
• Continuous verification
• Honeycomb
• Prometheus
• Grafana
• AWS EKS
• Load testing
• Stress testing
• Endurance testing
• Playwright
• Synthetic testing
• Test data generation
• PII masking
• Cron jobs
• Tracing
• Metrics
• Automated rollback
• Deployment verification
• Dependency graph