Software Engineer II, Autonomy Evaluation & Metrics Platform

Quick Facts

• Role: Software Engineer II (Autonomy Evaluation & Metrics Platform)

• Focus: Building a metrics platform for autonomy model evaluation, reproducible measurements, and trusted release scorecards

Description

You will build backend services, Python libraries, and data pipelines that convert autonomy model outputs into reproducible metrics and comparisons. You’ll own well-scoped components from implementation through deployment and operation, with support from senior engineers. You will help engineers understand model behavior, investigate regressions, and turn evaluation findings into better models and evaluation datasets.

Responsibilities

• Build and test services, APIs, and libraries for registering metric definitions, computing results, and retrieving them consistently across tools

• Implement scalable metric-processing pipelines with safe retries, incremental computation, and backfills; surface missing inputs, failed jobs, and incomplete results

• Capture code, metric, model, dataset, and scenario versions so reported results are traceable and reproducible

• Build tools for comparing model versions and exploring results by scenario, geography, vehicle platform, and operating conditions

• Partner with model developers to run evaluation experiments, inspect predictions, investigate failure patterns, and translate findings into targeted datasets and regression tests

• Integrate metrics into evaluation workflows, CI regression checks, and release scorecards; connect results to recordings and triage tools

• Improve automated tests, performance, observability, and deployment tooling; debug production issues and document supported components

• Collaborate across locations to deliver maintainable software through design and code reviews

Requirements

• 2+ years of software engineering experience (or equivalent) building and maintaining substantial software systems

• Strong Python skills; solid foundation in data structures, algorithms, software design, and debugging

• Practical SQL skills, including writing/refining queries and validating correctness and performance

• Experience building backend services, reusable libraries, or data-processing pipelines with automated tests, version control, and code review

• Ability to reason about APIs, data models, and component design; identify failure cases such as duplicate, delayed, or missing data

• Experience with cloud services, containers, and CI/CD; understanding of production deployment and monitoring

• Care about data correctness and basic statistical reasoning (sample sizes, rates, variability)

• Clear written communication and collaborative problem-solving

Benefits

• Medical/Rx, dental, and vision insurance coverage for full-time employees and eligible dependents; coverage effective on the first day, company covers most premiums