Software Engineer, AI Quality & Evaluation

Bask Health

16m ago • Remote

Mid

Remote

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Quick Facts

  • Role: Software Engineer, AI Quality & Evaluation

  • Focus: Build infrastructure for AI testing, evaluation, and quality assurance across the software lifecycle

Description

You will design, build, and maintain backend systems that help engineering teams test, evaluate, and improve AI-powered applications during development and in production. The role centers on reusable tooling—APIs, testing frameworks, dataset management, automated regression pipelines, and dashboards—to make AI evaluation more efficient, consistent, and accessible. Because this involves healthcare-related environments, the systems must handle sensitive data appropriately with proper access controls and privacy considerations.

Responsibilities

  • Build automated AI evaluation execution and management services

  • Develop APIs for submitting evaluation jobs and retrieving results

  • Create tooling for dataset/test case management and expected outcomes

  • Implement prompt/model/application comparison workflows and AI regression testing

  • Integrate evaluation checks into CI/CD pipelines

  • Record, store, and organize evaluation results; provide structured comparison and inspection tools

  • Build interfaces/dashboards to review evaluation outcomes and investigate failures

  • Support deterministic checks and model-based evaluations, including tool-calling and structured output testing

  • Create infrastructure for retrieval-augmented generation (RAG) evaluation

  • Implement asynchronous job processing (scheduling, retries, timeouts, error handling) for large evaluation workloads

  • Add monitoring and logging for evaluation infrastructure; optimize speed, reliability, and costs

  • Translate testing requirements from AI/ML specialists into reusable engineering systems

  • Implement secure handling of evaluation data with tenant boundary isolation

  • Troubleshoot evaluation infrastructure failures and improve reliability over time

  • Improve internal documentation and tooling for AI testing

  • Participate in code reviews and architecture discussions; own projects end-to-end

Benefits

The offer emphasizes developer experience and production reliability by providing the opportunity to build dependable evaluation infrastructure that other teams rely on.

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report