
Software Engineer, AI Quality & Evaluation
Bask Health
16m ago • Remote
Mid
Remote
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Quick Facts
Role: Software Engineer, AI Quality & Evaluation
Focus: Build infrastructure for AI testing, evaluation, and quality assurance across the software lifecycle
Description
You will design, build, and maintain backend systems that help engineering teams test, evaluate, and improve AI-powered applications during development and in production. The role centers on reusable tooling—APIs, testing frameworks, dataset management, automated regression pipelines, and dashboards—to make AI evaluation more efficient, consistent, and accessible. Because this involves healthcare-related environments, the systems must handle sensitive data appropriately with proper access controls and privacy considerations.
Responsibilities
Build automated AI evaluation execution and management services
Develop APIs for submitting evaluation jobs and retrieving results
Create tooling for dataset/test case management and expected outcomes
Implement prompt/model/application comparison workflows and AI regression testing
Integrate evaluation checks into CI/CD pipelines
Record, store, and organize evaluation results; provide structured comparison and inspection tools
Build interfaces/dashboards to review evaluation outcomes and investigate failures
Support deterministic checks and model-based evaluations, including tool-calling and structured output testing
Create infrastructure for retrieval-augmented generation (RAG) evaluation
Implement asynchronous job processing (scheduling, retries, timeouts, error handling) for large evaluation workloads
Add monitoring and logging for evaluation infrastructure; optimize speed, reliability, and costs
Translate testing requirements from AI/ML specialists into reusable engineering systems
Implement secure handling of evaluation data with tenant boundary isolation
Troubleshoot evaluation infrastructure failures and improve reliability over time
Improve internal documentation and tooling for AI testing
Participate in code reviews and architecture discussions; own projects end-to-end
Benefits
The offer emphasizes developer experience and production reliability by providing the opportunity to build dependable evaluation infrastructure that other teams rely on.
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
Software Engineer, AI Platform
Bask Health
Senior
Technology
Remote
N/A
16m ago

Software Engineer
Atlassian Inc.
Senior
Technology
Seattle, WA · Remote
$150k - $196k/yr
2 days ago
AI Software Evaluation Engineer (Siri AI Quality)
Apple
Senior
Technology
Cupertino, CA · On-site
$150k - $278k/yr
9 days ago

Software Engineer
Atlassian Inc.
Mid
Technology
Seattle, WA · On-site
$150k - $196k/yr
6 days ago

Lead Software Engineer
JPMorgan Chase & Co.
Senior
Technology
Hoboken, NJ · On-site
N/A
9 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.
AI Software Engineer
Hilton
Mid
Technology
Remote
$78k - $110k/yr
8 days ago

Senior Backend Software Engineer
Mirage
Senior
Technology
New York, NY · On-site
$175k - $300k/yr
6 days ago

Platform Engineer (APIs & Workflow Automation)
Bask Health
Mid
Technology
Remote
N/A
16m ago

Backend Software Engineer
Horizontal Talent
Mid
Technology
Brooklyn Park, MN · On-site
N/A
4h ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights