
Senior Machine Learning Engineer, Evaluation & Reliability
Bask Health
16m ago • Remote
Senior
Remote
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Quick Facts
Senior hands-on ML engineering role focused on evaluation infrastructure and production reliability for AI/LLM applications in healthcare-related workflows.
Description
Design, build, and maintain systems that measure and improve the quality of AI-powered applications. Create evaluation pipelines, automated regression tests, and production monitoring to assess accuracy, relevance, grounding, consistency, and safety—then investigate failures and drive improvements in prompts, retrieval, model configuration, and app behavior.
Responsibilities
- Design and build AI/ML evaluation infrastructure
- Develop automated testing systems for LLM apps and intelligent workflows
- Establish measurable quality standards for production AI features
- Create evaluation pipelines for accuracy, relevance, completeness, consistency, and reliability
- Build and maintain representative datasets for testing
- Develop regression testing systems to detect quality changes after updates to models, prompts, or application logic
- Design evaluation methods for retrieval-augmented generation and knowledge-grounded AI systems
- Measure retrieval quality, grounding accuracy, and unsupported model outputs
- Build tools for identifying hallucinations, incorrect reasoning, and unreliable responses
- Develop testing approaches for AI agents, tool calling, and multi-step workflows
- Evaluate whether AI systems follow application instructions and respect workflow boundaries
- Implement automated checks for structured outputs, API interactions, and tool execution
- Build monitoring systems that track AI performance in production
- Develop dashboards and reporting tools for model quality, latency, errors, and operational reliability
- Investigate production AI failures and identify their underlying technical causes
- Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations
- Establish testing procedures for model upgrades and provider changes
- Develop benchmarking tools to compare models based on quality, performance, and cost
- Implement evaluation processes for sensitive healthcare-related AI workflows
- Work with product and clinical stakeholders to define appropriate acceptance criteria and escalation requirements
- Support human review and feedback collection for AI outputs requiring additional validation
- Design reproducible experiments for evaluating AI system improvements
- Build reusable evaluation libraries, services, and internal engineering tools
- Integrate AI evaluation checks into development and deployment workflows
- Improve observability through structured logging, tracing, and production diagnostics
- Mentor engineers on AI testing methodologies and reliability practices
- Contribute to architecture discussions involving AI infrastructure, testing, and monitoring
- Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance
Requirements
- 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline
- Demonstrated experience developing and operating ML/AI systems in production
- Strong programming skills in Python and experience building production software services
- Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications
- Strong understanding of machine learning evaluation methodologies and performance measurement
- Experience developing reproducible experiments and benchmark datasets
- Familiarity with large language models and modern LLM application architectures
- Understanding of prompt engineering, model inference, and AI application integration patterns
- Experience evaluating model outputs for quality, accuracy, and reliability
- Familiarity with retrieval-augmented generation and retrieval evaluation methods
- Experience working with structured datasets, data pipelines, and analytical workflows
- Strong understanding of software testing, regression testing, and continuous integration
- Experience building or maintaining backend services and APIs
- Familiarity with cloud infrastructure and production deployment environments
- Experience with monitoring, logging, tracing, and production troubleshooting
- Understanding of model latency, throughput, and operational performance metrics
- Ability to investigate complex system failures and identify root causes
- Experience designing metrics and reporting systems for technical performance
- Familiarity with version control, code review, and collaborative software development
- Understanding of data quality and its impact on model evaluation
- Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost
- Strong analytical skills and attention to detail
- Experience collaborating with engineers, product managers, and technical stakeholders
- Ability to independently own complex engineering initiatives
- Strong communication skills and the ability to explain evaluation results clearly
Nice to Have
- Experience building evaluation frameworks specifically for LLM-powered applications
- Familiarity with tools such as LangSmith, DeepEval, Ragas, or comparable evaluation platforms
- Experience implementing LLM-as-a-judge evaluation methods and validating their limitations
- Experience building automated hallucination detection or groundedness evaluation systems
- Familiarity with agent evaluation and multi-step workflow testing
- Experience evaluating tool-calling accuracy and structured model outputs
- Familiarity with retrieval metrics such as precision, recall, and ranking quality
- Familiarity with vector databases, embeddings, and semantic search
- Experience with MLflow, Weights & Biases, or similar experiment tracking tools
- Experience developing machine learning observability infrastructure
- Familiarity with model drift detection and data quality monitoring
- Experience integrating evaluation checks into CI/CD pipelines
- Familiarity with distributed processing or large-scale evaluation workloads
- Familiarity with AWS and cloud-based machine learning infrastructure
- Experience building internal developer tools for AI engineering teams
- Experience evaluating systems built with OpenAI, Anthropic, or comparable model providers
- Familiarity with healthcare, telehealth, or other sensitive-data applications
- Understanding of HIPAA and healthcare data privacy considerations
- Experience designing human-in-the-loop evaluation or review systems
- Experience using AI-assisted development tools such as Cursor or Claude Code
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
ML Platform Engineer (Evaluation Systems)
Bask Health
Senior
Technology
Remote
N/A
1 days ago
AI/ML Engineer
BlueStone Solutions Group
Senior
Technology
Warren, MI, United States of America · On-site
N/A
10 days ago

Senior Backend Software Engineer (Platforms & Agentic/ML Systems)
Socure
Senior
Technology
New York, NY · On-site
$230k - $270k/yr
1 days ago

Software Engineer, AI Quality & Evaluation
Bask Health
Mid
Technology
Remote
N/A
16m ago
Senior LLM Inference Quality Engineer
Apple
Senior
Technology
Seattle, WA · On-site
$175k - $309k/yr
2 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

AI Infrastructure Software Engineer
Gray Swan AI
Mid
Technology
Pittsburgh, PA · On-site
$130k - $185k/yr
6 days ago
Machine Learning Engineer, AI Model Evaluation & Quality
Apple
Senior
Technology
California, CA · On-site
$142k - $263k/yr
3 days ago
Data Scientist – RAG & Document Intelligence
Craftware
Senior
Technology
Warsaw, Poland · Remote
354k zł - 416k zł/yr
12 days ago

Data Systems AI/ML Engineer
Oak Ridge National Laboratory
Mid
Technology
Oak Ridge, TN, United States of America · On-site
N/A
4 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights