Senior Machine Learning Engineer, Evaluation & Reliability

Bask Health

16m ago • Remote

Senior

Remote

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Quick Facts

Senior hands-on ML engineering role focused on evaluation infrastructure and production reliability for AI/LLM applications in healthcare-related workflows.

Description

Design, build, and maintain systems that measure and improve the quality of AI-powered applications. Create evaluation pipelines, automated regression tests, and production monitoring to assess accuracy, relevance, grounding, consistency, and safety—then investigate failures and drive improvements in prompts, retrieval, model configuration, and app behavior.

Responsibilities

  • Design and build AI/ML evaluation infrastructure
  • Develop automated testing systems for LLM apps and intelligent workflows
  • Establish measurable quality standards for production AI features
  • Create evaluation pipelines for accuracy, relevance, completeness, consistency, and reliability
  • Build and maintain representative datasets for testing
  • Develop regression testing systems to detect quality changes after updates to models, prompts, or application logic
  • Design evaluation methods for retrieval-augmented generation and knowledge-grounded AI systems
  • Measure retrieval quality, grounding accuracy, and unsupported model outputs
  • Build tools for identifying hallucinations, incorrect reasoning, and unreliable responses
  • Develop testing approaches for AI agents, tool calling, and multi-step workflows
  • Evaluate whether AI systems follow application instructions and respect workflow boundaries
  • Implement automated checks for structured outputs, API interactions, and tool execution
  • Build monitoring systems that track AI performance in production
  • Develop dashboards and reporting tools for model quality, latency, errors, and operational reliability
  • Investigate production AI failures and identify their underlying technical causes
  • Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations
  • Establish testing procedures for model upgrades and provider changes
  • Develop benchmarking tools to compare models based on quality, performance, and cost
  • Implement evaluation processes for sensitive healthcare-related AI workflows
  • Work with product and clinical stakeholders to define appropriate acceptance criteria and escalation requirements
  • Support human review and feedback collection for AI outputs requiring additional validation
  • Design reproducible experiments for evaluating AI system improvements
  • Build reusable evaluation libraries, services, and internal engineering tools
  • Integrate AI evaluation checks into development and deployment workflows
  • Improve observability through structured logging, tracing, and production diagnostics
  • Mentor engineers on AI testing methodologies and reliability practices
  • Contribute to architecture discussions involving AI infrastructure, testing, and monitoring
  • Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance

Requirements

  • 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline
  • Demonstrated experience developing and operating ML/AI systems in production
  • Strong programming skills in Python and experience building production software services
  • Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications
  • Strong understanding of machine learning evaluation methodologies and performance measurement
  • Experience developing reproducible experiments and benchmark datasets
  • Familiarity with large language models and modern LLM application architectures
  • Understanding of prompt engineering, model inference, and AI application integration patterns
  • Experience evaluating model outputs for quality, accuracy, and reliability
  • Familiarity with retrieval-augmented generation and retrieval evaluation methods
  • Experience working with structured datasets, data pipelines, and analytical workflows
  • Strong understanding of software testing, regression testing, and continuous integration
  • Experience building or maintaining backend services and APIs
  • Familiarity with cloud infrastructure and production deployment environments
  • Experience with monitoring, logging, tracing, and production troubleshooting
  • Understanding of model latency, throughput, and operational performance metrics
  • Ability to investigate complex system failures and identify root causes
  • Experience designing metrics and reporting systems for technical performance
  • Familiarity with version control, code review, and collaborative software development
  • Understanding of data quality and its impact on model evaluation
  • Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost
  • Strong analytical skills and attention to detail
  • Experience collaborating with engineers, product managers, and technical stakeholders
  • Ability to independently own complex engineering initiatives
  • Strong communication skills and the ability to explain evaluation results clearly

Nice to Have

  • Experience building evaluation frameworks specifically for LLM-powered applications
  • Familiarity with tools such as LangSmith, DeepEval, Ragas, or comparable evaluation platforms
  • Experience implementing LLM-as-a-judge evaluation methods and validating their limitations
  • Experience building automated hallucination detection or groundedness evaluation systems
  • Familiarity with agent evaluation and multi-step workflow testing
  • Experience evaluating tool-calling accuracy and structured model outputs
  • Familiarity with retrieval metrics such as precision, recall, and ranking quality
  • Familiarity with vector databases, embeddings, and semantic search
  • Experience with MLflow, Weights & Biases, or similar experiment tracking tools
  • Experience developing machine learning observability infrastructure
  • Familiarity with model drift detection and data quality monitoring
  • Experience integrating evaluation checks into CI/CD pipelines
  • Familiarity with distributed processing or large-scale evaluation workloads
  • Familiarity with AWS and cloud-based machine learning infrastructure
  • Experience building internal developer tools for AI engineering teams
  • Experience evaluating systems built with OpenAI, Anthropic, or comparable model providers
  • Familiarity with healthcare, telehealth, or other sensitive-data applications
  • Understanding of HIPAA and healthcare data privacy considerations
  • Experience designing human-in-the-loop evaluation or review systems
  • Experience using AI-assisted development tools such as Cursor or Claude Code

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report