
ML Platform Engineer (Evaluation Systems)
Bask Health
15m ago • Remote
Senior
Remote
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Quick Facts
Role: ML Platform Engineer (Evaluation Systems)
Focus: Infrastructure for reliable machine learning evaluation, benchmarking, and experiment repeatability at scale
Description
Build and maintain ML evaluation infrastructure that enables consistent and scalable assessment of machine learning and AI applications. You will design platform services, data pipelines, execution environments, orchestration, and monitoring to support reproducible experiments and accurate model comparisons before and after deployment. The role also requires secure, privacy-aware handling of sensitive healthcare-related evaluation data.
Responsibilities
Design, build, and maintain machine learning evaluation infrastructure
Develop platform services to execute and manage automated model evaluation workloads
Implement job orchestration, scheduling, and execution management for evaluation pipelines
Store, version, and manage evaluation datasets and evaluation artifacts/results
Build experiment tracking for model configurations (prompts/parameters) and evaluation outcomes
Create reproducible workflows to support consistent model comparisons
Implement automated benchmarking pipelines
Provide APIs/services for teams to submit, run, and monitor evaluation jobs
Compare evaluation results across model versions and experiments
Support evaluation workflows for large language models, predictive models, and retrieval systems
Build asynchronous processing for computationally intensive evaluations
Add monitoring, logging, and tracing for performance, failures, latency, and resource usage
Integrate evaluation infrastructure with model registries and ML development workflows
Implement automated quality checks for controlled releases
Add reliability mechanisms such as retries, timeouts, and failure recovery
Optimize execution performance, utilization, and operational costs; support distributed execution on cloud compute
Ensure secure access controls and tenant boundary isolation for datasets and results
Investigate incidents, improve reliability, and document operational standards
Own ML evaluation platform projects from design through deployment and continuous improvement
Requirements
3+ years of professional experience in software engineering, machine learning engineering, platform engineering, or a closely related field
Production experience building and maintaining backend services or infrastructure
Strong programming skills in Python, TypeScript, or comparable languages
Familiarity with machine learning development and evaluation workflows
Experience developing APIs and backend service integrations
Understanding of distributed systems and asynchronous processing
Experience working with cloud-based infrastructure and production deployments
Familiarity with automated testing and continuous integration practices
Understanding of machine learning model versioning and experiment reproducibility
Experience working with relational databases and structured datasets
Familiarity with data pipelines and batch processing workflows
Experience designing reliable background job systems
Understanding of infrastructure scalability, performance, and fault tolerance
Familiarity with logging, monitoring, and production observability
Experience implementing retries, timeouts, and error recovery
Understanding of authentication, authorization, and secure service communication
Familiarity with configuration management and deployment automation
Ability to write maintainable, well-tested production code
Ability to troubleshoot complex backend/infrastructure issues
Understanding of software release processes and operational reliability
Experience using Git and collaborative engineering workflows
Ability to translate ML requirements into scalable infrastructure
Strong analytical thinking and problem-solving skills
Ability to collaborate with ML engineers and applied scientists
Ability to independently own projects from design through deployment
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
AI Infrastructure Software Engineer
Gray Swan AI
Mid
Technology
Pittsburgh, PA · On-site
$130K - $185K/yr
5 days ago
Machine Learning Engineer
Apple
Mid
Technology
Cupertino, CA · On-site
$129K - $195K/yr
2 days ago
Senior Machine Learning Platform Engineer (Evaluation)
Apple
Senior
Technology
Seattle, WA · On-site
$175K - $309K/yr
8 days ago

Staff Machine Learning Engineer (Model Platform)
Wonder Group, Inc.
Senior
Technology
New York, NY · On-site
$209K - $217K/yr
5 days ago

ML/Data Engineer
Andersen
Senior
Technology
Warsaw, Poland · Remote
12K zł - 23K zł/yr
18h ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Lead Software Engineer
JPMorgan Chase & Co.
Senior
Technology
Hoboken, NJ · On-site
N/A
8 days ago
AI/ML Engineer
BlueStone Solutions Group
Senior
Technology
Warren, MI, United States of America · On-site
N/A
9 days ago
Machine Learning Platform Software Engineer
Cursor
Senior
Technology
Palo Alto, CA · On-site
$140K - $225K/yr
5 days ago
ML Engineer
Remodevs
Senior
Technology
Warsaw, Poland · Remote
N/A
9 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights
