Platform Engineer (AWS AgentCore Evaluation)

Quick Facts

• Role: Platform Engineer focused on AWS AgentCore evaluation and quality monitoring for AI agent systems
• Team size: Medium (10–20 people)
• Collaboration: Platform engineering, DevOps, AI engineering, cloud architecture

Description

You will build and maintain a scalable evaluation and monitoring platform for AI agents and intelligent applications. The platform provides automated deployment gates, online quality assessment, operational visibility, and regression detection. You will integrate automated evaluation frameworks into deployment workflows and develop monitoring and semantic discovery capabilities.

Responsibilities

• Design and implement semantic discovery using vector embeddings (OpenSearch or pgvector)
• Develop and maintain Python APIs for secure credential retrieval and secrets access
• Integrate and manage HashiCorp Vault and AWS Secrets Manager within platform services
• Build and optimize semantic indexing pipelines and search experiences
• Design and maintain semantic domain taxonomies to improve information discovery
• Implement monitoring, logging, and operational best practices for platform services
• Support MCP server capabilities and enterprise system integrations
• Participate in architecture discussions, technical design reviews, and platform roadmap initiatives
• Ensure secure access patterns and compliance with security, governance, and credential management standards

Requirements

• 3+ years of experience in platform engineering or backend engineering
• Hands-on experience with secrets management solutions (HashiCorp Vault or AWS Secrets Manager)
• Experience implementing vector search, semantic search, or embedding-based search solutions
• Semantic discovery implementation using vector embeddings with OpenSearch or pgvector
• HashiCorp Vault integration for credential and secrets management
• AWS Secrets Manager integration (alternative)
• Development of Python APIs for credential retrieval and access management
• MCP server capabilities and semantic domain taxonomy design

Benefits

• Up to 26 business vacation days per year and 10 illness/special paid days off
• Health and life insurance (Luxmed)
• MyBenefit platform with Multisport option
• Internal psychological support service
• English language classes from the first working day
• Work from office, home, or hybrid
• Access to external learning platforms (O’Reilly, LinkedIn Learning, Udemy) and internal training catalog
• Tech Skills Mentoring Program
• Opportunities to develop as a public speaker, mentor, or technical interviewer
• Fully paid idle (bench) when not assigned to a project
• Certification reimbursement (AWS, GCP, Microsoft, etc.)