Senior QA / ML Tester

Responsibilities

• Design and implement a functional test suite for the AWS AgentCore Evaluation API using pytest, covering known-good/known-bad session-pair validation, multi-evaluator execution correctness, and edge-case handling.
• Develop and maintain integration tests for on-demand evaluation mode, integrated into the CI/CD pipeline with automated execution on each build.
• Validate online-mode sampling accuracy; design test scenarios, define acceptance criteria, and report deviations with reproducible evidence.
• Conduct and document a feasibility assessment for non-AgentCore runtime evaluation: analyze alternative runtimes, define an evaluation methodology, and deliver a structured findings report.
• Test OpenTelemetry trace-based evaluation inputs and validate ADOT trace ingestion, trace-structure correctness, and evaluator-input integrity.
• Collaborate with platform engineers to clarify evaluation contracts, reproduce defects, and align on quality gates.
• Maintain test documentation, including test plans, test reports, defect logs, and evaluation feasibility artifacts in Confluence and Jira.
• Participate in Agile ceremonies, including sprint planning, daily standups, demos, and retrospectives.
• Contribute to engineering practices such as code review of test scripts, CI/CD pipeline integration, and test-coverage reporting.

Requirements

• 5+ years of production experience in QA automation or ML/AI system testing.
• Proven experience testing AI/LLM systems, including evaluation pipelines, model outputs, or agent-behavior validation.
• Proficiency in Python test automation, including pytest, fixtures, parametrization, and mocking with unittest.mock and moto.
• Knowledge of the AWS AgentCore Evaluation API, including on-demand and online evaluation modes.
• Familiarity with OpenTelemetry/ADOT for trace-based evaluation-input testing and trace-structure validation.
• REST API testing skills, including request/response validation and authentication with SigV4 and bearer tokens.
• CI/CD integration experience using GitHub Actions, Jenkins, or equivalent, including test-pipeline configuration.
• Test data management experience, including known-good/known-bad session-pair design and synthetic-trace generation.
• Expertise in functional and integration test design for AI/ML evaluation pipelines.
• Defect lifecycle management competency, including Jira, reproducible bug reports, and root-cause analysis.
• Understanding of LLM/agent evaluation concepts, including correctness scoring, sampling strategies, and evaluator chaining.
• Ability to work independently after onboarding, manage tasks, report status, and proactively escalate blockers.
• Strong analytical skills to define test scenarios from ambiguous or evolving specifications.
• English at B2+ level, written and verbal, for daily collaboration with distributed teams.

Nice to have

• Hands-on experience with the AWS AgentCore Evaluation API or AWS Bedrock testing.
• Experience testing OpenTelemetry or distributed-tracing pipelines.
• Familiarity with multi-evaluator execution patterns and correctness-validation strategies.
• Experience writing feasibility assessments or technical reports for stakeholders.
• Knowledge of agentic AI frameworks, including LangGraph and Strands Agents, sufficient to understand evaluation contracts.
• ISTQB CT-AI certification or an equivalent AI-testing qualification.
• Experience with prompt engineering and AI-assisted test design.

Benefits

• Hybrid work model and the opportunity to work remotely within Poland.
• Opportunity to work abroad for up to 60 days annually.
• Business-driven relocation opportunities.
• Career development programs, thought leadership, mentoring, soft-skills, and well-being programs.
• Certification opportunities in Anthropic, Gemini, GCP, Azure, and AWS, plus English classes.
• Stable pay and participation in an Employee Stock Purchase Plan with a 15% discount.
• Benefits package including health insurance, multisport, and shopping vouchers.
• Referral bonuses of up to $2,000.
• Offices with entertainment and relaxation zones, table tennis, football, free snacks, coffee, and more.
• Corporate, social, and well-being events.

Benefits listed above are available to employees only.

Contractor engagement is available; B2B cooperation terms are agreed individually.

Only selected candidates will be contacted.