Senior Automation Test Engineer (Python, GenAI)

Quick Facts

• Role: Senior Automation Tester in Python for GenAI-powered clinical applications

Description

You will build and maintain automated test frameworks for AI/GenAI clinical applications, validating non-deterministic AI outputs and ensuring GxP compliance using automated IQ/OQ/PQ qualification. The role focuses on AI-specific testing (accuracy, RAG relevance, hallucination checks, and prompt regression) and requires integrating quality checks into CI/CD pipelines with compliance-ready evidence. You will collaborate with Data Scientists and Backend Developers across multiple use cases.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest

• Design and execute test strategies for non-deterministic AI/GenAI outputs (accuracy, RAG relevance, hallucination checks, prompt regression)

• Integrate automated tests into CI/CD pipelines with quality gates (block-on-failure, coverage thresholds)

• Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages

• Manage AI-specific test data (edge cases, adversarial inputs, golden datasets)

• Monitor model drift, regression, and performance degradation over time

• Produce test strategy documents, test plans, and compliance-ready test evidence

• Collaborate with Data Scientists and Backend Developers on testable acceptance criteria for AI features

Requirements

• 5+ years of hands-on QA/test automation experience

• Ability to build and maintain automated test suites from scratch (Playwright, Selenium, Cypress, pytest)

• API and backend test automation skills (REST APIs, microservices, serverless) using Postman/Newman and pytest + requests

• Python proficiency as the primary automation language (write harnesses, fixtures, utilities)

• CI/CD test integration experience with quality gates (Jenkins, GitHub Actions)

• AI/GenAI testing background for non-deterministic AI/LLM outputs (accuracy, RAG relevance, hallucination, prompt regression)

• Familiarity with AI/ML model testing (accuracy, bias, performance) using RAGAS, LLM-as-judge, golden datasets

• English proficiency at B2 level or higher

Benefits

• Stable pay and participation in an Employee Stock Purchase Plan with a 15% discount

• Benefits package (health insurance, Multisport, shopping vouchers)

• Referral bonuses up to $2,000

• Hybrid work model with opportunity to work remotely within Poland

• Option to work abroad up to 60 days annually

• Growth opportunities: career development programs, mentoring, soft skills and well-being programs

• Certifications (Anthropic, Gemini, GCP, Azure, AWS) and English classes

• Corporate social and well-being events; offices with entertainment and relaxation zones and free snacks/coffee

• Open to B2B contractors (terms agreed individually)