Senior Automation Tester (Python) for GenAI Applications

Quick Facts

Role: Senior Automation Tester (Python) for GenAI applications

Focus: Automated testing frameworks for AI/GenAI-powered clinical applications (not manual QA)

Description

You will build and maintain automated test frameworks for AI/GenAI-powered clinical applications, validate non-deterministic AI outputs, and ensure GxP compliance through automated IQ/OQ/PQ qualification. The work spans multiple use cases across Safety, Regulatory, Clinical Operations, and platform tooling, including managing AI test data and monitoring drift/regression over time.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest
• Design and execute test strategies for non-deterministic AI/GenAI outputs (accuracy, RAG relevance, hallucination checks, prompt regression)
• Integrate automated tests into CI/CD pipelines with quality gates (block-on-failure and coverage thresholds)
• Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages
• Manage AI-specific test data including edge cases, adversarial inputs, and golden datasets
• Monitor model drift, regression, and performance degradation over time
• Create test strategy documents, test plans, and compliance-ready test evidence
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features

Requirements

• 5+ years of hands-on QA/test automation experience
• Experience building and maintaining automated test suites from scratch using Playwright, Selenium, and pytest (and related tools such as Cypress)
• Strong API and backend test automation skills, including REST APIs, microservices, and serverless architectures
• Proficiency in Python as the primary automation language
• Experience integrating automated tests into CI/CD pipelines (e.g., Jenkins, GitHub Actions) with quality gates
• Experience testing AI/GenAI applications and validating non-deterministic outputs (accuracy, RAG relevance, hallucinations, prompt regression)
• Familiarity with AI/ML model testing approaches such as RAGAS, LLM-as-judge, and golden datasets
• English proficiency at B2 level or higher

Benefits

• Stable pay and Employee Stock Purchase Plan with a 15% discount
• Benefits package including health insurance and multisport; shopping vouchers
• Referral bonuses up to $2,000
• Hybrid-by-design and opportunity to work remotely within Poland
• Chance to work abroad up to 60 days annually; business-driven relocation opportunities
• Career development, mentoring, thought leadership, and soft-skills/well-being programs
• Certification support (Anthropic, Gemini, GCP, Azure, AWS) and English classes
• Employee-only benefits and corporate social and well-being events