Senior Automation Tester (Python, AI/GenAI)

Quick Facts

• Role: Senior Automation Tester (Python, AI/GenAI)
• Focus: Automated QA for AI/GenAI-powered clinical applications and GxP qualification

Description

Build and maintain automated test frameworks for AI-powered clinical applications. Validate non-deterministic AI outputs—accuracy, RAG relevance, hallucination checks, and prompt regression—and ensure GxP compliance through automated IQ/OQ/PQ qualification evidence. Integrate tests into CI/CD pipelines with quality gates, manage AI test data (edge cases, adversarial inputs, golden datasets), and monitor model drift and performance degradation over time.

Responsibilities

• Build automated test frameworks from scratch using Playwright, Selenium, and pytest
• Design and execute test strategies for non-deterministic AI/GenAI outputs (accuracy, RAG relevance, hallucination checks, prompt regression)
• Integrate automated tests into CI/CD pipelines with quality gates (block-on-failure, coverage thresholds)
• Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages
• Manage AI-specific test data, including edge cases, adversarial inputs, and golden datasets
• Monitor model drift, regression, and performance degradation over time
• Produce test strategy documents, test plans, and compliance-ready test evidence
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features

Requirements

• 5+ years of hands-on QA/test automation experience
• Ability to build and maintain automated test suites from scratch using Playwright, Selenium (Cypress also relevant) and pytest
• API and backend test automation skills for REST APIs, microservices, and serverless architectures (Postman/Newman and pytest + requests)
• Python proficiency as the primary automation language, including writing test harnesses, fixtures, and utilities independently
• CI/CD test integration experience (e.g., Jenkins, GitHub Actions) with quality gates
• Background in testing AI/GenAI applications and validating non-deterministic AI/LLM outputs (accuracy, RAG relevance, hallucination, prompt regression)
• Familiarity with AI/ML model testing using RAGAS, LLM-as-judge, and golden datasets
• English proficiency at B2 level or higher

Benefits

• Stable pay
• Employee Stock Purchase Plan with a 15% discount
• Benefits package: health insurance, multisport, shopping vouchers
• Referral bonuses up to $2,000
• Hybrid by design, with opportunity to work remotely within Poland
• Chance to work abroad for up to 60 days annually
• Business-driven relocation opportunities
• Career development programs, mentoring, and soft skills & well-being programs
• Certification support (Anthropic, Gemini, GCP, Azure, AWS)
• English classes
• Corporate, social and well-being events
• Office perks: entertainment/relaxation zones, table tennis and football, free snacks and coffee