Senior Python Test Automation Engineer (GenAI)

Quick Facts

• Role: Senior Python Test Automation Engineer (GenAI) for AI/GenAI-powered clinical applications
• Focus: Automated testing frameworks, non-deterministic AI output validation, and GxP IQ/OQ/PQ qualification evidence

Description

You will build and maintain automated test frameworks for AI-powered clinical applications and validate non-deterministic AI/GenAI outputs. The role includes automated GxP qualification (IQ/OQ/PQ), continuous integration of tests into CI/CD quality gates, and managing AI-specific test data and model drift monitoring. You will also produce compliance-ready test evidence and work with Data Scientists and Backend Developers on acceptance criteria.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest
• Design and execute test strategies for non-deterministic AI/GenAI outputs (accuracy, RAG relevance, hallucination checks, prompt regression)
• Integrate automated tests into CI/CD pipelines with quality gates (block-on-failure, coverage thresholds)
• Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages
• Manage AI-specific test data (edge cases, adversarial inputs, golden datasets)
• Monitor model drift, regression, and performance degradation over time
• Produce test strategy documents, test plans, and compliance-ready test evidence
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features

Requirements

• 5+ years of hands-on QA/test automation experience
• Ability to build and maintain automated test suites from scratch using Playwright, Selenium, Cypress, and pytest
• API and backend test automation skills (REST APIs, microservices, serverless) with Postman/Newman and pytest + requests
• Proficiency in Python as the primary automation language
• Knowledge of CI/CD test integration with quality gates in tools such as Jenkins and GitHub Actions
• Background in testing AI/GenAI applications for non-deterministic AI/LLM outputs (accuracy, RAG relevance, hallucination, prompt regression)
• Familiarity with AI/ML model testing (accuracy, bias, performance) using RAGAS, LLM-as-judge, and golden datasets
• English proficiency at B2 level or higher

Benefits

• Stable pay and Employee Stock Purchase Plan with a 15% discount
• Health insurance, Multisport, shopping vouchers
• Referral bonuses up to $2,000
• Hybrid working model with remote opportunity within Poland
• Chance to work abroad up to 60 days annually and business-driven relocation opportunities
• Career development, mentoring, soft skills and well-being programs, and certifications (Anthropic, Gemini, GCP, Azure, AWS)
• English classes; corporate, social and well-being events; office amenities (snacks, coffee, entertainment/recreation zones)