Senior Python Automation Test Engineer (AI/GenAI)

Quick Facts

• Role: Senior Python Automation Test Engineer (AI/GenAI)

Description

Build and maintain automated test frameworks for AI/GenAI-powered clinical applications. Validate non-deterministic AI outputs (accuracy, RAG relevance, hallucination checks, and prompt regression), and ensure GxP compliance through automated IQ/OQ/PQ qualification. Integrate tests into CI/CD pipelines with quality gates and provide compliance-ready test evidence.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest

• Design and execute test strategies for non-deterministic AI/GenAI outputs

• Integrate automated tests into CI/CD pipelines with quality gates (block-on-failure, coverage thresholds)

• Execute GxP-compliant IQ/OQ/PQ scripts and produce qualification evidence packages

• Manage AI-specific test data (edge cases, adversarial inputs, golden datasets)

• Monitor model drift, regression, and performance degradation over time

• Create test strategy documents, test plans, and compliance-ready test evidence

• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria

Requirements

• 5+ years of hands-on QA/test automation experience

• Ability to build and maintain automated test suites from scratch using Playwright, Selenium, Cypress, and pytest

• API and backend test automation experience (REST APIs, microservices, serverless) using Postman/Newman and pytest + requests

• Python as the primary automation language (write harnesses, fixtures, and utilities independently)

• CI/CD test integration experience with Jenkins and GitHub Actions and quality gates

• Experience testing AI/GenAI applications by validating non-deterministic AI/LLM outputs (accuracy, RAG relevance, hallucinations, prompt regression)

• Knowledge of AI/ML model testing (accuracy, bias, performance) using RAGAS, LLM-as-judge, and golden datasets

• English proficiency at B2 level or higher

Benefits

• Stable pay

• Employee Stock Purchase Plan with a 15% discount

• Benefits package: health insurance, multisport, shopping vouchers

• Referral bonuses up to $2,000

• Hybrid-by-design with potential remote work within Poland

• Chance to work abroad for up to 60 days annually

• Business-driven relocation opportunities

• Career development programs, mentoring, soft skills, well-being programs

• Certification support (Anthropic, Gemini, GCP, Azure, AWS)

• English classes

• Corporate, social, and well-being events

• Offices with entertainment and relaxation zones, table tennis and football, free snacks and coffee