Senior Automation QA Engineer (Python, GenAI)

Quick Facts

Role: Senior Automation QA Engineer (Python, GenAI)

Focus: Automated testing for AI/GenAI-powered clinical applications and GxP compliance

Work model: Hybrid (remote within Poland possible)

Description

Build and maintain automated test frameworks for AI/GenAI-powered clinical applications. Validate non-deterministic AI outputs (accuracy, RAG relevance, hallucination checks, and prompt regression) and ensure GxP compliance through automated IQ/OQ/PQ qualification. Work across clinical and platform use cases and produce compliance-ready test evidence.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest
• Design and execute test strategies for non-deterministic AI/GenAI outputs (accuracy, RAG relevance, hallucination checks, prompt regression)
• Integrate automated tests into CI/CD pipelines with quality gates such as block-on-failure and coverage thresholds
• Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages
• Manage AI-specific test data, including edge cases, adversarial inputs, and golden datasets
• Monitor model drift, regression, and performance degradation over time
• Produce test strategy documents, test plans, and compliance-ready test evidence
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features

Requirements

• 5+ years of hands-on QA/test automation experience
• Experience building and maintaining automated test suites from scratch using Playwright, Selenium, Cypress, and pytest
• API and backend test automation experience (REST APIs, microservices, serverless) using Postman/Newman and pytest + requests
• Proficiency in Python to independently write harnesses, fixtures, and utilities
• Experience integrating tests into CI/CD pipelines (e.g., Jenkins, GitHub Actions) with quality gates
• Background testing AI/GenAI applications for non-deterministic AI/LLM outputs (accuracy, RAG relevance, hallucination, prompt regression)
• Familiarity with AI/ML model testing (accuracy, bias, performance) using RAGAS, LLM-as-judge, and golden datasets
• English proficiency at B2 level or higher

Nice to have

• Pharmaceutical/life sciences domain background and GxP awareness
• Experience with IQ/OQ/PQ qualification
• Test management tools: Jira/Xray, TestRail, Zephyr
• Performance/load testing: JMeter, Locust
• Security testing basics: OWASP
• AWS familiarity: Lambda, API Gateway, S3, CloudWatch

Benefits

• Stable pay and participation in the Employee Stock Purchase Plan with a 15% discount
• Benefits package (health insurance, multisport, shopping vouchers)
• Referral bonuses up to $2,000
• Hybrid by design and opportunity to work remotely within Poland
• Chance to work abroad for up to 60 days annually
• Career development programs, mentoring, soft skills and well-being programs
• Certifications (Anthropic, Gemini, GCP, Azure, AWS) and English classes
• Corporate, social and well-being events; office amenities (snacks, coffee, entertainment and relaxation zones)