Senior Automation QA Engineer (Python, GenAI)

Quick Facts

Role: Senior Automation QA Engineer (Python, GenAI)
Focus: Automated QA for AI/GenAI-powered clinical applications and GxP compliance

Description

Build and maintain automated test frameworks for AI/GenAI-powered clinical applications. Validate non-deterministic AI outputs (accuracy, RAG relevance, hallucination checks, and prompt regression) and integrate tests into CI/CD with quality gates. Ensure GxP compliance by executing automated IQ/OQ/PQ qualification scripts and generating qualification evidence packages.

Responsibilities

• Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest.
• Design and execute test strategies for non-deterministic AI/GenAI outputs, including accuracy, RAG relevance, hallucination checks, and prompt regression.
• Integrate automated tests into CI/CD pipelines with quality gates such as block-on-failure and coverage thresholds.
• Execute GxP-compliant IQ/OQ/PQ scripts and generate compliance-ready qualification evidence packages.
• Manage AI-specific test data, including edge cases, adversarial inputs, and golden datasets.
• Monitor model drift, regression, and performance degradation over time.
• Create test strategy documents, test plans, and compliance-ready test evidence.
• Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features.

Requirements

• 5+ years of hands-on QA/test automation experience.
• Experience building and maintaining automated test suites from scratch using Playwright, Selenium, Cypress, and pytest.
• Proficiency in Python for writing automation harnesses, fixtures, and utilities independently.
• API and backend test automation experience (REST APIs, microservices, serverless) using Postman/Newman and pytest with requests.
• Knowledge of CI/CD test integration into pipelines such as Jenkins and GitHub Actions with quality gates.
• Experience testing AI/GenAI applications and validating non-deterministic AI/LLM outputs: accuracy, RAG relevance, hallucination, and prompt regression.
• Familiarity with AI/ML model testing (accuracy, bias, performance) using RAGAS, LLM-as-judge, and golden datasets.
• English proficiency at B2 level or higher.

Benefits

• Stable pay and Employee Stock Purchase Plan participation with a 15% discount.
• Comprehensive benefits package (health insurance, Multisport, shopping vouchers).
• Referral bonuses up to $2,000.
• Opportunities for growth: career development programs, mentoring, soft skills and well-being programs, and certifications (Anthropic, Gemini, GCP, Azure, AWS) plus English classes.
• Hybrid work with the option to work remotely within Poland and chance to work abroad up to 60 days annually.
• Business-driven relocation opportunities.
• Office amenities and social/well-being events (snacks, coffee, and recreation zones).