New offer - be the first one to apply!

June 20, 2026

Automation QA Engineer

Mid • Remote

2,800 - 3,600 EUR/hr

Gdansk, Poland

Salary range: B2B 20–25 EUR/h + VAT

Ciklum is looking for an Automation QA Engineer to join the team full-time in Poland.

About the role

As an Automation QA Engineer, you will become part of a cross-functional development team engineering next-generation experiences.

Responsibilities

  • Programmatic RAG Evaluation: Build and run automated validation test suites using Python frameworks like RAGAS to continuously score pipeline faithfulness, answer relevance, and context precision.
  • CI/CD Quality Gates: Integrate automated LLM evaluation scorecards directly into Azure DevOps pipelines to prevent regression in code-generation quality.
  • Data Integrity Validation: Develop automated checks to monitor documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy.
  • Phase-Gate Reporting: Generate programmatic quality scorecards and regression reports to satisfy technical criteria at each project gate approval milestone.
  • Squad Collaboration: Work closely with senior and mid-level ML implementation engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by validation metrics.

Requirements

  • Core Experience Baseline: 3+ years of experience in backend test automation, data quality engineering, or Python software development within structured delivery environments.
  • Automation Engineering Stack: Mastery of Python for backend testing, scripting, and data manipulation (traditional UI automation tools like Selenium or Cypress are not applicable).
  • AI Evaluation Frameworks: Hands-on experience or strong technical familiarity with LLM validation frameworks (specifically RAGAS or equivalent automated validation engines).
  • CI/CD Infrastructure: Practical experience configuring automated script execution, regression test triggers, and quality gates inside Git-based pipelines, preferably Azure DevOps.
  • Data & Search Familiarity: Basic understanding of relational databases (PostgreSQL), data profiling techniques, and search indexing concepts.
  • Location: Must be physically located in Poland or Eastern Europe.

What’s in it for you?

  • Strong community: Work in a friendly, open-door environment with experienced professionals.
  • Growth focus: Participate in large-scale, globally impactful projects.
  • Tailored learning: Access to internal events, Udemy, language courses, and company-paid certifications.
  • Endless opportunities: Internal mobility across domains and projects.
  • Flexibility: Full remote working possibilities.
  • Care: Company-paid medical insurance.