Automation QA Engineer

Salary range: B2B 20–25 EUR/h + VAT

Ciklum is looking for an Automation QA Engineer to join the team full-time in Poland.

About the role

As an Automation QA Engineer, you will become part of a cross-functional development team engineering next-generation experiences.

Responsibilities

• Programmatic RAG Evaluation: Build and run automated validation test suites using Python frameworks like RAGAS to continuously score pipeline faithfulness, answer relevance, and context precision.
• CI/CD Quality Gates: Integrate automated LLM evaluation scorecards directly into Azure DevOps pipelines to prevent regression in code-generation quality.
• Data Integrity Validation: Develop automated checks to monitor documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy.
• Phase-Gate Reporting: Generate programmatic quality scorecards and regression reports to satisfy technical criteria at each project gate approval milestone.
• Squad Collaboration: Work closely with senior and mid-level ML implementation engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by validation metrics.

Requirements

• Core Experience Baseline: 3+ years of experience in backend test automation, data quality engineering, or Python software development within structured delivery environments.
• Automation Engineering Stack: Mastery of Python for backend testing, scripting, and data manipulation (traditional UI automation tools like Selenium or Cypress are not applicable).
• AI Evaluation Frameworks: Hands-on experience or strong technical familiarity with LLM validation frameworks (specifically RAGAS or equivalent automated validation engines).
• CI/CD Infrastructure: Practical experience configuring automated script execution, regression test triggers, and quality gates inside Git-based pipelines, preferably Azure DevOps.
• Data & Search Familiarity: Basic understanding of relational databases (PostgreSQL), data profiling techniques, and search indexing concepts.
• Location: Must be physically located in Poland or Eastern Europe.

What’s in it for you?

• Strong community: Work in a friendly, open-door environment with experienced professionals.
• Growth focus: Participate in large-scale, globally impactful projects.
• Tailored learning: Access to internal events, Udemy, language courses, and company-paid certifications.
• Endless opportunities: Internal mobility across domains and projects.
• Flexibility: Full remote working possibilities.
• Care: Company-paid medical insurance.