Automated QA Engineer

Mid • Remote

Warsaw, Poland

We are looking for an Automated QA Engineer to join our team who will focus on end-to-end quality assurance.

Your responsibilities:

● RAG Quality Evaluation: Design and curate datasets for RAG evaluation, develop and execute tests for RAG benchmarking, evaluating the accuracy and relevance of AI-generated content and information retrieval.

● Data-Driven Validation: Use data analysis techniques to evaluate test outcomes, track AI performance trends, and detect regression or drift across releases.

● Automation Test Solution Development: Design and implement automated test suites for both OutSystems front-end modules and backend microservices interacting with RaaS from the Galileo platform.

● Regression Testing: Build and maintain automated regression pipelines to ensure continuous platform stability after deployments and LLM-model updates.

● Integration Testing: Verify seamless communication between LUMIN and Galileo APIs, ensuring data integrity and correct processing across layers.

● CI/CD Integration: Embed automated tests into CI/CD pipelines (GitLab CI preferred) to enable consistent, repeatable quality gates before releases.

● Performance & Reliability Testing: Monitor and test system performance and reliability across environments, identifying performance bottlenecks or RAG degradation issues.

We are looking for you, if you have:● Experience: 4+ years of professional experience in automated software testing, including experience with web, API, and backend systems. Strong understanding of regression and integration testing principles. Proven experience in AI-app testing, specifically with applications using LLMs and RAG.

● Automation Skills: Hands-on experience with automated testing tools (e.g., Playwright, Selenium, or OutSystems test automation frameworks). Experience developing custom test scripts in Python or JavaScript/TypeScript.

● AI Quality Testing: Experience testing LLM and RAG applications, including prompt validation, factuality testing, and output benchmarking.

● Backend & API Testing: Proven experience validating RESTful APIs, microservices, and asynchronous workflows. Familiarity with API testing tools (e.g., Postman, pytest, REST Assured).

● CI/CD & DevOps: Working knowledge of GitLab CI or equivalent systems (e.g., Github Actions), with test automation integrated into build pipelines.

We offer:● Interesting and challenging projects

● Flexible working hours

● Friendly, non-corporate atmosphere

● Stable working conditions (CoE or B2B)

● Possibility for self-development and promotion in the company

● Rich benefits package

● Work model: remote or hybrid (2 days per week from the office)

Transition Technologies MS

Transition Technologies MS is a company operating in the software development industry, focusing on enterprise projects and products. The company values flexibility, offering remote or hybrid work models, and emphasizes a non-corporate atmosphere, suggesting a more relaxed and innovative work environment. Transition Technologies MS is committed to stable employment conditions and career development opportunities for its employees. While the company does not require experience in healthcare, it acknowledges the value of familiarity with healthcare systems, indicating a potential focus or interest in this sector. The company is distinguished by its involvement in interesting and challenging projects, and it provides a comprehensive benefits package to its employees.