2x QA Engineer - Mid and Senior - Project: CaaS

🏢 Summary: Mid and Senior QA Engineers are needed for a cloud-based CaaS project to ensure quality of APIs and data workflows in an automation-first environment. The role focuses on automated API and data validation, JSON schema verification, and testing of AWS-based services and data pipelines. Engineers will design and maintain automated tests to ensure reliability of enterprise platform services. 🗂️ Requirements: Minimum 2 years of automated testing with Python and pytest, Experience with API testing for REST services, Experience with AWS services: S3, Aurora, Glue, Step Functions, CloudWatch, Strong SQL skills and validation of relational databases, Experience with JSON schema validation and contract testing, Experience testing data pipelines and data transformations 📃 Skills: Python, pytest, REST, FastAPI, AWS, S3, Aurora, Glue, StepFunctions, CloudWatch, SQL, JSON 🏢 Description: Industry: education Budget: Mid: 127PLN/h + VAT Senior: 170PLN/h + VAT REMOTE FROM POLAND Introduction & Summary We are seeking two QA Engineers, at mid and senior levels, to join our team working on the CaaS project. This role is crucial in validating data and APIs within a cloud-based workflow environment, emphasizing an automation-first mindset. Candidates should possess hands-on experience in automated testing with Python and familiarity with AWS services, API testing, and data validation techniques. Main Responsibilities The QA Engineers will play a key role in ensuring the quality and reliability of critical enterprise capabilities across Sanoma Learning. Design, develop, and maintain automated tests for APIs, data pipelines, and platform services. Ensure data quality and correctness across content ingestion, enrichment, and delivery flows. Validate JSON schemas and shared data models across multiple consuming systems. Test REST APIs, including contract, integration, and regression testing. Verify data persistence and transformations across relational databases and object storage. Monitor and validate platform behavior using logs, metrics, and alerts. Collaborate with product owners and engineers to contribute to quality practices early in the development lifecycle. Key Requirements Hands-on experience with Python and pytest for automated testing (at least 2 years). Experience with API testing (FastAPI-based services). Experience with AWS services, in particular: S3, Aurora, Glue, Step Functions, CloudWatch. Good SQL skills and experience validating data stored in relational databases. Experience with JSON data validation, schema checks, and contract testing. English proficiency at B2/C1 level. Nice to Have Understanding of AI or ML-driven search, content enrichment, or recommendation systems. Experience with vector databases. Experience with Bitbucket Pipelines or similar CI/CD tooling. Experience validating LLM or AI-driven outputs, including quality, consistency, and determinism checks.

🏢 Summary: Full-time remote B2B role for a Senior QA Engineer combining manual and automation testing within advanced technology and Generative AI projects. The position focuses on designing and expanding test automation solutions while actively contributing to modern QA practices. The role involves close collaboration with development teams and ownership of automation initiatives across projects. 🗂️ Requirements: Minimum 5 years of experience in test automation, Minimum 3 years of experience with Python, Hands-on experience in manual and automated testing, Experience with cloud environments (AWS or Azure), Ability to design and implement test automation solutions, Experience in defect identification and resolution process, Experience with SITACH, UAT, QAT, and security testing, ISTQB certification, Fluency in English, Fluency in Polish 📃 Skills: Python, AWS, Azure, Automation, Manual, QA, Cloud, AI, ISTQB, SITACH, UAT, QAT, Pentesting 🏢 Description: PROJECT DETAILS: Workload: Full-time Model of working: 100% Remote Contract: B2B, long-term 12-months+ Rate: depending on experience 5+ years: 110-125 PLN/h nett + VAT 7-8+ years: 130-140 PLN/h nett + VAT Senior QA Engineer - Manual & Automation (AI, Python, Cloud) We are looking for an experienced QA Engineer to join a growing QA team working on advanced technology projects , including exciting Generative AI initiatives . This is a great opportunity for someone who combines strong manual testing foundations with solid automation expertise and wants to play a visible role in shaping modern QA practices. What makes this opportunity interesting? You will join a supportive QA team made up of 1 QA Manager and 3 testers , where onboarding is described as smooth, well-organised, and team-driven. The company is strongly investing in test automation , so this role is ideal for someone who wants not only to work with automation but also to help expand it across the organisation. Your responsibilities Design, create, and execute manual and automated test cases Test new features and perform regression testing Identify, report, and help drive the resolution of defects together with development teams Participate in test planning , estimation, and prioritisation Design and implement test automation solutions , strategies, and concepts Collaborate with developers on automation scripts from early development stages Coordinate the execution of automated tests within assigned areas or projects Help grow and strengthen test automation practices across the company Coach and support the testing team in the area of automation Contribute to activities related to SITACH, UAT, QAT, and security testing / pentesting What we are looking for Fluency in English and Polish At least 5 years of experience in test automation 3+ years of experience with Python Hands-on experience in both manual and automated testing Knowledge of cloud environments such as AWS or Azure ISTQB certification Willingness to learn and work with AI-related solutions Nice to have Openness to working with changing or evolving technology stacks Experience in performance testing , ideally with Locust Experience with Xray, Zephyr , or similar test management tools Knowledge of Jira or Azure DevOps

🏢 Summary: Senior QA Engineer role in the insurance/financial services industry, focused on ensuring high-quality deliverables through manual and automated testing, including projects involving generative AI. The position emphasizes building and scaling test automation practices in a 100% remote, long-term B2B engagement. The role involves close collaboration with developers and active contribution to automation strategy and quality processes. 🗂️ Requirements: Fluency in English, Fluency in Polish, 5+ years of experience in test automation, 3+ years of experience with Python, Experience in manual testing, Experience in automated testing, Knowledge of AWS or Azure, ISTQB certification, Willingness to work with AI-related solutions 📃 Skills: Python, AWS, Azure, ISTQB, Locust, Xray, Zephyr, Jira, AzureDevOps, SITACH, UAT, QAT, Pentesting 🏢 Description: GENERAL INFORMATION: Industry: Insurance / Financial Services Location: 100% remote Rate: up to 135 PLN/hour net + VAT Contract type: B2B – long-term cooperation Summary: The role of the Senior QA Engineer focuses on ensuring high-quality deliverables through both manual and automated testing, particularly in advanced technology projects, including generative AI initiatives. The primary goal is to enhance the testing processes and foster a culture of test automation within the team. Responsibilities: Designing, creating, and executing manual and automated test cases according to agreed standards. Testing new functionalities and performing regression testing . Reporting defects identified during testing and collaborating with developers for resolution and closure. Participating in test planning activities , including estimation and task prioritization. Designing, building, testing, and deploying test automation solutions , including test concepts and automation strategies. Collaborating with developers on automation scripts from the early stages of the development process. Coordinating execution of automated tests within assigned areas or projects. Participating in the development and expansion of test automation practices within the organization. Consulting and coaching the testing team in the area of test automation. Performing testing activities related to SITACH, UAT, QAT, and security testing (pentests) . Must Haves: Fluency in English and Polish . 5+ years of experience in test automation. 3+ years of experience with Python . Experience with manual and automated testing . Knowledge of cloud solutions (AWS or Azure) . ISTQB certification . Willingness to learn and work with new AI-related solutions . Nice to Haves: Openness to changing technology stacks. Experience in performance testing (preferably using Locust). Experience with Xray, Zephyr , or other test management tools. Knowledge of Jira or Azure DevOps . Other Details: Team Structure: The QA team consists of 1 QA Manager and 3 testers. Onboarding Process: Described as very supportive and smooth. Organizational Focus: Moving strongly toward test automation, with support for skill development.

🏢 Summary: Senior QA Engineer role focused on automation testing for the Credit Assessment and Decision team, with responsibility for API and regression test automation, release support, and quality assurance throughout the delivery lifecycle. The position emphasizes an automation-first approach, collaboration with developers and analysts, and maintaining high-quality testing practices in enterprise environments. Fully remote work is available with a B2B contract. 🗂️ Requirements: 5+ years of senior-level QA experience, Automation testing experience, Robot Framework, Python, API testing, System testing, Automated test suite maintenance, CI/CD pipelines, English 📃 Skills: Robot, Python, API, CI/CD, Jenkins, qTest, Pega, BizDevOps 🏢 Description: Industry: Retail Rate: up to 120pln/h net+VAT Location: Warsaw, Tricity, Łódź (Fully remote option available) Type of contract: B2B Summary: The Senior QA Engineer role is designed to enhance the quality assurance process within the Credit Assessment and Decision team by focusing on automation testing and ensuring project quality from early stages of delivery. Responsibilities: Design, create, and maintain automated tests for API-based systems and regression testing. Support release activities by preparing, executing, and following up on quality assurance tasks. Perform selective exploratory and release-focused manual testing where automation is not the most effective. Manage and maintain test assets with proper traceability and reporting in enterprise test management tools. Investigate failed test runs to identify issues related to product defects, test defects, data problems, or environment instability. Collaborate closely with developers and business analysts to ensure quality is addressed early in the delivery process. Act as a senior QA voice within the project by promoting best testing practices and a strong automation-first approach. Contribute to knowledge sharing and participate in cross-project tester ceremonies. Requirements: 5+ years of senior-level QA experience with a focus on automation testing. Hands-on experience with Robot Framework and Python for test automation. Strong experience in API and system testing. Proficiency in maintaining automated test suites. Experience with CI/CD pipelines to support automated test execution. Good spoken and written English. Nice to have: Familiarity with qTest as a test management tool. Experience in financial services or regulated environments. Exposure to Pega-based solutions or comparable enterprise platforms. Experience in BizDevOps environments. Familiarity with Jenkins and similar CI/CD tools.

🏢 Summary: Automation-focused QA Engineer role within a PaaS Customer Delivery team, responsible for ensuring high-quality software delivery through manual and automated testing. The position involves designing and executing test cases, contributing to automation frameworks, and integrating tests into CI/CD pipelines in a hybrid work model in Warsaw. The role supports enterprise clients by maintaining rigorous quality standards across the full development lifecycle. 🗂️ Requirements: 2–4 years of experience in software quality assurance or testing, Experience in manual testing, including test planning, execution, and defect tracking, Experience working in Agile/Scrum environments, Experience with test case and issue management tools, Ability to design and maintain automated test suites, Experience with API testing tools, Basic knowledge of CI/CD workflows, Availability to work onsite in Warsaw 3 days per week 📃 Skills: Selenium, Cypress, Playwright, Postman, Swagger, Jira, TestRail, CI/CD, Agile, Scrum 🏢 Description: QA Engineer – Automation-Focused Software Testing 📍 Hybrid in Warsaw (3 days/week onsite required) | 💼 Full-time | B2B contract up to $5,500/month Our client is a US-based technology company headquartered in New York City, delivering digital solutions and consulting services that transform businesses and drive measurable value. With offices in multiple countries, the company is now investing in a new engineering centre in Warsaw, recognizing the strong talent and culture of Polish software professionals. We are currently seeking a QA Analyst- Software Testing to join the Platform-as-a-Service (PaaS) Customer Delivery team. This role is essential in ensuring the delivery of high-quality software to enterprise clients by proactively identifying issues, improving testing coverage, and contributing to automation frameworks. You will collaborate closely with engineering, product delivery, and business analysis teams to uphold rigorous quality standards across all phases of the development lifecycle. This is a hybrid position — you will be expected to work from the Warsaw office 3 days per week. Key Responsibilities Design and execute functional, integration, regression, and exploratory test cases Participate in requirement reviews and define clear acceptance criteria in collaboration with cross-functional teams Identify, document, and track defects; communicate risks and inconsistencies to stakeholders Maintain accurate test documentation and support User Acceptance Testing (UAT) with implementation teams Contribute to the design and maintenance of automated test suites, particularly using Selenium or similar frameworks Assist in expanding automation test coverage and integrating tests into CI/CD workflows Promote continuous improvement of QA processes, test tooling, and reporting standards Actively support quality assurance efforts throughout the entire software delivery lifecycle Key Requirements Required 2–4 years of experience in software quality assurance or testing Proficiency in manual testing practices, including test planning, execution, and defect tracking Working knowledge of Agile/Scrum methodologies and experience in cross-functional teams Familiarity with tools such as Jira, TestRail, or similar for test case and issue management Strong attention to detail and analytical thinking, with a focus on identifying edge cases and system weaknesses Preferred Experience with automation testing frameworks such as Selenium, Cypress, or Playwright Exposure to API testing tools (e.g., Postman, Swagger) Understanding of multi-tenant, configurable platforms, or SaaS products Basic knowledge of scripting or automation workflows in CI/CD environments What’s Offered B2B contract with monthly compensation up to $5 500 Strong career growth opportunities in a global fintech environment High-impact projects in a fast-growing sector Friendly, open, and ambitious team culture Hybrid model – minimum 3 days/week in the Warsaw office

🏢 Summary: Fully remote Senior AI QA Engineer role focused on testing AI agents and LLM-based applications through both manual and automated approaches. The position centers on building scalable LLM test harnesses, designing Gen AI evaluation frameworks, and ensuring reliability, accuracy, and performance of AI-driven systems. The role requires strong Python-based automation skills and experience with AI evaluation metrics, cloud environments, and CI/CD integration. 🗂️ Requirements: 5+ years of software QA experience, Minimum 1 year testing AI agents, agentic solutions or LLM-based systems, Experience with manual and automated testing of AI agents, Strong Python programming skills for test automation, Experience with pytest or equivalent testing frameworks, Expertise in AI agent frameworks and prompt engineering, Experience evaluating Gen AI/LLM systems (grounding, accuracy, hallucination, determinism), Knowledge of evaluation metrics: precision, recall, criteria recall, efficiency, Experience with Jira, QMetry or TestRail, Experience with version control systems, Experience integrating tests into CI/CD pipelines, Understanding of AWS cloud environments, Ability to work 13:00–21:00 Polish time 📃 Skills: Python, pytest, LLM, GenAI, LangChain, OpenAI, AWS, Jira, QMetry, TestRail, CI/CD, Git, Copilot 🏢 Description: We are seeking a skilled Senior AI QA Engineer with strong experience in both manual and automated testing and extensive exposure to AI-based application testing. The ideal candidate will test a variety of applications, including projects involving AI agents and integrations with APIs and databases. You will help ensure our solutions are reliable and accurate and meet business requirements, while also contributing to the development of our automation capabilities. This is a fully remote position with a requirement to work from 13:00 to 21:00 Polish time, due to the client team's location. Responsibilities Research and evolve automation frameworks in line with Gen AI tooling and best practices Design and automate evaluation of Gen AI features — grounding, answer accuracy, determinism/reproducibility, precision, recall, and criteria recall Build automated LLM test harnesses that scale evaluation beyond human-in-the-loop Selection and application of Gen AI evaluation frameworks, measuring answer quality and pipeline efficiency Perform manual testing as needed to validate new features, integrations, and user stories Build and maintain test cases from requirements and user stories Test applications that may include AI agents, APIs, databases, and other integrations Collaborate with product, engineering, and operations teams to understand requirements and deployment environments Track and report test results, defects, and quality metrics Assist with troubleshooting production issues and escalate risks as needed Guide and support team members, including onshore and offshore consultants Requirements 5+ years of experience in software QA, with at least 1 year focused on testing AI agents, agentic solutions or LLM-based systems Hands-on experience with both manual and automated testing of AI agents, including prompt/instruction testing and evaluation of agentic workflows Strong programming skills in Python test automation — pytest or equivalent, scripting and AI/ML library integration Expertise in AI agent frameworks, prompt engineering and evaluation metrics for LLM-based systems Demonstrated experience testing and evaluating Gen AI / LLM applications — grounding, answer accuracy and hallucination/determinism checks Applied knowledge of Gen AI / LLM evaluation frameworks and metrics — precision, recall, criteria recall and efficiency Familiarity with issue and test management tools such as Jira, QMetry and TestRail Experience with version control systems and integrating tests into CI/CD pipelines Flexibility to use AI-powered tools for QA such as GitHub Copilot and LLM-based test generation Understanding of cloud environments, particularly AWS Excellent communication, collaboration and leadership skills Nice to have Experience with agentic AI platforms such as LangChain, OpenAI Function Calling or similar Experience with AI safety, bias and reliability testing Experience with test data generation for AI/ML systems We offer We gather like-minded people: Engineering community of industry professionals Friendly team and enjoyable working environment Flexible schedule and opportunity to work remotely within Poland Chance to work abroad for up to 60 days annually Business-driven relocation opportunities We provide growth opportunities: Outstanding career roadmap Leadership development, career advising, soft skills, and well-being programs Certification (GCP, Azure, AWS) Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru English classes We cover it all: Stable income (Employment Contract or B2B) Participation in the Employee Stock Purchase Plan Benefits package (health insurance, multisport, shopping vouchers) Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more Referral bonuses Corporate, social and well-being events Please, note: The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview. We will reach out to selected candidates exclusively. EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

🏢 Summary: Fully remote Senior AI QA Engineer role focused on testing and automating evaluation of Gen AI and LLM-based applications, including AI agents and integrated systems. The position combines manual and automated testing, development of LLM test harnesses, and implementation of AI evaluation frameworks to ensure reliability, accuracy, and performance. Work is conducted in collaboration with cross-functional teams, with working hours aligned to 13:00–21:00 Polish time. 🗂️ Requirements: 5+ years of software QA experience, Minimum 1 year testing AI agents, agentic solutions or LLM-based systems, Hands-on experience in manual and automated testing of AI agents, Strong Python programming skills for test automation, Experience with pytest or equivalent testing frameworks, Expertise in AI agent frameworks and prompt engineering, Experience evaluating Gen AI / LLM systems (grounding, accuracy, hallucination, determinism), Knowledge of Gen AI / LLM evaluation metrics (precision, recall, criteria recall, efficiency), Experience with Jira, QMetry or TestRail, Experience with version control systems and CI/CD integration, Understanding of AWS cloud environments, Ability to work 13:00–21:00 Polish time 📃 Skills: Python, pytest, LLM, GenAI, LangChain, OpenAI, AWS, Jira, QMetry, TestRail, CI/CD, Git 🏢 Description: We are seeking a skilled Senior AI QA Engineer with strong experience in both manual and automated testing and extensive exposure to AI-based application testing. The ideal candidate will test a variety of applications, including projects involving AI agents and integrations with APIs and databases. You will help ensure our solutions are reliable and accurate and meet business requirements, while also contributing to the development of our automation capabilities. This is a fully remote position with a requirement to work from 13:00 to 21:00 Polish time, due to the client team's location. Responsibilities Research and evolve automation frameworks in line with Gen AI tooling and best practices Design and automate evaluation of Gen AI features — grounding, answer accuracy, determinism/reproducibility, precision, recall, and criteria recall Build automated LLM test harnesses that scale evaluation beyond human-in-the-loop Selection and application of Gen AI evaluation frameworks, measuring answer quality and pipeline efficiency Perform manual testing as needed to validate new features, integrations, and user stories Build and maintain test cases from requirements and user stories Test applications that may include AI agents, APIs, databases, and other integrations Collaborate with product, engineering, and operations teams to understand requirements and deployment environments Track and report test results, defects, and quality metrics Assist with troubleshooting production issues and escalate risks as needed Guide and support team members, including onshore and offshore consultants Requirements 5+ years of experience in software QA, with at least 1 year focused on testing AI agents, agentic solutions or LLM-based systems Hands-on experience with both manual and automated testing of AI agents, including prompt/instruction testing and evaluation of agentic workflows Strong programming skills in Python test automation — pytest or equivalent, scripting and AI/ML library integration Expertise in AI agent frameworks, prompt engineering and evaluation metrics for LLM-based systems Demonstrated experience testing and evaluating Gen AI / LLM applications — grounding, answer accuracy and hallucination/determinism checks Applied knowledge of Gen AI / LLM evaluation frameworks and metrics — precision, recall, criteria recall and efficiency Familiarity with issue and test management tools such as Jira, QMetry and TestRail Experience with version control systems and integrating tests into CI/CD pipelines Flexibility to use AI-powered tools for QA such as GitHub Copilot and LLM-based test generation Understanding of cloud environments, particularly AWS Excellent communication, collaboration and leadership skills Nice to have Experience with agentic AI platforms such as LangChain, OpenAI Function Calling or similar Experience with AI safety, bias and reliability testing Experience with test data generation for AI/ML systems We offer We gather like-minded people: Engineering community of industry professionals Friendly team and enjoyable working environment Flexible schedule and opportunity to work remotely within Poland Chance to work abroad for up to 60 days annually Business-driven relocation opportunities We provide growth opportunities: Outstanding career roadmap Leadership development, career advising, soft skills, and well-being programs Certification (GCP, Azure, AWS) Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru English classes We cover it all: Stable income (Employment Contract or B2B) Participation in the Employee Stock Purchase Plan Benefits package (health insurance, multisport, shopping vouchers) Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more Referral bonuses Corporate, social and well-being events Please, note: The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview. We will reach out to selected candidates exclusively. EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

🏢 Summary: Fully remote Senior AI QA Engineer role focused on testing and evaluating AI agents and LLM-based systems using both manual and automated approaches. The position emphasizes building scalable LLM test harnesses, designing GenAI evaluation frameworks, and ensuring reliability, accuracy, and performance of AI-driven applications. Work is conducted in collaboration with cross-functional teams with required availability from 13:00 to 21:00 Polish time. 🗂️ Requirements: 5+ years of experience in software QA, Minimum 1 year of experience testing AI agents, agentic solutions or LLM-based systems, Hands-on experience in manual and automated testing of AI agents, Strong Python programming skills for test automation, Experience with pytest or equivalent automation frameworks, Experience integrating AI/ML libraries in test automation, Expertise in AI agent frameworks and prompt engineering, Experience evaluating GenAI/LLM systems including grounding and hallucination checks, Knowledge of evaluation metrics: precision, recall, criteria recall, determinism, Experience with Jira, QMetry or TestRail, Experience with version control systems, Experience integrating tests into CI/CD pipelines, Understanding of AWS cloud environments, Availability to work 13:00–21:00 Polish time 📃 Skills: Python, pytest, LLM, GenAI, LangChain, OpenAI, AWS, Jira, QMetry, TestRail, Git, CI/CD 🏢 Description: We are seeking a skilled Senior AI QA Engineer with strong experience in both manual and automated testing and extensive exposure to AI-based application testing. The ideal candidate will test a variety of applications, including projects involving AI agents and integrations with APIs and databases. You will help ensure our solutions are reliable and accurate and meet business requirements, while also contributing to the development of our automation capabilities. This is a fully remote position with a requirement to work from 13:00 to 21:00 Polish time, due to the client team's location. Responsibilities Research and evolve automation frameworks in line with Gen AI tooling and best practices Design and automate evaluation of Gen AI features — grounding, answer accuracy, determinism/reproducibility, precision, recall, and criteria recall Build automated LLM test harnesses that scale evaluation beyond human-in-the-loop Selection and application of Gen AI evaluation frameworks, measuring answer quality and pipeline efficiency Perform manual testing as needed to validate new features, integrations, and user stories Build and maintain test cases from requirements and user stories Test applications that may include AI agents, APIs, databases, and other integrations Collaborate with product, engineering, and operations teams to understand requirements and deployment environments Track and report test results, defects, and quality metrics Assist with troubleshooting production issues and escalate risks as needed Guide and support team members, including onshore and offshore consultants Requirements 5+ years of experience in software QA, with at least 1 year focused on testing AI agents, agentic solutions or LLM-based systems Hands-on experience with both manual and automated testing of AI agents, including prompt/instruction testing and evaluation of agentic workflows Strong programming skills in Python test automation — pytest or equivalent, scripting and AI/ML library integration Expertise in AI agent frameworks, prompt engineering and evaluation metrics for LLM-based systems Demonstrated experience testing and evaluating Gen AI / LLM applications — grounding, answer accuracy and hallucination/determinism checks Applied knowledge of Gen AI / LLM evaluation frameworks and metrics — precision, recall, criteria recall and efficiency Familiarity with issue and test management tools such as Jira, QMetry and TestRail Experience with version control systems and integrating tests into CI/CD pipelines Flexibility to use AI-powered tools for QA such as GitHub Copilot and LLM-based test generation Understanding of cloud environments, particularly AWS Excellent communication, collaboration and leadership skills Nice to have Experience with agentic AI platforms such as LangChain, OpenAI Function Calling or similar Experience with AI safety, bias and reliability testing Experience with test data generation for AI/ML systems We offer We gather like-minded people: Engineering community of industry professionals Friendly team and enjoyable working environment Flexible schedule and opportunity to work remotely within Poland Chance to work abroad for up to 60 days annually Business-driven relocation opportunities We provide growth opportunities: Outstanding career roadmap Leadership development, career advising, soft skills, and well-being programs Certification (GCP, Azure, AWS) Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru English classes We cover it all: Stable income (Employment Contract or B2B) Participation in the Employee Stock Purchase Plan Benefits package (health insurance, multisport, shopping vouchers) Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more Referral bonuses Corporate, social and well-being events Please, note: The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview. We will reach out to selected candidates exclusively. EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

🏢 Summary: Fully remote Senior AI QA Engineer role focused on testing AI agents and LLM-based systems through both manual and automated approaches. The position emphasizes building scalable automation frameworks, LLM test harnesses, and evaluating GenAI features using advanced metrics. The role requires strong Python automation skills, experience with AI evaluation frameworks, and collaboration across product and engineering teams. 🗂️ Requirements: 5+ years of software QA experience, At least 1 year testing AI agents or LLM-based systems, Experience with manual and automated testing of AI agents, Strong Python programming for test automation, Experience with pytest or equivalent framework, Expertise in AI agent frameworks and prompt engineering, Experience evaluating Gen AI/LLM systems (grounding, accuracy, hallucination, determinism), Knowledge of evaluation metrics (precision, recall, criteria recall, efficiency), Experience with Jira, QMetry, or TestRail, Experience with version control systems, Experience integrating tests into CI/CD pipelines, Understanding of AWS cloud environments, Availability to work 13:00–21:00 Polish time 📃 Skills: Python, pytest, LLM, GenAI, LangChain, OpenAI, AWS, Jira, QMetry, TestRail, CI/CD, Git 🏢 Description: We are seeking a skilled Senior AI QA Engineer with strong experience in both manual and automated testing and extensive exposure to AI-based application testing. The ideal candidate will test a variety of applications, including projects involving AI agents and integrations with APIs and databases. You will help ensure our solutions are reliable and accurate and meet business requirements, while also contributing to the development of our automation capabilities. This is a fully remote position with a requirement to work from 13:00 to 21:00 Polish time, due to the client team's location. Responsibilities Research and evolve automation frameworks in line with Gen AI tooling and best practices Design and automate evaluation of Gen AI features — grounding, answer accuracy, determinism/reproducibility, precision, recall, and criteria recall Build automated LLM test harnesses that scale evaluation beyond human-in-the-loop Selection and application of Gen AI evaluation frameworks, measuring answer quality and pipeline efficiency Perform manual testing as needed to validate new features, integrations, and user stories Build and maintain test cases from requirements and user stories Test applications that may include AI agents, APIs, databases, and other integrations Collaborate with product, engineering, and operations teams to understand requirements and deployment environments Track and report test results, defects, and quality metrics Assist with troubleshooting production issues and escalate risks as needed Guide and support team members, including onshore and offshore consultants Requirements 5+ years of experience in software QA, with at least 1 year focused on testing AI agents, agentic solutions or LLM-based systems Hands-on experience with both manual and automated testing of AI agents, including prompt/instruction testing and evaluation of agentic workflows Proficiency in at least one programming language for test automation (Python, Java, JavaScript, TypeScript, C#) Expertise in AI agent frameworks, prompt engineering and evaluation metrics for LLM-based systems Demonstrated experience testing and evaluating Gen AI / LLM applications — grounding, answer accuracy and hallucination/determinism checks Applied knowledge of Gen AI / LLM evaluation frameworks and metrics — precision, recall, criteria recall and efficiency Familiarity with issue and test management tools such as Jira, QMetry and TestRail Experience with version control systems and integrating tests into CI/CD pipelines Flexibility to use AI-powered tools for QA such as GitHub Copilot and LLM-based test generation Understanding of cloud environments, particularly AWS Excellent communication, collaboration and leadership skills Nice to have Experience with agentic AI platforms such as LangChain, OpenAI Function Calling or similar Experience with AI safety, bias and reliability testing Experience with test data generation for AI/ML systems We offer We gather like-minded people: Engineering community of industry professionals Friendly team and enjoyable working environment Flexible schedule and opportunity to work remotely within Poland Chance to work abroad for up to 60 days annually Business-driven relocation opportunities We provide growth opportunities: Outstanding career roadmap Leadership development, career advising, soft skills, and well-being programs Certification (GCP, Azure, AWS) Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru English classes We cover it all: Stable income (Employment Contract or B2B) Participation in the Employee Stock Purchase Plan Benefits package (health insurance, multisport, shopping vouchers) Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more Referral bonuses Corporate, social and well-being events Please, note: The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview. We will reach out to selected candidates exclusively. EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

🏢 Summary: Fully remote Senior AI QA Engineer role focused on testing AI agents and LLM-based systems through both manual and automated approaches. The position emphasizes building scalable automation frameworks, LLM test harnesses, and robust evaluation processes for Gen AI features. You will collaborate cross-functionally to ensure reliability, accuracy, and quality of AI-driven applications. 🗂️ Requirements: 5+ years of software QA experience, Minimum 1 year testing AI agents or LLM-based systems, Experience in manual and automated testing of AI agents, Strong Python programming skills for test automation, Experience with pytest or equivalent framework, Expertise in prompt engineering and LLM evaluation metrics, Experience evaluating Gen AI/LLM applications (grounding, hallucination, determinism), Knowledge of precision, recall, criteria recall metrics, Experience with Jira, QMetry or TestRail, Experience with CI/CD integration and version control systems, Understanding of AWS cloud environments, Availability to work 13:00–21:00 Polish time 📃 Skills: Python, pytest, LLM, GenAI, LangChain, OpenAI, Jira, QMetry, TestRail, AWS, CI/CD, Git 🏢 Description: We are seeking a skilled Senior AI QA Engineer with strong experience in both manual and automated testing and extensive exposure to AI-based application testing. The ideal candidate will test a variety of applications, including projects involving AI agents and integrations with APIs and databases. You will help ensure our solutions are reliable and accurate and meet business requirements, while also contributing to the development of our automation capabilities. This is a fully remote position with a requirement to work from 13:00 to 21:00 Polish time, due to the client team's location. Responsibilities Research and evolve automation frameworks in line with Gen AI tooling and best practices Design and automate evaluation of Gen AI features — grounding, answer accuracy, determinism/reproducibility, precision, recall, and criteria recall Build automated LLM test harnesses that scale evaluation beyond human-in-the-loop Selection and application of Gen AI evaluation frameworks, measuring answer quality and pipeline efficiency Perform manual testing as needed to validate new features, integrations, and user stories Build and maintain test cases from requirements and user stories Test applications that may include AI agents, APIs, databases, and other integrations Collaborate with product, engineering, and operations teams to understand requirements and deployment environments Track and report test results, defects, and quality metrics Assist with troubleshooting production issues and escalate risks as needed Guide and support team members, including onshore and offshore consultants Requirements 5+ years of experience in software QA, with at least 1 year focused on testing AI agents, agentic solutions or LLM-based systems Hands-on experience with both manual and automated testing of AI agents, including prompt/instruction testing and evaluation of agentic workflows Proficiency in at least one programming language for test automation (Python, Java, JavaScript, TypeScript, C#) Expertise in AI agent frameworks, prompt engineering and evaluation metrics for LLM-based systems Demonstrated experience testing and evaluating Gen AI / LLM applications — grounding, answer accuracy and hallucination/determinism checks Applied knowledge of Gen AI / LLM evaluation frameworks and metrics — precision, recall, criteria recall and efficiency Familiarity with issue and test management tools such as Jira, QMetry and TestRail Experience with version control systems and integrating tests into CI/CD pipelines Flexibility to use AI-powered tools for QA such as GitHub Copilot and LLM-based test generation Understanding of cloud environments, particularly AWS Excellent communication, collaboration and leadership skills Nice to have Experience with agentic AI platforms such as LangChain, OpenAI Function Calling or similar Experience with AI safety, bias and reliability testing Experience with test data generation for AI/ML systems We offer We gather like-minded people: Engineering community of industry professionals Friendly team and enjoyable working environment Flexible schedule and opportunity to work remotely within Poland Chance to work abroad for up to 60 days annually Business-driven relocation opportunities We provide growth opportunities: Outstanding career roadmap Leadership development, career advising, soft skills, and well-being programs Certification (GCP, Azure, AWS) Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru English classes We cover it all: Stable income (Employment Contract or B2B) Participation in the Employee Stock Purchase Plan Benefits package (health insurance, multisport, shopping vouchers) Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more Referral bonuses Corporate, social and well-being events Please, note: The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview. We will reach out to selected candidates exclusively. EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.