
Azure QA/DevOps Engineer
CGI
Plano, TX • 1h ago • On-site
Mid
On-site
103,300 - 223,700 USD/yr
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Potential salary for Azure QA/DevOps Engineer role
Improve your resume now
$103,300 – $223,700/year
Quick Facts
- Role: QA/DevOps Engineer for GenAI and agentic AI products on Microsoft Azure
- Focus: automated testing and AI evaluation, plus Azure CI/CD and reliability
Description
You will own the quality, reliability, and delivery pipeline for GenAI and agentic AI products on Microsoft Azure. The role combines software testing, cloud operations, and AI evaluation to enable fast, confident releases of LLM-powered applications (RAG pipelines, multi-step agents, and the APIs/UIs around them). You will define testing approaches for non-deterministic behaviors driven by prompt/model changes and multi-step agent execution.
Responsibilities
- Design and maintain automated test suites for Python/FastAPI services (unit, integration, contract, load) and React front ends (component and end-to-end).
- Build evaluation pipelines for RAG and agentic workflows, including retrieval quality, groundedness, answer relevance, hallucination rates, tool call accuracy, and multi-step LangGraph trajectories.
- Create and curate golden datasets, regression sets, and adversarial test cases (prompt injection, jailbreaks, PII leakage, unsafe output) and use them to gate releases.
- Use LangSmith for tracing, datasets, experiments, and online/offline evaluation to catch regressions when prompts, models, or retrieval configs change.
- Define quality metrics and thresholds with engineers and product owners and report results in a way the team can act on.
- Build and maintain CI/CD pipelines (Azure DevOps or GitHub Actions) with automated quality, security, and evaluation gates.
- Provision and manage Azure infrastructure as code (Bicep, Terraform, or ARM) for services including Azure Container Apps/AKS, App Service, Azure OpenAI, Azure AI Search, Key Vault, Storage, and networking.
- Containerize applications with Docker and manage environments, secrets, configuration, and promotion across dev, test, and production.
- Implement monitoring, alerting, and cost tracking (Azure Monitor, Application Insights, Log Analytics) covering token usage, latency, and error budgets for LLM calls.
- Support incident response, root-cause analysis, and post-incident improvements for AI services in production.
- Collaborate with AI engineers, backend/frontend developers, and security/compliance to embed quality and reliability into design.
- Document test strategies, runbooks, and evaluation methodology.
Requirements
- 4+ years in QA automation, DevOps, SDET, or Site Reliability Engineering, with at least 1 year on or alongside ML/AI-enabled applications.
- Strong Python skills (pytest) and automated testing for REST APIs; working knowledge of FastAPI.
- Hands-on experience with Azure services and cloud deployment fundamentals (containers, networking, identity/RBAC, Key Vault, monitoring).
- Proven experience building CI/CD pipelines using Azure DevOps or GitHub Actions.
- Experience with infrastructure as code (Bicep, Terraform, or equivalent) and Docker; Kubernetes is a plus.
- Experience testing React applications using Playwright, Cypress, Jest, or React Testing Library.
- Understanding of how LLM applications work (prompts, embeddings, vector search, retrieval, agent tool use) and why testing differs from deterministic software.
- Solid grasp of test strategy, API and performance testing, and quality metrics.
- Clear written and verbal communication; comfortable working across engineering, product, and security teams.
Benefits
- Competitive compensation
- Comprehensive insurance options
- 401(k) plan and share purchase plan matching contributions
- Paid time off, paid parental leave, learning opportunities, and tuition assistance
- Wellness and wellbeing programs
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See moreAI Platform Engineer (Azure & GenAI)
KHS GmbH
Senior
Technology
Dortmund, NW, Germany · On-site
€57k - €89k/yr
22h ago

Lead Azure DevOps Engineer (.NET)
EPAM Systems
Senior
Technology
Wrocław, DS, Poland · Remote
N/A
15h ago

Lead Azure DevOps Engineer (.NET)
EPAM Systems
Senior
Technology
Poznan, Poland · Remote
N/A
15h ago

Lead Azure DevOps Engineer (.NET)
EPAM Systems
Senior
Technology
Gdańsk, GD, Poland · Remote
N/A
15h ago

GenAI Engineer (Full Stack or Backend Engineer)
StoneGate-Technologies LLC
Senior
Technology
Remote
N/A
3 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

DevOps / Platform Engineer
Alpha-Hub
Senior
Technology
Toronto, ON, Canada · Remote
N/A
8 days ago
DevOps Engineer – Agentic AI
CLOUDFIDE SPÓŁKA Z OGRANICZONĄ ODPOWIEDZIALNOŚCIĄ
Mid
Technology
Warsaw, Poland · Remote
N/A
16 days ago

Lead Azure DevOps Engineer (.NET)
EPAM Systems
Senior
Technology
Kraków, ML, Poland · Remote
N/A
15h ago

DevOps Engineer (K/M)
Alior Bank SA
Senior
Technology
Kraków, Poland · On-site
N/A
12 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights