Azure QA/DevOps Engineer

CGI

Plano, TX • 1h ago • On-site

Mid

On-site

103,300 - 223,700 USD/yr

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Potential salary for Azure QA/DevOps Engineer role

Improve your resume now

$103,300 – $223,700/year

Quick Facts

  • Role: QA/DevOps Engineer for GenAI and agentic AI products on Microsoft Azure
  • Focus: automated testing and AI evaluation, plus Azure CI/CD and reliability

Description

You will own the quality, reliability, and delivery pipeline for GenAI and agentic AI products on Microsoft Azure. The role combines software testing, cloud operations, and AI evaluation to enable fast, confident releases of LLM-powered applications (RAG pipelines, multi-step agents, and the APIs/UIs around them). You will define testing approaches for non-deterministic behaviors driven by prompt/model changes and multi-step agent execution.

Responsibilities

  • Design and maintain automated test suites for Python/FastAPI services (unit, integration, contract, load) and React front ends (component and end-to-end).
  • Build evaluation pipelines for RAG and agentic workflows, including retrieval quality, groundedness, answer relevance, hallucination rates, tool call accuracy, and multi-step LangGraph trajectories.
  • Create and curate golden datasets, regression sets, and adversarial test cases (prompt injection, jailbreaks, PII leakage, unsafe output) and use them to gate releases.
  • Use LangSmith for tracing, datasets, experiments, and online/offline evaluation to catch regressions when prompts, models, or retrieval configs change.
  • Define quality metrics and thresholds with engineers and product owners and report results in a way the team can act on.
  • Build and maintain CI/CD pipelines (Azure DevOps or GitHub Actions) with automated quality, security, and evaluation gates.
  • Provision and manage Azure infrastructure as code (Bicep, Terraform, or ARM) for services including Azure Container Apps/AKS, App Service, Azure OpenAI, Azure AI Search, Key Vault, Storage, and networking.
  • Containerize applications with Docker and manage environments, secrets, configuration, and promotion across dev, test, and production.
  • Implement monitoring, alerting, and cost tracking (Azure Monitor, Application Insights, Log Analytics) covering token usage, latency, and error budgets for LLM calls.
  • Support incident response, root-cause analysis, and post-incident improvements for AI services in production.
  • Collaborate with AI engineers, backend/frontend developers, and security/compliance to embed quality and reliability into design.
  • Document test strategies, runbooks, and evaluation methodology.

Requirements

  • 4+ years in QA automation, DevOps, SDET, or Site Reliability Engineering, with at least 1 year on or alongside ML/AI-enabled applications.
  • Strong Python skills (pytest) and automated testing for REST APIs; working knowledge of FastAPI.
  • Hands-on experience with Azure services and cloud deployment fundamentals (containers, networking, identity/RBAC, Key Vault, monitoring).
  • Proven experience building CI/CD pipelines using Azure DevOps or GitHub Actions.
  • Experience with infrastructure as code (Bicep, Terraform, or equivalent) and Docker; Kubernetes is a plus.
  • Experience testing React applications using Playwright, Cypress, Jest, or React Testing Library.
  • Understanding of how LLM applications work (prompts, embeddings, vector search, retrieval, agent tool use) and why testing differs from deterministic software.
  • Solid grasp of test strategy, API and performance testing, and quality metrics.
  • Clear written and verbal communication; comfortable working across engineering, product, and security teams.

Benefits

  • Competitive compensation
  • Comprehensive insurance options
  • 401(k) plan and share purchase plan matching contributions
  • Paid time off, paid parental leave, learning opportunities, and tuition assistance
  • Wellness and wellbeing programs

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report