
Senior AI Quality Engineer (GenAI/LLM Evaluation & Agentic AI)
ClifyX
Austin, TX • 2m ago • Hybrid
Senior
Hybrid
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Quick Facts
- Role: Senior AI Quality Engineer (GenAI / LLM Evaluation & Agentic AI)
- Location: Austin, TX (Hybrid)
- Contract: Long term
- Eligibility: USC only
Description
Establish and scale quality and evaluation frameworks for next-generation conversational AI and agentic systems. You will design automated evaluation pipelines and safety/guardrail assessments, including test suites that handle non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend integrations. You’ll integrate automated regression suites into build/release pipelines.
Responsibilities
- Architect automated AI evaluation frameworks to measure faithfulness, answer completeness, hallucination rates, and task completion.
- Develop, calibrate, and maintain rubric-based LLM-as-a-judge evaluators.
- Create systematic adversarial and safety stress tests for prompt injection, jailbreaks, PII leakage, and guardrail boundaries.
- Partner with ML/platform engineers to analyze distributed traces, logs, and telemetry to identify failure sources (prompts, models, RAG, downstream connectors).
- Build synthetic test data pipelines for diverse persona simulations, multi-turn edge cases, and golden evaluation datasets.
- Integrate automated regression suites into CI/CD to gate prompt changes, model updates, and backend API changes.
Requirements
- USC status only
- Experience designing and scaling AI quality/evaluation frameworks for GenAI/LLM systems
- Hands-on ability to build automated evaluation pipelines and test suites for non-deterministic multi-turn and agentic workflows
- Experience with LLM-as-a-judge and rubric/heuristic scoring approaches
- Experience with guardrail and adversarial safety testing (prompt injection/jailbreak/PII leakage)
- Experience using telemetry (traces/logs) to diagnose issues across prompts, models, RAG, and integrations
- Experience integrating evaluation/regression tests into CI/CD pipelines
Benefits
- Long-term contract opportunity
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
AI Quality Engineer
ClifyX
Mid
Technology
Austin, TX · Hybrid
N/A
2m ago

AI Quality Engineer
ClifyX
Mid
Technology
Austin, TX · Hybrid
N/A
2m ago
Senior AI Tools Testing Engineer
select minds
Senior
Technology
Dallas, TX · On-site
$120k - $140k/yr
10 days ago

Senior AI Engineer
Photon
Senior
Technology
Dallas, TX · On-site
$100k - $120k/yr
2 days ago

Senior AI Engineer (M/F/D)
DSV
Senior
Technology
Warsaw, Poland · On-site
N/A
13 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Senior AI Engineer
Motion Recruitment Partners, LLC
Senior
Technology
Boston, MA, United States of America · On-site
$200k - $225k/yr
1m ago

Senior Agentic AI Engineer NDA
GT
Senior
Technology
Warsaw, Poland · Remote
N/A
8 days ago

Senior AI Engineer
T-Mobile
Senior
Technology
Washington, WA · On-site
$153k - $276k/yr
23h ago

Senior AI Engineer
WPP Enterprise Solutions
Senior
Technology
Poznań, Pl-15, Poland · Remote
312k zł - 395k zł/yr
14 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights