
AI Quality Engineer
ClifyX
Austin, TX • 16m ago • Hybrid
Mid
Hybrid
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Quick Facts
Role: AI Quality Engineer
Location: Austin, TX (Hybrid Onsite)
Contract: 12+ months
Description
Design and scale quality and evaluation frameworks for next-generation conversational AI and agentic systems. You will build automated evaluation pipelines, create LLM-as-a-judge and heuristic scoring, and develop guardrail/safety stress tests for non-deterministic multi-turn dialogues, tool-use workflows, and backend integrations.
Responsibilities
Architect automated AI evaluation pipelines measuring faithfulness, answer completeness, hallucination rates, and task completion
Develop and maintain rubric-based LLM-as-a-judge evaluators
Design adversarial and safety testing for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and edge cases
Partner with ML/Platform engineers by analyzing distributed traces, logs, and telemetry to identify failure sources (prompts, models, RAG, or downstream connectors)
Build programmatic synthetic test data generation for diverse personas, multi-turn edge cases, and golden evaluation datasets
Integrate automated regression suites into CI/CD build/release pipelines to gate prompt changes, model updates, and backend API updates
Requirements
Experience designing and building automated AI evaluation pipelines
Proficiency with LLM-as-a-judge or rubric-based evaluation approaches
Hands-on experience with adversarial safety testing (prompt injection/jailbreaks/PII leakage)
Ability to evaluate non-deterministic multi-turn conversational outputs and tool-use workflows
Experience using logs/telemetry/traces to diagnose failures across prompts, models, retrieval, and downstream services
Experience generating synthetic test data and maintaining golden evaluation datasets
Experience integrating regression suites into CI/CD pipelines
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
AI Quality Engineer
ClifyX
Mid
Technology
Austin, TX · Hybrid
N/A
16m ago

Senior AI Quality Engineer (GenAI/LLM Evaluation & Agentic AI)
ClifyX
Senior
Technology
Austin, TX · Hybrid
N/A
15m ago

AI Engineer (Agentic AI / LLM)
DCV Technologies
Mid
Technology
Warsaw, Poland · Remote
N/A
5 days ago

AI Software Engineer
Photon
Mid
Technology
Sunnyvale, CA · On-site
N/A
1 days ago

AI Software Engineer
Photon
Mid
Technology
Sunnyvale, CA, United States of America · On-site
N/A
5 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Software Engineer, AI Quality & Evaluation
Bask Health
Mid
Technology
Remote
N/A
1 days ago

Techno-Functional QA Engineer (AI Automation)
Tek Inspirations LLC
Senior
Technology
Houston, TX, United States of America · Hybrid
N/A
1 days ago

AI Engineer
Rivago infotech inc
Mid
Technology
Irving, TX · On-site
N/A
7 days ago

Techno-Functional QA Engineer (AI Automation)
Fervorly
Senior
Technology
Houston, TX · On-site
N/A
11m ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights