
AI Quality Engineer
ClifyX
Austin, TX • 16m ago • Hybrid
Mid
Hybrid
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Quick Facts
- Role: AI Quality Engineer
- Location: Austin, TX (Hybrid Onsite)
- Contract Duration: 12+ Months
Description
AI Quality Engineering (AI QE) Engineer to establish and scale quality and evaluation frameworks for next-generation conversational AI and agentic systems. The role includes designing automated evaluation pipelines, leading guardrail and safety assessments, and developing test suites for non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend service integrations.
Responsibilities
- Design and build automated AI evaluation pipelines to measure faithfulness, answer completeness, hallucination rates, and task completion.
- Develop, calibrate, and maintain rubric-based LLM-as-a-judge evaluators.
- Create adversarial and safety test plans for prompt injection, jailbreaks, PII leakage, guardrail boundary testing, and adversarial edge cases.
- Partner with ML and platform engineers to use distributed traces, logs, and telemetry to identify whether failures come from prompts, models, RAG retrieval, or downstream connectors.
- Build programmatic synthetic test data generation pipelines for persona simulations, multi-turn edge cases, and golden evaluation datasets.
- Integrate automated regression suites into CI/CD build and release pipelines to gate prompt changes, model updates, and backend API updates.
Benefits
- 12+ months contract role with hybrid onsite work in Austin, TX.
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
AI Quality Engineer
ClifyX
Mid
Technology
Austin, TX · Hybrid
N/A
16m ago

Senior AI Quality Engineer (GenAI/LLM Evaluation & Agentic AI)
ClifyX
Senior
Technology
Austin, TX · Hybrid
N/A
16m ago

Techno-Functional QA Engineer (AI Automation)
Tek Inspirations LLC
Senior
Technology
Houston, TX, United States of America · Hybrid
N/A
1 days ago

Techno-Functional QA Engineer (AI Automation)
Fervorly
Senior
Technology
Houston, TX · On-site
N/A
11m ago

Techno-Functional QA Engineer (AI Automation)
Logging-In
Senior
Technology
Houston, TX · On-site
N/A
6 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Techno-Functional QA Engineer (AI Test Automation)
Tek Inspirations LLC
Senior
Technology
Houston, TX · Hybrid
N/A
7 days ago

AI Software Engineer
Photon
Mid
Technology
Sunnyvale, CA · On-site
N/A
1 days ago

AI Software Engineer
Photon
Mid
Technology
Sunnyvale, CA · On-site
N/A
6 days ago

Automation QA Engineer
Ideate Technologies LLC
Senior
Technology
Charlotte, NC · On-site
N/A
9 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights