Senior AI Quality Engineer (GenAI/LLM Evaluation & Agentic AI)

ClifyX

Austin, TX • 2m ago • Hybrid

Senior

Hybrid

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Quick Facts

  • Role: Senior AI Quality Engineer (GenAI / LLM Evaluation & Agentic AI)
  • Location: Austin, TX (Hybrid)
  • Contract: Long term
  • Eligibility: USC only

Description

Establish and scale quality and evaluation frameworks for next-generation conversational AI and agentic systems. You will design automated evaluation pipelines and safety/guardrail assessments, including test suites that handle non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend integrations. You’ll integrate automated regression suites into build/release pipelines.

Responsibilities

  • Architect automated AI evaluation frameworks to measure faithfulness, answer completeness, hallucination rates, and task completion.
  • Develop, calibrate, and maintain rubric-based LLM-as-a-judge evaluators.
  • Create systematic adversarial and safety stress tests for prompt injection, jailbreaks, PII leakage, and guardrail boundaries.
  • Partner with ML/platform engineers to analyze distributed traces, logs, and telemetry to identify failure sources (prompts, models, RAG, downstream connectors).
  • Build synthetic test data pipelines for diverse persona simulations, multi-turn edge cases, and golden evaluation datasets.
  • Integrate automated regression suites into CI/CD to gate prompt changes, model updates, and backend API changes.

Requirements

  • USC status only
  • Experience designing and scaling AI quality/evaluation frameworks for GenAI/LLM systems
  • Hands-on ability to build automated evaluation pipelines and test suites for non-deterministic multi-turn and agentic workflows
  • Experience with LLM-as-a-judge and rubric/heuristic scoring approaches
  • Experience with guardrail and adversarial safety testing (prompt injection/jailbreak/PII leakage)
  • Experience using telemetry (traces/logs) to diagnose issues across prompts, models, RAG, and integrations
  • Experience integrating evaluation/regression tests into CI/CD pipelines

Benefits

  • Long-term contract opportunity

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report