AI Quality Engineer

ClifyX

Austin, TX • 16m ago • Hybrid

Mid

Hybrid

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Quick Facts

  • Role: AI Quality Engineer

  • Location: Austin, TX (Hybrid Onsite)

  • Contract: 12+ months

Description

Design and scale quality and evaluation frameworks for next-generation conversational AI and agentic systems. You will build automated evaluation pipelines, create LLM-as-a-judge and heuristic scoring, and develop guardrail/safety stress tests for non-deterministic multi-turn dialogues, tool-use workflows, and backend integrations.

Responsibilities

  • Architect automated AI evaluation pipelines measuring faithfulness, answer completeness, hallucination rates, and task completion

  • Develop and maintain rubric-based LLM-as-a-judge evaluators

  • Design adversarial and safety testing for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and edge cases

  • Partner with ML/Platform engineers by analyzing distributed traces, logs, and telemetry to identify failure sources (prompts, models, RAG, or downstream connectors)

  • Build programmatic synthetic test data generation for diverse personas, multi-turn edge cases, and golden evaluation datasets

  • Integrate automated regression suites into CI/CD build/release pipelines to gate prompt changes, model updates, and backend API updates

Requirements

  • Experience designing and building automated AI evaluation pipelines

  • Proficiency with LLM-as-a-judge or rubric-based evaluation approaches

  • Hands-on experience with adversarial safety testing (prompt injection/jailbreaks/PII leakage)

  • Ability to evaluate non-deterministic multi-turn conversational outputs and tool-use workflows

  • Experience using logs/telemetry/traces to diagnose failures across prompts, models, retrieval, and downstream services

  • Experience generating synthetic test data and maintaining golden evaluation datasets

  • Experience integrating regression suites into CI/CD pipelines

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report