AI Quality Engineer

ClifyX

Austin, TX • 16m ago • Hybrid

Mid

Hybrid

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Quick Facts

  • Role: AI Quality Engineer
  • Location: Austin, TX (Hybrid Onsite)
  • Contract Duration: 12+ Months

Description

AI Quality Engineering (AI QE) Engineer to establish and scale quality and evaluation frameworks for next-generation conversational AI and agentic systems. The role includes designing automated evaluation pipelines, leading guardrail and safety assessments, and developing test suites for non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend service integrations.

Responsibilities

  • Design and build automated AI evaluation pipelines to measure faithfulness, answer completeness, hallucination rates, and task completion.
  • Develop, calibrate, and maintain rubric-based LLM-as-a-judge evaluators.
  • Create adversarial and safety test plans for prompt injection, jailbreaks, PII leakage, guardrail boundary testing, and adversarial edge cases.
  • Partner with ML and platform engineers to use distributed traces, logs, and telemetry to identify whether failures come from prompts, models, RAG retrieval, or downstream connectors.
  • Build programmatic synthetic test data generation pipelines for persona simulations, multi-turn edge cases, and golden evaluation datasets.
  • Integrate automated regression suites into CI/CD build and release pipelines to gate prompt changes, model updates, and backend API updates.

Benefits

  • 12+ months contract role with hybrid onsite work in Austin, TX.

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report