Senior AI Tools Testing Engineer

Quick Facts

• Role: Senior/Lead AI Tools & Testing Engineer (Agentic AI & LLM Evaluation)
• Work model: Hybrid
• Location: Dallas, TX
• Duration: Long-term
• Experience: 10+ years

Description

Hands-on engineering role focused on building, orchestrating, monitoring, and evaluating production-grade agentic AI and generative AI applications. You will design agent workflows, implement RAG solutions, and create automated LLM evaluation and regression-testing frameworks. You will also provide observability and monitoring using tracing/telemetry to continuously improve reliability, accuracy, and performance.

Responsibilities

• Design, develop, and deploy production-grade agentic and generative AI applications
• Orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore; implement multi-agent workflows, tool calling, state, memory, and orchestration
• Build RAG pipelines for ingestion, chunking, embeddings, retrieval, reranking, and context management; optimize quality, latency, and cost
• Create and automate LLM evaluation frameworks (DeepEval and other tools), including LLM-as-a-Judge methods and regression-testing pipelines
• Evaluate agent behavior, tool execution, workflow accuracy, and response consistency; identify failure patterns and improve quality
• Implement AI observability and monitoring with OpenTelemetry, LangSmith, and Langfuse; track requests, execution flows, tool calls, latency, token usage, errors, and performance
• Deploy and operate AI apps on Amazon Bedrock and Bedrock AgentCore; integrate models, agent frameworks, APIs, and enterprise data sources
• Support CI/CD integration and automated evaluation; optimize model selection, inference cost, latency, and resource utilization
• Provide technical leadership: establish reusable frameworks and best practices; mentor engineers; participate in design discussions and code reviews

Requirements

• 10+ years of software engineering or IT experience
• Hands-on experience developing and deploying GenAI or agentic AI applications
• Expert knowledge of LangGraph, LangChain, Amazon Bedrock, and AgentCore
• Strong experience implementing RAG solutions and optimizing retrieval accuracy, response quality, latency, and cost
• Experience building LLM evaluation frameworks and automated AI testing (including LLM-as-a-Judge)
• Experience with AI observability/monitoring using OpenTelemetry, LangSmith, and Langfuse
• Experience deploying production-grade GenAI applications on AWS and applying security, scalability, reliability, and performance practices
• Ability to establish AI quality standards and evaluation methodologies; strong collaboration with engineering and QA teams

Benefits

• Flexible work from home options
• Compensation: $120,000–$140,000 per year