Applied AI Engineer (LLM Systems)

ICF International, Inc.

Rockville, MD • 25m ago • Remote

Mid

Remote

98,614 - 167,644 USD/yr

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Potential salary for Applied AI Engineer (LLM Systems) role

Improve your resume now

$98,614 – $167,644/year

Quick Facts

  • Role: Applied AI Engineer (LLM systems) for global health and survey programs
  • Location: Remote within the United States (occasional travel)
  • Condition: Contingent on award of program funding
  • Eligibility: U.S. citizenship required for a federal government contract

Description

Build frontier-model pipelines and agents that power survey workflows, including AI-assisted data-to-report, natural-language analysis of survey data, multilingual questionnaire translation, and field support tools for interviewers and supervisors. Define acceptance criteria with technical and survey experts, create evaluation harnesses using expert-produced ground truth, and perform comparisons to determine whether tools can move forward. Manage model routing and inference cost while monitoring production performance and documenting results for transfer and public release.

Responsibilities

  • Design and build LLM pipelines and agents for survey workflows
  • Develop evaluation harnesses as first-class deliverables
  • Create test sets from expert ground truth and run non-inferiority comparisons for ship/no-ship decisions
  • Route tasks across models based on reasoning complexity, volume, cost, latency, and data-residency requirements
  • Track token usage, context limits, prompt caching, and batch processing to manage inference cost
  • Integrate models with data services and application layers built by full-stack developers
  • Use agentic coding tools to generate software from specifications; review, test, and own the code
  • Monitor production systems, conduct disciplined error analysis, and report results
  • Document methods and outcomes (including negative results) and support tool transfer to partner-country institutions

Requirements

  • U.S. citizenship required
  • Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent years of experience)
  • At least 4 years of professional software development experience, including production LLM-powered systems used by real users
  • Proficiency in Python or another modern language (e.g., JavaScript/TypeScript, Go)
  • Hands-on experience with frontier-model APIs, including tool use and retrieval-augmented generation
  • Demonstrated LLM evaluation experience: test-set construction, automated/human grading, regression testing, and ship/no-ship decisions
  • Working knowledge of LLM cost, latency, and model-selection trade-offs
  • Experience building and deploying on AWS, Azure, or Google Cloud
  • Experience using agentic coding tools (e.g., Claude Code, OpenAI Codex, Cursor, Antigravity, GitHub Copilot agent mode) to specify, review, test, and own production code
  • Experience with agent frameworks and AI evaluation tooling

Benefits

  • Pay range (full-time): $98,614.00 - $167,644.00
  • Nationwide remote office (US99)
  • Reasonable accommodations available for eligible candidates

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report