
Applied AI Engineer (LLM Systems)
ICF International, Inc.
Rockville, MD • 25m ago • Remote
Mid
Remote
98,614 - 167,644 USD/yr
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Potential salary for Applied AI Engineer (LLM Systems) role
Improve your resume now
$98,614 – $167,644/year
Quick Facts
- Role: Applied AI Engineer (LLM systems) for global health and survey programs
- Location: Remote within the United States (occasional travel)
- Condition: Contingent on award of program funding
- Eligibility: U.S. citizenship required for a federal government contract
Description
Build frontier-model pipelines and agents that power survey workflows, including AI-assisted data-to-report, natural-language analysis of survey data, multilingual questionnaire translation, and field support tools for interviewers and supervisors. Define acceptance criteria with technical and survey experts, create evaluation harnesses using expert-produced ground truth, and perform comparisons to determine whether tools can move forward. Manage model routing and inference cost while monitoring production performance and documenting results for transfer and public release.
Responsibilities
- Design and build LLM pipelines and agents for survey workflows
- Develop evaluation harnesses as first-class deliverables
- Create test sets from expert ground truth and run non-inferiority comparisons for ship/no-ship decisions
- Route tasks across models based on reasoning complexity, volume, cost, latency, and data-residency requirements
- Track token usage, context limits, prompt caching, and batch processing to manage inference cost
- Integrate models with data services and application layers built by full-stack developers
- Use agentic coding tools to generate software from specifications; review, test, and own the code
- Monitor production systems, conduct disciplined error analysis, and report results
- Document methods and outcomes (including negative results) and support tool transfer to partner-country institutions
Requirements
- U.S. citizenship required
- Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent years of experience)
- At least 4 years of professional software development experience, including production LLM-powered systems used by real users
- Proficiency in Python or another modern language (e.g., JavaScript/TypeScript, Go)
- Hands-on experience with frontier-model APIs, including tool use and retrieval-augmented generation
- Demonstrated LLM evaluation experience: test-set construction, automated/human grading, regression testing, and ship/no-ship decisions
- Working knowledge of LLM cost, latency, and model-selection trade-offs
- Experience building and deploying on AWS, Azure, or Google Cloud
- Experience using agentic coding tools (e.g., Claude Code, OpenAI Codex, Cursor, Antigravity, GitHub Copilot agent mode) to specify, review, test, and own production code
- Experience with agent frameworks and AI evaluation tooling
Benefits
- Pay range (full-time): $98,614.00 - $167,644.00
- Nationwide remote office (US99)
- Reasonable accommodations available for eligible candidates
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
Staff Machine Learning Engineer
ServiceNow
Senior
Technology
Santa Clara, CA · On-site
$176k - $308k/yr
7 days ago

Senior AI Engineer (LLM / Agents)
CodeTalent
Mid
Technology
Gdańsk, PM, Poland · Remote
29k zł - 37k zł/yr
15 days ago

Senior AI Agent Engineer
Traba
Senior
Technology
New York City, NY · On-site
$200k - $240k/yr
8 days ago

Data Engineer (AI/ML)
UnitedHealth Group
Senior
Technology
Minnetonka, MN · On-site
$73k - $130k/yr
6 days ago

AI Engineer (Agentic AI / LLM)
DCV Technologies
Mid
Technology
Warsaw, Poland · Remote
N/A
5 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Software Engineer, AI Agents
Epiq
Senior
Technology
New York, NY, United States of America · On-site
$180k - $260k/yr
8 days ago
Senior Data Scientist
Margo
Senior
Technology
Warsaw, Poland · On-site
N/A
4 days ago

Senior AI Solutions Engineer with Python
EPAM Systems
Senior
Technology
Krakow, Poland · Remote
N/A
14 days ago
AI/ML Architect (Agentic AI Lead)
Relanto, Inc.
Senior
Technology
San Francisco, CA · Hybrid
N/A
32m ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights