AI Operations & Applied AI Research Developer

Quick Facts

• Role: AI Operations & Applied AI Research Developer

• Focus: technical design, feasibility validation, rapid POCs, and evaluation for human-centered AI

Description

Own the technical validation lifecycle—from partnering with product to turning pitches into feasible solutions and delivering rapid POCs. You will build and govern golden datasets and evaluations for training, testing, and benchmarking, then translate results into engineering handoffs and post-launch validation. You’ll also continuously research the state of AI to drive technical strategy and product roadmap recommendations.

Responsibilities

• Act as a technical expert with the Operations Product Manager to assess feasibility while adhering to responsible AI principles

• Curate and own golden datasets and evals for training, testing, and benchmarking

• Lead rapid POCs by creating a testing plan and measuring performance against baselines

• Own the technical handoff to engineering with reports, demos, and hands-on guidance

• Drive post-launch validation by benchmarking production performance vs POC results and, where applicable, human-level parity

• Research cutting-edge AI and translate insights into actionable recommendations for product roadmap and tooling

• Build POCs and automated workflows combining approved LLMs and multimodal models (text, image, video, audio) with tools and data (prompt design, structured outputs, retrieval, and failure handling)

• Design and run evaluations using ground-truth datasets, choose business-relevant metrics, and run controlled experiments across models, prompts, preprocessing, and scoring

• Turn results into clear recommendations, including honest reporting of limits and failure modes

Requirements

• Bachelor’s or Master’s degree in Computer Science, AI/ML, Data Science, or a related field (completed or expected by start date)

• Strong Python for data manipulation and analysis (pandas, SQL), plus REST APIs and structured data (JSON/XML)

• Hands-on experience building AI applications on LLM APIs (prompting, structured outputs, RAG, or tool-using workflows), including judgment on when to use LLMs vs classical ML vs deterministic approaches

• Working knowledge of ML evaluation (precision/recall/F1, building test sets, experiment design, validating against baselines and benchmarks)

• Understanding of AI guardrails: safety, privacy, compliance, and responsible deployment

• Proficiency with LLM-assisted coding tools to design, implement, and iterate on software

• Clear written and verbal communication, including technical documentation and explaining complex results to non-technical stakeholders

Benefits

• Lead end-to-end technical validation and evaluation of state-of-the-art, human-centered AI systems

• Own rapid POCs and multi-step LLM/multimodal workflows supporting high-quality content metadata and video/audio outputs

• Drive post-launch benchmarking to ensure production outcomes match original POC expectations