Data Scientist, Agent Evaluations & Quality
About this role
About the Role
This role sits at the intersection of applied data science and AI product quality for a small, fast-moving AI productivity startup building autonomous agents that handle email, calendar, browser, and business software tasks. You will own the measurement of agent quality end-to-end: turning ambiguous product behavior into rigorous, actionable evaluation systems that directly guide engineering and product decisions.
What You'll Do
Architect and maintain automated evaluation pipelines that measure agent quality across capabilities and product surfaces.
Translate agent capabilities into explicit success criteria, including pass, partial-pass, and failure definitions for complex multi-step tasks.
Build representative gold datasets and regression suites covering common workflows, edge cases, ambiguous requests, and adversarial scenarios.
Define and track metrics such as task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability.
Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and measure grader agreement, false positives, and false negatives.
Analyze traces, tool calls, model outputs, and production outcomes to identify root causes and build a useful failure taxonomy.
Compare models, prompts, tools, and capability implementations using rigorous offline experiments and production evidence.
Build dashboards and release-quality signals that make evaluation results understandable and actionable for engineering, product, and leadership.
Partner with capability engineers to recommend improvements and verify that fixes raise quality without unacceptable regressions in cost, latency, or reliability.
What We're Looking For
5+ years in data science, machine learning, or analytics roles, with a focus on evaluation systems, metrics frameworks, or quality measurement for production systems.
Demonstrated experience designing and implementing evaluation frameworks, grading systems, and success criteria for ML or AI systems in production.
Strong Python and SQL proficiency with the ability to build automated data pipelines and production-quality analysis code at scale.
Solid statistical and experimental design knowledge: sampling, variance, uncertainty quantification, bias detection, confounding variables, and significance testing for non-deterministic systems.
Experience with ground-truth data development: labeling guideline design, annotation quality control, ambiguity resolution, and dataset maintenance.
Working knowledge of LLM behavior, tool use, retrieval systems, multi-step execution, and practical failure modes of language model systems.
Ability to connect quantitative patterns to individual system traces and identify failure origins across model, prompt, context, tools, data, and application logic.
Experience communicating evaluation results, methodology, uncertainty, and trade-offs to both technical and non-technical stakeholders.
Comfort operating with high ownership in ambiguous, fast-moving environments, independently turning open-ended quality questions into evaluation systems.
Experience with LLM-as-a-judge systems, agentic or multi-step task evaluation, or benchmarking platforms for AI systems is a strong plus.
Location
On-site in Palo Alto, California, United States. Visa sponsorship is not available for this role.
Originally posted on Himalayas
