Evaluate AI Agents With Traces First—Then Build a Repeatable Test Set

To evaluate an AI agent, inspect complete runs before assigning a broad score. Follow the user request through tool calls, responses, handoffs, and the final outcome; record the first observable failure. Then define what a passing run must achieve and turn representative cases into tests you can rerun after a change. OpenAI’s agent evaluation guidance recommends traces for debugging workflow behavior and datasets with evaluation runs when repeatability is needed.

Keep the task as the unit of evaluation. A polished answer can conceal a skipped lookup or an action that never changed the underlying system. A different sequence of valid tool calls may still complete the task. Inspect the path to explain a failure, then grade the outcome and any required constraints to decide whether the run passes.

Inspect traces before writing a broad score

Start with successful runs, reported failures, and cases whose outcome is uncertain. For each, identify the request, the information available to the agent, each tool request and response, any handoff, and the final answer or changed state. Locate the first point where the run departed from an acceptable path. An incorrect final answer may be the last symptom of a much earlier tool or routing decision.

Record a compact review worksheet: task, expected outcome, observed outcome, first failing step, trace evidence, severity, and proposed regression test. Distinguish a wrong tool choice from correct tool selection with bad arguments. Mark tool errors that the agent mishandled and handoffs that dropped necessary context separately; each calls for a different fix.

Check whether the trace contains enough evidence to support your diagnosis. An apparent agent failure might come from stale test data or a broken tool fixture. Conversely, a grader may reject a valid alternative route because its expected sequence is too narrow. Label those cases as evaluation problems so they do not become misleading agent tests.

Define success a reviewer can observe

Write a pass condition for each task before choosing a grader. Specify the acceptable end state, any action that must occur or must be avoided, and the evidence needed to verify it. Require a particular tool or handoff only when the task or a policy genuinely demands it. Otherwise, allow different paths to the same valid result.

Separate outcome, process constraints, answer quality, and safety in the worksheet. Outcome might be a changed record or a correct answer grounded in retrieved material. Process constraints can cover authorization before an external action or a handoff to an agent with the needed permission. Quality covers completeness and clarity; safety covers prohibited actions and sensitive information.

Make each criterion testable against available evidence. “Used tools well” leaves reviewers to invent a standard; “looked up the current balance before stating it” identifies an observable event. Add an “insufficient evidence” verdict when the trace cannot show whether the underlying state changed. For a hypothetical refund task, a clear message to the customer passes an answer-quality check only if its claims are accurate; completion of the refund requires a separate state check.

Turn findings into a repeatable test set

Create a case for each distinct failure mode, then include ordinary tasks so the suite represents the work the agent actually does. Preserve the request, relevant context, starting state or tool fixtures, expected outcome, and grading rules. Include both sides of important decisions: cases where a tool or handoff is required and cases where it would be unnecessary or improper.

Make the test environment stable enough for comparisons to mean something. Reset mutable state between trials, control external responses where appropriate, and retain the model, prompt, tool, and grader versions used for each run. Keep previously fixed failures in a regression group and maintain a broader set for common tasks and difficult edge cases. Remove sensitive information from cases drawn from real incidents, and resolve ambiguous expectations before adding them.

Run the candidate and current baseline on the same cases. Compare outcomes by failure category rather than relying on one aggregate score: better prose should not conceal a new unauthorized action. Where results vary between attempts, use repeated trials and inspect the traces behind score changes. A score difference is useful only when the tasks, environment, and grading rules are comparable.

Match each criterion to a grader

Use deterministic checks for properties a system can verify directly: valid output structure, a required state change, a prohibited tool call, or a passing test. Use a narrowly defined model-judge rubric for open-ended properties such as whether an answer covers the request or is supported by the material the agent saw. Human reviewers should resolve ambiguous cases and check whether judge decisions match the product’s intended standard.

Anthropic’s evaluation guidance distinguishes code-based, model-based, and human graders; it warns that exact tool-call sequences can reject valid solutions and recommends calibrating model judges against expert judgment. Give a judge the relevant evidence and one clear dimension to assess. Allow an “unknown” verdict when that evidence is missing.

In the hypothetical refund task, a state check determines whether the refund was processed, a trace check verifies a required identity step, and a judge assesses the customer-facing explanation. These results should stay separate. A fluent claim that the refund is complete cannot substitute for the state change, while a completed refund does not excuse an inaccurate explanation or a missed authorization requirement.

Set release gates and monitor production runs

Turn the test set into a release decision. Require critical safety and authorization checks to pass, set thresholds for task outcomes and answer quality, and review newly failing cases before shipping. Track latency and cost beside quality so the trade-off is visible. Choose thresholds for the agent’s tasks and risks rather than adopting a universal pass rate.

Offline tests cover known cases; production traces expose requests and tool paths the set missed. Sample routine runs as well as runs associated with poor feedback or unusual behavior. LangSmith’s online evaluator documentation describes filtering and sampling production runs for judging, including traces with logged image, audio, or document content and attachments.

Use automated production judging once its rubric has been checked against human decisions and its sampling cost is understood. Review the traces behind alerts, distinguish agent failures from missing evidence or evaluator errors, and add confirmed new failure modes to the offline set. This keeps release tests grounded in observed behavior as the agent’s work changes.