Why AI Workflows Must Be Discovered Through Evals

Since April 2026, the case for treating AI development as measured discovery has gained a concrete institutional example. Last updated on July 24, NIST’s AITE overview describes a new program, still in its initial phase, that tests models on blind data in a sequestered environment using common tasks, metrics, and scoring.

The practical conclusion is not that AI systems cannot be engineered. It is that an AI workflow begins as a hypothesis: teams must discover where a model, its instructions, tools, data, controls, and human operators produce an acceptable result together. That distinction matters because a polished demonstration does not establish reliability, productivity, or safety in the conditions where the system will actually operate.

Start with an outcome, not an agent diagram

The first design decision should be the result you need to observe. “Use AI for customer support” is too broad to test; “classify incoming requests into the correct queue while escalating uncertain cases” defines an outcome, a boundary, and a possible human handoff.

Before choosing a complex architecture, collect a small set of representative tasks and record the existing baseline. Depending on the workflow, that baseline might include completion time, correction rate, cost per completed case, policy violations, or the proportion of work that requires expert review. The useful comparison is not whether the model produces impressive text, but whether the proposed workflow improves a defined result without creating unacceptable failures elsewhere.

A conditional example makes the difference clearer. Suppose an operations team wants AI to turn meeting notes into assigned actions. Success is not a fluent summary; it is whether the correct owner, deadline, dependency, and source context reach the task system without invented commitments. Those observable conditions become the first evaluation cases.

Capability belongs to the workflow, not just the model

A model’s benchmark score does not describe the reliability of a deployed system. Production behavior also depends on the prompt, available context, retrieval quality, tool definitions, permissions, retry logic, memory, and the point at which a person takes over. Changing any one of those components can change both what the system can accomplish and how it fails.

This is especially important for agents, which act across multiple turns and can modify external state. Anthropic’s January 2026 evaluation guidance distinguishes the final outcome from the transcript that produced it and says an agent evaluation measures the model and its harness together. It also recommends multiple trials because outputs can vary between runs, with code-based, model-based, and human graders serving different purposes.

For a newcomer, the implication is simple: do not record only the final answer. Preserve the inputs, tool calls, retrieved material, intermediate decisions, final state, latency, and cost. A response claiming that a reservation was created is weaker evidence than checking whether the reservation actually exists in the relevant system.

Real work can contradict confident impressions

AI assistance can feel productive even when the measured result moves in the opposite direction. In a randomized study of 16 experienced open-source developers completing 246 real issues, participants expected AI to reduce their completion time by 24%; the observed result was that tasks took 19% longer when AI was allowed. The METR study of early-2025 tools explicitly limits that finding to its setting and does not claim that AI slows most developers or other kinds of work.

The study is valuable here because of that narrow scope, not despite it. It shows why a general belief about “AI productivity” cannot settle whether one configuration helps one team perform one class of tasks. Experience level, familiarity with the work, quality requirements, tool choice, and the definition of completion all affect the answer.

A public benchmark, vendor demonstration, or colleague’s success story can justify an experiment. It cannot replace a local comparison against the work you need done. The closer the test resembles the real environment, inputs, constraints, and acceptance process, the more useful its result becomes.

Use a small discovery loop

The goal of an early trial is to reduce uncertainty cheaply. A useful loop can be run without building an autonomous agent or committing an entire department to a new process.

Choose one bounded task. Prefer frequent work with a visible result, available examples, and a reversible failure. Avoid starting with a process whose success depends on many unrecorded judgments.
Write acceptance criteria. Specify what must be correct, what may vary, what must never happen, and when the system should abstain or escalate. Include quality, time, and cost where they affect the decision.
Build a representative task set. Include normal cases, difficult edge cases, incomplete inputs, and situations in which the correct action is to request clarification. Keep some cases unchanged so later versions can be compared against the same baseline.
Test the simplest configuration first. A single model call with structured output may be enough. Add retrieval, tools, memory, or autonomous planning only when the results show a specific limitation that the extra component could address.
Review failures by type. Separate instruction errors, missing context, tool failures, incorrect reasoning, policy breaches, and grading problems. Change one meaningful variable at a time, then rerun both the failed cases and the stable regression set.

This process discovers two things at once: where AI adds value and where conventional software or human judgment remains the better component. A deterministic validation rule, for example, should not be replaced by a probabilistic model merely because the larger workflow contains AI.

Promote evidence, not the best demonstration

A prototype is ready to advance when its result survives variation. It should work across representative cases and repeated trials, not only on a carefully selected prompt. The team should also know which failures are tolerable, which require an immediate stop, and who owns the decision when the evidence is ambiguous.

Before a limited deployment, define a release threshold and compare the candidate against the current process. Evaluate the complete outcome, including human correction time and downstream cleanup. If the AI step saves five minutes but creates ten minutes of review, its apparent speed is not an operational gain.

Keep the first deployment narrow enough to reverse. Limit permissions, preserve logs, require approval for consequential actions, and establish a fallback path. These controls do not prove that the system is good; they contain the cost of learning where it is not.

Discovery continues after launch

Passing a pre-release evaluation establishes performance only under the tested conditions. Production inputs change, users find new interaction patterns, external tools fail, and a model or prompt update can shift behavior. Monitoring therefore belongs to the workflow rather than being an optional dashboard added later.

Feed verified incidents and recurring corrections back into the evaluation set. Track results by task type instead of hiding uneven performance inside one average score. A system may be dependable for classification, costly for research, and unsafe for taking an irreversible action even when all three functions use the same underlying model.

The durable asset is not a diagram frozen at launch. It is a documented cycle that connects real tasks, measurable outcomes, controlled changes, regression tests, production evidence, and human accountability. That cycle turns experimentation from improvisation into engineering—and makes the useful AI system something a team can discover, verify, and maintain.

AI Workflows Aren’t Designed Once — Evals Reveal What Actually Works