Why Effective AI Systems Need Explicit Controls

Treating an LLM as a colleague remains a design mistake. The practical update is sharper: current production guidance defines an agent as a system assembled from a model, tools, instructions, guardrails, and human handoffs—not a personality whose judgment can be trusted implicitly.

That distinction matters because fluent replies can conceal missing context, stale evidence, or an unsafe action path. Effective systems supply the relevant information, constrain available actions, store durable state outside the conversation, measure outcomes, and stop or escalate when risk crosses a defined boundary.

Replace personality assumptions with an operating contract

Words such as “knows,” “remembers,” and “understands” are convenient shorthand, but they are poor engineering specifications. A team cannot test whether an assistant “understands the customer.” It can test whether the system retrieves the correct account record, applies the current policy, identifies missing fields, and produces an answer that satisfies defined criteria.

Start by writing an operating contract for each task. It should state the permitted inputs, required evidence, available tools, expected output, failure conditions, and actions that need approval. This turns an ambiguous role—“act like an experienced support representative”—into observable behavior that can be evaluated and revised.

The current OpenAI agent-building guide identifies the model, tools, and instructions as foundational components, then recommends performance baselines, layered guardrails, and human intervention for failure thresholds or high-risk actions. The important unit of design is therefore the complete execution loop, not the model response in isolation.

Put memory and truth in systems you can inspect

A conversational history is not the same thing as an authoritative record. Even when a model can process a long context, the application still has to decide which records to retrieve, which version is current, what may be retained, and whether the model is allowed to write anything back.

Separate at least three kinds of state. Conversation state supports the immediate interaction. Operational state records durable facts such as ticket status, approvals, and completed actions. Knowledge state contains policies, product documentation, or other material the system may cite. Each needs an owner, an update path, access controls, and a rule for resolving conflicts.

This architecture prevents a common category error: asking the model to remember a correction when the correction belongs in a database, policy repository, or versioned instruction set. If a user changes a delivery address, for example, a successful response is not evidence that the account was updated. The workflow must call the authorized system, verify the result, record the event, and tell the user what actually happened.

Use the least autonomous pattern that fits the task

Not every LLM feature should become an agent. Classification, extraction, drafting, or answering from a controlled document set may need one model call plus deterministic validation. A fixed sequence is preferable when the steps are known in advance; flexible agent control becomes useful when the route cannot be reliably hard-coded.

Anthropic’s engineering guidance distinguishes predefined workflows from systems in which the model directs its own process and tool use. It recommends beginning with the simplest workable design, noting that greater autonomy can add latency, cost, and compounding errors; it also calls for environmental feedback, stopping conditions, sandbox testing, and appropriate guardrails.

A useful autonomy decision has four parts:

Can success be verified from an external result rather than the model’s own explanation?
Are actions reversible, limited in scope, and attributable to a specific run?
Can the system detect missing information or repeated failure and stop?
Is a person available for exceptions whose cost exceeds the automation benefit?

If those conditions are absent, restrict the model to proposing or drafting. Autonomy should follow evidence that the surrounding controls work; it should not be granted because the assistant sounds confident.

Design tools as narrow permissions, not digital abilities

A tool is both a capability and a permission boundary. “Access to email” is too broad to be a safe specification. Reading a message, searching a mailbox, preparing a draft, and sending to an external recipient carry different consequences and should be represented by distinct operations where practical.

Tool definitions should describe required parameters, valid outputs, side effects, authorization requirements, and recoverable errors. The application should validate arguments before execution and return structured results afterward. For consequential operations, use idempotency controls, logs, spending or volume limits, and an approval step close to the action itself.

This also clarifies human responsibility. People should not be asked to supervise every harmless generation, but they should retain control over irreversible, sensitive, or high-impact actions. Escalation is part of the normal workflow, not proof that the AI has failed to become sufficiently human.

Evaluate the workflow, including its failures

A polished demonstration usually exercises a clean path. Production evaluation must include incomplete requests, conflicting documents, unavailable tools, malicious instructions inside retrieved content, expired permissions, and cases where the correct result is to refuse or ask for help.

Build a test set from real task categories and known failure modes, with sensitive data removed or handled under the organization’s controls. Score the properties that matter to the workflow: evidence selection, factual accuracy, tool choice, argument validity, policy compliance, action completion, escalation behavior, latency, and cost. Averages alone can hide unacceptable failures, so establish release thresholds for critical categories.

The NIST Generative AI Profile, published on July 26, 2024 and maintained as a voluntary cross-sector companion to the AI Risk Management Framework, places trustworthiness considerations across the design, development, use, and evaluation of AI systems. That lifecycle view is more useful than treating safety as a final prompt added before launch.

A practical build sequence

The system can be developed in a controlled progression without pretending that better prose equals greater reliability:

Define one bounded outcome and the evidence that proves completion.
Establish a baseline with the simplest model call or deterministic workflow that could work.
Add retrieval only for information the task genuinely requires, with ownership and version rules.
Expose narrow, documented tools and validate every proposed action.
Create tests for normal cases, edge cases, refusals, and tool failures before expanding autonomy.
Add explicit budgets for turns, time, cost, and repeated attempts.
Require approval for sensitive or irreversible actions and preserve an audit trail.
Monitor production failures, turn them into evaluation cases, and reassess whether each autonomous step still earns its complexity.

The durable principle is simple: do not assign human qualities where the product needs testable controls. Let the model handle language, pattern recognition, and flexible selection where those capabilities help; let software own state, permissions, validation, and records. The result is not a digital employee. It is a system whose behavior can be observed, limited, and improved.

Treating AI Like a Colleague Hides the Controls Your System Needs