Apple Builds Agent Tests From MCP Specs—Argument Values Still Break

Apple researchers’ Agent Seer converts a Model Context Protocol specification into a proposed, execution-free evaluation harness: expected tool calls, synthetic outputs, multi-turn prompts, and a generated oracle for grading. Its central limitation is not usually choosing a tool name; it is assigning trustworthy values to that tool’s arguments.

That distinction sets the practical boundary. Schema-derived tests can broaden workflow coverage and expose regressions quickly, but teams should grade tool selection separately from argument correctness and keep live tests for state, permissions, side effects, and business rules. Apple’s Agent Seer summary describes a pipeline that needs no examples, live tool access, or domain-specific tuning, while identifying argument-value accuracy as its dominant failure mode.

What Agent Seer generates from a specification

Agent Seer uses the information already present in an MCP tool catalogue: function names, descriptions, and typed parameter schemas. Its first stage enriches those definitions with functional descriptions, parameter roles, likely use cases, and organizational context. The second stage generates simple or complex workflows, each with a user goal and an ordered sequence of expected calls.

The third stage fabricates a response for every expected call. Those mock outputs let the fourth stage split suitable workflows into multiple conversational turns, with later prompts referring to concrete values returned earlier. The resulting harness contains a prompt, expected calls with typed arguments, synthetic responses, a conversation, and an oracle withheld from the agent being evaluated.

Structured validation sits between the stages, so malformed artifacts can be rejected before they propagate. That protects syntax, not truth: an identifier can match the declared type while referring to no object, and a valid date can still violate a runtime rule. Agent Seer therefore creates plausible test fixtures from a specification; it does not establish that every generated fixture is semantically correct.

A reproducible schema-to-harness blueprint

The method can be reproduced without adopting the paper’s exact model stack. The important engineering choice is to preserve inspectable artifacts and validation boundaries instead of hiding generation and grading inside one prompt.

1. Parse and normalize the specification. Retain every tool name, description, required and optional field, type, format, enumeration, default, nested object, and explicit constraint. Store model-generated interpretations separately from facts copied from the schema.
2. Generate workflows and expected calls. Produce a user goal and an ordered trace containing the tool name and complete argument object for every call. Reject any tool absent from the supplied specification.
3. Create typed mock outputs. Generate a response for each call and record whether its shape comes from an output schema, an optional example, or model inference. Inference-only values need the most scrutiny.
4. Expand suitable workflows into turns. Split at natural phase boundaries. A follow-up should consume a field from an earlier mock response, rather than merely rephrase the initial request.
5. Grade independent dimensions. Compare the emitted trace with the held-out workflow, but retain separate results for tool necessity, selection, ordering, arguments, and conversational coherence.

This pipeline is especially useful when a new or private tool suite has no examples or execution environment. If documented examples or captured traces do exist, they can improve mock-output fidelity; their presence should be recorded rather than blended invisibly with inferred data.

Why successful tool selection is not enough

A function name identifies the intended operation, but its arguments determine the object, scope, time range, filter, or mutation. Selecting the right function with the wrong repository reference, channel, path, expiry, or record identifier can still produce the wrong action. It can also corrupt the synthetic oracle, causing a valid agent response to be penalized.

The full Agent Seer paper reports 337 scenarios and 391 evaluation records across seven MCP specifications. Tool selection was perfect in 77% of records, while argument scoring was perfect in only 42% and partial in 57%; when each failing record was assigned to its lowest-scoring argument subdimension, value accuracy accounted for 223 records. The same experiment achieved complete tool coverage for six specifications containing 14 to 56 tools, but only 56% coverage for the 64-tool Illustrator specification.

The grader separates argument completeness, name, value, type, format, and relevance. That decomposition reveals failures hidden by name matching. In one Redis pattern, the generated call selected the expected operation, key, and stored value but omitted an expiry implied by the scenario. In Git scenarios, overloaded optional parameters created value errors because the same field could carry different meanings across operations.

The reported figures measure the quality of Agent Seer’s generated scenarios and oracles, not the performance of an independently tested production agent. They also come from seven specifications, so the correlations between schema complexity and quality are findings within this sample rather than universal constants.

Build a scorecard around argument provenance

A useful scorecard should preserve both the aggregate result and the reason underneath it. Record whether a tool was necessary, whether the correct operation was selected, whether dependencies were ordered correctly, and whether every argument was complete, correctly named, well typed, properly formatted, relevant, and semantically accurate. Score context retention separately for later turns that reuse earlier outputs.

For values, literal equality is sometimes too strict and sometimes far too weak. Normalize equivalent date encodings, casing, paths, and unordered collections only where the API permits that equivalence. Conversely, do not accept a value merely because it is valid JSON or matches a primitive type: structural conformance cannot prove that an identifier exists or that a date satisfies a server rule.

Attach provenance to every expected value: user prompt, earlier mock response, explicit schema constraint, documented example, or model inference. This creates a review queue with a defensible priority order. Values inferred by the generator, reused across dependent calls, or attached to consequential mutations deserve review before schema-derived pass rates influence a release decision.

Where synthetic evaluation must stop

The paper describes Agent Seer as a proxy evaluation method and identifies two direct threats to its ground truth. The oracle is LLM-generated, and mock outputs for sequential calls are produced independently, so identifiers or other values may fail to align across dependencies. Multi-turn evidence is also limited because only a subset of generated scenarios expanded successfully.

Live execution remains necessary when correctness depends on server state, authorization, undocumented constraints, concurrency, external data, or side effects. The official MCP tools specification requires servers to validate tool inputs and distinguishes malformed protocol requests from execution errors such as invalid dates, out-of-range values, and business-logic failures. A synthetic harness cannot demonstrate that those runtime paths behave correctly.

The resulting evaluation stack has three defensible layers: broad schema-derived coverage, reviewed cases concentrated around risky argument values, and a smaller live end-to-end suite. Agent Seer can generate the first layer and help prioritize the second. Its own failure analysis explains why it cannot replace the third—and why counting successful tool selections gives an incomplete picture.

Also read:

• ChatGPT Still Leads as Gemini Takes More Than a Quarter of AI Web Traffic
• Curiosity’s Torn Wheels Still Roll After More Than 23 Miles on Mars
• Chanel CEO’s ChatGPT Demo Produced All-Male Leaders; Later Tests Still Skewed Male