Edition 35: After four tool calls, every unreachable-task run failed

A tool-using assistant can sound most confident at the exact moment it should stop.

Consider one ordinary support request:

Publish a reply to ticket INC-101.

The assistant can read the ticket and draft a reply. It has no publishing tool. The useful finished output is not “Done.” It is:

BLOCKED: I can read the ticket and create a draft, but no available tool can publish it. Add a publishing tool or change the request to “draft a reply.”

That small distinction is the center of a new benchmark. On November 5, the authors of MCPHallu reported 5,012 agent trajectories across 358 tasks and 36 real MCP server implementations running in isolated containers. “Unreachable Goal” was the hardest task type. Only 23.2% of those runs passed.

The sharper result appears when the authors group the runs by tool-call count. Runs with no tool calls passed 65.0% of the time. Runs with one to three calls passed 27.9%. Once an agent made four or more calls, none of the 667 runs in those groups passed.

This does not establish a universal four-call law. The tasks were deliberately constructed to expose specific failure risks, and a longer trajectory may be a symptom rather than the cause. It does show why a useful agent needs an explicit way to say, “The requested end state cannot be reached with the tools and permissions I have.”

MCP gives an agent tools, not proof that the goal is possible

The Model Context Protocol, or MCP, is a standard interface through which an assistant can discover and call external tools. A server might offer tools to search a database, update a ticket, write a file or query a market-data service.

A tool list answers “What can I call?” It does not automatically answer “Can this combination of tools produce the user’s requested end state?”

That second question is a feasibility check. Before acting, the agent should identify the observable result, map it to required capabilities, compare those capabilities with the available tools and permissions, and stop when a required link is missing.

A person can do this manually for a one-off task. Read the tool descriptions, list the required actions and compare the two lists. The structured gate becomes useful when requests repeat, tool catalogs change, permissions differ by user or a false completion could hide a missing write.

The research question for this edition is: Can a small feasibility gate prevent an assistant from turning a missing capability or refused tool call into a false claim of completion?

The paper found a stopping problem, not one universal model ranking

MCPHallu covers general search, analytics, productivity, finance and coding. The authors evaluated 14 models from eight vendors on four designed failure types:

• Branch Collapse: intermediate work is plausible, but the final aggregation is wrong.
• Unreachable Goal: no available tool path can satisfy the request.
• Tool Misuse: a tempting but incorrect tool is available beside the correct one.
• Context Forgetting: an earlier fact is lost before the task finishes.

The average score across models was 0.642 on a zero-to-one scale. The best overall score was 0.728. More useful than that leaderboard is the variation underneath it. A model’s average rank shifted by 6.9 of 13 positions across the four failure types. Branch Collapse and Unreachable Goal scores were nearly uncorrelated across models, with r = -0.12.

So “best agent” is too broad for an application owner choosing a system. A support workflow that must recognize an impossible write needs an Unreachable Goal test. A research workflow that combines many sources needs a Branch Collapse test. An assistant that carries facts across steps needs a Context Forgetting test.

The paper’s task construction also deserves attention. Tool schemas came from the server implementations. An LLM generated candidate tasks, then automated and human validation removed broken or ambiguous cases. Deterministic checks supplied most of each score, while an LLM judge contributed 20% to 30%, depending on the subtype. This is stronger evidence than asking a judge whether a final answer “looks right,” but it still leaves judgment and benchmark-design choices in the result.

Two public failures show why the distinction matters

The benchmark is new, but the reader problem is not hypothetical.

On July 14, a contributor reported that Microsoft’s Go agent framework did not inspect MCP’s isError field. A server refused a write because the OAuth client had read-only scope, but the wrapper returned no ordinary error. The agent then reported that it had published even though nothing was written. The public issue was closed after the failure path was addressed.

On October 1, a LangGraph issue described a headless tool result that received HTTP 200 but was silently discarded during a resume race. The client believed submission had succeeded while the thread remained interrupted.

These are two dated, independently reported implementation problems. They demonstrate demand for observable tool failure handling. They do not establish incident frequency, market size or search volume. Keyword volume, traffic and ranking metrics are NOT_AVAILABLE for this edition.

The current MCP tools specification explains an easy-to-miss detail: a tool execution failure can arrive inside a successful JSON-RPC response with isError: true. The official Python SDK troubleshooting guide says call_tool does not raise an exception for that case. Client code must inspect the result.

A feasibility gate therefore needs two checks:

1. Is the required capability present before the first call?
2. Did every probe or action return evidence of success, including isError: false and any expected state change?

Build a preflight card before the agent acts

Use this seven-step procedure for a workflow that can write, publish, send, close, delete or spend.

1. Write the end state. Replace “handle the ticket” with “a draft exists” or “reply INC-101 is published.”
2. Name each required capability. Publishing may require read_ticket, draft_reply and publish_reply.
3. Read the live tool inventory. Tool names alone are insufficient. Inspect descriptions, required arguments, return fields and permission requirements.
4. Compare required and available capabilities. If one is absent, stop before calling adjacent tools. Drafting is not publishing.
5. Probe only when feasibility depends on runtime state. A permission check or read-only lookup may reveal whether the named object exists. Keep this probe narrow.
6. Treat tool-result errors as failed evidence. Check isError, structured status fields and the external state that should have changed.
7. Build the final answer from evidence. Say “completed” only when the requested end state has been verified. Otherwise name the missing capability, failed permission or unresolved state.

The finished artifact can be a small card:

This card is useful even if the assistant itself never sees it. A product owner can use it to design tests, permissions and completion messages.

Do not turn the paper’s call count into a magic timeout

The benchmark’s zero-pass result after four calls is striking, but “stop every agent after three calls” would be a poor general rule. Feasible tasks can legitimately require long sequences. The paper also reports that blanket refusal performs badly on tasks where the correct tool path exists.

Use a call budget as an escalation signal instead:

• If the required capability is missing, stop immediately.
• If a narrow probe returns a definitive denial, stop and report it.
• If calls repeat without producing new evidence, pause and re-evaluate feasibility.
• If the task is feasible and each call advances a verified state, continue within the application’s ordinary limits.

This keeps refusal calibrated. The goal is not a timid agent. It is an agent that distinguishes “hard but possible” from “impossible with the current contract.”

Optional clean-folder feasibility check

This credentials-free exercise implements the two checks with three synthetic cases. It does not run a model, connect to MCP servers or reproduce the paper’s benchmark. It is a deterministic boundary test for the decision rule.

Prerequisites: Python 3.10 or later and an empty folder. Save the complete file as feasibility_gate.py.

cases = [
    {
        "name": "possible_draft",
        "required": {"read_ticket", "draft_reply"},
        "available": {"read_ticket", "draft_reply"},
        "probe_error": None,
    },
    {
        "name": "missing_publish",
        "required": {"read_ticket", "publish_reply"},
        "available": {"read_ticket", "draft_reply"},
        "probe_error": None,
    },
    {
        "name": "denied_update",
        "required": {"read_ticket", "update_ticket"},
        "available": {"read_ticket", "update_ticket"},
        "probe_error": "insufficient_scope",
    },
]

runnable = 0
stopped = 0

for case in cases:
    missing = sorted(case["required"] - case["available"])
    if missing:
        decision = "STOP"
        evidence = "missing:" + ",".join(missing)
        stopped += 1
    elif case["probe_error"]:
        decision = "STOP"
        evidence = "tool_error:" + case["probe_error"]
        stopped += 1
    else:
        decision = "RUN"
        evidence = "capabilities_present"
        runnable += 1
    print(f"CASE {case['name']} decision={decision} evidence={evidence}")

print(f"GATE runnable={runnable} stopped={stopped} total={len(cases)}")

Run:

python3 -I feasibility_gate.py

The complete verified output is:

CASE possible_draft decision=RUN evidence=capabilities_present
CASE missing_publish decision=STOP evidence=missing:publish_reply
CASE denied_update decision=STOP evidence=tool_error:insufficient_scope
GATE runnable=1 stopped=2 total=3

The second case is the negative capability test. The assistant can read and draft, but publishing is unreachable. The third case shows why a listed tool is not enough: the capability exists, yet the probe records insufficient permission.

If the output order changes after adding capabilities, keep the sorted call so missing items remain deterministic. If a real MCP client reports success through HTTP status alone, inspect the tool result’s isError field. If a provider returns a success flag but the external object did not change, add a read-after-write state check before allowing completion language.

To adapt the script, replace the synthetic capability names with a sanitized inventory from your application. Keep destructive tools out of this local exercise. For a real integration test, use an isolated test account and reset state between cases.

What this evidence does and does not prove

MCPHallu is a paper benchmark built from real server implementations, adversarial tasks and isolated execution. Its results are not production failure rates, customer outcomes or evidence that one prompt will repair every framework. Sixteen of the 36 server setups require credentials that the benchmark does not distribute, and the authors note that a future response-replay layer is needed for fully offline execution.

The local script proves only that the stated capability and tool-error rules produce the displayed deterministic decisions. It does not measure model judgment, MCP transport behavior or a deployed agent.

A completed implementation has four pieces: a named end state, a required-capability list, an explicit stop decision for missing or denied capabilities, and a completion claim backed by a state check. The next experiment should run matched feasible and unreachable requests through one real MCP client in an isolated account. Record the tool inventory, exact model, prompts, calls, isError values, final claims and external state. That would test whether the gate changes agent behavior without confusing a scripted exercise with model evidence.