Small deltas, big claims
Multi-agent coordination papers report benchmark gains of a few points as evidence that one architecture beats another. Almost nobody asks the prior question: how much do two runs that shouldn't differ, differ?
Two LLM agents work in parallel on the same multi-step itinerary. The first attempts to book a flight, discovers there is no inventory, abandons the path, and moves on. The second — started moments later by the same orchestrator — attempts the same booking through the same API and re-discovers the same dead end. The first agent's failure lived only in its private scratchpad.
This is a structural property of how multi-agent LLM systems coordinate today, and it is measured, not anecdotal: the MAST taxonomy attributes 36.94% of failures across 1,600+ production agent traces to inter-agent coordination breakdowns — the single largest category. Recent architectures address it with coordination channels (structured handoffs, shared memory, server-side state) and report small benchmark deltas as evidence one design beats another.
We set out to run exactly that kind of comparison — and the measurement forced a question we should have asked at the start: what is the run-to-run variance between protocols whose API inputs are equivalent? A single number turns out to be enough to invert the read of much of the recent coordination literature.
Before you claim your coordination mechanism added +6 pp, you must know that two mechanisms doing nothing at all can differ by +18 pp on one seed — and by −3 pp on the next.
Three protocols, one store, matched payloads
ET-MCP — a task-scoped negative-knowledge store conformant with MCP 2026-07-28 — is the substrate that makes the measurement possible, not the contribution. Agents write typed failure events; two reader-side designs decide who reads them, and when.
All runs use τ²-bench retail (state-validated passk on a customer-service tool-use domain),
with Claude Haiku 4.5 as both assistant and user simulator, temperature 0, a 20-turn budget,
n = 100 tasks × 2 trials per protocol. When any trial ends with reward < 1, the final
K = 3 tool calls are written to the task-scoped store as failed_trial_action events.
The three protocols differ only in the reader interface:
The store is written but never read. Pure baseline: whatever this arm scores is what the benchmark plus harness plus API produce with zero coordination.
The agent is the active reader: a curated peer-warnings summary is prepended to its system prompt at trial start. The agent pays decision overhead and an enlarged tool surface for control.
The framework intercepts every tool call; if (name, arguments) match a peer failure event, it
prepends a [PEER-WARNING] block to the tool response. No agent changes, no prompt changes —
the same hook surface already shipped in Strands, AgentCore Policy, LangChain middleware, and
Microsoft Agent Framework.
trace.query when peer state might matter.[PEER-WARNING] block is prepended to the response.Coordination didn't move the metric
At trial 1 — where the store is populated and coordination is logically active — all three protocols collapse to the same success rate: 0.54. And where the mechanism had a real chance to help, it directionally hurt.
The coordination-active subset: no detectable effect, mild interference
The pre-specified architectural test conditions on tasks where trial 0 failed — there, the store
necessarily contains peer events, so the mechanism is active. Pull and intercept tie exactly, and each
directionally underperforms no_coord at recovering from a prior failure. With 8–17 informative
pairs after ties, the test is underpowered for a medium effect — this is no detectable effect at the
available power, not a clean null.
| Contrast | subset n | t1 success | W / L / T | sign p |
|---|---|---|---|---|
| pull vs. intercept | 29 | 0.28 / 0.28 | 5 / 5 / 19 | 1.00 |
| pull vs. no_coord | 28 | 0.14 / 0.29 | 2 / 6 / 20 | 0.29 |
| intercept vs. no_coord | 35 | 0.20 / 0.34 | 6 / 11 / 18 | 0.33 |
The noise floor
At trial 0 every reader is a no-op — any paired gap between protocols is noise, not architecture. We measured that gap across two seeds, two models, and two domains.
The protocol caught its own artifact
At seed 1, pull-vs-intercept showed a +18 pp gap with Bonferroni-corrected p = 0.012 — the kind of number that gets published as a finding. We re-ran the full n = 100 sweep at a second seed: the gap became −3 pp, pcorr = 1.0. The sign flipped. We retracted the hypothesized pull-specific perturbation (E4) as an established source. Pooled over both seeds, no trial-0 contrast is significant after Bonferroni — at either seed or pooled.
What the trial-0 gap decomposes into: E1 byte-identical first requests (audited, holds); E2 within-protocol API stochasticity, bounded at ≤3 pp by a paired n=30 replicate — the Messages API exposes no seed, so T=0 is not bit-deterministic; E3 harness drift once the user-simulator emits its first sampled token; E4 the retracted structural perturbation. The envelope any coordination claim must clear is E1–E3 combined.
A single seed's pcorr = 0.012 on a contrast that pools to +7.5 pp (pcorr = 0.316) over two seeds is exactly the artifact a paired noise-floor protocol should catch — and ours did, on our own result.
Where published gains sit relative to the floor
Ten recent multi-agent coordination papers, each reporting a single-number headline delta. Seven fall below the clean-contrast pooled upper CI; one more sits inside the envelope.
| Paper | Benchmark / task | Reported effect | vs. floor |
|---|
Coordination-active passk
The minimum reporting protocol: passk restricted to trials where the coordination mechanism is logically active — the store is non-empty, so the mechanism had an opportunity to act.
Marginal passk answers “should I deploy the whole system?” Coordination-active passk answers “did coordination help when it had the chance?” The two diverge whenever empty-store trials carry unrelated configuration perturbations — which is exactly the situation we measure. The metric is causally pre-specifiable from protocol semantics: when the store is empty the mechanism is inactive, so any cross-protocol comparison on empty-store trials measures noise.
The Sonnet probe shows why this matters. At n = 30, pull posts a +23 pp marginal gain — but it started trial 0 at −16.7 pp behind the baseline, so much of that “gain” is recovery of an unrelated deficit. Marginal pass1 obscures the direction flip; the coordination-active metric makes it visible.
Runtime monitoring, shipped
Three alarms operationalize the metric on AgentCore / SageMaker Model Monitor deployments: (R1) CUSUM on coordination-active passk over a golden-task set, alarm at ±1.5σfloor; (R2) rolling-N 95th-percentile injection-event count, alarm at ≥50% above release baseline (right-skew defeats Gaussian 2σ); (R3) coordination-key collision rate, alarm at >0.5%.
Why naive coordination fails — and what to try next
Three candidate failure modes surface in the trial-level data, each matched to a refinement on hook surfaces that already exist in production frameworks. All three are preliminary — hypotheses for confirmatory work.
Writer mis-attribution
The causally relevant call is often upstream; a last-K writer records downstream symptoms instead of the root cause.
Causally-attributed writers
An LLM judge consumes the trajectory and reward and re-weights which calls get written, replacing the recency-only heuristic.
Per-match injection noise
The store accumulates events (mean 2.77 per trial, max 8) and every matching call fires a separate warning — repeated interruptions, not curated signal.
Selective intercept
A score gate fires only when a peer event is relevant, novel, and decision-proximal. Preliminary: +15.8 pp over vanilla intercept on the coordination-active subset (n=30, p=0.18 — directionally consistent, underpowered).
Brittle literal keying
Literal argument tuples don't transfer: cancel_order(#W123) doesn't match
cancel_order(#W456) even when both target shipped orders.
Predicate constraint extraction
An extractor turns the trajectory into a typed predicate — tool=cancel_order with
order.status=shipped — so intercept fires on the predicate, not the literal tuple.
What this paper does not claim
The negative reading applies to naive coordination under a measurement-honest protocol — not to coordination as a research program. Six limitations bound the claims.
L1Configuration-equivalent, not identity-replay
L2Writer was pilot-tuned
L3Two seeds, one within-protocol replicate
L4Single-judge M1 audit
L5Cross-model / cross-domain probes are n=30
L6Reader-side flows only
Read, reproduce, cite
All 600 trials are logged as JSONL artifacts; analysis is reproducible via the
analysis CLI in the repository. The harness and ET-MCP substrate ship together, open source.
The core insight here is simple and compelling: before claiming coordination helps, you need to know how much variance exists between runs that shouldn't differ. Nobody in the multi-agent coordination space has done this measurement, and it's overdue. […] especially the self-retraction of the seed-1 p=0.012 result when it didn't reproduce — that last bit is rare and commendable.
@inproceedings{kaliyev2026noisefloor,
title = {How Much Coordination Gain Is Real? A Paired Noise-Floor
Protocol for Multi-Agent {LLM} Benchmarks},
author = {Kaliyev, Alibek and Maryanskyy, Artem},
booktitle = {KDD 2026 Workshop on Agentic AI Evaluation
and Trustworthiness},
year = {2026},
url = {https://abekek.github.io/coordination-noise-floor-protocol/}
}