RecursiveMAS: What Happens When AI Agents Stop Talking in Tokens

Every multi-agent system you have ever used is, at bottom, a group chat. A new paper from UIUC, Stanford, NVIDIA and MIT asks the question that only looks obvious after you've read it: why are we forcing neural networks to talk to each other in English?

Open the logs of any multi-agent framework and you will find the same scene. A planner writes three paragraphs of prose. A critic reads them, thinks it over, writes three paragraphs back. A solver wades through all of it and produces the answer. Token by token, the system decodes rich internal representations into text, then immediately re-encodes that text back into internal representations on the other side. Twice per hop. At every hop.

Text is a compression format designed for humans. For a language model, it is a bottleneck: the last hidden state of a transformer is a dense vector that carries far more information than the single token it gets collapsed into. When agent A verbalizes its reasoning for agent B, the system pays three times — compute to decode, information lost in the projection, and latency while B waits for A to finish generating.

Recursive Multi-Agent Systems — RecursiveMAS, published on arXiv in April 2026 by a team spanning UIUC, Stanford, NVIDIA and MIT — removes the group chat entirely. Agents pass each other their latent states directly, through a small trained adapter. No decoding, no re-reading, no tokens until the final answer. The result, across nine benchmarks: +8.3% average accuracy, 1.2×–2.4× faster inference, and 34.6%–75.6% fewer tokens than the same system communicating in text.

The paper is not just an efficiency trick. It reframes what a multi-agent system is: instead of a committee of models exchanging messages, the whole system becomes one recursive computation, where each agent behaves like a layer that the latent state flows through — again and again, until the answer is ready.

Why Text Is the Bottleneck

Every message between two text-based agents hides a wasteful round trip. The sender runs its full transformer stack, reaches a hidden state h of dimension d_h — and then projects it onto a vocabulary of size |V| to pick one token. It repeats this for every token of the message. The receiver then tokenizes the text and re-encodes it from scratch.

The paper quantifies the asymmetry. Generating m reasoning tokens costs m·|V|·d_h just in vocabulary-space decoding. Passing the equivalent reasoning as latent vectors costs m·d_h². Since the hidden dimension is orders of magnitude smaller than the vocabulary — a few thousand versus 100K+ — latent transfer wins by construction (Proposition 3.1 in the paper formalizes the full end-to-end complexity gap).

Compute is only half the argument. The projection from hidden state to token is lossy: one vector that encodes nuance, uncertainty and intermediate structure collapses into a single discrete symbol. Whatever did not fit into the chosen token is gone. Multiply that loss across every token, every message and every round of a multi-agent debate, and "agents collaborating in text" starts to look like scientists collaborating exclusively through tweets.

The RecursiveLink: A Tiny Bridge Between Minds

The obvious objection: you cannot just wire Llama's output into Qwen's input. Different models have different hidden dimensions, different embedding geometries, different "mental languages". A raw hidden state from one model is noise to another.

The paper's answer is the RecursiveLink, and it is deliberately tiny: two linear layers with a GELU in between, plus a residual connection. That is the whole module.

It comes in two variants:

• The inner link lives inside a single agent. As the model generates, its last-layer hidden state is mapped back into the input embedding space and fed into the next forward pass: R(h) = h + W₂ σ(W₁h). Instead of emitting a token, the agent keeps "thinking" in vectors — the paper calls each of these steps a latent thought.
• The outer link connects two different agents. An extra linear layer W₃ in the residual branch handles the dimension mismatch: R(h) = W₃h + W₂ σ(W₁h). This is what lets a Gemma planner hand its reasoning to a Llama critic, which hands it to a Qwen solver. Directly, no text in between.

The residual connection is the quiet design win. Because the input passes through mostly unchanged, the trainable branch only needs to learn the distributional shift between two embedding spaces — not a full projection from scratch. The paper's ablation backs this up: the residual two-layer design beats one-layer, no-residual and plain two-layer alternatives on every benchmark tested, reaching 88.0 on MATH500 versus 85.6 for the same module without the residual.

Chaining Agents into One Recursive Loop

With links in place, the architecture snaps together. The first agent reads the actual question — real input tokens — and generates a sequence of latent thoughts through its inner link. The outer link translates those thoughts into the next agent's embedding space, where they become conditioning input alongside that agent's own instructions. And so on down the chain.

Then comes the recursive part — the one that gives the paper its name. When the last agent finishes, its latent output — effectively the system's current draft of an answer — loops back to the first agent. Round two begins with everything the system concluded in round one, still in vector form. Each agent can reflect on the earlier outputs of the whole system and refine its contribution.

Only at the final recursion round does the last agent decode text: the answer you actually read.

This is a direct extension of an existing scaling axis. Recursive and looped language models — LoopLM, tiny recursive networks, latent-reasoning depth transformers — already showed that reusing the same layers over several rounds deepens reasoning without adding parameters. RecursiveMAS asks: if layers can recur inside one model, why can't agents recur inside one system? The paper is, by its own claim, the first attempt to extend recursive scaling to the system level.

Training the System, Not the Agents

Now the part that makes all of this workable outside a lab: no LLM is fine-tuned. Every agent's weights stay frozen, start to finish. The only trainable parameters in the entire system are the RecursiveLinks — about 13 million, against the several billion of the models they connect.

Training runs in two stages:

1. Inner loop — each agent, in parallel, warms up its inner link. The target: given the training question, the latent thoughts produced through the link should align with the embedding of the ground-truth answer. The loss is a cosine regression, L = 1 − cos(R(H), Emb(y)). After this stage, each frozen model can generate latent thoughts that stay in-distribution instead of drifting into noise.
2. Outer loop — the whole chain is unrolled for n recursion rounds, the final agent decodes a text answer, and a standard cross-entropy loss on that answer backpropagates through the entire recursive computation. Every outer link receives its share of credit in proportion to what it contributed to the final prediction. The system is optimized as a single organism.

The theory section explains why this works where naive approaches fail. Theorem 4.1 shows that applying text-based supervised fine-tuning across recursion rounds suffers from vanishing gradients — once tokens become confident, the gradient norm collapses toward zero and earlier rounds stop learning. The latent path through RecursiveLink keeps gradient norms near one across the loop. Differentiability is not a nice-to-have here; it is the reason the whole system can be trained end-to-end at all.

The Numbers

The evaluation covers nine benchmarks across mathematics (MATH500, AIME2025, AIME2026), science (GPQA-Diamond), medicine (MedQA), code generation (LiveCodeBench-v6, MBPP+) and search-based QA (HotpotQA, Bamboogle). The most telling comparison is against Recursive-TextMAS: the identical system — same agents, same collaboration structure, same recursion budget — with agents communicating through text instead of latent states. Any difference is attributable to the communication channel alone.

The gap widens as recursion deepens. At one round, latent communication improves accuracy by 3.4 points on average; at three rounds, by 7.2. Text-based recursion actually degrades past round two on several benchmarks — consistent with the gradient-vanishing analysis — while the latent system keeps climbing.

Efficiency moves in the same direction, which is the unusual part. Deeper recursion normally means paying more for more quality. Here the relative advantage grows with depth:

The mechanism is simple: Recursive-TextMAS re-decodes and re-reads intermediate text at every round, so its token bill compounds. RecursiveMAS only ever decodes once, at the very end. By round three the latent system uses 75.6% fewer tokens and answers 2.4× faster.

Against the broader field — fine-tuned single agents, Mixture-of-Agents, TextGrad, LoopLM — the whole-system comparison lands the same way:

The margins are largest exactly where multi-step reasoning matters most: +13.4 points on AIME2025 and +10.0 on AIME2026 over the strongest baseline.

There is also a genuine scaling law hiding in the results. Training with more recursion rounds and inferring with more recursion rounds improve performance independently and together — the best cell is always the corner where both are largest:

Four Patterns, Heterogeneous Models

If this only worked on one pipeline calibrated to the millimeter, it would be a lab curiosity. Instead, the paper instantiates RecursiveMAS on four common collaboration patterns, mixing model families and sizes with little ceremony:

The results hold everywhere. In mixture style, the recursive system beats the strongest individual domain specialist by 6.2% on average — evidence that latent collaboration composes knowledge across domains rather than just selecting the best expert. In distillation style, a small learner recovers most of a larger expert's capability while keeping a 1.5× speed advantage. In deliberation style, latent coordination improves a tool-calling agent by 4.8%.

The heterogeneity matters more than it may seem. The outer link's W₃ projection is what aligns a BioMistral's embedding geometry with a Qwen's — two models that were never trained to interoperate, produced by different labs, now exchanging thoughts through a two-layer adapter trained for a few dollars. Everything is public: code on GitHub under MIT license, checkpoints and training data on Hugging Face, with a demo playground on the project page.

Limits and Open Questions

The honest caveats, some from the paper and some mine.

You lose the transcript. Intermediate rounds are vectors, not text. There is no log of "what the critic said to the solver" to audit after the fact. You can decode any intermediate latent state on demand through the model's language head — but then you are paying the decoding cost you were trying to avoid, and an approximate verbalization of a latent thought is not the thought itself. For regulated or safety-critical deployments, unreadable inter-agent channels are a real concern, and the interpretability of latent communication is an open research problem, not a solved footnote.

Links are pairwise and trained. Every agent pair in the loop needs its outer link trained on data from the target domain. This is cheap — dollars, not GPU-weeks — but it is not plug-and-play: you cannot hot-swap a new model into the system without a training pass, the way you can with text-based frameworks where the interface is just prompts.

The evaluation lives where ground truth exists. Math, science QA, code: domains with verifiable answers, which the outer loop's cross-entropy needs like oxygen. Open-ended collaborative work — writing, design review, long-horizon agentic tasks with tools and side effects — is a different regime, and the paper does not claim it. The deliberation-style results with a tool-caller are encouraging but early.

Latent budgets saturate. The ablation shows accuracy climbing as agents are allowed more latent thoughts per turn, then flattening around 80 steps. That is remarkably few — one more hint that a lot of inter-agent token traffic in today's systems is redundant — but it also suggests latent channels are not a magic bandwidth upgrade; they are a more efficient encoding of a bounded amount of collaboration.

Practical Assessment

RecursiveMAS is best read as an existence proof with unusually good engineering: multi-agent collaboration does not need natural language as its internal protocol, and dropping it makes systems simultaneously cheaper, faster and more accurate — on benchmarks where correctness is checkable.

If you build multi-agent systems today, none of this replaces your stack tomorrow morning. Orchestration frameworks assume text at every boundary, and most production value still depends on traces you can inspect. But the direction is set. Inter-agent text is the internal bus of current systems, and this paper shows that bus can be swapped for something an order of magnitude more efficient — for the price of a coffee and a croissant in training compute. The economics will do the rest of the arguing.

The deeper shift is conceptual. We have spent two years treating "more agents" and "more reasoning tokens" as separate scaling levers. RecursiveMAS folds them into one: a multi-agent system as a single recursive model whose layers happen to be entire LLMs, trained end-to-end through a 13M-parameter seam. Whether or not this exact architecture wins, system-level recursion now has a scaling law, a training recipe, and public code.

The token era of agent communication is not over. But it just received its first credible eviction notice.

Sources: Recursive Multi-Agent Systems (arXiv 2604.25917), project page, official GitHub repository, released checkpoints and datasets on Hugging Face linked therein.