Second post in the reasoning series, and a deliberate change of register from the first. Landscape of Thoughts asked where does reasoning actually go by looking at trajectories empirically. This paper asks the question one level down: what can a transformer that emits chain-of-thought tokens compute at all, as a matter of circuit complexity — independent of any particular model, dataset, or training run. It’s the theoretical floor and ceiling that every empirical CoT result in this series will implicitly sit inside.

0. The Picture in One Paragraph

Without chain of thought, a transformer that has to answer immediately after reading its input is provably stuck inside TC0 — the class of problems solvable by constant-depth, polynomial-size threshold circuits. That’s a real ceiling: TC0 can’t even reliably do things like check whether two nodes in a directed graph are connected, because that kind of problem is inherently serial — each step depends on the result of the previous one, and a constant-depth circuit has no way to unroll an unbounded number of sequential steps. Chain of thought changes the computational model itself: each generated token is fed back in and attended to, so the transformer effectively gets one more “layer” of computation per CoT token. The paper’s central result is that this isn’t a vague intuition — it’s an exact trade curve. Zero or constant CoT steps: still TC0. Logarithmically many steps: only a marginal gain. Linearly many steps: the transformer can recognize every regular language. Polynomially many steps: the transformer recognizes exactly P, no more and no less. Length of chain of thought isn’t a knob you turn for style — it’s the resource that literally determines which complexity class you’re computing in.

1. The Problem — Why “TC0” Is the Right Question to Ask

Merrill and Sabharwal’s own prior work had already pinned down what a transformer without CoT can do: under standard assumptions (log-precision, poly-size), a transformer’s forward pass is simulable by a uniform TC0 circuit family. TC0 is a narrow class — it’s below NL (nondeterministic log-space, the class containing graph reachability) and believed to be strictly below P. So there’s a concrete, well-known list of problems a no-CoT transformer cannot solve at scale, no matter how it’s trained: directed graph connectivity, simulating a finite-state automaton, evaluating a boolean formula with unbounded nesting. These aren’t exotic edge cases — they’re the textbook shape of “the next step depends on everything before it,” which is exactly the shape of a lot of multi-step reasoning.

Empirically, chain-of-thought prompting was already known to help transformers with exactly this flavor of task. The open theoretical question was whether that’s a real expansion of computational power or just a more favorable way of asking the same TC0-bounded model for an answer. The paper’s answer is unambiguous: it’s a real expansion, and its size is governed almost entirely by how many CoT tokens you allow.

2. The Result — A Trade Curve, Not a Threshold

The framing is circuit complexity throughout: a “decoding step” is one generated CoT token, fed back into the transformer before the next step. The paper walks the number of allowed decoding steps up from constant to polynomial and tracks what complexity class falls out at each point.

2.1 Constant and logarithmic steps — barely more than nothing

A constant number of CoT steps doesn’t change the class at all — still TC0, since a constant number of extra “layers” doesn’t help a constant-depth circuit escape its own definition. Allowing O(log n) steps (n = input length) buys only a modest amount of extra power, still well short of what’s needed for NL-complete problems like graph connectivity. This is the paper’s most counter-intuitive finding for anyone who assumes “more CoT tokens = proportionally more reasoning power”: logarithmically many tokens is a rounding error, computationally speaking.

2.2 Linear steps — all regular languages

Once the number of decoding steps scales linearly with input length (and with a mild architectural condition the paper calls “projected pre-norm”), the transformer can recognize every regular language — i.e., simulate an arbitrary finite-state automaton step by step, writing its running state into the CoT and reading it back. This is the point where the earlier example — “can a transformer track state across an unboundedly long sequence” — flips from impossible to solvable.

2.3 Polynomial steps — exactly P

With polynomially many decoding steps and a further generalization of pre-norm, the paper proves the transformer’s expressive power is exactly P — not an upper bound, an exact characterization, in both directions: anything in P can be computed by some CoT transformer with polynomially many steps, and no CoT transformer with a polynomial step budget can compute more than P. The general form of the argument is almost a simulation lemma: any problem decidable in time t(n) can be solved by a transformer using roughly t(n) chain-of-thought tokens, because each CoT step can simulate one step of a general computation. The paper calls this the first exact characterization of a transformer variant in terms of a standard, unconditional complexity class — most prior transformer-expressivity results are one-sided (upper bounds only).

2.4 The example problems that carry the intuition

Three recurring examples do the work of making this concrete: s-t reachability in a directed graph (NL-complete, needs more than log-length CoT), simulating a finite-state automaton (the canonical regular-language task that linear-length CoT unlocks), and composing a sequence of permutations (an inherently serial, non-parallelizable task used to illustrate why constant-depth circuits struggle with “chained” operations even when each individual operation is trivial).

3. Why It Matters

For a blog whose reasoning series so far has been almost entirely architectural — looped transformers, latent iteration, adaptive compute — this paper is the piece that explains why those architectures are chasing the effect they’re chasing. Think-at-Hard, LoopFormer, and Huginn all spend extra computation per token to approximate what unrolled, serial reasoning buys you; this paper says precisely what that serial computation is worth, in classical complexity terms, when it’s spent as explicit CoT tokens instead of implicit latent iterations. It also gives a principled reading of a purely empirical observation that recurs across the CoT literature: short reasoning traces plateau on genuinely serial problems not because the model “isn’t trying hard enough,” but because a chain of thought that’s too short is, provably, still stuck below the complexity class the problem lives in. If a task is NL-hard-flavored, no amount of prompting cleverness inside a logarithmic-length CoT budget will get a TC0-bounded model there — the length of the chain has to actually grow with the problem.

4. Limitations Worth Knowing

This is a computability result, not a learnability result. The paper shows what a transformer of appropriate size and weights could compute given enough CoT steps — it says nothing about whether gradient descent on realistic pretraining data would ever find those weights, or how much CoT-annotated data that would take. The polynomial-steps-equal-P result also depends on a specific architectural assumption (“generalized pre-norm”) rather than holding for every transformer variant unconditionally, so the exact correspondence is a statement about a well-defined model family, not a claim that any transformer with enough CoT tokens automatically reaches P. And “chain of thought” here means any generated intermediate token sequence used as scratch space — the paper is not making claims about whether human-readable, semantically coherent reasoning traces (the kind actually produced by CoT-prompted LLMs) achieve these bounds in practice, only that some sequence of tokens of the given length could.

5. The Takeaway for a First Reader

Chain-of-thought length is not a stylistic hyperparameter — it is, provably, the resource that determines which complexity class a transformer’s answer can come from. Constant or logarithmic CoT barely moves the needle past TC0; it takes linear CoT to reach all regular languages and polynomial CoT to reach all of P. Every subsequent paper in this series that tries to make reasoning cheaper by shortening or compressing the chain of thought is implicitly trading against this curve, and it’s worth having Merrill & Sabharwal’s exact version of the trade-off in mind as a reference point for what “shorter” is actually giving up.

References

• Merrill, W., & Sabharwal, A. (2024). The Expressive Power of Transformers with Chain of Thought. ICLR 2024. arXiv:2310.07923
• Related, complementary ICLR 2024 result: Li, Z., et al. Chain of Thought Empowers Transformers to Solve Inherently Serial Problems. arXiv:2402.12875
• This blog: Landscape of Thoughts — the empirical companion to this post’s theory, first in the reasoning series.
• This blog: Think-at-Hard, LoopFormer, recurrent-depth / Huginn — architectures that spend extra per-token computation to approximate the effect this paper attributes to explicit CoT length.

This is the first post in what will be an ongoing series of reasoning paper reviews on this blog, tagged reasoning. It felt right to start with a paper that isn’t proposing a new reasoning method, but a way to actually look at what existing reasoning methods are doing — a lens that every later review in this series can implicitly use.

0. The Picture in One Paragraph

Chain-of-thought reasoning is evaluated almost exclusively by its final-answer accuracy — a single scalar that says nothing about how the model got there, or where it went wrong along the way. Landscape of Thoughts (LoT) opens up that black box with a simple but under-explored idea: take every intermediate reasoning state in a trajectory (each partial chain-of-thought so far), compute how close that state is to each candidate answer using perplexity as a distance metric (asking the same LLM how likely it is to complete the trajectory with each answer choice), and project all these per-state distance vectors into 2D via t-SNE. The result is a literal landscape: the answer choices sit as fixed points, and a reasoning trajectory becomes a path that either converges toward the correct answer, drifts, or wanders. This single visualization — backed by three quantitative metrics (perplexity, uncertainty, consistency) — turns out to reliably distinguish strong models from weak ones, correct trajectories from incorrect ones, and can even be repurposed into a lightweight verifier that improves test-time scaling.

1. The Problem — Accuracy Tells You Nothing About the Path

Reasoning research has produced a large zoo of prompting and search strategies — chain-of-thought, self-consistency, tree-of-thoughts, MCTS-guided search — and the standard way to compare them is a leaderboard of final-answer accuracy on math and QA benchmarks. This is a reasonable summary statistic, but it collapses everything interesting about how a model reasons into one number:

• Two models can reach the same accuracy while one converges confidently and the other gets there by essentially guessing late.
• A single model can look identical on aggregate accuracy across two datasets while behaving completely differently at the trajectory level.
• Failure analysis usually means reading individual chain-of-thought transcripts by hand — it doesn’t scale, and it’s hard to turn into a quantitative signal.

LoT’s premise is that the sequence of intermediate states a reasoning trajectory passes through is itself informative, and that information is currently thrown away by accuracy-only evaluation.

2. The Method — From Text States to a 2D Landscape

2.1 Sampling and segmenting trajectories

For a given multi-choice question, a reasoning method (chain-of-thought being the base case, though the tool is designed to work with its derivatives too) is used to sample a reasoning trajectory from an LLM. That trajectory is segmented into individual thoughts — the intermediate reasoning steps — so a single trajectory becomes a sequence of partial states: state after thought 1, state after thought 1+2, and so on up to the final answer.

2.2 Turning each state into a distance vector

The core trick: for each intermediate state, LoT asks the same LLM that generated the trajectory to estimate how well that partial reasoning path would lead into each candidate answer, using perplexity as the distance metric. This produces, for every state, a vector of distances — one number per answer choice — without needing a separate reward model or verifier. A state’s distance vector is a self-consistent measure of “given what’s been reasoned so far, how compatible is this with each possible answer.”

2.3 Projecting into a landscape

These distance vectors across all states in all sampled trajectories are projected into 2D with t-SNE, with the answer choices treated as fixed anchor points. A single trajectory then reads as a literal path across the plot: it starts somewhere ambiguous and either converges toward the correct-answer anchor, converges toward a wrong one, or fails to converge at all — visually distinguishable at a glance, and aggregatable across many trajectories to compare models, datasets, or reasoning methods side by side.

2.4 Three quantitative metrics

The visualization is backed by three scalar metrics that don’t require looking at a plot at all:

• Perplexity — the raw distance-to-answer-choice signal itself, usable to estimate how compatible a state is with each candidate across different thought lengths.
• Uncertainty — how confident the model is about its prediction at intermediate steps (not just at the final answer).
• Consistency — roughly, whether the model already “knows” the eventual answer early in the trajectory, rather than only converging on it very late (or not at all).

3. Findings

3.1 Scale changes the shape of the landscape, not just the accuracy

Using the Llama-3.1 family swept from 1B to 70B parameters with CoT prompting on AQuA, the landscape visibly changes shape with scale: larger models converge faster toward the correct-answer region, with higher density of late-trajectory states clustered near the correct answer — consistent with, and visually explaining, their higher accuracy. Quantitatively, larger models show higher consistency, lower uncertainty, and lower perplexity — the three metrics move together with scale, not just the final accuracy number.

3.2 The landscape separates correct from incorrect trajectories

Across models and datasets, LoT reliably distinguishes correct trajectories from incorrect ones — correct trajectories show a different convergence pattern (faster, more monotonic movement toward the right answer) than incorrect ones. This holds not just as a qualitative visual pattern but as a quantifiable signal: convergence speed itself predicts whether a trajectory will land on the correct answer, distinct between success and failure cases.

3.3 It surfaces failure modes accuracy alone can’t show

Because the tool exposes per-state uncertainty and consistency, it can flag reasoning patterns that look fine in the final answer but are fragile underneath — low consistency (the model doesn’t settle on an answer until very late, essentially getting there by chance) and high uncertainty (the model never becomes confident even when it ends up correct). These are exactly the patterns that aggregate accuracy metrics cannot distinguish from confident, well-grounded correct reasoning.

3.4 From visualization to verifier

The same per-state distance/convergence features that make the landscape interpretable are also useful as a standalone signal: the authors adapt LoT into a lightweight verifier that scores trajectory correctness using these features, without training a separate reward model from scratch. Used to select among multiple sampled trajectories, this verifier improves reasoning accuracy and strengthens the test-time scaling effect — i.e., the same interpretability tool doubles as a practical component for best-of-N-style inference.

4. Why It Matters

Two things make this a useful paper to have read before diving into method-proposing reasoning papers:

1. It reframes “does this method work” as “how does this method’s trajectory move through state space.” Every reasoning method review that follows in this series — search-based, budget-aware, latent-iteration, whatever the mechanism — is implicitly making a claim about how it shapes the trajectory’s path toward the answer. LoT gives a vocabulary (convergence speed, consistency, uncertainty) for that claim that doesn’t require a new benchmark number.
2. The verifier repurposing is the sharper practical point. A visualization tool that also functions as a free, training-free-ish scoring signal for trajectory selection is a different kind of contribution than “we made a prettier plot” — it suggests that the geometry of intermediate reasoning states carries real, exploitable signal about correctness, not just a post-hoc explanation of it.

5. Limitations Worth Knowing

• Requires multi-choice structure. The distance-to-answer-choice feature construction depends on having a fixed, enumerable set of candidate answers — it isn’t a direct fit for open-ended generation tasks where there’s no small set of anchor points to project against.
• Perplexity as a distance metric is model-relative. The distance vector for a given state is computed using the same LLM that produced the trajectory, so landscapes are not directly comparable across models without care — a “close to the correct answer” reading from one model’s self-assessment isn’t necessarily calibrated the same way as another’s.
• t-SNE distortion. Like any t-SNE projection, the 2D layout is a nonlinear compression of a higher-dimensional space; visual distances and cluster shapes can mislead if read too literally, and the quantitative metrics (perplexity, uncertainty, consistency) are the more trustworthy artifacts than the plot geometry itself.
• Verifier gains are reported at a general level in the sources used for this review — exact accuracy deltas and which datasets/ reasoning methods the verifier was tested against were not fully confirmed against the primary PDF, so treat the “boosts accuracy and test-time scaling” claim as directionally correct pending a closer read of the paper’s tables.

6. The Takeaway for a First Reader

If you remember three things:

1. LoT visualizes reasoning trajectories, not just final accuracy — every intermediate state in a chain-of-thought is turned into a distance-to-each-answer-choice vector (via perplexity) and projected into 2D with t-SNE, so a trajectory becomes a literal path toward (or away from) the correct answer.
2. Three metrics — perplexity, uncertainty, consistency — move together with model scale and correctness. Bigger models converge faster and more confidently; correct trajectories are quantitatively distinguishable from incorrect ones by how they move, not just where they end up.
3. The same features double as a lightweight, training-free-ish verifier for selecting among sampled trajectories — turning an interpretability tool into a practical test-time-scaling component.

References

• Zhou, Z., Zhu, Z., Li, X., Galkin, M., Feng, X., Koyejo, S., Tang, J., & Han, B. (2026). Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models. ICLR 2026. arXiv:2503.22165 · project page · code.

0. The Picture in One Paragraph

Ask an LLM to generate a novel idea and it tends to default to high-probability, familiar concepts — the paper’s framing is that generation gets stuck in training-data “gravity wells.” Tree-of-Thoughts-style search tries to escape this by exploring multiple branches, but it relies on the LLM’s own self-evaluation to score and prune those branches — an unprincipled and, empirically, unreliable signal for judging genuine novelty. Magellan reframes creative generation as guided MCTS over an LLM’s latent conceptual space, replacing self-evaluation with a hierarchical guidance system: a semantic compass vector (built via orthogonal projection of concept embeddings) sets the long-range direction of the search toward a promising, non-obvious region of the solution space, while a landscape-aware value function — combining coherence (token log-probability), novelty (semantic distance from established knowledge), and progress (“semantic momentum,” how much new information a step adds relative to its parent) — makes the local, step-by-step tactical decisions inside the UCT selection formula. On a scientific idea generation task, Magellan wins 92% of head-to-head comparisons against zero-shot, CoT, ReAct, and ToT baselines, using only a Qwen3-1.7B backbone.

1. The Problem — Gravity Wells and Unreliable Self-Evaluation

Two separable failures compound in naive creative generation with LLMs:

1. Gravity wells. Sampling from an LLM’s output distribution over-weights common, familiar concepts from training data — genuine novelty is, almost by definition, lower-probability, so unstructured generation rarely reaches it.
2. Unreliable self-evaluation. Search-based mitigations like Tree-of-Thoughts try to explore multiple candidate branches and prune with the LLM’s own judgment of which branch is “better” — but an LLM judging its own candidate ideas for novelty and promise is exactly the same distribution-following process that produced the gravity-well problem in the first place, just applied one level up.

2. The Magellan Method

2.1 Semantic compass — global guidance

A semantic compass vector is computed via orthogonal projection of concept embeddings: it is constructed to preserve the core problem context (so the search doesn’t drift off-topic) while maximizing the influence of directions that lead toward novel mechanistic pathways. This vector gives the whole MCTS search a consistent long-range target — a fixed sense of “which direction is interesting” that doesn’t have to be re-derived and re-judged at every node.

2.2 Landscape-aware value function — local guidance

Instead of asking the LLM to self-evaluate each candidate node, a value function combines three explicit, computable terms:

• Coherence — average token log-probability, measuring local fluency/plausibility.
• Novelty — semantic distance from established (training-data- typical) knowledge.
• Progress — “semantic momentum”: how much substantially new information a step introduces relative to its parent node, penalizing branches that just restate or trivially rephrase prior content.

This weighted combination is added directly as a guidance term inside the standard UCT (Upper Confidence bound applied to Trees) selection formula, replacing the role that self-evaluation would otherwise play in choosing which branch to expand next.

3. Results

Evaluated on a scientific idea generation task, judged by an LLM judge (DeepSeek-V3.1-Think) scoring 1–10 on Plausibility, Clarity, and Innovation, with all methods using a Qwen3-1.7B generation backbone:

ReAct and ToT score notably below even the zero-shot baseline, with the paper attributing this to “thematic drift” and shallow, repetitive exploration — a pointed illustration of the self-evaluation failure mode Magellan is designed to avoid, though the severity of this gap is worth treating cautiously (see limitations below).

Ablation

Removing the guidance term from the UCT selection formula — i.e., falling back to plain UCT exploration without the semantic compass/value function — drops the win rate from 90.0% to 10.0%, a near-total collapse that indicates the guidance system, not the MCTS scaffolding itself, is doing essentially all of the work.

Efficiency

MCTS is configured for a maximum of 30 iterations, but the search reportedly converges in roughly 3 iterations on average — the guidance signal is precise enough to avoid the kind of broad, unguided exploration that would need many more iterations.

4. Why It Matters

The core claim worth taking seriously is architectural, not just empirical: self-evaluation is not a free source of judgment — asking an LLM to score its own candidate outputs for a property like novelty inherits the same biases that make naive generation gravity-well-prone in the first place. Magellan’s answer — decompose “is this idea good” into explicit, separately-computable signals (fluency, distance-from-known, information-gain-over-parent) rather than one holistic self-judgment — is a pattern that could generalize well beyond scientific idea generation to any LLM-search setting where self-evaluation is currently the default (which includes most ToT/MCTS-for-reasoning systems).

5. Limitations Worth Knowing

• Single-author, independent research, evaluated on one domain (scientific idea generation) with one LLM judge — independent replication at larger scale and across different creative domains (e.g., open-ended design, creative writing) would strengthen the claim.
• Small backbone (1.7B). Whether the gap over baselines persists, narrows, or widens with a larger generation backbone is untested.
• ReAct/ToT baseline failures are unusually severe (overall scores of 4.56 and 4.93, both below zero-shot) — this could reflect a genuine limitation of those methods for this task, but it is also consistent with under-tuned baseline implementations; the magnitude of the gap should be interpreted cautiously until independently reproduced.
• LLM-judge evaluation (DeepSeek-V3.1-Think scoring 1–10) inherits whatever biases that judge model has about what counts as “innovative” — a single automated judge is a narrower validation than human expert evaluation on genuinely novel scientific ideas.

6. The Takeaway for a First Reader

If you remember three things:

1. LLMs default to familiar, high-probability concepts (“gravity wells”), and Tree-of-Thoughts-style self-evaluation is an unreliable way to escape them, since the judge and the generator share the same underlying bias.
2. Magellan replaces self-evaluation with two explicit guidance signals: a global semantic-compass vector (orthogonal projection of concept embeddings) and a local landscape-aware value function (coherence + novelty + progress) injected directly into MCTS’s UCT formula.
3. 92% win rate against zero-shot/CoT/ReAct/ToT baselines on scientific idea generation with a small 1.7B backbone — and an ablation showing the guidance term, not the search scaffolding, is responsible for nearly all of the gain (90% → 10% win rate when removed).

References

• Chang, L. (2025). Magellan: Guided MCTS for Latent Space Exploration and Novelty Generation. 1st Open Conference on AI Agents for Science (agents4science 2025). arXiv:2510.21341.
• Related on this site: PriorZero — another MCTS-plus-LLM-guidance system, applied to sequential decision-making rather than creative idea generation; BG-MCTS — budget-aware MCTS for reasoning, a related use of guided search under a different objective.

0. The Picture in One Paragraph

MuZero-family agents (and their scalable descendant, UniZero) learn a latent world model and plan over it with MCTS — powerful, but they learn everything about the environment’s dynamics and semantics from scratch through interaction. LLMs, in contrast, carry rich static world knowledge but no grounding in a specific environment’s actual transition dynamics — injecting their priors naively risks a prior-dynamics mismatch that corrupts the agent’s own learned lookahead. PriorZero resolves this with a deliberately narrow injection point: an LLM policy prior (computed from prompt log-probabilities over admissible actions, using a chain-of-thought prefix) is blended into the action distribution only at the MCTS root node. Every deeper simulation step — dynamics rollout, value backup, expansion — remains purely world-model-driven. The LLM cheaply biases which actions get explored first without ever touching the model’s own learned notion of how the environment actually behaves. On sparse-reward Jericho text-adventures and the compositional- generalization BabyAI suite, this gives PriorZero a clear edge over plain UniZero — including reaching 0.96 on a compositional level (SynthLoc) where UniZero alone learns almost nothing.

1. The Problem — Prior-Dynamics Mismatch

An LLM’s world knowledge is static and general; a MuZero-style world model’s knowledge is learned, task-specific, and grounded in actual observed transitions. If an LLM prior is injected deeply into a planning process — say, at every internal MCTS node — it risks steering the search toward actions that sound plausible in general but are wrong for the actual (possibly idiosyncratic) dynamics of the specific environment, corrupting the value estimates the world model would otherwise have produced through honest simulation.

2. The PriorZero Method

Three gradient-decoupled components:

1. UniZero world model + latent MCTS — the base scalable MuZero-family planner, unchanged in its own dynamics/value/policy learning.
2. LLM prior module — scores admissible action strings via prompt log-probabilities, using a chain-of-thought prefix, normalized with a softmax (temperature 1) into a smooth prior distribution over the actions available at the current state.
3. Alternating trainer — the world model continually updates on interaction data (dynamics, policy, value losses) while its own value estimates, in alternation, provide a fine-grained credit-assignment signal used to fine-tune the LLM.

Root-prior injection

The core design decision: the LLM’s policy prior is blended into the action distribution only at the root node of each MCTS search. Internal nodes, dynamics rollouts, and value backups are left entirely to the world model. This means the LLM’s contribution is limited to biasing which branches get explored/expanded first from the current real state — a cheap, low-risk way to inject semantic knowledge without letting a potentially wrong static prior propagate through deep, compounding simulation steps.

3. Results

Jericho (text-adventure interactive fiction — Detective, Acorncourt, Zork1, Omniquest):

• On the sparse-reward games (Acorncourt, Omniquest), PriorZero reaches high-return regions substantially earlier than UniZero.
• On Detective, UniZero has an early advantage but PriorZero overtakes it as training progresses.
• On Zork1 (a large action space, >50 admissible actions/step, long horizon), PriorZero reaches a higher asymptotic return than UniZero.

BabyAI (18 instruction-following gridworld levels):

• PriorZero’s average asymptotic score across levels is 0.82 versus UniZero’s 0.79.
• On the compositional-generalization level SynthLoc, where UniZero learns almost nothing (near-zero), PriorZero reaches 0.96 — the clearest single result in the paper for where language priors help most: generalizing to novel action/object compositions the world model has not directly experienced.

LLM capacity ablation

Swapping the default Qwen2.5-3B-Instruct prior source for Qwen2.5-7B-Instruct converges faster and reaches a higher asymptotic score on Zork1 — the quality of the injected prior scales with the underlying LLM’s capacity, as one would hope.

4. Ablations

• Alternating vs. non-alternating fine-tuning schedule — confirms the alternating (decoupled rollout/training) design is preferable to jointly optimizing world model and LLM prior in lockstep.
• CoT-prefixed prior extraction with weighted token loss vs. plain action scoring — the CoT-prefixed variant is the one used in the main results.
• LLM capacity (3B vs. 7B) — larger LLM priors help, as above.
• Necessity of MCTS lookahead depth — confirms that PriorZero’s gains depend on retaining genuine world-model search depth, not just on the LLM prior alone; a shallow-search variant loses much of the advantage.

5. Relationship to Prior Work

PriorZero is built directly on UniZero, itself part of the MuZero/EfficientZero lineage and released alongside the LightZero benchmark codebase. Its contribution relative to that lineage is narrow and specific: rather than proposing a new world model or a new MCTS variant, it identifies the root node as the right, minimally-invasive injection point for external LLM priors — a design choice that is easy to describe but whose value is only demonstrated empirically through the sparse-reward and compositional generalization results above.

6. Why It Matters

MCTS-based planners are excellent at exploiting a well-learned world model but can be slow to discover good actions in sparse-reward or combinatorially large action spaces, precisely because undirected search has to stumble onto the right branch before the world model can even evaluate it well. PriorZero’s root-only injection is a minimal-risk way to use an LLM’s general knowledge to narrow that initial search — a design pattern that generalizes beyond text adventures to any MCTS-based agent operating in an action space large enough that undirected exploration is the actual bottleneck, not world-model accuracy.

7. Limitations Worth Knowing

• Evaluated only on text/language-grounded environments (Jericho, BabyAI) — no results on classic MCTS domains like Atari, board games, or continuous-control robotics, where action spaces are not naturally describable as admissible strings for an LLM to score.
• Inference-cost overhead of the added LLM scoring step at the MCTS root (extra LLM forward passes per search) is not reported in the sources reviewed here.
• Root-only injection may leave value on the table in settings where useful semantic guidance would help deeper in the tree, not just at the root — the paper doesn’t explore that trade-off space.
• Generalization beyond Jericho/BabyAI-scale action spaces is untested; both benchmarks are relatively small/discrete compared to many real-world decision-making settings.

8. The Takeaway for a First Reader

If you remember three things:

1. Naively injecting LLM priors deep into MCTS risks corrupting the world model’s own learned lookahead — a prior-dynamics mismatch problem PriorZero avoids by design.
2. Root-prior injection blends the LLM’s policy prior into the action distribution only at the MCTS root node, leaving all deeper simulation, value backup, and dynamics rollout purely world-model-driven.
3. The gains are largest exactly where undirected search struggles most: sparse-reward text-adventures and compositional generalization (0.96 vs. near-zero on BabyAI’s SynthLoc level) — suggesting the technique targets exploration bottlenecks specifically, not general sample efficiency.

References

• Xiong, J., Pu, Y., Tang, J., & Niu, Y. (2026). PriorZero: Bridging Language Priors and World Models for Decision Making. arXiv:2605.12289.
• Related on this site: BG-MCTS — budget-aware MCTS for LLM reasoning, a complementary use of search budget rather than search guidance; Magellan — another MCTS-plus-LLM-guidance system, applied to creative idea generation rather than sequential decision-making.

0. The Picture in One Paragraph

Long chain-of-thought reasoning is expensive not because of the total number of FLOPs alone, but because tokens are generated strictly sequentially — one forward pass, one token, repeat. Prior attempts to reason in continuous latent space (Coconut, CODI, CoLaR) sidestep discrete decoding but tend to be unstable to train and don’t scale past short reasoning chains. SuperThoughts takes a middle path: during the CoT phase, a Compressor module folds each pair of consecutive CoT tokens into a single latent vector; from that one latent, a Main module predicts the next token and a lightweight Multi-Token Prediction (MTP) module predicts the token after that — two tokens decoded per forward step, while discrete token supervision is preserved throughout training (the key stabilizer missing from pure continuous-latent methods). Because compressing every step this aggressively is fragile on hard reasoning tokens, a confidence-based adaptive mechanism checks the MTP module’s max softmax probability at each step against a threshold: below it, the model falls back to the slower, more reliable Main-module decoding. Evaluated on Qwen2.5-Math models, this yields roughly 20–30% CoT length reduction with near-parity accuracy on MATH500, AMC, OlympiadBench, and GPQA-Diamond.

1. The Problem — Sequential Decoding Is the Bottleneck

Standard autoregressive CoT generates one token per forward pass, in order. Even if two consecutive tokens are jointly “easy” to predict together, the model still pays for two full forward passes. Prior attempts to fix this by reasoning in continuous latent space entirely (no discrete tokens at all) run into two problems: training instability (there’s no clean supervision signal for a latent that was never meant to be decoded) and short-horizon-only effectiveness — Coconut, CODI, and CoLaR are reported to work for reasoning chains of roughly 20–60 tokens but don’t transfer to the longer, more complex chains typical of real math/reasoning benchmarks.

2. The SuperThoughts Method

2.1 Three modules

• Compressor: encodes a pair of consecutive CoT tokens into one latent vector.
• Main module: from that latent, predicts the immediate next token — this is the “primary,” higher-quality decoding path.
• MTP module: a lightweight head that, from the same latent, predicts the token after the Main module’s prediction — a second token “in superposition” with the first, decoded without an additional full forward pass.

2.2 Two-stage training

1. Latent-space alignment: the compressed latent space is distilled from a discrete-token teacher, so the compressed representation starts out anchored to real token semantics rather than drifting into an arbitrary continuous space.
2. Joint end-to-end training: all three modules are trained together, with discrete token supervision preserved throughout — unlike Coconut-style approaches that drop discrete supervision once latent reasoning kicks in. This is the paper’s claimed source of training stability at longer horizons.

2.3 Confidence-gated adaptive inference

At each step, the MTP module’s max softmax probability is compared against a threshold τ. If confidence is high, the fast two-token superposition path is used. If confidence falls below τ, the model discards the MTP shortcut and falls back to feeding the latent into the Main module for standard, single-token decoding — self-regulating speed so that easy spans compress aggressively while hard reasoning steps get full-resolution treatment.

3. Results

Backbones: Qwen2.5-Math-1.5B, -7B, -14B (Instruct variants), evaluated on MATH500, AMC, OlympiadBench, GPQA-Diamond.

With adaptive (confidence-gated) inference:

Without adaptive fallback (fixed 2-token superposition on every step):

Two things stand out here. First, the adaptive mechanism is doing most of the work — pure fixed-rate compression is aggressive but expensive in accuracy, while confidence-gated compression recovers near-parity accuracy at a more modest (but still substantial) compression rate. Second, there is a clear scale-tolerance effect: the 7B model degrades less than the 1.5B model under equally aggressive fixed compression, suggesting larger models have more redundant capacity to absorb the loss of per-token discrete supervision.

4. Why Speculative Decoding Isn’t the Right Comparison

The paper is explicit that speculative decoding is not a fair baseline: speculative decoding targets latency under low hardware utilization (using spare compute to verify guesses in parallel), not a reduction in the total FLOPs spent on reasoning — which is SuperThoughts’ actual target. A method judged by tokens/sec under speculative decoding could look identical to SuperThoughts in latency while doing strictly more total computation.

5. Relationship to Prior Work

The MTP module design builds on prior multi-token-prediction work (Gloeckle et al., 2024, Meta; Liu et al., 2024). Relative to continuous-latent reasoning methods:

Since Qwen2.5 models lack native multi-token-prediction heads, the authors train the MTP module from scratch; they note that models with native MTP support (Qwen3-Next, MiMo) would likely align more easily and could simplify training.

6. Why It Matters

SuperThoughts is a useful data point in the broader “compress reasoning without losing discrete supervision” agenda: rather than betting everything on a fully continuous latent reasoning space (which has repeatedly struggled to scale past short chains), it keeps one foot in discrete-token land while still getting a real 2× decode-step reduction on the fast path. The confidence gate is the load-bearing piece — it is what turns an aggressive-but-fragile compression scheme into one that holds up at benchmark-length reasoning chains.

7. Limitations Worth Knowing

• Requires training an MTP module from scratch for backbones without native multi-token-prediction support, adding training complexity absent in models that ship with MTP already.
• Non-adaptive (fixed-rate) compression is fragile, especially at smaller scale — the adaptive mechanism is not optional for maintaining accuracy.
• Throughput-doubling is a per-step decode claim, not an independently reported wall-clock tokens/sec measurement in the sources reviewed here — real-world speedup will depend on how the confidence-gated fallback rate behaves across different task distributions.
• Scale-tolerance trend is only shown up to 14B parameters; whether even larger models could tolerate 3+ tokens in superposition is speculated but not tested.

8. The Takeaway for a First Reader

If you remember three things:

1. SuperThoughts compresses pairs of consecutive CoT tokens into a single latent, decoding two tokens per forward step via a Main module (primary) and a lightweight MTP module (secondary) — while keeping discrete token supervision throughout training, unlike prior continuous-latent reasoning methods.
2. A confidence-gated adaptive mechanism falls back to standard single-token decoding on hard steps, which is what makes the method hold up on long reasoning chains rather than degrading sharply.
3. ~20–30% CoT length reduction with near-parity accuracy on Qwen2.5-Math models across MATH500, AMC, OlympiadBench, and GPQA-Diamond — with larger models tolerating more aggressive compression than smaller ones.

References

• Xiong, Z., Garg, S., Yu, M., Shrivastava, V., Zhao, H., Kyrillidis, A., & Papailiopoulos, D. (2026). SuperThoughts: Reasoning Tokens in Superposition. ICML 2026. arXiv:2606.13862.
• Related on this site: SOL — token-level efficiency policies via GRPO, a complementary axis of inference-time compute reduction; Reasoning Cache — another approach to controlling the cost of long reasoning traces.

0. The Picture in One Paragraph

Most test-time scaling scales compute by writing more chain-of-thought tokens — which requires specialized reasoning training data, burns context-window budget, and represents “thinking” as literal words. This paper (whose resulting model is nicknamed Huginn, after one of Odin’s two ravens — “Thought,” as opposed to its counterpart Muninn, “Memory”) takes a different route: a transformer split into a prelude, a small recurrent core (a handful of shared blocks), and a coda, where the recurrent core is looped R times in latent space before the coda decodes an output. No specialized chain-of- thought data is needed, small context windows suffice (the “thinking” never leaves latent space to consume context tokens), and R is a free knob at test time — more recurrence, more effective compute, without changing the parameter count. Trained at 3.5B parameters and 800B tokens on Oak Ridge’s Frontier supercomputer, the model shows reasoning-benchmark gains that scale with recurrence depth up to a compute load roughly equivalent to a 50B-parameter non-recurrent model — though it still trails the strongest explicit-CoT models on some benchmarks like GSM8K.

1. The Problem — Chain-of-Thought Is One Specific Way to Spend Test-Time Compute

Explicit CoT test-time scaling works, but it has costs baked into its form: it needs reasoning-annotated or RL-shaped training data, it consumes the context window with literal reasoning tokens, and it assumes that useful intermediate computation can always be represented in words. The paper’s premise is that at least some of that computation is better represented as continued latent-space refinement of a hidden state, unconstrained by having to be periodically projected back into vocabulary space and re-embedded.

2. Architecture — Prelude, Recurrent Core, Coda

The model is split into three parts:

• Prelude (~2 transformer blocks): embeds the input into an initial latent state.
• Recurrent core (~4 transformer blocks): a small shared bank of blocks that is looped R times, injected with a randomly initialized latent state at the start of recurrence and iteratively refining it.
• Coda (~2 blocks + unembedding head): reads the final latent state after R iterations and decodes the output.

Because the recurrent core is weight-tied across iterations, looping it R times adds no parameters — only compute. R is chosen at test time (commonly 16–128 in the paper’s experiments), giving per-token adaptive compute: easy tokens can be decoded after few iterations, hard tokens can keep looping. The design is also compatible with KV-cache sharing across recurrence steps and with speculative decoding, since the recurrent core’s weight-tying means cached key/value projections from earlier iterations remain valid.

Training the recurrence

Rather than fixing R during training, the depth per training step is sampled from a Poisson-log-normal distribution with mean 32, and backpropagation is truncated to the last 8 recurrence passes to bound memory and compute. This lets the model see a wide range of recursion depths during training without paying the full back-through-time cost of the deepest ones.

3. Training Scale

The model was trained at 3.5B parameters on 800B tokens, using 21 training segments of 4,096 AMD MI250X GPUs on the Oak Ridge Frontier supercomputer — a substantial systems effort in its own right, reflecting the difficulty of training recurrent-depth models at scale (variable computational graphs per step complicate standard data/pipeline parallelism).

4. Results

Reasoning-benchmark performance improves — “sometimes dramatically” — as recurrence depth R increases, up to a compute budget roughly equivalent to a 50B-parameter non-recurrent model, at which point gains from further recurrence plateau or mildly decline (tested out to R ≈ 128–256). On GSM8K without explicit CoT prompting, accuracy rises with more recurrence steps before plateauing — outperforming similarly-sized non-recurrent open baselines like Pythia-6.9B and Pythia-12B on ARC and GSM8K. Gains are most pronounced on math- and logic-heavy benchmarks (GSM8K, ARC-Challenge); they are flatter on knowledge-heavy benchmarks like MMLU and HellaSwag, where extra latent recurrence has less to work with. Notably, plain recurrent-depth scaling still lags the strongest explicit-CoT models on GSM8K, even as it substantially closes the gap against non-recurrent baselines of similar parameter count.

5. Emergent Latent-Space Behavior

Qualitative analysis of the recurrent core’s trajectories through latent space finds recognizable geometric patterns tied to task type:

• Orbits — rotating, cyclic trajectories on tasks with an iterative structure.
• Sliders — progressive, monotonic movement through latent space on counting-like tasks.
• Fixed-point convergence — the latent state settling into a stable point once further recurrence stops changing the answer.

The authors also report a genuine self-correction behavior: on some examples, the model detects and revises an earlier wrong intermediate value during later recurrence steps — computation that happens entirely in latent space, never surfacing as a visible, correctable chain-of-thought token.

6. Why It Matters

This paper is a foundational reference point for the whole looped/recurrent-depth transformer line — both LoopFormer’s elastic-depth training and Think-at-Hard’s selective per-token iteration (both reviewed alongside this post) explicitly position themselves relative to it. Huginn establishes that (a) latent-space recurrence is trainable at real scale (3.5B/800B tokens) without specialized reasoning data, and (b) depth genuinely functions as a test-time compute knob with measurable accuracy gains — the base result that later work refines by making depth elastic (LoopFormer) or selective per token (TaH) rather than a single global recurrence count.

7. Limitations Worth Knowing

• Proof-of-concept scale. The authors describe this as an early demonstration; 3.5B parameters is modest by frontier-model standards, and it is unclear how the recurrence-depth scaling curve behaves at 10× or 100× the parameter count.
• Gains plateau and can mildly degrade at very high recurrence counts (roughly R ≈ 128–256 in the reported experiments) — more test-time compute is not unboundedly beneficial.
• Still trails top explicit-CoT models on some benchmarks (e.g., GSM8K), meaning recurrent-depth latent reasoning is not (yet) a strict replacement for chain-of-thought, more a complementary or partially substitutable mechanism.
• Wall-clock cost of recurrence versus explicit CoT tokens is not directly compared in a hardware-normalized way in the sources reviewed here; the “compute-equivalent to 50B parameters” framing is a FLOPs-style comparison, not a latency benchmark.

8. The Takeaway for a First Reader

If you remember three things:

1. Test-time compute doesn’t have to be spent on explicit CoT tokens — this paper scales it instead through R iterations of a shared recurrent block in latent space, with no specialized reasoning training data required.
2. The architecture is prelude → recurrent core (looped R times) → coda, trained by sampling R per step from a Poisson-log-normal distribution (mean 32) with truncated backprop through the last 8 passes.
3. At 3.5B parameters / 800B tokens, the model (nicknamed Huginn) shows accuracy gains scaling with recurrence up to a ~50B-parameter compute-equivalent, with emergent latent-space self-correction — while still trailing top explicit-CoT models on some benchmarks.

References

• Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B.R., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv:2502.05171.
• Related on this site: LoopFormer — trains a single model for elastic, budget-conditioned loop depth, directly building on this recurrent-depth foundation; Think-at-Hard — pushes the same latent-iteration idea down to per-token selectivity rather than a single global recurrence count.

0. The Picture in One Paragraph

Looped transformers — a shared block applied M times instead of M distinct layers — have become a popular way to add algorithmic and latent-reasoning inductive bias without adding parameters. But every prior design (Universal Transformers, ACT, recurrent-depth models like Huginn) fixes the loop count at training time, or samples it randomly without ensuring different depths are actually consistent with each other. LoopFormer asks a different question: can a single trained model support elastic depth at inference — pick any budget M ≤ L after training, with quality degrading gracefully rather than collapsing? The mechanism is shortcut modulation: each loop iteration is conditioned on its cumulative normalized time and step size via sine-cosine embeddings feeding a FiLM-style layer that scales norms and gates residual updates. Training adds a shortcut-consistency objective — coarse (shortcut) schedules are distilled to match the representation a fine-grained (full) schedule would have produced — so that fewer, coarser steps at inference still approximate what many fine steps would give. The result is a model that scales smoothly with added inference budget rather than requiring a separately trained model per depth.

1. The Problem — Depth as a Training-Time Hyperparameter

Weight-tied looped transformers repeat one block M times. Prior designs pick M once and bake it in:

• Universal Transformers / ACT: per-token adaptive halting, but the halting policy is learned jointly with a specific training-time schedule.
• Huginn (recurrent-depth): samples the recurrence depth per training step from a distribution, discarding any explicit halting mechanism, and treats depth as a test-time knob — but does not explicitly train different depths to be consistent with each other.

None of these give a single model a principled, budget-conditioned way to trade inference compute for quality at deployment time, with a guarantee that a shorter schedule is a genuine approximation of a longer one rather than an arbitrary truncation.

2. The LoopFormer Mechanism

2.1 Time and step-size conditioning

Each loop iteration i is given two scalars: the cumulative normalized time t_{i-1} (how far along the 0→1 schedule the model already is) and the step size Δ_i = t_i − t_{i-1} it is about to take. Both are turned into sine-cosine frequency embeddings, passed through small MLPs, and summed into a single conditioning vector e_i — the same style of timestep conditioning used in diffusion models, repurposed for transformer loop depth.

2.2 Shortcut modulation

The conditioning vector e_i drives a small FiLM-style MLP that produces scale parameters (γ1, γ2) for two RMSNorm layers and gates (α1, α2) applied before the attention and FFN residual connections. This lets the same shared block behave differently depending on where in the schedule — and how coarse a step — the current iteration represents.

2.3 Shortcut-consistency training

Within a batch, LoopFormer runs both a full-length route (many fine-grained steps) and a shorter “shortcut” route (a coarser step schedule) through the same weights. The shortcut route’s final representation is trained — via a stop-gradient target from the full route — to match what the full route would have produced. This is a form of self-distillation within the loop: coarse schedules learn to approximate the outcome of fine-grained ones, rather than just being truncated fine-grained runs.

2.4 Elastic-depth inference

At inference, a user picks any budget M ≤ L and any step schedule (e.g., coarser early, finer late — an ablation finding), with no retraining required. The architecture itself is a NanoGPT fork with RMSNorm in place of LayerNorm, kept close to the original for reproducibility.

3. Results

LoopFormer is evaluated across a budget sweep of loop counts K ∈ {1, 3, 6, 9, 12}, with language-modeling perplexity and zero-shot accuracy across roughly ten benchmarks — COPA, HellaSwag, LAMBADA, OpenBookQA, PIQA, RACE, Social IQA, ARC (Easy + Challenge), SciQ, and WinoGrande — compared at matched compute against vanilla non-looped transformers, fixed-depth looped baselines, and adaptive early-exit baselines.

The headline qualitative finding: LoopFormer scales smoothly and monotonically with added inference budget, whereas fixed-depth baselines that are asked to run at an off-schedule depth degrade sharply rather than gracefully. (The paper’s precise numeric perplexity/accuracy tables were not independently verifiable while drafting this review — arXiv access was intermittently blocked — so treat the exact magnitudes as reported by the authors rather than independently re-derived here.)

4. Ablations

• Schedule shape matters. The best-performing schedules take coarser steps early in the loop and finer steps late — mirroring a common pattern in diffusion-model noise schedules.
• Perplexity-optimal ≠ accuracy-optimal. The schedule that minimizes language-modeling perplexity is not always the one that maximizes downstream reasoning accuracy — a reminder that perplexity is an imperfect proxy for reasoning quality.
• Training cost. Running both a full and a shortcut route per batch increases training cost roughly 1.5× relative to training a single fixed-depth schedule.

5. Relationship to Prior Work

The key differentiator from Huginn — the most directly comparable prior system, and reviewed separately on this site — is that Huginn samples depth randomly during training without an explicit mechanism ensuring a short rollout approximates a long one; LoopFormer makes that approximation an explicit training objective.

6. Why It Matters

Elastic inference-time depth, if it holds up at larger scale, turns “how much do I want to spend on this token/prompt” into a genuine runtime knob rather than a model-selection decision made before training. Combined with token-level selective iteration methods (like Think-at-Hard, reviewed alongside this post), the looped-transformer line is converging on a picture where compute allocation is both elastic at the sequence level (LoopFormer) and selective at the token level (TaH) — two largely orthogonal axes of the same broader “stop paying uniform compute for reasoning” agenda.

7. Limitations Worth Knowing

• Global, sequence-level budget. The chosen depth M applies to the whole sequence, not per-token or per-instance — LoopFormer does not currently combine its elastic budget with the kind of per-token selectivity TaH introduces.
• ~1.5× training overhead from running both full and shortcut routes per batch.
• Representation-consistency analysis is correlational, not a causal guarantee that shortcut routes always faithfully approximate full routes on out-of-distribution inputs.
• Scale and exact benchmark numbers were not independently re-verified for this review beyond what search-indexed sources reported; readers should consult the primary PDF for exact figures.

8. The Takeaway for a First Reader

If you remember three things:

1. Prior looped transformers fix loop depth at training time; LoopFormer instead trains one model to support elastic, user-chosen depth at inference.
2. The mechanism is shortcut modulation (time/step-size conditioning via FiLM-style gates) plus shortcut-consistency training — coarse schedules are explicitly distilled to approximate fine-grained ones.
3. LoopFormer scales smoothly with added budget where fixed-depth and early-exit baselines degrade sharply off their trained schedule — depth becomes a genuine runtime trade-off knob.

References

• Jeddi, A., Ciccone, M., & Taati, B. (2026). LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. ICLR 2026. arXiv:2602.11451.
• Related on this site: Think-at-Hard — token-level selective iteration, an orthogonal axis of compute allocation to LoopFormer’s sequence-level elastic depth; SOL — token-level efficiency policies for frozen LLMs, another point in the same design space of learned, granular compute allocation.

0. The Picture in One Paragraph

Looped transformers give every token a second (or third) latent forward pass before it commits to an output — a cheap, training-data-free form of test-time reasoning. The implicit assumption is that more iteration is always at least neutral. Think-at-Hard (TaH) shows this assumption is false: a fixed “always iterate twice” policy flips more correct first-pass predictions into wrong ones than it fixes — a latent overthinking phenomenon, the looped-transformer analogue of overthinking in explicit chain-of-thought. TaH replaces the fixed policy with a lightweight neural decider that predicts, per token, whether the first pass is likely wrong and only then triggers a second latent iteration. To make this selective computation actually fast in parallel hardware, TaH introduces duo-causal attention — a causal mask extended across a 2D (sequence position, iteration depth) grid — plus depth-aware LoRA adapters that activate only on the extra pass. The result: TaH beats a fixed-two-iteration baseline by 8–11% while skipping 94% of second iterations, and beats a same-data single-iteration Qwen3 baseline by roughly 4–5%.

1. The Problem — Latent Overthinking

A looped transformer processes a token through a shared block once, then optionally again, refining the hidden state before it is decoded. The implicit bet made by every prior looped-transformer design (Universal Transformers, Huginn-style recurrent depth) is that iterating uniformly across all tokens can only help or be neutral.

TaH’s diagnostic experiment — an “AlwaysThink” ablation that forces a second iteration on every token — shows the opposite: tracking per-token prediction transitions across iteration depths reveals that more predictions flip from correct to incorrect than from incorrect to correct. Most tokens are already right after the first pass; the second pass, applied indiscriminately, does more harm than good on average. This mirrors the well-documented overthinking failure mode in explicit CoT reasoning, but at the level of a single token’s hidden state rather than a whole reasoning trace.

2. The TaH Framework

2.1 Lightweight neural decider

A small classifier reads the first-pass hidden state at each token position and predicts whether that token’s first-pass top-1 prediction is likely wrong. It is trained against oracle labels: a token is labeled “iterate” if running the second pass actually corrects it relative to a reference. At inference, the decider gates the second iteration per token — most tokens skip it entirely.

2.2 Duo-causal attention

Selective, per-token iteration creates a structural problem: different tokens in the same sequence are now at different iteration depths, but training and inference still need to run in parallel, not token-by-token. TaH’s fix is to extend the causal mask from the usual 1D (sequence position) axis to a 2D grid of (position, depth): a token attends to all earlier positions and to states at or below its own iteration depth at the same and earlier positions. This preserves causality on both axes while keeping full sequence-level parallelism — no serialization penalty for mixing 1-pass and 2-pass tokens in one forward call.

2.3 Depth-aware LoRA adapters

Small LoRA adapters are active only during the second (extra) pass. This decouples the backbone’s original next-token objective (used on the first pass) from a narrower refinement objective (used only when the decider triggers a second pass), letting the extra parameters specialize in fixing likely-wrong tokens rather than diluting the general-purpose first-pass behavior.

2.4 TaH+ — a cheap upper bound

A variant, TaH+, skips the decider and its selective gating entirely, always running the second pass with the LoRA adapters active (under 3% extra parameters). It is not meant to be efficient — it is a cheap way to check how much of TaH’s decider-gated gain comes from the adapters themselves versus the selectivity.

3. Training

TaH is fine-tuned from Qwen3-Base at 0.6B and 1.7B scale on the Open-R1 reasoning dataset. The decider is trained as a binary classifier against oracle iterate/don’t-iterate labels derived from per-token correctness deltas between the first and second pass. Evaluation spans nine benchmarks across math, QA, and coding, including GSM8K, MATH500, and AIME24/AIME25.

4. Results

On AIME25, TaH reaches 17.9% versus a 13.3% standard single-iteration baseline. The gap between TaH’s realized gains and the oracle ceiling of +7.3% is itself informative: the trained decider captures most, but not all, of the available headroom — there is still knowable-but-unexploited signal about which tokens need a second pass.

5. Why It Matters

Looped/recurrent-depth transformers (this blog has covered several — see the recurrent-depth and elastic-depth family below) treat “how many iterations” as either a fixed hyperparameter or a global, sequence-level budget. TaH pushes the granularity down to individual tokens, and — more importantly — demonstrates empirically that uniform iteration is not merely wasteful but actively harmful on net. That reframes the design question for the whole looped-transformer line: the goal isn’t just “skip iterations to save compute,” it’s “skip iterations because most of them are actively hurting accuracy.”

6. Limitations Worth Knowing

• Decider imperfectly approximates the oracle. The 7.3% oracle ceiling versus TaH’s realized 8–11% gain over AlwaysThink (a different, weaker baseline) shows there is still room for a better decider — the selectivity mechanism is good but not optimal.
• Small-scale evaluation. Results are reported at 0.6B and 1.7B parameter scale; whether the overthinking phenomenon and the decider’s effectiveness hold at larger scales is untested here.
• Binary, per-token gating is coarse. The decider makes a single iterate/don’t-iterate call; it does not decide how much extra computation a hard token deserves beyond one more pass.
• Fine-tuned rather than pretrained from scratch. TaH is built on top of an existing Qwen3-Base checkpoint; interactions between the decider and pretraining-time objectives are not explored.

7. The Takeaway for a First Reader

If you remember three things:

1. Looped transformers can overthink. Forcing a second latent iteration on every token flips more correct predictions to incorrect than the reverse — indiscriminate iteration is a net negative, not just a compute cost.
2. TaH fixes this with three pieces: a lightweight decider that gates the second pass per token, duo-causal attention that lets mixed-depth tokens be processed in parallel, and depth-aware LoRA adapters that specialize the extra pass toward refinement.
3. 8–11% accuracy gain over a fixed-iteration baseline while skipping 94% of second passes — selective computation beats uniform computation, and the decider captures most (not all) of the oracle’s theoretical headroom.

References

• Fu, T., You, Y., Chen, Z., Dai, G., Yang, H., & Wang, Y. (2026). Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models. ICML 2026. arXiv:2511.08577.
• Related on this site: LoopFormer — elastic, budget-conditioned loop depth trained via shortcut-consistency, a complementary approach to token-level selective iteration; BG-MCTS — another budget-aware test-time mechanism, at the level of search rather than latent iteration.

0. The Picture in One Paragraph

In math and code, intermediate reasoning steps are locally verifiable — you can check each algebraic step or run the code. In knowledge-intensive domains like medicine, step verification requires cross-referencing vast external knowledge bases, and a subtly wrong clinical inference may propagate through an entire reasoning trace undetected. Process Reward Agents (PRA) address this by training a dedicated reward agent — distinct from the reasoning policy — to observe the unfolding CoT trace, selectively query external knowledge via retrieval, and assign scalar reward scores to candidate continuations during beam search. The policy model is never fine-tuned; only the reward agent is trained. The result is a modular, backbone-agnostic plug-in: a frozen Qwen3-4B-Instruct reaches 80.8% on MedQA (new SOTA at the 4B scale), with up to +25.7% improvement across frozen policy models ranging from 0.5B to 8B parameters. Unlike Self-Consistency, which degrades on hard benchmarks as more samples are added, PRA continues improving monotonically as inference compute scales.

1. The Problem — Non-Local Verifiability

Process reward models (PRMs) have been applied successfully to mathematics: you train a reward model to score each step of a proof or calculation, then use those scores to guide beam search toward correct solutions. The recipe works because math steps are locally verifiable — you can determine whether an intermediate algebraic transformation is correct without knowing the final answer.

Medical reasoning is structurally different:

Prior retrieval-augmented PRMs score completed trajectories after the fact. They cannot steer what is generated — by the time a bad reasoning path is scored, the trajectory is already done. PRA’s contribution is making the process reward signal online and integrated into the generation loop.

2. The Architecture — Frozen Policy + Reward Agent

PRA decouples two roles that prior work combined:

Policy model (frozen)   →   generates candidate CoT continuations
         ↓
Reward agent (trained)  →   reads partial trace
                        →   decides whether to retrieve
                        →   synthesizes retrieved docs with trace
                        →   assigns reward score per candidate
         ↓
Beam search             →   prunes low-reward candidates
                        →   propagates high-reward candidates

2.1 Frozen policy model

The reasoning policy generates text. It is never updated during PRA deployment. This separation means: when a better backbone model becomes available, you swap the frozen policy with no retraining of the reward agent. Domain expertise transfers.

2.2 Reward agent

A separate LLM trained to:

1. Read the partial CoT trace up to the current step.
2. Decide whether to issue a retrieval query.
3. If retrieval is useful: select a query, retrieve relevant documents (medical literature, PubMed, clinical guidelines), and synthesize them with the trace.
4. Assign a scalar reward score to each candidate continuation produced by the frozen policy.

The reward agent’s training labels are generated by a teacher model: given the partial trace and retrieved documents at each step, the teacher assesses whether the partial step is on track. This produces “reasoning labels” that train the reward agent to assign reliable intermediate rewards at inference time.

2.3 Integration into beam search

At each beam search step:

1. The frozen policy autoregressively proposes K candidate continuations.
2. The reward agent scores each candidate.
3. Beam search uses scores to rank and prune: low-scoring candidates are discarded, high-scoring ones propagate.
4. Repeat until the final answer is generated.

This is online (rewards computed during generation, not after) and process-level (rewards at intermediate steps, not just the final answer).

3. Results

3.1 MedQA (USMLE-style)

All improvements are achieved without modifying the policy model.

3.2 Generalization benchmarks

PRA is evaluated out-of-distribution on:

• MedBullets Op4 / Op5 (harder USMLE-style)
• MedMCQA (Indian medical licensing exam)
• MMLU-Med (medical subset of MMLU)
• GPQA (graduate-level “Google-proof” expert questions)
• Lancet clinical cases (real-world Lancet journal cases)
• NEJM clinical cases (New England Journal of Medicine cases)

PRA consistently outperforms baselines (greedy CoT, RAG, Self-Consistency, Best-of-N, post-hoc PRM scoring) across all benchmarks under matched sampling budgets.

3.3 Inference-time scaling behavior

This is the most practically important finding:

Self-Consistency degrades on hard benchmarks. As the number of sampled trajectories increases past ~8, majority vote begins amplifying errors — the policy frequently produces incorrect responses across samples, so the plurality answer is wrong. Self-Consistency actively hurts accuracy on GPQA and Lancet at high sample counts.

PRA continues improving. Because step-wise steering recovers from early errors (rather than averaging over them), PRA’s accuracy increases monotonically with inference compute. There is no observed saturation point in the evaluated range.

The mechanism is clear: when the policy starts down a wrong path, the reward agent’s low score causes that branch to be pruned before the error compounds. Self-Consistency has no such pruning — it samples independently and votes.

3.4 Ablations

4. Relation to Prior Work

Post-hoc PRMs

Standard PRMs and retrieval-augmented PRMs score completed trajectories. They can identify good vs. bad completions after the fact, but they cannot intervene during generation to prevent bad paths from developing. PRA makes the reward signal generative — it shapes what is produced, not just evaluates what was produced.

RAG (Retrieval-Augmented Generation)

Standard RAG appends retrieved documents to the prompt at the beginning. PRA retrieves selectively at each step based on what has been reasoned so far. The reward agent decides whether to retrieve and what to query based on the partial trace — a more targeted and dynamic use of retrieval.

Best-of-N

Best-of-N samples N complete trajectories and picks the highest-reward one. PRA prunes during generation, preventing bad branches from completing. This is more efficient and produces better results because early pruning stops error compounds before they occur.

5. Why It Matters

Three reasons:

1. Process-level rewards for knowledge-intensive domains. The PRM literature has largely focused on math. PRA is a concrete recipe for extending the process reward paradigm to domains where step verification requires external knowledge — a much larger class of real-world tasks.
2. Frozen policy + trained reward agent = modular deployability. The decoupled design means domain expertise lives in the reward agent, not the policy. You can upgrade the backbone reasoning model without retraining domain knowledge. This is a practically important architectural property for medical AI deployment.
3. Online steering vs. post-hoc scoring changes the scaling curve. Self-Consistency’s failure mode at high compute (degrading on hard tasks) is well-documented but underappreciated in practice. PRA’s demonstration that online process rewards maintain monotonic improvement where Self-Consistency fails is a direct argument for moving beyond best-of-N and majority vote as inference scaling strategies.

6. Limitations Worth Knowing

• Not a deployment system. The paper explicitly states this is a method contribution, not a ready-to-deploy medical decision-support system.
• Hallucination not eliminated. PRA reduces the propagation of incorrect intermediate steps but does not guarantee their absence.
• Inference cost. Every generation step incurs retrieval overhead plus reward computation, making PRA more expensive than greedy decoding or standard RAG.
• Hard tasks remain hard. On very challenging benchmarks (GPQA, Lancet), PRA is more stable than Self-Consistency but still operates under the inherent difficulty of the domain.

7. The Takeaway for a First Reader

If you remember three things:

1. Knowledge-intensive reasoning has a non-local verification problem — you can’t check a clinical inference without consulting external knowledge. Prior PRMs were post-hoc; PRA trains a dedicated reward agent that reads the partial trace at each step, selectively retrieves relevant knowledge, and assigns step-level reward scores during beam search — without touching the policy model.
2. Self-Consistency degrades on hard benchmarks as samples increase (majority vote amplifies errors). PRA continues improving monotonically because early pruning prevents bad paths from compounding into completed trajectories.
3. PRA + Qwen3-4B-Instruct achieves 80.8% on MedQA (SOTA at 4B scale), with up to +25.7% improvement across frozen 0.5B–8B policy models and consistent gains across 7 medical reasoning benchmarks.

References

• Sohn, J., Sternal, T., Styppa, K., Hoefler, T., & Moor, M. (2026). Process Reward Agents for Steering Knowledge-Intensive Reasoning. ICML 2026. arXiv:2604.09482.
• Related on this site: RLHF → RULER trend note — the process reward model context this work extends; Ch.8 Adversarial Feedback Design — external patterns in critic/verifier agent design.

0. The Picture in One Paragraph

A persistent question in CoT interpretability: do LLMs actually plan globally in their hidden states before verbalizing the reasoning steps, or are they making purely local token-by-token transitions? Prior evidence suggests both directions — hidden states seem to encode future information, yet removing explicit CoT steps destroys compositional reasoning. Tele-Lens settles this question empirically with a probing framework that tracks three signals simultaneously across the full CoT trajectory: predicted next tokens, predicted final answer, and predicted total reasoning length. The answer is myopic: for compositional tasks (multi-step algorithmic reasoning), the model’s hidden states carry near-chance final-answer predictions (0.49–0.51) throughout most of the trajectory, spiking to 0.94–0.99 only in the final one or two steps before completion. LLMs do not pre-plan; they compute step by step. This insight has two practical payoffs: a sparse set of “pivot” positions (identified by Tele-Lens signals near CoT completion) can represent the full trajectory’s uncertainty with +6% accuracy improvement, and automatic CoT bypass achieves a 16.2% bypass rate with negligible performance loss by detecting when hidden states already carry high early confidence.

1. The Open Question — Do LLMs Plan Ahead in Their Hidden States?

The empirical evidence before this paper pulled in two directions:

Evidence suggesting latent planning: “Future Lens” (Pal et al., ACL 2023) showed that intermediate Transformer layer activations encode information about tokens and even final answers before those outputs are verbalized. This implies some degree of latent forward planning.

Evidence against global planning: For tasks requiring compositional computation — multi-step algorithms, logical deduction chains — removing explicit CoT steps causes catastrophic performance drops. If the answer were truly pre-planned in latent space, removing intermediate verbalizations should matter less.

Tele-Lens asks: at what point in the CoT trajectory do hidden states reliably encode the final answer? If the model is planning globally, that signal should appear early. If it is computing locally, the signal should appear only near the end.

2. Tele-Lens — A Probing Framework for Teleological Information

2.1 Conceptual lineage

Tele-Lens generalizes two prior probing tools:

• Logit Lens (Nostalgebraist, 2020): bridges each Transformer layer’s hidden state to the LM head, showing how token predictions evolve through depth.
• Future Lens (Pal et al., 2023): extends this to predict subsequent tokens from a single hidden state.

Tele-Lens adds a third dimension — reasoning length prediction — and applies all three probes simultaneously across the temporal dimension (CoT trajectory position) rather than just depth.

2.2 Three probing dimensions

For each hidden state at each step in a CoT trajectory, Tele-Lens trains probes to predict:

2.3 Architecture

A bottleneck low-rank adapter with added nonlinearity is trained on top of each hidden state to produce the three probing outputs. The adapter has far fewer parameters than the LLM itself, reducing overfitting risk. One adapter per task family is trained on held-in data.

2.4 Two model settings

• In-Domain LLM: Qwen2.5-7B-Instruct fine-tuned via GRPO on the target tasks. Chosen because this model lacks a native “thinking” mode, enabling a clean bootstrap of CoT.
• Off-the-shelf LLM: Publicly available models without task-specific training.

2.5 Task suite

12 datasets across three categories:

3. The Core Empirical Finding — Myopic Planning

3.1 Final-answer probability trajectory

For compositional tasks (the hardest and most diagnostic category), the final-answer probability extracted from hidden states follows a characteristic profile:

Most of CoT trajectory:   ~0.49–0.51 (near chance)
Final 1–2 steps:          0.94–0.99 (decisive spike)

The model has essentially no reliable internal prediction of the final answer until the very last steps of the chain, regardless of how long the chain is. This directly contradicts a “latent global planning” hypothesis.

Concrete examples:

• Parity task (is the number of 1s in a binary string odd or even?): The model cannot predict parity until it has counted every bit. The final-answer hidden state signal only appears after the last digit is processed.
• Cycle task (does a graph contain a cycle?): The answer crystallizes only after the model has verbally traced a complete path or detected a cycle step-by-step.

3.2 Reasoning length prediction — no global clock

Early hidden states show no reliable internal estimate of total reasoning length — with an important caveat:

For Parity and Subsum, early states appear to correlate with true reasoning length. But this turns out to be a confound: in those tasks, reasoning length is proportional to input length (a surface shortcut). For Cycle tasks, reasoning length scales with path depth — which is not proportional to input length — and here early states completely fail to predict reasoning length.

This exposes the Parity/Subsum correlation as shallow pattern matching, confirming the absence of genuine global planning.

3.3 Simple tasks — coarse gist, not plan

For semantics/knowledge tasks, early hidden states do carry some predictive signal about the final answer. But the paper characterizes this as pattern-matching gist — recognizing a coarse category from surface features — not step-by-step planning. The signal is imprecise and degrades on unfamiliar inputs.

3.4 What this means for CoT

The findings land squarely on one side of a foundational debate:

CoT = scratchpad computation, not “decoding a latent plan.”

For compositional reasoning tasks, the intermediate steps are not verbalizations of a pre-existing internal plan — they are the computation. Removing them removes the reasoning itself.

4. Practical Applications

4.1 Pivot-based uncertainty estimation

The finding that useful hidden-state signal concentrates in the final 1–2 steps of CoT suggests that a sparse set of “pivot positions” (positions where Tele-Lens signals are most informative) can stand in for full-trajectory uncertainty estimation.

Result: +6% accuracy improvement compared to naive full-trajectory uncertainty estimation.

Comparison: positions selected by standard LM entropy spread across the whole trajectory; positions selected by Tele-Lens cluster near CoT completion — confirming that Tele-Lens adds signal beyond vanilla entropy.

4.2 Automatic CoT bypass

When Tele-Lens signals show high early confidence — the model’s hidden states already encode the final answer clearly from the start — CoT may be unnecessary overhead.

Result: 16.2% bypass rate (proportion of inputs where full CoT is skipped entirely) with negligible performance loss.

This provides an inference-time efficiency win without requiring a separately labeled router.

5. Why It Matters

Three reasons:

1. It resolves the latent-planning debate empirically. The question “are LLMs pre-computing in latent space?” has significant implications for understanding what CoT is doing and how to improve it. Tele-Lens gives a definitive answer for compositional tasks: no. The scratchpad is the computation.
2. It reframes the value of CoT. If CoT were just verbalizing a latent plan, you could in principle extract the plan directly and skip the verbalizations. Tele-Lens shows this is not possible: each CoT step genuinely updates the model’s internal state in a way that previous steps did not accomplish. This validates the investment in long CoT for hard tasks.
3. The uncertainty estimation application has practical value. A method that can reliably identify the handful of positions in a CoT trajectory that carry the most uncertainty information — using learned probes, not heuristics — enables cheaper and better calibration of model confidence, with direct deployment relevance.

6. Limitations Worth Knowing

• Training requirement. Tele-Lens requires a trained probing adapter, which needs task-specific data. It is heavier than zero-shot interpretability approaches.
• In-domain focus. Key quantitative results are on the GRPO- trained Qwen2.5-7B-Instruct model on the curated 12-task suite. Direct generalization to frontier closed-source models (GPT-4, Claude, Gemini) is not confirmed.
• Task scope. The 12 tasks skew toward controlled algorithmic and knowledge-QA settings. Whether the myopic pattern holds for open-ended long-horizon tasks (multi-document synthesis, large codebase reasoning) remains open.
• Correlational, not causal. Probing establishes that hidden states do not encode future planning signals early — but does not establish why, or whether interventions could force global planning.
• Per-domain adapter calibration. The adapter needs domain- specific calibration, limiting plug-and-play deployability across arbitrary tasks.

7. The Takeaway for a First Reader

If you remember three things:

1. LLMs are myopic planners. For compositional tasks, hidden states carry near-chance final-answer predictions (0.49–0.51) throughout CoT, spiking to 0.94–0.99 only in the final 1–2 steps. There is no evidence of global latent planning — CoT is scratchpad computation, and each step genuinely advances the reasoning in a way hidden states alone could not.
2. Tele-Lens probes three signals simultaneously across the CoT trajectory: subsequent tokens, final answer, and reasoning length — using a bottleneck low-rank adapter trained per task family. The reasoning-length result exposes a surface confound in simpler tasks, strengthening the myopia conclusion.
3. Two practical payoffs: a sparse set of pivot positions near CoT completion can represent full-trajectory uncertainty with +6% accuracy improvement, and automatic CoT bypass achieves 16.2% bypass rate with negligible performance loss by detecting early high-confidence hidden states.

References

• Xu, L., Yu, M., Meng, F., & Zhou, J. (2026). How Far Ahead Do LLMs Plan? Uncovering the Latent Horizon in Chain-of-Thought Reasoning. ICML 2026. arXiv:2602.02103.
• Code & data: https://github.com/lxucs/tele-lens
• Pal, A., et al. (2023). Future Lens: Anticipating Subsequent Tokens from a Single Hidden State. ACL 2023.
• Related on this site: MCTD review — inference-time search as an alternative to relying on a single CoT trace.