Why this, why now
For most of the scaling era, the levers were two: parameters and data. Then o1 and R1 made a third lever legible - test-time compute - and the entire reasoning-model boom has been one particular way of pulling it: generate more tokens. Chain-of-thought is, mechanically, a loop: the model writes a thought, reads it back, writes the next one. The state of the loop is the context window, the step size is one token, and the loop runs in vocabulary space.
There is another way to pull the same lever. Instead of looping over generated tokens, loop over depth: take a block of transformer layers, tie its weights, and run it again and again on the same hidden state. The state of this loop is the sequence of token embeddings - a continuous, high-dimensional object - and the step is a full block of attention and MLP compute. Nothing is verbalized. The parameter count stays flat while the effective depth, and therefore the compute per token, becomes a runtime dial.
That is a looped transformer. The idea is old (Universal Transformer, 2018 [1]), but between early 2025 and mid-2026 it produced: a 3.5B-parameter model, trained on 800B tokens on the Frontier supercomputer, whose scores keep climbing as you unroll it deeper at test time [6]; a Google DeepMind paper arguing with experiments and theorems that reasoning tasks want depth, not parameters [3]; a 27M-parameter model that went viral for beating o3-mini on ARC-AGI [8], and its 7M-parameter successor that beat it again [9]; and a token-level adaptive version from KAIST and Google that sets a new compute-quality Pareto frontier [7]. It also produced a serious backlash: the ARC Prize Foundation’s independent replication found the viral result worked for reasons nobody advertised [13].
This note is the deep version. The mechanics, the exact training recipes, the real tables, and the criticism.
The core idea
A standard transformer computes, at each position, xₗ₊₁ = xₗ + Attnₗ(xₗ) + MLPₗ(·) with fresh parameters θₗ at every layer l = 1..L. A looped transformer replaces the stack with one block R of k layers with shared weights, iterated r times:
sr+1 = R(sr, θ), r = 1..N
where s is the whole n x h matrix of token embeddings. Unrolled, the computation graph is a k·N-layer transformer; rolled up, it is a k-layer parameter budget. All standard training machinery applies unchanged to the unrolled graph. The two things that make this interesting rather than just a compression trick:
N is a free variable at test time. Because the weights are the same at every depth, nothing in the architecture fixes N. Train with a distribution of depths, run deeper at inference. This is the entire basis of the test-time scaling results.
The state is latent and shared. The thing that flows from iteration to iteration is not text; it is the full n x h hidden state, updated in place. Attention at every iteration lets every position read every other position’s current state. The loop is parallel over the sequence - which is what separates it from an RNN, where recurrence is over the sequence and training can’t parallelize across time.
Notation used below
Saunshi et al. write (k ⊗ L) for a k-layer model looped L times. The two natural baselines: the iso-param model (k ⊗ 1), same parameters, no loop; and the iso-FLOP model (kL ⊗ 1), same effective depth and compute but L times more parameters. A loop is only impressive if it beats the first and approaches the second.
Anatomy of one loop pass
What exactly happens to information during one recurrence? Using the Huginn design [6] as the reference (it is the most carefully documented), a token’s journey is:
1. Prelude. Tokens are embedded (scaled by √h) and passed through a small stack of ordinary, non-shared layers P. Output: an initial working representation e. The prelude runs once.
2. State initialization. The recurrent state s₀ is not the prelude output - it is sampled from a truncated normal distribution (variance 2/5 in Huginn). The state starts as structured noise; the loop’s job is to shape it.
3. The loop body. Each iteration: concatenate sᵢ with the prelude output e along the hidden dimension, and map the 2h-vector back down to h through a learned adapter matrix A. (Addition instead of concatenation works equally well at small scale; concatenation won at 3.5B.) Then run the k shared transformer layers, and rescale the output with an RMSNorm. So per iteration, every position: reads the current state of all positions via attention, mixes in a fresh copy of the input via the adapter, and writes its updated state.
4. Coda. After N iterations, a few more non-shared layers plus the unembedding (tied to the input embedding) map the final state to logits.
Two subtleties worth internalizing. First, the input is re-injected at every iteration through the adapter, so the block never “forgets” what question it is answering - this also means the prelude gets gradient signal at every step even when gradients are truncated over iterations. Second, because the same attention parameters are reused, the KV cache does not grow with loop count; you hold one block’s worth of cache and can even bound it (Huginn shows a cache budget of 4 steps, overwriting old entries cyclically, leaves MT-Bench essentially unchanged at 5.86).
The lineage, with details
2018 - Universal Transformer. Dehghani et al. [1] start from an observation that still motivates the field: vanilla transformers fail to generalize on simple algorithmic tasks - copying strings, executing short programs - once inputs exceed training lengths, while recurrent models handle those trivially. Their fix is to apply one shared encoder/decoder block repeatedly over the whole sequence at once, with two additions: a per-step time embedding added to the position encoding so the shared block knows which iteration it is on, and dynamic halting (ACT, from Graves 2016 [17]): each position maintains a halting probability that accumulates over iterations; when the cumulative mass passes 1 - ε, that position stops updating and its state is copied forward unchanged. So even in 2018 the loop count was per-position and data-dependent. Results: SOTA on bAbI question answering at the time, a new SOTA on LAMBADA language modeling, +0.9 BLEU over the base transformer on WMT14 En-De, and a proof that under mild assumptions the UT is Turing-complete while the fixed-depth transformer is not. The paper framed the UT as “a parallel-in-time self-attentive recurrent sequence model” - every looped transformer paper since is a variation on this sentence.
2019 - ALBERT. Lan et al. [2] tie all layer weights in a BERT-scale model and lose remarkably little accuracy while cutting parameters by roughly 10x. Not a reasoning paper - a parameter-efficiency paper - but it established the load-bearing empirical fact the whole field leans on: depth-wise weight sharing is nearly free in quality. If tying weights barely hurts at fixed depth, the door is open to tie weights and vary depth.
2023 - Looped transformers as programmable computers. Giannou et al. [4] ask the expressiveness question and answer it constructively. With hand-set weights, a constant number of encoder layers in a loop can implement: a scratchpad with read/write via attention-based copying, a program counter, conditional branching (if cond then goto), and function calls. Assembled, these emulate a one-instruction-set computer (SUBLEQ is Turing-complete), and the input sequence acts as a punchcard holding both program and data. Their headline artifact: a 13-layer looped transformer that, instructed entirely by its input, runs a calculator, a small linear algebra library, and - the foreshadowing - an in-context learning algorithm that internally performs backpropagation with SGD. Existence result, hand-programmed, not learned: it proves the architecture can express general computation at constant depth, not that gradient descent will find it. That gap is what the next three years of papers attacked.
2024 - Length generalization. Fan et al. [5] pick up the UT’s original failure case with modern tools. They study tasks whose solutions are repeated applications of one RASP-L operation (parity, copy with repeats, binary addition/sum/multiplication, unique-set), and train looped transformers with a randomized number of loop steps per training batch - the single most copied trick in the field since. Two results matter: looped models length-generalize far beyond the training lengths where fixed-depth transformers collapse, and at inference a simple maximum-confidence stopping criterion (halt when the output distribution stops changing) recovers the right number of loops per input. Learn the local update; let the runtime supply the iterations.
Worked example: ripple-carry addition
Addition is the running example in both Saunshi et al. and Fan et al., so make it concrete. Compute 847 + 629. The gradeschool algorithm is sequential: sum a column, carry, move left. The carry is a wave that propagates right to left and can ripple arbitrarily far (999…9 + 1). A fixed-depth transformer has to embed the entire worst-case ripple into its layers; a looped transformer only has to express one step of the wave and run it as many times as the input demands.
Give each digit column a position. The loop body performs a purely local update at every position simultaneously: add my operand digits, read any carry my right neighbor emitted last iteration, output my final digit and my carry. Unrolled:
Figure 1 - One loop iteration = one wave of carry propagation
| hundreds | tens | units | |
|---|---|---|---|
| operand a | 8 | 4 | 7 |
| operand b | 6 | 2 | 9 |
| loop 1: raw column sums | 14 | 6 | 16 |
| loop 2: apply carry from the right | 14 | 6+1=7 | 6, carry 1 |
| loop 3: carry reaches hundreds | 14+1=15 | 7 | 6 |
| loop 4: final carry emitted | 5, carry 1 | 7 | 6 |
| readout | 1 4 7 6 ✓ |
The carry wave. Each iteration of the shared block advances the carry one column. A 3-digit sum settles in four loops; a 40-digit sum needs more loops but zero new parameters and zero new training data. This is precisely the mechanism behind the length-generalization results: learn the local update, loop until quiescent.
Three observations, all of which show up later in the real experiments:
The update is trivially small. One iteration is a lookup-table-grade function of a column and its neighbor - well within one transformer’s capacity (formally, one RASP-L operation [5]). All the difficulty of addition lives in the iteration count, which is free. Saunshi et al. make this vivid: a 1-layer transformer looped 12 times solves n-ary addition with 1/12th of the parameters of the 12-layer baseline [3].
The right loop count is data-dependent. 847 + 629 settles in four iterations; 999 + 1 ripples through every column. A fixed-depth model must be sized for the worst case it will ever see; a looped model can keep looping, or learn to halt when the state stops changing - the 2018 ACT idea [17], reborn as MoR’s router [7].
The pattern is a class, not a trick. Swap “carry” for “a graph node’s new value” and you have graph algorithms; for “next digit of a partial quotient” and you have long division; for “one step of gradient descent” and you have in-context learning - the exact construction Giannou et al. hand-coded [4]. Iterative algorithms are a class of computations, and a looped transformer is the architecture-shaped hole the class fits.
Inside Huginn: the recurrent-depth recipe
Geiping et al. [6] (University of Maryland / ELLIS; model code-named Huginn) is the paper that answered “does any of this survive real pretraining scale?” The answer came with receipts, including the failed runs. This section is the recipe in full.
The model
Decoder-only transformer with the prelude / loop / coda split of section 3. Layer counts (2, 4, 2): 2 prelude layers, 4 shared recurrent layers, 2 coda layers. Hidden size h = 5280, giving 55 attention heads of size 96; MLP inner dimension 17920; RMSNorm with ε = 10-6. Parameter split: about 1.5B in prelude + head, 1.5B in the shared recurrent block, 0.5B in the tied embedding - 3.5B total. Mean training recurrence r̄ = 32, so the effective depth at training-time average is 2 + 32·4 + 2 = 132 layers. Model and code are public (tomg-group-umd/huginn-0125, seal-rg/recurrent-pretraining).
The training tricks - this is where the paper lives
Sampled loop counts. Every training step draws its recurrence count r from a log-normal Poisson distribution: sample τ ~ N(log(r̄) - σ²/2, σ) with σ = 1/2, then r ~ Poisson(e^τ) + 1. Most steps get r well below 32 (cheap), with a heavy tail of occasional deep unrolls (which teach the model depths it will need at test time). The loss is the expectation over both data and r. At large scale they synchronize one sampled r per micro-batch across all workers so no worker idles waiting for a deep unroll - “locked-step sampling.”
Truncated backprop. Backpropagate through only the last k = 8 iterations of the loop. Activation memory and backward compute are then independent of r, which is what makes the heavy-tailed depth distribution affordable. This is truncated BPTT, except recurrence is over depth rather than time - and, importantly, the prelude still receives gradients every step because its output e is re-injected into every iteration through the adapter.
Initialization is load-bearing. All weights from a truncated normal with variance σh² = 2/(5h) (Takase et al.), except out-projection layers, which get variance 1/(5h·l) with l = 132 the effective layer count - very small initial outputs, keeping the residual stream well-behaved at effective depth. The initial recurrent state s₀ is sampled fresh per sequence with variance 2/5.
The two failed runs. The paper documents what happens if you skip this: “Bad Run 1” used parameter-free RMSNorms, no embedding scale, a parameter-free additive adapter (A(s,e) = s+e), and peak LR 4e-4. The run stalled, and the stall mechanism is diagnostic gold: the correlation of hidden states across the token dimension goes to 1.0 - the model’s representation collapses, predicting the same hidden state for every token. A second failed run collapsed the recurrence itself (the loop degenerated to near-identity). The sandwich of learned norms, scaled embeddings, concatenation adapter, and small out-projection init is not decoration; it is what makes the loop a loop.
The run
Trained on the Frontier supercomputer (Oak Ridge; 9408 nodes of AMD MI250X) in 21 segments of up to 12 hours each, mostly in December 2024, on up to 4096 GPUs at 41-51% achievable-FLOP utilization, ~1-1.2M tokens/second, global batch 16M tokens, bf16, data-parallel only (weight sharing keeps the model small enough to skip tensor parallelism). AdamW with β = (0.9, 0.95), lr 5e-4, constant schedule with 4096-step warmup, gradient clip 1.0. Because the LR was constant, extra segments were bolted on whenever allocation appeared, to a final 795B tokens. A feedforward twin (same everything, one pass through the core block) was trained on 180B tokens as the control.
The results, with numbers
Zero-shot lm-eval-harness at 800B tokens, as a function of test-time recurrence r (Table 1 of the paper):
Figure 2 - Huginn-3.5B zero-shot accuracy vs. test-time recurrence
| Model | r | ARC-E | ARC-C | HellaSwag | MMLU | OBQA | PiQA | SciQ | WinoGrande |
|---|---|---|---|---|---|---|---|---|---|
| Ours, 3.5B / 0.8T | 4 | 49.07 | 27.99 | 43.46 | 23.39 | 28.20 | 64.96 | 80.00 | 55.24 |
| 8 | 65.11 | 35.15 | 58.54 | 25.29 | 35.40 | 73.45 | 92.10 | 55.64 | |
| 16 | 69.49 | 37.71 | 64.67 | 31.25 | 37.60 | 75.79 | 93.90 | 57.77 | |
| 32 | 69.91 | 38.23 | 65.21 | 31.38 | 38.80 | 76.22 | 93.50 | 59.43 | |
| Pythia-2.8B / 0.3T | - | 58.00 | 32.51 | 59.17 | 25.05 | 35.40 | 73.29 | 83.60 | 57.85 |
| Amber 7B / 1.2T | - | 65.70 | 37.20 | 72.54 | 26.77 | 41.00 | 78.73 | 88.50 | 63.22 |
| OLMo-2 7B / 4T | - | 82.79 | 57.42 | 80.50 | 60.56 | 46.20 | 81.18 | 96.40 | 74.74 |
Read it honestly. Going from r=4 to r=32 buys +20.8 points on ARC-E and +9.0 on MMLU at zero training cost. But also note the last row: OLMo-2 7B (4T tokens) beats it everywhere, and Huginn’s MMLU at 31.4 is weak in absolute terms. The paper’s claim is about the direction - a knob that exists and works - not about SOTA. Saturation is task-dependent: HellaSwag nearly peaks at r=8, GSM8K keeps improving to r=32. On ARC-C the saturation point shifts later with more few-shot examples: more context in, more loops used. The model spends compute in proportion to what it’s given.
Three findings that didn’t fit in the table:
The gains live in the loop, not the shell. Evaluated at r=1, the model flatlines between the 180B and 800B checkpoints - all improvement from the extra 600B tokens is encoded in the iterated block, not the prelude or coda. Against its feedforward twin at 180B, the recurrent model is already ahead everywhere, and on GSM8K it is 5x better. The authors’ “equivalent to a 50B-parameter compute load” headline comes from this direction of comparison.
Free inference-time features. Because r is a runtime choice, the model gets three things zero-shot that standard transformers need whole research programs for: per-token adaptive compute (exit when the state stops changing - on MMLU the model ponders ~3.5 steps longer on moral-scenarios questions than on high-school math, with no training for this); self-speculative decoding (draft N tokens with few iterations, verify with more - no separate draft model); and KV-cache sharing (cyclic cache budget of 4 steps costs nothing on MT-Bench: 5.86).
The latent space is doing recognizable geometry. Tracking token trajectories s₁..sᵣ: most tokens converge to fixed points, but key tokens in hard questions trace orbits (loops in latent space, observed on arithmetic), and some tokens are sliders - steady drifts in one direction, which can implement iteration counting. These behaviors were never in the training objective; they emerged with scale. And the dynamics are path-independent: re-initialize s₀ and the same orbits and fixed points reappear.
Latent thoughts: the reasoning/memorization split
Saunshi, Dikkala, Li, Kumar, Reddi (Google DeepMind; ICLR 2025) [3] is the cleanest argument for why loops should help reasoning specifically, built in four claims.
Claim 1: reasoning needs depth, not parameters. On three procedurally generated tasks - n-ary addition, p-hop induction (a recursive backtracking version of induction heads), and i-GSM (symbolic math word problems with 7B+ unique solution templates) - a (k ⊗ 12/k) looped model nearly matches the iso-FLOP 12-layer baseline and crushes the iso-param k-layer model, on input difficulty far beyond the training mix. The extreme data point: 1 layer looped 12 times solves addition with 1/12th the parameters. The theory section backs this: problems solvable by iterative algorithms with short descriptions are solvable by looped models at nearly optimal depth - the loop count tracks the problem’s iteration complexity, not its size.
Claim 2: in language modeling, loops trade perplexity for reasoning. This is the paper’s most interesting empirical move. They pretrain 24-layer, 1B-parameter GPT-2-style models on 250B tokens of the Pile, then compare the looped (k ⊗ 24/k) against both baselines on 25 downstream tasks grouped into memorization-flavored (closed-book QA) and reasoning-flavored (open-book QA, math word problems, reasoning primitives) buckets. The numbers for k = 12:
Figure 3 - 1B Pile models: the loop is worse on perplexity, better on reasoning
| Model | Params/FLOPs | PPL ↓ | Closed-book QA | Open-book QA | Math word | All-task avg | Reasoning primitives |
|---|---|---|---|---|---|---|---|
| Baseline (24 ⊗ 1) | 24x/24x | 7.40 | 11.2 | 33.9 | 29.3 | 26.0 | 47.5 |
| Base (12 ⊗ 1) | 12x/12x | 8.16 | 8.2 | 26.9 | 26.7 | 21.8 | 35.7 |
| Loop (12 ⊗ 2) | 12x/24x | 7.90 | 9.3 | 30.8 | 34.3 | 26.5 | 51.2 |
| Middle loop (4 ⊗ 1,4,1) | 12x/24x | 7.81 | 11.0 | 32.3 | 28.3 | 25.0 | 56.5 |
The dichotomy, in one table. The looped model has worse perplexity than the iso-FLOP baseline (7.90 vs 7.40) - memorization capacity tracks parameters, and it has half of them. But it closes 131% of the gap on reasoning primitives (i.e., beats the 24-layer baseline outright with half the parameters), and the middle-loop variant - unique layers at the ends, loop in the center - does even better at 56.5, also beating the baseline on closed-book QA. Their ”% Gap” metric shows the pattern across all k: loops close little of the memorization gap (34-37%) and most or all of the reasoning gap (56-282%). Perplexity and downstream reasoning decouple. Facts live in parameters; multi-step inference lives in depth.
Claim 3: loops generate latent thoughts, and can simulate CoT. Their conceptual framing: a CoT model is a loop that emits one thought-token per iteration; a looped model emits a full n x h latent state per iteration - strictly more bandwidth. Formally, they prove a looped model with a scratchpad can simulate T steps of CoT with T loops. Empirically, downstream accuracy scales with the log of effective depth for both looped and non-looped models, structurally the same law as CoT scaling with generated tokens. Looping is CoT in continuous space, with the same scaling shape and a much wider channel.
Claim 4: you can get some of the benefit without looping. Inspired by the inductive bias, they add a regularizer that pulls successive layers of a non-looped model toward functional similarity. It inherits part of the reasoning benefit with no perplexity cost - evidence that “layers that can survive being iterated” is itself a useful training signal, not just an architecture.
HRM: hierarchy as a stability trick
The Hierarchical Reasoning Model [8] (Wang et al., Sapient Intelligence) attacks a different failure of loops than efficiency: training stability at extreme effective depth. A weight-tied loop is a dynamical system, and a naive one either converges instantly (later iterations become inert) or explodes. HRM’s answer, borrowed from multi-timescale processing in the brain, is two coupled recurrent modules: a low-level module fL that updates fast, and a high-level module fH that updates once per cycle of T low-level steps - and when fH updates, fL’s state is reset to start a fresh phase. They call this hierarchical convergence: the fast module is repeatedly driven to a local equilibrium, the slow module moves only on equilibria, and the system avoids both premature convergence and blow-up. Over N cycles of T steps with 4-layer modules and up to Nsup = 16 deep-supervision segments, the effective depth reaches 4·(2+1)·2·16 = 384 layers - an absurd depth that trains stably at 27M parameters.
No backprop through time. HRM trains with a one-step gradient approximation, justified via deep equilibrium models: if fL converges to a fixed point zL* given zH, the implicit function theorem gives the gradient of the fixed point without unrolling. In practice: backprop only through the last fL and fH step, and between supervision segments the state is detached from the graph. Memory stays O(1) in depth instead of BPTT’s O(T). The TRM paper later showed the fixed-point assumption is barely true at the moment it’s invoked - more on that below.
Deep supervision and halting. Training runs up to 16 segments per example: each segment forward-passes from the previous segment’s (detached) state, computes a loss, updates. A Q-learning head on the final zH predicts Q-values for “halt” vs “continue” - Adaptive Computation Time - so easy examples stop early (under 2 segments on average on Sudoku-Extreme, vs 16 max), while all 16 are used at test time. Architectural odds and ends that matter at this scale: Post-Norm blocks, RoPE, SwiGLU, stablemax instead of softmax for small-data generalization, Adam-atan2 optimizer, fixed random initial state.
The results that went viral: trained from scratch on ~1000 examples per task, no pretraining, no CoT: near-perfect Sudoku-Extreme and optimal pathfinding on 30x30 mazes - tasks where o3-style CoT models score 0% - and 40.3% on ARC-AGI-1 with a 900-token context, above o3-mini-high (34.5%) and Claude 3.7 8K (21.2%). 27 million parameters.
TRM: how much of HRM was actually necessary
Alexia Jolicoeur-Martineau’s Tiny Recursive Model [9] (Samsung SAIL Montréal) is a controlled demolition of HRM’s machinery, and the most instructive paper in the whole area for how to read this literature. Her moves, each ablated on Sudoku-Extreme (1K train, 423K test):
Drop the fixed-point theory entirely. HRM’s one-step gradient is justified by the IFT at a fixed point - but TRM checks the residuals and finds zL is nowhere near converged when the theorem is invoked. So TRM simply backprops fully through one complete recursion process (n updates of the latent plus 1 answer update) and keeps the BPTT-free segments on top. No theorem needed, and generalization jumps: 56.5% → 87.4% on Sudoku-Extreme.
Reinterpret the two features. The biological “hierarchy” story dissolves under inspection: zH is just the current answer (apply the output head and argmax and you read it off), and zL is a latent reasoning scratchpad. So TRM renames them y and z: the network updates the reasoning z given (question x, current answer y, current z), then updates the answer y given (y, z). Why exactly two features? Because z plays the role a chain-of-thought would - drop it and the model forgets how it got its answer; drop y and the model is forced to store its answer inside its reasoning channel. Splitting z into more features hurts (77.6%), collapsing to one hurts (71.9%). Two is the natural number, no brain required.
One network, not two. The low-level update sees x (z ← f(x+y+z)) and the high-level update doesn’t (y ← f(y+z)) - the task is fully specified by whether x is in the input. So a single network can do both jobs: 82.4% → 87.4%, at half the parameters.
Less is more, literally. Adding layers hurt; removing them helped. With only ~1000 training examples, a 4-layer net overfits; a 2-layer net with proportionally more recursions (same effective depth) generalizes best: 79.5% → 87.4%, half the parameters again. Echoes of fixed-point diffusion models, where 2 layers were also optimal.
Attention is optional on small grids. When context length L ≤ hidden size D, a token-mixing MLP (à la MLP-Mixer) is cheaper than attention’s projections and generalizes better: +10 points on Sudoku. It loses on the 30x30 tasks, so TRM keeps attention there - architecture per task, not one architecture for everything.
Cheap halting and stability. HRM’s Q-learning ACT needs a second forward pass; TRM replaces it with a plain binary-cross-entropy “is the answer correct yet” head (86.1% → 87.4%, one pass). And an EMA over weights stops the overfit-then-diverge pattern on tiny data (79.9% → 87.4%).
Bottom line: one 2-layer, 5M-parameter network, T=3 answer updates of n=6 latent recursions each (effective depth 42 per supervision step, up to 16 steps): 87.4% Sudoku-Extreme (HRM: 55.0%), 85.3% Maze-Hard (74.5%), and with the 7M attention variant, 44.6% ARC-AGI-1 and 7.8% ARC-AGI-2 - vs HRM’s 40.3% / 5.0% at 4x the parameters, and vs Gemini 2.5 Pro’s 37.0% / 4.9% and o3-mini’s 34.5% / 3.0%. Only Grok-4-thinking (66.7% / 16.0%, ~1.7T parameters) sits clearly above among LLMs.
Figure 4 - Depth vs. accuracy on Sudoku-Extreme, at matched effective depth
| Effective depth / step | HRM (4 layers) | TRM (2 layers) |
|---|---|---|
| ~9 / 7 | 46.4% | 63.2% |
| 24 / 20 | 55.0% | 81.9% |
| 48 / 42 | 61.6% | 87.4% |
| 80 / 72 | 59.5% | 84.2% |
| 168 / 156 | 57.5% | OOM |
Depth helps until it doesn’t. Both models improve with effective depth up to a sweet spot, then degrade (HRM) or run out of memory (TRM backprops through the full recursion, so deep unrolls OOM). Depth is a dial with an optimum, not a monotonic good - the same saturation shape as Huginn’s test-time curves.
What people are saying
The reaction to this wave has been unusually two-sided, and the critical literature is as informative as the papers.
The ARC Prize Foundation’s audit [13] is the centerpiece. HRM went viral in June 2025 (4M+ views on X threads, 475K+ on YouTube). The Foundation scored it on the hidden semi-private sets and ran ablations. Findings: it approximately reproduces (32% ARC-AGI-1, 2% ARC-AGI-2 on the hidden sets, vs 40.3%/5.0% public) - impressive for the size, not state of the art, and “not material progress on ARC-AGI-2.” More importantly, where the performance comes from: the hierarchical architecture itself had minimal impact over a same-size plain transformer; the “outer loop” of deep-supervision refinement drove the gains (deep supervision roughly doubled accuracy, 19% → 39%; hierarchical recursion added only 35.7% → 39.0%); cross-task transfer was limited - much of the score is memorizing solutions to the specific evaluation tasks, which ARC-AGI supplies at eval time; and task augmentation was critical (though ~300 augmentations sufficed, not the 1000 used). Their conclusion: the approach is “fundamentally similar” to an existing augment-and-refine pipeline, and the brain-inspired framing oversold the mechanism.
TRM’s author says the quiet part too [9,14]: “the question of why recursion helps so much compared to using a larger and deeper network remains to be explained; we suspect it has to do with overfitting, but we have no theory.” And these models are supervised, deterministic input-output machines, not generative models - nothing here writes a poem, and extending the trick to generative tasks is explicitly future work.
Is there a latent chain-of-thought? A follow-up probing study of Huginn [15] (Logit Lens and a purpose-built “Coda Lens” on arithmetic) found limited evidence of interpretable latent CoT: the intermediate loop states do not decode into clean intermediate calculations the way token-level CoT does. The loop is doing something structured (orbits, sliders, path-independent dynamics per the original paper), but it is not a readable reasoning trace. For alignment-minded readers this is the sharpest concern in the whole area: latent reasoning’s advantage - no tokens - is also its audit problem.
The 2026 mechanistics wave is trying to close exactly that gap. The convergence-selection paper [11] argues that standard representational-similarity metrics are “tail instruments”: they provably saturate at the fixed points loops converge to, while the algorithm lives in the head of the trajectory. Their proposed head instrument - convergence-time scaling τ(n,i), the first loop at which position i’s decoded output stabilizes - is validated causally by activation patching, predicts which training seeds generalize (tail metrics don’t), and reproduces on the public easy-to-hard benchmark with 4x exact-match length extrapolation. DeepLoop [10] attacks the stability side: a DeepNorm-style scaling rule for Post-LN looped blocks (α = (2N)1/2, β = (8N)-1/2), derived from a depth-untied bound, with GPT-2-small/medium ablations showing the correction matters precisely when loop count exceeds one. And the ARC Prize analysis plus TRM together reset expectations for the tiny-reasoner claims: recursion works, but via supervision steps, augmentation, and refinement more than via any single architectural idea.
The meta
Why is this resurging now, when the Universal Transformer was 2018? Three forces converged. First, the complexity ceiling got formalized: fixed-depth transformers sit in AC⁰/TC⁰-like classes and provably can’t do polynomial-time sequential computation end-to-end; CoT escaped that ceiling by spending tokens, and loops escape it by spending depth - after o1 made test-time compute the frontier, every lab needed a position on all the ways to spend it. Second, the economics: FLOPs are cheap and getting cheaper, while parameters are what you pay to store, ship, and serve - a loop trades exactly the cheap resource for the expensive one, and the KV cache doesn’t grow with depth. Third, the tooling caught up: randomized-depth training, truncated backprop, and modern init/norm schemes (each rediscovered the hard way, cf. Huginn’s collapsed runs) made loops trainable at scale for the first time.
Who is betting. The affiliations are the tell: Google DeepMind on both the theory (Saunshi et al.) and the adaptive-efficiency side (MoR, with KAIST and Google Cloud); UMD/ELLIS with the open 3.5B proof-of-concept; Samsung SAIL Montréal on tiny recursive reasoners; Sapient Intelligence’s whole company bet on HRM; ByteDance Seed appearing in the recurrent-depth citation graph by 2026 [11]; Meta FAIR adjacent via continuous-thought (Coconut [16]), which loops at the token level by feeding hidden states back as embeddings - the same idea one axis over. This is no longer a niche; it is a named contender for the post-CoT scaling direction, with frontier labs holding positions.
The honest synthesis. Loops buy: test-time compute without tokens, parameter efficiency, depth-on-demand, and strong wins on iterative/verifiable domains. They cost: worse memorization per FLOP (Saunshi’s perplexity gap, Huginn’s 31.4 MMLU), a saturation ceiling set by training depth, harder serving (variable per-token depth vs. batch-parallel hardware; MoR’s expert-choice routing is the current best answer [7]), opaque internal computation, and - per the ARC audit - a tendency for headline claims to decompose into more mundane mechanisms on inspection. The most defensible 2026 position is that recurrence over depth is a real, load-bearing third axis of scale that composes with the other two - not a replacement for parameters, and not (yet) a replacement for chain-of-thought.
Open questions
1. Why does recursion beat a bigger, deeper network at all? TRM’s author names it directly: suspected overfitting control, no theory. Saunshi’s regularizer result hints the inductive bias itself is the active ingredient. Nobody has the account.
2. Saturation. Every test-time curve in this literature bends over - Huginn’s by task, TRM/HRM by depth, with OOM or degradation past the sweet spot. Training-depth distributions, loop-aware normalization (DeepLoop), and curriculum over depth are all being tried; nobody has a curve that rivals token-scaling all the way up.
3. Reading the loop. Latent CoT probing found little that is human-legible; convergence-time instruments are the first mechanistic handle. Faithful auditing of 32 iterations of continuous thought is the alignment problem for this whole direction.
4. Generative tiny reasoners. HRM/TRM are supervised one-answer machines. Making recursive refinement generative (multiple valid answers, open-ended outputs) without losing the small-data magic is unsolved.
5. Systems. Per-token variable depth fights batch-parallel serving. MoR’s expert-choice routing and continuous depth-wise batching are the start; a serving stack that treats loop count as a first-class scheduling dimension doesn’t exist yet.
6. Composition. Depth loops, token loops (CoT/Coconut), tool use, and retrieval are four independent axes of test-time compute. Almost nothing is known about their interactions - multiply, substitute, or interfere. The interesting 2027 paper lives there.
References
- Dehghani, Gouws, Vinyals, Uszkoreit, Kaiser. Universal Transformers. 2018. arXiv:1807.03819
- Lan et al. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. 2019. arXiv:1909.11942
- Saunshi, Dikkala, Li, Kumar, Reddi. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025. arXiv:2502.17416
- Giannou, Rajput, Sohn, Lee, Lee, Papailiopoulos. Looped Transformers as Programmable Computers. ICML 2023. arXiv:2301.13196
- Fan, Lou, et al. Looped Transformers for Length Generalization. 2024. arXiv:2409.15647
- Geiping, McLeish, Jain, et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. 2025. arXiv:2502.05171
- Bae et al. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. NeurIPS 2025. arXiv:2507.10524
- Wang et al. (Sapient Intelligence). Hierarchical Reasoning Model. 2025. arXiv:2506.21734
- Jolicoeur-Martineau (Samsung SAIL Montréal). Less is More: Recursive Reasoning with Tiny Networks. 2025. arXiv:2510.04871
- DeepLoop: Depth Scaling for Looped Transformers. 2026. arXiv:2607.13491
- When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers. 2026. arXiv:2607.20594
- Goyal et al. Think Before You Speak: Training Language Models With Pause Tokens. 2023. arXiv:2310.02226
- ARC Prize Foundation. The Hidden Drivers of HRM’s Performance on ARC-AGI. Aug 2025. arcprize.org/blog/hrm-analysis
- Jolicoeur-Martineau. Tiny Recursion Models (author’s blog post). Sep 2025. alexiajm.github.io
- Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer. 2025. arXiv:2507.02199
- Hao et al. (Meta FAIR). Training Large Language Models to Reason in a Continuous Latent Space (Coconut). 2024. arXiv:2412.06769
- Graves. Adaptive Computation Time for Recurrent Neural Networks. 2016. arXiv:1603.08983
- Bai, Kolter, Koltun. Deep Equilibrium Models. NeurIPS 2019. arXiv:1909.01377
Eternal Horizons · a#evergreen note - permanent, constantly growing. Companion piece to Transformers from first principles. Format modeled on the Transformer Circuits article style.