In September of this year, The Information reported that OpenAI’s upcoming GPT-6 Astra uses a recurrent-depth architecture similar to Huginn (The Information, 2026). Huginn is a 3.5B model open-sourced by Geiping et al. in early 2025 that routes the hidden state repeatedly through a handful of middle layers for every generated token (Geiping et al., 2025). After the story broke, OpenAI chief scientist Jakub Pachocki dismissed it on X as “confused reporting,” clarifying that the computational-graph depth of OpenAI’s current frontier models, Astra included, remains within twice that of GPT-4 (Pachocki, 2026).
More concrete details soon surfaced. On October 6, an X user posted a screenshot of a public Microsoft page showing that GPT-6 Sol and GPT-6.1 Sol share the same base model weights, differing only in how many times they run them at inference: three passes for GPT-6 Sol, two for GPT-6.1 Sol (Algorithon, 2026). The Information later followed up, reporting that OpenAI had deliberately capped Astra’s loop count to keep its chain of thought readable, and that the company’s next flagship model carries the loop design forward (Palazzolo et al., 2026).
All of these leaks point to the same idea: executing a group of layers inside the model multiple times. Looping a Transformer is hardly new. As early as 2018, the Universal Transformer shared weights across layers (Dehghani et al., 2019), yet as language models scaled up over the next several years, giving every layer its own parameters remained the default. The idea returned to the frontier only in 2025, when Huginn (800B tokens) and ByteDance’s Ouro (7.7T tokens) (Zhu et al., 2025) pushed looped architectures to modern pretraining scales. Then in July of this year, BOSS Zhipin’s Nanbeige team open-sourced Nanbeige4.2-3B, a model pretrained on 28 trillion tokens that runs its entire layer stack twice (Nanbeige LLM Lab, 2026). Once static parameter count is decoupled from effective computational depth, what does a model actually gain, and what does it pay in return?
What’s a loop
A standard decoder-only Transformer is a one-way pipeline. It stacks $L$ physical layers in series, each with its own weights, and a token’s hidden state moves sequentially from layer 1 to layer $L$, visiting each layer once.
A looped Transformer rewires that forward path. It groups several consecutive layers into a block; once a vector exits the block’s final layer, it loops back to the first layer and passes through again, carrying forward the state from the previous iteration. After $R$ passes, the result moves on to the rest of the network. Figure 1 illustrates this setup. The two outer panels show the folded physical layout, with only the middle block looping on the left and the entire stack inside the block on the right, while the center panel unrolls the computation graph across passes.
In the unrolled graph, the block’s layers appear multiple times, yet every pass reuses the same physical weights; only the hidden vectors moving between layers change. Let $h_0$ denote the input entering the block and $h_r$ the output after pass $r$. Each pass updates the previous state via $h_r = \text{Block}(h_{r-1})$. Under causal self-attention, a token on pass $r$ attends to earlier tokens’ representations from that same pass $r$. Unrolled, the forward pass looks much like a deeper standard Transformer, except that subsets of layers share tied weights.
Depending on whether unshared layers sit outside the block, architectures generally organize their stacks in one of two ways:
- Sandwich: As shown on the left of Figure 1, a few fixed, independent layers bookend the looped block. Leading prelude layers run once to project input tokens into the block’s representation space, the middle block iterates, and trailing coda layers run once at the end to map the final hidden state to vocabulary probabilities. Huginn, mentioned earlier, uses this layout.
- Full stack: The standalone layers at both ends are dropped, placing every Transformer layer inside the loop. After the final layer, the hidden vector loops straight back to the first layer without passing through the LM head to emit a token, producing an output only after all $R$ passes finish. Both the 2018 Universal Transformer and Ouro follow this design.
This design decouples physical parameter count from actual computational depth. In Figure 1, both models store only four layers’ worth of weights in memory, yet at $R = 3$, each token passes through an effective depth of $1 + 2 \times 3 + 1 = 8$ layers on the left and $4 \times 3 = 12$ layers on the right. Raising $R$ scales computational depth linearly while leaving the memory footprint of the weights unchanged. Parameter count alone therefore reveals little about how deep a model really is, and the “computational-graph depth” Pachocki cited most likely refers to this unrolled effective depth. By the same logic, the Microsoft screenshot showing three passes for GPT-6 Sol does not mean its total depth tripled. That would hold for a full-stack loop, whereas in a sandwich architecture like Huginn, the fixed layers at either end run only once, keeping the overall multiplier below three.
Nor does the pass count $R$ have to be frozen after training. Sampling $R$ randomly for each batch during training forces the model to handle computation trajectories of varying lengths, turning $R$ into a tunable knob at inference: fewer passes for easy tokens or latency-sensitive traffic, more passes when a hard reasoning problem warrants deeper computation. The Microsoft screenshot describing GPT-6 Sol and GPT-6.1 Sol as sharing base weights while running three and two passes most likely refers to this knob.
Why loop
Ask a reasoning model, “What is the capital of the state where Miami is located?” With thinking mode enabled, it spells out the steps in its response: first identifying that Miami is in Florida, then retrieving Tallahassee as Florida’s capital. In the literature, this is a 2-hop question. The first hop finds the intermediate bridge entity (here, Florida), and the second hop uses it to reach the final answer.
Even with thinking mode switched off and Qwen3.6-27B restricted to a single output word, it still answers Tallahassee on the spot, without emitting a single visible reasoning token. If the model genuinely executes both steps internally, the entire derivation must unfold within the single forward pass that produces that word. Reasoning carried out inside the hidden states rather than written out as text is known as latent reasoning (Chen et al., 2025).
Looping sends the hidden state around the model a few extra times, stretching the computation path of a single forward pass. To see what running the same block repeatedly actually buys, we first need to look at what constrains a single forward pass in a standard Transformer, and why conventional remedies fall short.
Latent reasoning
Within a single forward pass, every token’s vector travels upward through the layers along the residual stream (Elhage et al., 2021). Two components divide the work along the way: self-attention moves information horizontally across tokens, while the feed-forward network (FFN) acts like a key-value memory stored in the weights, retrieving by content the facts memorized during pretraining (Geva et al., 2021).
That division of labor imposes a strict sequence. Because each layer computes its attention queries and keys at the start of the layer, information written into the residual stream by one layer cannot be read by other sequence positions until at least the next layer. Returning to Miami, completing both hops internally requires roughly three stages: the FFN at Miami’s position first retrieves “Miami is in Florida” from its weights; attention layers higher up then copy the intermediate concept Florida to the end of the sequence; and only then can an FFN even higher up at the final position look up the capital, Tallahassee.
Since the second hop depends on the output of the first, consecutive reasoning steps have to queue up sequentially along the depth of the network. The later the first hop finishes, the fewer layers remain for the second. Worse, memorized facts are distributed across the FFNs of different layers rather than replicated everywhere. If the fact needed for the second hop (such as “Florida’s capital is Tallahassee”) resides in a lower layer that the hidden state already passed while resolving the first hop, the upper layers have no way to retrieve it.
This layer-by-layer progression can be observed directly inside the model. Vectors in the residual stream are not directly interpretable. The classic logit lens projects an intermediate vector straight through the model’s unembedding head (nostalgebraist, 2020), which mostly surfaces the token the model is about to emit; because Florida is merely a stepping stone that never gets spoken, it rarely shows up this way. The Jacobian lens (J-lens), introduced by Gurnee et al. in July of this year, goes further (Gurnee et al., 2026). By averaging the Jacobian of downstream layers with respect to a given layer’s state over a large corpus, it constructs a fixed linear map that translates intermediate states into final-layer space, revealing which tokens that layer’s contents tend to promote. Feeding the Miami question into Neuronpedia’s public J-lens demo page (Neuronpedia, 2026) for Qwen3.6-27B shows this: at the position immediately following the prompt, the top readout across the middle layers is the unspoken bridge word Florida (Figure 2).
<|im_start|>, the first token after the question. The paper reads the token right before the answer, but with thinking off that token is a line break after an empty think block whose middle layers mostly read line breaks, probably a trait of the thinking template. The swap is the demo’s intervention with its default settings: at layers 30 to 50 of every token, including the ones the model generates, it pushes the hidden state away from what the lens reads as ␣Florida and toward ␣Texas, while the input text stays the same. The model then answers Austin, and underlines mark the tokens whose readouts changed, darker where more layers changed. ␣ marks a space and \n a line break. Data from the Neuronpedia J-lens demo.A readout only shows that Florida appears in the residual stream; proving it participates causally in the computation requires an intervention. Using the demo page’s built-in steering tool to swap the Florida direction for Texas across layers 30 to 50 flips the model’s answer from Tallahassee to Austin without changing the prompt. Across 50 two-hop questions on three Claude models, Gurnee et al. found that substituting the bridge entity shifted the final answer 54% to 70% of the time. On the remaining questions, a direct shortcut from question to answer may operate in parallel, a dual-path setup Lindsey et al. also observed in Claude 3.5 Haiku (Lindsey et al., 2025). Gurnee et al. further showed that swapping the bridge entity alters the answer only when applied in lower layers, whereas swapping the final answer directly works only in higher layers, confirming that the intermediate concept emerges early and gets consumed downstream.
In practice, however, a standard single forward pass frequently stalls before finishing the second hop. Testing LLaMA-2 on more than 40,000 two-hop questions, Yang et al. checked whether each hop occurred by measuring how often internal readouts shifted in the expected direction under input or state perturbations (against a 50% random baseline). Scaling from 7B to 70B parameters and from 32 to 80 layers lifted first-hop success from roughly 71% to 78%, yet second-hop success stayed flat between 61% and 65% (Yang et al., 2024).
Diagnosing the failed examples, Biran et al. found that the model actually knew both facts; it simply resolved Florida too late on the first hop (Biran et al., 2024). By the time Florida entered the residual stream, the lower layers storing the second fact had already executed. When the authors copied a late layer’s hidden state back into an earlier layer at the same token position and ran the forward pass again, 32% to 66% of the wrong answers flipped to right. Because the target layer was selected post hoc, the experiment serves as a diagnostic rather than a practical method, yet it pinpoints the bottleneck. A standard Transformer is a one-way ticket: computation moves strictly upward, and once the forward pass moves past the floor holding a fact, there is no going back.
Stacking depth
The two most obvious ways around this one-way ticket are chain of thought (CoT) and simply stacking more layers. Both run into clear limits.
Chain of thought follows a straightforward logic: if one forward pass is too short, write the intermediate answer out (Wei et al., 2022). Once the model emits Florida as a token, the forward pass for the next token starts fresh at layer 1, putting every lower-layer weight and attention head back in play. CoT can emit as many intermediate tokens as a reasoning chain demands, at a familiar price: each step costs a full forward pass and forces the intermediate state through the bottleneck of a discrete word. CoT and looping also compose cleanly. CoT adds steps while looping deepens each step; the more work a single pass can absorb, the fewer tokens need to be written out, and a looped model remains free to generate CoT.
The alternative is to skip intermediate tokens and build a deeper network. Empirical evidence shows that adding layers does not automatically produce longer reasoning chains. Comparing shallower and deeper models across families such as Llama 3.1, Qwen 3, and OLMo 2, Csordás et al. found that layers in the second half of deep models spend most of their capacity making minor refinements to predictions already formed earlier, so much so that skipping them outright barely changes the output (Csordás et al., 2025). On multi-hop questions, deeper models showed no sign of chaining additional steps either.
At least in these models, stacking extra layers mostly stretches the same computation proportionally across a taller network instead of carving out a distinct second stage of reasoning. That fits the LLaMA-2 finding where jumping from 32 to 80 layers left second-hop accuracy unchanged. The pretraining objective itself exerts little pressure on depth: Kaplan et al. showed that at fixed parameter count, varying the depth-to-width ratio across a 40x range leaves pretraining loss virtually flat (Kaplan et al., 2020). Because tokens that genuinely require multi-step latent reasoning are rare in web text, they barely register in an average next-token loss, so the network likely has little incentive to reorganize labor across layers. Facts stored near the bottom still get visited only once; if the first hop finishes late, dozens of extra layers above cannot recover the missing second-hop fact.
What loop adds
Looping also increases effective depth, except the added depth comes from running the same block multiple times. That reuse gives looped architectures two capabilities that standard layer stacking cannot match.
First, lower-layer knowledge becomes accessible for a second lookup. In a standard model, once the hidden state passes layer 10, layer 10 is gone for good. In a loop, once the state finishes its first pass through the block with Florida encoded in the vector, it routes back to the block’s first layer, placing the bottom attention and FFN layers back in front of it. The architecture effectively builds in the state-copying trick Biran et al. performed by hand, letting the second hop query parameters it missed earlier. In a sandwich layout, the state returns to the first layer of the block, while the single-pass prelude is not revisited.
Small synthetic experiments bear this out. Training an 8-layer Transformer from scratch on two-hop factual reasoning, Wang et al. found that facts seen only in isolation during training were stored exclusively in the lower layers, leaving the upper layers unable to retrieve them on the second hop (Wang et al., 2024). Tying the weights of the first four and last four layers, equivalent to running a 4-layer block twice, allowed the model to master these compositions, though far more slowly. Ouro’s authors similarly observed on synthetic three-hop QA that looped models needed fewer training examples to acquire compositional generalization.
Second, the same transition rule can be applied iteratively. Many reasoning tasks boil down to repeating a single operation: summing a long list of numbers, simplifying an expression tree bottom up, or tracing backward through a chain of variable assignments. A standard deep network has to learn that operation separately in every layer, burning parameters along the way, whereas a looped model learns the rule in one set of weights and iterates until the chain finishes. Saunshi et al. demonstrated this cleanly: on mentally summing 32 three-digit numbers without CoT, a 2-layer network run once scored just 38.8%, while the same 2-layer block looped six times hit 99.5%, nearly matching a 12-layer model with independent weights (Saunshi et al., 2025).
Closer to natural-language scale, Saunshi et al. trained roughly 1B-parameter models on 250 billion tokens. A 4-layer block looped six times matched the depth of a 24-layer standard baseline at one-sixth the parameter count, beating it 56.9% to 47.5% on reasoning primitives like multi-step variable tracking, though at the cost of worse language-modeling perplexity. In a control run, they kept all 24 layers of independent weights and added a regularizer pulling each 4-layer group toward its neighbor; that model also reached 57.2%, with no perplexity penalty. On these tasks, the gain comes from applying similar operations across steps, with strict weight identity mattering little, and weight sharing simply buys that behavior without extra parameters.
Most of these clean wins, however, come from controlled small-scale or synthetic setups. On real language models and natural-language benchmarks, the evidence splits. Evaluating open-source Huginn on GSM8K without written CoT, Lu et al. found that increasing the loop count from 4 to 32 nudged accuracy only from 3.11% to 4.93%, with no gains beyond that, whereas letting the same model generate CoT reached 24.87% and 38.13% under two scoring protocols (Lu et al., 2025). Training over a hundred smaller models at fixed effective depth, Schwethelm et al. found no architectural advantage beyond seed-to-seed noise on either math word problems or reasoning primitives (Schwethelm et al., 2026), clashing with Saunshi et al.'s findings. Because the two studies differ in model scale and token budget, which regime better reflects frontier models remains unsettled.
A further limitation is that we still struggle to tell what each loop pass actually computes. Decoding Huginn’s intermediate states pass by pass with both the logit lens and a coda lens (feeding the block’s output through the coda), Lu et al. saw readouts shift dramatically depending on the probe, with no orderly handoff from intermediate subgoals to final answers across passes. Two-hop questions can be probed and edited as in Figure 2 because the bridge result corresponds to a named entity; open-ended reasoning rarely offers a single token to track. For complex real-world reasoning, the picture of “one hop per loop pass” remains an intuition, and evidence for looping still rests on behavioral metrics like accuracy and sample efficiency.
How to train
As we saw in the previous section, looping calls the same Transformer block repeatedly within a single forward pass, feeding the hidden state from pass $r$ directly into pass $r+1$. Running in place saves parameters, yet it creates serious training traps: Huginn failed twice in a row while scaling up, and Ouro had to cut its planned eight-pass loop in half to four.
Viewed purely as a computation graph, training looks straightforward. Unrolling an $R$-pass loop yields a deep network that chains the same block $R$ times, and backpropagation simply sums the gradients from every pass onto the shared weights, much like tying input and output embedding matrices.
The real difference lies in the context in which those weights are called. In a standard Transformer, layer 5 only ever processes layer 4’s output, with each layer specialized for its slot. In a looped model, from the second pass onward, the block processes its own output from a moment ago. That sets off a chain reaction: any tiny bias in the block’s weights (such as slightly amplifying one feature direction in the hidden state) gets compounded $R$ times within a single forward pass, and a single gradient update alters the behavior of all $R$ passes simultaneously.
Beyond ordinary parameter fitting, training a looped model therefore demands tight control over how the hidden state evolves across passes, keeping it from snowballing out of control while preventing it from freezing early and leaving later passes idling.
Stability
Huginn’s two training failures at the 3.5B scale offer the clearest illustration of how easily state dynamics go off the rails, landing at opposite extremes of a dynamical system.
Pretrained from random initialization, the 3.5B Huginn model uses the sandwich layout introduced earlier: two prelude layers encode input tokens into vectors $e$, four middle layers form the looped block, and two coda layers map the final state to output logits. The first run used the simplest possible setup: adding $e$ straight onto the previous pass’s state without a learnable projection, placing parameter-free RMSNorm before and after every attention and MLP sublayer in the block, leaving input embeddings unscaled, and using a standard peak learning rate of $4\times10^{-4}$. Training loss quickly stalled. Diagnostics revealed representation collapse: token vectors across different positions in the same sequence grew increasingly similar until their correlation hit 1.0, leaving all positions nearly identical and unable to predict the next token. The authors traced this to the loop itself. Attention mixes information across positions on every pass, and running the same block dozens of times compounded that mixing until the entire sequence collapsed into a single state, a failure mode that never appeared in smaller runs and surfaced only at 3.5B.
On the second run, the team modified the architecture to prevent collapse, concatenating the state and input through a learnable adapter, switching to standard pre-norm, and scaling the input embeddings. Collapse subsided after the first 150 steps, only for training to swing to the opposite extreme: the model learned to coast. Validation perplexity was identical whether the model ran 1 pass or 32. As the authors diagnosed it, the network learned early on to ignore the incoming recurrent state, turning the remaining 31 passes into dead weight.
On the third attempt, keeping the adapter and embedding scaling, reverting to pre- and post-sublayer normalization, and cutting the peak learning rate tenfold to $4\times10^{-5}$ finally carried a 750-billion-token run to completion, keeping token correlations healthy and steadily lowering perplexity with additional passes. Because several changes were introduced at once across the three runs, the paper does not isolate the individual contribution of each tweak.
To uncover the mechanics behind blowing up on contact and playing dead in place, Prairie et al. decomposed the per-pass block update when introducing Parcae in April of this year (Prairie et al., 2026).
On pass $r$, the block draws on at most two ingredients: the previous state $h_{r-1}$ and the original input $e$ encoded by the prelude. Both Huginn and Parcae re-inject $e$ into the block on every pass, whereas Ouro omits re-injection and feeds only the prior state, making this an architectural choice. Huginn’s authors liken re-injection to gradient descent, which must consult the data at every step for its solution to depend on that data; if the input appears only on pass one, an iteration that converges toward a stable state will settle somewhere independent of the prompt, forgetting the original question.
Parcae splits the mixing of these two inputs and the subsequent computation into a linear term and a nonlinear residual:
\[h_r = A h_{r-1} + B e + \mathcal{R}(h_{r-1}, e).\]Here $A$ governs how much of the previous state carries over linearly, $B$ controls how the input $e$ is injected, and $\mathcal{R}$ captures the remaining nonlinear computation, primarily from the attention and MLP sublayers.
Setting the nonlinear term $\mathcal{R}$ aside for a moment reduces state evolution to a pure linear dynamical system, where any small perturbation in the prior state is multiplied by $A$ on each pass. Much like repeatedly multiplying a scalar by $a$, perturbations shrink and vanish whenever $\lvert a \rvert < 1$, and blow up exponentially once $\lvert a \rvert > 1$. For a matrix, that amplification factor is the spectral radius $\rho(A)$, the largest absolute eigenvalue of $A$. When $\rho(A) > 1$, at least one eigen-direction grows with every pass, causing the state norm to diverge whenever it has any projection along that direction. When $\rho(A) < 1$, perturbations decay from any starting point and the linear iteration converges to a unique equilibrium, or fixed point, satisfying $h^\ast = A h^\ast + B e$. These properties describe only the linear system; a full block pass also includes $\mathcal{R}$.
To be clear, converging to a fixed point here means a single token’s internal representation stabilizes as the loop count grows. That differs from the representation collapse in Huginn’s first failure, where distinct tokens blurred together; because each token has its own input $e$, each converges to a distinct fixed point. It also differs from training loss convergence, where loss stops falling across optimizer steps.
Viewed through this simplified model, earlier training failures become much easier to read. Adding $e$ directly onto the state sets the merge matrix $A$ to the identity, putting $\rho(A) = 1$ right on the stability boundary, with residual connections inside the Transformer layers adding further contributions of their own. Huginn’s later adapter placed no constraint on $A$ at all. Prairie et al. observed that in every diverging run, the learned spectral radius crept to 1 or above, causing the state norm to explode across passes.
Parcae applies a direct fix: parameterizing $A$ structurally as a diagonal matrix with entries strictly inside $(0, 1)$ to lock in $\rho(A) < 1$, while normalizing $e$ before multiplying by $B$. This bound governs only the merge step; whether a full pass remains stable still depends on the full update including $\mathcal{R}$, which $\rho(A) < 1$ alone cannot guarantee. Its payoff is empirical: with this constraint in place, Parcae converged smoothly even at an aggressive peak learning rate of $10^{-3}$ (where the unconstrained design broke down at $4\times10^{-4}$), late-training loss spikes after roughly 170k steps vanished, and validation perplexity dropped by up to 6.3% below prior looped models matched on data and parameter count.
Architectures without explicit spectral constraints, such as Ouro, have to walk an engineering tightrope instead: cutting passes back from eight to four after hitting loss spikes and gradient oscillations, ramping batch size from 4M to 8M tokens to smooth gradients, and using a smaller learning rate than a standard Transformer of the same size. Both spectral constraints and conservative learning rates work by holding down perturbation amplification. Yet as Huginn’s second failure showed, the opposite failure mode lurks just as close: that run never diverged, yet later passes contributed nothing to predictions. An iteration that damps per-pass updates too aggressively can easily turn dozens of trailing passes into empty idling.
Starting point
Huginn, Ouro, and Parcae all opted for full pretraining from random initialization. The difficulty with starting from scratch is that the model has to solve two hard problems at once: absorbing vast amounts of language and world knowledge, and staying stable while running the same block dozens of times in a row.
Separating the two problems makes the appeal of an alternative clear. Pretraining standard Transformers is already a mature playbook with off-the-shelf checkpoints everywhere, whereas the divergence and idling issues just discussed come entirely from the second challenge. A natural shortcut is to start from a pretrained standard Transformer, carve out a subset of layers as the shared block, and teach it to loop through continued training.
Bae et al. tested this conversion strategy on Gemma 2B (Bae et al., 2024). Starting from the 18-layer base model, they compressed it into a 9-layer block run twice to build a half-size Recursive Gemma 1B. Mapping two original layers onto a single shared layer raises an obvious initialization question. Under strict weight sharing, subsampling half the layers at a fixed stride while keeping the first and last layers worked best. Relaxing sharing slightly by adding a small per-pass LoRA on top of the shared base allowed them to average each pair of corresponding layers for the base weights and initialize the LoRA from a truncated singular value decomposition (SVD) of their difference. With just 15 billion tokens of continued training, the converted model reached 51.7% average few-shot accuracy, far ahead of a parameter-matched model trained from scratch on the same recipe (38.2%).
McLeish et al. used a split closer to Huginn’s sandwich layout (McLeish et al., 2025). Slicing the 22-layer TinyLlama into segments, they kept the first four layers as the prelude, layers 10 through 17 as the looped block, and the final four layers as the coda, discarding the six remaining layers in between. Applying the same recipe to Llama-3.2-1B yielded a loss after 120 billion tokens of continued training that remained markedly lower than a matched architecture trained from scratch, which scaling curves estimated would need at least 950 billion tokens to catch up.
Retrofitting a pretrained checkpoint comes with its own hurdles, however. Layers in a standard LLM are trained for a fixed depth and a single pass per token, never repeated application. Chen et al. showed that forcing middle layers of a frozen model to loop in place without training usually degrades performance (Chen et al., 2026); only after breaking each pass into damped micro-steps to curb state drift did they coax a 2.64-point gain on MMLU-Pro out of Qwen3-4B-Instruct.
Post-conversion training therefore has to juggle several jobs at once. First comes recovery: dropping or subsampling layers disrupts existing language-modeling fluency, so McLeish et al. ran 26 billion tokens of plain web text to let the model heal before mixing in math reasoning data, beating immediate joint training by 5 points on ARC-Challenge. Second, stored knowledge and representations continue to adapt during this phase: at matched training compute, the original model’s GSM8K accuracy plateaued as extra compute was added, while the converted model kept climbing. Finally, these pretrained layers still have to learn iterative stability; during conversion training, standard AdamW at a $5\times10^{-5}$ learning rate hit mid-run loss spikes and diverged to NaN, stabilizing only after switching to the Muon optimizer.
At first glance, this seems at odds with the earlier point that training from scratch is harder, yet the two comparisons start from different baselines. Bae and McLeish et al. matched models on additional post-conversion tokens, giving the retrofitted model a head start from its full pretraining run. When training Nanbeige4.2-3B, the Nanbeige team asked a different question: if you are already paying for a full pretraining run, is it better to train a standard Transformer first and retrofit it later, or train the looped architecture from step one? In their experiments, training the loop from scratch came out clearly ahead. As their report explains, learning from random initialization lets internal representations adapt gradually to repeated layer reuse over the entire pretraining trajectory, whereas forcing a loop onto weights already settled into single-pass habits misses that adaptation window. Retrofitting thus makes sense when you already have a checkpoint and a modest continued-training budget, while full-scale pretraining from scratch should include the loop from day one.
Training depth
Whether weights start from random initialization or a retrofitted checkpoint, training immediately forces an unavoidable design choice: how many loop passes should the model run on each training step?
If every training step rigidly executes a fixed $R$ passes, the loss lands solely on the output of pass $R$, leaving passes 1 through $R-1$ without any direct gradient supervision. A model trained that way becomes brittle. Turning the pass-count knob dynamically at inference according to problem difficulty (fewer passes for easy inputs, more for hard ones) requires exposing the model to a range of loop depths during training.
Teams achieve this depth generalization in different ways. Huginn samples a random pass count per micro-batch from a heavy-tailed distribution centered around 32, though sharing one depth across an entire batch makes gradient norms fluctuate from step to step with the sampled depth. Parcae goes further by sampling an independent pass count for every example inside the same micro-batch, blending shallow and deep gradients in each update, which smoothed out training loss spikes and improved low-pass performance. Ouro skips random sampling altogether: every example runs all four passes, with an LM head attached at the exit of each pass to compute a loss, forcing the model to produce valid next-token predictions after 1, 2, 3, and 4 passes alike.
How far that inference knob can turn depends closely on the depth range seen during training. Ouro never ran beyond four passes in training, and pushing its 1.4B reasoning model to eight passes on AIME 2024 dropped accuracy from 65.0% to 38.7%. By contrast, Huginn’s heavy-tailed sampling occasionally explored well beyond 32 passes during training, so running more than 32 passes at inference stays closer to familiar territory. How sharply a model degrades outside its training horizon varies with the architecture and training recipe.
Pushing up the training pass count quickly triggers a memory crunch. Running $R$ passes forward is unavoidable, and differentiating faithfully through all of them requires caching intermediate activations across every pass while scaling backward compute linearly with $R$. Whenever Huginn sampled an unusually deep trajectory from its long tail, memory usage spiked well above average.
Huginn solved this with truncated backpropagation through time (truncated BPTT). The forward pass still executes all $R$ passes, but only the final eight passes retain their computation graph for backpropagation. Between pass $R-8$ and pass $R-7$, the backward chain is severed: earlier passes advance the hidden state without storing activations, and no gradients flow back through the state into those iterations.
Activation memory and backward compute are thus capped at eight passes regardless of sampled depth, leaving only forward compute to scale with $R$. The shared block still receives rich gradients from the final eight passes, and the prelude remains well trained because its output $e$ is re-injected on every one of those last eight iterations. To curb the forward cost of deep trajectories early in training, McLeish et al. borrowed from gradual stacking, ramping the mean pass count smoothly up to 32 over the course of training to reach a lower validation loss for the same total compute.
An immediate worry is whether dropping gradients from the first $R-8$ passes biases the optimization.
That brings us back to the contraction analysis from earlier. A simple scalar recurrence, $h_r = a h_{r-1} + b$ (with $b$ representing the input injected on each pass), makes the intuition clear. After $R$ passes, the derivative of the final state $h_R$ with respect to $b$ forms a geometric series:
\[\frac{\partial h_R}{\partial b} = \sum_{j=0}^{R-1} a^j.\]The $j$-th term captures the echo of the input injected $j+1$ passes from the end after $j$ rounds of scaling. Backpropagating through only the last $K$ passes retains the first $K$ terms of this sum.
Whenever the iteration is contractive ($\lvert a \rvert < 1$), earlier terms decay exponentially, so the discarded tail shrinks on the order of $a^K$ and truncation loses almost nothing. Conversely, if $\lvert a \rvert \ge 1$, the earliest terms are at least as large as the recent ones (and strictly larger for $\lvert a \rvert > 1$), meaning truncation throws away the dominant part of the gradient. In a full looped model, $a$ is replaced by the per-pass state Jacobian, which varies across iterations. Huang et al. derived the corresponding result (Huang et al., 2026): near a stable fixed point, backpropagating through only the last $K$ passes matches the first $K$ terms of the full gradient series, with truncation error decaying geometrically in $K$.
In short, truncated backpropagation safely discards early gradients only when the retained passes operate inside a contractive regime; far from a fixed point, or whenever updates amplify perturbations, that approximation loses its guarantee.
Early exit
Once a model handles variable depths and its states settle smoothly under contractive updates, a clear inference optimization opens up. Tokens vary in difficulty, and once a token’s prediction stops changing, pushing it through the remaining passes mostly wastes compute. Letting easy tokens exit early saves FLOPs for harder ones; the catch is knowing when a token is ready to stop.
Two main strategies have emerged. Huginn relies on an unsupervised heuristic requiring no extra training: it compares the vocabulary probability distributions from the current and previous passes, halting as soon as their KL divergence drops below $5\times10^{-4}$. Across MMLU subjects, the model brakes early on high school math while spending an average of 3.5 extra passes on moral scenarios; on MT-Bench, early exit virtually matched a fixed pass count (5.56 vs. 5.63). Because this criterion watches only the output distribution, the hidden state may still be evolving along directions invisible to the unembedding layer, or the model may simply have locked onto a confident wrong answer.
Ouro takes a supervised approach, attaching a learned brake made of a linear layer and a sigmoid to every pass to predict whether further computation will meaningfully reduce the token’s loss. At inference, stopping probabilities accumulate across passes until they cross a threshold $q$. On MMLU, where fixed one-pass (around 40%) and four-pass (67.35%) runs sit far apart, this gate reached 66% while averaging just 2.5 passes. Mixture-of-Recursions (MoR), proposed by Bae et al., pulls routing decisions directly into training (Bae et al., 2025). Before each pass, a router scores all tokens and advances only the top-scoring fraction, restricting both training and inference compute to active tokens. Selecting top-k across a full sequence introduces a causality snag, however, since whether a token makes the cut depends on future tokens not yet generated during autoregressive decoding. MoR resolves this by adding an auxiliary training loss that teaches the router to predict top-k selection from the current token alone, allowing independent per-token routing at inference.
Dropping the learned gating network in favor of a simple rule, halting whenever the hidden state shifts by less than a threshold between consecutive passes, lands within 1 to 2 percentage points of the trained gate at an average of two to three passes. Most of the signal needed to decide when to stop is already encoded in whether the state is still moving.
In production autoregressive decoding, however, early exit creates a thorny engineering side effect: gaps in the KV cache.
Under causal attention, a token on pass $r$ attends to the keys and values computed by all preceding tokens on that same pass $r$. In a standard model, every token traverses the exact same depth. Under early exit, suppose an earlier pronoun finishes and exits at pass 3 while a later verb keeps deliberating through pass 10: where does the verb look for the pronoun’s keys and values on passes 4 through 10?
Three workarounds have emerged for these missing cache entries. The first is Huginn’s fallback reuse: if an earlier token exited at pass 3, deeper passes simply reuse the deepest KV pair it produced, the one from pass 3. Because every pass applies the same K and V projection matrices, those vectors live in a shared space, though states from different passes are not identical, so whether the substitution holds up is an empirical question (the MT-Bench score cited above used this scheme). The second is MoR’s approach of hiding exited tokens from deeper passes altogether: pass $r$ computes attention exclusively among tokens still active at pass $r$ and caches only their keys and values, so later tokens never query words that have already exited. The third is the final-state-only cache proposed by Huang et al. Because their trained model converges close to a fixed point, they retain only the final pass’s KV pair for each token during decoding, letting subsequent tokens consult that single cache regardless of their own pass depth. Whereas storing a separate cache per pass scales memory linearly with effective depth, this trick cut KV cache memory by two-thirds at their five-pass training depth on models under 1.6B, raising perplexity by at most 0.85%.
Taken together, this toolkit of training and inference techniques hinges on how the hidden state evolves across passes, yet each piece imposes a slightly different demand. Preventing training divergence requires perturbations not to grow across passes; truncated backpropagation saves memory only when early passes’ gradient contributions decay; inference early exit needs an observable signal that extra passes will barely move the output or state; and final-state KV compression actually requires the state to sit near a fixed point.
Beneath these requirements lies a tension built into looped models. The whole reason to introduce a loop is so that extra passes perform fresh computation; yet once the state settles onto a fixed point, $h_r$ and $h_{r-1}$ become nearly indistinguishable, turning subsequent iterations into dead cycles. Huginn, trained with an average of 32 passes, still improves on ARC-Challenge when scaled from 4 to 16 passes before flattening out past 16, a plateau that likely reflects this effect, while its early run where 31 extra passes yielded zero gain is the extreme case of wasted iterations.
Taming a looped model means threading the needle between these two poles: using architectural constraints and training hyperparameters to damp perturbations so iteration stays stable and gradients truncate cleanly, while giving the hidden state enough room to travel before settling so later passes still have useful work to do.
Why not
In principle, the big draw of looping is turning computational depth into a free dial: keep parameter count fixed, spin the block a few more times, and get deeper reasoning, just as early Huginn pushed its average depth to 32 passes. Look at models trained at production token scales, however, and the loops actually deployed are remarkably shallow. Nanbeige4.2-3B, pretrained on 28 trillion tokens, runs its stack only twice, and its technical report notes that going past two passes yields minimal capability gains while noticeably slowing and destabilizing training; GPT-6.1 Sol in the Microsoft screenshot likewise runs two passes, and ByteDance’s Ouro stepped back from a planned eight passes to four. If looping can deepen a network indefinitely on paper, why have shipped models all stopped at two to four passes? Answering that means tallying the bills that looping brings along.
Compute
Folding a 16-layer network into four layers looped four times saves memory only for storing weights; every forward pass still executes 16 layers of matrix multiplications, cutting FLOPs by zero. Under a fixed training compute budget, that trade looks unfavorable. In an iso-FLOP comparison, Schwethelm et al. found that a 410M-parameter model looped four times matched the per-token compute of a 1B standard model, yet its pretraining loss only equaled that of a 580M standard baseline. In a dense architecture, spending FLOPs on extra passes over the same weights lowers pretraining loss less efficiently than spending that budget on more independent parameters.
Trailing a 1B model in language-modeling loss despite spending 1B FLOPs most likely traces to memorization: a major share of pretraining loss reflects how much world knowledge a model stores, and storage capacity scales with unique parameter count rather than computational depth. In Ouro’s biography memorization benchmark, information stored per parameter stayed flat at roughly 2 bits whether the model ran one pass or four; evaluating the same weights four times does not let them store four times as many facts. That same trade-off likely explains why Saunshi et al.'s looped model beat an equal-depth standard baseline on reasoning tasks while posting worse perplexity: looping lengthened the reasoning chain at the expense of independent layers that would otherwise hold knowledge.
If dense loops lose at iso-compute because they lack the parameter capacity to store enough facts, offloading memorization to a mixture of experts (MoE) offers a natural fix. MoE expands total parameter count through a large pool of experts while activating only a small slice per token, supplying the missing storage capacity without inflating per-token FLOPs. The SMELT team put this idea to the test (Wang et al., 2026). Looping the middle half of an MoE model twice while narrowing hidden width, expanding the expert pool, and retuning attention heads, they matched a standard MoE baseline on per-token FLOPs, total non-embedding parameters, and KV cache footprint. Scaling this setup to 54 billion non-embedding parameters yielded 6.8% to 18% training compute savings along the compute-optimal frontier (a gain that depends on co-tuning width, expert count, and looping together, rather than bolting a loop onto an existing model). Read side by side, the two lines of experiments suggest that whether looping wins or loses on compute most likely depends on whether the capacity gap is filled: in parameter-constrained dense models, looping crowds out factual storage, whereas pairing it with sparse experts lets layer reuse pay off in training FLOPs.
Latency
Beyond training FLOPs, inference latency and memory bandwidth often decide whether a model can actually be served. Because a loop executes the same block $R$ times in series, with each pass waiting for the prior pass to finish, per-token generation latency grows linearly with pass count. Memory pressure is even trickier. Storing a separate KV cache for every pass expands cache footprint in proportion to effective depth, quickly eating back the memory saved by weight sharing. Researchers have tried compressing intra-loop caches to close that gap, though at large scale that can hurt quality: the Nanbeige team tested sharing a single KV cache across both passes, cutting cache memory in half, yet the shared-cache variant trailed the full per-pass cache across benchmarks.
With every pass costing real latency and cache memory, the adaptive pass count sketched earlier, running few passes by default and more only on hard queries, looks like an appealing way to bring average latency down. In current implementations, though, that path is narrower than it appears. For one thing, dynamic early exit in production easily saves nominal depth without saving actual compute. During batched GPU inference, sequences in a batch halt at different depths, yet every decoding step advances all sequences in lockstep and finishes only when the slowest token completes. As long as a single token’s gate keeps running, tokens that exited early sit idle alongside it, and at larger batch sizes nearly every step contains at least one token that runs the maximum number of passes (Figure 3). Even Ouro’s open-source inference code runs all four passes unconditionally before using the gate signal to select an earlier output (ByteDance Seed, 2025).
Even in single-request serving where pass count can be tuned freely, the useful operating window remains narrow. As seen in the training-depth section, dialing the pass count past the ceiling covered in training causes accuracy to drop steeply; within the training window, a model like Huginn, trained with an average of 32 passes, already flattens out on ARC-Challenge past 16 passes. When marginal accuracy gains top out early while every additional pass adds its full share of latency, deployment usually has to keep pass counts within a tight band.
Monitorability
When The Information reported in September that GPT-6 Astra uses a recurrent-depth architecture, Redwood Research’s Ryan Greenblatt posted on X that it “may be the single worst development for AI security/safety to date” (Wright, 2026). A clear argument explains why an architectural tweak set off such a loud alarm. In a standard reasoning model, a single forward pass has a fixed layer budget, so carrying out an extended multi-step plan forces the model to externalize intermediate steps as chain-of-thought (CoT) tokens for subsequent passes to read back, giving auditors and automated monitors a window into its intentions. A position paper signed by Korbak and over forty co-authors sounded a warning built on this premise (Korbak et al., 2025). Calling out latent-recurrence designs such as Huginn and Coconut (Hao et al., 2024), they argued that once upper-layer states can loop back to lower layers without passing through discrete tokens, long serial chains no longer require a readable CoT, closing off that monitoring window.
That concern identifies the core mechanism behind CoT monitoring, yet following our computation-graph breakdown reveals two distinctions worth keeping separate. First, reasoning inside hidden states without emitting text is hardly new to looping. As we saw earlier, non-looped Qwen3.6-27B answers the capital of Miami’s state directly with thinking disabled, keeping the bridge word Florida entirely inside the residual stream; standard Transformers already perform latent reasoning up to a serial ceiling of $L$ layers per pass, spilling into tokens only when derivations exceed that budget. Second, looping the stack $R$ times raises the ceiling on serial computation within a forward pass, yet it raises it by a fixed constant factor. At the rumored two or three passes, the longest serial path in a single forward pass doubles or triples, remaining strictly bounded (Figure 4). What actually removes the length ceiling is cross-token latent recurrence like Coconut, which Korbak et al. cite in the same breath: by feeding the final-layer hidden state of one step directly as the input to the next, reasoning can propagate across arbitrarily many positions without ever touching text. Lumping a two- or three-pass loop within a single forward pass together with an unbounded cross-token loop overstates the architectural shift.
The GPT-6 Astra system card provides concrete measurements from a deployed frontier model (OpenAI, 2026). Compared with its predecessor, Astra’s CoT monitorability did drop noticeably, driven mainly by shorter, sparser chains of thought and more frequent minimal or empty reasoning traces; on examples where Astra still produced longer CoTs, monitorability largely recovered. Alongside that shift came a sharp jump in the model’s ability to solve problems without emitting CoT at all. Independent evaluations by the UK AI Security Institute found that with reasoning output completely disabled, Astra solved math problems equivalent to 30.9 minutes of human effort, compared with 3.6 minutes for GPT-6 Sol of the same generation. Astra’s CoT controllability, the share of cases in which the model follows instructions about what its chain of thought may contain (for instance, never using a given word), reached 60.9%, against 23.2% for GPT-6 Sol; the higher it is, the more easily a model could reshape its CoT to slip past a monitor. OpenAI noted in the report that it was quite confident the change in CoT controllability stemmed from something other than the architectural change, tracking instead with stronger no-CoT capability. Korbak, who signed the original warning and also works at OpenAI, echoed this in response to press coverage, attributing the drop in monitorability primarily to increased capability while reiterating deep concern over losing transparent chains of thought.
Treating loops as the end of monitorability also rests on an unspoken assumption: that adopting a loop stops a model from writing CoT, or renders its intermediate states inherently opaque. Neither has materialized in models disclosed so far. Astra still emits chains of thought, albeit terser ones; Nanbeige4.2 deliberately retained a thinking mode in post-training and used reinforcement learning to regulate reasoning length; and Ouro released a dedicated Thinking variant that outputs its reasoning. Nor are recurrent hidden states doomed to remain a black box. With LOTUS, Fan et al. supervised each loop pass’s hidden state using steps from a standard CoT, enabling the post-loop vectors to decode straight into the corresponding reasoning steps through the unembedding head (Fan et al., 2026). That squares cleanly with Lu et al.'s inability to extract orderly steps from unsupervised Huginn: whether intermediate states are legible depends heavily on the training objective, and the loop topology itself has not welded that door shut.
Even when stronger models write far less of their reasoning down, CoT monitoring still retains bite. Researchers have long observed that written CoTs do not always reflect a model’s true decision process: planting subtle hints in prompts, Chen et al. found that when Claude 3.7 Sonnet and DeepSeek R1 changed their answers using a hint, they acknowledged it in their CoT only 25% and 39% of the time on average (Chen et al., 2025). Those tasks, however, were solvable without CoT in the first place, making the written text look more like post-hoc rationalization. Emmons et al. drew a sharp line between post-hoc rationalizations and CoTs serving as a genuine computational scratchpad (Emmons et al., 2025). When a complex or dangerous task can only be solved by using the CoT as a working scratchpad, a model struggles to complete the task while simultaneously fooling a monitor, though the authors cautioned that once training or architecture allows such computation without natural language, evasion becomes far easier. Astra’s evaluations bear out the first half of that claim: the model showed no capacity for hiding steganographic messages in its CoT, suggesting that evasion risk may be concentrated mainly on tasks already solvable without CoT.
Taken as a whole, the evidence suggests that treating looping itself as the worst news yet for AI safety overstates the threat. Latent reasoning predates looped architectures, and looping has not stopped models from emitting chains of thought; it simply multiplies the serial depth of a single forward pass by a finite constant. Rather than fixating on whether a model shares weights across passes, safety work is better served by tracking how complex a task a model can solve without writing CoT, and which high-risk behaviors still require CoT as a working scratchpad. Deeper loops do push the boundary of silent capability outward, yet they push in the same direction as larger parameter counts and stronger reinforcement learning, making this risk one cost among several rather than an anomaly that breaks safety defenses.
Outlook
Returning to the leaks at the start of this post, the picture they sketch is notably restrained: GPT-6 Sol running two and three passes in the Microsoft screenshot, Astra with its reportedly capped loop count, and open-weight Nanbeige4.2-3B likewise running two passes. Frontier practice has avoided folding dozens of layers endlessly, settling instead on shallow loops of two or three passes. Within that band, weight reuse can buy extra effective depth, enabling a second look at lower-layer facts and repeated application of the same computational rule, while scaling up the base model or adding sparse experts compensates for memory capacity and keeps serial latency, KV cache overhead, and latent-reasoning safety risks within manageable bounds.
How much additional reasoning power two or three shallow passes actually unlock at frontier scale remains an open empirical question. How much of the striking multi-hop and algorithmic gains seen in small controlled experiments carries over to the complex distribution of large language models awaits more public controlled experiments.