Parcae: Scaling Laws for Stable Looped Language Models

Created in May 18, 2026

2026   ·   llm   transformer   architecture   scaling-laws   looped-models   ·   paper-review

Paper. Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, Daniel Y. Fu. Parcae: Scaling Laws For Stable Looped Language Models. arXiv:2604.12946, April 2026. UC San Diego · Together AI. [arXiv] · [project page] · [Together AI blog]

0. The Picture in One Paragraph

A standard Transformer is a tall stack of $L$ distinct blocks. You pay for the depth twice: once in parameters (each block has its own weights) and once in compute (each block runs once). What if you could keep just one block of weights but run it $N$ times in a loop? You’d pay parameters for one block and compute for N. This is the looped transformer. It’s been tried before and it tends to blow up during training (the residual stream grows without bound; the loss spikes). Parcae diagnoses why (a spectral-norm argument borrowed from control theory), fixes it with a parameterization trick borrowed from state-space models, and shows that the resulting recipe gives clean scaling laws: a 770M Parcae model matches a 1.3B fixed-depth Transformer.

1. The Background You Need

What a Transformer block does (sketch)

Each Transformer block takes a sequence of vectors — the residual stream $x_t$ — and produces an updated sequence

\[x_{t+1} = x_t + f_\theta(x_t),\]

where $f_\theta$ is attention + MLP + normalization. The “+” is the residual connection: each block adds its contribution to a shared running representation. The depth of a Transformer is just the number of these additions.

What “looped” means

In a looped transformer, you keep one block of weights $\theta$ (or a small group) and apply it repeatedly:

\[x_{t+1} = x_t + f_\theta(x_t), \quad t = 1, 2, \dots, N.\]

The block is reused $N$ times. Parameters stay the same; compute scales with $N$. This is appealing because most of inference cost is compute, and you can dial $N$ up at test time to spend more compute on harder inputs (a poor man’s “thinking longer”).

Earlier work on this idea includes Universal Transformers, CoTFormer, and the recent line of recurrent-depth models (RDMs) the paper positions itself against.

Why it usually breaks

The problem is sitting inside that “$+$”. Every loop iteration adds something to the residual stream. If the added pieces tend to grow, the residual stream explodes after a handful of iterations. In practice this shows up as loss spikes and divergent training runs. Fixed-depth Transformers don’t have this problem because each layer has its own LayerNorm + weights tuned for its specific point in the stack; loops do because the same block is responsible for the update at every depth.

2. The Reframing: Looping as a Dynamical System

The paper’s first move is to stop thinking of the loop as “apply a neural network $N$ times” and start thinking of it as a discrete-time dynamical system over the residual stream:

\[x_{t+1} = A x_t + B u_t + \text{(nonlinearity)}.\]

After a linear approximation of the actual block, the loop looks like a linear recurrence with state $x_t$, input $u_t$ (token-level information injected at each step), and matrices $A, B$ baked out of $f_\theta$. This is exactly the setting where classical control theory has a clean answer:

Spectral norm $\rho(A)$ Behavior
$< 1$ Stable — state stays bounded
$= 1$ Marginally stable
$> 1$ Unstable — state grows without bound (residual explosion)

The empirical finding: in existing looped architectures, $\rho(A) > 1$, and the offending matrices are the injection parameters — the weights that decide how token-level information enters the residual stream at each loop step. That’s the diagnosis.

3. The Fix: Parcae’s Parameterization

Now they need to force $\rho(A) < 1$ by construction, without killing the model’s expressiveness. They borrow a trick from state-space models like S4 and Mamba:

  1. Work in continuous time. Replace the discrete recurrence $x_{t+1} = A x_t + \dots$ with its continuous-time counterpart $\dot{x}(t) = A_c x(t) + \dots$.
  2. Constrain $A_c$ to be negative diagonal. Each diagonal entry is a learned negative scalar. A negative-diagonal $A_c$ has all eigenvalues on the negative real axis, which after discretization gives $\rho(A) < 1$ automatically.
  3. Discretize with a learned step size. Convert $A_c$ back to the actual discrete matrix $A$ using zero-order hold (ZOH) or Euler integration with a step size $\Delta$ that the model learns. The step size lets the loop control how aggressively each iteration updates the residual stream.

The intuition: the negative diagonal makes every iteration a slight decay toward zero, while the rest of the block ($B u_t$ + the nonlinear residual) supplies the actual signal. The residual stream can grow, but only as long as the block is actively pushing — it can’t run away on its own.

Why this is more than a hack. It generalizes the standard residual recipe (which corresponds to a specific, ungoverned choice of $A$) to a small parameterized family of stable recurrences — and the parameters are learned end-to-end.

4. Scaling Laws for Looped Training

Once training is stable, the natural next question is: how should you spend your compute? A fixed-depth Transformer at a given FLOP budget has one main knob (parameters vs. tokens, à la Chinchilla). A looped model has two: parameters/tokens and loop count $N$.

The paper measures this carefully and reports two scaling laws.

4.1 Training-time looping (FLOP-optimal recipe)

When you’re allowed to tune $N$ during training:

Increase the mean recurrence $N$ and the training data together, following a power law.

At a fixed FLOP budget, you do better by training with more loops and fewer tokens than by training with one loop and lots of tokens. The optimal $N$ and the optimal token count both scale as power laws in the FLOP budget, with consistent exponents across the two scales they tested.

4.2 Test-time looping

If you train at $N$ and then dial $N$ up at inference time:

Validation loss decays as a saturating exponential in $N$.

Each extra loop helps less than the previous one. There’s a soft ceiling beyond which more iterations stop helping. This matters for practitioners: you can spend extra inference compute on hard inputs, but with strongly diminishing returns.

4.3 The Pareto frontier

Plotting downstream quality vs. FLOPs, looped models trained at their optimal $N$ sit on a strictly better frontier than fixed-depth models at every FLOP budget the authors tested. Looping isn’t a trick that pays off only in special regimes — it’s Pareto-dominant.

5. The Headline Number

The result everyone is quoting:

A 770M-parameter Parcae model matches the downstream quality of a 1.3B-parameter fixed-depth Transformer trained on the same data — by reusing the same layers across multiple loop iterations instead of adding more weights.

Other numbers:

  • −6.3% validation perplexity vs. prior RDMs at matched parameters and data.
  • −9.1% perplexity on WikiText.
  • +1.8 points on the average of several downstream benchmarks.

This is not a “5%-better-than-the-baseline” paper. The headline is a ~2× parameter-efficiency at downstream quality, with stable training as the enabling condition.

6. Why It Matters

Two beats:

  1. Decoupling parameters from compute. Parcae is one of the cleanest demonstrations that you don’t need to grow the parameter count to grow the compute spent on each token. For deployment (memory bandwidth, KV cache, on-device inference) this is enormous — parameters cost RAM, loops cost only time, and you can choose how much time to spend per token.
  2. The right abstraction for “stability in recurrent depth”. The paper’s contribution is not just an architecture — it’s the diagnosis. Once you frame looping as a dynamical system and look at the spectral norm, the fix is almost forced, and you can reason about future looped designs (different $A_c$ families, different discretization schemes) inside the same framework. That transferability is what makes me bet on this line of work.

7. Limitations Worth Knowing

  • Inference latency. “Parameter-efficient” doesn’t mean “wall-clock-efficient.” A 770M Parcae model that loops $N$ times costs roughly $N \times$ the per-token compute of a 770M fixed-depth model. The win is vs. a 1.3B model, not vs. its own parameter twin.
  • Saturating returns at test time. You can’t keep looping forever for free — the saturating exponential decay caps the practical benefit of test-time recursion.
  • Linearization. The stability argument relies on a linear approximation to the actual nonlinear block. The empirical results show this is enough in practice, but it’s not a global guarantee.
  • Architecture coverage. Reported results are at the 360M and 770M scale. Whether the scaling laws extrapolate cleanly to multi-billion parameter, instruction-tuned regimes is the next paper.

8. The Takeaway for a First Reader

If you remember three things:

  1. Looped transformers reuse the same block $N$ times — fewer parameters, more compute per token.
  2. They blow up because the residual recurrence is unstable; formally, the spectral norm of the linearized update matrix exceeds 1.
  3. Parcae forces stability by parameterizing the update as a discretized continuous-time recurrence with a negative-diagonal generator (a trick from state-space models), which is enough to make a 770M model match a 1.3B fixed-depth Transformer with clean scaling laws.

That’s the whole arc: dynamical-system framing → spectral-norm diagnosis → SSM-style fix → cleaner scaling laws → 2× parameter efficiency.

References



    Enjoy Reading This Article?

    Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition