LeWorldModel — A 15M-Parameter JEPA That Actually Trains End-to-End from Pixels

Paper. Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312, March 2026. [arXiv] · [project page] · [code]

0. The Picture in One Paragraph

The Joint-Embedding Predictive Architecture (JEPA) has been LeCun’s proposed answer to “how do we learn world models from pixels without generative modeling?” The catch is that JEPA training is notoriously fragile: the encoder and predictor have a trivial fixed point — map every input to the same vector and the prediction loss is zero — so prior JEPAs have leaned on stop-gradient targets, EMA updates, predictor asymmetries, and six tunable loss hyperparameters to keep representations from collapsing. LeWorldModel replaces all of that with a single regularizer (SIGReg) that pushes the embedding distribution toward an isotropic Gaussian, leaving a clean two-term loss with one tunable weight. At ~15M parameters trainable on a single GPU in a few hours, the resulting world model outperforms PLDM and DINO-WM on PushT (96% success, +18% over PLDM, beats DINO-WM even without proprioception) and plans ~48× faster at inference (1s vs 47s per planning cycle on an L40S).

1. The Problem: Why JEPAs Are Hard to Train

A JEPA learns by predicting the embedding of the next observation, not the next observation itself. The setup is appealing — no pixel-space reconstruction, no GAN, no diffusion — but it has a structural failure mode:

If the encoder maps every input to the same constant vector, the predictor trivially achieves zero loss. Representation collapse.

Existing JEPAs (I-JEPA, V-JEPA, DINO-WM) prevent this collapse with architectural and training-side asymmetries:

• EMA target encoder — the target embedding comes from an exponentially-averaged copy of the encoder, not the encoder itself.
• Stop-gradient on the target — gradients only flow through the predictor side.
• Predictor asymmetry — the predictor must be cheaper / smaller / bottlenecked relative to the encoder.
• Multi-term loss with six tunable hyperparameters in the best prior end-to-end alternative — variance, covariance, invariance, prediction, plus weights and schedules.

Each of these is a workaround for the same underlying problem: the encoder has no direct pressure to use the full latent space. They work, but they make the training recipe brittle, hyperparameter-heavy, and hard to scale.

2. The Move: Make Collapse Impossible by Construction

LeWM’s contribution is a single regularizer that directly forbids the collapse mode, and lets everything else (EMA, stop-grad, predictor asymmetry) go away.

The intuition: a collapsed encoder produces a degenerate distribution over embeddings — a delta at one point, or a line in latent space. If we force the embedding distribution to be an isotropic Gaussian $\mathcal{N}(0, I)$, collapse is mathematically impossible. The distribution has full-rank covariance by construction.

The technical question is how to enforce that without writing down the whole $D$-dimensional density. That’s where SIGReg comes in.

2.1 SIGReg — Sketched Isotropic Gaussian Regularization

SIGReg uses two classical results:

• Cramér-Wold theorem (1936). A multivariate distribution equals $\mathcal{N}(0, I)$ if and only if every one-dimensional projection of it is $\mathcal{N}(0, 1)$.
• Epps-Pulley test. A standard one-dimensional normality test with a closed-form characteristic-function-based statistic.

The recipe:

1. Sample $M$ random unit directions $u_1, \dots, u_M \in \mathbb{R}^D$.
2. For each direction, project the batch’s embeddings onto $u_m$ to get a 1-D distribution.
3. Apply the Epps-Pulley statistic to compare each projection to $\mathcal{N}(0, 1)$.
4. Sum (or average) the $M$ statistics into the SIGReg loss.

                      M random directions u_m
                           │
                           ▼
  embeddings  z ∈ ℝ^D ─→ ⟨z, u_m⟩ ∈ ℝ ─→ Epps-Pulley vs N(0,1)
                                                   │
                            sum over m  ────────── ▼
                                              L_SIGReg

Why this works in practice:

• Linear cost in $M$. Each projection is a dot product; each Epps-Pulley evaluation is closed-form. Default $M = 1024$.
• Bounded gradients. No moments to estimate, no batch-norm statistics to track. Stable across scale.
• Single hyperparameter that actually matters. Ablations show little sensitivity to $M$ or to numerical-integration knots; the regularizer weight $\lambda$ (default $0.1$) is the only knob.

2.2 The Full LeWM Objective

That’s it. The full training loss is just:

Two terms. One hyperparameter. No EMA target, no stop-gradient, no predictor asymmetry. The whole pipeline is end-to-end differentiable from raw pixels through the encoder, through the action-conditioned predictor, to the next-step embedding.

3. Architecture and Training

The system is deliberately small. ~15M parameters split into:

• Encoder — maps $(image_t) \to z_t \in \mathbb{R}^D$.
• Predictor — maps $(z_t, action_t) \to \hat z_{t+1}$.
• No decoder. Reconstruction-free; latent-space planning only.

Both modules train jointly under a single optimizer. The reported runtime is a few hours on a single GPU — a deliberate counterpoint to the foundation-model-scale world-modeling work the field has been trending toward.

4. Headline Results

4.1 Planning on PushT and Reacher

The DINO-WM comparison is the one that matters. DINO-WM gets to use pretrained DINO features and additional proprioceptive sensors; LeWM uses only raw pixels and still surpasses it on PushT. On Reacher the same pattern holds.

4.2 Planning Speed

The speed comes from the small latent and the absence of pretrained heavy-weight features in the planning loop. This is the practical result that makes LeWM interesting for robotics: planning latency in the seconds rather than minutes is a different deployment regime.

4.3 Violation-of-Expectation (IntPhys-style)

The probe of whether the latent space actually encodes physical structure rather than visual similarity:

• Surprise signals spike when an object teleports (a physical violation).
• Surprise signals do not spike when an object’s color changes (a visual perturbation that should not affect dynamics).

This is the strongest evidence in the paper that the regularizer isn’t just preventing collapse but is producing a latent space whose geometry is aligned with physical state.

4.4 Ablations

• Drop SIGReg → training collapses. As expected; this is the load-bearing piece.
• Vary $M$ (number of random projections) → little effect. Suggesting the Cramér-Wold approximation is robust.
• Vary $\lambda$ → moderate effect, but a wide plateau around the default. The one hyperparameter that needs tuning isn’t fussy.

5. Why This Matters

Three reasons:

1. A clean theoretical handle on JEPA collapse. Prior JEPAs prevented collapse with engineering tricks; LeWM replaces them with one principled regularizer derived from a 90-year-old probability theorem. That kind of substitution — workaround → first-principles constraint — usually marks the right level of abstraction for an idea to scale.
2. Small, fast, on-pixels world models become tractable. A 15M-parameter world model that trains in hours on one GPU and plans in sub-second wall-clock is a different deployment story from foundation-scale pixel models. For robotics specifically, this is the regime where on-board, real-time use becomes plausible.
3. The fewer-hyperparameters claim isn’t cosmetic. Six tunable loss weights → one means an actual practitioner can search over $\lambda$ instead of running grid sweeps over a six-dimensional space. This is the kind of simplification that makes a method reproducible in places other than the original lab.

In the broader context of recent self-supervised work, LeWM is part of a small family of papers (LeJEPA being the other obvious one) arguing that SSL doesn’t need heuristics if you specify the right distributional constraint on the latent space. The bet is that the field’s accumulated complexity (multi-view augmentations, momentum encoders, predictor bottlenecks) is mostly compensating for one missing piece: a direct, end-to-end-differentiable handle on the embedding distribution.

6. Limitations Worth Knowing

• Pixel-space generality. Results are on procedurally generated / robotics-style environments (PushT, Reacher, IntPhys-style perturbations). Whether the same recipe scales to large, diverse, natural-video datasets is the obvious next experiment.
• No language / multimodality. The paper stays inside the visual-and-action world. Combining LeWM-style training with a text-conditioned predictor is plausible but unverified.
• Planner is task-conditioned. Like most latent-space planners, LeWM still relies on a known task / reward / goal at planning time. The world model is unsupervised; the use of it is not.
• Single dataset scale shown. ~15M parameters is a feature for some deployments and a limitation for others. Whether SIGReg continues to be sufficient at, say, 1B parameters and a V-JEPA-scale dataset is the question that determines whether this becomes the default JEPA recipe or stays in the small-model niche.

7. The Takeaway for a First Reader

If you remember three things:

1. JEPAs collapse because the encoder has a trivial constant-output fixed point. Prior fixes (EMA, stop-grad, predictor asymmetry, six-term losses) all attack this indirectly.
2. LeWM forbids collapse by construction by adding SIGReg — a regularizer that uses random 1-D projections + the Epps-Pulley test (via Cramér-Wold) to push the embedding distribution toward isotropic Gaussian. The whole training loss is MSE + $\lambda \cdot$ SIGReg.
3. The resulting ~15M-parameter, single-GPU world model outperforms PLDM by ~18 pp and beats DINO-WM (even without proprioception) on PushT, while planning ~48× faster at inference.

That’s the arc: theoretical fix → architectural simplification → small, fast, surprisingly capable world model.

References

• Maes, L., Le Lidec, Q., Scieur, D., LeCun, Y., & Balestriero, R. (2026). LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312.
• Project page & code: https://le-wm.github.io/ · https://github.com/lucas-maes/le-wm
• Background: Cramér & Wold (1936) on random projections; Epps & Pulley (1983) on the normality test SIGReg borrows.
• Related on this site: Parcae paper review (another “diagnose the instability, fix by parameterization” paper), Meta-Harness review.