LAPA — Latent Action Pretraining from Action-Label-Free Video

Paper. Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, Minjoon Seo. Latent Action Pretraining from Videos. ICLR 2025. KAIST · UW · Microsoft · NVIDIA · AI2. [arXiv] · [project page] · [code] · [HF checkpoint]

📑 Companion slide deck. A self-contained HTML walk-through of this review (in Korean) is available at LAPA 논문 해설 (standalone).html. The deck and this post share the same structure; the post is the long-form English version.

0. The Picture in One Paragraph

Modern Vision-Language-Action (VLA) models — OpenVLA, RT-2-X, Octo — are trained on robot demonstration data in which every frame has a ground-truth action label. That label is expensive: it requires a human teleoperator (or motion-captured expert) recording joint commands at every timestep, on a specific robot, for every task. The world has enormous amounts of unlabeled video (humans manipulating objects, robots in the wild, household scenes) but none of it has action labels in the VLA sense. LAPA’s question is simple: can we pretrain a VLA on action-label-free video, and recover the action label only at fine-tuning time with a small amount of robot data? The answer is yes, via a two-stage recipe. First, a VQ-VAE-style inverse-dynamics model learns to encode each pair of consecutive frames into a discrete latent action — the “label” the video implicitly carries. Then a VLA backbone is pretrained to predict these discrete latent codes from observations

• language, on internet-scale video. Finally, a tiny decoder is fine-tuned on robot data to translate the latent code into actual joint actions. The result: +6.22% over OpenVLA on real-robot evaluation with ~30× lower pretraining compute and zero ground-truth action labels at the pretraining stage.

1. The Problem — VLAs Are Pinned to Labeled Robot Data

The dominant VLA recipe (OpenVLA, RT-2-X, Octo) treats a robot policy as a language-conditioned next-action predictor:

To train, you need (o_t, ℓ, a_t) triples — and the $a_t$ has to be in the target robot’s joint/end-effector action space. This ties the training distribution to:

• Teleoperated robot data. Expensive, slow to collect.
• A specific embodiment. Cross-embodiment transfer (Open X-Embodiment) helps but the action spaces still have to be reconciled.

Meanwhile, the world has orders of magnitude more action-label- free video:

The structural complaint: the supervision pipeline is wasting ~99% of the available training data because action labels are gating access to it. LAPA’s bet is that the action labels are implicit in the video — two consecutive frames already encode “what action was taken.” We just have to extract them.

2. The Reframing — Recover Implicit Actions from Frame Pairs

LAPA’s framing has three moves:

1. Define a latent action space. Not the robot’s joint space; a learned, discrete latent space whose codes describe the transitions between consecutive frames.
2. Pretrain a VLA to predict latent codes, not joint commands. This uses any video — no action labels required.
3. Fine-tune a small decoder from latent codes to real robot actions, using a small amount of labeled robot data.

The latent space is the universal currency. Pretraining happens in that currency; only the final fine-tune touches the embodiment- specific action space.

3. Stage 1 — Latent Action Quantization

The first model LAPA trains is an inverse-dynamics VQ-VAE:

  frame x_t  ─┐
              ├─→ Encoder ─→ continuous latent ─→ VQ codebook ─→ discrete code z_t
 frame x_{t+1}┘                                                       │
                                                                      ▼
              ┌──────── Decoder ──────────────────────────────────────┘
              │ generates x_{t+1} from (x_t, z_t)
              ▼
            x_{t+1}'  ←  reconstruction loss vs ground-truth x_{t+1}

3.1 What the encoder learns

The encoder reads two consecutive frames and emits a discrete code $z_t$ that captures what changed between them. This is the classic inverse dynamics model target: infer the action that caused the transition $(x_t, x_{t+1})$.

VQ-VAE quantization forces the latent space to be discrete, which is what makes downstream language-modeling-style pretraining clean. The codebook size is a hyperparameter (a few thousand codes); each code corresponds to one “kind” of inter-frame transition.

3.2 What the decoder learns

The decoder is the forward dynamics model: given the current frame $x_t$ and a discrete code $z_t$, predict the next frame $x_{t+1}$. The reconstruction loss between the predicted and true $x_{t+1}$ trains the whole encoder-decoder + codebook end to end.

3.3 NSVQ — gradient collapse fix

Vanilla VQ-VAE training suffers from gradient collapse: most codebook entries go unused because the nearest-neighbor lookup has zero gradient. LAPA uses NSVQ (Vali & Bäckström, 2022), which replaces the quantization error with the product of the original error and a normalized noise vector. This keeps gradients flowing through more of the codebook and prevents the collapse.

3.4 What you have at the end of Stage 1

A discrete vocabulary of inter-frame actions, learned purely from video with no action labels needed. Every frame pair in your video corpus now has an implicit “label” — its latent code.

4. Stage 2 — Latent VLA Pretraining

Now treat the latent codes as the target tokens for a language-style VLA model.

4.1 Setup

• Backbone: a Vision-Language model (Prismatic-7B class).
• Inputs: current observation $o_t$ + task description $\ell$.
• Target: the discrete latent code $z_t$ produced by the Stage-1 encoder on the next frame.
• Loss: standard next-token prediction (or classification over the codebook).

4.2 The data unlock

This pretraining task uses any video corpus, because the only “label” needed is the Stage-1 encoder’s output on a frame pair — and the Stage-1 encoder runs on any frames. LAPA’s headline pretraining run uses the Open-X Embodiment corpus, but the recipe is compatible with internet-scale unlabeled video.

The compute story is dramatic. For pretraining on Open-X:

That’s roughly 30–40× lower pretraining compute to beat OpenVLA on the downstream benchmark, while never seeing a single ground-truth action label at this stage.

4.3 The intuition

The model is learning the same competence OpenVLA learns — “given this observation and instruction, what kind of motion should follow?” — but it expresses the answer in a vocabulary (the codebook) that doesn’t require the model to commit to a specific joint angle in a specific embodiment. That’s why it can absorb training signal much more cheaply: the target is lower- dimensional and embodiment-agnostic.

5. Stage 3 — Fine-tune a Latent → Action Decoder

The final step bridges the latent code to actual robot actions.

5.1 Setup

• Take the latent-pretrained VLA.
• Add a small action decoder head that maps the latent code $z_t$ (plus the current observation) to the real action $a_t$ in the target robot’s space.
• Fine-tune on a small amount of labeled robot data — orders of magnitude less than what OpenVLA used.

5.2 What survives, what changes

• Survives: the VLA backbone, which has already learned “observation + language → latent action” from huge video data.
• Changes: only the latent-to-real-action mapping. This is the embodiment-specific part, and it’s the only thing that needs embodiment-specific data.

This is the cross-embodiment version of “freeze the backbone, re-train the head.” LAPA’s recipe makes that pattern work for action spaces, not just classification spaces.

6. Headline Results

6.1 Real-robot evaluation vs OpenVLA

The pattern: LAPA matches or beats a VLA trained on millions of labeled trajectories, with a tiny fraction of the compute, and without using any action labels during pretraining.

6.2 Cross-embodiment

LAPA significantly outperforms OpenVLA on 2 out of 3 unseen embodiments. The latent-action representation transfers across robots better than the per-embodiment action label does, because the latent vocabulary was never tied to one robot in the first place.

6.3 The fine-tune efficiency

Fine-tuning a small action decoder on a small amount of labeled data is enough to unlock the policy. The compute asymmetry — heavy on data-cheap pretraining, light on data-expensive fine-tuning — is the practical win for any team that doesn’t own a teleoperation rig.

7. What the Method Actually Says

Stripped of acronyms, the structural claim:

Action labels are not the unique source of supervision for robot policies. The transition between two consecutive video frames is already a noisy label of what action was taken. If we extract that label as a discrete code via a VQ-VAE inverse- dynamics model, we can pretrain a VLA on any video — action-labeled or not — and then specialize to a particular robot with a tiny labeled fine-tune. The pretraining bottleneck moves from labeled-data quantity to video-corpus quantity, which is orders of magnitude larger.

This is the pretraining analog of the architecture argument DreamZero makes a year later: instead of treating action as the primary modality and adding a vision module, treat video as the primary modality and recover action from it. LAPA does this through a discrete code; DreamZero does it through joint video+action generative modeling. Both arguments start with the same observation about data availability.

8. Why It Matters

Three reasons:

1. It dissolves the labeled-data bottleneck for VLA pretraining. Robot foundation models have been compute-bound and data-bound simultaneously — labels are scarce and expensive. LAPA shifts the bottleneck to video availability, which is effectively unbounded.
2. The discrete-latent-action vocabulary is a reusable primitive. Treating action as a sequence of discrete codes slots cleanly into LLM-style training pipelines (next-token prediction, tokenizer reuse, mixture-of-data training). The subsequent work — UniVLA, GR00T, the DreamZero lineage — borrows variations of this primitive.
3. It validates the “pretrain on video, specialize on robot data” recipe. This was a stated dream for years; LAPA is the first paper to ship it convincingly on standard VLA benchmarks without losing to action-supervised baselines.

The trend connection is direct. The same first author (Seonghyeon Ye) led both LAPA (Oct 2024, ICLR 2025) and DreamZero (Feb 2026). The arc is internally consistent: LAPA proved that video contains action supervision; DreamZero replaces the discrete codebook with a video-diffusion backbone that learns the dynamics implicitly and emits actions as a parallel head. The two papers are different architectural answers to the same underlying question — and reading them together is the cleanest way to see where robot foundation modeling is going.

9. Limitations Worth Knowing

• The latent codebook is fixed at Stage 1. If the codebook is too small, you lose action granularity; too large and the pretraining task becomes harder. The paper picks a working size but doesn’t fully ablate this knob.
• Inverse dynamics assumes near-Markov frame pairs. Two frames are usually enough to identify the action, but for slow or fine-grained motions, two-frame inverse dynamics undercounts. Multi-frame windows would help; they aren’t in the recipe.
• Fine-tune data is still required. LAPA reduces but does not eliminate the need for robot-specific labeled data — the latent-to-action decoder needs something to ground the codes. How small that something can get is the practical question.
• Internet video isn’t yet the main pretraining corpus. The reported run uses Open-X. The paper’s framing strongly implies internet video will work, but the headline numbers are on Open-X. Whether the gain holds on truly unconstrained internet video is the obvious next experiment.
• The latent space isn’t interpretable. Each code is “some inter-frame transition” — it doesn’t correspond to a named primitive like “grasp” or “push.” For debugging and safety audits, this is harder than a hand-engineered action vocabulary.

10. The Takeaway for a First Reader

If you remember three things:

1. VLAs were data-bottlenecked by action labels. LAPA breaks the bottleneck by extracting implicit actions from video via a VQ-VAE inverse-dynamics model that emits a discrete latent code for each frame pair.
2. Pretraining the VLA to predict latent codes is action-label-free, scales to any video corpus, and costs ~30× less compute than the action-supervised alternative (OpenVLA) while ultimately winning on downstream real-robot evaluation by +6.22%.
3. Fine-tuning a small latent → action decoder on a tiny amount of labeled robot data is enough to deploy the policy on a real robot. The pretrained backbone transfers; only the embodiment-specific head needs replacement.

That’s the arc: action labels are a bottleneck → extract them from video as discrete codes → pretrain a VLA on those codes → fine-tune a small decoder for the target robot → beat the action-supervised baseline at a fraction of the compute.

References

• Ye, S., Jang, J., et al. (2024). Latent Action Pretraining from Videos. ICLR 2025. arXiv:2410.11758.
• Project page: https://latentactionpretraining.github.io/
• Code: https://github.com/LatentActionPretraining/LAPA
• Checkpoint: https://huggingface.co/latent-action-pretraining/LAPA-7B-openx
• Vali, M. H. & Bäckström, T. (2022). NSVQ: Noise Substitution in Vector Quantization.
• Background: OpenVLA (Kim et al., 2024); Open X-Embodiment (Open X-Embodiment Collaboration, 2024); RT-2-X.
• Related on this site: DreamZero paper review — the same author’s follow-up that replaces the discrete codebook with a video-diffusion world model; Flow Matching paper review — the training objective the video-diffusion backbones in this lineage are built on.