Video Prediction Policy — Predictive Visual Representations as the Policy Backbone

Paper. Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, Jianyu Chen. Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations. ICML 2025 Spotlight / arXiv:2412.14803, Dec 2024. Tsinghua · UC Berkeley · RobotEra · Shanghai AI Lab · Shanghai Qi Zhi Institute. [arXiv] · [project page] · [code] · [OpenReview]

📑 Companion slide deck. A self-contained HTML walk-through of this review (in Korean) is available at VPP 논문 해설 (standalone).html. The deck and this post share the same structure; the post is the long-form English version.

0. The Picture in One Paragraph

Generalist robot policies need a visual backbone — something that turns a camera image into a feature vector the action head can operate on. The dominant backbones (R3M, MVP, DINO-v2, CLIP) are pretrained with single-image reconstruction or two-image contrastive objectives. These produce static features: they encode what’s currently in the scene, but they don’t encode what’s about to happen. For embodied control, that’s the wrong shape — the policy needs predictive features. VPP’s argument is that modern video diffusion models already carry predictive features inside them as a side effect of being trained to generate the next frames, and these features can be used directly as the backbone of a robot policy. The recipe is two parts. (1) Fine-tune a general-purpose video diffusion foundation model on a mix of robot data and internet human manipulation video to make it a text-conditioned manipulation video predictor. (2) Take the internal representations of that fine-tuned video model — not the generated video, but the intermediate features — as the conditioning signal for a Diffusion Transformer action head, which learns an implicit inverse-dynamics mapping from “predicted future representation” to “action that gets you there.” The result: +41.5% relative improvement on Calvin ABC-D generalization vs. previous SOTA, an average task length of 4.33 on Calvin ABC, +31.6% on complex real-world dexterous manipulation, and a single VPP policy solving 100+ tasks on a real-world dexterous hand.

1. The Problem — Static Features Don’t Carry Dynamics

The standard “vision encoder for robot policy” recipe pretrains a backbone with one of two objectives:

Family	Pretraining objective	What it captures
R3M, MVP, time-contrastive	contrastive learning on video clips	semantic / spatial similarity
DINO-v2, MAE, CLIP	reconstruction / self-supervised on still images	static scene content

These backbones produce good static features: “what’s in the image right now.” But embodied control is fundamentally a temporal problem. The right feature for “should I move left or right?” is not just “there’s a cup at (x, y) in the current frame” — it’s “the cup is currently here, and if I act, it will be there.” That predictive component is what static encoders miss.

The structural complaint VPP makes: the backbone has been optimized for the wrong task. A backbone optimized to predict the future frame is a backbone that has, by construction, internalized the dynamics that policy learning needs.

2. The Move — Video Diffusion as a Predictive Feature Extractor

Video diffusion models (VDMs) are trained to generate the next $K$ frames given prior frames and a text instruction. To do that well, they have to internalize:

Object permanence and trajectories — objects move continuously, don’t teleport.
Contact and articulation — fingers close around handles, cups tilt under gravity.
Causality — actions implied by the prompt produce specific visual consequences.

This is exactly the dynamics knowledge robot policies need. VPP’s bet is that you don’t need the VDM to actually generate the video at policy inference time — its intermediate features already carry the predictive information.

The mental shift: the video model is not used as a generator at deployment. It is used as a feature extractor whose features happen to encode predicted dynamics. The action head reads those features and emits actions.

3. The Architecture

VPP has two trainable components plus one frozen-after-fine-tune backbone.

3.1 Component 1 — The video prediction model

Start from a general-purpose video diffusion foundation model (image-to-video). Fine-tune on:

Robot teleoperation data (the target embodiment’s videos).
Internet human manipulation video (broad action prior).

The fine-tuned model becomes a text-conditioned manipulation video predictor: given a current frame and a language instruction, it can predict how the scene will evolve.

This is the same first step in DreamGen — the generative model is adapted to the embodiment. The difference is what comes next.

3.2 Component 2 — The action head

A Diffusion Transformer (DiT) policy that:

Reads the internal features of the video prediction model (not the generated video — the intermediate activations) as conditioning.
Reads the current observation and the language instruction.
Outputs an action chunk via standard diffusion-policy denoising.

The action head is small relative to the VDM. It’s the only part that touches the robot’s action space.

3.3 Implicit inverse dynamics

The action head learns an implicit inverse-dynamics mapping:

Given the VDM’s predicted next-state representation (implicitly encoded in its features), output the action that, when executed, will actually reach that state.

This is the same structural pattern as LAPA’s inverse-dynamics trick, but at the feature level instead of the frame level. LAPA’s IDM read two frames and produced a discrete code; VPP’s implicit IDM reads the VDM’s predictive features and produces a continuous action. Both are recovering the action implicit in a visual prediction.

3.4 The inference-time efficiency trick

A naive use of a VDM at inference would require generating the full video — slow, with many denoising steps. VPP sidesteps this:

The features used for conditioning are read early in the denoising trajectory (or even from a single forward pass), so the policy doesn’t wait for a full clean video to be sampled.
The DiT action head itself runs on these features, so the full-video generative cost is avoided.

This is what makes VPP usable as a real-time policy despite a heavy video-model backbone.

4. Why Predictive Features Beat Static Ones

The intuition the paper formalizes:

A frame’s static features describe what is. The internal features of a video diffusion model trained to predict the next frames describe what is and what will be. For inverse- dynamics-style action learning, you need both.

A static-feature backbone forces the policy head to do all the dynamics modeling on its own — from limited demonstration data. A predictive-feature backbone gives the policy head a head start: the dynamics are already implicit in the conditioning signal, distilled from internet-scale video pretraining.

The mechanism is the same one DreamZero leans on at the backbone level — internet video is a dense, scalable source of physical dynamics, and a model trained on it inherits a usable world prior. VPP’s contribution is showing that you don’t need to collapse the action head into the video backbone (as DreamZero does later); a clean separation where the VDM is just a feature extractor already wins decisively.

5. Headline Results

5.1 Calvin ABC-D (sim, generalization)

Metric	VPP vs. previous SOTA
Relative improvement	+41.5%
Average task length on Calvin ABC	4.33

Calvin ABC-D measures generalization to unseen task combinations across rooms. The 41.5% relative gain is the headline benchmark result.

5.2 Real-world dexterous manipulation

Metric	Result
Success rate improvement vs. baselines	+31.6%
Real-world tasks solved by a single VPP policy	100+ on dexterous hand

The dexterous-hand result is the most striking. A single policy solving 100+ tasks on a high-DOF hand is the kind of generality result that, until recently, required heavy multi-task pretraining on per-task labeled data. VPP gets it from the predictive-feature conditioning.

5.3 Evaluation breadth

Simulated: Calvin, MetaWorld.
Real: Panda arm manipulation, XHand dexterous hand manipulation.

The cross-platform consistency of the gains supports the claim that the predictive-feature backbone — not benchmark-specific tuning — is what’s doing the work.

6. The Taxonomy of Using a Video Model for Robot Learning

Reading VPP alongside the three NVIDIA GEAR papers reviewed on this site produces a clean taxonomy of how a modern video model can power robot learning. Each paper uses the video model at a different layer of the pipeline:

Paper	The video model is used as…
LAPA	a label extractor — encodes frame pairs as discrete latent action codes for VLA pretraining
DreamGen	a data factory — dreams synthetic videos for new behaviors / environments, with pseudo-actions recovered downstream
VPP (this paper)	a feature extractor — its internal predictive representations condition a separate action head
DreamZero	a policy backbone — the video model itself emits both predicted frames and actions through parallel heads

The four occupy a Pareto-like frontier. LAPA is cheapest at deploy time (the VDM doesn’t run; only the label-extracted VLA does). VPP is the cleanest separation of concerns (VDM features → action head). DreamGen trades online efficiency for offline data-amortization. DreamZero maximally collapses the architecture at the cost of needing the largest model at deploy time.

Choosing between them is a decision frame (Ch. 2-style) based on what you actually have:

Lots of unlabeled video, small embodied dataset → LAPA.
A few demos for the source task, want behavior coverage → DreamGen.
Strong VDM, limited real-time GPU budget for the policy → VPP.
Plenty of real-time compute, want the cleanest architecture → DreamZero.

VPP is the most practitioner-friendly point on this frontier because the action head is small and the VDM only runs as a feature extractor.

7. Why It Matters

Three reasons:

It establishes predictive features as a backbone primitive. For two years the field’s default backbones (R3M, MVP, DINO-v2) have been static. VPP shows that swapping to a VDM-derived predictive backbone produces a measurable jump in generalization benchmarks. That’s a reusable substrate, not a one-off result.
It makes “use the VDM as a feature extractor” a respectable choice. Before VPP, the implicit assumption was “if you have a VDM, you should generate videos with it.” VPP shows that just reading the intermediate features is often enough, and is dramatically cheaper at inference. This is the same lesson transformer-based feature reuse taught vision two years ago — the inside of the model is at least as useful as the output.
It defines the action head minimally. The DiT action head is small relative to the VDM. This means the policy-specific tuning is cheap; you can adapt to new embodiments by retraining the action head while keeping the VDM features fixed. The architecture mirrors the LAPA fine-tune-the-decoder pattern, generalized to continuous actions and predictive features.

8. Limitations Worth Knowing

VDM quality caps the policy. Just like DreamGen, if the video model is bad at instruction-following or physics, the predictive features inherit those flaws. There is no DreamGen Bench-style explicit measurement in this paper, but the dependency is real.
The feature-extraction step has a wall-clock cost. The efficiency tricks bring inference into a usable regime, but the VDM is still a much heavier backbone than R3M or DINO-v2. On edge robots without GPU support, this is a serious deployment constraint.
Inverse dynamics is implicit. The action head is not trained against an explicit inverse-dynamics target — it learns the mapping by gradient descent on action-prediction loss. This works in practice but lacks the interpretability of an explicit IDM head.
VDM fine-tuning data still has to exist. The video prediction model needs robot + human-manipulation data to be fine-tuned. This is cheaper than action-labeled robot data (which is what was avoided), but it’s not zero.
Calvin-style benchmarks aren’t open-world. Calvin and MetaWorld are constrained simulated environments. The dexterous- hand real-world result is impressive, but the gap to fully unconstrained deployment remains.

9. The Takeaway for a First Reader

If you remember three things:

Vision backbones for robot policy have been pretrained for the wrong task — static reconstruction / contrastive objectives capture what is, not what will be. Video diffusion models already carry the missing predictive signal inside their features.
VPP uses a fine-tuned VDM as a feature extractor, reads its intermediate features as conditioning, and trains a small Diffusion Transformer action head that learns an implicit inverse-dynamics mapping from predicted future representation to action.
The result is +41.5% on Calvin ABC-D, +31.6% on real-world dexterous manipulation, and a single policy solving 100+ real-world dexterous-hand tasks, while keeping the action head small and the deployment story clean.

That’s the arc: identify the missing predictive signal in robot backbones → notice that VDMs already encode it → use VDM features (not videos) → train an implicit-IDM action head → win across sim and real with one policy architecture.

References

Hu, Y., Guo, Y., Wang, P., Chen, X., Wang, Y.-J., Zhang, J., Sreenath, K., Lu, C., & Chen, J. (2025). Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations. ICML 2025 Spotlight. arXiv:2412.14803.
Project page: https://video-prediction-policy.github.io/
Code: https://github.com/roboterax/video-prediction-policy
Background backbones: R3M, MVP, DINO-v2, CLIP.
Related on this site (the “video model in robot learning” taxonomy): LAPA paper review — video model as label extractor; DreamGen paper review — video model as data factory; DreamZero paper review — video model as policy backbone; Flow Matching paper review — the training objective these video models depend on.