DreamGen — Scaling Robot Learning by Dreaming Trajectories

Created in June 13, 2026

2026   ·   robotics   world-models   video-diffusion   data-synthesis   vla   ·   paper-review

Paper. Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loic Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zhu, Linxi Fan. DreamGen: Unlocking Generalization in Robot Learning through Video World Models. CoRL 2025 / arXiv:2505.12705, May 2025. NVIDIA GEAR Lab. [arXiv] · [project page] · [code]

📑 Companion slide deck. A self-contained HTML walk-through of this review (in Korean) is available at DreamGen 논문 해설 (standalone).html. The deck and this post share the same structure; the post is the long-form English version.

0. The Picture in One Paragraph

Robot learning has been stuck on a structural problem: behaviors and environments don’t transfer cheaply. To get a robot to do a new task in a new room, you typically need to teleoperate demonstrations of that task in that room. The teleoperation rig becomes the bottleneck — humans are slow, robots are expensive, and the data scales linearly with the number of (behavior, environment) pairs you want to cover. DreamGen reframes the problem: what if we dream the new trajectories instead of collecting them? The recipe is four stages. (1) Fine-tune an image-to-video diffusion model with a LoRA on a small amount of real teleoperation data so it produces videos in the target robot’s visual / kinematic style. (2) Roll out the fine-tuned video model with new initial frames and new language instructions to generate photorealistic synthetic videos of behaviors and environments the robot has never been teleoperated through. (3) Recover pseudo-actions from those dreamed videos via an inverse- dynamics model (IDM) or a LAPA-style latent action model. (4) Train a visuomotor policy on the resulting (synthetic video, recovered action) pairs — “neural trajectories.” The headline: a humanoid robot performs 22 new behaviors in both seen and unseen environments while having been teleoperated on only a single pick-and-place task in one environment. The paradigm shift the paper argues for: scaling human teleoperation → scaling GPU compute through world models.

1. The Problem — Teleoperation Doesn’t Scale

The dominant recipe for training a VLA or visuomotor policy needs (observation, language, action) triples for every (behavior, environment) combination you want the robot to handle. The practical math:

Axis Cost
New behavior on familiar robot in familiar env hours of teleoperation
New environment (lighting, layout, objects) recollect everything
New embodiment recollect everything again
Tail behaviors (the actually-useful ones) impossible at scale

This is the wall LAPA’s predecessor work (review) addressed at the pretraining stage by extracting implicit actions from video. DreamGen attacks the same wall at the trajectory generation stage: instead of waiting for a human to teleoperate the behaviors, manufacture the trajectories with a generative model.

2. The Move — Neural Trajectories

A neural trajectory is a synthetic robot trajectory composed of:

  1. A photorealistic video of the robot performing the task (generated by a video world model).
  2. A sequence of pseudo-actions recovered from that video (inferred by an IDM or latent action model).

Crucially, the pseudo-actions are in the target robot’s action space. Once you have the (video, action) pair, any policy that trains on (observation, action) triples can consume it.

DreamGen’s bet: if the video world model is good enough at modeling the embodiment, the policies trained on its dreams will transfer to the real robot.

3. The Four-Stage Pipeline

Stage 1                        Stage 2                       Stage 3                  Stage 4
─────────                      ─────────                     ─────────                 ─────────
small real                      fine-tuned                    pseudo-action            visuomotor
teleoperation        ─────→     video world         ─────→    recovery        ─────→   policy on
demos (one task,                model (LoRA)                  (IDM or                  neural
one env)                                                      LAPA-style)              trajectories

                                ↓ rollouts
                                synthetic videos
                                of NEW behaviors
                                + NEW environments

3.1 Stage 1 — Video World Model Fine-Tuning

Take an off-the-shelf image-to-video generative model (the paper evaluates Cosmos, WAN 2.1, Hunyuan, CogVideoX) and fine-tune it with LoRA on a small set of real teleoperation videos.

The LoRA’s job is to inject the robot’s kinematics and the environment’s dynamics into the prior:

  • Kinematics: how this robot’s arm / gripper / base actually moves (joint limits, link lengths, gripper closure dynamics).
  • Embodiment appearance: what the robot looks like in the generated frames.
  • Scene physics: how objects in this lab respond to contact.

Critically, the LoRA preserves the general world prior in the base model. The fine-tuned model can still generate scenes and objects that were never in the teleoperation data — it just generates them with the right robot in the foreground.

3.2 Stage 2 — Video World Model Rollout

Now generate synthetic videos. Inputs:

  • An initial frame (from simulation or a single captured photo) showing the start state.
  • A language instruction (“pour the cup”, “open the drawer”, “wipe the table”) describing the behavior.

The fine-tuned model produces a photorealistic video of the robot performing the behavior. The initial frames don’t have to match the teleoperation distribution — they can be from new environments. This is the step where behavior × environment combinatorial coverage actually happens.

In the paper’s headline experiment, this stage generates 50 neural trajectories per behavior, for 14 novel behavior tasks.

3.3 Stage 3 — Pseudo-Action Recovery

The generated videos contain no action labels. DreamGen recovers them by running each video through one of:

  • An inverse-dynamics model (IDM) — supervised on the real teleoperation data to predict actions from observation sequences. Same data as Stage 1; different head.
  • A LAPA-style latent action model — extracts discrete latent action codes from frame pairs (LAPA review). The decoder maps codes to robot actions.

Both pathways produce per-frame pseudo-actions. The pseudo-actions are noisy by construction; the policy training step has to absorb that noise. The fact that it does is itself a finding.

3.4 Stage 4 — Policy Training on Neural Trajectories

Now train a visuomotor policy on the neural trajectories as if they were real teleoperation data. The paper shows two regimes:

  • Pure neural-trajectory training (no real data after Stage 1’s one task) — surprisingly works.
  • Co-training — mix neural trajectories with whatever small real trajectory set you have. This yields consistent gains on both IDM-pseudo-action and LAPA-pseudo-action variants across all data regimes.

4. DreamGen Bench — The World-Modeling Benchmark

A practical question DreamGen has to answer: which video model should I use as the world model? The paper ships DreamGen Bench to make this decision empirically rather than by vibes.

4.1 What it measures

Two metrics:

  1. Instruction following — does the generated video actually depict the behavior the prompt asked for?
  2. Physics following — is the motion in the video physically plausible (no teleporting objects, no impossible contact, no broken kinematics)?

4.2 The four contenders

Cosmos, WAN 2.1, Hunyuan, CogVideoX — the major open image-to-video models at the time of writing.

4.3 The most useful finding

DreamGen Bench scores correlate strongly with downstream policy success.

This is the result that turns DreamGen from a recipe into a research program. It means improving the video world model (better instruction following, better physics) provably improves the downstream robot policy. The robot people now have a direct reason to care about progress in video generation, and the video people have a benchmark that measures something other than “is the video pretty.”

7,000 neural trajectories per model are generated for the benchmark.

5. Headline Results

5.1 The humanoid generalization claim

A humanoid robot performs 22 new behaviors in both seen and unseen environments while requiring teleoperation data from only a single pick-and-place task in one environment.

The 22 novel behaviors include pouring, opening/closing articulated objects, and manipulating a variety of tools — tasks that are very different in kinematic structure from pick-and-place.

The setup is the strongest possible test of the data-amortization claim: real teleoperation supervision is constant (one task, one environment); behavior diversity comes entirely from the dreamed trajectories.

5.2 Co-training consistency

Across both pseudo-action recovery pathways (IDM and LAPA) and across all data regimes (small / medium / large real-trajectory budgets), co-training with neural trajectories improves the policy. This is the robust version of the result — the gain isn’t limited to a particular regime.

5.3 Scaling per task

A few real teleoperation trajectories per task — 10 to 13 — combined with neural trajectories, suffice to lift policy success across all evaluated robots. This is the data-budget DreamGen wants people to think in: a handful of real demos plus thousands of dreamed ones.

6. What the Method Actually Says

Stripped of acronyms, the structural claim:

A modern image-to-video diffusion model has internalized enough physical and visual structure that, with a small LoRA on teleoperation data, it becomes a usable simulator of the target robot. Once you have a usable simulator, you can dream the trajectories for behaviors and environments you’ve never teleoperated, recover pseudo-actions via inverse dynamics or latent action models, and train policies on the result. The cost of new behaviors and new environments moves from human-hours at a teleoperation rig to GPU-hours at a video model.

This is the data-side analog of the architecture-side argument DreamZero makes a year later. DreamGen treats the video model as a dataset generator; DreamZero treats it as the policy backbone itself. Both rely on the same belief: video is the right abstraction for robot learning because it carries dynamics that action labels alone cannot.

7. Why It Matters

Three reasons, in increasing order of generality:

  1. It changes the cost structure of robot data. Robot data has been priced in human-hours of teleoperation. DreamGen prices it in GPU-hours of dreaming. GPU-hours scale with money; teleoperation-hours scale with people. This is the kind of substitution that, if it holds up, reshapes the field’s resource model.
  2. It turns video-generation progress into robot-policy progress. Through DreamGen Bench, every improvement in instruction-following or physics-following video generation now directly translates into a downstream robot win. The two research communities (generative video, robot learning) now have a measurable common interest.
  3. It completes the NVIDIA GEAR triad.
    • LAPA (Oct 2024) — pretrain a VLA on action-label-free video by extracting discrete latent actions.
    • DreamGen (May 2025) — generate the videos themselves and recover pseudo-actions to train policies.
    • DreamZero (Feb 2026) — collapse both into a single video-diffusion backbone that jointly predicts video and action.

    Read in order, these three papers tell one consistent story: video is the substrate; action is a derived quantity; the research question is where in the pipeline you do the extraction.

8. Limitations Worth Knowing

  • Video-model fidelity caps the policy. If the world model hallucinates physics (sliding cups, ghost contacts) the policy will learn the hallucinations. DreamGen Bench’s physics metric exists to measure this exactly; the implication is that some behaviors are simply not yet dreamable well enough for policy training.
  • Pseudo-actions are noisy. Both IDM and LAPA recover actions from observation sequences with error. The paper shows the policy absorbs this, but extreme-precision tasks (sub-millimeter insertion, delicate manipulation) may not survive the noise.
  • Initial-frame coverage limits behavior coverage. Stage 2 needs an initial frame for each scenario. If the only initial frames you have come from one room, the dreams can extend to new behaviors but not to genuinely new visual environments. The paper handles this with simulation-rendered initial frames, but the dependency exists.
  • Embodiment-specific LoRA. Each new robot embodiment needs its own LoRA fine-tune (and the teleoperation data to do that fine-tune). DreamGen reduces but does not eliminate per-embodiment supervision.
  • CoRL benchmarks aren’t real homes. The 22 humanoid behaviors are an impressive proof-of-concept; the gap between “humanoid pours water in a lab” and “household robot deploys to customers” is still large.

9. The Takeaway for a First Reader

If you remember three things:

  1. Robot data was priced in teleoperation-hours. DreamGen reprices it in GPU-hours by dreaming synthetic robot trajectories from a video world model and recovering pseudo-actions from those dreams.
  2. The pipeline is four stages: fine-tune the video model on a small amount of real demos (LoRA) → roll out videos of new behaviors/environments → recover actions via IDM or LAPA-style latent action model → train policies on the resulting neural trajectories.
  3. The headline: a humanoid performs 22 new behaviors in seen and unseen environments from a single pick-and-place teleoperation source task. DreamGen Bench scores correlate with downstream policy success, turning video-generation progress into robot-policy progress.

That’s the arc: teleoperation doesn’t scale → dream the trajectories → recover the actions → train the policy → measure the world model’s quality as a robot-learning signal.

References



    Enjoy Reading This Article?

    Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition