Daily AI Paper

Imagine trying to learn a single action representation that works for a human hand, a robot arm, a different camera view, and even a different scene. That is the core problem this paper tackles. It matters because modern robot learning is hungry for data, but high-quality robot demonstrations are expensive to collect at scale. Human videos are abundant, and they often show the kinds of manipulation behaviors we want robots to imitate. The catch is that the action labels in human video and robot demonstrations are not naturally shared. A human opening a drawer and a robot opening a drawer can look similar at the level of intent and motion, but the underlying control signals are completely different. So if we want to transfer behavior across embodiments, we need a common action language that captures what the motion is doing, not just how it looks in one particular setting. That is where latent action models come in. The basic idea is to infer a compact hidden representation of action from visual transitions. Instead of requiring an explicit action label, the model learns from videos by trying to explain how the scene changes over time. This is appealing, because video is much easier to collect than robot control data. But there is a known weakness: if you only supervise the model using local frame-to-frame reconstruction, the learned representation can entangle action with irrelevant details like viewpoint, background, object appearance, or embodiment-specific cues. In other words, the model may learn a code that helps predict pixels, but does not cleanly isolate the underlying motion. That makes transfer hard. DreamX proposes a different route. Rather than learning an action encoder from scratch, it repurposes a pretrained video diffusion model as the backbone for latent action learning. The intuition is that large video generation models already learn rich spatiotemporal structure: they understand how objects move, how scenes evolve, and how geometry and appearance interact over time. They also have a strong generative prior, meaning they can reconstruct plausible future frames from an initial frame and some conditioning signal. DreamX uses those two properties together. It reads action-relevant information directly from the video model’s internal features, and then uses the same model to reconstruct the future, so the latent code is pressured to capture motion rather than appearance. A useful way to think about the method is this: the video diffusion model sees the clip, and DreamX attaches a small set of learnable query tokens that act like information gatherers. These queries attend over the model’s spatiotemporal features and pull out the dynamics that matter. The result is a very compact latent action representation. The paper emphasizes the compression: it reduces each latent video frame from 1,200 tokens down to just 4 tokens, a 300-times reduction. That is not just a neat efficiency trick; it is also a sign that the model is trying to distill motion into a small shared interface. The backbone they build on is a pretrained video diffusion transformer. In broad terms, a video diffusion model learns to generate or reconstruct video by denoising latent representations over time. It is trained on large-scale video data, so it acquires a prior over what kinds of scene evolution are plausible. DreamX uses this pretrained model in two roles. First, as an encoder: learnable queries are appended alongside the video tokens, and through spatiotemporal attention they extract compact latent actions. The queries are organized so that each group corresponds to one latent video frame, and the attention is designed to be temporally grounded. Second, as a decoder: the model reconstructs future frames from the initial frame plus the latent actions, without relying on text conditioning. Since the initial frame already provides scene appearance, the latent actions are encouraged to encode the missing ingredient, which is the motion needed to explain what happens next. This is an important design choice. If the decoder already knows the starting appearance, then any information about colors, textures, or static scene layout is less useful for the latent action code. The model is pushed toward action-centric information: what moved, where it moved, and how the scene changed. The paper also includes an auxiliary branch for action decoding when action labels are available. In that case, the model can also predict continuous robot action chunks from the same latent representation. This helps connect the learned latent space to executable control, not just future video prediction. The attention masking is also worth understanding at a high level. The latent-action queries are not allowed to attend arbitrarily in the decoder; instead, they serve as conditioning signals. The video stream and the action stream can attend to the queries, but not to each other. This keeps the latent actions in the middle as the shared interface. The training objective is the standard flow-matching objective used by the video diffusion backbone, applied to both video generation and action generation when action labels exist. For clips without action labels, only the video objective is used. So the method can learn from heterogeneous data: human videos, robot videos, labeled or unlabeled. What makes DreamX especially interesting is that cross-embodiment transfer is not directly hard-coded as a separate objective. Instead, it emerges from the combination of pretrained video priors and reconstruction-based supervision. The authors argue that because the backbone already understands motion and scene dynamics at a high level, the extracted latent actions become more invariant to embodiment and appearance than codes learned from scratch on one-step transitions. For evaluation, the paper trains DreamX on heterogeneous human and robot videos and tests it on LARYBench, which is the benchmark they use for latent action models. They compare against prior latent action methods and report state-of-the-art performance. The paper also looks beyond standard benchmark scores and tests whether the latent space actually transfers. One experiment is any-to-any video transfer: actions encoded from one setting can be used to drive generation in another. Another is zero-shot human-to-robot action transfer, where motion extracted from a source video can be decoded into continuous actions for a target embodiment without paired source-target training data. That is a strong claim, because it suggests the representation is not just good at compressing motion, but also at serving as a shared action interface. The results show several things. First, DreamX outperforms other latent action models on LARYBench, which supports the claim that reading actions from a pretrained video diffusion backbone is a better starting point than learning a dedicated encoder from scratch. Second, the latent space appears transferable across embodiments and environments. The fact that it supports any-to-any video transfer and zero-shot action transfer suggests that the model has captured a more abstract notion of action than a pure pixel-transition code would. Third, the compact latent actions can be used as prediction targets for downstream policy learning. In other words, a robot policy can predict these latent codes from observations and then map them to low-level actions, without needing to generate future video during execution. That matters because video generation is expensive, so using the latent codes as an intermediate supervision signal makes policy learning more practical. There is also an efficiency angle here. By compressing 1,200 frame tokens into 4 latent-action tokens, DreamX reduces the token burden dramatically. That can lower computational cost when using video-diffusion-based policies, which is important because these models are powerful but often heavy. So the paper is not only about representation quality; it is also about making that representation compact enough to be useful in downstream control pipelines. As for limitations, the paper hints at a few. The method depends on a strong pretrained video diffusion backbone, so its performance likely inherits whatever biases and coverage that backbone has from pretraining. It also relies on the quality of the generative prior: if the backbone does not model some embodiment or motion well, the latent actions may be less reliable there. And while the results show promising zero-shot transfer, the paper does not claim that the latent space solves all embodiment mismatch problems. The learned interface is shared and transferable, but it is still learned from data, so broad generalization will depend on how diverse the training videos are. The main takeaway is that DreamX reframes latent action learning as a problem of extracting motion from a pretrained video world model, rather than inventing a new action encoder from scratch. That lets the model use the video backbone’s understanding of dynamics to separate action from appearance, compress motion aggressively, and support transfer across humans, robots, and environments. In the broader landscape of robot learning, that is a compelling direction: use large-scale video pretraining not just to predict the future, but to discover a universal action space that different agents can share.