Daily AI Paper

When you watch a video model make a monkey leap over a crossbar, it can look astonishingly alive. But there is a hidden limitation: what you get is pixels, not reusable 3D motion. If you want that same leap to drive a specific rigged asset in a 3D scene, or a robot body, or a character with a very different skeleton, the video alone is not enough. You would still need to recover the motion, recover the camera, and make sure the motion actually sits correctly inside the scene. That is the problem this paper tackles. The key idea is simple to state but hard to do well: instead of generating only video, or generating motion and then trying to infer the rest afterward, the model should co-generate both the video and the 3D motion at the same time. And not just for human skeletons. The goal here is arbitrary rigs: any supplied skeleton structure, whether human, animal, or robot-like. The model should also learn where that motion lives in the scene, so the 3D animation can be replayed in the right place with the right camera. This matters for a few reasons. In animation, you often want a text prompt to become a usable asset animation, not just a clip. In robotics, a motion prior is more useful if it can be attached to a physical body and scene. And for synthetic data generation, a model that can produce both rendered video and the underlying 3D motion can serve as a much better training or editing tool than a video-only generator. The broader trend in the field has been to make generative models more world-like, but most video models still stop at the image plane. There is prior work on joint video and motion generation, but the paper argues that existing approaches have important limits. Some are human-specific. Some use very large motion branches tied closely to the video backbone. Others generate 2D pose maps from video and then triangulate to 3D, which requires multi-view data and adds an indirect recovery step. That indirectness is exactly what this paper wants to avoid. If the model is going to produce 3D motion, it should do so directly, and if the motion is going to be placed in a scene, the model should learn that registration too. The architecture is built around a division of labor. Video and motion are kept as separate pretrained backbones, then connected with a small number of cross-attention layers. That is an important design choice. Instead of forcing everything into one giant unified model, the authors let a video model retain its learned visual priors and let a motion model learn skeleton dynamics on its own. The motion model is designed for arbitrary rigs by treating the skeleton as a graph: joints are nodes, bones are edges. Because it is trained on motion alone first, it can learn from motion data that is easier to obtain and augment than paired video-motion data. The motion representation itself is fairly rich. Each joint carries rotation and position information, plus frame-to-frame changes, so the model has enough structure to represent articulated movement in 3D. The rig is encoded into per-joint descriptors using information such as the joint’s name, its rest-pose geometry, whether it is a root, chain, or leaf joint, and, when available, its location in a reference image. A graph attention network passes information along the skeleton, and then full self-attention lets distant joints communicate too. Those descriptors condition the motion denoiser throughout generation. On the video side, the model starts from a pretrained video generator. The two systems are connected with zero-initialized bidirectional cross-attention layers. Zero initialization matters because it preserves the behavior of both pretrained models at the start of training, so coupling can begin gently rather than destroying what each model already knows. During denoising, the video and motion streams influence one another, so the motion is not just an afterthought attached at the end; it is shaped jointly with the visual generation. The paper also introduces what it calls bridge guidance. At inference time, it compares predictions with the video-motion connections enabled and disabled, then amplifies the change caused by that interaction. In plain language, this is a way of encouraging the model to pay attention to the agreement between the two modalities. It is a bit like asking, “What changes when motion and video talk to each other?” and then strengthening that effect. But co-generating motion and video still leaves one major question unanswered: where is the motion in the scene? A jump can be generated, but is the character on the left side of the frame, near the crossbar, or somewhere else entirely? To solve this, the model learns to localize each joint in the image. It predicts where every 3D joint appears in the generated frame. Those 2D joint locations, together with the 3D joint positions, give correspondences that can be used to fit a camera. That camera then registers the motion back into the scene, so the 3D animation can be replayed consistently with the generated video. Training data is another major challenge, because there is no large corpus that naturally pairs diverse rigs, real scenes, and 3D motion. The paper combines two complementary sources. First, human video provides rich motion and rich backgrounds, but only a fixed human skeleton. Second, animated 3D assets provide diverse rigs and exact motion labels, but their renders are simple. To bridge that gap, the authors use a generative video editor to place rendered animated assets into scenes that suit their actions. They carefully verify the edits so that the known 3D motion remains valid. They also add robot video. Altogether, this gives them nearly 200,000 video-motion pairs spanning diverse skeletons. The experimental setup is therefore broad in both data and tasks. The model is evaluated on text-driven animation of diverse rigs, and the paper emphasizes that the motion is more faithful than in video-first pipelines. It is also tested in the reverse directions: video to motion, and motion to video. In other words, if you supply video, the model can recover motion for a rig; if you supply motion, it can generate video; and if you provide a few reference image keyframes, it can produce a full 3D rig trajectory and a matching video. That last capability gives a form of image-based control over motion, which is quite useful for directing animation. What do the results actually show? The main claim is not just that the model can produce plausible motion and plausible video separately, but that the two stay aligned. The motion responds to the surrounding scene, which is important because in realistic action, the environment often determines what movement makes sense. A crossbar should change how a monkey jumps; a scene with different obstacles should lead to different motion. The paper argues that this scene sensitivity is better than what video-first pipelines deliver, because those pipelines do not explicitly maintain a reusable 3D motion representation tied to the scene. Another result is that the model can turn video into rig motion and rig motion into video within the same framework. That suggests the learned representation is not one-way. It is not just a generator that can be prompted in one direction; it is a coupled model that can bridge between the visual and articulated domains. The learned joint localization and camera fitting are what make the generated motion replayable in the scene, rather than merely plausible in isolation. There are, however, important limitations to keep in mind. The paper itself points to the data challenge: because paired real video and 3D motion across arbitrary rigs is scarce, the training corpus has to be assembled from multiple sources and edited renders. That means the system still depends on a carefully curated pipeline rather than purely natural data. The scene-registration step also relies on predicted joint locations and camera fitting, so errors there could affect replay quality. And while the model supports arbitrary rigs, it is still learning from the statistics of the rigs and motions it has seen, so unusual skeletons or very out-of-distribution actions may remain difficult. A final point is that the method is parameter-efficient compared with approaches that bolt a huge motion branch onto a video backbone, but it is still a complex coupled system. The benefit of separating motion and video is flexibility, yet the tradeoff is that success depends on the quality of the coupling, the guidance, and the training data pipeline. The paper’s contribution is to show that this separation can work well enough to make arbitrary-rig co-generation practical. So the big picture is this: Joints, Camera, Action! tries to turn video generation from a pixel-only trick into something closer to a usable 3D animation engine. It does that by combining a rig-aware motion model, a pretrained video generator, learned joint localization, and a data pipeline that spans human, robot, and edited asset videos. The result is a single framework that can generate video and 3D motion together, move in both directions between them, and recover the camera needed to place the motion back into the scene. For animation and embodied generation, that is a meaningful step from making things look alive to making them actually usable.