Daily AI Paper

Humanoid robots are getting impressively athletic. They can run, dance, even do acrobatic motions that look almost effortless. But the paper’s starting point is that none of that automatically makes a humanoid useful in an ordinary home. A robot that can backflip is not the same as a robot that can walk into an unfamiliar living room, find a chair, sit down, and do it safely without having seen that room before. That gap matters because the tasks people actually want from general-purpose humanoids are not staged demonstrations. They happen in cluttered, unstructured spaces, with furniture in new places, doors half open, tables at awkward heights, and instructions that are expressed in natural language. To succeed, the robot has to solve two problems at once. First, it has to understand the instruction semantically: when someone says “sit on the chair near the window,” which object is the chair, and what action should be taken? Second, it has to execute the motion physically: how do you actually move your body so you make contact with the chair in the right place, keep balance, and complete the task without falling? The central bottleneck, according to the authors, is data. Mapping pixels and free-form language to low-level joint commands requires paired examples: what the robot saw, what it was told, and the exact whole-body motion that solved the task. In language and vision, scale has transformed performance because data is abundant. But for humanoid control, that kind of paired supervision is scarce and expensive. Teleoperation systems and wearable capture setups can produce demonstrations, but they tie data creation to human labor. If you want an hour of robot experience, you often need at least an hour of person-time to produce it. HALO, which stands for Humanoid Autonomy, Learned in simulation, running Onboard, is the paper’s answer to that bottleneck. The key idea is to generate the training corpus entirely in simulation, so data scales with compute rather than with human effort. Instead of asking people to physically act out every episode, HALO synthesizes the environment, synthesizes the motion, renders the robot’s egocentric observations, and uses that as training data. To understand the landscape, it helps to separate three pieces of the problem. One is motion supervision: how do you get whole-body trajectories that are good enough to train on? Another is the environment: if you only train in a few scanned rooms, you may overfit to those rooms and fail in new homes. And the third is control architecture: do you try to solve everything end to end, or do you break the problem into layers? HALO takes a hierarchical approach. At the top is an orchestrator, which is an off-the-shelf vision-language model invoked every couple of seconds. Its job is to turn a free-form request into a sequence of subtasks, such as “walk to the chair,” “turn,” or “sit.” In the middle is a motion generator that takes the current egocentric RGB observation and the active subtask and produces a short-horizon whole-body kinematic plan. At the bottom is a motion tracker that turns that plan into real robot actions using onboard depth and proprioception. So the top layers decide what to do, and the bottom layer makes sure the body actually gets there in the physical world. The synthetic data pipeline is the engine that makes this possible. Each training episode is built from three ingredients. First, a motion diffusion model generates whole-body motion. Second, a procedural scene generator creates a home-like environment. Third, the simulator places the robot in the scene and renders the resulting egocentric sensory input. The paper says the corpus spans 1,000 procedurally generated homes and amounts to 300,300 hours of episodes. The important point is not just the size, but that this scale is achieved without human demonstrations. There is also a quality-control step, because generated motion can be physically implausible. A kinematic model may produce a trajectory that looks reasonable in isolation but actually clips through the floor, slides a foot unrealistically, or contains discontinuities. HALO addresses this by generating multiple candidate trajectories for each segment and then scoring them with geometric verifiers. These verifiers check things like foot skating, floor penetration, flight phases, and joint limits. The best candidate is kept, and invalid ones are discarded. In other words, the system does not trust raw generation blindly; it filters for trajectories that are geometrically more plausible. The motion itself is built in segments. A task like “sit on the armchair” is broken into pieces such as walking forward, turning around, and sitting. Some segments use waypoint guidance, and the final segment uses a keyframe pose that defines the interaction with the object. The same decomposition can then be reused across many different procedurally generated rooms. For navigation, the system samples valid start and target placements and uses a non-holonomic unicycle planner to compute a path. Then the motion diffusion model, referred to in the paper as Kimodo, fills in the dense whole-body motion for each segment. Those segments are stitched into one continuous trajectory. The environments are generated with Infinigen, producing 1,000 distinct rooms across three types: living rooms, offices, and bedrooms. The rooms vary in size and furniture layout. This is important because the system is not relying on scanned real houses or manually annotated assets. The training world is synthetic, but diverse. To place a task into a room, the system builds a 2D occupancy grid from the layout and then samples scene placements until it finds one that avoids collisions between the robot motion, the target object, and surrounding furniture. Because every episode is defined by a random seed and a lightweight placement record, the same scene can be regenerated later without needing a full scan. On the control side, the motion tracker is trained with large-scale reinforcement learning to follow the reference motion while respecting balance, contact, and actuator limits. This is a crucial distinction. Many prior systems can generate a motion plan but then hand it to a blind tracker that lacks awareness of the real environment. HALO instead gives the low-level controller depth and proprioceptive feedback, so it can adjust while executing. That makes the tracker perceptive rather than purely reactive to joint targets. The paper positions this against several prior directions. Some systems rely on teleoperation or wearable capture, which is expensive to scale. Some train in reconstructed rooms, which can work well if you know the deployment environment ahead of time, but that is not true autonomy. Some hierarchical methods use compact task commands or latent skills, but do not pass full whole-body trajectories. And some perceptive controllers can handle narrow skills like traversing clutter, but they do not combine that with a general motion-generating interface. HALO’s claim is that it bridges these gaps: it uses simulation for scale, a hierarchical interface for compositionality, and a perceptive tracker for physical grounding. The experimental protocol is also unusually strict. The authors evaluate on six distinct real-world home environments, and none of them were scanned, reconstructed, or seen during training. They test six whole-body skills across 360 physical trials. That matters because sim-to-real papers can look good if the real test spaces are effectively part of the training pipeline. Here, the point is zero-shot transfer to previously unseen physical environments. The reported result is a 91% success rate across those 360 trials. The paper’s claim is not just that HALO can do isolated motions, but that it can perform perceptive whole-body skills in real homes it has never encountered before. Those skills include sitting on chairs, navigating to named objects, stepping sideways through half-open doors, crouching and crawling beneath tables, and pushing boxes. It can also chain these skills together for longer, multi-step instructions. So the result suggests that the hierarchy is doing real work: the orchestrator decomposes the request, the motion generator proposes a local plan, and the tracker adapts that plan to the actual geometry of the room. What do these results actually show? First, they show that a large synthetic corpus can be enough to train a humanoid policy that transfers beyond the simulator, at least for the set of skills tested. Second, they show that the system is not just memorizing one room layout, because it works in six unseen real environments. Third, they show that perception at the low level matters: the robot is not simply replaying a blind trajectory, but using onboard sensing to ground the motion in the physical scene. At the same time, there are limitations to keep in mind. The paper is explicit that the training data comes from a thousand procedurally generated homes, which is diverse, but still synthetic and still limited to the kinds of rooms and furniture the generator produces. The tested skills are impressive but bounded: sitting, navigation, side-stepping through doors, crouching, crawling, and pushing boxes. That is a meaningful repertoire, but not the full range of household manipulation. The system also depends on a layered pipeline with several learned and engineered components, so success likely reflects careful integration as much as any single model. Another thing to watch is that the motion generator is kinematic, so physical plausibility still has to be enforced by filtering and by the tracker. And while the paper demonstrates zero-shot transfer to real homes without scanning them, it does not imply the problem is solved for arbitrary environments or arbitrary instructions. Generalist humanoid autonomy will likely require broader skills, richer object interactions, and even more robust adaptation to unexpected situations. Still, the main contribution is clear: HALO shows that if you can generate large-scale vision-language-motion data in simulation, and if you couple that with a perceptive low-level controller and a task-level orchestrator, you can get a humanoid to perform useful whole-body behaviors in real, unseen homes without ever having trained on those homes directly. That is a meaningful step toward robots that are not just athletic in the lab, but actually useful in the world people live in.