Daily AI Paper

General-purpose robots need to handle a huge range of behaviors. A legged robot may have to cross irregular terrain, while an arm may need to grasp, align, insert, or thread parts together with very little margin for error. This paper is about how to train those kinds of robots with reinforcement learning in simulation, and specifically how to make large-scale simulation actually pay off when the task is hard. The basic promise of sim-to-real reinforcement learning is attractive: train in a physics simulator, then transfer the learned policy to the real robot. Over the last several years, this has produced impressive results in locomotion and manipulation. But there is a catch. In practice, these systems often depend on a lot of task-specific engineering. You need carefully shaped rewards, hand-designed curricula, demonstrations, or other priors that make exploration easier. That means every new task becomes a new engineering project. The paper argues that this is exactly the opposite of what we want if the goal is a general-purpose robot training recipe. A more recent line of work has tried to reduce that burden by improving the starting states the robot sees during training. Instead of always beginning from a standard reset, the simulator can reset the robot into useful intermediate states, so the policy does not have to discover everything from scratch. This is especially helpful for contact-rich tasks like assembly, where the probability of stumbling into a useful state by random exploration is tiny. The authors build on that idea, and on prior work like OmniReset, which showed that diverse, programmatically generated resets can make difficult tasks learnable with massive parallel simulation. But they identify a second bottleneck. Even if you have many useful reset states, you still have to decide how to allocate training across them. If you sample them uniformly, a lot of experience gets wasted. Some resets are already too easy, so the policy gets little new learning signal. Others are far too hard, so the policy cannot yet make progress. As you scale up to more and more parallel environments, this problem gets worse, because a growing fraction of your batch can be spent on configurations that are not currently informative. In other words, the bottleneck is not just exploration in the usual sense, but exploration allocation. That is the central idea behind Success Guided Sampling, or SGS. The method is simple: keep track of how successful the policy currently is on different task configurations, and preferentially sample the configurations where the policy is in the productive middle ground. Those are the cases that are neither solved nor hopeless, but right near the frontier of what the policy can do. The intuition is very similar to a curriculum, but importantly it is not a manually ordered curriculum. The policy’s own success rate determines where training should focus. To make this concrete, the paper sets up goal-conditioned reinforcement learning. Each episode starts from a task configuration that includes the initial state, the goal, and the environment setup. In locomotion, that means things like terrain type, terrain geometry, and the robot’s starting and target poses. In manipulation, it means the assembly task, object and robot initial states, and the receptacle pose. The policy receives a shared reward within each domain: one reward function for all locomotion tasks and one for all manipulation tasks. That is important, because it means they are not hand-tuning a separate reward for every terrain or every assembly problem. SGS itself works over a fixed set of sampled configurations. For each configuration, it maintains a rolling estimate of recent success, based on the last several episode outcomes. Then it scores each configuration with a smooth beta-shaped weighting function that favors a target success rate. You can think of that target as the sweet spot of learning: not too easy, not too hard. Configurations near that target are sampled more often, while configurations with very high or very low success still remain in the mix with nonzero probability. That nonzero probability matters, because it keeps the success estimates fresh and prevents the sampler from abandoning difficult configurations entirely. The paper uses this same basic mechanism in two domains. The first is difficult-terrain quadruped locomotion, where the robot must learn whole-body control across a range of terrains. The second is dexterous assembly, including tasks such as threaded assembly, insertion, connector assembly, and gear meshing. These are contact-rich tasks where precise alignment and sustained interaction matter, and they are exactly the kind of problems where naive exploration tends to struggle. The experimental setup is notable for scale. The authors train with PPO and massive parallel simulation, using up to about one million parallel environments. For manipulation they use a fixed set of 32,768 task configurations, and for locomotion 104,000. They use a rolling window of recent successes, and they choose different target success levels and kernel settings for locomotion and manipulation. The paper also emphasizes that SGS is not especially sensitive to most of these hyperparameters, which is encouraging because it suggests the method is not brittle. A key design choice is that SGS is paired with shared rewards and no demonstrations. That makes the setup much cleaner than many prior robot RL pipelines. Instead of relying on per-task reward engineering or demonstration bootstrapping, the system tries to let the reset distribution and the adaptive sampler do the heavy lifting. The authors are very explicit that the contribution is not a brand-new theory of curriculum learning. Rather, it is a practical and surprisingly effective way to combine success-based sampling with diverse resets and large-scale RL. So what do the results show? The headline result is that SGS makes scale actually useful on tasks where uniform sampling runs into diminishing returns. Across their benchmarks, SGS outperforms uniform sampling and Prioritized Level Replay on the hardest locomotion and assembly problems. In particular, it helps on challenging multi-terrain whole-body locomotion and on nut-and-bolt assembly, where prior methods fail to solve the tasks. On simpler tasks, SGS performs about comparably to uniform sampling, which is also what you would hope to see: the method is not meant to hurt easy problems, but to concentrate learning where the frontier is moving. The broader takeaway is that more parallel environments are only helpful if those environments are producing informative experience. If you simply add more workers and keep sampling configurations uniformly, you can end up with more wasted data rather than more learning. SGS changes that balance by directing the batch toward the configurations that are currently most useful for improvement. That is why the paper frames the problem as an exploration bottleneck in mega-scale RL: the issue is not just whether the robot can explore at all, but whether the training system is spending its compute on the right part of the task space. The paper also includes a real-robot transfer result. The learned manipulation policies are distilled into RGB-based policies and then transferred zero-shot to several challenging assembly tasks on hardware. Zero-shot transfer is important because it means the learned behavior carries over without additional real-world fine-tuning. The paper presents this as evidence that the training recipe is not just succeeding in simulation, but learning skills that are robust enough to survive the sim-to-real gap for precise assembly tasks. There are, of course, limitations and open questions. SGS depends on having a meaningful set of task configurations to sample from in the first place, so it is not a replacement for good environment design. It also uses success rate as the main signal for where to sample next, which is simple and effective, but may not capture every aspect of learning progress. The authors do compare against a few related methods, including uniform sampling, PLR, and in locomotion an SFL-style baseline, but they are careful not to claim that SGS is a fundamentally new curriculum principle. It is a practical instantiation of a broader idea. Another thing to watch is that the strongest gains appear on difficult tasks at large scale. That suggests the method’s value grows when the configuration space is broad and the environment count is high. On simpler tasks, the benefit is smaller. So SGS seems best understood as a scaling tool: it helps large-scale simulation avoid wasting its own capacity. In the end, the paper’s message is quite clear. If you want general-purpose robot learning, it is not enough to make resets diverse. You also have to sample those resets intelligently. Success Guided Sampling is a lightweight way to do that, and in this work it turns massive parallel simulation from a blunt instrument into something much more targeted. That lets the authors solve harder locomotion and assembly problems than prior approaches, while also showing real-world transfer for manipulation. For anyone interested in robot RL, the key lesson is that data quality is not only about what states you include, but about when and how often you train on each one.