Daily AI Paper

Generalist robot policies are an exciting step toward robots that can handle many tasks with one model. Instead of training a separate policy for each job, a Vision-Language-Action model can take a language instruction and produce actions across a broad range of manipulation tasks. That’s powerful, but there’s a gap between being able to do many things in principle and being reliable enough to deploy. In practice, these models often need adaptation after pretraining. The question this paper asks is: can we improve a generalist robot policy online, across many tasks at once, and can that improvement carry over to new tasks the robot never saw during adaptation? That question matters because most prior reinforcement-learning-based post-training for robots has been studied in a narrow setting: one deployment task, one environment, one objective. That makes sense if the only goal is to boost performance on a single task. But it’s a much poorer match for the promise of foundation models, where the same policy is supposed to generalize across a whole family of tasks. If online RL only helps on the exact task you train on, then every new task still needs its own expensive fine-tuning phase. This paper is trying to move beyond that single-task mindset. To understand the landscape, it helps to separate a few different ways of improving a robot policy. One option is direct policy adaptation: collect successful trajectories and fine-tune the policy on them, often with filtered supervised fine-tuning. This is simple and scalable. Another option is critic-guided test-time refinement: keep the policy fixed, learn a value function or critic that scores actions, and use that critic to choose better actions at inference time, for example with Best-of-N sampling. A third option is critic-guided policy extraction: use the critic during training to update the policy itself so it produces higher-value actions. The paper studies representative versions of all three, including filtered supervised fine-tuning, Best-of-N, and off-policy generative policy optimization. The core idea is to ask not just whether these methods improve performance on the tasks they were trained on, but whether they also produce broader generalization. The authors define this as “generalist improvement”: the policy is adapted online on a set of training tasks, but the goal is to improve performance on a test distribution of tasks that may be different from the training tasks. Importantly, the test tasks are not assumed to be arbitrary or unrelated; they should be related enough that transfer is plausible, but different enough to make generalization meaningful. Their experimental setup is structured around a contextual Markov decision process. In plain terms, each task is represented by a context, such as a language prompt paired with a manipulation environment. The policy sees the state and the task context and outputs actions. The researchers assume an initial generalist policy that already has non-zero but imperfect zero-shot performance. Then they run repeated online iterations: for each round, they freeze the policy, collect episodes on a set of training tasks, and then update the policy using that newly gathered experience. This is meant to resemble a realistic deployment setting, where a robot fleet gathers experience over time rather than being retrained from scratch in a lab. A key part of the paper is that they do not just compare algorithms; they also study the task distribution itself. They vary both how many training tasks are used and how diverse those tasks are. That distinction turns out to matter a lot. It is not enough to simply have more tasks; the tasks need to span different solution strategies. The authors argue that diversity in the required behavior is especially important for learning policies that generalize beyond the training set. What do they find? First, yes, generalist policies can be improved online across diverse tasks. More than that, the right training recipe can enable generalization to tasks that were not part of the online training set. That is one of the main headline results: improvement is not confined to the exact tasks used for adaptation. Second, the amount of generalization grows as the number and diversity of training tasks increase. So the benefit is not just from seeing more data, but from seeing a broader range of task structure. Third, among the algorithmic choices, critic-guided methods are clearly stronger than direct policy adaptation alone when it comes to generalization. The paper reports that critic-guided approaches can generalize across training scenes and, somewhat surprisingly, across object types. That is an important point: the policy is not merely memorizing a particular visual setup or a fixed set of objects. It is learning something more abstract about how to solve tasks. Even within the critic-guided family, not all methods are equal. Critic-guided policy extraction, using an off-policy generative policy optimization style update, outperforms test-time refinement with Best-of-N sampling, while showing similar scaling trends as task diversity increases. In other words, using the critic not just to select actions at inference time, but to actually shape the policy, appears to be more effective for building a broadly improved robot foundation model. The paper’s interpretation is that critic learning is the key ingredient enabling scalable generalist improvement. Direct supervised-style updates can help, but they are less effective at producing transfer. The critic seems to provide a richer learning signal that can organize experience across tasks and support generalization to new ones. The authors also emphasize that this is not just about the algorithm; the training task distribution is equally critical. If the training tasks do not contain enough diversity in the kinds of solutions required, generalization is much weaker. A useful way to think about the result is this: if you only train on one task, reinforcement learning can specialize the policy to that task. But if you train on a carefully chosen family of tasks, online RL can act more like a compressor of useful behavior across the task family, improving the shared policy in a way that transfers. That is a much closer match to the promise of robot foundation models. The paper’s limitations are also worth keeping in mind. First, the setting still assumes that the training and test task distributions are related; the authors explicitly say generalization to arbitrary tasks is unrealistic. So this is not a claim of universal transfer. Second, the work studies structured evaluations and representative algorithmic recipes, but it is still an empirical study rather than a complete theory of when generalist improvement must happen. Third, the strongest results depend on value-based RL and on having a sufficiently diverse training distribution, so there is still a practical design problem: how do we choose those tasks efficiently in real deployments? The broader takeaway is encouraging. The paper provides some of the first evidence that online reinforcement learning can improve robot foundation models in a genuinely generalist way: across multiple tasks, not just one, and with benefits that extend to unseen tasks when the training setup is right. That suggests a path toward amortizing adaptation costs over a diverse set of tasks instead of paying for separate fine-tuning runs every time a robot encounters a new deployment setting. In the long run, that could make generalist robot policies not just versatile at pretraining time, but truly adaptable in the field.