Daily AI Paper

Sequential decision-making gets much harder when the agent cannot fully observe the world. In robotics, navigation, and many embodied AI tasks, the agent often sees only a partial, noisy slice of what is really going on. That means the policy cannot just react to the current observation; it has to gather information first, remember what it has already tried, and then use that history to decide what to do next. This is the core challenge of partially observable decision-making, and it is what this paper is about. Why does this matter? Because in many real problems, the key hidden information is exactly what determines success. A household robot looking for an object does not know where it is at the start. A navigation agent may not know which route is blocked or which goal is correct. A manipulation policy may need to infer the state of an object from limited sensing. If you try to learn these behaviors from scratch with reinforcement learning, the agent must simultaneously learn how to explore and how to exploit what it discovers. That is notoriously sample-inefficient. The paper starts from a common workaround: training with a privileged expert. During training, the expert can see the true state, while the deployed student only gets partial observations and history. This is attractive because the expert can teach the student what to do once the hidden state is known. But there is a catch. A teacher that already knows the answer usually does not need to explore. So if you simply imitate the teacher’s actions, the student learns how to act after information is available, but not how to find that information in the first place. This paper argues that the standard imitation setup is missing something crucial: the student must be trained on the kinds of histories it will actually encounter when it explores on its own. If the student only sees expert-generated trajectories, it never practices the act of searching while uncertain. Instead, it tends to average over incompatible expert actions, which can lead to indecision rather than exploration. The method the authors propose is called ASTEROID, short for Asymmetric Student Teacher Roll-Out In-context Distillation. The idea is simple but powerful. Instead of letting the expert generate the whole trajectory, the student first rolls out on its own under partial observability. That produces a real interaction history: observations, actions, and rewards collected from the student’s own behavior. Then the privileged expert is asked to label the action that would be taken from the true hidden state reached by that student history. So the student is not trained on expert trajectories from scratch. It is trained on its own on-policy contexts, with the expert providing labels for those contexts. That means failed searches, probes, and detours are no longer wasted. They become useful evidence about what the student has ruled out. Over time, the student learns to condition on this history, eliminate inconsistent hypotheses, and choose actions that continue the search when uncertain and switch to exploitation once enough information has been gathered. A helpful intuition in the paper is the three-door or three-location example. Suppose the target could be at A, B, or C. If the student checks A and finds nothing, that observation rules out A but still leaves B and C possible. The expert’s label depends on the true hidden state, so for histories consistent with B it will suggest searching B, and for histories consistent with C it will suggest searching C. As the student accumulates more evidence, the set of plausible hidden states shrinks, and the expert labels become more specific. In this way, the student learns not just a final action, but a whole information-gathering strategy. The paper connects this behavior to posterior sampling, a classic idea in exploration. In posterior sampling, an agent maintains a belief over hidden states or parameters, samples one possibility from that belief, and acts optimally under that sampled hypothesis. That tends to balance exploration and exploitation naturally. The authors’ claim is that ASTEROID can approximate this kind of behavior without explicitly maintaining a posterior. Instead, the posterior-like behavior emerges from iterative supervised learning on the student’s own histories. This is important because explicit posterior methods are often hard to scale. They require either maintaining a belief over hidden variables or solving for policies under sampled hypotheses, which becomes impractical in large or high-dimensional problems. ASTEROID avoids that by staying within the familiar teacher-student distillation framework. It uses modern sequence models and supervised learning, but changes what data the student is trained on: on-policy histories rather than offline expert rollouts. The paper also places this in context with related work. Meta-RL learns to adapt through recurrent policies, but usually needs a lot of online interaction. Other imitation methods can use teacher supervision, but often rely on assumptions like full coverage of possible histories in an offline dataset, or on extra machinery such as differentiable experts or RL-style bootstrapping. ASTEROID’s main contribution is to show that you can get exploration behavior from a much simpler supervised setup, as long as the student generates the right contexts itself during training. The formal setup is a partially observable Markov decision process, where the agent does not see the true state directly. The student policy depends on the current observation and the full interaction history. The teacher, by contrast, can access the true state during training. The goal is to train the student to maximize expected return under partial observability. This asymmetric setting is natural in robotics, navigation, sim-to-real transfer, and other domains where a privileged simulator or oracle is available in training but not at test time. Now, what do the experiments show? The authors evaluate ASTEROID across eleven environments, spanning discrete exploration benchmarks, robotic manipulation, navigation, and Procgen-style environments. This matters because the method is not being tested only on toy problems. The environments include settings where exploration is genuinely needed, and also higher-dimensional domains where simple posterior methods would be difficult to apply directly. The headline result is that ASTEROID achieves about twice the return under fixed exploration budgets compared with baselines, while using roughly one hundred times fewer interactions than reinforcement-learning baselines. That is a very strong sample-efficiency story. In other words, it does not just eventually learn a good policy; it learns it with far less training data than RL methods that have to discover exploration behavior by trial and error. The paper also reports that ASTEROID outperforms frontier models at exploration. The key point here is not just raw performance, but the kind of behavior learned. The student does not simply wander randomly. It learns to search while uncertain and then exploit once the hidden state becomes clear. That is exactly the behavior you want in a partially observed task. A particularly interesting result is the sim-to-real transfer experiment. The authors transfer policies trained in simulation to real-world robot manipulation in a setting where vision is denied. Even there, ASTEROID shows efficient exploration and improves success by forty percent over standard distillation. That suggests the method is not merely fitting simulation quirks; it is learning a robust exploration strategy that survives a harder deployment setting. The paper also mentions a curriculum over the student rollout horizon. In plain terms, the student is not forced to solve the hardest long-horizon exploration problem immediately. Training can begin with shorter rollouts and gradually extend the context length. This improves sample efficiency and helps the model learn to adapt over longer histories. That is a practical detail, but an important one, because long-horizon partial observability is exactly where memory and history conditioning become difficult. So what should we take away from the results? First, that privileged supervision can teach exploration, but only if the student is trained on its own interaction histories rather than on expert trajectories alone. Second, that exploration can emerge from supervised learning without explicit reinforcement-learning-style search over policies. And third, that this works not just in theory, but across a broad set of environments, including real robot transfer. At the same time, there are limitations and open questions. ASTEROID still relies on access to a privileged expert during training, so it is not solving exploration from nothing. Its success depends on the quality of that teacher and on the assumption that the teacher can label the student’s visited histories. The method also depends on the student actually reaching informative histories during training; if the on-policy rollouts never cover useful regions, learning can still be limited. The paper suggests that a curriculum helps with this, but coverage remains a practical concern. Another thing to watch is that the theoretical connection to posterior sampling is described as holding on the covered history support, which is an important qualifier. In other words, the posterior-sampling-like behavior is not magic everywhere; it is tied to the histories the student actually experiences and learns from. That is a strength, because it makes the method practical, but also a reminder that exploration quality still depends on training dynamics. Overall, the message of ASTEROID is refreshingly simple. If you want a partially observed agent to learn how to explore, do not just show it expert actions from fully informed trajectories. Let the student generate its own uncertain histories, and then distill the expert’s action choices conditioned on those histories. That turns failed searches into training signal, makes exploration learnable with supervised learning, and gives a scalable alternative to expensive reinforcement learning for test-time exploration.