Daily AI Paper

Large language models are getting better at reasoning, but many of the best test-time methods depend on extra labels, repeated sampling, or expensive verification. This paper asks a different question: can a model improve itself at test time using only its own guesses? The authors propose Test-Time Policy Optimization, or TTPO. The key insight is that when a majority-vote pseudo-label is wrong, the model’s rollouts that disagree with it are often wrong too, while agreeing rollouts are usually the safer signal. TTPO turns that asymmetry into an optimization rule: it distills the agreeing answers and applies reinforcement learning pressure to the disagreeing ones, while also focusing the update on the most informative tokens. The result is a label-free method that can still sharpen reasoning, especially in math tasks. What makes this interesting is that it brings together self-training, policy optimization, and test-time adaptation in a way that is both practical and surprisingly strong, narrowing the gap to supervised methods without needing ground-truth answers.