Rethinking On-Policy Distillation of Large Language Models II: One Training Example
Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang, Jinqian Zhang, Ruhang Xiao, et al.
arXiv:2609.04172v1Today’s most broadly interesting paper is about how little data large language models may actually need to learn a task through on-policy distillation. The authors ask a simple but surprising question: if a student model is trained from its own rollouts with teacher supervision, do you really need a full dataset, or could one example be enough? They find that one-shot distillation keeps improving for hundreds of steps and can recover most of the gain from full-data training. The key idea is state coverage: a single query can already drive the student through a large fraction of the same internal states that many examples would, so the training signal is much broader than it looks. More diverse queries help, but the bigger bottleneck is not data quantity so much as how slowly the student absorbs the supervision. That matters because it suggests we may be overpaying for data in some post-training pipelines, and that future progress could come from making optimization more step-efficient rather than simply scaling up the dataset.
Also spotted that day
Previous daily papers