Daily AI Paper

Today’s paper looks at a practical weakness in modern language model training: what happens when we run out of fresh human-written text and start reusing the same data over and over. The authors compare dense Transformers with Mixture-of-Experts, or MoE, models, which are popular because they can scale to many total parameters while only activating a small subset at each step. Their key finding is surprising: MoE models overfit repeated data much faster than dense models, and the problem gets worse as sparsity increases. In some settings, dense models can tolerate eight repetitions with little loss, while MoEs start degrading around four. The paper also tests regularization tricks and finds that methods like dropout can help, but none fully recover the benefit of truly unique data. This matters because repeated training data is becoming the norm, and it shows that the efficiency gains of sparse models may come with a hidden cost in data hunger and robustness.