Daily AI Paper

Text-to-image generation has become one of the clearest examples of how large-scale generative AI can turn language into a visual medium. But most modern systems rely on a very expensive assumption: that you need internet-scale language pretraining before you can make a strong text-to-image model. This paper asks a simple but important question: is that actually necessary? That question matters for both scientific and practical reasons. Scientifically, it gets at what language pretraining is really buying you in multimodal generation. Practically, language pretraining often dominates the total compute budget, and that cost is sometimes hidden when people talk about the price of building a text-to-image system. If a model can learn to follow captions well without first becoming a giant language model, then the field may be overpaying for a capability that is not essential. The authors introduce Halcyon, a family of early-fusion multimodal autoregressive models trained entirely from scratch on image-text data. “From scratch” here is important: they train the text tokenizer, the image tokenizer, and the generative model itself without initializing from a pretrained language model. The core claim is not that language pretraining is useless for language tasks in general, but that for text-to-image generation specifically, paired image-text supervision may already be enough. To understand the landscape, it helps to distinguish the two main design patterns in modern text-to-image systems. The first is late fusion. In this setup, a language model is pretrained separately on text-only data, and its representations are then used to condition an image generator. This is the dominant approach in many recent systems. The second is early fusion. Here, text and image tokens live in one unified sequence, and the model learns both modalities together. Even in early-fusion systems, though, many recent methods still rely on text-only pretraining before multimodal training. So the difference is not just architectural; it is also about where the language knowledge comes from. The authors focus on a dataset called GPIC, which contains about 100 million image-text pairs. They use it to run controlled experiments along three dimensions. First, they train everything from scratch, including tokenizers. Second, they vary how the training budget is split among text-only, text-to-image, and image-to-text examples. Third, they compare Halcyon against representative baselines under matched or comparable compute settings, so they can ask what language pretraining actually contributes when compute is counted honestly. The model itself is a decoder-only Transformer with a joint vocabulary for text and image tokens. The key idea is straightforward: instead of treating captioning and image generation as separate problems, Halcyon learns to predict the next token in a single sequence that can contain either text or image tokens. That is a standard autoregressive formulation, but applied to a multimodal token stream. For images, the paper trains a discrete image tokenizer from scratch. Each 256-by-256 image is compressed into a sequence of 1,024 image tokens by downsampling by a factor of 8 in each spatial dimension. The tokenizer uses an encoder-decoder design with finite scalar quantization. For text, they train a byte-level BPE tokenizer from scratch on GPIC captions plus OpenWebText. The final system uses a shared embedding table and output head over a combined vocabulary of text tokens, image tokens, and special symbols. The training data is organized into four sequence types. Text-only sequences are used for language modeling. Image-only sequences predict image tokens from a start token. Text-to-image sequences predict image tokens conditioned on a caption. And image-to-text sequences do the reverse: the model sees an image and then predicts the caption. This last type matters because it lets the same model learn image understanding and captioning, not just generation. They also use classifier-free guidance, which is standard in text-to-image generation. In practice, that means sometimes dropping the caption during training so the model learns both a conditional and an unconditional branch. At inference time, those two branches are combined to steer the output toward the prompt. The authors note that image-to-text sequences also help train the unconditional branch because they share the same image prefix as unconditional text-to-image examples. A particularly important part of the paper is the set of controlled pretraining schedules. The authors compare five configurations. One is a baseline that uses only text-to-image examples. Another is staged training where the model first gets text-only pretraining and then switches to text-to-image. A reverse staged setup starts with image-to-text and then moves to text-to-image. They also test mixed training, where image-to-text and text-to-image are interleaved throughout training. Finally, they test a variant that removes caption dropout and replaces some text-to-image examples with image-to-text supervision, while keeping the same overall budget. The experimental setup is designed to be as controlled as possible. All main configurations are trained for one epoch on GPIC at 256-by-256 resolution, with the same global batch size and the same learning-rate schedule. The model family is scaled across five sizes: 0.1B, 0.25B, 0.52B, 1.6B, and 4B parameters. They use a sequence length of 1280 and a caption-dropout rate of 0.1 unless otherwise noted. The paper also evaluates the tokenizers themselves. The image tokenizer is trained with reconstruction and perceptual losses, followed by a short adversarial phase. Compared with the Cosmos-DI-8x8 tokenizer, the authors report that their tokenizer performs better across all reconstruction metrics, although, as with prior discrete tokenizers, small text and fine glyphs remain difficult. For the text tokenizer, they report that even with a much smaller vocabulary than Qwen3, it achieves comparable compression on GPIC test captions, and better compression than the FLAN-T5 tokenizer. These results matter because they show the authors are not relying on a hidden advantage from a much better tokenizer; the system is being built from the ground up, and the components are competitive. Now to the main question: does language pretraining help? The answer from these experiments is no, not in the way one might expect. The headline result is that Halcyon improves steadily as model size increases, and the larger models are strong despite having no language pretraining at all. Halcyon-0.52B achieves a generation Fréchet distance of 48.0, outperforming LlamaGen-XL/8, which gets 63.3, while using 18.2 times less total pretraining compute. Halcyon-1.6B reaches 35.3 gFD, outperforming JiT-T2I-XXL/8++, which gets 41.0, while using 275 times less pretraining compute. Since lower gFD is better, these are meaningful gains, not just small differences. The compute comparison is especially important. The paper argues that once you count total pretraining compute, including the language side, the cost of systems that depend on pretrained text models is much higher than it first appears. Halcyon’s results suggest that strong text-to-image performance does not require spending most of your budget on text-only pretraining. The controlled scheduling experiments reinforce that point. When the authors vary how much text-only pretraining is used, they find that external text-only pretraining does not improve text-to-image generation. In other words, giving the model a large dose of generic language learning before multimodal training does not translate into better image generation from captions, at least under their setup. That is one of the central empirical claims of the paper. They also show something more nuanced and arguably more useful: Halcyon can learn image captioning without sacrificing generation quality. By replacing caption-dropped text-to-image examples with image-to-text supervision, the model learns to caption images while preserving competitive text-to-image performance under the same pretraining budget. That suggests the multimodal objective is flexible enough to support both understanding and generation, without needing a separate language-pretraining phase. What do these results actually show? First, they show that paired image-text data is a sufficient and effective supervision signal for strong text-to-image generation. Second, they show that the benefits of text-only pretraining are, at best, not obvious in this controlled setting. Third, they show that early-fusion autoregressive models trained from scratch can compete with, and in these tests outperform, strong baselines that rely on pretrained language models. There are, of course, limitations to keep in mind. The experiments are centered on GPIC and a specific training recipe, so the conclusions are strongest for that regime. The paper does not claim that language pretraining is never useful in any multimodal system, or that it would not help in other data regimes, other resolutions, or other downstream tasks. Also, the image tokenizer still struggles with fine text and glyphs, which is a known weakness of discrete image tokenization. And while the paper compares against representative baselines, the broader T2I ecosystem includes many architectural and training differences, so absolute generalization beyond these comparisons should be made carefully. Still, the main takeaway is clear: for text-to-image generation, language pretraining is not a prerequisite for strong performance. Halcyon shows that a model can be trained entirely from scratch, using only image-text supervision, and still reach competitive or better generation quality than systems that spend far more compute on text-only pretraining. The broader implication is that multimodal generation may benefit more from better use of paired data and better training design than from simply scaling generic language pretraining. That is a useful correction to a prevailing assumption in the field, and it gives researchers a simpler, cheaper recipe to explore next.