Daily AI Paper

Today’s paper is about a very practical problem in modern AI: once you’ve trained a large transformer, how do you serve it efficiently when the hardware, traffic, and latency requirements are constantly changing? The usual answer has been to publish a few fixed sizes of the model, like small, base, or large. But real deployments are not that tidy. One day the system is under heavy load and you want something cheaper and faster; another day you have more compute available and you’d like the best possible quality. If your only options are a handful of fixed checkpoints, you often end up with a compromise that is not ideal for either case. This paper argues for something more flexible: a single pre-trained transformer that can be adjusted at inference time to fit a range of budgets. That idea is often called elastic inference. Instead of choosing one compressed model ahead of time, you want one model that can be pruned on the fly, depending on the available compute. That matters not only for latency-sensitive applications, but also for systems with fluctuating workloads, heterogeneous hardware, and even retrieval pipelines where incompatible embeddings can become a headache if you maintain many separate pruned variants. There is already a large pruning literature behind this. Structured pruning removes whole functional units, such as MLP neurons or attention heads, and that is especially useful because it can speed up inference on ordinary hardware, without needing special sparse kernels. The challenge is deciding what to remove. Classical methods use magnitude or saliency scores, and more recent methods try to estimate importance more carefully. But most of these methods still produce one sparse model for one target budget. If you want several budgets, you often have to build several separate models. The closest prior direction here is a recent elastic-vision-transformer approach that made pre-trained models adaptable without retraining, by searching for a global ranking of units. But that method used one importance score per MLP block and attention head, which kept the search manageable while preserving the original ordering of neurons inside each MLP block. That detail is important: if a neuron with a low initial score turns out to matter, you still can’t keep it without also keeping all the higher-ranked neurons above it in that block. In other words, there is hidden elasticity inside the block that the coarse ranking cannot express. The core idea of this paper is to recover that hidden flexibility. Instead of ranking blocks or heads as coarse groups, the authors assign an importance score to every individual MLP neuron and every attention head, and they optimize those scores jointly. The key technical move is to make the pruning process differentiable. Hard pruning is a discrete decision: keep this unit, remove that one. That is not something you can optimize directly with gradient descent. So they replace the hard cutoff with a soft mask, which lets the model’s loss vary smoothly as the importance scores change. Then they optimize the scores to reduce loss, while keeping the original pre-trained weights frozen. A useful way to think about this is that they are not retraining the transformer, and they are not changing its weights. They are only learning a better ordering of its units. Once that ordering is learned, any sparsity level can be obtained by simply cutting off the bottom of the ranking. That makes the model elastic: one ranking, many budgets. The method starts from an initial importance estimate. For each weight, they approximate second-order saliency using a diagonal Hessian-style estimate, implemented through the empirical Fisher over a calibration set. The exact proxy loss depends on the modality: for vision encoders they use the DINO objective, and for text encoders they use a symmetrized in-batch InfoNCE objective. Since pruning operates on whole units rather than individual weights, they aggregate these per-weight scores into one score per MLP neuron and one per attention head. That gives an initial ranking, but the authors point out several problems with relying on it directly: diagonal approximations ignore interactions, the proxy loss may not line up perfectly with downstream performance, and scores from different blocks or unit types are not necessarily comparable. To fix that, they introduce learnable log-scores for all neurons and heads, and they center the initial scores so the two types live on a common scale. That matters because you want to compare a head against a neuron in the same ranking, not maintain separate rankings that are hard to reconcile. Then, for each sparsity target they care about, they identify the cutoff that would be used in hard pruning and replace the abrupt keep-or-remove decision with a sigmoid-shaped soft mask around that cutoff. The model is evaluated under several such soft-pruned budgets at once, and the losses are averaged. Gradient descent then updates all the importance scores jointly. There are two especially nice consequences of this setup. First, the gradients are informative near the pruning boundary, so using multiple sparsity targets exposes different parts of the ranking to useful updates. Second, because the scores are optimized jointly across budgets, the final ranking is not just good for one target; it tends to work smoothly across a whole range of sparsities. The authors emphasize that this lets the ranking be repaired within each MLP block, which was impossible in the earlier block-level search. The experimental evaluation is broad. They test 27 transformer encoders across vision and text, with model sizes ranging from 22 million parameters up to 8 billion. The tasks include image classification, semantic segmentation, text embedding, and retrieval. That breadth is important because the method is supposed to be general, not just a trick for one benchmark or one architecture family. The headline result is that the method gives smooth degradation up to 60% sparsity and generally outperforms one-shot baselines, especially when pruning is deep. In the example highlighted in the paper, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427 million parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, and this is done without any weight correction or retraining. That is a strong result because it suggests the ranking is good enough that you can apply a fairly aggressive structural cut and still preserve almost all of the useful representation quality. They also report actual speed gains, not just parameter savings. For AugReg ViT-B/16 at 60% sparsity, the pruned model reaches up to 1.80 times the dense throughput on an A100 GPU and up to 1.26 times lower single-query latency. So the compression is not merely theoretical; it translates into practical inference benefits. Another important result is that the construction itself is fast. Elastic models can be produced in minutes, and the construction time scales sub-linearly with parameter count. That is a big deal operationally, because a method that works only after hours or days of adaptation would be much less attractive for large model fleets. The paper also draws a careful comparison with baselines. Under shallow pruning, the older methods remain competitive, which is not surprising: when you remove only a little, almost any reasonable ranking can work. The advantage of this method becomes clearer as pruning gets deeper, where the quality of the ranking matters much more. That is exactly where a fine-grained, jointly optimized ordering of neurons and heads should have the most room to help. As for limitations, the paper is still in the structured-pruning regime, so it cannot exploit arbitrary unstructured sparsity. That means the achievable speedups depend on how well the structured cuts map to real hardware efficiency. The method also still relies on a calibration set and on a proxy loss such as DINO or InfoNCE, so if the proxy is poorly matched to the downstream use case, the ranking may not be perfect. And although the approach is retraining-free, it is still an optimization procedure over importance scores, not a purely analytic solution, so it inherits the usual sensitivity to calibration data and optimization choices. A broader caveat is that elastic inference is only useful if your deployment stack can actually switch budgets dynamically and if the resulting pruned subnetworks remain compatible with the downstream pipeline. Still, the paper’s main contribution is very clear: it shows that you can take a pre-trained transformer and make it much more elastic by learning a single, fine-grained ranking of all its neurons and heads, without touching the original weights. That makes the model far more adaptable to real-world compute constraints, and it does so in a way that is practical, fast, and surprisingly effective under deep pruning. What makes this especially interesting is the way it changes the role of pruning from a one-time compression trick into something closer to a deployment control knob. Instead of asking, “What is the one best smaller model?” the paper asks, “Can we make the model itself span a useful range of operating points?” That shift is subtle, but it matters a lot in production settings. A service might need one budget during quiet periods, another during traffic spikes, and yet another for edge or mobile constraints. If the ranking is learned once and can be cut at different points later, then the same checkpoint can support all of those scenarios without maintaining a zoo of separate variants. In practice, that also simplifies storage, model management, and potentially even retrieval pipelines, since you are no longer juggling incompatible embeddings from many differently pruned networks. The method’s emphasis on keeping the pre-trained weights frozen is also a major practical advantage. Fine-tuning large models for each target budget can be expensive, brittle, and data-hungry, especially when the original pre-training distribution is unknown. Here, the calibration data is only used to shape the ranking, not to rewrite the model. That makes the approach much easier to apply when you have a strong foundation model but limited room to adapt it. Another nice aspect is that the authors do not rely on a single sparsity point during optimization. By exposing the ranking to multiple targets at once, they encourage the score order to make sense across a whole continuum of budgets, not just at one chosen cutoff. That is likely part of why the degradation stays smooth rather than collapsing abruptly as sparsity increases. In a broader sense, this paper sits at the intersection of pruning, efficient inference, and model elasticity, and it suggests that fine-grained structure inside transformers still has untapped redundancy that coarse methods miss. For anyone building transformer systems, the takeaway is that elasticity does not necessarily require a special architecture from scratch; sometimes it can be recovered from an existing model by learning a better ranking of its internal units.