Daily AI Paper

Large language model systems are no longer just about picking one model and sending every request to it. In practice, many deployments are becoming heterogeneous: a small model might handle simple queries for speed, a larger model might be invoked for harder cases, and routing can change from request to request depending on latency, cost, or resource availability. But there is a catch. Even when the same conversation history, retrieved documents, or task state is reused, the newly selected model usually has to read all of that context again and rebuild its internal memory from scratch. As contexts get longer and are reused across models more often, that repeated computation becomes expensive. This paper addresses that inefficiency. The key idea is simple to state but nontrivial to make work: if one model has already processed the context, can another model in the same family reuse that work instead of re-encoding the raw text? Standard KV caching helps within a single model, because it avoids recomputing attention states for earlier tokens during generation. Prefix caching can also reuse states across requests served by the same model. But these caches are tied to the particular model that produced them. They are generally not directly interchangeable across models, even within the same family, because the internal representations depend on the model’s parameters and architecture. ACCT, short for Asymmetric Cross-LLM Context Transfer, is proposed to solve exactly that. The goal is to transfer context computation from a source model to a target model of a different scale, so the target can answer a query without reprocessing the full context. The word “asymmetric” matters here: the source does the expensive context encoding first, and the target consumes a transformed version of that work under a limited KV budget. In other words, context computation becomes a reusable resource. To understand the landscape, it helps to separate two things that are often bundled together in normal inference. One is context encoding: reading the prompt, documents, or conversation history and building internal attention states. The other is downstream reasoning and generation: using those states to answer the query and produce new tokens. ACCT tries to decouple those two stages. The source model handles context encoding, and the target model handles query processing and generation, but without having to reread the entire context. The method has three lightweight pieces, and each one handles a different part of the transfer problem. First is the Cartographer. Its job is to decide which context tokens matter for each target layer and each KV head. That granularity is important because different heads and layers can care about different parts of the context. A token that is crucial for one unit may be less important for another. The Cartographer uses information from the source context and the query to rank tokens for each target unit. Second is the Allocator. Once the Cartographer has produced importance scores, the Allocator decides how much of the context budget each target layer and head should receive. This matters because the total KV memory is limited, and a uniform split is often not ideal. Some units may need more context than others. The Allocator learns per-unit capacities under a global budget, and it also reserves the final few context tokens so that the text immediately before the query is not accidentally dropped. That tail reservation is a practical detail, but an important one, because recent context often carries strong local relevance. Third is the Compiler. The source model’s KV states cannot simply be copied into the target as-is. So the Compiler converts the selected source representations into target-readable keys and values. It uses the source final-layer states and a few selected intermediate-layer states to make context-dependent corrections. The result is sparse target KV that the target model can load directly. After that, the target does not re-encode the raw context; it only processes the query and then generates the answer. A useful way to think about ACCT is that it builds a kind of translation layer between models. The Cartographer decides what to keep, the Allocator decides how much to keep, and the Compiler decides how to express it in the target model’s internal language. Training is also staged carefully. First, the authors learn the scoring and allocation behavior using target-native KV, meaning KV produced by the target model itself when it encodes context. That gives supervision for which tokens are useful and how capacities should be assigned. Then, once the token sets are fixed, they train the Compiler to convert source representations into the corresponding target KV. Importantly, the source and target base models themselves remain frozen; only the transfer interface is trained. The scoring supervision is based on deletion utility. In plain terms, they ask: if you remove a token’s key and value and keep everything else fixed, how much does the attention output change? A larger change means the token mattered more. That signal is collected at the query position and also on later generated tokens, and then used to teach the Cartographer which tokens should rank higher. This is a sensible choice because it ties token importance to actual downstream effect rather than to a heuristic. The paper evaluates ACCT in two complementary settings: five general-capability benchmarks and six highly context-dependent tasks, across six directed model pairs. The reference point is called Target Full, where the target model processes the complete context on its own. That is the standard to compare against, because it represents what the target can do when it sees everything directly. The headline result is that ACCT closely matches Target Full on many tasks and even surpasses it on some, despite reducing the KV cache size by 75%. That is a strong result, because it suggests the transferred memory is not just a lossy compression that broadly degrades performance. Instead, the transferred KV can preserve the parts of the context that matter for the target’s reasoning. There is also a speed result that is especially important operationally. At a context length of about 19,000 tokens, ACCT’s KV conversion is reported to be 6.5 to 21.2 times faster than letting the target perform full context prefill. That speedup is about the expensive part of inference that ACCT avoids: re-encoding the raw context in the target model. In a heterogeneous serving system, that can translate into much lower latency and better throughput when the same long context must be reused across different models. What do these results actually show? First, they show that context information can be transferred across models in a way that remains useful to the target’s own attention mechanism. The target is not reading text, but it is still able to reason over transferred internal state. Second, they show that the transfer can be done under a fairly aggressive KV budget reduction without collapsing performance. Third, they suggest that in the same model family, context computation is separable enough from downstream reasoning that one model can do the expensive reading while another does the answering. There are, of course, limitations and things to watch next. ACCT is designed for models within the same family and for source-target pairs where the transfer interface can learn a meaningful mapping. The paper’s own framing is not that any arbitrary model can share caches with any other arbitrary model. Also, the method relies on a trained transfer interface, so it is not a zero-cost plug-and-play mechanism. There is training involved, and the quality of transfer will depend on how well the source and target align. Another point is that the budget is finite, so some context must always be discarded or compressed. The method tries to choose what matters most, but long contexts can contain many interacting facts, and no selection method can guarantee perfect preservation. The paper’s results are encouraging precisely because they show strong performance under reduced memory, not because they eliminate all information loss. Finally, ACCT is best viewed as part of a broader shift in LLM systems design. As deployments become more dynamic, we may want to think less about a single monolithic model and more about a pipeline in which context processing, routing, and generation can be distributed across different models. ACCT demonstrates one concrete way to make that practical: by turning context computation into something that can be transferred, rather than always recomputed from scratch.