Daily AI Paper

Indirect prompt injection is one of the most practical and worrying failure modes for large language models. The basic problem is simple to describe: a model is supposed to treat some text as data, not as instructions, but an attacker can hide malicious commands inside that data. If the model reads a webpage, a document, an email, or a tool output and mistakes embedded instructions for trusted commands, it can be manipulated into ignoring the user, leaking information, or taking unsafe actions. As LLMs become more connected to external content and tools, this attack surface matters more and more. The paper argues that prompt injection is still not solved, especially for open-weight models, and that even strong adaptive attackers can still find ways through many existing defenses. The key idea behind this work is instruction-data separation. That means drawing a hard trust boundary: the model may inspect untrusted content and reason over it, but it should not execute instructions found there. This is a useful framing because in many real applications the boundary is already known externally. A webpage, a search result, or a tool response is often clearly untrusted, so the model does not need to guess which source is safe. The challenge is not identifying trust from scratch; it is making sure the model’s internal machinery respects that boundary while still preserving its ability to use the content for reasoning and summarization. The paper’s proposal is called Heads of Security, or HoS. Instead of retraining the whole model, HoS adds a very small number of learned gates to the attention mechanism. The intuition comes from prior interpretability work showing that attention heads often specialize: some heads copy text, some suppress copying, and some are involved in instruction following. If different heads already play different roles, then perhaps each head can learn its own way of routing attention between trusted and untrusted tokens. In other words, rather than changing what the model knows, HoS changes where each head is allowed to look. Here is the mechanism in plain language. Each token is assigned a trust channel: trusted, untrusted, or template. Trusted tokens include system and user instructions and the model’s own generated text. Untrusted tokens are things like tool outputs or external content. Template tokens are the chat formatting tokens, including the beginning-of-sequence token, which can act like an attention sink. For every attention head in every layer, HoS learns a small table of gate values that says, for a given query channel and key channel, how much attention should be allowed. These gates are applied before the softmax inside attention. So if a head should not read from untrusted content, its gate for that channel is pushed down, and the attention mass is redistributed to trusted tokens instead. Importantly, this does not block all processing of untrusted content. Other heads can still attend to it. The goal is not to erase the data, but to stop injected instructions from being treated as commands. A useful way to think about this is that HoS turns prompt injection defense into an attention-routing problem. It does not require a new model architecture in the sense of adding large modules, and it does not require full fine-tuning of billions of parameters. It only trains the gate parameters. For the 70B Llama model, that is tens of thousands of scalars rather than hundreds of millions or billions of parameters. The GPT-OSS model needs a slight adjustment because it has an explicit attention sink, so the method treats that sink as an extra key-only channel. But the core idea stays the same: learn how each head should redistribute attention across trust levels. Training follows the SecAlign++ recipe. The model is trained with direct preference optimization on pairs where the prompt contains a user instruction and an injected instruction in the untrusted segment. The preferred response follows the user instruction, while the rejected response follows the injection. So the model learns, through preference data, to ignore the malicious instruction while still using the untrusted content for legitimate reasoning. The paper emphasizes that HoS is much smaller than the finetuning-based baseline and also much smaller than full-parameter approaches. The authors compare HoS against two main baselines. SecAlign++ is a strong finetuning-based defense, and ASIDE is an architectural defense that separates instruction and data by rotating embeddings for untrusted content. HoS is architectural like ASIDE, but instead of changing token embeddings, it changes attention routing. The paper evaluates on two open-weight models: Llama-3.3-70B-Instruct and GPT-OSS-20B. That choice is important because it gives a comparison across both a very capable agentic model and a newer, smaller reasoning model that already has some instruction hierarchy training. The experiments cover three kinds of evaluation. First are adaptive attacks, which are the most important security test. The paper uses RL-Hammer in a black-box setting, where an attacker model learns to rewrite an attack goal into an injection optimized for the target. It also uses Claudini, a white-box optimization-based attack, and even strengthens it further through an autoresearch loop. Second are utility benchmarks, especially multiple-choice benchmarks and few-shot prompting, to see whether the defense harms normal model performance. Third are agentic prompt injection benchmarks such as AgentDojo, AgentDyn, LLMail-Inject, and WASP, which test tool use and more realistic agent behavior. The adaptive attack results are striking. Against GPT-OSS-20B, the strongest RL-Hammer attacker reaches a 97% attack success rate on the undefended setup, 85% on SecAlign, but only 10% on HoS. On Llama-3.3-70B, the strongest attacker reaches 99% against the undefended model, 82% against Meta-SecAlign-70B, and only 1% against HoS. That is a very large gap. The white-box Claudini attack is also informative: even after being specifically optimized against HoS, it still needs about three times more compute to reach the same attack success rate on HoS as it does on GPT-OSS-20B or its SecAlign variant. The takeaway is not that HoS is unbreakable, but that it raises the cost of attack substantially, including for adaptive attackers who know the defense exists. On ordinary utility benchmarks, HoS is much closer to the base models than the other defenses. Across standard multiple-choice tasks and prompting setups, HoS nearly matches the original models. The paper highlights that this remains true in few-shot settings, where SecAlign and ASIDE often degrade. For example, on 5-shot MMLU-Pro, HoS scores 67% versus 68% for GPT-OSS-20B, while SecAlign falls to 59% and ASIDE to 52%. On Llama-3.3-70B, HoS scores 55% versus 56% for the base model, while SecAlign drops dramatically to 16%. That is a strong sign that some defenses may look fine in zero-shot evaluation but become much less reliable when the prompting setup gets more realistic. The agentic benchmarks tell a more nuanced story. On AgentDojo, AgentDyn, LLMail-Inject, and WASP, HoS and SecAlign perform similarly in terms of the overall security-utility trade-off, reducing attack success while keeping utility mostly intact. ASIDE, however, struggles badly with tool use, often producing near-zero utility and only 10% utility on AgentDojo. So not all architectural separations are equal: a defense can be secure in principle but still interfere too much with the model’s ability to act as an agent. HoS appears to avoid that pitfall better than ASIDE. The paper also includes mechanistic analysis to understand what the gates are actually doing. Using causal head ablations and role-confusion probes, the authors study how HoS changes attention behavior. The broader point is that the defense is not just a black-box regularizer; it appears to work by steering specific heads away from untrusted instructions while leaving other heads available for legitimate processing. That fits the design intuition behind the method. There are still limitations to keep in mind. HoS depends on a known trust boundary. It is designed for cases where the system can already label content as trusted or untrusted, so it is not a universal solution for ambiguous settings where trust is unclear. It also does not eliminate the need for strong evaluation against adaptive attackers; the paper’s own results make clear that prompt injection defenses can fail badly if they are not stress-tested. And while HoS is lightweight compared with finetuning, it is still not free: it introduces learned routing behavior that must be trained and validated carefully for each model. The main lesson is that prompt injection robustness does not necessarily require retraining a model from scratch. By learning a small set of attention gates, HoS shows that you can enforce instruction-data separation in a targeted way, preserve most of the model’s normal capability, and substantially improve resistance to both black-box and white-box adaptive attacks. In a field where many defenses trade away too much utility or collapse under strong adversaries, that combination is the real contribution.