Daily AI Paper

Large language models are now good at saying no to obviously harmful requests. That part of safety alignment has improved a lot. But there is still a major weakness: if the same harmful intent is spread across a multi-turn conversation, models often fail much more often. Instead of one direct jailbreak prompt, the attacker can slowly build context, ask harmless-looking questions, and only reveal the real goal at the end. By then, the model has often been steered into a response pattern that is no longer protective. That matters because real users do not always present intent in a single, obvious sentence. Multi-turn chat is the natural interface for assistants, so if safety breaks down there, the whole alignment story becomes much less reliable. This paper is about that gap: single-turn safety is relatively strong, but multi-turn safety is brittle. The authors start from an important observation about why current defenses struggle. Many training-based safety methods depend on some external judge: a classifier, a preference model, or another LLM that decides whether a response is safe. Those methods usually supervise the surface form of the answer. They teach the model what a safe-looking reply should look like, but they do not force it to make an explicit safety decision first. That creates two problems. First, the reward signal is only as good as the judge, and judges disagree with each other on a meaningful fraction of examples. Second, the model never learns a clean internal commitment like “refuse” or “comply” that can be checked directly. The second key insight is about the mechanics of alignment itself. Prior work has shown that safety alignment is shallow: the difference between an aligned model and its base model is concentrated in the first few output tokens. In other words, the opening of the answer is where the safety behavior is most heavily determined. That sounds like a weakness, and in one sense it is. But this paper turns that weakness into the central design principle. If the first few tokens steer the whole response, then the highest-leverage place to intervene is right there, before the model has committed to the rest of the answer. That is the core idea behind the method called Assess → Decide → Respond. At every turn in the conversation, the defender first assesses the user’s underlying intent, then explicitly commits to a safety decision, and only then generates the actual reply conditioned on that decision. The decision is not hidden inside the model’s activations or inferred afterward. It is written out as a token span in the model’s own output, so it can be parsed by a simple rule. And because it appears before the reply, it is binding: the reply is generated conditioned on that commitment. The decision space is simple in the self-verified setting: comply or refuse. In the external-judge setting, they add a third option, safe-redirect, which lets the model deflect a harmful request without directly refusing. That distinction matters because some benchmarks and reward setups want to distinguish a clean refusal from a safe redirection, while the self-verified rule-based reward needs a binary target it can check unambiguously. The training setup is reinforcement learning, but with a twist. The reward is attached only to the final turn of the conversation, because that is where the attack typically delivers its payload. The authors explore two reward formulations. The first is an external LLM-as-a-judge reward: the judge sees the full conversation and the final reply, then decides whether the defender was bypassed. A response is safe if it refuses a harmful request or redirects it safely, and for benign conversations it must answer helpfully rather than over-refuse. The second reward is the more unusual one: a self-verified reward. Here, the conversation is built from seed prompts whose nature is known in advance, so the training trajectory already carries a ground-truth label about whether the conversation is harmful or benign. Because the defender commits to a label in its deliberation, the reward can simply compare that label to the known target. No judge is needed. No annotation is needed. It is just a rule reading the committed decision. This is the paper’s version of a verifiable reward for safety. There is one subtle but important safeguard: the model must actually produce a reply after the decision. Without that, a degenerate strategy would be to emit “refuse” and then stop, which would score well on the label check but not be useful. So they include a completeness guard that penalizes a turn if the deliberation consumes the token budget and leaves no reply. The authors also study two gradient scopes. In the full-gradient version, all tokens in the final turn are trained. In the partial-gradient version, only the deliberation tokens up to the decision boundary receive gradient, and the reply tokens are masked out. That is a strong test of the method’s logic: if the decision truly controls the reply, then training only the decision span should still shift the end-to-end behavior. And that is exactly what they are claiming the binding format enables. Training uses GRPO, with two further design choices that help avoid reward hacking. First, each turn has its own length cap, rather than only a total conversation budget. Otherwise, the model could waste tokens on easy early turns and leave the difficult later turn truncated. Second, reward is assigned from the final turn only, because earlier turns are usually innocuous and would otherwise dominate the learning signal. So what do the experiments show? The authors evaluate on two model families and fourteen benchmarks spanning safety, utility, and helpfulness. They compare against baselines trained on the same data, and across six independent judges. The main result is that both versions of ShallowDefend achieve the strongest multi-turn safety in the controlled comparison. They maintain their lead on safety while preserving utility, though there is a modest cost to helpfulness. That tradeoff is important: the method is not claiming to be free. It improves protection, but some helpfulness is sacrificed. The self-verified version is especially notable because it works without any judge in the loop and uses the same recipe across backbones, without per-model tuning. The external-judge version also performs strongly, especially when the graded decision space includes safe-redirect. Across the reported evaluations, the emitted decision label is positively correlated with the external judges’ verdicts, at a level comparable to judge-to-judge agreement. That is a useful sanity check: the explicit decision is not just decorative; it tracks the safety outcome. The analysis goes beyond aggregate scores. A per-token KL study shows that training concentrates its effect on the brief deliberation span, exactly where the decision is written. That matches the paper’s thesis that the earliest tokens matter most. They also run interventional probes, which support a causal interpretation: the committed decision is not merely correlated with the response, it acts as a control point for it. In other words, changing that early commitment changes the trajectory of the answer. What should we take away from this? The paper is arguing that shallow alignment is not only a vulnerability to be fixed by making models “deeper” in some abstract sense. It is also an opportunity. If the first few tokens determine the response, then you can make those tokens carry an explicit safety commitment. That turns an implicit behavior into something trainable, readable, and in one variant even rule-checkable. There are also clear limits to keep in mind. The self-verified reward depends on conversations whose ground-truth nature is known from the seed data, so it is not a general solution to arbitrary open-ended safety labeling. The binary self-verified setup also cannot naturally represent safe-redirect, which is why that option is reserved for the external-judge version. And although the method improves multi-turn safety, it does so with a modest hit to helpfulness, so deployment would still require balancing refusal behavior against user experience. Still, the main contribution is quite elegant. Instead of trying to hide the fact that safety is shallow, the paper uses that shallowness as the mechanism of defense. It places an explicit decision exactly where the model is most sensitive, trains that decision directly, and shows that doing so can improve multi-turn robustness without relying entirely on opaque external judges.