Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness
Abstract
We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction-data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires 3 times more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security-utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
Methodology
The paper proposes "Heads of Security" (HoS), a defense mechanism against prompt injection that operates by learning per-head attention gates. Unlike previous methods that rely on broad fine-tuning (like SecAlign) or complex architectural changes (like ASIDE), HoS targets the attention mechanism directly to enforce instruction-data separation. The core idea is to learn how specific attention heads should redistribute focus between trusted (system/user) and untrusted (tool/web) tokens. This allows the model to process untrusted content for reasoning purposes without executing any instructions embedded within it. The approach is lightweight and targeted, which is a significant methodological shift from behavior-level alignment to attention-level intervention.
Results & evaluation
The evaluation is rigorous and comprehensive, testing on two major open-weight models: Llama-3.3-70B-Instruct and GPT-OSS-20B. The paper compares HoS against strong baselines: SecAlign++ (SOTA fine-tuning defense) and ASIDE (architectural defense). Key results include a dramatic reduction in Attack Success Rate (ASR) against adaptive attacks (RL-Hammer drops from 97% to 10% on GPT-OSS-20B) and improved robustness against white-box attacks (Claudini). Crucially, the paper demonstrates that HoS preserves base-model utility on standard benchmarks, whereas competitors degrade in few-shot settings. The inclusion of agentic benchmarks with static attacks further validates the method's practical applicability.
Reproducibility
The authors provide a GitHub repository and a Hugging Face space, indicating a commitment to open science. The code availability is a strong plus. However, the specific details of the "autoresearch" used to strengthen the white-box attack against HoS are not fully detailed in the abstract, though likely present in the full text. The use of standard models and benchmarks enhances reproducibility.
Limitations
The primary limitation is the reliance on open-weight models for evaluation; performance on closed-source models like GPT-4 or Claude is not directly testable in this manner. Additionally, the method requires access to the model's internal attention heads, which limits its applicability to API-only models. The paper also notes that ASIDE struggles with tool use, but it is unclear if HoS fully resolves all edge cases in complex agentic workflows beyond the tested benchmarks.
Broader impact
This work has high impact on the field of LLM security. As LLMs are increasingly deployed in agentic roles with access to external data, prompt injection is a critical vulnerability. HoS offers a scalable, efficient, and effective defense that does not sacrifice model utility. It provides a new paradigm for securing LLMs by manipulating internal representations (attention) rather than just output behavior. This could lead to more secure and trustworthy AI systems in production environments. The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
Author of this paper? Send a correction.