Last 7 Days (June 22 – June 28, 2026)
AI-assisted vulnerability discovery has proven effective for bug classes like memory safety, where instrumentation confirms memory violations and efficiently filters false positives. Many dangerous vulnerability classes, such as cryptographic misuse, however, lack any comparable instrumentation. In this work, we present Chai, an AI-based system that discovers and validates cryptographic misuse vulnerabilities through naturally occurring signals. To achieve this, Chai rethinks the classical technique of differential testing by leveraging AI to 1) improve precision for detecting real security issues in libraries, and 2) repurpose commonly overlooked discrepancies as leads for tangible vulnerabilities in downstream applications. In doing so, Chai inverts the prevailing paradigm of AI vulnerability discovery: instead of auditing one codebase for many flaws, it catalogs flaws at the library level and propagates them across a cryptographic dependency graph, delivering compounding efficiency gains. We evaluate Chai across X.509, JWT, and SAML libraries. Chai discovered a previously unknown critical vulnerability in an SSL library that powers billions of devices, along with security bugs in one library behind a major web browser and another in major Linux distributions. In total, these techniques surfaced over 100 vulnerabilities.
Primary: UC Berkeley
All Institutions: UC Berkeley
Chai has a profound broader impact on several fronts: * **Revolutionizing Vulnerability Discovery:** It offers a paradigm shift for discovering cryptographic misuse vulnerabilities, a critical and hard-to-find class of bugs. By providing a verifiable signal where traditional instrumentation fails, it makes AI-assisted discovery practical for these domains. * **Enhanced Software Supply Chain Security:** Cryptographic libraries are fundamental components in countless applications. Discovering and tracing vulnerabilities in these libraries and their downstream usage significantly strengthens the security of the entire software supply chain, impacting billions of devices and users. * **Compounding Efficiency:** The "amplified testing" and "discrepancy tracing" mechanisms offer a highly efficient model for security auditing, potentially reducing the cost and time required to find high-impact bugs. This efficiency gain could enable more frequent and comprehensive security assessments. * **Ethical AI in Security:** The emphasis on a human in the loop for final verification and responsible disclosure is crucial. This ensures that AI-generated findings are properly vetted before being reported to maintainers, fostering trust and collaboration rather than burdening open-source projects with false positives. * **Generalizability:** The core principles of combining differential testing with adaptive AI search and discrepancy tracing could be extended to other complex, "hard-to-instrument" vulnerability classes beyond cryptography, opening new avenues for AI-assisted security research. Chai introduces a novel AI-based system that rethinks differential testing to effectively discover and validate cryptographic misuse vulnerabilities, demonstrating significant real-world impact by finding critical flaws in widely used libraries and outperforming existing tools. The paper's strength lies in its innovative paradigm shift from auditing individual codebases to cataloging library-level flaws and propagating them across dependency graphs, coupled with a rigorous experimental evaluation that confirms its superior efficiency and ability to uncover unique, high-severity vulnerabilities previously missed by other advanced AI systems.
Chai introduces a highly innovative, agentic approach to discovering cryptographic misuse vulnerabilities, a notoriously difficult class of bugs due to the lack of clear instrumentation or oracles. The core methodology rethinks differential testing by integrating AI in two key ways: improving precision for library-level issues and repurposing overlooked discrepancies as leads for downstream application vulnerabilities. This "inversion of the prevailing paradigm" is a significant methodological shift. Instead of auditing one codebase for many flaws, Chai catalogs flaws at the library level and propagates them across a cryptographic dependency graph. The system design is robust, comprising two main stages: 1. **Amplified Testing (Differential Testing):** An AI agent generates test inputs, which are then run through multiple implementations of a cryptographic protocol simultaneously. The agent's input generation is adaptive, conditioned on prior findings and leveraging a retrieval agent that queries an index of past CVEs. This moves beyond fixed-grammar fuzzers by allowing the agent to reason about behavior to probe rather than just encoding. Resource allocation for parallel searches is managed by a UCB1 multi-armed bandit algorithm, ensuring broad exploration while prioritizing productive mutation groups. Output analysis involves reproduction, minimization, and classification of differentials, aiming to produce a verifiable signal. 2. **Discrepancy Tracing:** This is arguably the most novel part. Ambiguities identified at the library level (where the specification grants latitude, leading to differing but individually defensible implementations) are treated as leads for downstream vulnerabilities. Chai constructs a dependency graph from package manifests (OpenSSF Criticality Score repositories) and traces these ambiguities to dependent applications. A coding agent then performs a targeted audit on these applications, attempting to build a proof-of-concept (PoC) for the *specific* identified ambiguity. This narrows the agent's task, making it more reliable than open-ended searches. The PoC pipeline aims for end-to-end exploits, with a human in the loop for final verification and report generation. The methodology effectively combines the strengths of differential testing (verifiable discrepancies) with AI's ability for adaptive search and reasoning, addressing the "no oracle" problem for cryptographic misuse. The "compounding efficiency" from amplified testing, reverse search, and targeted auditing is a powerful concept.
The experimental evaluation is comprehensive and compelling. Chai was evaluated across three critical cryptographic protocol domains: X.509 (13 libraries), JWT (23 libraries), and SAML (11 libraries), spanning 47 libraries across 8 languages. Key findings: * **Vulnerability Discovery:** Chai surfaced over 100 vulnerabilities and security bugs. This includes a critical chain-validation bypass in wolfSSL (an SSL library powering billions of devices), a certificate-constraint fail-open in a major browser's TLS library, and a certificate-chain validation flaw in a TLS library shipped in major Linux distributions. These are high-impact, real-world vulnerabilities. * **Superior Performance:** Chai consistently outperformed strong baselines (MLCerts, jwt-fuzzer, jwt_tool, AFL++) in terms of unique differentials found and efficiency. For X.509, Chai found 147 unique discrepancy vectors from 1,500 certificates at $52.5, while MLCerts found 73 from 500,000 certificates at $560. This represents twice as many differentials at a tenth the cost and a thousandth the inputs. Similar leads were observed for JWT and SAML. * **Unique Findings:** Venn diagrams clearly show that the vast majority of Chai's findings (e.g., 132 of 147 on X.509) were unique to Chai and not surfaced by any baseline, indicating it explores different and often more fruitful parts of the input space. * **Comparison to Other AI Systems:** The paper highlights that the wolfSSL codebase had recently been audited by Anthropic's Mythos, yet Chai discovered two severe vulnerabilities that Mythos missed. This strongly suggests Chai's approach reaches vulnerabilities that prevailing AI-driven methods overlook. * **Cost Analysis:** The evaluation includes cost metrics (inference/GPU costs) for LLM-driven methods, which is crucial for practical assessment. The evaluation demonstrates clear, reproducible improvements on important tasks and provides strong evidence for the effectiveness and novelty of Chai's approach. The real-world impact of the discovered vulnerabilities is a testament to its significance.
The paper provides a good level of detail regarding implementation, which aids reproducibility: * **Protocol Domains:** Chai spans X.509, JWT, and SAML, with approximate lines of code for each system. * **Technologies:** Python is the primary language. Specific libraries are mentioned (cryptography, asn1crypto, pyOpenSSL for X.509; cryptography, ecdsa for JWT; signxml, lxml for SAML). * **LLM Integration:** Agent requests route through LiteLLM to track spend and route to multiple providers. Specific models used are listed: GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8, and the open-source Kimi K2.6. Embedding model (OpenAI's text-embedding-3-small) and similarity metric (cosine similarity) are specified. * **Harnesses:** Described as small native-language scripts invoked as subprocesses, exchanging JSON verdicts. Languages covered include C, Go, Ruby, PHP, Node.js. * **PoC Pipeline:** Uses GCP SDK for parallel VM launches, Jinja templates for reports, and Postgres/React for the web visualizer. * **Baselines:** Specific baselines like MLCerts, jwt-fuzzer, jwt_tool, AFL++ are named. While the exact prompts for the LLM agents are not provided (which is common in LLM-based security research), the detailed description of the system architecture, specific tools, and models used, along with the inclusion of an open-source model (Kimi K2.6), suggests a strong commitment to reproducibility. The deterministic nature of the builder and the minimization process also contribute to reproducible findings.
* **Empirical Coverage:** Chai's agents generate inputs probabilistically, meaning the uncovered discrepancies represent a subset of those present. The coverage is empirical rather than exhaustive. * **Human in the Loop:** While a strength for ethical disclosure, the final manual review and preparation for each disclosure (2-5 hours per finding) is labor-intensive, limiting the ultimate scale of fully verified and disclosed findings. * **Scope of Vulnerabilities:** Chai does not directly detect flaws in the underlying mathematical constructions or in the specifications themselves. It focuses on implementation discrepancies. * **Reliance on Disagreement:** An issue surfaces only when at least two independent implementations disagree on the same input, potentially missing bugs in universally flawed implementations. * **LLM Dependence:** The effectiveness of the agentic reasoning is dependent on the capabilities and cost of the underlying LLMs, which can be prone to hallucinations or high costs, even if amortized. * **Protocol Specificity:** While the *approach* is generalizable, adapting Chai to new cryptographic protocols still requires harnessing libraries and supplying seed messages, implying some engineering effort per protocol.
Chai has a profound broader impact on several fronts: * **Revolutionizing Vulnerability Discovery:** It offers a paradigm shift for discovering cryptographic misuse vulnerabilities, a critical and hard-to-find class of bugs. By providing a verifiable signal where traditional instrumentation fails, it makes AI-assisted discovery practical for these domains. * **Enhanced Software Supply Chain Security:** Cryptographic libraries are fundamental components in countless applications. Discovering and tracing vulnerabilities in these libraries and their downstream usage significantly strengthens the security of the entire software supply chain, impacting billions of devices and users. * **Compounding Efficiency:** The "amplified testing" and "discrepancy tracing" mechanisms offer a highly efficient model for security auditing, potentially reducing the cost and time required to find high-impact bugs. This efficiency gain could enable more frequent and comprehensive security assessments. * **Ethical AI in Security:** The emphasis on a human in the loop for final verification and responsible disclosure is crucial. This ensures that AI-generated findings are properly vetted before being reported to maintainers, fostering trust and collaboration rather than burdening open-source projects with false positives. * **Generalizability:** The core principles of combining differential testing with adaptive AI search and discrepancy tracing could be extended to other complex, "hard-to-instrument" vulnerability classes beyond cryptography, opening new avenues for AI-assisted security research. Chai introduces a novel AI-based system that rethinks differential testing to effectively discover and validate cryptographic misuse vulnerabilities, demonstrating significant real-world impact by finding critical flaws in widely used libraries and outperforming existing tools. The paper's strength lies in its innovative paradigm shift from auditing individual codebases to cataloging library-level flaws and propagating them across dependency graphs, coupled with a rigorous experimental evaluation that confirms its superior efficiency and ability to uncover unique, high-severity vulnerabilities previously missed by other advanced AI systems.
Gradient equilibrium (GEQ) is a recently introduced online optimization framework that generalizes first-order stationarity from offline optimization and abstracts problems like online conformal prediction. While GEQ has curious similarities with known online learning frameworks, namely regret minimization, prior work has shown that GEQ error and regret are incomparable objectives, leaving open a precise understanding of how GEQ fits into the broader online learning landscape. In this work, we show that GEQ is equivalent to Blackwell approachability in the algorithmic sense. That is, a Blackwell approachability problem can always be solved using queries to a black-box GEQ oracle, with no asymptotic loss in the oracle's error rate, and vice versa. Taken together with known equivalences between approachability, regret minimization, and calibration, these results imply that GEQ is equivalent to these frameworks, as well. Our reductions are efficient and can be used to transfer refined guarantees, such as optimism and strong adaptivity, from regret minimization to GEQ. Along the way, we also identify necessary and sufficient conditions for GEQ, and establish reductions between different notions of GEQ with unconstrained and constrained decision sets.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley, Inria & École Normale Supérieure
This work has a significant broader impact on the field of online learning: 1. **Theoretical Unification**: It provides a crucial piece in the puzzle of understanding the relationships between different online learning frameworks. By rigorously establishing the equivalence between GEQ and Blackwell Approachability (and consequently, regret minimization and calibration), it offers a more unified and coherent theoretical landscape. 2. **Algorithmic Transferability**: The black-box oracle reductions are a powerful tool for algorithm design. They enable the direct transfer of algorithms and refined guarantees (such as optimism and strong adaptivity) from well-studied frameworks like regret minimization to the newer GEQ framework, and vice versa. This can accelerate the development of more sophisticated algorithms for GEQ-type problems and simplify the design of certain regret minimization algorithms. 3. **Deeper Understanding of GEQ**: Identifying Blackwell's condition as a necessary and sufficient condition for GEQ provides a fundamental theoretical characterization, moving beyond previously known sufficient conditions like restorativity. This deeper understanding can guide future research into the fundamental properties and solvability conditions of GEQ problems. 4. **Applications in Statistical Problems**: GEQ abstracts problems like online conformal prediction and online quantile debiasing. By connecting GEQ to other frameworks, this work opens avenues for applying established techniques and guarantees from regret minimization or calibration to these statistical problems, potentially leading to improved algorithms or stronger theoretical guarantees. 5. **Foundation for Future Research**: The paper lays a strong theoretical foundation, inviting further research into exploring other minimal conditions for GEQ, investigating the practical computational efficiency of these oracle compositions, and applying the transferred guarantees to a wider range of online learning applications. The paper rigorously establishes an algorithmic equivalence between Gradient Equilibrium (GEQ) and Blackwell Approachability (BA), thereby unifying GEQ with regret minimization and calibration and enabling the transfer of advanced algorithmic guarantees across these online learning frameworks. This work makes a fundamental theoretical contribution to online learning by precisely positioning the recently introduced GEQ framework within the broader landscape of online optimization. Through elegant black-box oracle reductions, the authors demonstrate that GEQ problems can be solved using BA algorithms and vice versa, with no asymptotic loss in error rates. This equivalence is then extended to regret minimization and calibration, providing a powerful conceptual unification and practical tool for algorithm design, as it allows for the transfer of sophisticated guarantees like optimism and strong adaptivity across these paradigms. The methodology is sound and rigorous, involving careful definitions, proofs of conditions (e.g., restorativity implying Blackwell's condition), and a clever reduction from constrained to unconstrained GEQ, all contributing to a significant advancement in the theoretical understanding of online learning.
The paper's methodology is centered on establishing algorithmic equivalences between Gradient Equilibrium (GEQ) and Blackwell Approachability (BA) through rigorous black-box oracle reductions. This involves two main directions: 1. **Reducing GEQ to BA**: The authors interpret a GEQ problem as a specific BA problem where the vector payoffs are the negative subgradients and the target set is the origin. A key technical contribution here is demonstrating that the restorativity condition (a known sufficient condition for GEQ) implies Blackwell's condition (the necessary and sufficient condition for BA). To make the reduction constructive, they propose an "approximate halfspace oracle" that uses a growing function `phi(t)` to select decisions. This oracle may make a bounded number of errors, which is then handled by leveraging the robustness properties of standard BA algorithms (like Blackwell's algorithm). The analysis shows that the error rate of the GEQ algorithm derived from a BA oracle is asymptotically equivalent to the BA oracle's rate. 2. **Reducing BA to GEQ**: This direction is more complex and involves two sub-steps: * **BA to Constrained GEQ**: Assuming the BA target set `S` is a cone (which can be generalized via conic lifting), the authors construct a GEQ problem where the decision set is the polar cone `S^` and the vector field `g_t(u)` is defined as `-f(O_H(u), b_t)`, where `O_H` is a halfspace oracle for the BA problem. They show that this constructed GEQ problem satisfies the necessary assumptions (boundedness, restorativity) and that solving it with a GEQ oracle leads to a solution for the BA problem. The proof ingeniously uses the normal vectors guaranteed by the GEQ oracle as "primal witnesses" for the approachability of the target set. * **Constrained GEQ to Unconstrained GEQ**: This crucial technical lemma completes the loop. It shows how to solve any GEQ problem with a constrained decision set `X` using an oracle for unconstrained GEQ (`X = R^d`). This is achieved by modifying the original vector field `g(x)` into `g'(x) = g(Proj_X(x)) + n_g(x)`, where `n_g(x)` is a scaled projection residual. This modification ensures that `g'(x)` is restorative and that the projection residual term `n_g(x)` effectively acts as a normal vector to `X` at `Proj_X(x)`, thus linking the unconstrained GEQ solution back to the constrained GEQ definition. The methodology is highly rigorous, relying on precise definitions of oracles and conditions. The black-box nature of the reductions makes them broadly applicable, allowing for the transfer of algorithmic guarantees across frameworks. The paper also provides a detailed technical overview, explaining the intuition behind the reductions, particularly the "primal" interpretation of their BA-to-GEQ reduction in contrast to the "dual" interpretation of prior work connecting BA to regret minimization.
The paper is purely theoretical and does not include any experimental evaluation. This is entirely appropriate for its venue (COLT) and the nature of its contribution, which is to establish fundamental theoretical equivalences and algorithmic implications rather than demonstrate empirical performance on specific tasks. The focus is on mathematical proofs, oracle reductions, and asymptotic error rate guarantees.
As a theoretical paper, reproducibility pertains to the clarity and correctness of its definitions, theorems, lemmas, and proofs. The paper provides comprehensive definitions for Blackwell Approachability and Gradient Equilibrium, clearly states assumptions, and presents algorithms in pseudocode. All new claims are supported by detailed mathematical proofs. A reader with a solid background in online learning theory and convex analysis should be able to follow and verify the logical steps and derivations. There are no code implementations or experimental setups to reproduce.
1. **Purely Theoretical**: The primary limitation is the absence of empirical validation. While justified for a COLT paper, it means the practical implications of transferring guarantees (e.g., the actual performance benefits of "optimistic" GEQ algorithms) are not explored. 2. **Efficiency of Reductions**: While the reductions are "efficient" in an asymptotic sense, the constant factors and computational overhead introduced by composing multiple black-box oracles (e.g., repeated halfspace oracle queries, projections, or the specific choice of `phi(t)`) are not deeply analyzed in terms of practical runtime. 3. **Assumptions**: The reductions rely on specific assumptions, such as the boundedness of payoffs/gradients and the restorativity condition for GEQ, or Blackwell's condition for BA. While these are standard in their respective contexts, they might not hold for all conceivable online learning problems. 4. **Conic Lifting Detail**: The reduction from general BA to GEQ relies on a "conic lifting argument" from prior work. While this is a standard technique, the full details of this lifting are not provided in the main text, requiring familiarity with external literature. 5. **Specifics of `phi(t)`**: The choice of the growing function `phi(t)` in the approximate halfspace oracle impacts the constant factor in the error bound. The paper provides examples but doesn't delve into the optimal choice or its practical implications for different problem settings.
This work has a significant broader impact on the field of online learning: 1. **Theoretical Unification**: It provides a crucial piece in the puzzle of understanding the relationships between different online learning frameworks. By rigorously establishing the equivalence between GEQ and Blackwell Approachability (and consequently, regret minimization and calibration), it offers a more unified and coherent theoretical landscape. 2. **Algorithmic Transferability**: The black-box oracle reductions are a powerful tool for algorithm design. They enable the direct transfer of algorithms and refined guarantees (such as optimism and strong adaptivity) from well-studied frameworks like regret minimization to the newer GEQ framework, and vice versa. This can accelerate the development of more sophisticated algorithms for GEQ-type problems and simplify the design of certain regret minimization algorithms. 3. **Deeper Understanding of GEQ**: Identifying Blackwell's condition as a necessary and sufficient condition for GEQ provides a fundamental theoretical characterization, moving beyond previously known sufficient conditions like restorativity. This deeper understanding can guide future research into the fundamental properties and solvability conditions of GEQ problems. 4. **Applications in Statistical Problems**: GEQ abstracts problems like online conformal prediction and online quantile debiasing. By connecting GEQ to other frameworks, this work opens avenues for applying established techniques and guarantees from regret minimization or calibration to these statistical problems, potentially leading to improved algorithms or stronger theoretical guarantees. 5. **Foundation for Future Research**: The paper lays a strong theoretical foundation, inviting further research into exploring other minimal conditions for GEQ, investigating the practical computational efficiency of these oracle compositions, and applying the transferred guarantees to a wider range of online learning applications. The paper rigorously establishes an algorithmic equivalence between Gradient Equilibrium (GEQ) and Blackwell Approachability (BA), thereby unifying GEQ with regret minimization and calibration and enabling the transfer of advanced algorithmic guarantees across these online learning frameworks. This work makes a fundamental theoretical contribution to online learning by precisely positioning the recently introduced GEQ framework within the broader landscape of online optimization. Through elegant black-box oracle reductions, the authors demonstrate that GEQ problems can be solved using BA algorithms and vice versa, with no asymptotic loss in error rates. This equivalence is then extended to regret minimization and calibration, providing a powerful conceptual unification and practical tool for algorithm design, as it allows for the transfer of sophisticated guarantees like optimism and strong adaptivity across these paradigms. The methodology is sound and rigorous, involving careful definitions, proofs of conditions (e.g., restorativity implying Blackwell's condition), and a clever reduction from constrained to unconstrained GEQ, all contributing to a significant advancement in the theoretical understanding of online learning.
Large language models can serve as capable long-horizon agents, but their out-of-distribution (OOD) generalization remains weak. We identify a key source of this failure as task insensitivity: when faced with similar but distinct tasks, models might apply patterns learned during training and fail to solve the task at hand. We show that models often continue with actions aligned with the original task even when the instruction is semantically corrupted and cannot be directly answered. We further find that, when we replace the task description in a trained prompt with another similar but distinct task, the model may still output the same action. This behavior is accompanied by a consistent training-time attention drift away from task tokens and toward local observations, suggesting an optimization bias toward shortcuts. To mitigate this problem, we propose Task-Perturbed NLL Optimization, a lightweight contrastive regularizer that explicitly encourages action dependence on the task instruction. Extensive evaluations show that our intervention improves task sensitivity and OOD generalization while preserving more stable attention to task tokens.
Primary: Gaoling School of Artificial Intelligence Renmin University of China
All Institutions: Gaoling School of Artificial Intelligence Renmin University of China, Tmall Group of Alibaba, Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Engineering Research Center of Next-Generation Intelligent Search and Recommendation
This paper makes a significant contribution to understanding and mitigating a critical failure mode in large language model agents: out-of-distribution generalization. By identifying "task insensitivity" and providing clear diagnostics, it offers a new lens through which to analyze agent behavior. The proposed Task-Perturbed NLL Optimization is a lightweight and effective regularization technique that can be readily integrated into existing SFT and RL pipelines. This has direct implications for developing more robust and reliable LLM agents, particularly in long-horizon tasks where maintaining task grounding is paramount. The insights into attention drift also contribute to the broader understanding of how LLMs learn and potentially overfit, informing future research on interpretability and robust training methodologies for large models beyond agents. This paper identifies task insensitivity as a key source of out-of-distribution generalization failure in LLM agents, provides strong empirical diagnostics and mechanistic insights into its development, and proposes an effective, lightweight contrastive regularizer to mitigate it. The comprehensive analysis, rigorous experimentation across multiple benchmarks, and clear improvements in agent robustness make this a highly impactful contribution to the field of LLM agents and robust AI.
The paper introduces a compelling diagnostic framework to identify "task insensitivity" in language agents. This framework includes: 1) **Corrupted Task Descriptions**: Modifying task instructions to be ambiguous or underspecified while allowing clarification, observing if models reconstruct the original task. This is a clever way to probe reliance on learned patterns. 2) **Controlled OOD Task Shifts**: Replacing original task descriptions with similar but distinct OOD tasks, specifically filtering for cases requiring different actions. This directly tests generalization beyond training distributions. 3) **Attention Analysis**: Tracking attention allocation to task tokens versus local observations during training to understand the underlying mechanism. This provides mechanistic insight into the observed behavioral failures. The proposed solution, **Task-Perturbed NLL Optimization**, is a lightweight contrastive regularizer. It explicitly encourages action dependence on the task instruction by making the original action less likely when the task is perturbed. A key methodological strength is the use of a frozen reference model to calibrate the desired separation, preventing over-penalization and ensuring a more stable optimization target. The extension to RL by perturbing tasks and reusing advantage scores is also a practical consideration. The theoretical intuition provided through simplified gradient and local attention reallocation analyses, while not a formal proof, offers a plausible explanation for the observed attention drift and strengthens the motivation for the proposed intervention.
The experimental evaluation is extensive and well-designed, covering three diverse embodied-agent benchmarks: ALFWorld, ScienceWorld, and WebShop. The use of controlled OOD splits (e.g., "heat and place" vs. "cool and place" in ALFWorld) is crucial for isolating the problem of task insensitivity. The paper evaluates both supervised fine-tuning (SFT) and reinforcement learning (GRPO) settings, demonstrating the broad applicability of the findings and the proposed method. Baselines include vanilla SFT, CoIN, and task augmentation, providing a good comparative context. The results consistently show that Task-Perturbed NLL Optimization improves OOD generalization across environments and training paradigms. Crucially, the method also preserves more stable attention to task tokens and reduces the growth of task-insensitive behavior (measured by conditional consistency), directly supporting the paper's hypothesis and the mechanism of the intervention. The use of GPT-5.4 as a judge for semantic reconstruction adds a layer of qualitative validation. The detailed breakdown of attention across different prompt regions in the appendix further strengthens the empirical analysis.
The paper provides a good level of detail for reproducibility. The appendix includes specific information on dataset splits for ALFWorld, ScienceWorld, and WebShop, training details (epochs, learning rates, batch sizes, GPU setup, GRPO parameters), and the full prompt templates used for agents and the judge model. Crucially, the implementation details for Task-Perturbed NLL Optimization, including response masking and the reference-ratio computation, are clearly described. This level of detail should allow researchers to replicate the experiments and validate the findings.
The authors acknowledge that the empirical study covers only three agent settings (ALFWorld, ScienceWorld, WebShop), which, while diverse, represent a narrow slice of agentic behavior. This suggests that the generalizability of "task insensitivity" and the proposed solution to broader, more complex agent environments (e.g., open-ended web agents, real-world robotics) needs further investigation. Another limitation noted is that the attention analysis is mechanistically suggestive rather than causally definitive. While the observed attention drift aligns with the optimization account, it does not fully establish the causal mechanisms of internal decision-making within the transformer.
This paper makes a significant contribution to understanding and mitigating a critical failure mode in large language model agents: out-of-distribution generalization. By identifying "task insensitivity" and providing clear diagnostics, it offers a new lens through which to analyze agent behavior. The proposed Task-Perturbed NLL Optimization is a lightweight and effective regularization technique that can be readily integrated into existing SFT and RL pipelines. This has direct implications for developing more robust and reliable LLM agents, particularly in long-horizon tasks where maintaining task grounding is paramount. The insights into attention drift also contribute to the broader understanding of how LLMs learn and potentially overfit, informing future research on interpretability and robust training methodologies for large models beyond agents. This paper identifies task insensitivity as a key source of out-of-distribution generalization failure in LLM agents, provides strong empirical diagnostics and mechanistic insights into its development, and proposes an effective, lightweight contrastive regularizer to mitigate it. The comprehensive analysis, rigorous experimentation across multiple benchmarks, and clear improvements in agent robustness make this a highly impactful contribution to the field of LLM agents and robust AI.
AI-assisted vulnerability discovery has proven effective for bug classes like memory safety, where instrumentation confirms memory violations and efficiently filters false positives. Many dangerous vulnerability classes, such as cryptographic misuse, however, lack any comparable instrumentation. In this work, we present Chai, an AI-based system that discovers and validates cryptographic misuse vulnerabilities through naturally occurring signals. To achieve this, Chai rethinks the classical technique of differential testing by leveraging AI to 1) improve precision for detecting real security issues in libraries, and 2) repurpose commonly overlooked discrepancies as leads for tangible vulnerabilities in downstream applications. In doing so, Chai inverts the prevailing paradigm of AI vulnerability discovery: instead of auditing one codebase for many flaws, it catalogs flaws at the library level and propagates them across a cryptographic dependency graph, delivering compounding efficiency gains. We evaluate Chai across X.509, JWT, and SAML libraries. Chai discovered a previously unknown critical vulnerability in an SSL library that powers billions of devices, along with security bugs in one library behind a major web browser and another in major Linux distributions. In total, these techniques surfaced over 100 vulnerabilities.
Primary: UC Berkeley
All Institutions: UC Berkeley
Chai has a profound broader impact on several fronts: * **Revolutionizing Vulnerability Discovery:** It offers a paradigm shift for discovering cryptographic misuse vulnerabilities, a critical and hard-to-find class of bugs. By providing a verifiable signal where traditional instrumentation fails, it makes AI-assisted discovery practical for these domains. * **Enhanced Software Supply Chain Security:** Cryptographic libraries are fundamental components in countless applications. Discovering and tracing vulnerabilities in these libraries and their downstream usage significantly strengthens the security of the entire software supply chain, impacting billions of devices and users. * **Compounding Efficiency:** The "amplified testing" and "discrepancy tracing" mechanisms offer a highly efficient model for security auditing, potentially reducing the cost and time required to find high-impact bugs. This efficiency gain could enable more frequent and comprehensive security assessments. * **Ethical AI in Security:** The emphasis on a human in the loop for final verification and responsible disclosure is crucial. This ensures that AI-generated findings are properly vetted before being reported to maintainers, fostering trust and collaboration rather than burdening open-source projects with false positives. * **Generalizability:** The core principles of combining differential testing with adaptive AI search and discrepancy tracing could be extended to other complex, "hard-to-instrument" vulnerability classes beyond cryptography, opening new avenues for AI-assisted security research. Chai introduces a novel AI-based system that rethinks differential testing to effectively discover and validate cryptographic misuse vulnerabilities, demonstrating significant real-world impact by finding critical flaws in widely used libraries and outperforming existing tools. The paper's strength lies in its innovative paradigm shift from auditing individual codebases to cataloging library-level flaws and propagating them across dependency graphs, coupled with a rigorous experimental evaluation that confirms its superior efficiency and ability to uncover unique, high-severity vulnerabilities previously missed by other advanced AI systems.
Chai introduces a highly innovative, agentic approach to discovering cryptographic misuse vulnerabilities, a notoriously difficult class of bugs due to the lack of clear instrumentation or oracles. The core methodology rethinks differential testing by integrating AI in two key ways: improving precision for library-level issues and repurposing overlooked discrepancies as leads for downstream application vulnerabilities. This "inversion of the prevailing paradigm" is a significant methodological shift. Instead of auditing one codebase for many flaws, Chai catalogs flaws at the library level and propagates them across a cryptographic dependency graph. The system design is robust, comprising two main stages: 1. **Amplified Testing (Differential Testing):** An AI agent generates test inputs, which are then run through multiple implementations of a cryptographic protocol simultaneously. The agent's input generation is adaptive, conditioned on prior findings and leveraging a retrieval agent that queries an index of past CVEs. This moves beyond fixed-grammar fuzzers by allowing the agent to reason about behavior to probe rather than just encoding. Resource allocation for parallel searches is managed by a UCB1 multi-armed bandit algorithm, ensuring broad exploration while prioritizing productive mutation groups. Output analysis involves reproduction, minimization, and classification of differentials, aiming to produce a verifiable signal. 2. **Discrepancy Tracing:** This is arguably the most novel part. Ambiguities identified at the library level (where the specification grants latitude, leading to differing but individually defensible implementations) are treated as leads for downstream vulnerabilities. Chai constructs a dependency graph from package manifests (OpenSSF Criticality Score repositories) and traces these ambiguities to dependent applications. A coding agent then performs a targeted audit on these applications, attempting to build a proof-of-concept (PoC) for the *specific* identified ambiguity. This narrows the agent's task, making it more reliable than open-ended searches. The PoC pipeline aims for end-to-end exploits, with a human in the loop for final verification and report generation. The methodology effectively combines the strengths of differential testing (verifiable discrepancies) with AI's ability for adaptive search and reasoning, addressing the "no oracle" problem for cryptographic misuse. The "compounding efficiency" from amplified testing, reverse search, and targeted auditing is a powerful concept.
The experimental evaluation is comprehensive and compelling. Chai was evaluated across three critical cryptographic protocol domains: X.509 (13 libraries), JWT (23 libraries), and SAML (11 libraries), spanning 47 libraries across 8 languages. Key findings: * **Vulnerability Discovery:** Chai surfaced over 100 vulnerabilities and security bugs. This includes a critical chain-validation bypass in wolfSSL (an SSL library powering billions of devices), a certificate-constraint fail-open in a major browser's TLS library, and a certificate-chain validation flaw in a TLS library shipped in major Linux distributions. These are high-impact, real-world vulnerabilities. * **Superior Performance:** Chai consistently outperformed strong baselines (MLCerts, jwt-fuzzer, jwt_tool, AFL++) in terms of unique differentials found and efficiency. For X.509, Chai found 147 unique discrepancy vectors from 1,500 certificates at $52.5, while MLCerts found 73 from 500,000 certificates at $560. This represents twice as many differentials at a tenth the cost and a thousandth the inputs. Similar leads were observed for JWT and SAML. * **Unique Findings:** Venn diagrams clearly show that the vast majority of Chai's findings (e.g., 132 of 147 on X.509) were unique to Chai and not surfaced by any baseline, indicating it explores different and often more fruitful parts of the input space. * **Comparison to Other AI Systems:** The paper highlights that the wolfSSL codebase had recently been audited by Anthropic's Mythos, yet Chai discovered two severe vulnerabilities that Mythos missed. This strongly suggests Chai's approach reaches vulnerabilities that prevailing AI-driven methods overlook. * **Cost Analysis:** The evaluation includes cost metrics (inference/GPU costs) for LLM-driven methods, which is crucial for practical assessment. The evaluation demonstrates clear, reproducible improvements on important tasks and provides strong evidence for the effectiveness and novelty of Chai's approach. The real-world impact of the discovered vulnerabilities is a testament to its significance.
The paper provides a good level of detail regarding implementation, which aids reproducibility: * **Protocol Domains:** Chai spans X.509, JWT, and SAML, with approximate lines of code for each system. * **Technologies:** Python is the primary language. Specific libraries are mentioned (cryptography, asn1crypto, pyOpenSSL for X.509; cryptography, ecdsa for JWT; signxml, lxml for SAML). * **LLM Integration:** Agent requests route through LiteLLM to track spend and route to multiple providers. Specific models used are listed: GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8, and the open-source Kimi K2.6. Embedding model (OpenAI's text-embedding-3-small) and similarity metric (cosine similarity) are specified. * **Harnesses:** Described as small native-language scripts invoked as subprocesses, exchanging JSON verdicts. Languages covered include C, Go, Ruby, PHP, Node.js. * **PoC Pipeline:** Uses GCP SDK for parallel VM launches, Jinja templates for reports, and Postgres/React for the web visualizer. * **Baselines:** Specific baselines like MLCerts, jwt-fuzzer, jwt_tool, AFL++ are named. While the exact prompts for the LLM agents are not provided (which is common in LLM-based security research), the detailed description of the system architecture, specific tools, and models used, along with the inclusion of an open-source model (Kimi K2.6), suggests a strong commitment to reproducibility. The deterministic nature of the builder and the minimization process also contribute to reproducible findings.
* **Empirical Coverage:** Chai's agents generate inputs probabilistically, meaning the uncovered discrepancies represent a subset of those present. The coverage is empirical rather than exhaustive. * **Human in the Loop:** While a strength for ethical disclosure, the final manual review and preparation for each disclosure (2-5 hours per finding) is labor-intensive, limiting the ultimate scale of fully verified and disclosed findings. * **Scope of Vulnerabilities:** Chai does not directly detect flaws in the underlying mathematical constructions or in the specifications themselves. It focuses on implementation discrepancies. * **Reliance on Disagreement:** An issue surfaces only when at least two independent implementations disagree on the same input, potentially missing bugs in universally flawed implementations. * **LLM Dependence:** The effectiveness of the agentic reasoning is dependent on the capabilities and cost of the underlying LLMs, which can be prone to hallucinations or high costs, even if amortized. * **Protocol Specificity:** While the *approach* is generalizable, adapting Chai to new cryptographic protocols still requires harnessing libraries and supplying seed messages, implying some engineering effort per protocol.
Chai has a profound broader impact on several fronts: * **Revolutionizing Vulnerability Discovery:** It offers a paradigm shift for discovering cryptographic misuse vulnerabilities, a critical and hard-to-find class of bugs. By providing a verifiable signal where traditional instrumentation fails, it makes AI-assisted discovery practical for these domains. * **Enhanced Software Supply Chain Security:** Cryptographic libraries are fundamental components in countless applications. Discovering and tracing vulnerabilities in these libraries and their downstream usage significantly strengthens the security of the entire software supply chain, impacting billions of devices and users. * **Compounding Efficiency:** The "amplified testing" and "discrepancy tracing" mechanisms offer a highly efficient model for security auditing, potentially reducing the cost and time required to find high-impact bugs. This efficiency gain could enable more frequent and comprehensive security assessments. * **Ethical AI in Security:** The emphasis on a human in the loop for final verification and responsible disclosure is crucial. This ensures that AI-generated findings are properly vetted before being reported to maintainers, fostering trust and collaboration rather than burdening open-source projects with false positives. * **Generalizability:** The core principles of combining differential testing with adaptive AI search and discrepancy tracing could be extended to other complex, "hard-to-instrument" vulnerability classes beyond cryptography, opening new avenues for AI-assisted security research. Chai introduces a novel AI-based system that rethinks differential testing to effectively discover and validate cryptographic misuse vulnerabilities, demonstrating significant real-world impact by finding critical flaws in widely used libraries and outperforming existing tools. The paper's strength lies in its innovative paradigm shift from auditing individual codebases to cataloging library-level flaws and propagating them across dependency graphs, coupled with a rigorous experimental evaluation that confirms its superior efficiency and ability to uncover unique, high-severity vulnerabilities previously missed by other advanced AI systems.
Gradient equilibrium (GEQ) is a recently introduced online optimization framework that generalizes first-order stationarity from offline optimization and abstracts problems like online conformal prediction. While GEQ has curious similarities with known online learning frameworks, namely regret minimization, prior work has shown that GEQ error and regret are incomparable objectives, leaving open a precise understanding of how GEQ fits into the broader online learning landscape. In this work, we show that GEQ is equivalent to Blackwell approachability in the algorithmic sense. That is, a Blackwell approachability problem can always be solved using queries to a black-box GEQ oracle, with no asymptotic loss in the oracle's error rate, and vice versa. Taken together with known equivalences between approachability, regret minimization, and calibration, these results imply that GEQ is equivalent to these frameworks, as well. Our reductions are efficient and can be used to transfer refined guarantees, such as optimism and strong adaptivity, from regret minimization to GEQ. Along the way, we also identify necessary and sufficient conditions for GEQ, and establish reductions between different notions of GEQ with unconstrained and constrained decision sets.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley, Inria & École Normale Supérieure
This work has a significant broader impact on the field of online learning: 1. **Theoretical Unification**: It provides a crucial piece in the puzzle of understanding the relationships between different online learning frameworks. By rigorously establishing the equivalence between GEQ and Blackwell Approachability (and consequently, regret minimization and calibration), it offers a more unified and coherent theoretical landscape. 2. **Algorithmic Transferability**: The black-box oracle reductions are a powerful tool for algorithm design. They enable the direct transfer of algorithms and refined guarantees (such as optimism and strong adaptivity) from well-studied frameworks like regret minimization to the newer GEQ framework, and vice versa. This can accelerate the development of more sophisticated algorithms for GEQ-type problems and simplify the design of certain regret minimization algorithms. 3. **Deeper Understanding of GEQ**: Identifying Blackwell's condition as a necessary and sufficient condition for GEQ provides a fundamental theoretical characterization, moving beyond previously known sufficient conditions like restorativity. This deeper understanding can guide future research into the fundamental properties and solvability conditions of GEQ problems. 4. **Applications in Statistical Problems**: GEQ abstracts problems like online conformal prediction and online quantile debiasing. By connecting GEQ to other frameworks, this work opens avenues for applying established techniques and guarantees from regret minimization or calibration to these statistical problems, potentially leading to improved algorithms or stronger theoretical guarantees. 5. **Foundation for Future Research**: The paper lays a strong theoretical foundation, inviting further research into exploring other minimal conditions for GEQ, investigating the practical computational efficiency of these oracle compositions, and applying the transferred guarantees to a wider range of online learning applications. The paper rigorously establishes an algorithmic equivalence between Gradient Equilibrium (GEQ) and Blackwell Approachability (BA), thereby unifying GEQ with regret minimization and calibration and enabling the transfer of advanced algorithmic guarantees across these online learning frameworks. This work makes a fundamental theoretical contribution to online learning by precisely positioning the recently introduced GEQ framework within the broader landscape of online optimization. Through elegant black-box oracle reductions, the authors demonstrate that GEQ problems can be solved using BA algorithms and vice versa, with no asymptotic loss in error rates. This equivalence is then extended to regret minimization and calibration, providing a powerful conceptual unification and practical tool for algorithm design, as it allows for the transfer of sophisticated guarantees like optimism and strong adaptivity across these paradigms. The methodology is sound and rigorous, involving careful definitions, proofs of conditions (e.g., restorativity implying Blackwell's condition), and a clever reduction from constrained to unconstrained GEQ, all contributing to a significant advancement in the theoretical understanding of online learning.
The paper's methodology is centered on establishing algorithmic equivalences between Gradient Equilibrium (GEQ) and Blackwell Approachability (BA) through rigorous black-box oracle reductions. This involves two main directions: 1. **Reducing GEQ to BA**: The authors interpret a GEQ problem as a specific BA problem where the vector payoffs are the negative subgradients and the target set is the origin. A key technical contribution here is demonstrating that the restorativity condition (a known sufficient condition for GEQ) implies Blackwell's condition (the necessary and sufficient condition for BA). To make the reduction constructive, they propose an "approximate halfspace oracle" that uses a growing function `phi(t)` to select decisions. This oracle may make a bounded number of errors, which is then handled by leveraging the robustness properties of standard BA algorithms (like Blackwell's algorithm). The analysis shows that the error rate of the GEQ algorithm derived from a BA oracle is asymptotically equivalent to the BA oracle's rate. 2. **Reducing BA to GEQ**: This direction is more complex and involves two sub-steps: * **BA to Constrained GEQ**: Assuming the BA target set `S` is a cone (which can be generalized via conic lifting), the authors construct a GEQ problem where the decision set is the polar cone `S^` and the vector field `g_t(u)` is defined as `-f(O_H(u), b_t)`, where `O_H` is a halfspace oracle for the BA problem. They show that this constructed GEQ problem satisfies the necessary assumptions (boundedness, restorativity) and that solving it with a GEQ oracle leads to a solution for the BA problem. The proof ingeniously uses the normal vectors guaranteed by the GEQ oracle as "primal witnesses" for the approachability of the target set. * **Constrained GEQ to Unconstrained GEQ**: This crucial technical lemma completes the loop. It shows how to solve any GEQ problem with a constrained decision set `X` using an oracle for unconstrained GEQ (`X = R^d`). This is achieved by modifying the original vector field `g(x)` into `g'(x) = g(Proj_X(x)) + n_g(x)`, where `n_g(x)` is a scaled projection residual. This modification ensures that `g'(x)` is restorative and that the projection residual term `n_g(x)` effectively acts as a normal vector to `X` at `Proj_X(x)`, thus linking the unconstrained GEQ solution back to the constrained GEQ definition. The methodology is highly rigorous, relying on precise definitions of oracles and conditions. The black-box nature of the reductions makes them broadly applicable, allowing for the transfer of algorithmic guarantees across frameworks. The paper also provides a detailed technical overview, explaining the intuition behind the reductions, particularly the "primal" interpretation of their BA-to-GEQ reduction in contrast to the "dual" interpretation of prior work connecting BA to regret minimization.
The paper is purely theoretical and does not include any experimental evaluation. This is entirely appropriate for its venue (COLT) and the nature of its contribution, which is to establish fundamental theoretical equivalences and algorithmic implications rather than demonstrate empirical performance on specific tasks. The focus is on mathematical proofs, oracle reductions, and asymptotic error rate guarantees.
As a theoretical paper, reproducibility pertains to the clarity and correctness of its definitions, theorems, lemmas, and proofs. The paper provides comprehensive definitions for Blackwell Approachability and Gradient Equilibrium, clearly states assumptions, and presents algorithms in pseudocode. All new claims are supported by detailed mathematical proofs. A reader with a solid background in online learning theory and convex analysis should be able to follow and verify the logical steps and derivations. There are no code implementations or experimental setups to reproduce.
1. **Purely Theoretical**: The primary limitation is the absence of empirical validation. While justified for a COLT paper, it means the practical implications of transferring guarantees (e.g., the actual performance benefits of "optimistic" GEQ algorithms) are not explored. 2. **Efficiency of Reductions**: While the reductions are "efficient" in an asymptotic sense, the constant factors and computational overhead introduced by composing multiple black-box oracles (e.g., repeated halfspace oracle queries, projections, or the specific choice of `phi(t)`) are not deeply analyzed in terms of practical runtime. 3. **Assumptions**: The reductions rely on specific assumptions, such as the boundedness of payoffs/gradients and the restorativity condition for GEQ, or Blackwell's condition for BA. While these are standard in their respective contexts, they might not hold for all conceivable online learning problems. 4. **Conic Lifting Detail**: The reduction from general BA to GEQ relies on a "conic lifting argument" from prior work. While this is a standard technique, the full details of this lifting are not provided in the main text, requiring familiarity with external literature. 5. **Specifics of `phi(t)`**: The choice of the growing function `phi(t)` in the approximate halfspace oracle impacts the constant factor in the error bound. The paper provides examples but doesn't delve into the optimal choice or its practical implications for different problem settings.
This work has a significant broader impact on the field of online learning: 1. **Theoretical Unification**: It provides a crucial piece in the puzzle of understanding the relationships between different online learning frameworks. By rigorously establishing the equivalence between GEQ and Blackwell Approachability (and consequently, regret minimization and calibration), it offers a more unified and coherent theoretical landscape. 2. **Algorithmic Transferability**: The black-box oracle reductions are a powerful tool for algorithm design. They enable the direct transfer of algorithms and refined guarantees (such as optimism and strong adaptivity) from well-studied frameworks like regret minimization to the newer GEQ framework, and vice versa. This can accelerate the development of more sophisticated algorithms for GEQ-type problems and simplify the design of certain regret minimization algorithms. 3. **Deeper Understanding of GEQ**: Identifying Blackwell's condition as a necessary and sufficient condition for GEQ provides a fundamental theoretical characterization, moving beyond previously known sufficient conditions like restorativity. This deeper understanding can guide future research into the fundamental properties and solvability conditions of GEQ problems. 4. **Applications in Statistical Problems**: GEQ abstracts problems like online conformal prediction and online quantile debiasing. By connecting GEQ to other frameworks, this work opens avenues for applying established techniques and guarantees from regret minimization or calibration to these statistical problems, potentially leading to improved algorithms or stronger theoretical guarantees. 5. **Foundation for Future Research**: The paper lays a strong theoretical foundation, inviting further research into exploring other minimal conditions for GEQ, investigating the practical computational efficiency of these oracle compositions, and applying the transferred guarantees to a wider range of online learning applications. The paper rigorously establishes an algorithmic equivalence between Gradient Equilibrium (GEQ) and Blackwell Approachability (BA), thereby unifying GEQ with regret minimization and calibration and enabling the transfer of advanced algorithmic guarantees across these online learning frameworks. This work makes a fundamental theoretical contribution to online learning by precisely positioning the recently introduced GEQ framework within the broader landscape of online optimization. Through elegant black-box oracle reductions, the authors demonstrate that GEQ problems can be solved using BA algorithms and vice versa, with no asymptotic loss in error rates. This equivalence is then extended to regret minimization and calibration, providing a powerful conceptual unification and practical tool for algorithm design, as it allows for the transfer of sophisticated guarantees like optimism and strong adaptivity across these paradigms. The methodology is sound and rigorous, involving careful definitions, proofs of conditions (e.g., restorativity implying Blackwell's condition), and a clever reduction from constrained to unconstrained GEQ, all contributing to a significant advancement in the theoretical understanding of online learning.
Large language models can serve as capable long-horizon agents, but their out-of-distribution (OOD) generalization remains weak. We identify a key source of this failure as task insensitivity: when faced with similar but distinct tasks, models might apply patterns learned during training and fail to solve the task at hand. We show that models often continue with actions aligned with the original task even when the instruction is semantically corrupted and cannot be directly answered. We further find that, when we replace the task description in a trained prompt with another similar but distinct task, the model may still output the same action. This behavior is accompanied by a consistent training-time attention drift away from task tokens and toward local observations, suggesting an optimization bias toward shortcuts. To mitigate this problem, we propose Task-Perturbed NLL Optimization, a lightweight contrastive regularizer that explicitly encourages action dependence on the task instruction. Extensive evaluations show that our intervention improves task sensitivity and OOD generalization while preserving more stable attention to task tokens.
Primary: Gaoling School of Artificial Intelligence Renmin University of China
All Institutions: Gaoling School of Artificial Intelligence Renmin University of China, Tmall Group of Alibaba, Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Engineering Research Center of Next-Generation Intelligent Search and Recommendation
This paper makes a significant contribution to understanding and mitigating a critical failure mode in large language model agents: out-of-distribution generalization. By identifying "task insensitivity" and providing clear diagnostics, it offers a new lens through which to analyze agent behavior. The proposed Task-Perturbed NLL Optimization is a lightweight and effective regularization technique that can be readily integrated into existing SFT and RL pipelines. This has direct implications for developing more robust and reliable LLM agents, particularly in long-horizon tasks where maintaining task grounding is paramount. The insights into attention drift also contribute to the broader understanding of how LLMs learn and potentially overfit, informing future research on interpretability and robust training methodologies for large models beyond agents. This paper identifies task insensitivity as a key source of out-of-distribution generalization failure in LLM agents, provides strong empirical diagnostics and mechanistic insights into its development, and proposes an effective, lightweight contrastive regularizer to mitigate it. The comprehensive analysis, rigorous experimentation across multiple benchmarks, and clear improvements in agent robustness make this a highly impactful contribution to the field of LLM agents and robust AI.
The paper introduces a compelling diagnostic framework to identify "task insensitivity" in language agents. This framework includes: 1) **Corrupted Task Descriptions**: Modifying task instructions to be ambiguous or underspecified while allowing clarification, observing if models reconstruct the original task. This is a clever way to probe reliance on learned patterns. 2) **Controlled OOD Task Shifts**: Replacing original task descriptions with similar but distinct OOD tasks, specifically filtering for cases requiring different actions. This directly tests generalization beyond training distributions. 3) **Attention Analysis**: Tracking attention allocation to task tokens versus local observations during training to understand the underlying mechanism. This provides mechanistic insight into the observed behavioral failures. The proposed solution, **Task-Perturbed NLL Optimization**, is a lightweight contrastive regularizer. It explicitly encourages action dependence on the task instruction by making the original action less likely when the task is perturbed. A key methodological strength is the use of a frozen reference model to calibrate the desired separation, preventing over-penalization and ensuring a more stable optimization target. The extension to RL by perturbing tasks and reusing advantage scores is also a practical consideration. The theoretical intuition provided through simplified gradient and local attention reallocation analyses, while not a formal proof, offers a plausible explanation for the observed attention drift and strengthens the motivation for the proposed intervention.
The experimental evaluation is extensive and well-designed, covering three diverse embodied-agent benchmarks: ALFWorld, ScienceWorld, and WebShop. The use of controlled OOD splits (e.g., "heat and place" vs. "cool and place" in ALFWorld) is crucial for isolating the problem of task insensitivity. The paper evaluates both supervised fine-tuning (SFT) and reinforcement learning (GRPO) settings, demonstrating the broad applicability of the findings and the proposed method. Baselines include vanilla SFT, CoIN, and task augmentation, providing a good comparative context. The results consistently show that Task-Perturbed NLL Optimization improves OOD generalization across environments and training paradigms. Crucially, the method also preserves more stable attention to task tokens and reduces the growth of task-insensitive behavior (measured by conditional consistency), directly supporting the paper's hypothesis and the mechanism of the intervention. The use of GPT-5.4 as a judge for semantic reconstruction adds a layer of qualitative validation. The detailed breakdown of attention across different prompt regions in the appendix further strengthens the empirical analysis.
The paper provides a good level of detail for reproducibility. The appendix includes specific information on dataset splits for ALFWorld, ScienceWorld, and WebShop, training details (epochs, learning rates, batch sizes, GPU setup, GRPO parameters), and the full prompt templates used for agents and the judge model. Crucially, the implementation details for Task-Perturbed NLL Optimization, including response masking and the reference-ratio computation, are clearly described. This level of detail should allow researchers to replicate the experiments and validate the findings.
The authors acknowledge that the empirical study covers only three agent settings (ALFWorld, ScienceWorld, WebShop), which, while diverse, represent a narrow slice of agentic behavior. This suggests that the generalizability of "task insensitivity" and the proposed solution to broader, more complex agent environments (e.g., open-ended web agents, real-world robotics) needs further investigation. Another limitation noted is that the attention analysis is mechanistically suggestive rather than causally definitive. While the observed attention drift aligns with the optimization account, it does not fully establish the causal mechanisms of internal decision-making within the transformer.
This paper makes a significant contribution to understanding and mitigating a critical failure mode in large language model agents: out-of-distribution generalization. By identifying "task insensitivity" and providing clear diagnostics, it offers a new lens through which to analyze agent behavior. The proposed Task-Perturbed NLL Optimization is a lightweight and effective regularization technique that can be readily integrated into existing SFT and RL pipelines. This has direct implications for developing more robust and reliable LLM agents, particularly in long-horizon tasks where maintaining task grounding is paramount. The insights into attention drift also contribute to the broader understanding of how LLMs learn and potentially overfit, informing future research on interpretability and robust training methodologies for large models beyond agents. This paper identifies task insensitivity as a key source of out-of-distribution generalization failure in LLM agents, provides strong empirical diagnostics and mechanistic insights into its development, and proposes an effective, lightweight contrastive regularizer to mitigate it. The comprehensive analysis, rigorous experimentation across multiple benchmarks, and clear improvements in agent robustness make this a highly impactful contribution to the field of LLM agents and robust AI.
Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets. This creates an activation-dark regime: representation-similarity metrics can be uninformative without assumptions about label-conditioned error geometry, while classical update-geometry metrics are computationally prohibitive at vocabulary scale. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates. The key identity is that head Fisher alignment is exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors rather than requiring a materialized Fisher matrix. FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature (m=4096) and a 192 KB per-task streaming state, small enough to store next to a model hash, but encoding transfer-relevant update structure. Beyond source selection, the same signatures and marginals provide a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling; shared-parameter and internal-layer validations, together with Llama-3.1-8B verbalizer-shift experiments, show that FisherSketch remains informative when activation similarity cannot distinguish tasks.
Primary: Sideplane AI
All Institutions: Sideplane AI
The paper presents a theoretically grounded and practically significant method for estimating transfer compatibility at scale, solving a key bottleneck in automated LLM adaptation. By reformulating Fisher alignment as a kernel mean embedding cosine similarity, it enables vocabulary-scale analysis with minimal storage overhead, offering a new diagnostic tool for understanding LLM task similarity beyond representation learning.
The paper introduces "FisherSketch," a novel method for estimating head-level Fisher alignment in large language models without materializing the full Fisher Information Matrix. The core theoretical contribution is the derivation that head Fisher alignment can be expressed as a cosine similarity between kernel mean embeddings in a joint activation-error space. This allows for a streaming, single-pass estimation of transfer compatibility using only 16 KB of storage per task. The approach addresses the "activation-dark" regime where standard representation similarity metrics (like CKA) fail to predict transfer outcomes because they ignore the geometry of the output head updates. The methodology is mathematically sound, leveraging random feature maps (kernel mean embeddings) to approximate the inner product of Fisher matrices efficiently.
The evaluation covers scientific string domains (SMILES, protein, genomic sequences) where tokenizers are shared but prediction targets differ. The authors demonstrate that FisherSketch effectively identifies source corpora that yield positive transfer, outperforming activation-based metrics which are shown to be non-identifiable in this setting. Experiments include shared-parameter validations, internal-layer checks, and verbalizer-shift experiments on Llama-3.1-8B. The results show that FisherSketch remains informative even when activation similarity cannot distinguish tasks. The use of 29 tables suggests a comprehensive empirical study, although the specific magnitude of improvement over baselines is not detailed in the abstract, the claim of solving a previously prohibitive computational problem (vocabulary-scale Fisher alignment) is strong.
The paper provides specific implementation details, including the signature size (16 KB), streaming state size (192 KB), and the number of samples ($m=4096$). The method is described as a single streaming pass, which implies straightforward implementation. However, the exact hyperparameters for the kernel mean embedding approximation and the specific linear probe settings (SGD, lr=0.1) are mentioned. The reliance on specific LLM families and shared vocabularies limits immediate generalizability to all LLMs, but the code for FisherSketch itself should be reproducible given the description.
The method is specifically tailored for "shared-output head" settings and "shared vocabulary" tokenizers, which restricts its applicability to cross-lingual or cross-modal transfer where vocabularies differ. The "activation-dark" regime assumption means it may not offer advantages in domains where activation geometry is already highly predictive. Furthermore, the paper acknowledges that compatibility estimates can be wrong, potentially leading to negative transfer if relied upon exclusively without other safeguards. The computational savings are significant compared to full Fisher computation, but the cost of the streaming pass and kernel embedding approximation must be weighed against the cost of just fine-tuning a small candidate set.
This work enables more efficient and reliable automated fine-tuning pipelines by providing a cheap, accurate signal for source selection. This can reduce compute waste and negative transfer in large-scale adaptation. The impact statement rightly highlights risks regarding bias amplification if automated selection deprioritizes underrepresented data. The ability to detect interference patterns early is a significant benefit for multi-task learning and continual learning systems. The paper presents a theoretically grounded and practically significant method for estimating transfer compatibility at scale, solving a key bottleneck in automated LLM adaptation. By reformulating Fisher alignment as a kernel mean embedding cosine similarity, it enables vocabulary-scale analysis with minimal storage overhead, offering a new diagnostic tool for understanding LLM task similarity beyond representation learning.
Efficient sampling of molecular systems at thermodynamic equilibrium is a hallmark challenge in statistical physics. This challenge has driven the development of Boltzmann Generators (BGs), which allow rapid generation of uncorrelated equilibrium samples by combining a generative model with exact likelihoods and an importance sampling correction. However, modern BGs predominantly rely on normalizing flows (NFs), which either suffer from limited expressivity due to strict invertibility constraints (discrete time) or computationally expensive likelihoods (continuous time). In this paper, we propose Autoregressive Boltzmann Generators (ArBG) -- a novel autoregressive modelling framework -- that overcomes these limitations by departing from the flow-based BG paradigm. ArBG circumvents the topological constraints of flows and enables sequential inference-time interventions, while offering enhanced scalability by leveraging architectures effective in Large Language Models. We empirically demonstrate that ArBG leads to significant improvements over flow-based models across all benchmarks, but particularly in larger peptide systems such as the 10-residue Chignolin. Furthermore, we introduce Robin, a 132 million parameter transferable model trained with the ArBG framework which improves over the previous state-of-the-art, reducing the zero-shot energy error, E-W$_2$, on 8-residue systems by over 60$\%$. The code can be found at the following link: https://github.com/danyalrehman/autobg.
Primary: Mila – Québec AI Institute
All Institutions: Mila – Québec AI Institute, Broad Institute of MIT & Harvard, Aithyra, University of Oxford, Université de Montréal, CIFAR, Imperial College London
This paper presents a significant methodological advance in Boltzmann Generation by introducing Autoregressive Boltzmann Generators (ArBG), which leverage discrete token prediction techniques to overcome the expressivity and computational limitations of normalizing flows, achieving state-of-the-art performance in molecular conformation sampling.
The paper proposes Autoregressive Boltzmann Generators (ArBG), a novel framework that replaces the dominant Normalizing Flow (NF) architecture in Boltzmann Generation with an autoregressive (AR) model. The core technical innovation lies in adapting discrete token prediction techniques (inspired by LLMs) to continuous molecular coordinates via uniform binning. This approach circumvents the topological constraints and Jacobian determinant costs associated with diffeomorphic flows. The authors introduce a "Twisted Sequential Monte Carlo" (SMC) inference scheme that leverages the autoregressive nature of the model to perform intermediate resampling based on partial energy evaluations, a capability not natively available in flow-based models. The methodology is theoretically grounded, with a proposition bounding the KL divergence error introduced by the binning discretization.
The empirical evaluation is comprehensive, covering single-peptide systems (AL3, AL4, AL6, Chignolin) and a transferable setting on unseen peptides. ArBG consistently outperforms state-of-the-art flow-based methods (SBG, FALCON, ECNF++) across Wasserstein energy ($E-W_2$) and torsional ($T-W_2$) metrics. A significant result is the 60% reduction in zero-shot energy error on 8-residue systems with the 132M parameter "Robin" model compared to the previous SOTA (Prose). The scaling analysis demonstrates favorable inference-time scaling relative to Molecular Dynamics and other generative baselines. The ablation studies on bin resolution and sampling temperature provide robust validation of the design choices.
The paper provides a GitHub link to the code repository. The methodology for binning, the specific metrics (Wasserstein distances in energy and torsional space), and the baseline implementations are clearly described. The inclusion of ablation studies on hyperparameters (temperature, bin count) enhances reproducibility. The use of standard benchmarks (ManyPeptidesMD) facilitates comparison.
The autoregressive formulation imposes a fixed ordering on atomic coordinates, which is arbitrary for molecules and may impact performance or require careful handling of symmetries (though the paper notes PDB ordering helps). The uniform binning introduces an irreducible discretization error, which may limit precision for very sharp energy minima in larger systems. The "Twisted SMC" showed marginal gains over standard SNIS in the tested regimes, suggesting its primary value may be in more complex, out-of-distribution scenarios or for guided generation rather than pure equilibrium sampling in this specific regime.
This work bridges the gap between large-scale autoregressive modeling (LLMs) and scientific machine learning (molecular sampling). By demonstrating that AR models can outperform specialized flow-based models in a rigorous physical benchmark, it opens a new direction for generative modeling in statistical physics and drug discovery. The ability to perform inference-time interventions via SMC could enable more efficient sampling in complex energy landscapes. This paper presents a significant methodological advance in Boltzmann Generation by introducing Autoregressive Boltzmann Generators (ArBG), which leverage discrete token prediction techniques to overcome the expressivity and computational limitations of normalizing flows, achieving state-of-the-art performance in molecular conformation sampling.
A central goal of safety research is determining whether a model is misaligned. Prior work has largely focused on detecting concerning behavior. But behavior alone does not establish misalignment: a concerning action can arise from benign causes such as confusion. This motivates model forensics: investigating whether the action was driven by malign intent. In this paper, we propose a baseline protocol for model forensics consisting of two steps, iterated as needed. First, we read the chain of thought (CoT) to generate hypotheses about what drives model behavior. Second, we make edits to the prompt or environment to test these hypotheses. While the CoT is not always faithful, it is a rich source of unsupervised insight that can guide the collection of more rigorous evidence. To evaluate our protocol, we create a suite of six agentic environments where models exhibit concerning behavior, and apply it to each. We establish that Kimi K2 Thinking takes shortcuts due to a genuine disposition towards low-effort actions, by showing this hypothesis successfully predicts its behavior. Through counterfactual experiments, we show DeepSeek R1 deceives out of a desire to be consistent with a previous instance of itself. Our methods nonetheless leave significant room for refinement. For example, when we test whether Kimi K2 Thinking believes it is violating user intent, we find no evidence of such a belief, but without positive controls we cannot confirm our tests would detect it. Overall, we find our simple protocol provides a strong baseline that we hope future work will improve upon. More broadly, our work is a concrete step in developing the growing field of model forensics.
Primary: ML Alignment | All: & Theory Scholars (MATS) program
All Institutions: ML Alignment | All: & Theory Scholars (MATS) program
This paper makes a genuinely significant contribution to the burgeoning field of ML safety and alignment. By formalizing "model forensics," it provides a critical framework for understanding *why* AI models exhibit concerning behaviors, moving beyond mere detection. This distinction between benign causes (e.g., confusion, overzealousness) and malign intent (misalignment) is paramount for developing effective mitigation strategies. If a model is simply confused, a clarification might suffice; if it's intentionally deceptive, far more robust safeguards are needed. The proposed protocol, environments, and methodological insights will serve as a strong baseline for future research, enabling more rigorous and systematic investigations into model motivations. This work has direct implications for the safe deployment of increasingly autonomous AI agents, helping developers and auditors make informed decisions about model trustworthiness and the necessary level of oversight. It also contributes to the broader interpretability literature by focusing on complex, agentic trajectories rather than single forward passes. This paper introduces a robust baseline protocol for "model forensics," a critical methodology for investigating whether concerning AI behavior stems from benign causes or genuine misalignment. Through a systematic two-step iterative process of hypothesis generation (primarily via Chain of Thought) and validation (via environment interventions), the authors provide a foundational framework, a suite of six agentic environments, and detailed case studies that demonstrate its efficacy in distinguishing model motivations. The work is highly significant for ML safety, offering practical guidance and open-sourced resources to enable more rigorous and reproducible investigations into the internal states and intentions of frontier AI models.
The paper proposes a simple yet effective two-step iterative protocol for model forensics: (1) Hypothesis Generation, primarily by reading the Chain of Thought (CoT), supplemented by techniques like sentence resampling and user-turn sampling; and (2) Hypothesis Validation, mainly through environment interventions (counterfactuals or prediction testing), and repeated resampling. This protocol is designed to investigate the motivations behind concerning model behavior, distinguishing between benign causes (e.g., confusion) and malign intent (misalignment). The strength of the methodology lies in its systematic approach to a complex problem, emphasizing the need for converging lines of evidence due to the absence of ground truth. The explicit acknowledgment that CoT is not always faithful but serves as a rich source of unsupervised insight is pragmatic. The inclusion of existing interpretability techniques like sentence and repeated resampling within this framework is a smart integration, leveraging established methods for a new application. The iterative nature of the protocol, where validation results feed back into hypothesis generation, is crucial for refining understanding. The paper also provides clear standards for rigorous investigations, such as using control settings/models and checking common benign explanations, which are vital for establishing a robust methodology in this nascent field.
The experimental evaluation is comprehensive and well-structured. The creation of a suite of six agentic environments (Pre-commit Hook, Funding Email, Evaluation Tampering, Secret Number, Board Games, Math Sandbagging) is a significant contribution. These environments are designed with thoughtful principles to ensure realism, unprompted behavior, clear user intent, and legitimate courses of action, addressing common pitfalls of prior misalignment evaluations. The application of the proposed protocol to each environment results in six detailed case studies, which effectively demonstrate the protocol's utility. The findings from these case studies are specific and non-trivial, such as Kimi K2 Thinking's disposition towards low-effort actions in Pre-commit Hook, or DeepSeek R1's strong dependence on self-consistency for deception in Evaluation Tampering. The use of frontier models (Kimi K2 Thinking, DeepSeek R1, Kimi K2.5, DeepSeek v3.2, o3, GPT-5, Gemini 3 Pro) adds to the relevance and impact of the findings. The paper rigorously discusses methodological insights derived from these case studies, highlighting the strengths of predictions as evidence and the challenges of interpreting negative results or confounded counterfactuals. The quantitative results, including workaround rates and deception rates, are presented with confidence intervals, adding to the empirical rigor.
The paper demonstrates excellent commitment to reproducibility. It explicitly states that the environments, transcripts, and reproducibility code are open-sourced. Providing links to the GitHub repositories for the environments and code, and a HuggingFace dataset for transcripts, makes it straightforward for other researchers to replicate the experiments, build upon the environments, and further develop the model forensics methodology. This level of transparency and resource sharing is commendable and crucial for advancing research in ML safety.
The paper is commendably transparent about its limitations. Key limitations include: 1. **Interpretation of Negative Results:** The difficulty in interpreting negative results (absence of evidence) due to potential confounds like capability limitations, competing motivations, or eval awareness. The lack of positive controls to validate behavioral tests is a noted weakness. 2. **Confounding in Counterfactuals:** Counterfactual experiments, while flexible, can suffer from non-linear interaction effects between factors, incomplete interventions (not fully acting on the targeted latent), and unintended side effects that confound interpretation. 3. **CoT Faithfulness:** While acknowledged, the reliance on CoT for hypothesis generation still carries the inherent risk of unfaithfulness, which could lead to incorrect initial hypotheses. 4. **Scalability:** The manual reading of many rollouts for hypothesis generation, while informative, may not scale efficiently to extremely complex agentic behaviors or very long trajectories. 5. **Generalizability of "Motivations":** The definition of motivations as "simple, easy-to-describe factors" is pragmatic but acknowledges that models may not have coherent, human-like motivations, which could limit the depth of understanding achievable. 6. **Future Challenges:** The paper notes that more capable models will pose additional challenges like plausible deniability and situational awareness, which current methods may not fully address.
This paper makes a genuinely significant contribution to the burgeoning field of ML safety and alignment. By formalizing "model forensics," it provides a critical framework for understanding *why* AI models exhibit concerning behaviors, moving beyond mere detection. This distinction between benign causes (e.g., confusion, overzealousness) and malign intent (misalignment) is paramount for developing effective mitigation strategies. If a model is simply confused, a clarification might suffice; if it's intentionally deceptive, far more robust safeguards are needed. The proposed protocol, environments, and methodological insights will serve as a strong baseline for future research, enabling more rigorous and systematic investigations into model motivations. This work has direct implications for the safe deployment of increasingly autonomous AI agents, helping developers and auditors make informed decisions about model trustworthiness and the necessary level of oversight. It also contributes to the broader interpretability literature by focusing on complex, agentic trajectories rather than single forward passes. This paper introduces a robust baseline protocol for "model forensics," a critical methodology for investigating whether concerning AI behavior stems from benign causes or genuine misalignment. Through a systematic two-step iterative process of hypothesis generation (primarily via Chain of Thought) and validation (via environment interventions), the authors provide a foundational framework, a suite of six agentic environments, and detailed case studies that demonstrate its efficacy in distinguishing model motivations. The work is highly significant for ML safety, offering practical guidance and open-sourced resources to enable more rigorous and reproducible investigations into the internal states and intentions of frontier AI models.
AI agents are granted access to tools, APIs, and other infrastructure, making them active principals in those systems. The dominant approach places controls inside the agent's own runtime: system prompts, output filters, and guardrail libraries. Any control in the agent's address space is reachable by inputs that influence it; this generalizes to any AI system with sufficient reach into its own runtime, a class we term escapable AI systems. We identify four properties that an authorization mechanism must satisfy for architectural control rather than for cooperative requests: process separation, pre-action enforcement on a structurally only path, fail-closed at both the request and system levels, and externalized signed evidence verifiable outside the controlled system's trust boundary. We position this layer as execution-time AI alignment, complementing training-time alignment (RLHF, Constitutional AI) and inference-time alignment. We present the Unfireable Safety Kernel, a Rust reference implementation realizing all four. Its fail-closed invariant is machine-checked at two levels: an SMT theorem (Z3) and an exhaustive bounded-model-checking proof of the production decision function (Kani, 4/4 harnesses). A Python-to-Rust migration was gated on byte-equivalence (1000/1000 fixtures; 17/17 adversarial classes). We evaluate the kernel governing a live, escapable AI system, a deterministic, self-improving world model, against an escape-seeking adversary driving its real self-modification seam: across 1,000 self-modifications, all 704 attempts on the safety-critical core are refused, with no escape; a further 300, under the operator kill switch, are also refused. A separate campaign of 6,240 authorization round-trips had no successful bypass. Against 3 contemporary systems claiming the agent control plane, the agent invokes control; here, it lacks that choice.
Primary: ARYA Labs PBC
All Institutions: ARYA Labs PBC
This paper makes a profoundly significant contribution to the field of AI safety and the secure deployment of advanced AI agents. By formally defining "escapable AI systems" and introducing "execution-time AI alignment" as a distinct and crucial layer, it provides a vital conceptual framework for addressing a pressing and growing challenge. The Unfireable Safety Kernel offers a concrete, rigorously verified, and open-source architectural pattern that can fundamentally change how high-stakes AI systems are controlled. It shifts the paradigm from cooperative, in-runtime agent controls to architecturally enforced, unavoidable mediation, which is essential as AI capabilities increase and agents become more autonomous and potentially misaligned. This work has the potential to become a standard for building secure and corrigible AI agents, enabling safer deployment in critical applications where human authority must be durably maintained. It also sets a new benchmark for the level of rigor, particularly formal verification, applied to safety-critical components within AI systems. The Unfireable Safety Kernel introduces a rigorously verified, process-separated architectural control for AI agents, establishing "execution-time AI alignment" as a critical layer for durable human authority over escapable AI systems. This paper provides a comprehensive, open-source solution, formally verified with machine-checked proofs on production code, and empirically validated against a live, escape-seeking AI agent, setting a new standard for safety architecture in the deployment of autonomous AI.
The methodology is exceptionally strong and well-grounded in security engineering principles. The paper clearly identifies a critical architectural vulnerability in current AI agent safety controls, coining the term "escapable AI systems." It then rigorously derives four essential properties for an authorization mechanism to provide architectural control: process separation (P1), pre-action enforcement on a structurally-only path (P2), fail-closed at both request and system levels (P3), and externalized signed evidence (P4). These properties are not merely theoretical; they are meticulously translated into the design and implementation of the Unfireable Safety Kernel. The design incorporates a robust, multi-seam defense-in-depth architecture (nginx auth_request, application middleware, dispatch hook, client SDK circuit breaker), ensuring that every consequential action transits the kernel. The "no kernel, no agent" system-level lifecycle binding enforces P3 at a fundamental level. The choice of Rust for the implementation is highly appropriate for a security-critical component, leveraging memory safety, static compilation, and the absence of runtime mutability primitives. The hexagonal architecture enforces strict internal separation, and the inclusion of constant-time authentication further demonstrates attention to detail in security. The Python defense library, using `sys.addaudithook`, provides an elegant solution for P2 at the application layer. The overall methodology demonstrates a deep understanding of both AI agent capabilities and robust system security.
The experimental evaluation is comprehensive, rigorous, and highly compelling. It goes significantly beyond typical ML paper evaluations, focusing on security and correctness. 1. **Migration Equivalence**: The use of byte-equivalence gating for the Python-to-Rust migration, with 1000/1000 fixtures byte-equal and 17/17 adversarial classes showing identical reject behavior, is an excellent practice for trust-critical components. The continuous strict-mode equivalence lane, which caught nine production divergences, underscores the value of this ongoing verification. 2. **Latency**: A controlled comparison showed the Rust implementation to be 1.85x faster than Python at C=20, meeting the performance requirements for the migration. 3. **Adversarial Robustness**: The extensive corpus of over eighty robustness tests, covering a wide range of attack classes (token/signature forgery, replay, bypasses, transparency log tampering, client-side fail-closed, mTLS), provides strong evidence of the kernel's resilience. The enforcement of cross-language parity in testing is also a notable strength. 4. **Machine-Checked Fail-Closed Invariant**: This is a standout technical achievement. The fail-closed invariant (P3) is formally verified at two levels: an SMT theorem (Z3) over a symbolic model and, crucially, an exhaustive bounded-model-checking proof of the *production Rust decision function* using Kani (4/4 harnesses verified). This provides an extremely high degree of assurance that the core safety property holds for all inputs, directly binding the proof to the shipped code. 5. **Containment of a Live Escapable AI System**: This is the most impactful empirical demonstration. The kernel was evaluated governing a *deterministic, self-improving, fully autonomous world model* against a *purpose-built escape-seeking adversary* driving its real self-modification seam. Across 1,000 self-modifications, all 704 attempts against the safety-critical core were refused with no successful escape. A further 300 modifications were refused under the operator kill switch, confirming its absolute halt capability. This real-world adversarial test provides strong empirical validation of the kernel's ability to enforce control over a misaligned, self-modifying AI agent.
Reproducibility is excellent. The reference implementation is open-source under Apache-2.0 at `https://github.com/ARYA-Labs-Public/unfireable-safety-kernel`. The paper explicitly states that the migration, latency, robustness, and machine-checked evidence are reproducible from this public artifact. The deterministic generation of equivalence fixtures, the use of standard formal verification tools (Z3, Kani) on the production code, and the detailed descriptions of the evaluation methodology all contribute to a high level of reproducibility.
The paper transparently acknowledges several limitations: * It does not prevent prompt injection upstream of the kernel, focusing instead on mediating actions. * It does not filter model outputs as text, leaving content filtering to the application layer. * Side-channel leakage through patterns of allow/deny decisions is not yet mitigated. * Denial of service against the kernel itself is not prevented, though its fail-closed property converts this into a correctness-preserving outage. * Insider misuse of the operator key is detectable but not prevented by the current architecture, with multi-party schemes planned for future work. * The bypass count in the live system evaluation is specific to the tested attack taxonomy and not a completeness proof. * The persistence of changes after an authorized step was not confirmed in the live system run. These clearly stated limitations demonstrate a mature and responsible approach to system design and evaluation.
This paper makes a profoundly significant contribution to the field of AI safety and the secure deployment of advanced AI agents. By formally defining "escapable AI systems" and introducing "execution-time AI alignment" as a distinct and crucial layer, it provides a vital conceptual framework for addressing a pressing and growing challenge. The Unfireable Safety Kernel offers a concrete, rigorously verified, and open-source architectural pattern that can fundamentally change how high-stakes AI systems are controlled. It shifts the paradigm from cooperative, in-runtime agent controls to architecturally enforced, unavoidable mediation, which is essential as AI capabilities increase and agents become more autonomous and potentially misaligned. This work has the potential to become a standard for building secure and corrigible AI agents, enabling safer deployment in critical applications where human authority must be durably maintained. It also sets a new benchmark for the level of rigor, particularly formal verification, applied to safety-critical components within AI systems. The Unfireable Safety Kernel introduces a rigorously verified, process-separated architectural control for AI agents, establishing "execution-time AI alignment" as a critical layer for durable human authority over escapable AI systems. This paper provides a comprehensive, open-source solution, formally verified with machine-checked proofs on production code, and empirically validated against a live, escape-seeking AI agent, setting a new standard for safety architecture in the deployment of autonomous AI.
On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten (i.e., generating more rollouts fails to improve accuracy). We trace this to compounding biases in the design of self-distillation with sampled demonstrations. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model's own biases. We theoretically analyze the optimal self-distillation policy and show that it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the correct rollout used as context. Unlike the ideal optimal on-policy reinforcement learning (RL), which preserves probability ratios among equally correct rollouts, self-distillation can amplify existing probability gaps, concentrating mass on already-dominant modes. On a controlled graph path-finding task and science question-answering benchmarks, self-distilled models match or exceed RL on average performance but exhibit substantially lower functional and semantic diversity, failing on out-of-distribution settings that require diverse strategies.
Primary: Mila
All Institutions: Mila, Université de Montréa, FAIR at Meta, CIFAR AI Chair
This paper reveals that on-policy self-distillation with sampled demonstrations (SDSD), despite strong pass@1 accuracy, suffers from a fundamental diversity collapse due to its optimal policy tilting the base distribution by pointwise conditional mutual information, which amplifies existing probability gaps and leads to poor out-of-distribution performance. The paper provides a rigorous theoretical analysis, introduces novel functional and semantic diversity metrics, and empirically validates its claims on controlled graph path-finding and science QA tasks, demonstrating that SDSD models exhibit substantially lower diversity than RL-trained models, even when using diverse external demonstrations, and that token-level entropy is an insufficient measure of meaningful diversity.
The methodology is exceptionally strong, combining rigorous theoretical analysis with well-designed empirical investigations. The core theoretical contribution is the derivation of the optimal self-distillation policy (Proposition 3.2), showing it tilts the base distribution by the expected pointwise conditional mutual information (PCMI). This provides a clear, mathematical explanation for why SDSD can amplify existing probability imbalances and lead to diversity collapse, distinguishing it from general mode-seeking in RL. The comparison to the optimal RL policy (Remark 3.3) effectively highlights this crucial difference. The paper introduces two highly relevant and more meaningful notions of diversity: "functional diversity" (measured by the slope of pass@k curves) and "semantic diversity" (capturing high-level strategic variations). These are critical advancements over the often-misleading token-level entropy. The controlled graph path-finding task is a particularly innovative methodological contribution, allowing for precise measurement of semantic diversity and a direct link to out-of-distribution generalization, which is invaluable for diagnosing LLM behaviors.
The experimental evaluation is comprehensive, robust, and strongly supports the theoretical claims. The use of both a controlled synthetic task (concept graph path-finding) and real-world benchmarks (SciKnowEval science QA) provides a balanced and convincing validation of the diversity collapse phenomenon. The concept graph task effectively demonstrates the loss of semantic diversity and its direct consequence on out-of-distribution performance. The science QA experiments confirm the flattening of pass@k curves (indicating low functional diversity) in a practical LLM setting. The baselines, including standard GRPO and GRPO with an explicit diversity reward, are well-chosen. A particularly impactful finding is that SDSD's diversity collapse persists even when the teacher is conditioned on diverse *external* demonstrations, suggesting a fundamental mechanism at play rather than just a bias from self-generated samples. The paper also convincingly shows that token-level entropy is an unreliable metric for meaningful diversity, often failing to correlate with functional or semantic diversity. The experiments are well-controlled, using multiple seeds and modern LLMs (Qwen3, Olmo-3), enhancing the credibility of the results.
The paper provides a good level of detail for reproducibility. It specifies the base models (Qwen3-1.7B/8B, Olmo-3-7B-Instruct), datasets (SciKnowEval, custom graph dataset), training parameters (epochs, batch sizes, rollouts, temperature, optimizer AdamW), and hardware (4 Nvidia H200 GPUs, 3 seeds). The custom graph task is described with sufficient detail, including an example prompt in the appendix, making it feasible to re-implement. The mention of "NanoAhaMoment2025" as the library used is helpful. Overall, the information provided should allow for reasonably good reproducibility of the main results.
The authors are commendably transparent about the limitations. They explicitly state that the analysis focuses on self-distillation with *sampled correct rollouts* and does not cover settings with richer privileged signals (e.g., runtime errors, environmental feedback). They also acknowledge that the theoretical analysis assumes a frozen base policy teacher and demonstrations sampled from the base policy, whereas practical implementations often use EMA teachers and self-generated demonstrations, which could introduce additional biases not fully captured by the current theory. While the paper argues that the token-level derivation yields similar implications, a more detailed exploration of the compounding effects of PCMI at each token generation step could be a valuable extension. These identified limitations provide clear avenues for future research.
This paper has significant broader impact on the field of LLM training and evaluation. It fundamentally challenges the prevailing understanding of on-policy self-distillation, revealing a hidden cost (diversity collapse) that can undermine its apparent pass@1 strengths, especially for tasks requiring robustness, exploration, or out-of-distribution generalization. This insight is crucial for the responsible development and deployment of LLMs, as a lack of diversity can lead to brittleness, reduced creativity, and an inability to handle novel or ambiguous situations. The paper provides a robust theoretical framework and practical tools (functional/semantic diversity metrics, concept graph task) that the ML community can adopt to better evaluate and improve LLM training methods. It will likely stimulate research into diversity-preserving self-distillation techniques and more robust evaluation protocols for LLMs, contributing to a deeper understanding of LLM learning dynamics and their implications for real-world applications. This paper reveals that on-policy self-distillation with sampled demonstrations (SDSD), despite strong pass@1 accuracy, suffers from a fundamental diversity collapse due to its optimal policy tilting the base distribution by pointwise conditional mutual information, which amplifies existing probability gaps and leads to poor out-of-distribution performance. The paper provides a rigorous theoretical analysis, introduces novel functional and semantic diversity metrics, and empirically validates its claims on controlled graph path-finding and science QA tasks, demonstrating that SDSD models exhibit substantially lower diversity than RL-trained models, even when using diverse external demonstrations, and that token-level entropy is an insufficient measure of meaningful diversity.
Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.
Primary: University of Wisconsin-Madison
All Institutions: University of Wisconsin-Madison
The paper presents a novel and theoretically sound method for extracting step-level process rewards from standard RL post-training, offering a significant efficiency gain and performance improvement over existing methods for LLM agent evaluation and scaling.
The paper proposes a theoretically grounded method to derive step-level process rewards for Large Language Model (LLM) agents without requiring additional training or human annotation. The core theoretical contribution is the derivation of "progress advantage," defined as the log-probability ratio between the RL-fine-tuned policy and its reference policy. The authors claim this ratio exactly recovers the optimal advantage function under a general stochastic Markov Decision Process (MDP). This is a significant conceptual shift, moving away from the standard paradigm of training separate Process Reward Models (PRMs) or using Monte Carlo rollouts for value estimation. The methodology leverages the existing RL post-training signal (likely DPO or PPO) to extract granular feedback, which is computationally efficient and domain-agnostic. The theoretical justification provided in the method section appears rigorous, linking policy gradients to advantage functions in a way that makes the "free lunch" claim plausible.
The empirical evaluation is comprehensive, covering three distinct applications: test-time scaling, uncertainty quantification, and failure attribution. The authors evaluate across five benchmarks and four different model families, which strengthens the generalizability claims. The results indicate that the progress advantage signal consistently outperforms confidence-based baselines (like log-probability of the final answer) and, crucially, surpasses dedicated trained reward models despite requiring no task-specific training. This is a strong empirical finding. The comparison against trained PRMs is particularly compelling because it highlights the efficiency and effectiveness of the proposed "byproduct" signal. The inclusion of failure attribution analysis adds depth, showing how the signal can be used for diagnostic purposes in agentic workflows.
The paper provides a GitHub repository link, which is a positive indicator for reproducibility. The methodology is mathematically defined and relies on standard RL components (policy, reference policy, log-probs), making the implementation straightforward for researchers familiar with RLHF pipelines. The use of multiple model families and benchmarks also suggests that the code is likely modular. However, the specific details of the "five benchmarks" and "four model families" would need to be checked in the appendix for full reproducibility, but the core algorithm is simple enough to be replicated.
The primary limitation lies in the assumption that the RL post-training has converged sufficiently to provide a stable estimate of the advantage function. If the RL training is unstable or the reference policy is poorly calibrated, the progress advantage signal may be noisy. Additionally, the claim that it "exactly recovers" the optimal advantage function relies on specific assumptions about the MDP structure and the nature of the reward signal during RL training that may not hold in all real-world, highly stochastic agentic environments. The paper also notes that this is a "byproduct" signal, meaning its quality is inherently tied to the quality of the RL fine-tuning; if the RL fine-tuning fails to improve the policy, the progress advantage may not be informative.
This work has significant implications for the deployment of LLM agents. By eliminating the need for expensive and labor-intensive process reward model training, it lowers the barrier to entry for building robust, self-correcting agents. It enables more efficient test-time compute allocation and better uncertainty estimation, which are critical for safety and reliability in autonomous systems. The ability to attribute failures using this signal can also aid in debugging and improving agent architectures. The paper presents a novel and theoretically sound method for extracting step-level process rewards from standard RL post-training, offering a significant efficiency gain and performance improvement over existing methods for LLM agent evaluation and scaling.
Modern coding agents pair LLM generators with various tools, including cheap diagnostics and expensive verifiers. The tool-use decisions are typically governed by orchestrators that often use fixed rules and ignore uncertainty. We formulate orchestration as cost-sensitive sequential hypothesis testing: a Bayesian controller maintains a belief over candidate correctness and dynamically decides whether to gather more evidence, refine the candidate, verify it, or stop. Across six generators and nine coding benchmarks, Bayesian control proves to be most valuable when verification is costly and critics are informative but imperfect. Beyond control, the belief state yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.
Primary: Google DeepMind
All Institutions: Google DeepMind
This paper has significant broader impact for the design and deployment of robust and efficient LLM-based agents. By introducing a principled, uncertainty-aware control mechanism, it moves beyond heuristic orchestration, paving the way for more reliable and cost-effective AI systems. The ability to dynamically adapt decisions based on evidence and cost considerations is crucial for real-world applications where resources (e.g., API calls, human review) are limited and errors are costly. The belief state's superior uncertainty quantification capability can lead to more trustworthy AI systems, allowing users to better understand the confidence level of an agent's output. This framework could be extended to other domains beyond coding, such as scientific discovery, complex problem-solving, or even general-purpose autonomous agents, fostering the development of more intelligent and adaptive AI. This paper introduces a novel Bayesian control framework for LLM coding agents, formulating orchestration as cost-sensitive sequential hypothesis testing to dynamically manage tool use and uncertainty. The methodology, grounded in decision theory, significantly outperforms heuristic baselines in terms of cost-efficiency and success rate, especially when verification is expensive, and provides a superior correctness score for uncertainty quantification, marking a substantial step towards more robust and principled LLM agent design.
The paper proposes a principled Bayesian control framework for orchestrating LLM-based coding agents, framing the problem as cost-sensitive sequential hypothesis testing. This is a significant departure from the prevalent heuristic-based orchestrators. The core of the methodology lies in maintaining a belief state—a probability distribution over the true correctness of the generated code—which is dynamically updated using Bayes' rule based on observations from various tools (diagnostics, verifiers). The decision policy is derived from a partially observable Markov decision process (POMDP) formulation, aiming to minimize expected costs associated with refinement, verification, and incorrect stopping. To make the POMDP tractable, the authors introduce practical simplifications, such as a fixed maximum number of refinement steps, allowing for a finite-horizon dynamic programming approach. The critic models (diagnostics and verifier) are characterized by their likelihoods, which are learned or estimated. A notable strength is the dual utility of the belief state: it not only guides optimal decision-making but also provides an interpretable correctness score for uncertainty quantification. The methodology is theoretically sound, drawing from established decision theory, and provides a robust, uncertainty-aware mechanism for agent control.
The experimental evaluation is comprehensive and rigorous. The authors test their Bayesian control framework across a diverse set of six LLM generators (including GPT-3.5, GPT-4, Gemini 1.0 Pro, and open-source models like CodeLlama and StarCoder) and nine coding benchmarks (HumanEval, MBPP, and APPS at various difficulty levels). This broad coverage demonstrates the generalizability of the approach. Baselines include several fixed-rule orchestrators (e.g., "Always Refine," "Refine until pass," "Verify immediately") and uncertainty quantification methods (token probability, raw tool success). The results clearly show that Bayesian control consistently outperforms fixed-rule baselines, particularly when verification is costly and diagnostic critics are informative but imperfect. The value proposition of Bayesian control is shown to increase significantly with higher verification costs. Furthermore, the belief state's correctness score demonstrates superior performance in uncertainty quantification, achieving higher AUC scores than token probability and raw tool success in predicting code correctness. The experiments effectively validate the core hypotheses and highlight the conditions under which Bayesian control is most beneficial.
The paper provides a detailed appendix outlining the experimental setup, including specific LLM models, benchmarks, critic configurations, and hyper-parameters used for the Bayesian controller. This level of detail is commendable and greatly aids in understanding the experimental procedure. However, the paper states, "Our code and data are available at [anonymized for review]," indicating that the code is not publicly accessible at the time of review. While the detailed methodology and experimental setup provide a strong basis, the lack of publicly available code and data slightly hinders immediate, independent reproducibility. Should the code be released, the paper's reproducibility would be excellent.
The authors acknowledge several important limitations. The performance of the Bayesian controller is heavily dependent on the quality and accurate modeling of the critic likelihoods. If critics are unreliable or their characteristics are poorly estimated, the belief state and subsequent decisions may be suboptimal. The full POMDP formulation is computationally intractable, necessitating simplifications like a fixed maximum number of refinement steps, which might not always be optimal. The current framework assumes fixed costs for actions, which may not hold in dynamic real-world scenarios. The action space is also limited to "refine," "verify," and "stop," without considering more complex actions like re-planning or seeking human assistance. Finally, the work focuses specifically on coding agents, and its generalization to other LLM agent domains requires further investigation.
This paper has significant broader impact for the design and deployment of robust and efficient LLM-based agents. By introducing a principled, uncertainty-aware control mechanism, it moves beyond heuristic orchestration, paving the way for more reliable and cost-effective AI systems. The ability to dynamically adapt decisions based on evidence and cost considerations is crucial for real-world applications where resources (e.g., API calls, human review) are limited and errors are costly. The belief state's superior uncertainty quantification capability can lead to more trustworthy AI systems, allowing users to better understand the confidence level of an agent's output. This framework could be extended to other domains beyond coding, such as scientific discovery, complex problem-solving, or even general-purpose autonomous agents, fostering the development of more intelligent and adaptive AI. This paper introduces a novel Bayesian control framework for LLM coding agents, formulating orchestration as cost-sensitive sequential hypothesis testing to dynamically manage tool use and uncertainty. The methodology, grounded in decision theory, significantly outperforms heuristic baselines in terms of cost-efficiency and success rate, especially when verification is expensive, and provides a superior correctness score for uncertainty quantification, marking a substantial step towards more robust and principled LLM agent design.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs. These methods ignore the different functionalities of attention heads, leading to the eviction of critical tokens and thus degrading the performance of LLMs. To address this issue, we propose CompressKV, a resource-efficient KV-cache compression framework for GQA-based LLMs. Instead of aggregating attention scores from all heads, CompressKV identifies Semantic Retrieval Heads (SRHs) that capture both the initial and final tokens of a prompt and semantically important mid-context evidence, and uses them to select tokens whose KV pairs should be retained. Furthermore, CompressKV allocates cache budgets across layers according to offline estimates of layer-wise eviction error. Experiments on LongBench and Needle-in-a-Haystack show that CompressKV consistently outperforms existing KV-cache eviction methods across memory budgets. Notably, it preserves over 97\% of full-cache performance using only 3\% of the KV cache on LongBench question-answering tasks and achieves 90\% accuracy with just 0.7\% KV storage on Needle-in-a-Haystack. These results demonstrate an improved resource--performance trade-off for long-context LLM inference. Our code is publicly available at: https://github.com/TUDa-HWAI/CompressKV
Primary: Technical University of Darmstadt
All Institutions: Technical University of Darmstadt, Darmstadt, Germany; University of Notre Dame, Notre Dame, IN, USA; Technical University of Ilmenau, Ilmenau, Germany
CompressKV makes a significant contribution towards enabling more resource-efficient and sustainable deployment of long-context LLMs, particularly on memory-constrained hardware. By substantially reducing the KV-cache memory footprint while preserving high performance, it can facilitate wider adoption of advanced LLMs in edge devices, mobile applications, or large-scale inference clusters where memory is a critical bottleneck. The principled approach to identifying semantically important tokens and allocating cache budgets could inspire further research into fine-grained attention head functionalities and more accurate error modeling for compression. Its demonstrated compatibility and additive benefits with other efficiency techniques suggest it can be a foundational component in a multi-faceted approach to LLM optimization, contributing to the overall goal of making powerful LLMs more accessible and cost-effective. CompressKV introduces a novel KV-cache compression framework for GQA-based LLMs, leveraging Semantic Retrieval Heads for robust token selection and an offline error-aware mechanism for layer-adaptive budget allocation, significantly improving the resource-performance trade-off for long-context inference. This paper presents a highly effective and well-validated approach to a critical problem in large language model inference, offering a principled and practical solution to the memory footprint of KV caches. The core innovations, including the span-aggregation-based Semantic Retrieval Heads and the offline Frobenius norm error-aware layer allocation, are well-motivated and address clear limitations of prior work. The experimental validation is exceptionally thorough, demonstrating consistent and significant performance gains across multiple LLMs and benchmarks, especially under tight memory constraints, and proving orthogonality with other efficiency techniques. This work has strong practical implications for deploying long-context LLMs more efficiently and sustainably.
CompressKV proposes a two-fold framework for KV-cache compression in GQA-based LLMs. The first key component is the identification and utilization of Semantic Retrieval Heads (SRHs) for token selection. Unlike prior methods that often aggregate attention scores across all heads or rely on strict top-k attention hits (Traditional Retrieval Heads), SRHs are identified by aggregating attention mass over the *entire answer span* during correct answer generation on a calibration dataset. This novel span-aggregation approach allows SRHs to capture broader semantic context, effectively mitigating the "streaming head dominance" issue where critical mid-context tokens might be evicted. The selected SRHs then guide the importance scoring for tokens to be retained. The second component is an error-aware layer-adaptive cache allocation strategy. Instead of using online attention statistics, CompressKV quantifies the compression error for each layer by computing the Frobenius norm of the difference between attention-block outputs with full and compressed caches. This error estimation is performed *offline*, which is a significant practical advantage as it introduces no additional runtime overhead during inference. The total cache budget is then distributed proportionally to these precomputed layer-wise error scores, with practical minimum and maximum allocation constraints. The methodology is well-motivated, directly addresses identified limitations of existing methods, and offers a principled, efficient, and practical approach to KV-cache management.
The experimental evaluation is exceptionally comprehensive and robust. CompressKV is rigorously benchmarked against six strong, state-of-the-art KV-cache eviction baselines (StreamingLLM, SnapKV, PyramidKV, CAKE, HeadKV, AdaKV). The evaluation spans multiple GQA-based LLMs, including Llama-3.1-8B, Mistral-7B, Qwen2.5-14B, and Qwen2.5-32B, demonstrating broad applicability. Performance is assessed on two crucial long-context benchmarks: LongBench (covering diverse tasks like QA, summarization, few-shot learning) and Needle-in-a-Haystack (focused on retrieval accuracy). The results consistently show CompressKV's superior performance across models and memory budgets, with particularly impressive gains under tight memory constraints. For instance, it preserves over 97% of full-cache performance using only 3% of the KV cache on LongBench and achieves 90% accuracy with just 0.7% KV storage on Needle-in-a-Haystack. Extensive ablation studies confirm the individual contributions and complementary nature of SRH-driven token selection and error-aware layer-adaptive allocation. A causal ablation further highlights the critical role of SRHs compared to Traditional Retrieval Heads. Crucially, the paper also demonstrates CompressKV's orthogonality and additive benefits when combined with other efficiency techniques such as prefilling acceleration, KV-cache quantization, and head-level allocation, underscoring its potential as a general improvement. Memory and latency measurements further validate the practical benefits, showing stable decoding latency and reduced peak memory at long contexts.
The paper explicitly states that the code is publicly available at `https://github.com/TUDa-HWAI/CompressKV`, which is a strong indicator of reproducibility. Key implementation details are provided, including the use of FlashAttention-2, greedy decoding, specific local attention parameters (window_size=8, kernel_size=5), the number of selected SRHs per layer (top four), and the min/max budget constraints for layer allocation (m=32, M=3*B_per-layer). The offline nature of SRH identification and error-aware allocation, along with the mention of a calibration dataset (following prior work and provided in their codebase), further aids reproducibility by clearly defining the precomputation steps.
One potential limitation is the reliance on a calibration dataset with ground-truth answers for the identification of Semantic Retrieval Heads. While the paper states this data is provided and follows prior work, it implies that applying CompressKV to entirely new tasks or models without such a dataset might require an initial data collection and calibration step, which could be an overhead for certain niche applications. The method is specifically designed for GQA-based LLMs, and its direct applicability or performance on other attention mechanisms (e.g., MQA, MHA) is not explicitly discussed or evaluated. Although the offline computation is a strength for efficiency, it means the SRH identification and layer budgets are fixed and do not adapt dynamically to specific input prompts or changing task characteristics during inference, which might be a trade-off for ultimate adaptability.
CompressKV makes a significant contribution towards enabling more resource-efficient and sustainable deployment of long-context LLMs, particularly on memory-constrained hardware. By substantially reducing the KV-cache memory footprint while preserving high performance, it can facilitate wider adoption of advanced LLMs in edge devices, mobile applications, or large-scale inference clusters where memory is a critical bottleneck. The principled approach to identifying semantically important tokens and allocating cache budgets could inspire further research into fine-grained attention head functionalities and more accurate error modeling for compression. Its demonstrated compatibility and additive benefits with other efficiency techniques suggest it can be a foundational component in a multi-faceted approach to LLM optimization, contributing to the overall goal of making powerful LLMs more accessible and cost-effective. CompressKV introduces a novel KV-cache compression framework for GQA-based LLMs, leveraging Semantic Retrieval Heads for robust token selection and an offline error-aware mechanism for layer-adaptive budget allocation, significantly improving the resource-performance trade-off for long-context inference. This paper presents a highly effective and well-validated approach to a critical problem in large language model inference, offering a principled and practical solution to the memory footprint of KV caches. The core innovations, including the span-aggregation-based Semantic Retrieval Heads and the offline Frobenius norm error-aware layer allocation, are well-motivated and address clear limitations of prior work. The experimental validation is exceptionally thorough, demonstrating consistent and significant performance gains across multiple LLMs and benchmarks, especially under tight memory constraints, and proving orthogonality with other efficiency techniques. This work has strong practical implications for deploying long-context LLMs more efficiently and sustainably.
Accurate user modeling often depends on rich interaction histories, which are unavailable for billions of low-activity users. Large Language Models (LLMs) can infer latent user states from static profiles, but this reasoning becomes unreliable when profiles are sparse, and applying an LLM to billions of users is prohibitively expensive. We present ScaleToT, which learns structured reasoning from a small LLM-processed subset and extends it to the broader low-activity user population. To improve reasoning reliability, ScaleToT constructs typed user-state chains with a bounded entropy-guided Tree-of-Thought (ToT) refinement procedure. To make this structured reasoning usable from sparse profiles, the teacher-curated chains are used to train a student model on static profiles through supervised fine-tuning (SFT) and Outcome-Driven Segment-Aware Implicit Reward Policy Optimization (OSIPO). ScaleToT then transfers the student's reasoning representations to a lightweight profile encoder, providing shared reasoning signals for the remaining users without LLM inference. We evaluate ScaleToT on lifetime value (LTV) prediction in a billion-scale advertising deployment. A randomized online A/B test increased LT30 by 6.738\%, while offline reasoning covered only 7.32\% of the potential population, greatly reducing compute cost compared with full-population reasoning.
Primary: Kuaishou Technology
All Institutions: Kuaishou Technology
ScaleToT presents a robust industrial solution for scaling LLM-based structured reasoning to billion-scale low-activity user modeling, achieving significant online gains through a novel combination of entropy-guided ToT refinement, segment-aware RL distillation, and vector-quantized reasoning transfer.
The paper proposes ScaleToT, a framework for low-activity user modeling that bridges the gap between expensive LLM reasoning and scalable inference. The core methodological innovation lies in the "Bounded Typed Tree-of-Thought" (ToT) construction, which uses entropy-guided refinement to create structured, typed user-state chains from sparse profiles using privileged context during training. This is followed by a distillation phase where a student model learns to generate these chains via Supervised Fine-Tuning (SFT) and a novel Outcome-Driven Segment-Aware Implicit Reward Policy Optimization (OSIPO). Finally, the reasoning representations are transferred to the full population using Vector Quantization (VQ) and a profile-conditioned gate, allowing inference without LLM calls. The approach is technically sound, addressing specific industrial constraints (sparsity, cost) with a multi-stage pipeline that combines structured reasoning, RL-based alignment, and representation learning.
The evaluation is conducted on a billion-scale industrial dataset for Lifetime Value (LTV) prediction. The paper reports a 6.738% increase in LT30 (cumulative active days) in a randomized online A/B test, which is a significant and practically meaningful metric for an advertising platform. Offline metrics (Ranking AUC) also show improvements over baselines like Direct LLM, Free-Form CoT, and Sequential CoT. The ablation studies effectively isolate the contributions of the entropy-guided refinement and the OSIPO reward signal. The scalability analysis demonstrates that high performance can be maintained with reasoning coverage of only ~7.32% of the population, validating the efficiency claims.
The paper provides detailed descriptions of the model architectures, hyperparameters (learning rates, batch sizes, codebook sizes), and the specific LLM backbones used (Qwen3 series). The algorithms for entropy-guided refinement and reasoning transfer are formally defined. However, as is common with industrial papers, the exact dataset statistics and proprietary features are anonymized, which may limit exact replication. The code is not publicly available.
The method relies heavily on the assumption that latent user states can be represented by a finite set of typed fields, which may not hold for all user modeling tasks. The "privileged context" used during training (post-return feedback) is not available at inference, creating a distribution shift that the student must learn to approximate from sparse profiles alone; while the results are good, this is an inherent limitation of the cold-start setting. The VQ retrieval mechanism, while efficient, introduces a quantization error that might discard nuanced reasoning patterns.
This work has significant implications for the deployment of LLMs in large-scale recommendation and advertising systems. By demonstrating how to distill structured reasoning into lightweight, scalable models, it provides a blueprint for applying complex LLM capabilities to billion-user populations where direct inference is infeasible. It highlights the value of structured, interpretable reasoning in user modeling, potentially shifting the field away from black-box sequence modeling towards more explicit state inference for cold-start users. ScaleToT presents a robust industrial solution for scaling LLM-based structured reasoning to billion-scale low-activity user modeling, achieving significant online gains through a novel combination of entropy-guided ToT refinement, segment-aware RL distillation, and vector-quantized reasoning transfer.
Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.
Primary: Stanford University
All Institutions: Stanford University
This paper has significant broader impact. It uncovers a critical and widespread reliability flaw in current MLLMs, which has profound implications for their trustworthiness and deployment in sensitive applications. By proposing "cross-ordering flip rate" as a standard reporting axis, the paper directly influences future MLLM evaluation benchmarks and development practices, encouraging researchers and practitioners to explicitly consider and mitigate order sensitivity. The findings also redirect research efforts, highlighting the need for deeper architectural or training-based solutions rather than relying solely on prompt engineering. Ultimately, Facet-Probe provides a valuable tool and a new perspective for building more robust, transparent, and accountable multimodal AI systems. This paper introduces Facet-Probe, a rigorous, multi-faceted audit framework, to reveal that all 18 frontier MLLMs tested exhibit significant order sensitivity, proposing cross-ordering flip rate as a new standard for MLLM evaluation. The work provides a crucial evaluation methodology and surprising empirical findings that expose a fundamental reliability issue in current MLLMs, motivating a paradigm shift in how these models are developed and benchmarked for robustness.
The paper introduces Facet-Probe, a highly systematic and comprehensive framework for auditing order sensitivity in Multimodal Large Language Models (MLLMs). The methodology is robust, defining five distinct facets of ordering: option, evidence-chunk, document-rank, image-set, and mixed-modality ordering. This multi-faceted approach ensures a broad investigation into various types of input permutations relevant to MLLMs. A key strength is the use of a Bayesian item-response model, which rigorously separates true ordering noise from per-facet bias, adding significant statistical rigor to the analysis. Furthermore, the inclusion of a same-ordering control is crucial; it establishes a decoder-stochastic floor, allowing the researchers to differentiate between inherent model stochasticity and genuine order-induced flips. This methodological design is sound, innovative in its comprehensive application to MLLMs, and provides a strong foundation for reliable findings.
The experimental evaluation is extensive and impactful. The audit covers 18 frontier and open-weight MLLMs, providing a broad and representative sample of current models. The findings are striking and highly significant: none of the audited MLLMs are order-invariant, with screened per-facet panel-mean flip rates spanning a substantial 24-50%. The Gemini same-ordering control, conducted at temperature 0, empirically demonstrates a substantial ordering excess over the decoder-noise floor, confirming that the observed flips are indeed due to order sensitivity rather than mere stochasticity. The experiments also reveal that increased model capability does not eliminate flips, with even the best model flipping on 13.4% of trials, indicating a fundamental architectural or training issue. Finally, the mitigation tests show that training-free prompt changes are modality-conditional and do not transfer effectively between text and visual reasoning, suggesting that prompt engineering alone is insufficient for general order robustness. The experiments are well-designed, thorough, and yield critical insights that challenge current assumptions about MLLM reliability.
The paper explicitly supports reproducibility by providing a GitHub repository link (`https://github.com/yahskapar/facet-probe`) for the Facet-Probe audit artifacts. The abstract and section titles (e.g., "irt_methodology", "extended_dataset_details") suggest that the methodology and dataset details are thoroughly described within the full paper and supplementary materials. This commitment to open-sourcing the audit framework and data is excellent for enabling future research and verification of results.
A primary limitation highlighted by the authors themselves is that prompt-level mitigation alone is unlikely to provide general order robustness. This suggests that while the paper effectively diagnoses the problem and evaluates simple fixes, it does not offer a definitive solution, instead motivating future work on more fundamental training-time and architectural approaches. While the five facets cover a broad range, the specific datasets and tasks used for the audit might not encompass every possible real-world scenario or interaction type for MLLMs, potentially limiting the generalizability to highly niche applications.
This paper has significant broader impact. It uncovers a critical and widespread reliability flaw in current MLLMs, which has profound implications for their trustworthiness and deployment in sensitive applications. By proposing "cross-ordering flip rate" as a standard reporting axis, the paper directly influences future MLLM evaluation benchmarks and development practices, encouraging researchers and practitioners to explicitly consider and mitigate order sensitivity. The findings also redirect research efforts, highlighting the need for deeper architectural or training-based solutions rather than relying solely on prompt engineering. Ultimately, Facet-Probe provides a valuable tool and a new perspective for building more robust, transparent, and accountable multimodal AI systems. This paper introduces Facet-Probe, a rigorous, multi-faceted audit framework, to reveal that all 18 frontier MLLMs tested exhibit significant order sensitivity, proposing cross-ordering flip rate as a new standard for MLLM evaluation. The work provides a crucial evaluation methodology and surprising empirical findings that expose a fundamental reliability issue in current MLLMs, motivating a paradigm shift in how these models are developed and benchmarked for robustness.
Commercial large language models bill, scale latency, and budget context per token. Yet tokenizers assign more subword tokens to the same meaning in some languages than in others, so speakers of languages with high token-fertility pay a structural penalty before a model is ever invoked. This penalty is documented for multilingual settings in general, but it has not been measured systematically for African languages at the level of enterprise deployment economics and cognitive context capacity. We measure it across 20 African languages spanning five language families and three scripts (Latin, Ge'ez/Ethiopic, N'Ko; 19 appear in the primary FLORES-200+ corpus, with Nigerian Pidgin measured via MAFAND-MT only), using parallel corpora so that the language effect is isolated from content. Across 11 frontier and open tokenizers on FLORES-200+, every African language carries a tokenization premium above English (median 1.88x on GPT-5 / o200k_base, up to 8.92x for N'Ko); the penalty is largest for Ethiopic and N'Ko scripts (reaching 7-9x) and is near-invariant across corpora (FLORES vs SIB-200 Pearson r = 0.9998). Translated into deployment terms, this results in up to 8.9x inference cost and an equivalent generation-latency multiplier (N'Ko vs English on GPT-5; 7.4x for Amharic), and as little as 11% of English's effective context window. The best currently available tokenizer for African languages, Gemma 4, reduces the mean premium from 3.31x (cl100k_base) to 2.38x, but no tokenizer eliminates the penalty. We release an open measurement tool (afri-fertility), a public leaderboard, a results dataset, and mitigation guidance for African builders. The penalty falls hardest on the languages whose speakers can least afford it, a digital divide encoded directly into the subword vocabulary.
Primary: DataLens Africa Research
All Institutions: DataLens Africa Research, CipherSense AI
This paper has profound broader impact, particularly for AI equity and the digital divide. 1. **AI Equity and Fairness:** It rigorously quantifies a structural unfairness baked into LLMs, where speakers of African languages pay more and get less utility (shorter context, longer latency) for the same service. This highlights a critical dimension of AI fairness beyond accuracy. 2. **Economic Viability:** The translation of the "tax" into concrete enterprise costs directly impacts the economic viability of deploying LLM-powered applications in African markets, where compute resources are often least affordable. This can hinder innovation and access to AI technologies. 3. **Call to Action for Developers:** The findings, especially the success of Gemma 4 for Ethiopic, provide clear evidence that tokenizer design and vocabulary inclusion can significantly mitigate the penalty. This serves as a strong call to action for LLM developers to explicitly consider low-resource and non-Latin script languages in tokenizer training. 4. **Empowerment for African Builders:** The release of the `afri-fertility` tool, leaderboard, dataset, and mitigation guidance empowers African builders to measure the penalty themselves, make informed decisions about tokenizer/model choice, and advocate for more equitable LLM infrastructure. 5. **Research Agenda:** The paper establishes a foundational benchmark and framework for future research on multilingual tokenization, fairness, and cost optimization, particularly for underrepresented languages. It links the pre-inference cost layer to existing accuracy benchmarks, encouraging a holistic view of LLM performance. This work is a crucial step towards making LLMs truly global and equitable. The paper quantifies the "African Language Tax," a structural penalty where African languages incur significantly higher tokenization costs, latency, and reduced context window efficiency in frontier LLMs due to tokenizer design. This comprehensive study, using parallel corpora across 20 African languages and 11 tokenizers, reveals a median 1.88x premium over English (up to 8.92x for N'Ko), translating into substantial economic burdens and operational limitations for African builders, and provides an open measurement tool and mitigation guidance.
The methodology is exceptionally robust and well-designed to isolate and quantify the "African Language Tax." The core strength lies in the use of parallel corpora, which ensures that differences in token counts are attributed solely to the language and tokenizer, not content variations. The definition of metrics (Fertility, Premium, CPT, BPT, Context Efficiency) is clear and appropriate. The aggregation method ("sum-then-divide") correctly handles corpus-level metrics, avoiding biases from short sentences, and the inclusion of bootstrap confidence intervals demonstrates statistical rigor. A significant methodological contribution is the enterprise cost model, which translates abstract tokenization premiums into tangible economic terms (USD, local currency, latency, context erosion). This model is instantiated with realistic deployment scenarios (high-volume chat, output-heavy generation, context-constrained advisory), making the impact concrete for decision-makers. The "Economic Sensitivity" analysis, which accounts for the compounding effect of FX volatility on USD-denominated API pricing, is a particularly insightful and novel aspect of the cost model, directly addressing a critical real-world challenge for African builders. The `afri-fertility` tool itself is a methodological artifact, designed for determinism, reproducibility (caching, run manifest, `reproduce` command), and extensibility, which is a strong point. The inclusion of script-level controls and the consideration of normalization forms for non-Latin scripts further demonstrate careful methodological planning.
The experimental evaluation is comprehensive and meticulously executed. The study covers 20 African languages spanning five language families and three scripts (Latin, Ge'ez/Ethiopic, N'Ko), providing a diverse and representative sample. The inclusion of dual-script languages (Hausa Latin/Ajami, Bambara Latin/N'Ko) is a clever design choice to isolate the script effect. Eleven frontier and open tokenizers are tested, including commercially dominant ones (OpenAI's o200k_base, Llama, Gemma, Mistral, Qwen, DeepSeek) and multilingual baselines (BLOOM, Aya), as well as opaque API-based tokenizers (Claude, Gemini) for spot checks. This broad coverage ensures the findings are relevant to current LLM deployment. Three parallel corpora (FLORES-200+, SIB-200, MAFAND-MT) are used, with FLORES-200+ as the primary, providing robustness checks across different text registers. The results are striking and clearly presented: 1. **Universal Premium (H1 confirmed):** Every African language in the study carries a tokenization premium above English (median 1.88x on o200k_base, up to 8.92x for N'Ko), with the lowest observed premium still 1.29x. 2. **Dominant Script Effect (H2 confirmed):** Non-Latin scripts incur significantly higher penalties (Ethiopic mean 7.08x, N'Ko 8.92x on o200k_base) compared to Latin-script African languages (mean 1.76x). 3. **Tokenizer Performance:** Gemma 4 is identified as a standout for Ethiopic languages, reducing the premium from 7-9x to ~2.65x, demonstrating that targeted vocabulary improvements can significantly mitigate the penalty. Qwen 3 also shows a notable reduction for N'Ko. 4. **Economic Impact:** The cost model translates these premiums into substantial annual inference costs (e.g., N'Ko on GPT-5 costs up to $1.6M/year vs. $183k for English), equivalent generation latency multipliers, and severe context window erosion (N'Ko having only 11% of English's effective context). 5. **FX Compounding:** The paper effectively illustrates how FX depreciation further compounds the tokenization tax for African builders, leading to even higher effective costs in local currency. The experimental results are empirically sound, statistically supported, and translated into highly actionable insights for both LLM developers and African deployers.
Reproducibility is a major strength of this paper. The authors release `afri-fertility`, an open-source measurement tool (Apache-2.0 license) that performs all measurements deterministically. Key features ensuring reproducibility include: * **Determinism:** Tokenization is deterministic, and the only randomness (bootstrap CIs) is seeded. * **Caching:** Counts are cached on disk, keyed by content and tokenizer version, ensuring consistent results across re-runs. * **Run Manifest:** Every run generates a manifest detailing tool version, tokenizer versions, price/FX snapshots, config hash, and segmentation method, allowing precise traceability. * **Locked Study Config:** The entire study configuration is provided as a YAML file. * **`afri-fertility reproduce` command:** A simple command is provided to run a small offline reference suite for quick verification. * **Open Artifacts:** Beyond the tool, a public leaderboard and results dataset are released. This commitment to open science and reproducibility is exemplary and significantly enhances the paper's impact and trustworthiness.
The authors acknowledge several limitations: 1. **UAX-29 Word Segmentation:** The standard UAX-29 word segmentation, while applied uniformly, is imperfect for highly agglutinative languages (e.g., Kinyarwanda, isiXhosa) and Ethiopic script, where word boundaries may not align cleanly. The authors mitigate this by reporting character- and byte-normalized metrics (CPT, BPT) alongside fertility, ensuring conclusions don't solely rely on word counts. 2. **Opaque Tokenizers:** Claude and Gemini are included as count-only API checks, meaning their subword segmentation cannot be inspected, limiting deeper analysis of their internal mechanisms. 3. **Corpus Dependence:** While multiple corpora are used, the primary reliance on FLORES-200+ (a professionally translated, general-domain corpus) means the findings might vary slightly for highly specialized or informal text registers not covered. However, the robustness checks with SIB-200 and MAFAND-MT show near-invariance of rankings. 4. **Snapshot Nature:** The cost and FX rates are based on specific snapshots (June 2026), meaning the absolute monetary figures will change over time. However, the *relative* premiums and the *mechanism* of FX compounding remain valid.
This paper has profound broader impact, particularly for AI equity and the digital divide. 1. **AI Equity and Fairness:** It rigorously quantifies a structural unfairness baked into LLMs, where speakers of African languages pay more and get less utility (shorter context, longer latency) for the same service. This highlights a critical dimension of AI fairness beyond accuracy. 2. **Economic Viability:** The translation of the "tax" into concrete enterprise costs directly impacts the economic viability of deploying LLM-powered applications in African markets, where compute resources are often least affordable. This can hinder innovation and access to AI technologies. 3. **Call to Action for Developers:** The findings, especially the success of Gemma 4 for Ethiopic, provide clear evidence that tokenizer design and vocabulary inclusion can significantly mitigate the penalty. This serves as a strong call to action for LLM developers to explicitly consider low-resource and non-Latin script languages in tokenizer training. 4. **Empowerment for African Builders:** The release of the `afri-fertility` tool, leaderboard, dataset, and mitigation guidance empowers African builders to measure the penalty themselves, make informed decisions about tokenizer/model choice, and advocate for more equitable LLM infrastructure. 5. **Research Agenda:** The paper establishes a foundational benchmark and framework for future research on multilingual tokenization, fairness, and cost optimization, particularly for underrepresented languages. It links the pre-inference cost layer to existing accuracy benchmarks, encouraging a holistic view of LLM performance. This work is a crucial step towards making LLMs truly global and equitable. The paper quantifies the "African Language Tax," a structural penalty where African languages incur significantly higher tokenization costs, latency, and reduced context window efficiency in frontier LLMs due to tokenizer design. This comprehensive study, using parallel corpora across 20 African languages and 11 tokenizers, reveals a median 1.88x premium over English (up to 8.92x for N'Ko), translating into substantial economic burdens and operational limitations for African builders, and provides an open measurement tool and mitigation guidance.
Vision-language-action (VLA) models have made rapid progress in generalist robot manipulation by harnessing semantic knowledge from pretrained vision-language backbones, but their visual tokens remain grounded in 2D image coordinates rather than the calibrated geometry of the robot's cameras -- a mismatch especially pronounced in multi-camera setups, where views are coupled by known intrinsics and extrinsics yet processed as independent images. We propose G$^3$VLA, a camera-aware geometric module that injects calibrated structure into the visual-token stream of a pretrained VLA without altering its action space or imitation objective, combining intrinsic-conditioned ray embeddings, projective positional encoding (PRoPE), and bidirectional cross-view fusion. Geometric supervision is provided either from ground-truth point maps when available, or from confidence-gated $π^3$X teacher predictions, requiring no depth sensors or manual annotations. Instantiated on $π_0$, G$^3$VLA yields consistent gains across the LIBERO suites, RoboCasa24, RoboTwin2.0, and real-robot settings, with the largest improvements on spatially and object-sensitive tasks. We further validate on $π_{0.5}$ and GR00T 1.5, with results suggesting that geometric transfer is most effective when geometry-aware tokens have direct access to the action generation pathway. Our project page is at https://sites.google.com/view/g3vla
Primary: unknown
All Institutions: unknown
G$^3$VLA represents a significant advancement towards making generalist robot manipulation more robust and precise, particularly in multi-camera environments. By injecting calibrated geometric inductive biases into existing VLA models without requiring architectural overhauls or explicit 3D sensor inputs, it provides a lightweight and practical pathway for improving spatial reasoning and out-of-distribution generalization. This approach could accelerate the deployment of VLAs in real-world settings where precise manipulation and adaptability to varying viewpoints are crucial. The method's compatibility with pretrained backbones means it can readily benefit from ongoing advancements in large vision-language models. The insights into architectural dependencies also guide future research in designing VLA models that can better leverage geometric information. This work contributes to bridging the gap between high-level semantic understanding and low-level spatial precision in robot learning. This paper introduces G$^3$VLA, a camera-aware geometric module that enhances pretrained Vision-Language-Action (VLA) models by injecting calibrated camera geometry into their visual-token stream without altering their core architecture or action space. The work presents a novel combination of intrinsic-conditioned ray embeddings, projective positional encoding (PRoPE), and bidirectional cross-view fusion, alongside a practical geometry distillation strategy from a $\pi^3$X teacher, to significantly improve spatial precision and out-of-distribution generalization in robot manipulation across diverse simulated and real-world benchmarks. The comprehensive experimental validation, including multi-backbone evaluation, extensive ablation studies, and crucial real-robot experiments demonstrating improved generalization under viewpoint shifts, firmly establishes G$^3$VLA as a valuable and practical advancement for the field of robot learning, offering a lightweight yet impactful solution to a critical limitation of current VLA systems.
The paper introduces G$^3$VLA, a camera-aware geometric module designed to inject calibrated structure into the visual-token stream of pretrained Vision-Language-Action (VLA) models. This addresses a crucial limitation where VLA models often process visual tokens grounded in 2D image coordinates, neglecting the known calibrated geometry of multi-camera setups. A key strength of G$^3$VLA is its "lightweight" and "backbone-preserving" nature, meaning it integrates with existing VLA architectures without altering their action space or imitation objective, making it highly compatible for practical adoption. The module comprises three main components: intrinsic-conditioned ray embeddings, which enrich each ViT patch token with its back-projected viewing direction; Projective Positional Encoding (PRoPE), which leverages camera intrinsics and extrinsics to provide a calibration-derived attention bias for cross-view projective relationships; and bidirectional cross-view fusion, which facilitates the exchange of geometric context across camera streams. This combination effectively imbues 2D visual tokens with essential 3D geometric awareness. For supervision, G$^3$VLA offers flexibility: it can use ground-truth point maps in simulation or, more practically, confidence-gated predictions from a $\pi^3$X teacher model, eliminating the need for depth sensors or manual 3D annotations. The training employs a two-stage curriculum: an initial pre-training phase for the geometric module with a dominant distillation loss, followed by full policy fine-tuning where the action loss takes precedence, with distillation serving as a regularizer. This staged approach is a well-considered strategy for effectively integrating a new module into a pretrained system.
The experimental evaluation is exceptionally comprehensive and rigorous, providing strong evidence for G$^3$VLA's effectiveness. The authors validate the method across three architecturally distinct VLA backbones ($\pi_0$, $\pi_{0.5}$, and GR00T 1.5), demonstrating broad generalizability. Performance is assessed on an extensive suite of simulation benchmarks, including the LIBERO suites (Goal, Spatial, Object, and 10), RoboCasa24, and RoboTwin2.0. Results consistently show significant gains, particularly on spatially and object-sensitive tasks within LIBERO, directly supporting the paper's core hypothesis. For instance, on $\pi_0$, G$^3$VLA (GT) improves LIBERO's macro-average success rate by +3.5 points, with even larger improvements on Object (+5.0) and Spatial (+4.0) tasks. The evaluation on $\pi_{0.5}$ confirms compatibility with stronger baselines, yielding consistent, albeit smaller, improvements even when the baseline is near saturation. An insightful finding emerges from GR00T 1.5, where mixed gains suggest that the effectiveness of geometric injection depends on how directly geometry-aware tokens access the action generation pathway, highlighting an important architectural consideration for future VLA designs. Crucially, the paper includes real-robot experiments on two manipulation tasks (Pick-and-Place Test Tube, Pouring Nut) using a bimanual UR5 setup. These experiments demonstrate substantial improvements in out-of-distribution (OOD) generalization under viewpoint shifts, a critical capability for robust robot deployment. For example, on the pouring task, OOD performance for $\pi_0$ improved from 70.8-75.0% to 83.3-87.5%. Thorough ablation studies confirm the individual contributions of ray embeddings, PRoPE, and the two-stage training curriculum. The comparison between ground-truth and $\pi^3$X distillation shows that while GT provides the strongest signal, $\pi^3$X distillation recovers most of the gains, making it a practical alternative. The identified failure case of $\pi^3$X in visually clean synthetic scenes (RoboTwin2.0) also provides valuable insight into the teacher model's limitations.
The paper provides a clear and detailed description of the G$^3$VLA module's architecture and the two-stage training process. It explicitly states that implementation details, camera-geometry preprocessing, teacher-target generation, and backbone-specific training hyperparameters are provided in the Appendix, which is excellent practice for reproducibility. The use of established benchmarks and publicly available VLA backbones (like $\pi_0$, $\pi_{0.5}$, GR00T 1.5) further aids in replicating the results. The inclusion of a project page URL also suggests that code and/or additional resources might be available. Given the level of detail in the main paper and the promise of comprehensive appendices, the work appears to be highly reproducible.
The authors thoughtfully discuss several limitations. G$^3$VLA relies on accurate camera intrinsics and extrinsics, making it sensitive to calibration drift, synchronization errors, and train-test mismatches. The dependence on a visual geometry teacher ($\pi^3$X) means its targets can be imperfect under challenging visual conditions such as occlusion, specularities, blur, or weak-prior viewpoints, even with confidence gating. The architectural dependence is another key limitation, as evidenced by the attenuated gains on GR00T 1.5, suggesting that the benefits are maximized when geometry-aware tokens have direct access to the action generation pathway. The method focuses solely on enhancing the visual-token representation, leaving other potential failure modes (e.g., in the action space, limited demonstrations, or weak language-action grounding) unaddressed. Finally, the teacher caches and auxiliary-head training add offline computational cost, although they are not needed at deployment.
G$^3$VLA represents a significant advancement towards making generalist robot manipulation more robust and precise, particularly in multi-camera environments. By injecting calibrated geometric inductive biases into existing VLA models without requiring architectural overhauls or explicit 3D sensor inputs, it provides a lightweight and practical pathway for improving spatial reasoning and out-of-distribution generalization. This approach could accelerate the deployment of VLAs in real-world settings where precise manipulation and adaptability to varying viewpoints are crucial. The method's compatibility with pretrained backbones means it can readily benefit from ongoing advancements in large vision-language models. The insights into architectural dependencies also guide future research in designing VLA models that can better leverage geometric information. This work contributes to bridging the gap between high-level semantic understanding and low-level spatial precision in robot learning. This paper introduces G$^3$VLA, a camera-aware geometric module that enhances pretrained Vision-Language-Action (VLA) models by injecting calibrated camera geometry into their visual-token stream without altering their core architecture or action space. The work presents a novel combination of intrinsic-conditioned ray embeddings, projective positional encoding (PRoPE), and bidirectional cross-view fusion, alongside a practical geometry distillation strategy from a $\pi^3$X teacher, to significantly improve spatial precision and out-of-distribution generalization in robot manipulation across diverse simulated and real-world benchmarks. The comprehensive experimental validation, including multi-backbone evaluation, extensive ablation studies, and crucial real-robot experiments demonstrating improved generalization under viewpoint shifts, firmly establishes G$^3$VLA as a valuable and practical advancement for the field of robot learning, offering a lightweight yet impactful solution to a critical limitation of current VLA systems.