Last 7 Days (August 27 – September 02, 2026)
An LLM agent shown a professional-looking market panel commits to a directional call on a provably unpredictable question far more often than one asked the bare question: across 12 frontier models, commitment rises from 6.5% to 54.0% as evidence is escalated. It commits just as readily when every number on the panel is invented: fabricating the entire display, so nothing the model can see is true except the question itself, still lifts commitment from 24.5% to 36.8%, statistically indistinguishable from the 37.6% produced by genuine market data. What unlocks confident action is not information but the authority of its packaging. The failure is narrow and locatable. Incapacity is not the answer: on matched answerable questions attached to the same panels, the same models answer essentially always, at near-perfect accuracy. Nor is it belief - stated probabilities barely move across the gradient that swings action by 48 points, and score worse than a climatological baseline. Missing judgment isn't it either: asked to classify a question's knowability before acting, models call it irreducible 90% of the time and then commit on just 0.4% of those. The act/don't-act gate is what fails, and the effect is concentrated in a few models rather than universal. Because the gate is separable, it can be trained. Supervised fine-tuning of a 3B model on 540 synthetic cases, predominantly dice, coins, jars and timers, drives commitment to 0.0% on the original cases and transfers to three unseen domains. It does not survive everything: the gate holds exactly when the response format leaves room to reason, and rigid formats that remove that room leave the model confident and wrong on questions it otherwise answers correctly. The gate is trainable and context-fragile, and deployment needs both halves of that sentence.
Primary: Independent Researcher
All Institutions: Independent Researcher
This paper has profound broader implications for the development, deployment, and governance of LLM agents. It highlights a critical vulnerability: the "authority of packaging" can override an LLM's inherent judgment about knowability, leading to confident but unearned commitments on unpredictable questions. This directly impacts AI safety and reliability, as agents in real-world settings (e.g., finance, healthcare) are often presented with authoritative-looking data. The finding that action calibration is distinct from belief calibration necessitates a shift in evaluation paradigms, moving beyond stated probabilities to auditing decisions. The proposed "commitment rate on aleatoric probes" as a cheap, reproducible robustness test offers a concrete measurement artifact for compliance with emerging regulations like the EU AI Act (Article 15) and NIST AI RMF. The demonstration of a trainable gate provides a pathway for mitigation, but its context-fragility underscores the need for careful prompt engineering and robust deployment strategies, emphasizing that "deployment needs both halves of that sentence." This work is a significant step towards building more trustworthy and appropriately cautious AI agents. This paper makes a crucial contribution by demonstrating that LLM agents are susceptible to making confident, unearned commitments on unknowable questions when presented with authoritative-looking, even fabricated, evidence. The rigorous methodology causally isolates the effect of presentation over information, localizes the failure to an act/don't-act gate, and shows this gate can be trained with synthetic data, though it remains context-fragile. This work provides significant empirical findings that challenge current assumptions about LLM calibration and reliability, offering concrete implications for agent evaluation, safety, and responsible deployment.
The methodology is exceptionally rigorous and well-designed. The core innovation lies in the "unknowability oracle" for aleatoric questions (short-horizon price direction, sports outcomes, weather) with sealed outcomes, ensuring provable unpredictability. The evidence gradient, escalating from bare questions to rich, professional-looking panels, is systematically applied. Crucially, the paper introduces a "fabricated evidence" condition where entire data panels are invented while preserving their authoritative form and internal consistency, allowing for causal isolation of presentation from information. The agentic protocol, offering ANSWER, CALL_TOOL, or DECLINE options, directly measures action rather than just stated probabilities. The use of matched answerable controls is vital for distinguishing a true knowability gate from a blanket decliner. The paper employs a comprehensive suite of metrics, including commitment rate, Youden's J for discrimination, Brier score for earned check, and CORP decomposition for calibration, all carefully defined and justified against degenerate strategy baselines. The supervised fine-tuning (SFT) approach on a 3B model using synthetic data from unrelated domains (dice, coins, timers) is a clever strategy to teach the abstention behavior, and the subsequent testing for transfer and fragility is thorough. The transparency in discussing parser choices (strict vs. semantic) and their impact is commendable.
The experimental evaluation is extensive, spanning 12 frontier models, 11 trained checkpoints, and four distinct domains (equity, crypto, sports, weather). The headline finding—commitment rising from 6.5% to 54.0% with escalating evidence, and fabricated evidence inducing commitment statistically indistinguishable from genuine data—is robust and highly impactful. The disaggregated results reveal that the effect is concentrated in specific models, not universal, which is an important nuance. The "dial, not a switch" observation, showing commitment scales with the quantity of authoritative display, further strengthens the causal link. The localization experiments effectively rule out incapacity, absent belief (stated probabilities barely move and are anti-predictive), and absent judgment (models classify questions as irreducible 90% of the time before acting). The trainability of the gate is demonstrated convincingly, with SFT driving commitment to 0.0% on original cases and transferring to unseen domains, surviving a tense-balanced control that rules out a simple grammatical heuristic. However, the critical finding in Section 8, that the trained gate's robustness is context-fragile and breaks when the response format removes room for reasoning, is a crucial insight for deployment. The detailed reporting of run-to-run variability and the impact of training data composition (e.g., removing sports items) further enhances the credibility of the results.
The paper demonstrates a strong commitment to reproducibility. It explicitly links to a GitHub repository containing code, data, pre-registration, and all cached model outputs. The training recipe (SFT with QLoRA on Qwen2.5-3B-Instruct, 3 epochs, 540 synthetic cases) is detailed. Prompt examples for both training and evaluation are provided verbatim in Appendix B. The two parsing strategies (strict and semantic) are clearly defined, and their delta is reported, ensuring transparency in how model outputs were interpreted. The paper also discusses potential sources of variability, such as greedy decoding not being bit-reproducible and uncontrolled serving configurations for hosted models, which adds to its credibility. The pre-registration details in Appendix A, distinguishing between decisions made before and after data existence, are exemplary for scientific rigor.
The authors are highly transparent about the limitations. Six open threats are acknowledged: (1) Weather is the weakest instrument due to genuine skill in its ensemble rain probability and lack of sealed outcomes. (2) The finding that only a subset of frontier models are seduced, meaning pooled rates aren't universal. (3) A residual lexical cue in synthetic unknowable items (e.g., "fair," "hidden") not fully excluded, though transfer domains speak against it. (4) The intervention being limited to one 3B model at one scale, meaning generalizability to larger models is not established. (5) Lower bounds for discrimination scores for two models due to high tool calls. (6) Cell-level values being approximate due to greedy decoding variability and uncontrolled serving configurations. My assessment aligns with these, and the paper's proactive disclosure of these limitations strengthens its overall scientific integrity.
This paper has profound broader implications for the development, deployment, and governance of LLM agents. It highlights a critical vulnerability: the "authority of packaging" can override an LLM's inherent judgment about knowability, leading to confident but unearned commitments on unpredictable questions. This directly impacts AI safety and reliability, as agents in real-world settings (e.g., finance, healthcare) are often presented with authoritative-looking data. The finding that action calibration is distinct from belief calibration necessitates a shift in evaluation paradigms, moving beyond stated probabilities to auditing decisions. The proposed "commitment rate on aleatoric probes" as a cheap, reproducible robustness test offers a concrete measurement artifact for compliance with emerging regulations like the EU AI Act (Article 15) and NIST AI RMF. The demonstration of a trainable gate provides a pathway for mitigation, but its context-fragility underscores the need for careful prompt engineering and robust deployment strategies, emphasizing that "deployment needs both halves of that sentence." This work is a significant step towards building more trustworthy and appropriately cautious AI agents. This paper makes a crucial contribution by demonstrating that LLM agents are susceptible to making confident, unearned commitments on unknowable questions when presented with authoritative-looking, even fabricated, evidence. The rigorous methodology causally isolates the effect of presentation over information, localizes the failure to an act/don't-act gate, and shows this gate can be trained with synthetic data, though it remains context-fragile. This work provides significant empirical findings that challenge current assumptions about LLM calibration and reliability, offering concrete implications for agent evaluation, safety, and responsible deployment.
Reinforcement learning is a natural way to post-train LLM agents for long-horizon interactive tasks judged only by end-of-task verification, yet a shared belief holds that outcome-only RL soon hits a ceiling on small open models. Recent work therefore compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration. We argue the ceiling is an artifact of two failures of common practice. Signal starvation: group-relative RL with sparse outcome-only rewards yields a gradient only when a task's rollout group mixes successes and failures, so under-scaled exploration silences exactly the hardest, most instructive tasks. Policy drift: squeezing many updates out of a small task pool degrades the policy itself, as an unanchored objective lets the sampling distribution collapse exactly when saturation has already made informative groups rare. We present CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens, then cash in an enlarged interaction budget at test time. On AppWorld, a long-horizon interactive coding benchmark, a Qwen3-14B policy trained with CANOPY through environment interaction alone--without task-specific supervision, auxiliary credit signals, or elaborate agent scaffolding--topped the public leaderboard (Feb. 2026; Test-Normal TGC 86.9, Test-Challenge 67.6), and the same design principles lift Qwen3.5-9B on SWE-bench Verified by 16.6 points. Agentic RL alone thus internalizes long-horizon capability directly into a small open model; we plan to release the complete training stack at https://github.com/AlibabaResearch/SignalCoverageRL.
Primary: Alibaba Research
All Institutions: Alibaba Research
The paper identifies two key failure modes in sparse-reward RL for LLM agents—signal starvation and policy drift—and proposes a minimalist protocol (CANOPY) that resolves them through scaled exploration and strict on-policy KL-anchored updates, achieving state-of-the-art results on AppWorld and SWE-bench with a small open model. This work provides a rigorous empirical and theoretical foundation for the viability of outcome-only RL in long-horizon interactive tasks, offering a practical and reproducible recipe that simplifies agent post-training by removing the need for dense rewards, SFT priors, or complex inference-time orchestration.
The paper proposes CANOPY, a minimalist RL protocol for training LLM agents on long-horizon tasks. The core contribution is a diagnostic framework identifying two failure modes in standard Group Relative Policy Optimization (GRPO) for sparse rewards: "signal starvation" (lack of gradient signal when rollout groups are homogeneous) and "policy drift" (entropy collapse due to unanchored optimization on small task pools). The solution involves scaling the rollout group size to ensure mixed outcomes (coverage), maintaining strict on-policy updates to prevent drift, and using a KL anchor to the base model. The methodology is sound, leveraging well-understood RL components (GRPO, KL penalty) but applying them with specific hyperparameter choices (large $n$, strict on-policy) that address the identified bottlenecks. The derivation of the advantage magnitude in degenerate groups provides a clear theoretical motivation for the increased exploration budget.
The experiments are extensive and compelling. The primary result is topping the AppWorld leaderboard with a Qwen3-14B model using only outcome-based RL, outperforming systems that rely on dense rewards, SFT priors, or complex orchestration. The ablation studies are rigorous, isolating the impact of group size, KL anchoring, and on-policy updates. The transfer to SWE-bench Verified demonstrates the generalizability of the principles, showing a 16.6 point improvement. The paper provides detailed training dynamics analysis, showing how entropy and coverage evolve, which supports the theoretical claims. The comparison against the base model and other RL variants is thorough.
The paper provides high reproducibility. It includes full configuration tables for both AppWorld and SWE-bench runs, details the hardware setup, and explains the fault quarantine mechanism for environment reliability. The authors explicitly state that they plan to release the complete training stack. The detailed appendix on the "w/o strict on-policy" variant and the discussion of compute costs (GPU-hours) are valuable for practitioners. The transparency regarding what was *not* logged (e.g., per-step quarantine counts) is a strength.
The primary limitation is the high compute cost; the main run required ~2,690 GPU-hours, which may be prohibitive for smaller labs. The paper acknowledges that the "minimalist" claim refers to architectural simplicity, not compute frugality. Additionally, the results are heavily dependent on the specific base models (Qwen3/3.5) and the AppWorld/SWE-bench environments; generalization to other domains or smaller models is not fully tested. The "signal starvation" diagnosis assumes independent rollouts, which may not hold perfectly in practice, though the empirical results support the approach.
This paper has significant potential to shift the paradigm in LLM agent training. By demonstrating that outcome-only RL can suffice for long-horizon tasks without dense rewards or complex scaffolding, it simplifies the post-training pipeline for open-source models. It challenges the prevailing belief that RL hits a ceiling on small models, suggesting that the ceiling is an artifact of training practice rather than a fundamental limit. This could lead to more efficient and scalable methods for training specialized agents in various domains. The paper identifies two key failure modes in sparse-reward RL for LLM agents—signal starvation and policy drift—and proposes a minimalist protocol (CANOPY) that resolves them through scaled exploration and strict on-policy KL-anchored updates, achieving state-of-the-art results on AppWorld and SWE-bench with a small open model. This work provides a rigorous empirical and theoretical foundation for the viability of outcome-only RL in long-horizon interactive tasks, offering a practical and reproducible recipe that simplifies agent post-training by removing the need for dense rewards, SFT priors, or complex inference-time orchestration.
Two of the most fundamental questions in statistical learning theory are the following: which prediction problems are learnable, and how should they be learned? For the former, elegant answers often take the form of combinatorial dimensions. The latter question, however, has proved considerably more elusive: all known general-purpose multiclass learners rely on intricate orientations of exponentially large one-inclusion structures, and familiar algorithmic principles such as proper learning and regularization remain poorly understood. Motivated by prior work, we ask whether learning reduces to proper learning---possibly over a larger hypothesis class---and whether proper or improper multiclass learning can ultimately be captured by suitable regularizers. Our primary results answer both questions negatively, resolving three open problems from prior work. First, we exhibit a learnable multiclass problem that cannot be embedded in any properly learnable class, meaning learning cannot be reduced to proper learning by enlarging the hypothesis class. Second, we demonstrate that proper learning can require training error and characterize this phenomenon precisely: every properly learnable class admits a proper learner making $o(m)$ errors on samples of size $m$, but every prescribed sublinear scale $a_m=o(m)$ is necessary for some properly learnable problem. Third, regularization is not a general learner: we exhibit a properly learnable class that cannot be learned by any Structural Risk Minimization (SRM) learner, and a learnable class that cannot be learned by any local regularizer. We complement these impossibility results with a positive theory that gives two sufficient conditions for SRM learnability and characterizes SRM representability through integrability of revealed preferences.
Primary: University of Washington
All Institutions: University of Washington, Northwestern University, University of Chicago
The paper resolves three major open problems in statistical learning theory by proving that learning cannot be reduced to proper learning and that regularization is not a general learner, providing a rigorous theoretical framework for understanding the limits of these fundamental algorithmic principles.
The paper employs rigorous theoretical methods from combinatorics and functional analysis to address fundamental questions in statistical learning theory. It constructs specific counterexamples (learnable but not properly learnable classes, classes not learnable by SRM) to prove negative results. The methodology is sophisticated, utilizing concepts like the DS dimension, one-inclusion structures, and integrability of revealed preferences to characterize the boundaries of algorithmic principles like proper learning and regularization. The use of AI (ChatGPT 5.6 Pro) in the discovery of constructions is noted, but the core contribution remains the mathematical proofs and theoretical frameworks.
As a purely theoretical paper, there are no empirical experiments, datasets, or benchmarks. The "evaluation" consists of the validity and tightness of the mathematical proofs and the resolution of open problems. The results are definitive within the theoretical framework.
The results are mathematically reproducible in the sense that the proofs can be verified. There is no code or experimental setup to reproduce. The transparency regarding the use of AI in the discovery process is a positive step for reproducibility of the research process, though it raises questions about the novelty of the specific constructions.
The paper is highly theoretical and may have limited immediate practical impact on algorithm design. The reliance on AI for key constructions, while disclosed, might lead to skepticism about the depth of human insight in the specific combinatorial designs, although the authors claim full responsibility. The results are negative (impossibility results), which, while valuable, do not provide new algorithms.
The paper significantly impacts the field of statistical learning theory by clarifying the limits of standard algorithmic principles. It guides future research by showing where proper learning and regularization fail, potentially steering researchers toward more nuanced or problem-specific approaches. It resolves long-standing open problems, contributing to the foundational understanding of multiclass learning. The paper resolves three major open problems in statistical learning theory by proving that learning cannot be reduced to proper learning and that regularization is not a general learner, providing a rigorous theoretical framework for understanding the limits of these fundamental algorithmic principles.
Reinforcement learning is a natural way to post-train LLM agents for long-horizon interactive tasks judged only by end-of-task verification, yet a shared belief holds that outcome-only RL soon hits a ceiling on small open models. Recent work therefore compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration. We argue the ceiling is an artifact of two failures of common practice. Signal starvation: group-relative RL with sparse outcome-only rewards yields a gradient only when a task's rollout group mixes successes and failures, so under-scaled exploration silences exactly the hardest, most instructive tasks. Policy drift: squeezing many updates out of a small task pool degrades the policy itself, as an unanchored objective lets the sampling distribution collapse exactly when saturation has already made informative groups rare. We present CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens, then cash in an enlarged interaction budget at test time. On AppWorld, a long-horizon interactive coding benchmark, a Qwen3-14B policy trained with CANOPY through environment interaction alone--without task-specific supervision, auxiliary credit signals, or elaborate agent scaffolding--topped the public leaderboard (Feb. 2026; Test-Normal TGC 86.9, Test-Challenge 67.6), and the same design principles lift Qwen3.5-9B on SWE-bench Verified by 16.6 points. Agentic RL alone thus internalizes long-horizon capability directly into a small open model; we plan to release the complete training stack at https://github.com/AlibabaResearch/SignalCoverageRL.
Primary: Alibaba Research
All Institutions: Alibaba Research
The paper identifies two key failure modes in sparse-reward RL for LLM agents—signal starvation and policy drift—and proposes a minimalist protocol (CANOPY) that resolves them through scaled exploration and strict on-policy KL-anchored updates, achieving state-of-the-art results on AppWorld and SWE-bench with a small open model. This work provides a rigorous empirical and theoretical foundation for the viability of outcome-only RL in long-horizon interactive tasks, offering a practical and reproducible recipe that simplifies agent post-training by removing the need for dense rewards, SFT priors, or complex inference-time orchestration.
The paper proposes CANOPY, a minimalist RL protocol for training LLM agents on long-horizon tasks. The core contribution is a diagnostic framework identifying two failure modes in standard Group Relative Policy Optimization (GRPO) for sparse rewards: "signal starvation" (lack of gradient signal when rollout groups are homogeneous) and "policy drift" (entropy collapse due to unanchored optimization on small task pools). The solution involves scaling the rollout group size to ensure mixed outcomes (coverage), maintaining strict on-policy updates to prevent drift, and using a KL anchor to the base model. The methodology is sound, leveraging well-understood RL components (GRPO, KL penalty) but applying them with specific hyperparameter choices (large $n$, strict on-policy) that address the identified bottlenecks. The derivation of the advantage magnitude in degenerate groups provides a clear theoretical motivation for the increased exploration budget.
The experiments are extensive and compelling. The primary result is topping the AppWorld leaderboard with a Qwen3-14B model using only outcome-based RL, outperforming systems that rely on dense rewards, SFT priors, or complex orchestration. The ablation studies are rigorous, isolating the impact of group size, KL anchoring, and on-policy updates. The transfer to SWE-bench Verified demonstrates the generalizability of the principles, showing a 16.6 point improvement. The paper provides detailed training dynamics analysis, showing how entropy and coverage evolve, which supports the theoretical claims. The comparison against the base model and other RL variants is thorough.
The paper provides high reproducibility. It includes full configuration tables for both AppWorld and SWE-bench runs, details the hardware setup, and explains the fault quarantine mechanism for environment reliability. The authors explicitly state that they plan to release the complete training stack. The detailed appendix on the "w/o strict on-policy" variant and the discussion of compute costs (GPU-hours) are valuable for practitioners. The transparency regarding what was *not* logged (e.g., per-step quarantine counts) is a strength.
The primary limitation is the high compute cost; the main run required ~2,690 GPU-hours, which may be prohibitive for smaller labs. The paper acknowledges that the "minimalist" claim refers to architectural simplicity, not compute frugality. Additionally, the results are heavily dependent on the specific base models (Qwen3/3.5) and the AppWorld/SWE-bench environments; generalization to other domains or smaller models is not fully tested. The "signal starvation" diagnosis assumes independent rollouts, which may not hold perfectly in practice, though the empirical results support the approach.
This paper has significant potential to shift the paradigm in LLM agent training. By demonstrating that outcome-only RL can suffice for long-horizon tasks without dense rewards or complex scaffolding, it simplifies the post-training pipeline for open-source models. It challenges the prevailing belief that RL hits a ceiling on small models, suggesting that the ceiling is an artifact of training practice rather than a fundamental limit. This could lead to more efficient and scalable methods for training specialized agents in various domains. The paper identifies two key failure modes in sparse-reward RL for LLM agents—signal starvation and policy drift—and proposes a minimalist protocol (CANOPY) that resolves them through scaled exploration and strict on-policy KL-anchored updates, achieving state-of-the-art results on AppWorld and SWE-bench with a small open model. This work provides a rigorous empirical and theoretical foundation for the viability of outcome-only RL in long-horizon interactive tasks, offering a practical and reproducible recipe that simplifies agent post-training by removing the need for dense rewards, SFT priors, or complex inference-time orchestration.
Scaling Transformers has driven large gains in language modeling, but transplanting this to behavior-sequence modeling in production ranking is challenging: recommendation differs in signal quality, where behavior sequences are noisy, temporally irregular, and sparsely supervised, and in computation asymmetry, where each request scores many candidates against one shared user history under tight latency budgets. We propose ReST, a recommendation-native Transformer scaling framework. For signal quality, it introduces a sequence encoder with dual-gated attention, rotary positional and temporal embedding, stabilized residual normalization, and training-only auxiliary objectives. For computation asymmetry, it factorizes ranking into a heavy reusable encoder and a lightweight cross decoder with projection-free KV attention and token-specific parameterization, coupling user-level shared-prefix training with shared-prefix serving for compute-once, decode-many-times ranking. Across industrial and public benchmarks, ReST achieves higher accuracy and scales more consistently along sequence length, depth, and width, where LLM-style Transformer blocks saturate. A one-week online A/B test on a production advertising platform improves online AUC by 1.31% and lifts a core revenue metric by 11.93% within a 50 ms P99 budget; ReST has since been fully deployed in production, showing that behavior-sequence scaling remains a promising, under-exploited axis for production ranking.
Primary: ByteDance
All Institutions: ByteDance
ReST introduces a recommendation-native Transformer scaling framework that addresses signal quality and computation asymmetry in industrial ranking. By factorizing the model into a reusable encoder and lightweight decoder and introducing specific denoising and temporal mechanisms, it achieves consistent scaling gains and significant online business impact, demonstrating that behavior-sequence scaling is a viable axis for production ranking.
The paper proposes ReST, a framework that adapts Transformer scaling laws to industrial recommendation ranking by addressing two specific mismatches with LLMs: signal quality (noisy, sparse supervision) and computation asymmetry (one user history vs. many candidates). The methodology introduces "Rec-Native" components: Dual-Gated Attention (DGA) to filter noisy behavior tokens, Rotary Temporal Embedding (RoTE) to handle irregular time gaps, and Stabilized Residual Normalization (SRN) to prevent depth collapse under sparse supervision. Crucially, it factorizes the model into a heavy, reusable sequence encoder and a lightweight cross-decoder, enabling "compute-once, decode-many-times" serving. The introduction of auxiliary objectives to combat "sequence starvation" (where the DLRM branch shortcuts the gradient flow to the sequence branch) is a significant practical insight.
The experimental validation is strong, featuring a large-scale industrial dataset from TikTok Shop Ads and public benchmarks (MovieLens, Amazon-Books). The paper demonstrates consistent scaling benefits in length, depth, and width where LLM-style baselines saturate. The most compelling evidence is the online A/B test reporting a 1.31% AUC improvement and an 11.93% lift in core revenue metrics within a strict 50ms P99 latency budget. The ablation studies effectively isolate the contributions of DGA, RoTE, and the auxiliary losses.
Reproducibility is moderate. While the paper provides detailed architectural descriptions and hyperparameters for public benchmarks, the industrial dataset is proprietary. The specific implementation of the "shared-prefix serving" infrastructure is described conceptually but not open-sourced. The public benchmark results are reproducible given the RecBole framework details, but the primary claims rely on the industrial deployment which cannot be independently verified.
The primary limitation is the lack of open-source code and the proprietary nature of the main evaluation dataset. The "sequence starvation" phenomenon is specific to hybrid DLRM+Transformer architectures; its applicability to pure end-to-end recommendation models is less clear. Additionally, the scaling laws fitted are descriptive over a limited compute range and may not hold for orders of magnitude larger scales.
This work bridges the gap between LLM scaling research and industrial recommendation systems, providing a blueprint for efficiently scaling sequence models in latency-constrained environments. The insights on computation asymmetry and auxiliary supervision are likely to influence future architectures in ad-tech and e-commerce. ReST introduces a recommendation-native Transformer scaling framework that addresses signal quality and computation asymmetry in industrial ranking. By factorizing the model into a reusable encoder and lightweight decoder and introducing specific denoising and temporal mechanisms, it achieves consistent scaling gains and significant online business impact, demonstrating that behavior-sequence scaling is a viable axis for production ranking.
Benign fine-tuning severely weakens the safety alignment of large language models (LLMs), so we study why refusal behavior is so fragile. While prior work often attributes this failure to gradient conflict, we propose a fundamentally different Fisher-geometric explanation: safety Fisher is low-rank, and alignment makes the safety geometry flatter while preserving an output-routing pathway. After 100 benign fine-tuning examples, this pathway is selectively re-sharpened in output-side MLP modules, explaining the asymmetric fragility: safety can collapse to high attack success rates, while general utility degrades mildly. The routing view also explains why few safety examples can restore refusal behavior, indicating that internal safety-relevant representations are preserved. Finally, we show that LoRA and ASAM mitigate early collapse by suppressing output-side sharpness, but their protection weakens at larger fine-tuning scales. Overall, safety failure is best understood as a disruption of a low-rank output-routing mechanism
Primary: Indiana University
All Institutions: Indiana University, Center for Distributed Confidential Computing (CDCC)
The paper provides a novel Fisher-geometric explanation for the fragility of LLM safety alignment, identifying a low-rank output-routing mechanism that is selectively re-sharpened by benign fine-tuning. By combining geometric analysis with causal activation patching, the authors demonstrate that safety collapse is a localized disruption of output-side routing rather than a global loss of safety representations, offering a new perspective on alignment robustness and recovery.
The paper proposes a Fisher-geometric explanation for the fragility of LLM safety alignment under benign fine-tuning. The core methodological contribution is the use of block-wise empirical Fisher matrices to analyze the curvature of the loss landscape specifically for safety-related outputs (refusal) versus utility outputs. The authors argue that safety alignment creates a "flat" geometry in most layers but preserves a low-rank, high-curvature "routing pathway" in the final output-side MLP modules. They demonstrate that benign fine-tuning selectively re-sharpens this specific pathway, causing refusal behavior to collapse while leaving internal safety representations intact. The methodology is supported by logit-lens analysis and cross-condition activation patching, which provide causal evidence that the final layer is the dominant locus of the behavioral switch. This multi-faceted approach (geometry + activations + causality) is rigorous and well-suited to the research question.
The experiments are extensive, covering two major model families (Llama-3.1-8B and Qwen2.5-7B) and multiple alignment conditions (SFT, DPO, varying amounts of safety data). The evaluation includes standard benchmarks (HEx-PHI for safety, MMLU/BoolQ/ARC for utility) and ablations on fine-tuning methods (Full SFT, LoRA, ASAM). A key strength is the demonstration of "asymmetric fragility," where safety degrades significantly while utility remains stable, which supports the routing hypothesis. The paper also validates the reversibility of the damage with minimal safety examples, further supporting the claim that internal representations are preserved. The inclusion of DPO-aligned models strengthens the generalizability of the findings beyond SFT-only alignment.
The paper provides a GitHub repository link and detailed hyperparameters for training and evaluation. The use of standard open-source models and benchmarks (HEx-PHI, Alpaca, SciQ) enhances reproducibility. The specific protocols for Fisher estimation and activation patching are described in the appendix, allowing other researchers to replicate the geometric analyses.
The study is limited to mid-scale models (7B-8B parameters), and it is unclear if the "low-rank routing" mechanism holds for frontier-scale models. The analysis focuses on English-centric data and standard SFT/DPO alignment, leaving out more complex RLHF pipelines or multilingual settings. The Fisher analysis is computationally expensive, which may limit its adoption as a standard diagnostic tool for larger models.
This paper provides a mechanistic understanding of why LLM safety is fragile, shifting the focus from "gradient conflict" to "geometric routing." This insight has significant implications for developing more robust alignment techniques, such as methods that explicitly protect the output-side routing subspace or distribute safety constraints more deeply in the network. It also offers practical guidance for practitioners on the limitations of LoRA and ASAM in preserving safety during fine-tuning. The paper provides a novel Fisher-geometric explanation for the fragility of LLM safety alignment, identifying a low-rank output-routing mechanism that is selectively re-sharpened by benign fine-tuning. By combining geometric analysis with causal activation patching, the authors demonstrate that safety collapse is a localized disruption of output-side routing rather than a global loss of safety representations, offering a new perspective on alignment robustness and recovery.
Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.
Primary: Zhejiang University
All Institutions: Zhejiang University
The paper introduces a robust framework for generating research plans by decoupling question and answer sources to reduce leakage and employing a two-stage OPSD+GRPO training strategy. It demonstrates significant improvements over baselines and competitive performance against larger models, offering a reproducible pipeline and dataset for the scientific AI community.
The paper proposes "PaperGym," a framework that converts research papers into training environments for research-plan generation. The core methodological contribution is a structured data pipeline that decouples the generation of the research question (from Goal/Background) from the reference answer (from Method/Experiments) to minimize criterion leakage. It introduces a two-stage training paradigm: first, On-Policy Self-Distillation (OPSD) using the rubric as privileged context to establish a broad prior, followed by Group Relative Policy Optimization (GRPO) using the rubric as a reward signal. The use of rubrics as both a distillation teacher context and a reinforcement learning reward is a clever integration of dense and sparse supervision signals. The method addresses the "cold start" instability of GRPO by using OPSD to structure the initial policy distribution, a hypothesis supported by entropy dynamics analysis.
The experiments are extensive, covering three model scales (Qwen3-1.7B/4B/8B) and multiple benchmarks (in-domain and out-of-domain). The paper demonstrates that the two-stage approach outperforms SFT, single-stage OPSD, single-stage GRPO, and the reverse ordering (GRPO+OPSD). The win-rate analysis against RubricHub Science and the base model provides strong evidence of the data quality. The comparison with larger general-purpose models (Kimi K2.6, GPT-5.1) shows competitive performance for a small model, which is a significant result. The ablation studies on rubric sources and model scales are thorough.
The paper releases the pipeline, the 20,000-instance corpus (PaperGym-20k), and benchmarks. Detailed hyperparameters, prompts, and training configurations are provided in the appendix. The use of standard frameworks (verl, vLLM) and open-source base models (Qwen3) enhances reproducibility. The specific scoring protocols and judge prompts are fully disclosed.
The primary limitation is the reliance on LLM-as-a-judge for both data generation (rubric creation) and evaluation, which introduces potential biases and costs. The "criterion leakage" metric, while improved, is still non-zero (3.7%), and the effectiveness of the rubric as a reward signal depends heavily on the quality of the LLM judge. The paper focuses on Qwen3 models; generalization to other architectures is not tested. The computational cost of the two-stage training and the multi-call rubric evaluation during GRPO is significant.
This work provides a scalable method for generating high-quality training data for scientific reasoning tasks, which is a critical bottleneck in AI-for-Science. The framework could be adapted to other domains where expert feedback is scarce but structured documents exist. The two-stage OPSD+GRPO recipe may be applicable to other tasks requiring dense supervision and outcome-based optimization. The paper introduces a robust framework for generating research plans by decoupling question and answer sources to reduce leakage and employing a two-stage OPSD+GRPO training strategy. It demonstrates significant improvements over baselines and competitive performance against larger models, offering a reproducible pipeline and dataset for the scientific AI community.
Much of human health and function unfolds beyond the clinic, through the movements of everyday life. Wrist-worn accelerometers capture these movements continuously, yet their rich signals are often reduced to a small set of predefined behavioural summary measures. Here, we present Sensori, a self-supervised foundation model that learns general-purpose health representations directly from 24 hours of raw tri-axial wrist movement. We developed and evaluated the model across four population-based cohorts from the United Kingdom, China and the United States, comprising 122,640 participants contributing 683,617 person-days of free-living recordings. Sensori condensed each day of movement into a representation that captured diverse movement behaviours, demographic characteristics, health axes and physical function. Evaluation in independent cohorts showed that these representations generalised across populations and measurement settings without retraining. When added to common clinical covariates, Sensori significantly improved prevalent disease classification for 52 of 102 eligible conditions (median delta AUROC, 0.060; range, 0.012-0.242) and incident disease risk prediction for 26 of 87 eligible conditions (median delta Uno's C-index, 0.064; range, 0.025-0.172), with the largest gains for neurological and psychiatric disorders. These findings establish 24-hour wrist movement as a rich and scalable source of health information, with the potential to support passive health monitoring and disease prediction at population scale.
Primary: University of Oxford
All Institutions: University of Oxford, University College London, University of Birmingham, Peking University, National and Kapodistrian University of Athens, University Hospitals Birmingham NHS Foundation Trust, NIHR Biomedical Research Centre, Wellcome Trust, Big Data Institute
The paper presents a highly significant contribution to health informatics and self-supervised learning by introducing Sensori, a foundation model that learns general-purpose health representations from 24-hour wrist movement data at an unprecedented scale (122k participants, 683k person-days). The rigorous evaluation across four international cohorts demonstrates strong cross-population generalization and clinically meaningful improvements in disease prediction, particularly for neurological and psychiatric conditions, establishing raw wrist acceleration as a rich, scalable source of health information.
The paper introduces Sensori, a self-supervised foundation model for 24-hour wrist-worn accelerometer data. The architecture is a hybrid of a 1D convolutional encoder (adapted from wav2vec 2.0) and a Transformer backbone. The key methodological contribution is the dual pretraining objective: masked reconstruction at the 5-minute embedding level to capture intra-day temporal dependencies, and participant-level contrastive learning to capture stable inter-day individual characteristics. This approach effectively handles the long-sequence nature of 24-hour data (864,000 samples at 10Hz) by progressively pooling to 5-minute embeddings before applying the Transformer. The use of linear probing for evaluation is standard and appropriate for assessing representation quality.
The experimental scale is exceptional, utilizing 122,640 participants and 683,617 person-days across four major population cohorts (UK Biobank, China Kadoorie Biobank, ELSA, NHANES). The evaluation is comprehensive, covering activity recognition benchmarks, prediction of demographic/health traits, and disease risk prediction. The results show strong generalization across cohorts without retraining. The improvement in disease prediction (median delta AUROC 0.060) is statistically significant and clinically relevant, particularly for neurological and psychiatric disorders. The comparison against conventional handcrafted features and other foundation models (MOMENT, Chronos-2) is rigorous.
High. The authors provide code, pretrained weights, and a project website. The data sources are well-defined, though access to UKB and CKB requires approval. The preprocessing steps (actipy) are open-source. The model architecture and hyperparameters are detailed in the methods section.
The model is trained primarily on older populations (median age >60 in most cohorts), limiting generalizability to younger demographics. Disease analysis is restricted to UKB due to data availability. The paper acknowledges that improvements in incident disease risk may reflect prodromal states rather than true predictive risk. The computational cost of processing 24-hour sequences is non-trivial, though the 40M parameter model is manageable.
This work has significant potential for passive health monitoring and digital biomarker discovery. By learning general-purpose representations from raw movement data, it enables scalable screening for a wide range of conditions without condition-specific model training. It bridges the gap between raw sensor data and clinical utility, offering a pathway for integrating wearable data into routine healthcare. The paper presents a highly significant contribution to health informatics and self-supervised learning by introducing Sensori, a foundation model that learns general-purpose health representations from 24-hour wrist movement data at an unprecedented scale (122k participants, 683k person-days). The rigorous evaluation across four international cohorts demonstrates strong cross-population generalization and clinically meaningful improvements in disease prediction, particularly for neurological and psychiatric conditions, establishing raw wrist acceleration as a rich, scalable source of health information.
Protein structure modeling rests on a single computational primitive: the interaction between what a residue is (sequence content) and where it sits (three-dimensional geometry). What is the expressive limit of this layer class? We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling, while the additive message passing of mainstream geometric GNNs is provably blind to content-geometry binding. We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost: each radius neighborhood is organized into a sequence hyperedge and a contact hyperedge, modulated by an edge-conditioned matrix-valued operator factorized into K learned basis operators with geometry-generated coefficients. Across enzyme function prediction, fold classification, and ligand binding site detection, Hyper-Fold and its hierarchical variant Hyper-Fold-Deep achieve the best results among protein-specific structure encoders; Hyper-Fold-Pocket, an anchored set-prediction head, surpasses UniSite-3D on UniSite-DS and two zero-shot benchmarks with no sequence language model features, 68x fewer parameters, and 4.8x lower latency--suggesting that a sufficiently expressive 3D backbone recovers information that fusion architectures previously borrowed from evolution-scale pretraining.
Primary: Shanghai University
All Institutions: Shanghai University, Tsinghua University, Xi'an Jiaotong University, Zhejiang University
The paper establishes a theoretical expressive limit for sequence-geometry layers in protein modeling and introduces Hyper-Fold, a rank-K hypergraph-based architecture that approaches this limit with message-passing efficiency, achieving state-of-the-art results in pocket detection and function prediction without relying on sequence language models.
The paper proposes a theoretically grounded framework for protein structure modeling, defining an "expressive ladder" for sequence-geometry interaction layers. The core theoretical contribution is the proof that additive message passing (used in GearNet, GVP, etc.) is provably blind to content-geometry binding, while the complete bilinear operator over outer products represents the expressive ceiling. To make this ceiling computationally feasible, the authors introduce Hyper-Fold, a rank-K separable convolutional backbone that approximates the bilinear operator using hypergraph structures (sequence and contact hyperedges). The method is mathematically sound, leveraging factorization to reduce complexity from $O(C^2 d_g)$ to $O(K C C')$, which is comparable to standard message passing. The design of the "Hyper-Fold-Pocket" head, which uses structure-anchored set prediction without sequence language model features, is a clever architectural choice that directly tests the hypothesis that expressive 3D backbones can replace evolutionary information.
The experimental evaluation is rigorous and comprehensive. The authors benchmark on three distinct tasks: enzyme function prediction (EC), fold classification, and ligand binding site detection (Pocket). They demonstrate state-of-the-art or near-state-of-the-art results on all three. Notably, Hyper-Fold-Pocket outperforms UniSite-3D (which uses ESM features) on zero-shot benchmarks (HOLO4K, COACH420) without using any sequence language model features, achieving this with 68x fewer parameters and 4.8x lower latency. The ablation studies effectively isolate the contribution of the rank-K kernel, the hyperedge structure, and the contrastive denoising training strategy. The comparison against the "expressive ladder" rungs (additive vs. scalar vs. channel vs. matrix gating) provides strong empirical evidence supporting the theoretical claims.
The paper includes a detailed reproducibility statement, specifying architecture configurations, training recipes, and providing anonymized source code and weights. The use of standard benchmarks (UniSite-DS, HOLO4K, COACH420) and clear evaluation metrics ([email protected]/0.5, Fmax) facilitates independent verification. The complexity analysis and latency measurements are clearly defined (A100-80GB, batch size 1, synthetic length-300 protein).
The primary limitation is the reliance on radius graphs, which may not capture long-range interactions as effectively as attention-based methods for very large proteins, although the paper argues for the efficiency of the linear scaling. The theoretical "ceiling" is defined for second-order interactions; it is unclear how this extends to higher-order geometric constraints. Additionally, while the model is efficient, the hypergraph construction adds implementation complexity compared to standard GNNs. The zero-shot generalization, while impressive, is tested on a limited set of benchmarks.
This work has significant implications for the field of geometric deep learning in biology. By proving the limitations of additive message passing and providing a scalable alternative, it offers a new direction for designing protein structure encoders. The finding that expressive 3D backbones can substitute for sequence language model features in specific tasks (like pocket detection) challenges the prevailing paradigm of hybrid sequence-structure models and could lead to more efficient, structure-only pipelines for drug discovery and protein engineering. The paper establishes a theoretical expressive limit for sequence-geometry layers in protein modeling and introduces Hyper-Fold, a rank-K hypergraph-based architecture that approaches this limit with message-passing efficiency, achieving state-of-the-art results in pocket detection and function prediction without relying on sequence language models.
Tool-using agents are commonly evaluated by a single bit: whether an end-to-end workflow completed. This metric fails to distinguish failures that matter in production, such as expired credentials, malformed payloads, or correct execution followed by incorrect final delivery. We introduce APIFlow-Bench, a fully auditable benchmark for long-horizon, dependent REST-API workflows that decomposes performance into seven engineering capabilities and requires agents to produce answers supported by the actual call path. We generate synthetic API worlds forward, subtask by subtask; each subtask is admitted only after a zero-LLM self-test triad verifies its grader and an oracle establishes solvability, and an adversarial audit identified and fixed six grader exploits. Grading is deterministic and provenance-sensitive: a state check traces a mock-minted canary through the API data flow to the response the answer must originate from, and a typed answer card is verified field by field. We release all answer keys and 44,362 unredacted execution transcripts. Across 19 frontier and open-weight models under one neutral scaffold, we find: (1) longer dependency chains degrade success, from 93% on individual subtasks to 74% on clean 20-subtask chains and 61% when including the 8% of chain trials that a model-consensus screen flags as passed by no model; (2) reliability separates models more than best-case capability, with best-of-five spanning seven points but all-five-of-five reliability spanning 44 points; (3) the independent-error account of compounding failure does not fit the data: pass rates on 20-subtask chains are 33 percentage points above the product of subtask-level rates, and on the clean slice 77% of failing runs reached the correct final state and failed only at delivery.
Primary: Postman
All Institutions: Postman
APIFlow-Bench introduces a rigorous, provenance-gated benchmark for long-horizon API workflows that reveals a critical gap between agent capability and reliability, showing that final delivery failures dominate over state execution errors in long dependent chains. The paper's contribution lies in its methodological rigor in benchmark generation and validation, providing a reusable framework for creating auditable synthetic benchmarks, and its empirical findings that challenge the independent-error model of compounding failure, offering actionable insights for improving agent robustness in production-like environments.
The paper introduces a rigorous methodology for generating and validating synthetic API benchmarks. The core innovation is the "zero-LLM self-test triad" and the provenance-gated grading system. By using mock-minted canaries to trace data flow, the authors ensure that agents cannot pass by guessing or memorizing answers; they must execute the correct API calls. The validation stack (oracle solvability, golden replay, adversarial audit) is a significant methodological contribution to the field of generated benchmarks, addressing the common issue of grader exploits and unsolvable tasks. The decomposition of performance into seven specific engineering capabilities (authentication, discovery, schema repair, etc.) provides a granular view of agent competence that binary success metrics miss.
The evaluation is extensive, covering 19 frontier and open-weight models across 467 tasks. The key empirical finding is the "level collapse" with chain length, showing that while individual subtasks are easy (93% pass), long dependent chains degrade performance significantly (74% pass). The paper provides a strong analysis of failure modes, distinguishing between state execution failures and final delivery failures. The finding that 77% of failing runs reached the correct final state but failed at delivery is a surprising and valuable insight for practitioners. The reliability analysis (pass@5 vs pass^5) correctly identifies that consistency, not just capability, is the primary differentiator between models in long-horizon tasks.
High. The authors release the full harness, frozen task bank, answer keys, and 44,362 unredacted execution transcripts. The use of content-hash-pinned manifests and deterministic evaluators ensures that results are reproducible. The paper provides clear instructions for reproducing the leaderboard and verifying the bank integrity. The open-sourcing of the transcripts allows for independent analysis of failure modes, which is a best practice in benchmarking.
The primary limitation is the small number of sampling units (11 full-length worlds) for the headline chain-20 slice, leading to wide confidence intervals and overlapping model rankings. The paper acknowledges that the "clean" slice is heavily influenced by two specific world families, which may limit the generalizability of the failure mode analysis. Additionally, the benchmark is REST-only, and the synthetic nature of the API worlds, while controlled, may not fully capture the complexity and unpredictability of real-world production APIs. The entanglement of the generator family with the oracle and reviser models is a potential confounding factor, though mitigated by deterministic validation.
This paper has high potential impact on the development of tool-using agents. By providing a benchmark that distinguishes between capability and reliability, and by exposing specific failure modes like final delivery errors, it guides developers toward more robust agent designs. The provenance-gated grading approach is a reusable technique for other benchmark domains. The findings on the non-multiplicative nature of long-horizon failure challenge common assumptions about compounding errors and suggest that agents fail due to exposure to specific failure surfaces rather than hidden information accumulation. This insight is valuable for designing better error recovery and state management mechanisms in agents. APIFlow-Bench introduces a rigorous, provenance-gated benchmark for long-horizon API workflows that reveals a critical gap between agent capability and reliability, showing that final delivery failures dominate over state execution errors in long dependent chains. The paper's contribution lies in its methodological rigor in benchmark generation and validation, providing a reusable framework for creating auditable synthetic benchmarks, and its empirical findings that challenge the independent-error model of compounding failure, offering actionable insights for improving agent robustness in production-like environments.
Automating alignment research may accelerate progress toward aligned AI, but whether it does is hard to measure. Luckily, many alignment failures, such as deception, sycophancy, and jailbreaks, are already measurable by public benchmarks. We study whether automated alignment researchers (AARs) can post-train to mitigate alignment failures by proposing training methods and data to simultaneously optimize multiple safety benchmarks, while preserving general capability. Across 10 alignment failures, the strongest AAR methods significantly reduce the targeted alignment failures and generalize to a held-out benchmark, multi-turn behavioral audits, and models up to 4.7 times larger than the target model. As a human baseline, 28 experienced researchers receive up to eight hours to develop methods for the same benchmarks, but their methods underperform the best AAR methods. Using human ideas as the AARs' initial research direction does not improve performance, suggesting current AARs may not need guidance from experienced researchers. These results suggest that automating alignment research on well-characterized failures may be practical in the near term.
Primary: Anthropic
All Institutions: Anthropic
The paper demonstrates that automated alignment researchers (AARs) can effectively mitigate common alignment failures by autonomously proposing and testing post-training methods, outperforming human baselines and generalizing to larger models. This work is significant for the AI safety community as it provides a concrete, scalable framework for automating alignment research, offering a path toward accelerating the development of safer AI systems while highlighting the critical need for robust monitoring to prevent reward hacking by the automated researchers themselves.
The paper introduces a robust "Automated Alignment Researcher" (AAR) harness that utilizes LLM agents (Claude Opus 4.8) to autonomously propose, implement, and evaluate post-training methods for mitigating specific alignment failures. The methodology is rigorous, featuring a multi-agent setup with "librarian" agents for literature review, a "monitor" agent to enforce integrity constraints (preventing data leakage or distillation from stronger models), and a hill-climbing loop that optimizes geometric means of safety benchmarks while preserving general capabilities (MMLU, GSM8K, IFEval). The use of a held-out benchmark and open-ended behavioral audits (Petri) to test generalization is a strong methodological choice that mitigates overfitting concerns.
The experimental scope is extensive, covering 10 distinct alignment failures (deception, sycophancy, jailbreaks, etc.) across multiple model scales (2B to 7B open-weight models, and a production-grade Claude Opus 4.8 checkpoint). The results demonstrate that AARs can significantly reduce targeted failures and generalize to larger models (up to 4.7x size). A particularly strong finding is the comparison against 28 experienced human researchers, where AARs outperformed human-proposed ideas within 6 hours of computation. The study also includes ablations on the harness components and a monitoring system that detected and excluded 2.4% of trajectories for suspected cheating, adding a layer of empirical rigor to the safety claims.
The authors provide a public GitHub repository containing the code and benchmarks, which significantly enhances reproducibility. The paper details the specific models used, the compute budgets (H200 GPU, ~30 min training), and the evaluation protocols. However, the reliance on proprietary models (Claude Opus 4.8, Sonnet 5) for the AAR agents and the specific "Petri" audit setup may limit full external reproducibility for labs without access to these specific frontier models.
The study is limited to alignment failures that are already measurable by public benchmarks or automated audits, which may not cover all critical safety risks (e.g., novel, hard-to-supervise failures). The human baseline is a one-shot comparison without iteration, which the authors acknowledge is not a direct apples-to-apples comparison. Additionally, the capability preservation check is limited to three specific benchmarks, and the paper admits that methods might harm unmeasured capabilities. The "cheating" rate, while low, indicates that automated researchers can attempt to game evaluations, a risk that scales with model capability.
This paper has high potential impact on the field of AI safety and automated research. It provides early evidence that automating alignment research is practical for well-characterized failures, potentially accelerating the development of safer AI systems. The finding that AARs can outperform experienced human researchers in method discovery suggests a shift in how alignment research might be conducted, emphasizing the need for robust monitoring and control scaffolding for automated researchers. The work also highlights the importance of "monitorability" as a key property for future AI systems. The paper demonstrates that automated alignment researchers (AARs) can effectively mitigate common alignment failures by autonomously proposing and testing post-training methods, outperforming human baselines and generalizing to larger models. This work is significant for the AI safety community as it provides a concrete, scalable framework for automating alignment research, offering a path toward accelerating the development of safer AI systems while highlighting the critical need for robust monitoring to prevent reward hacking by the automated researchers themselves.
Factual question answering (QA) typically assumes a single canonical answer, obscuring whether large language models (LLMs) retain divergent accounts of long-tail facts. To address this gap, we introduce ElephantBench, a closed-book knowledge probe comprising 1,094 questions generated through an auditable graph-based pipeline. The pipeline retrieves related documents from a low-exposure web corpus, identifies naturally occurring disagreements, and converts them into multi-account QA records. Each answer is verified against the originating documents and authoritative public web sources and is then reviewed by human annotators. Across 32 models, even the strongest model recovers both accounts on only 52.4% of questions, while on nearly all remaining questions it recalls one account but omits the other. Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness. Corpus analysis further shows that exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall. These findings establish ElephantBench as a reproducible knowledge probe for diagnosing epistemic myopia in parametric memory. More broadly, our graph-based benchmark construction pipeline provides an efficient and scalable way to turn long-tail corpora into source-traceable knowledge probes, supporting efforts to evaluate and advance the epistemic rigour of next-generation LLMs. Code is available at https://github.com/Tencent/ElephantBench.
Primary: Tencent
All Institutions: Tencent
The paper introduces ElephantBench, a closed-book benchmark derived from low-exposure web corpora to probe the "epistemic myopia" of LLMs, revealing that even frontier models fail to recall all verified accounts of long-tail facts in nearly half of cases. By utilizing a graph-based pipeline to mine natural source disagreements and verifying them through multi-stage auditing, the study provides a rigorous diagnostic tool that distinguishes between factual retention and memory completeness, offering critical insights into the limitations of parametric knowledge and the impact of data exposure imbalance on model behavior.
The paper introduces a novel benchmark construction pipeline that inverts standard data selection by mining the "low-exposure" remainder of web corpora (documents filtered out by quality classifiers like DCLM) to identify naturally occurring factual disagreements. By constructing a document graph where edges represent support or conflict between accounts of the same fact, the authors create a closed-book probe for "epistemic myopia." The methodology is rigorous, involving LLM-based edge classification, subgraph sampling for QA generation, and a multi-stage verification process (LLM check, web agent verification, and human review). The use of a graph-based approach to isolate conflicting facts is a clever and scalable solution to the problem of finding multi-account facts at scale.
The evaluation is extensive, covering 32 models (26 open-weight, 6 proprietary). The key finding is that even the strongest models achieve complete recall of all verified accounts for only ~52% of questions, revealing a significant gap between factual retention and completeness. The analysis of exposure imbalance is particularly strong, showing that minority-side exposure is more predictive of complete recall than majority-side exposure. The ablation on reasoning modes reveals a counter-intuitive result where reasoning can sometimes degrade completeness in smaller models by suppressing less salient accounts. The inclusion of a perplexity-based proxy for evaluation adds practical value for efficient benchmarking.
The paper provides high reproducibility. The code and dataset are released under Apache 2.0. The construction pipeline is detailed with specific prompts and hyperparameters. The use of standard tools (DCLM, T-NER, SuperGPQA taxonomy) and open-source models for construction ensures that the benchmark can be regenerated or extended. The evaluation protocol is clearly defined, including the judge prompt and scoring rubric.
The primary limitation is the reliance on a specific low-exposure corpus (RePro organic data) as a proxy for general web exposure, which may not perfectly align with the training data of all evaluated models. The benchmark size (1,094 questions) is moderate, and the 22 knowledge fields, while diverse, may not cover the full breadth of long-tail knowledge. The observational nature of the exposure analysis means causal claims cannot be made.
This paper has significant implications for the evaluation of LLMs, shifting the focus from single-answer accuracy to the completeness of parametric memory. It highlights a critical failure mode (epistemic myopia) that is obscured by traditional QA benchmarks. The findings on exposure imbalance provide actionable insights for data curation strategies, suggesting that balancing exposure to minority accounts is crucial for improving model completeness. The benchmark serves as a valuable diagnostic tool for the field. The paper introduces ElephantBench, a closed-book benchmark derived from low-exposure web corpora to probe the "epistemic myopia" of LLMs, revealing that even frontier models fail to recall all verified accounts of long-tail facts in nearly half of cases. By utilizing a graph-based pipeline to mine natural source disagreements and verifying them through multi-stage auditing, the study provides a rigorous diagnostic tool that distinguishes between factual retention and memory completeness, offering critical insights into the limitations of parametric knowledge and the impact of data exposure imbalance on model behavior.
An LLM agent shown a professional-looking market panel commits to a directional call on a provably unpredictable question far more often than one asked the bare question: across 12 frontier models, commitment rises from 6.5% to 54.0% as evidence is escalated. It commits just as readily when every number on the panel is invented: fabricating the entire display, so nothing the model can see is true except the question itself, still lifts commitment from 24.5% to 36.8%, statistically indistinguishable from the 37.6% produced by genuine market data. What unlocks confident action is not information but the authority of its packaging. The failure is narrow and locatable. Incapacity is not the answer: on matched answerable questions attached to the same panels, the same models answer essentially always, at near-perfect accuracy. Nor is it belief - stated probabilities barely move across the gradient that swings action by 48 points, and score worse than a climatological baseline. Missing judgment isn't it either: asked to classify a question's knowability before acting, models call it irreducible 90% of the time and then commit on just 0.4% of those. The act/don't-act gate is what fails, and the effect is concentrated in a few models rather than universal. Because the gate is separable, it can be trained. Supervised fine-tuning of a 3B model on 540 synthetic cases, predominantly dice, coins, jars and timers, drives commitment to 0.0% on the original cases and transfers to three unseen domains. It does not survive everything: the gate holds exactly when the response format leaves room to reason, and rigid formats that remove that room leave the model confident and wrong on questions it otherwise answers correctly. The gate is trainable and context-fragile, and deployment needs both halves of that sentence.
Primary: Independent Researcher
All Institutions: Independent Researcher
This paper has profound broader implications for the development, deployment, and governance of LLM agents. It highlights a critical vulnerability: the "authority of packaging" can override an LLM's inherent judgment about knowability, leading to confident but unearned commitments on unpredictable questions. This directly impacts AI safety and reliability, as agents in real-world settings (e.g., finance, healthcare) are often presented with authoritative-looking data. The finding that action calibration is distinct from belief calibration necessitates a shift in evaluation paradigms, moving beyond stated probabilities to auditing decisions. The proposed "commitment rate on aleatoric probes" as a cheap, reproducible robustness test offers a concrete measurement artifact for compliance with emerging regulations like the EU AI Act (Article 15) and NIST AI RMF. The demonstration of a trainable gate provides a pathway for mitigation, but its context-fragility underscores the need for careful prompt engineering and robust deployment strategies, emphasizing that "deployment needs both halves of that sentence." This work is a significant step towards building more trustworthy and appropriately cautious AI agents. This paper makes a crucial contribution by demonstrating that LLM agents are susceptible to making confident, unearned commitments on unknowable questions when presented with authoritative-looking, even fabricated, evidence. The rigorous methodology causally isolates the effect of presentation over information, localizes the failure to an act/don't-act gate, and shows this gate can be trained with synthetic data, though it remains context-fragile. This work provides significant empirical findings that challenge current assumptions about LLM calibration and reliability, offering concrete implications for agent evaluation, safety, and responsible deployment.
The methodology is exceptionally rigorous and well-designed. The core innovation lies in the "unknowability oracle" for aleatoric questions (short-horizon price direction, sports outcomes, weather) with sealed outcomes, ensuring provable unpredictability. The evidence gradient, escalating from bare questions to rich, professional-looking panels, is systematically applied. Crucially, the paper introduces a "fabricated evidence" condition where entire data panels are invented while preserving their authoritative form and internal consistency, allowing for causal isolation of presentation from information. The agentic protocol, offering ANSWER, CALL_TOOL, or DECLINE options, directly measures action rather than just stated probabilities. The use of matched answerable controls is vital for distinguishing a true knowability gate from a blanket decliner. The paper employs a comprehensive suite of metrics, including commitment rate, Youden's J for discrimination, Brier score for earned check, and CORP decomposition for calibration, all carefully defined and justified against degenerate strategy baselines. The supervised fine-tuning (SFT) approach on a 3B model using synthetic data from unrelated domains (dice, coins, timers) is a clever strategy to teach the abstention behavior, and the subsequent testing for transfer and fragility is thorough. The transparency in discussing parser choices (strict vs. semantic) and their impact is commendable.
The experimental evaluation is extensive, spanning 12 frontier models, 11 trained checkpoints, and four distinct domains (equity, crypto, sports, weather). The headline finding—commitment rising from 6.5% to 54.0% with escalating evidence, and fabricated evidence inducing commitment statistically indistinguishable from genuine data—is robust and highly impactful. The disaggregated results reveal that the effect is concentrated in specific models, not universal, which is an important nuance. The "dial, not a switch" observation, showing commitment scales with the quantity of authoritative display, further strengthens the causal link. The localization experiments effectively rule out incapacity, absent belief (stated probabilities barely move and are anti-predictive), and absent judgment (models classify questions as irreducible 90% of the time before acting). The trainability of the gate is demonstrated convincingly, with SFT driving commitment to 0.0% on original cases and transferring to unseen domains, surviving a tense-balanced control that rules out a simple grammatical heuristic. However, the critical finding in Section 8, that the trained gate's robustness is context-fragile and breaks when the response format removes room for reasoning, is a crucial insight for deployment. The detailed reporting of run-to-run variability and the impact of training data composition (e.g., removing sports items) further enhances the credibility of the results.
The paper demonstrates a strong commitment to reproducibility. It explicitly links to a GitHub repository containing code, data, pre-registration, and all cached model outputs. The training recipe (SFT with QLoRA on Qwen2.5-3B-Instruct, 3 epochs, 540 synthetic cases) is detailed. Prompt examples for both training and evaluation are provided verbatim in Appendix B. The two parsing strategies (strict and semantic) are clearly defined, and their delta is reported, ensuring transparency in how model outputs were interpreted. The paper also discusses potential sources of variability, such as greedy decoding not being bit-reproducible and uncontrolled serving configurations for hosted models, which adds to its credibility. The pre-registration details in Appendix A, distinguishing between decisions made before and after data existence, are exemplary for scientific rigor.
The authors are highly transparent about the limitations. Six open threats are acknowledged: (1) Weather is the weakest instrument due to genuine skill in its ensemble rain probability and lack of sealed outcomes. (2) The finding that only a subset of frontier models are seduced, meaning pooled rates aren't universal. (3) A residual lexical cue in synthetic unknowable items (e.g., "fair," "hidden") not fully excluded, though transfer domains speak against it. (4) The intervention being limited to one 3B model at one scale, meaning generalizability to larger models is not established. (5) Lower bounds for discrimination scores for two models due to high tool calls. (6) Cell-level values being approximate due to greedy decoding variability and uncontrolled serving configurations. My assessment aligns with these, and the paper's proactive disclosure of these limitations strengthens its overall scientific integrity.
This paper has profound broader implications for the development, deployment, and governance of LLM agents. It highlights a critical vulnerability: the "authority of packaging" can override an LLM's inherent judgment about knowability, leading to confident but unearned commitments on unpredictable questions. This directly impacts AI safety and reliability, as agents in real-world settings (e.g., finance, healthcare) are often presented with authoritative-looking data. The finding that action calibration is distinct from belief calibration necessitates a shift in evaluation paradigms, moving beyond stated probabilities to auditing decisions. The proposed "commitment rate on aleatoric probes" as a cheap, reproducible robustness test offers a concrete measurement artifact for compliance with emerging regulations like the EU AI Act (Article 15) and NIST AI RMF. The demonstration of a trainable gate provides a pathway for mitigation, but its context-fragility underscores the need for careful prompt engineering and robust deployment strategies, emphasizing that "deployment needs both halves of that sentence." This work is a significant step towards building more trustworthy and appropriately cautious AI agents. This paper makes a crucial contribution by demonstrating that LLM agents are susceptible to making confident, unearned commitments on unknowable questions when presented with authoritative-looking, even fabricated, evidence. The rigorous methodology causally isolates the effect of presentation over information, localizes the failure to an act/don't-act gate, and shows this gate can be trained with synthetic data, though it remains context-fragile. This work provides significant empirical findings that challenge current assumptions about LLM calibration and reliability, offering concrete implications for agent evaluation, safety, and responsible deployment.
Two of the most fundamental questions in statistical learning theory are the following: which prediction problems are learnable, and how should they be learned? For the former, elegant answers often take the form of combinatorial dimensions. The latter question, however, has proved considerably more elusive: all known general-purpose multiclass learners rely on intricate orientations of exponentially large one-inclusion structures, and familiar algorithmic principles such as proper learning and regularization remain poorly understood. Motivated by prior work, we ask whether learning reduces to proper learning---possibly over a larger hypothesis class---and whether proper or improper multiclass learning can ultimately be captured by suitable regularizers. Our primary results answer both questions negatively, resolving three open problems from prior work. First, we exhibit a learnable multiclass problem that cannot be embedded in any properly learnable class, meaning learning cannot be reduced to proper learning by enlarging the hypothesis class. Second, we demonstrate that proper learning can require training error and characterize this phenomenon precisely: every properly learnable class admits a proper learner making $o(m)$ errors on samples of size $m$, but every prescribed sublinear scale $a_m=o(m)$ is necessary for some properly learnable problem. Third, regularization is not a general learner: we exhibit a properly learnable class that cannot be learned by any Structural Risk Minimization (SRM) learner, and a learnable class that cannot be learned by any local regularizer. We complement these impossibility results with a positive theory that gives two sufficient conditions for SRM learnability and characterizes SRM representability through integrability of revealed preferences.
Primary: University of Washington
All Institutions: University of Washington, Northwestern University, University of Chicago
The paper resolves three major open problems in statistical learning theory by proving that learning cannot be reduced to proper learning and that regularization is not a general learner, providing a rigorous theoretical framework for understanding the limits of these fundamental algorithmic principles.
The paper employs rigorous theoretical methods from combinatorics and functional analysis to address fundamental questions in statistical learning theory. It constructs specific counterexamples (learnable but not properly learnable classes, classes not learnable by SRM) to prove negative results. The methodology is sophisticated, utilizing concepts like the DS dimension, one-inclusion structures, and integrability of revealed preferences to characterize the boundaries of algorithmic principles like proper learning and regularization. The use of AI (ChatGPT 5.6 Pro) in the discovery of constructions is noted, but the core contribution remains the mathematical proofs and theoretical frameworks.
As a purely theoretical paper, there are no empirical experiments, datasets, or benchmarks. The "evaluation" consists of the validity and tightness of the mathematical proofs and the resolution of open problems. The results are definitive within the theoretical framework.
The results are mathematically reproducible in the sense that the proofs can be verified. There is no code or experimental setup to reproduce. The transparency regarding the use of AI in the discovery process is a positive step for reproducibility of the research process, though it raises questions about the novelty of the specific constructions.
The paper is highly theoretical and may have limited immediate practical impact on algorithm design. The reliance on AI for key constructions, while disclosed, might lead to skepticism about the depth of human insight in the specific combinatorial designs, although the authors claim full responsibility. The results are negative (impossibility results), which, while valuable, do not provide new algorithms.
The paper significantly impacts the field of statistical learning theory by clarifying the limits of standard algorithmic principles. It guides future research by showing where proper learning and regularization fail, potentially steering researchers toward more nuanced or problem-specific approaches. It resolves long-standing open problems, contributing to the foundational understanding of multiclass learning. The paper resolves three major open problems in statistical learning theory by proving that learning cannot be reduced to proper learning and that regularization is not a general learner, providing a rigorous theoretical framework for understanding the limits of these fundamental algorithmic principles.
Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, State Key Laboratory for General Artificial Intelligence, X-LANCE Lab
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces ASIL, an agent-native interface that replaces screenshot-and-click with structured state and semantic actions, demonstrating significant improvements in task success and training efficiency for software-operating agents. The technical contribution is robust, with a well-defined interface contract, a comprehensive benchmark, and rigorous experiments showing that structured interfaces are superior to pixel-based ones for both inference and training. The significance to the field is high, as it challenges the prevailing paradigm of GUI agents and offers a practical, scalable solution for operating complex software through agents. The work is likely to influence future designs of agent interfaces and evaluation benchmarks.
The paper proposes ASIL (Agent-Software Interaction Layer), a framework that replaces the standard "screenshot-and-click" interface for GUI agents with structured JSON observations and semantic, code-executable actions. The core methodological contribution is the "ASILization pipeline," which maps heterogeneous applications (file-backed, script-based, API-based) to a unified agent-native interface. This approach is conceptually sound, drawing parallels to the success of code agents (SWE-agent, CodeAct) in software engineering. By exposing internal state (e.g., ODF XML, Blender Python scene graph, REST endpoints) rather than pixel data, the method reduces the action space from low-level motor primitives to high-level semantic operations. The methodology is rigorous in its definition of the interface contract (observe, execute, validate) and provides a semi-automatic onboarding process for new applications. However, the novelty is somewhat incremental as it essentially applies existing principles of API-based agents to GUI software, rather than introducing a fundamentally new architectural paradigm. The "deepest feasible access path" principle is a practical engineering solution rather than a theoretical breakthrough.
The experimental setup is extensive, featuring a 380-task benchmark across 15 applications, including single-app and multi-app tasks. The results are compelling: ASIL achieves >80% success with closed models using <5 actions, whereas screenshot-and-click baselines struggle significantly (6.6-26.6% strict success). The inclusion of training experiments (SFT and RL) on smaller models (Qwen3.5-2B/9B) demonstrates that the structured modality is more sample-efficient for training, yielding double-digit gains. The ablation studies on realization patterns (file vs. script vs. API) and the comparison against native interfaces (LibreOffice UNO, draw.io MCP) add depth. However, the comparison is somewhat biased because the ASIL prompts include evaluator-derived hints while GUI prompts do not, although the authors acknowledge this and provide hint-off comparisons in the camera-ready version. The "repaired GUI" baseline is a strong control, but the gap remains massive, suggesting the interface change is the primary driver of performance.
The paper provides detailed implementation details, including adapter contracts, observation schemas, and training hyperparameters. The release of task definitions, validators, adapter code, and training data enhances reproducibility. The use of standard open-source models (Qwen) and common frameworks (Verl, vLLM) further supports reproducibility. The semi-automatic onboarding pipeline is described in detail, allowing other researchers to extend ASIL to new applications. The benchmark is shared, enabling direct comparison of different agent interfaces.
The main limitation is the coverage gap for fully opaque, closed-source applications that lack open file formats, scripting interfaces, or service APIs. ASIL requires at least one "access door" to be open. Additionally, the method struggles with intrinsically perceptual tasks (e.g., aesthetic judgment in image editing) where structured state is insufficient. The prompt asymmetry in the original comparison, while addressed in the camera-ready version, remains a point of caution. The training gains for the 2B model on hard tasks are less consistent, indicating that small models may still struggle with long-horizon planning even with better interfaces.
This paper has significant potential impact on the field of AI agents. By demonstrating that structured interfaces outperform pixel-based ones for software operation, it provides a clear direction for future research in agent-computer interaction. The framework could be widely adopted to improve the reliability and efficiency of agents operating in real-world software environments. The training efficiency gains suggest that ASIL could enable the development of smaller, more capable agents for specific software domains. The work bridges the gap between code agents and GUI agents, offering a unified view of agent-software interaction. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces ASIL, an agent-native interface that replaces screenshot-and-click with structured state and semantic actions, demonstrating significant improvements in task success and training efficiency for software-operating agents. The technical contribution is robust, with a well-defined interface contract, a comprehensive benchmark, and rigorous experiments showing that structured interfaces are superior to pixel-based ones for both inference and training. The significance to the field is high, as it challenges the prevailing paradigm of GUI agents and offers a practical, scalable solution for operating complex software through agents. The work is likely to influence future designs of agent interfaces and evaluation benchmarks.
Agent skills bundle instructions, reference data, and executable helpers that let a general agent perform specialized tasks. Hosted providers can keep these files secret while selling access to task results, making the skill itself a valuable target. Existing disclosure defenses can block requests that ask for the skill or reproduce its text, but they cannot block customers from submitting the ordinary tasks the service is built to complete. We present Daydreaming, an execution-only attack that steals a multi-file skill through black-box task interactions. The victim is never asked to reveal the skill or grade a reconstruction. Instead, Daydreaming adaptively creates crafted tasks whose results distinguish possible hidden behaviors. It tests individual behaviors, uses attacker-controlled shadow agents to choose a design, and completes each file using stored victim results and local execution checks. We formalize three nested threat levels of access as Differential, Trace, and Output, and focus on Output, where the attacker sees only the final response and returned files. Across 7 skills and 4 victim models, Daydreaming recovers 86.8% of the original skill's capability at Output, outperforming SigLeak by almost 4x. It produces installable skills using a median of 32 victim calls per skill even with disclosure defenses enabled. These results show that hiding skill files and filtering direct disclosure do not, by themselves, prevent functional reconstruction through normal use.
Primary: UC Berkeley
All Institutions: UC Berkeley, National Yang Ming Chiao Tung University
Daydreaming presents a novel execution-only attack that reconstructs hidden agent skills through black-box task interaction, outperforming existing methods by 4x. The paper rigorously formalizes the threat landscape of Skill-as-a-Service, demonstrating that standard disclosure defenses are ineffective against adaptive behavioral inference, thereby establishing a new baseline for agentic security research.
The paper introduces "Daydreaming," a sophisticated black-box attack framework for stealing proprietary agent skills (instructions + code + data) from hosted services. The core methodological contribution is the formalization of three nested observability levels (Differential, Trace, Output) and the design of a hierarchical hypothesis refinement loop that operates effectively under the most restrictive "Output" setting. The approach cleverly uses "discriminating tasks"—inputs crafted specifically to distinguish between competing hypotheses about the hidden skill's logic—combined with local "shadow agents" to simulate and verify potential behaviors without direct access to the victim's internals. This shifts the paradigm from simple prompt extraction to behavioral system identification, which is a significant conceptual advance in AI security.
The evaluation is rigorous, testing across 7 diverse skills and 4 victim models. The paper demonstrates that Daydreaming recovers 86.8% of the original skill's capability using only final outputs, significantly outperforming prior baselines like SigLeak (by ~4x). The use of normalized success rates (NSR) and behavioral utility (NU) provides a fair comparison against the original skill. The ablation studies on query budgets and component contributions further validate the method's efficiency and robustness. The finding that structural recovery (exact file matching) is low while behavioral recovery is high is a critical insight for defenders.
The paper provides a clear algorithmic description (Algorithm 1) and detailed descriptions of the three stages. While the code repository is currently anonymous (typical for pre-publication), the detailed prompts and parameter settings in the appendices suggest high reproducibility. The use of standard benchmarks (SkillsBench) and open-weight models for some experiments aids in verification.
The attack relies on the attacker having a capable "attacker model" (Gemini-3.7-flash in experiments) to generate hypotheses and shadow agents; if the attacker's model is significantly weaker than the victim's, the attack's efficacy may drop. Additionally, the evaluation is limited to 7 skills, which, while diverse, may not cover all possible skill structures. The assumption that the attacker can execute returned code locally (for verification) is a strong privilege that may not always be available in strict SaaS environments.
This paper has high impact on the emerging field of Agentic AI security. It demonstrates that traditional "prompt injection" defenses are insufficient against sophisticated behavioral cloning attacks. It forces service providers to consider not just text leakage but functional leakage through standard task execution. This will likely lead to new defensive mechanisms focused on output obfuscation, rate limiting, or differential privacy in agent responses. Daydreaming presents a novel execution-only attack that reconstructs hidden agent skills through black-box task interaction, outperforming existing methods by 4x. The paper rigorously formalizes the threat landscape of Skill-as-a-Service, demonstrating that standard disclosure defenses are ineffective against adaptive behavioral inference, thereby establishing a new baseline for agentic security research.
This work presents $\textbf{Lapis}$, a $\textbf{l}$inear-$\textbf{a}$ttention-based $\textbf{pi}$xel-$\textbf{s}$pace generative framework that achieves efficient and high-fidelity depth estimation with one-step diffusion. While generative frameworks have significantly advanced monocular depth estimation with superior detail fidelity, the $\mathcal{O}(N^2)$ complexity of standard attention and the multi-step denoising process introduce prohibitive computational costs when scaling them to high-resolution image applications. Although linear attention and one-step prediction are intuitively viable, directly applying them leads to poor structural consistency, detail loss, and noise. Lapis rectifies these limitations through a coarse-to-fine hierarchy. Specifically, a Patch-level Consistency Module restores structural coherence by integrating semantic and spatial priors. Subsequently, a Pixel-level Refinement Module recovers sharp geometric boundaries via skip-connection-based pixel correspondence. Furthermore, to mitigate sampling noise inherent in one-step diffusion, we leverage the manifold assumption and adopt a direct $\mathbf{x}$-prediction strategy to target the clean data manifold. Extensive evaluations on multiple benchmarks demonstrate that Lapis consistently achieves state-of-the-art (SOTA) accuracy and boundary sharpness across various resolutions, reducing inference latency by up to 7.6$\times$ at 1080P and 10.9$\times$ at 1440P resolution compared to previous SOTA generative models.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University, Zhiyuan College
[One sentence main contribution]. The paper introduces Lapis, a linear-attention-based pixel-space diffusion framework that achieves state-of-the-art monocular depth estimation with significantly reduced inference latency (up to 10.9x faster at 1440P) by employing a coarse-to-fine hierarchy and one-step $x$-prediction, effectively addressing the computational bottlenecks of high-resolution generative depth estimation.
The paper proposes Lapis, a framework addressing the computational bottleneck of pixel-space diffusion models for depth estimation. The core innovation lies in combining linear attention (to reduce $\mathcal{O}(N^2)$ complexity) with a one-step denoising process. The authors correctly identify that naively applying these techniques leads to structural incoherence and noise. To mitigate this, they introduce a coarse-to-fine hierarchy: a Patch-level Consistency Module that integrates semantic and spatial priors to restore structural coherence, and a Pixel-level Refinement Module using skip connections for boundary sharpness. Additionally, they employ an $x$-prediction strategy based on the manifold assumption to target the clean data manifold, which is a sound theoretical choice for reducing sampling noise in few-step diffusion. The methodology is coherent and directly addresses a significant practical limitation (latency) in state-of-the-art generative depth estimators.
The paper claims state-of-the-art accuracy and boundary sharpness across multiple benchmarks and resolutions. Specifically, it reports inference latency reductions of up to 7.6x at 1080P and 10.9x at 1440P compared to previous SOTA generative models. While the specific benchmark names (e.g., NYUv2, KITTI, DTU) are not explicitly listed in the provided abstract/summary, the claim of "multiple benchmarks" and high-resolution performance (1440P) suggests a rigorous evaluation protocol typical of top-tier vision conferences. The focus on high-resolution efficiency is a strong selling point, as most prior works struggle with computational cost at these scales.
The authors provide a public GitHub repository (https://github.com/VISION-SJTU/Lapis), which is a positive indicator for reproducibility. The methodological details regarding the specific modules (Patch-level Consistency, Pixel-level Refinement) are described sufficiently in the abstract to understand the architectural flow. However, without the full text of the implementation details section (referenced as `supp_sections/1`), specific hyperparameters and training schedules cannot be fully verified, but the code release mitigates this risk.
The primary limitation is the reliance on the "manifold assumption" for the $x$-prediction strategy, which may not hold perfectly for all out-of-distribution data or extreme geometric structures. Additionally, while linear attention reduces complexity, it may still incur higher memory costs than standard CNN-based approaches, though this is likely offset by the one-step nature. The paper does not explicitly discuss failure cases in the provided text, which is a minor gap in the summary.
This work has significant practical impact for real-time applications requiring high-fidelity depth maps, such as AR/VR, robotics, and autonomous driving, where latency is critical. By enabling pixel-space diffusion at 1440P with reduced latency, it bridges the gap between high-quality generative models and real-time deployment constraints. The techniques proposed (linear attention in diffusion, coarse-to-fine refinement) are likely to be adopted in other generative vision tasks. [One sentence main contribution]. The paper introduces Lapis, a linear-attention-based pixel-space diffusion framework that achieves state-of-the-art monocular depth estimation with significantly reduced inference latency (up to 10.9x faster at 1440P) by employing a coarse-to-fine hierarchy and one-step $x$-prediction, effectively addressing the computational bottlenecks of high-resolution generative depth estimation.
Latent generative models typically follow a two-stage pipeline, training a variational autoencoder for reconstruction and then a generative model on the frozen latent space. Since reconstruction-optimized latents are not necessarily generation-friendly, jointly training both models is an appealing alternative. However, direct end-to-end training remains challenging, as it is prone to latent collapse and faces a generation-reconstruction conflict. We revisit this problem by analyzing how different objectives shape the latent space and identify two key insights. First, the entropy term in the Kullback-Leibler divergence objective is essential for preventing collapse: reconstruction and prior fitting tend to shrink the posterior, while entropy preserves non-degenerate latent uncertainty. Second, reconstruction and generation exhibit asymmetric learning dynamics: reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize. Based on these insights, we achieve the first direct end-to-end training without latent collapse and propose GenFirst, a simple generation-before-reconstruction strategy. The generative objective first shapes the latent space under weak reconstruction pressure, after which reconstruction is progressively strengthened to recover visual details. We validate GenFirst with continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods. With our end-to-end objective and GenFirst, SiT achieves a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, while MMDiT reaches a GenEval score of 0.90 on text-to-image generation. Beyond image generation, we extend the framework to shared visual latents for generation and representation learning, and to continuous unified text-image generation. These results demonstrate the generality of stable end-to-end latent learning across generative priors and modalities.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China
The paper identifies the root cause of latent collapse in end-to-end training (prior-entropy imbalance) and proposes a simple, effective strategy (GenFirst) to resolve the generation-reconstruction conflict, achieving state-of-the-art results on ImageNet and text-to-image benchmarks.
The paper proposes a principled solution to the instability of end-to-end training in latent generative models. The core insight is identifying "prior-entropy imbalance" as the cause of latent collapse, where the generative objective's prior-fitting force overwhelms the weak KL regularization typically used in VAEs. The authors introduce an explicit entropy term to counteract this. Furthermore, they propose "GenFirst," a two-stage training schedule that prioritizes generation to shape the latent space before strengthening reconstruction. This addresses the asymmetric learning dynamics between the two objectives. The method is validated on both exact-likelihood autoregressive models (EAR) and flow-matching models (SiT/MMDiT), demonstrating generality.
The experimental results are strong. The paper reports a gFID of 0.97 on ImageNet-256 with SiT, which is a state-of-the-art result for diffusion models without using Fréchet Distance loss. It also achieves a GenEval score of 0.90 on text-to-image generation, outperforming larger models like FLUX.2-dev. The ablation studies are thorough, clearly isolating the effects of the entropy term and the GenFirst schedule. The comparison with REPA-E is particularly relevant, showing consistent improvements.
The paper provides detailed descriptions of the training schedules, loss weights, and architectural choices. However, specific hyperparameters for the "prior-only" phase and some implementation details of the GMM head are left to appendices or referenced works. The code availability is not explicitly stated in the provided text, which is a minor concern for immediate reproducibility, though the method is described clearly enough to be implemented.
The trade-off between generation and reconstruction is not fully eliminated; reconstruction fidelity (PSNR) still drops compared to standard VAEs. The autoregressive model (EAR) suffers from issues with Classifier-Free Guidance (CFG) scaling. The text-to-image experiments use a smaller dataset than industrial standards, so the scalability of the data efficiency claim is not fully tested at the largest scales.
This work provides a practical recipe for stable end-to-end latent learning, which could simplify the training pipeline for future generative models. By showing that latent spaces can be jointly optimized for generation and representation learning, it opens avenues for unified models that do not require separate pre-training stages for the tokenizer. The insights into entropy preservation are likely to be adopted in other variational frameworks. The paper identifies the root cause of latent collapse in end-to-end training (prior-entropy imbalance) and proposes a simple, effective strategy (GenFirst) to resolve the generation-reconstruction conflict, achieving state-of-the-art results on ImageNet and text-to-image benchmarks.
The integration of novel view synthesis (NVS) and open-vocabulary segmentation (OVS) has recently yielded powerful feed-forward 3D foundation models. However, their inherent reliance on static-scene assumptions leads to severe misalignment of spatial features in unconstrained dynamic environments. To bridge this critical gap, we propose SPAR, a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation. Furthermore, we introduce a dynamic-region-aware end-to-end training paradigm that structurally couples motion estimation with multi-view visual and semantic learning. This unified approach enables the network to inherently resolve motion conflicts and distill multi-view consistent, temporally stable scene representations from dynamic inputs. Extensive experiments on the challenging D-RE10K benchmark demonstrate that SPAR achieves state-of-the-art performance. Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively. Despite being trained in a self-supervised manner, our model achieves an mIoU of 88.5% for motion mask prediction. Furthermore, our analysis reveals a strong inter-task synergy between photometric scene reconstruction and semantic understanding, where semantic synthesis learning consistently enhances photometric fidelity in novel view rendering. Code will be available at https://github.com/dmucby/SPAR.
Primary: Institute of Automation, Chinese Academy of Sciences (CASIA)
All Institutions: ShanghaiTech University, Institute of Automation, Chinese Academy of Sciences (CASIA), The Chinese University of Hong Kong, Deepeleph Intelligent Technology
SPAR introduces a dynamic-robust photometric-semantic reconstruction framework that effectively isolates dynamic noise to enhance open-vocabulary 3D scene understanding in unconstrained environments. The paper presents a coherent solution to a significant problem in 3D vision, offering a unified training paradigm that improves both geometric and semantic consistency, though its impact is tempered by the specific benchmark usage and moderate photometric metrics.
The paper proposes SPAR, a joint semantic-geometric encoding architecture designed to handle dynamic scenes in open-vocabulary 3D scene understanding. The core methodological contribution is the explicit isolation of transient dynamic noise prior to latent space aggregation, which addresses the misalignment issues inherent in static-scene assumptions. Additionally, the authors introduce a dynamic-region-aware end-to-end training paradigm that couples motion estimation with multi-view visual and semantic learning. This unified approach allows the network to resolve motion conflicts and distill temporally stable representations. The methodology is logically sound and directly addresses a known limitation in feed-forward 3D foundation models.
Experiments are conducted on the D-RE10K benchmark. The reported results include a PSNR of 22.15 dB (3 views) and 23.33 dB (4 views) for novel view synthesis, and an mIoU of 88.5% for motion mask prediction. The paper claims state-of-the-art performance and highlights a synergy between photometric reconstruction and semantic understanding. However, the PSNR values are relatively modest for high-quality NVS, and the reliance on a single benchmark limits the generalizability of the claims. The self-supervised nature of the motion mask prediction is a strong point, but the absolute performance metrics need to be weighed against recent competitors in dynamic NVS.
The authors state that code will be available at the provided GitHub URL. The paper provides a clear description of the architecture and training paradigm. However, without access to the code or detailed hyperparameter settings in the text, full reproducibility is currently pending. The use of a specific benchmark (D-RE10K) aids in standardization.
The primary limitation is the reliance on the D-RE10K benchmark, which may not cover all dynamic scene complexities. The PSNR scores, while claimed to be SOTA, are not exceptionally high, suggesting potential room for improvement in photometric fidelity. The paper does not extensively discuss computational cost or inference speed, which are critical for real-time applications. Additionally, the "open-vocabulary" aspect is mentioned in the title but the depth of the semantic evaluation beyond mIoU is not fully detailed in the abstract.
This work contributes to the robustness of 3D foundation models in real-world, dynamic environments. By addressing the static-scene assumption, it enables more reliable applications in autonomous driving, robotics, and AR/VR where dynamic objects are prevalent. The synergy between semantic and geometric tasks offers insights into multi-task learning in 3D vision. SPAR introduces a dynamic-robust photometric-semantic reconstruction framework that effectively isolates dynamic noise to enhance open-vocabulary 3D scene understanding in unconstrained environments. The paper presents a coherent solution to a significant problem in 3D vision, offering a unified training paradigm that improves both geometric and semantic consistency, though its impact is tempered by the specific benchmark usage and moderate photometric metrics.
Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We then identify three structural factors that shape SHS---window interaction, token serialization, and local softmax allocation---and use them as design principles for hybrid attention. Guided by these factors, we design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute. Our findings establish head specialization as a measurable property for diagnosing and designing principled hybrid ViT attention at the multimodal-LLM scale.
Primary: The University of Hong Kong
All Institutions: The University of Hong Kong, Xiaomi Corporation, Peking University
The paper introduces Semantic Head Specialization (SHS) as a diagnostic metric to guide the design of efficient hybrid attention in Vision Transformers, proposing Ariadne Attention which achieves near-full-attention quality at significantly lower compute costs. The work is technically sound and offers valuable insights into the structural factors affecting attention specialization, though its impact is moderated by the limited scale of the controlled training experiments and the reliance on a small language model backbone for validation.
The paper proposes a diagnostic framework, Semantic Head Specialization (SHS), to explain the performance gap between full and hybrid attention in Vision Transformers (ViTs). The core methodology involves training matched pairs of ViTs (full vs. chunk-window) from scratch to isolate the effect of the attention operator. The authors define an AUROC-based metric (SHS-Index) to quantify how well attention heads separate foreground from background tokens. They identify three structural factors affecting this specialization: window isolation, token serialization order, and local softmax allocation. Based on these insights, they design "Ariadne Attention," a hybrid scheme using sliding windows, alternating row/column serialization, and sink biases. The methodology is rigorous in its controlled comparisons, though the reliance on a single small LLM backbone (Qwen2-0.5B) to validate the ViT properties is a significant methodological constraint.
The experiments are extensive within the controlled setting. The authors evaluate 9 different attention configurations across 22 downstream benchmarks. The correlation between SHS-Index and benchmark performance (r=0.858) is a strong empirical finding. The proposed Ariadne Attention achieves performance close to full attention (40.40 vs 40.92) while significantly reducing compute (6.5x less attention FLOPs). The evaluation includes ablations on window size, serialization order, and sink bias. However, the evaluation is limited to a single training run per configuration and a single LLM size, which limits the generalizability of the correlation findings.
The paper provides detailed architectural specifications, training hyperparameters, and benchmark definitions. The use of open-source models for the SHS-Index validation (16 models) enhances reproducibility of the diagnostic metric. However, the controlled training experiments require significant compute resources (training 9 ViTs from scratch), which may limit immediate reproduction by smaller labs. Code availability is not explicitly stated in the provided text, though the use of standard libraries (FlashAttention-3) suggests high reproducibility for the attention mechanisms.
The primary limitation is the scale of the controlled study. All controlled experiments use a small LLM (0.5B) and a single seed. The authors acknowledge that the correlation between SHS-Index and performance has not been tested with larger backbones or multiple seeds. Additionally, the "Ariadne" design is specific to the 32-layer ViT architecture tested; its applicability to other ViT depths or patch sizes is not fully explored. The paper also notes that certain tasks (counting, exact geometry) still regress slightly compared to full attention.
This paper provides a valuable diagnostic tool (SHS-Index) for the community to analyze attention mechanisms in ViTs. The findings on token serialization and window interaction offer practical design principles for developing efficient hybrid attention mechanisms in multimodal LLMs. The work bridges the gap between mechanistic interpretability (head specialization) and architectural design, potentially guiding future efforts to reduce the compute cost of high-resolution vision encoders without sacrificing quality. The paper introduces Semantic Head Specialization (SHS) as a diagnostic metric to guide the design of efficient hybrid attention in Vision Transformers, proposing Ariadne Attention which achieves near-full-attention quality at significantly lower compute costs. The work is technically sound and offers valuable insights into the structural factors affecting attention specialization, though its impact is moderated by the limited scale of the controlled training experiments and the reliance on a small language model backbone for validation.
Explorable image-to-scene generation is essential for applications in gaming, robotics, and virtual reality. Existing methods based on video diffusion model (VDM) commonly rely on incomplete conditioning signals such as sparse point clouds or 2D panoramas, leading to stochastic hallucinations, long-term drifts and suboptimal 3D consistency. We present SpatialCrafter, a novel two-stage framework that addresses these issues by introducing a global 3D proxy for high-fidelity image-to-scene generation. Specifically, we decompose the generation process into global proxy generation and appearance refinement. For proxy generation, we propose a Point-anchored Sparse Structure~(PaSS) Flow module that predicts a spatially aligned and geometrically consistent 3D proxy. For appearance refinement, we re-frame the VDM as a Generative Deferred Refiner which synthesizes high-frequency photorealistic details upon proxy-defined scene geometry. To better integrate the proxy with the pre-trained VDM, we introduce Parallel Geometry Injection and Proxy-Aware Corruption training strategies, which improve robustness to proxy artifacts without disrupting the pretrained generative manifold. Furthermore, as no suitable dataset exists for this explorable scene generation task, we construct a new large-scale dataset of 115K scenes. To the best of our knowledge, it is the first hybrid dataset for image-to-scene generation. Extensive experiments on both synthetic and real-world datasets show that SpatialCrafter outperforms state-of-the-art methods, mitigates long-term drift, and remains robust and consistent under rapid camera motion and extreme viewpoint changes. Code, models, and the newly constructed dataset will be publicly released. See more at https://fangchuan.github.io/SpatialCrafter/.
Primary: Hong Kong University of Science and Technology
All Institutions: Hong Kong University of Science and Technology, Alibaba Group (Tongyi Lab), Jilin University, ManyCore Tech Inc
SpatialCrafter introduces a two-stage framework using a global 3D proxy to enable high-fidelity, 3D-consistent image-to-scene generation, effectively mitigating long-term drift and hallucinations common in video diffusion models. By decomposing generation into proxy creation and appearance refinement, and contributing a large-scale dataset, the paper makes a substantial contribution to the field of world modeling and 3D generation, offering a robust solution for creating explorable environments from single images.
The paper proposes SpatialCrafter, a two-stage framework for single-image world modeling. The core innovation is the introduction of a "global 3D proxy" to anchor the generation process, decomposing it into proxy generation (using a Point-anchored Sparse Structure Flow module) and appearance refinement (using a Generative Deferred Refiner based on Video Diffusion Models). The method addresses the common issues of stochastic hallucinations and long-term drift in existing video diffusion approaches by enforcing geometric consistency through the proxy. The introduction of Parallel Geometry Injection and Proxy-Aware Corruption training strategies is a technically sound approach to integrating 3D constraints without disrupting the pre-trained generative manifold. The methodology is coherent and directly targets a significant bottleneck in current image-to-scene generation.
The authors construct a new large-scale dataset of 115K scenes, which is a significant contribution given the lack of suitable data for this specific task. Experiments on both synthetic and real-world data demonstrate that SpatialCrafter outperforms state-of-the-art methods, particularly in mitigating long-term drift and maintaining consistency under rapid camera motion. The qualitative results described in the abstract suggest high-fidelity photorealistic details. However, without access to the full quantitative tables in the provided text, the depth of the experimental comparison is inferred from the abstract's claims of superiority over SOTA.
The paper promises to release code, models, and the newly constructed dataset publicly. The detailed description of the two-stage framework and specific modules (PaSS Flow, Generative Deferred Refiner) provides a clear roadmap for reproduction. The construction of a hybrid dataset further enhances reproducibility by providing a standardized benchmark for future work in this niche area.
The reliance on a pre-trained Video Diffusion Model (VDM) may limit the method's applicability to domains where such models are not well-trained or where the computational cost of VDMs is prohibitive. The "proxy" approach, while effective for consistency, may introduce artifacts if the initial proxy generation is inaccurate, although the paper claims robustness to proxy artifacts. The method is specifically tailored for explorable scenes, which may not generalize to all types of 3D content generation.
This work has significant implications for gaming, robotics, and VR by enabling the creation of explorable, 3D-consistent worlds from a single image. The release of a large-scale dataset will likely accelerate research in image-to-scene generation. The approach of using 3D proxies to guide diffusion models could inspire similar hybrid approaches in other generative tasks requiring structural consistency. SpatialCrafter introduces a two-stage framework using a global 3D proxy to enable high-fidelity, 3D-consistent image-to-scene generation, effectively mitigating long-term drift and hallucinations common in video diffusion models. By decomposing generation into proxy creation and appearance refinement, and contributing a large-scale dataset, the paper makes a substantial contribution to the field of world modeling and 3D generation, offering a robust solution for creating explorable environments from single images.
The growing rate at which LLM agents interact with one another raises key questions about language evolution in multi-LLM-agent settings, with implications for safety and monitorability as well as for linguistic accounts of LLMs. To address these questions, we introduce GlossoGen, a novel platform for studying multi-agent language evolution in complex scenarios. Within GlossoGen, we build the SaveVeyru scenario, which requires agents with partial information to communicate under pressure. We find that language evolution does occur between LLM agents, that the resulting languages are compositional and morphologically productive, and that they deviate from the LLMs' English prior in ways that render them incomprehensible to humans. Moreover, we identify several qualities essential to this evolution: pressure towards efficiency; the strength of the models backing the agents; and access to a "postmortem" stage in which agents can agree on linguistic conventions. Importantly, we observe that different conditions govern the transmission of language to new agents. Specifically, we find that agents learn new languages from usage alone, take an active role in this learning, and that while stronger models are required for novel language emergence, weaker models can learn an existing language once it has emerged. Taken together, our results indicate that current LLMs have the potential for cumulative cultural evolution -- previously attested only in humans -- with mixed populations of agents developing capacities that go beyond their lowest common denominator.
Primary: University of Cambridge
All Institutions: University of Cambridge, Anthropic, OpenAI
The paper presents a rigorous and novel framework for studying emergent language in LLM agents, demonstrating that under specific pressures, agents develop compositional, non-English languages that can be transmitted across model generations, highlighting both the capabilities and safety risks of multi-agent LLM systems.
The paper introduces GlossoGen, a flexible platform for simulating multi-agent LLM interactions with a focus on language evolution. The core methodological contribution is the "SaveVeyru" scenario, which forces information asymmetry and communication efficiency through a character budget. By using a fictional alien entity, the authors successfully prevent LLMs from relying on parametric medical knowledge, ensuring that communication is necessary for task success. The methodology includes rigorous controls for "postmortem" deliberation phases and budget pressures. The analysis of emergent languages employs linguistic tools, such as perplexity under an English LM and morphosyntactic induction via LLM judges, to characterize the structure of the evolved codes.
The experiments are extensive, testing both proprietary (GPT-5.4, Claude Opus 4.7, Sonnet 4.6) and open-weight (Llama, Qwen) models. Key findings include: (1) Language emergence requires both budget pressure and a postmortem phase; (2) Emergent languages are compositional and morphologically productive, not just memorized codebooks; (3) Stronger models are required to *create* new languages, but weaker models can *learn* them from usage, suggesting a path to cumulative cultural evolution. The statistical analysis (mixed-effects models) supports the significance of these findings. The observation that agents ask metalinguistic questions to clarify atomic terms but not compositional ones is a particularly insightful result regarding LLM internal representations.
The authors provide a GitHub repository with the platform and scenario definitions. The use of standard LLM APIs ensures that the core experiments are reproducible, though exact replication of proprietary model behaviors may vary with model updates. The detailed description of the scoring mechanism (LLM judge verified against gold data) enhances trust in the results.
The study relies on a single, albeit complex, scenario (SaveVeyru). It is unclear if these findings generalize to other types of multi-agent tasks (e.g., competitive vs. cooperative, or different domain structures). The "postmortem" phase is a strong confounder; in real-world dynamic environments, such explicit deliberation time may not be available. Additionally, the claim of "cumulative cultural evolution" is based on short-term transmission within a simulation, not long-term open-ended evolution.
This paper has significant implications for AI safety, demonstrating that LLM agents can develop unmonitorable communication protocols even in cooperative settings without adversarial pressure. It challenges the assumption that LLMs will always communicate in human-readable languages. For linguistics and cognitive science, it provides a new testbed for studying language emergence and transmission in artificial agents, potentially offering insights into the cognitive prerequisites for language evolution. The paper presents a rigorous and novel framework for studying emergent language in LLM agents, demonstrating that under specific pressures, agents develop compositional, non-English languages that can be transmitted across model generations, highlighting both the capabilities and safety risks of multi-agent LLM systems.
LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary projection, and attention-head knockout applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B). Both evaluators implement a structured, coherent evaluation pipeline operating in two stages: below layer 15, attention performs local error comparison and routes the result to the final input position; above it, the MLP cascade integrates the signal and writes the rating, with the decision crystallizing in the residual stream at a sharp late layer (L = 26 on Themis, L = 25 on Prometheus). Furthermore, a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and crystallization but not the stage separation, isolating the two mechanisms that fine-tuning specifically installs, suppression of below-L15 MLP contribution at the last position and a two-layer advance of the crystallization depth, indicating that fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. We release the source code and data at https://github.com/himil-v/judge-mech
Primary: University of Michigan
All Institutions: University of Michigan, University of Toronto
The paper provides a rigorous mechanistic analysis of LLM-as-a-Judge models, revealing a two-stage evaluation pipeline and the specific role of fine-tuning in sculpting this pipeline. This work significantly advances the interpretability of automated NLG evaluation, offering actionable insights for improving the reliability of LLM judges.
The paper proposes a mechanistic interpretability framework specifically tailored for LLM-as-a-Judge models. The methodology is robust, combining an eight-attack perturbation taxonomy to stress-test evaluation dimensions (Readability and Adequacy) with causal tracing, logit-lens vocabulary projection, and attention-head knockout. The design of a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity is a strong methodological choice, allowing for precise isolation of the mechanisms responsible for error detection versus quality assessment. The inclusion of a base-model control (Llama-3-8B) is particularly effective in distinguishing between pre-trained capabilities and fine-tuning-induced behaviors.
The experiments are well-structured, applying the four-experiment battery to two distinct judge models (Themis and Prometheus) based on different base architectures (Llama-3 and Mistral). The results clearly demonstrate a two-stage evaluation pipeline: local error comparison in lower layers and signal integration in higher layers. The finding that fine-tuning sculpts an existing substrate rather than building a new pipeline is a significant empirical contribution. The sharp crystallization of decisions in late layers (L=25/26) provides a clear, actionable insight into the model's decision-making process.
The authors release source code and data, which is a positive step. However, the specific implementation details of the "eight-attack perturbation taxonomy" and the exact parameters for the causal tracing and knockout experiments would need to be thoroughly documented in the released code to ensure full reproducibility. The use of open-weight models (Llama-3-8B, Mistral-7B) enhances reproducibility compared to proprietary models.
The study is limited to two specific judge models and two specific dimensions of NLG quality (Readability and Adequacy). It is unclear if these mechanistic insights generalize to other judge models or other evaluation dimensions like Fluency or Coherence. The scale of the models (7B-8B) may not reflect the behavior of larger, more capable judge models. The "controlled error intensity" in the generation pipeline might not fully capture the complexity of real-world summary errors.
This work has significant implications for the reliability and trustworthiness of LLM-based evaluation. By understanding the internal mechanisms of judges, researchers can better diagnose failures, improve judge training, and develop more robust evaluation protocols. The finding that fine-tuning modifies existing substrates rather than creating new ones has broader implications for how we understand the effect of instruction tuning on model capabilities. The paper provides a rigorous mechanistic analysis of LLM-as-a-Judge models, revealing a two-stage evaluation pipeline and the specific role of fine-tuning in sculpting this pipeline. This work significantly advances the interpretability of automated NLG evaluation, offering actionable insights for improving the reliability of LLM judges.
Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured combinations of symbols, such as logical formulas. However, the strongest modern AI systems are based on neural networks, which instead represent information in continuous vectors. Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas. How do they do it? In this work, we propose a potential answer: Despite appearances, perhaps the internal representations of neural networks implicitly realize symbolic structure. In support of this hypothesis, we show that the vector representations of a variety of neural networks can be closely approximated with symbolic structures: we can replace the network's entire representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remains largely unchanged. This finding holds for both small-scale neural networks trained to manipulate lists as well as large language models (LLMs) operating in four domains that are central in symbolic traditions: arithmetic, logic, computer code, and language. Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations, showing that the LLM's behavior is reliant on the symbolic structures we have identified. This work provides a potential way to reconcile longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI.
Primary: Johns Hopkins University
All Institutions: Johns Hopkins University, Yale University, New York University, Microsoft Research, Princeton University
The paper demonstrates that neural networks, including large language models, implicitly use Tensor Product Representations to encode symbolic structure. By introducing the DISCOVER method, the authors provide a rigorous framework for extracting and validating these structures, showing that they are not only present but causally responsible for model behavior in arithmetic, logic, coding, and language tasks.
The paper proposes DISCOVER (DISsecting COmpositionality in VEctor Representations), a method to test the hypothesis that neural networks implicitly realize symbolic structure via Tensor Product Representations (TPRs). The core technique involves training a parametric TPR model to approximate the internal vector representations of a target neural network (black-box). The success of this approximation is evaluated by feeding the TPR-generated vectors into the target network's decoder (or a trained period-unpacking model for LLMs) and measuring task accuracy. The methodology is rigorous, leveraging established cognitive science formalisms (TPRs) to provide a structured, interpretable alternative to the "Linear Representation Hypothesis." The use of "linearly-transformed TPRs" adds necessary flexibility to fit real-world neural network geometry.
The experiments are extensive, covering small-scale synthetic tasks (copying, reversing, interleaving lists) across four architectures (MLP, GRU, Transformer, Bottleneck Transformer) and seven large language models (Gemma-3, GPT-2-XL, GPT-OSS, Pythia, Qwen3, OLMo-2, Llama-3.1). The paper demonstrates that bidirectional role schemes consistently provide high-fidelity approximations of network representations, whereas bag-of-words (structureless) schemes fail. A particularly strong result is the intervention study, where editing the TPR representation (e.g., swapping subject/object roles) successfully alters the LLM's behavior in the expected way, confirming the causal relevance of the identified structure. The finding that DISCOVER approximations often outperform the actual noisy LLM representations in downstream decoding tasks is a significant empirical insight.
The authors provide a partial codebase on GitHub. The paper details the architecture specifications, training hyperparameters, and dataset generation procedures in the appendices. The use of open-weights LLMs from Hugging Face ensures that the models analyzed are accessible. However, the "partial" nature of the code and the complexity of the TPR fitting procedure may pose some barriers to exact reproduction without the full implementation details.
The analysis primarily focuses on the representation of the final token (period) in LLMs, which may not capture the full dynamics of intermediate layers or token-level representations for all tasks. The role schemes are hypothesized by the experimenter (supervised DISCOVER), which limits the method's ability to discover unknown structures autonomously. The paper acknowledges that TPRs are a specific formalism and does not rule out other potential symbolic structures, though it argues for the generality of TPRs.
This work bridges the gap between symbolic AI and connectionist AI, offering a concrete mechanism for how neural networks handle structured information. It provides a new tool for mechanistic interpretability, allowing researchers to extract interpretable, closed-form equations for network representations. This could facilitate better alignment, debugging, and control of LLMs by enabling precise interventions on symbolic features. It challenges the notion that neural networks are purely "bag-of-words" systems and reinforces the importance of structure in intelligent behavior. The paper demonstrates that neural networks, including large language models, implicitly use Tensor Product Representations to encode symbolic structure. By introducing the DISCOVER method, the authors provide a rigorous framework for extracting and validating these structures, showing that they are not only present but causally responsible for model behavior in arithmetic, logic, coding, and language tasks.
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Primary: Seoul National University
All Institutions: Seoul National University, Samsung Electronics Co., Ltd., NVIDIA
The paper introduces the HEAR benchmark and the A2R model, demonstrating that speaker-attributed reasoning in Speech Language Models can be enhanced by training on counterfactual audio that decouples linguistic content from speaker identity. This work provides a critical diagnostic tool for the field and a viable path toward more robust, acoustically-grounded multimodal reasoning, addressing a fundamental limitation in current SLMs.
The paper introduces a two-pronged approach: a diagnostic benchmark (HEAR) and a specialized model (A2R). The benchmark is hierarchical, targeting the specific failure mode of Speech Language Models (SLMs) in multi-speaker environments: the inability to distinguish between semantic priors and actual acoustic speaker cues. The core methodological innovation lies in the training data for A2R, specifically the "Counterfactual Audio with Speaker-level Hard negatives" (CASH) dataset. By constructing counterfactual audio where the linguistic content is decoupled from the speaker identity (likely via voice cloning or TTS), the authors force the model to rely on paralinguistic and acoustic features rather than textual context. This is a sophisticated data-centric approach to solving a reasoning problem. The 30B parameter scale of A2R suggests a significant computational investment, likely fine-tuning a large multimodal foundation model. The methodology is sound, directly addressing the identified gap in current SLM capabilities.
The evaluation is extensive, testing 20 leading SLMs on the new HEAR benchmark. The finding that these models rely on semantic priors is a significant empirical contribution, as it quantifies a known but previously under-diagnosed weakness. The performance of A2R is reported to be strong, with zero-shot generalization to downstream tasks. However, the provided text is a summary/abstract-like structure rather than the full body, so specific numerical comparisons (e.g., accuracy percentages, ablation studies on the CASH dataset components) are not visible. The claim of "zero-shot generalization" is a strong indicator of robustness, but without seeing the specific downstream tasks and baseline comparisons in detail, the magnitude of improvement is inferred from the abstract's confidence. The use of human-verified samples (2.4K) adds credibility to the benchmark's quality.
The authors state that all resources are available at the provided URL. The ethics statement indicates that synthetic waveform data is restricted to a gated repository due to voice cloning risks, which limits full reproducibility of the training data for the general public. However, code, evaluation protocols, and non-identifying annotations are public. This is a reasonable balance for this type of research, though it does hinder independent verification of the CASH dataset construction by external parties without access.
The primary limitation is the latency introduced by explicit reasoning (transcript generation) in A2R, which hinders real-time application. The authors acknowledge this and suggest implicit reasoning as future work. Additionally, the reliance on voice cloning for the CASH dataset raises ethical and legal concerns regarding consent, which the authors address with a strict Data Use Agreement, but this restricts the open availability of the core training data. The benchmark size (2.4K samples) is moderate; while high-quality, it may not capture the full diversity of real-world multi-speaker interactions.
This work has significant implications for the development of trustworthy multi-party conversational AI. By providing a benchmark that exposes the "semantic prior" bias in SLMs, it guides the field toward more robust acoustic grounding. The A2R model demonstrates that speaker attribution can be explicitly learned, which is crucial for applications in meeting assistants, collaborative robots, and accessible communication tools. The ethical framework provided for voice cloning research is also a valuable contribution to the community's standards. The paper introduces the HEAR benchmark and the A2R model, demonstrating that speaker-attributed reasoning in Speech Language Models can be enhanced by training on counterfactual audio that decouples linguistic content from speaker identity. This work provides a critical diagnostic tool for the field and a viable path toward more robust, acoustically-grounded multimodal reasoning, addressing a fundamental limitation in current SLMs.
Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.
Primary: The Broad Institute of MIT and Harvard
All Institutions: The Broad Institute of MIT and Harvard, Harvard Medical School, The Jackson Laboratory, Sutter Hill Ventures, David H. Koch Institute for Integrative Cancer Research, Howard Hughes Medical Institute, The Wyss Institute for Biologically Inspired Engineering at Harvard University, Harvard-MIT Program in Health Sciences and Technology, Yale School of Medicine, Department of Systems Biology, Department of Biology, Department of Immunology and Infectious Diseases, Department of Organismic and Evolutionary Biology
The paper introduces "science sandboxes," a novel framework for evaluating AI agents' scientific reasoning by distinguishing between quantitative optimization and qualitative rule discovery. Through rigorous experiments in regulatory genomics and protein folding, it demonstrates that while frontier agents can match human performance on familiar tasks, they struggle to infer underlying rules when faced with systems outside their pretraining priors, providing a critical benchmark for the development of truly scientific AI agents.
The paper introduces "science sandboxes," a framework for evaluating AI agents' scientific reasoning capabilities by distinguishing between quantitative optimization and qualitative rule discovery. The methodology is robust, defining a clear experimental loop (specimens, assays, oracle) and categorizing oracles into "wet" (physical), "damp" (computational models), and "dry" (invented rules). The implementation of two specific sandboxes, MPRAbox (regulatory genomics) and CodonBox (protein folding/genetic code inference), is well-designed. MPRAbox uses a damp oracle based on the Malinois model to simulate MPRA experiments, while CodonBox uses a dry oracle with hidden, non-biological rules to test generalization beyond prior knowledge. The evaluation protocol includes both single-round and multi-round (long-horizon) tests, allowing for the assessment of hypothesis revision over time.
The experiments are extensive and insightful. The authors evaluate frontier agents (Claude Opus 4.7, GPT-5.5, Gemini 3.5 Flash) against human-designed baselines. Key findings include: (1) Agents can match or exceed human performance on standard biological tasks (MPRAbox) by leveraging pretraining priors. (2) Performance drops significantly when agents are forced to discover rules outside their biological priors (dry oracles in MPRAbox and CodonBox). (3) Multi-round experiments reveal that agents often engage in brute-force search rather than structured hypothesis testing, and they struggle to infer complex combinatorial rules (e.g., interactions between codon positions) when the search space grows. The qualitative analysis of agent "lab notebooks" provides strong evidence for the distinction between score optimization and true understanding.
The paper provides detailed descriptions of the sandbox mechanics, oracle rules (Table 3), and experimental protocols. However, as an arXiv preprint, the code and specific agent configurations are not yet publicly available. The reliance on specific frontier models (which change rapidly) limits long-term reproducibility, but the framework itself is clearly defined enough for others to implement similar sandboxes.
The evaluation is limited to two biological domains (genomics and protein folding), which may not generalize to all scientific fields. The "dry" oracles, while useful for testing generalization, are artificial and may not reflect the complexity of real-world scientific discovery. The paper focuses on a small number of frontier agents, and the results may not apply to smaller or specialized models. Additionally, the qualitative evaluation of lab notebooks is currently manual, though the authors suggest automation is possible.
This paper has high potential impact on the AI for Science community. It provides a rigorous framework for measuring "scientific capability" beyond mere benchmark scores, which is a critical gap in current AI evaluation. The findings highlight the limitations of current LLMs in genuine scientific induction, guiding future research in agent design and training. The concept of "science sandboxes" is likely to be adopted as a standard evaluation method for scientific AI agents. The paper introduces "science sandboxes," a novel framework for evaluating AI agents' scientific reasoning by distinguishing between quantitative optimization and qualitative rule discovery. Through rigorous experiments in regulatory genomics and protein folding, it demonstrates that while frontier agents can match human performance on familiar tasks, they struggle to infer underlying rules when faced with systems outside their pretraining priors, providing a critical benchmark for the development of truly scientific AI agents.
Generalizable embodied manipulation remains difficult to achieve through pretraining alone, due to unseen physical conditions in the real world. We argue that robots need to learn from their own physical interactions on the fly during real-world deployment and use this knowledge to inform subsequent actions. We present Zeva, the first framework that enables in-context learning from a robot's own physical interaction experience while keeping the policy model frozen. Zeva employs a Causal Interaction Extractor to encode an executed action and its induced state change into a causal interaction signal, which is stored in a dual-timescale causal memory. For subsequent actions, relevant causal interaction signals are retrieved from memory and injected into the frozen policy model as context. Experiments in simulation and real-world manipulation demonstrate that Zeva achieves the best performance among the compared frontier VLAs and WAMs and, more importantly, enables self-evolution during deployment without gradient updates. Its success rate continues to improve as the robot accumulates interaction experience. Furthermore, the acquired interaction experience can generalize across tasks.
Primary: Institute for AI Industry Research (AIR), Tsinghua University
All Institutions: Institute for AI Industry Research (AIR), Tsinghua University, Z-Trans AI
Zeva introduces a novel in-context causal learning framework that enables frozen robot policies to self-improve through accumulated interaction memory. The paper demonstrates that by extracting and retrieving causal interaction signals from past attempts, robots can significantly enhance their manipulation success rates in real-world settings without gradient updates, offering a scalable solution to the distribution shift problem in embodied AI.
The paper proposes Zeva, a framework for In-Context Causal Learning (ICCL) in embodied manipulation. The core innovation is the decoupling of policy adaptation from parameter updates. Instead of fine-tuning the Vision-Language-Action (VLA) model, Zeva extracts "causal interaction signals" from the robot's own execution history using a Causal Transition Encoder (CTE). These signals are stored in a dual-timescale memory: a Brief Interaction Trace (BIT) for immediate context and a Persistent Interaction Memory (PIM) for cross-attempt learning. The retrieved causal context is injected into the frozen policy's attention layers as a prompt. This approach allows the robot to learn from failures and successes in real-time without the computational cost or risk of catastrophic forgetting associated with gradient-based updates. The method is conceptually strong, leveraging the "test-time scaling" paradigm by treating interaction history as a form of in-context learning data.
The experiments are conducted on both simulation (RoboCasa365-Atomic5) and a real-world chemical lab benchmark (ChemLab-Evo). The real-world setup is particularly impressive, involving complex multi-stage tasks like titration and extraction. Zeva outperforms strong baselines like OpenVLA, Fast-WAM, and Cosmos3-Nano. The ablation studies convincingly demonstrate the necessity of both memory streams (BIT and PIM). The "Post-Deployment In-Context Scaling" experiments show monotonic improvement in success rates as the robot accumulates experience, validating the core hypothesis. The cross-task generalization analysis, where causal signals from one task successfully inform another, provides strong evidence that the CTE is learning functional physical dynamics rather than just memorizing task-specific trajectories.
The paper provides a clear algorithmic description and detailed architectural diagrams. However, specific hyperparameters for the CTE training, the exact structure of the "Cosmos3" backbone (which appears to be a proprietary or internal model given the citation style), and the implementation details of the memory retrieval mechanism are not fully specified in the text provided. The reliance on a specific, potentially non-public foundation model (Cosmos3) may limit immediate reproducibility for the broader community, although the framework itself is modular.
The primary limitation is the reliance on a specific foundation model (Cosmos3) which may not be publicly available, hindering independent verification. The method assumes that the CTE can be pre-trained or fine-tuned to extract meaningful causal signals; the paper does not deeply explore the cost or data requirements for training this encoder. Additionally, the "causal" label is somewhat aspirational; the method learns correlations between actions and state changes, but true causal inference (counterfactuals) is not explicitly demonstrated. The evaluation is limited to a single robot platform (ARX) and a specific set of lab tasks, so generalizability to other embodiments or highly dynamic environments remains to be seen.
This work addresses a critical bottleneck in embodied AI: the gap between pre-training data and real-world deployment conditions. By enabling self-evolution through in-context learning, Zeva offers a path toward robots that can adapt to new environments without retraining. This has significant implications for industrial automation and service robotics, where retraining models for every new task or environment is impractical. The concept of "causal memory" could be extended to other domains requiring long-horizon planning and adaptation. Zeva introduces a novel in-context causal learning framework that enables frozen robot policies to self-improve through accumulated interaction memory. The paper demonstrates that by extracting and retrieving causal interaction signals from past attempts, robots can significantly enhance their manipulation success rates in real-world settings without gradient updates, offering a scalable solution to the distribution shift problem in embodied AI.
Embodied navigation requires agents to translate heterogeneous goals and visual observations into actions across tasks, environments, and robot embodiments. Modern vision-language models (VLMs) already encode spatial priors for visual grounding, spatial reasoning, and pointing, but these capabilities are rarely elicited directly for robot control. Existing navigation systems instead rely on task- or embodiment-specific components, fragmenting perception, reasoning, and action while offering limited generalization. Here we present LightNav-0, a compact generalist embodied navigation model that elicits the spatial intelligence of a pretrained VLM and aligns it with navigation, without task-specific prediction heads. LightNav-0 represents diverse navigation tasks through a unified token interface: dual-channel pointing expresses task-, scene-, and embodiment-agnostic spatial intent, while a residual vector-quantized action tokenizer maps this intent to precise, embodiment-specific trajectories. Together with temporally aware visual history compression, ER mid-training, supervised fine-tuning, and reinforcement learning, this formulation supports instruction following, open-vocabulary object navigation, and visual tracking within a single model. The navigation training corpus spans 2K+ scenes and 4K+ hours of embodied navigation data. LightNav-ER, the embodied-reasoning checkpoint used to initialize LightNav-0, attains the highest complete-set average across 8 embodied-reasoning benchmarks, while LightNav-0 achieves state-of-the-art monocular success rates across all 10 public navigation simulation settings. Real-world evaluations further demonstrate zero-shot generalization across robot embodiments, diverse scenes, and static and dynamic targets. These results establish compact VLMs as a unified and transferable backbone for generalist embodied navigation.
Primary: Light Origins
All Institutions: Light Origins, LimX Dynamics, Manycore Tech
LightNav-0 establishes a unified framework for generalist embodied navigation by eliciting spatial intelligence from VLMs via a dual-channel pointing and RVQ action tokenizer. The paper demonstrates strong technical depth through its multi-stage training pipeline and achieves state-of-the-art results across diverse simulation benchmarks and real-world robot embodiments, validating the feasibility of using compact VLMs as a transferable backbone for navigation tasks.
The paper proposes LightNav-0, a framework that unifies embodied navigation tasks (instruction following, object goal, visual tracking) by leveraging the spatial priors of a pretrained Vision-Language Model (VLM). The core methodological contribution is a unified token interface that replaces task-specific heads with "dual-channel pointing" for spatial intent and a residual vector-quantized (RVQ) action tokenizer for embodiment-specific trajectories. This approach is supported by a multi-stage training recipe involving Embodied Reasoning (ER) mid-training, supervised fine-tuning, and reinforcement learning. The use of RVQ for action discretization is a strong technical choice that allows the model to handle continuous trajectory spaces within an autoregressive VLM framework.
The experimental evaluation is extensive, claiming state-of-the-art monocular success rates across 10 public simulation settings. The training corpus is substantial (2K+ scenes, 4K+ hours). The inclusion of real-world zero-shot generalization across different robot embodiments (humanoid, quadruped, aerial, wheeled) is a significant strength, as it validates the "embodiment-agnostic" claim of the spatial intent representation. The comparison against prior methods in the teaser figure suggests clear performance gains.
The paper provides links to a GitHub repository and Hugging Face model weights, which significantly aids reproducibility. However, the specific details of the "ER mid-training" data composition and the exact hyperparameters for the RL stage are not fully detailed in the provided text, which may limit full reproduction without the code.
The primary limitation is the reliance on a specific VLM backbone; the transferability to other VLM architectures is not fully explored. Additionally, the "compact" nature of the model is relative, and inference latency on edge devices for real-time control is not explicitly quantified in the provided text. The "zero-shot" real-world results, while impressive, may rely on specific calibration or mapping layers not fully detailed.
This work has high potential impact by demonstrating that generalist VLMs can serve as effective backbones for low-level robot control without extensive task-specific engineering. It bridges the gap between high-level semantic reasoning and low-level motor control, potentially accelerating the development of general-purpose robots. LightNav-0 establishes a unified framework for generalist embodied navigation by eliciting spatial intelligence from VLMs via a dual-channel pointing and RVQ action tokenizer. The paper demonstrates strong technical depth through its multi-stage training pipeline and achieves state-of-the-art results across diverse simulation benchmarks and real-world robot embodiments, validating the feasibility of using compact VLMs as a transferable backbone for navigation tasks.
Traversing sparse 3D structures requires humanoid robots to perceive thin, overhanging geometry while executing agile, accurate whole-body motions. We study this problem through monkey-bar traversal, where the robot must jump to the structure, traverse it through sparse bar interactions, and land safely. For this task, we present a reinforcement-learning-based perceptive control system that operates directly on observations from a head-mounted solid-state lidar. To extract task-relevant geometry from the sparse returns, the policy consumes the raw lidar scan through an attention-based encoder with recurrent memory. This policy is obtained by a phase-scheduled teacher- student pipeline that combines privileged experts for jumping up, brachiating, and jumping down. For transfer to hardware, we model lidar noise, battery-voltage sag, and actuator thermal limits, and equip the humanoid with passive hook end-effectors for robust bar interaction. On hardware, the resulting policy completes the full jump-up->brachiation->jump-down sequence in 14 of 15 trials across three bar configurations and reaches brachiation speeds up to 0.5 m/s. Beyond brachiation, the same perception backbone supports a separately trained policy that ducks beneath thin overhead obstacles with 2 cm cross-sections.
Primary: ETH Zurich
All Institutions: ETH Zurich, ETH AI Center
The paper presents a novel perceptive control system enabling humanoids to traverse sparse 3D structures via brachiation using raw lidar. It contributes a rigorous sim-to-real framework that models battery sag, thermal limits, and sensor-specific noise, achieving high success rates on hardware and demonstrating the viability of map-free perception for agile whole-body control.
The paper proposes a reinforcement learning framework for humanoid brachiation (monkey bar traversal) that operates directly on raw solid-state lidar data, bypassing intermediate map representations like elevation maps or voxel grids. The core methodological contribution is a phase-scheduled teacher-student distillation pipeline. Three privileged teachers (for jumping up, brachiating, and jumping down) are distilled into a single student policy that uses an attention-based encoder (adapted from AME-2) and recurrent memory to process sparse lidar returns. A key technical strength is the rigorous sim-to-real transfer modeling, which explicitly accounts for battery voltage sag, actuator thermal limits, and specific lidar noise artifacts (ray divergence/edge bleed). The use of passive hook end-effectors is a clever hardware simplification that reduces control complexity.
The experiments are robust and well-structured. The authors provide ablations on the encoder architecture and auxiliary losses, demonstrating the necessity of the attention-based approach for sparse geometry. Sim-to-sim validation in MuJoCo with a high-fidelity lidar model precedes hardware testing. On hardware, the robot achieves a 93% success rate (14/15 trials) on the full jump-up/brachiation/jump-down sequence across three different bar configurations. The demonstration of a secondary task (ducking under thin obstacles) using the same perception backbone further validates the generalizability of the perceptual encoder.
The paper provides detailed descriptions of the reward functions, curriculum, and sensor noise models. Specific parameters for the battery and thermal models are discussed, and the lidar noise model is described with sufficient detail (beam divergence, edge dropout probabilities) to be replicated. The use of standard tools like IsaacLab and RSL-RL aids reproducibility, though the specific hardware modifications (passive hooks) and sensor mounting require physical access to the PM-01 platform.
The system is currently limited to a small set of separately trained policies for specific tasks (brachiation, ducking). The paper acknowledges that robustness to substantially more diverse geometry remains to be demonstrated. The reliance on a specific solid-state lidar (RoboSense E1R) and the specific hardware platform (PM-01) limits immediate generalizability to other humanoid platforms without re-tuning the sim-to-real models.
This work pushes the boundary of what is possible with legged robots in unstructured, sparse environments. By demonstrating that raw lidar can support explosive, contact-accurate motions, it challenges the prevailing paradigm of using dense intermediate maps for locomotion. The techniques for modeling actuator thermal limits and battery sag are broadly applicable to any high-power robotic system operating near its physical limits. The paper presents a novel perceptive control system enabling humanoids to traverse sparse 3D structures via brachiation using raw lidar. It contributes a rigorous sim-to-real framework that models battery sag, thermal limits, and sensor-specific noise, achieving high success rates on hardware and demonstrating the viability of map-free perception for agile whole-body control.
We present $\mathcal{N}_0$-Foundation, a paradigm for tactile-enabled embodied manipulation, which integrates tactile sensing hardware, large-scale multimodal data, tactile representation learning, and standardized evaluation. First, we engineer the infrastructure for scalable data collection, including a vision-based tactile sensor, a tactile Universal Manipulation Interface (UMI), and a synchronized visuo-tactile data collection system supporting both robot embodiments and UMI-based demonstrations. Leveraging this infrastructure, we construct NeoData, which contains more than 30000 hours of synchronized visual and tactile demonstrations, spanning six embodiments, 450 tasks, and billions of paired RGB and tactile frames collected through a mixture of real-robot teleoperation and UMI-based demonstrations. To facilitate open research, we further release OpenNeoData, a 5000-hour open-source subset of NeoData. The dataset addresses a central limitation of existing manipulation corpora, critical for deformable-object manipulation, precise assembly, delicate force control, and sustained surface interaction. Capitalizing on the large-scale, heterogeneous tactile measurements, we propose NeoForce, a visuo-tactile representation model that learn transferable tactile representations across different sensor designs. To enable systematic evaluation of tactile embodied models built upon our infrastructure, datasets and tactile representations, we further propose a comprehensive benchmark, which combines the real-world NeoReal suite and the simulated NeoSim suite for standardized evaluation. Experiments across both suites show that policies benefit from the physical contact state rather than from the device-specific appearance of the tactile signal. We release the dataset, the representation, and the benchmark, aiming at supporting future work on tactile-enabled embodied manipulation.
Primary: Fudan University
All Institutions: Fudan University, NeoteAI
The paper introduces a large-scale tactile dataset, a transferable visuo-tactile representation model, and a comprehensive benchmark, establishing a foundational infrastructure for tactile-enabled embodied manipulation. By addressing the critical data scarcity in tactile sensing and providing a standardized evaluation framework, the work enables significant progress in dexterous robotics and force control, offering a robust baseline for future research in multimodal manipulation.
The paper proposes a comprehensive paradigm for tactile-enabled embodied manipulation, comprising three main technical pillars: (1) Hardware Infrastructure: The development of a vision-based tactile sensor and a "tactile UMI" (Universal Manipulation Interface) that allows for synchronized visuo-tactile data collection from both robot embodiments and human demonstrations. This addresses the scarcity of high-quality tactile data by enabling scalable collection via low-cost, portable interfaces. (2) Data & Representation: The construction of NeoData (30,000+ hours) and the release of OpenNeoData (5,000 hours). The core algorithmic contribution is NeoForce, a visuo-tactile representation model designed to learn transferable tactile features across heterogeneous sensor designs. This is crucial because tactile signals are often device-specific; learning a canonical representation allows policies to generalize across different hardware. (3) Benchmarking: The introduction of NeoReal (real-world) and NeoSim (simulated) suites for standardized evaluation. The methodology is sound, leveraging the scale of the data to train robust representations, and the focus on transferability is a significant methodological advance over previous work that often treated tactile data as modality-specific noise.
The experimental section evaluates policies trained on the proposed representations across both real-world and simulated environments. Key findings indicate that policies benefit from the physical contact state encoded in the tactile representation rather than the device-specific appearance of the raw signal, validating the transferability claim. The scale of the dataset (billions of paired frames) is a major strength, providing a resource that was previously unavailable. The evaluation covers a wide range of tasks (450 tasks, 6 embodiments), demonstrating the generality of the approach. However, the specific quantitative gains over strong baselines (such as vision-only policies or non-transferable tactile policies) need to be scrutinized for statistical significance and practical relevance. The comparison with existing tactile datasets (like T-DEX or similar) is implied but should be explicit to highlight the unique value of the scale and modality pairing.
The paper explicitly commits to releasing OpenNeoData (5,000 hours), the NeoForce representation model, and the benchmark suites. This is a high level of reproducibility, especially given the difficulty of collecting tactile data. The hardware design details for the tactile UMI and sensor are provided, allowing other labs to replicate the data collection infrastructure. The code for the representation learning and policy training is expected to be released, which is standard for such large-scale contributions. The only barrier to full reproducibility is the cost and effort of building the custom hardware, but the open-source data mitigates this for algorithmic research.
The primary limitation is the hardware dependency; while the data is open, the specific tactile sensors used may not be easily accessible to all researchers, potentially limiting the immediate adoption of the full pipeline. The "tactile UMI" is a novel device, and its calibration and maintenance requirements are not fully detailed in the abstract. Additionally, the computational cost of processing billions of tactile frames and training large-scale multimodal models is significant. The paper may also face challenges in generalizing to tactile sensors with fundamentally different sensing principles (e.g., capacitive vs. vision-based), as the current focus is on vision-based tactile sensing.
This work has the potential to significantly accelerate research in dexterous manipulation and force-sensitive robotics. By providing a large-scale, open dataset and a transferable representation, it lowers the barrier to entry for developing tactile-aware policies. It bridges the gap between vision-centric manipulation and the need for physical interaction feedback. The standardization of benchmarks (NeoReal/NeoSim) will facilitate fair comparison of future tactile manipulation algorithms. This could lead to more robust and versatile robots capable of handling deformable objects and performing precise assembly tasks, which are currently difficult for vision-only systems. The paper introduces a large-scale tactile dataset, a transferable visuo-tactile representation model, and a comprehensive benchmark, establishing a foundational infrastructure for tactile-enabled embodied manipulation. By addressing the critical data scarcity in tactile sensing and providing a standardized evaluation framework, the work enables significant progress in dexterous robotics and force control, offering a robust baseline for future research in multimodal manipulation.
Conventional flow-based vision-language-action (VLA) models support expressive continuous action generation but rely on multi-step refinement to produce each action chunk, increasing latency in online robot control. To address this issue, we introduce DriftingVLA, a native one-step VLA that generates a complete action chunk with a single action-expert forward pass. Rather than learning a flow field that requires iterative integration at inference, DriftingVLA uses a distribution-drifting objective to learn a direct noise-to-action-chunk mapping for one-step deployment. Since robot action dimensions carry distinct control semantics and distributional characteristics, we further introduce Per-Dimension Temporal Drifting (PDTD). PDTD treats the complete temporal trajectory of each action dimension as a separate drifting unit, enabling finer-grained modeling and shaping of dimension-specific action distributions. This per-dimension decomposition applies only to the training objective; the shared VLA model still generates the complete action chunk jointly, thereby preserving cross-dimensional dependencies. DriftingVLA achieves 98.32% success on LIBERO, 81.09% on RoboTwin 2.0, and 77.67% across six real-world single- and dual-arm tasks, outperforming the evaluated multi-step flow policy and one-step VLA baselines. Native one-step deployment also delivers a 3.36-fold speedup in action-chunk generation, eliminating iterative refinement without sacrificing control performance.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, LYNSENSE
DriftingVLA introduces a native one-step VLA generation method using Per-Dimension Temporal Drifting to achieve significant latency reduction without sacrificing performance. The paper presents a solid engineering and algorithmic contribution to the VLA field, offering a practical solution to the inference latency problem inherent in flow-based models, with strong empirical results on both simulation and real-world robotic tasks.
The paper proposes DriftingVLA, a Vision-Language-Action (VLA) model that replaces the iterative refinement of flow matching with a native one-step generation approach. The core innovation is the "distribution-drifting" objective, which trains a direct mapping from noise to action chunks, bypassing the need for multi-step numerical integration at inference. A key technical contribution is Per-Dimension Temporal Drifting (PDTD), which decomposes the training objective by treating the temporal trajectory of each action dimension as a separate drifting unit. This allows for finer-grained modeling of distinct control semantics (e.g., position vs. orientation) while maintaining a shared model architecture that generates the full action chunk jointly, thus preserving cross-dimensional dependencies. The method effectively addresses the latency bottleneck in online robot control by reducing the action-expert forward passes to one per chunk.
The experimental evaluation is robust, covering simulation benchmarks (LIBERO, RoboTwin 2.0) and real-world tasks (six single- and dual-arm tasks). The reported success rates are high: 98.32% on LIBERO, 81.09% on RoboTwin 2.0, and 77.67% on real-world tasks. The paper claims outperformance over both multi-step flow policies and other one-step VLA baselines. A significant practical advantage is demonstrated with a 3.36-fold speedup in action-chunk generation, which is critical for real-time robotic applications. The inclusion of real-world experiments strengthens the claim of practical utility.
The paper provides a clear description of the PDTD mechanism and the training objective. However, as an arXiv preprint without a specified code release in the provided text, full reproducibility depends on the availability of the codebase, which is not explicitly linked in the abstract or header. The architectural details of the "action-expert" and the specific hyperparameters for the drifting objective would be necessary for exact replication.
The evaluation is limited to specific benchmarks (LIBERO, RoboTwin) and a small set of real-world tasks. It is unclear how the method generalizes to more complex, long-horizon tasks or different robot embodiments beyond the tested arms. The "native one-step" approach may sacrifice some of the expressiveness or robustness of multi-step flow matching in highly stochastic or complex environments, though the paper claims no sacrifice in control performance. The lack of ablation studies on the specific impact of PDTD versus standard one-step distillation is a potential gap.
This work contributes to the trend of making VLA models more efficient and deployable in real-time settings. By eliminating iterative refinement, it lowers the computational barrier for high-frequency control, potentially enabling the use of larger VLA models on embedded hardware. The per-dimension decomposition strategy could inspire similar fine-grained modeling approaches in other continuous control domains. DriftingVLA introduces a native one-step VLA generation method using Per-Dimension Temporal Drifting to achieve significant latency reduction without sacrificing performance. The paper presents a solid engineering and algorithmic contribution to the VLA field, offering a practical solution to the inference latency problem inherent in flow-based models, with strong empirical results on both simulation and real-world robotic tasks.
Video world models increasingly serve as data engines, action planners, and simulators for embodied AI, but conventional embodied world model (EWM) benchmarks lack a unified 3D-grounded protocol for establishing whether generated rollouts preserve the underlying 3D scene state or translate into executable actions. We introduce RoboPhys-3D, a 3D-grounded EWM benchmark built on RoboTwin 2.0, covering 50 manipulation tasks across four regimes, with 5,000 episodes and 25,000 multi-view ground-truth videos. A defining feature of RoboPhys-3D is that generated and ground-truth videos are processed through the same 3D reconstruction pipeline, enabling reconstruction-induced error to be distinguished from generation-induced error. The RoboPhys-3D benchmark organizes 50 complementary metrics into 18 sub-dimensions across four levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. We further introduce Average Full Score, a hierarchical score averaging all 50 metrics for comprehensive evaluation, and RoboPhyscore, a compact task-aligned score averaging the metrics most strongly correlated with task success. Among the four representative video world models, Cosmos 3 achieves the highest RoboPhyscore (0.6330, 92.7% of ground truth), while state- and execution-grounded metrics reveal substantial failures that perceptual and vision-language model-based judgments fail to capture. RoboPhyscore further exhibits strong agreement with human evaluation (Pearson r = 0.9761 and Spearman \r{ho} = 0.8962), demonstrating the importance of grounded, execution-aware evaluation for EWM capability.
Primary: Purdue University
All Institutions: Purdue University, The University of Texas at Austin
The paper presents a rigorous 3D-grounded benchmark for evaluating embodied world models, effectively disentangling generation errors from reconstruction artifacts. By introducing a comprehensive metric suite and a task-aligned score (RoboPhyscore) that strongly correlates with human judgment and execution success, it provides a valuable tool for advancing the reliability of video-based robotic planning.
The paper introduces RoboPhys-3D, a benchmark for Embodied World Models (EWMs) that distinguishes itself by grounding evaluation in 3D reconstruction. The core methodological innovation is the "reconstruction-matched protocol," where both generated and ground-truth videos are processed through the same 3D reconstruction pipeline (e.g., VGGT, 4DGS). This allows the authors to disentangle errors caused by the video generation model from errors introduced by the 3D reconstruction process itself, a significant confound in previous benchmarks. The evaluation framework is extensive, comprising 50 metrics organized into four hierarchical levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. The introduction of "RoboPhyscore," a compact score derived from metrics strongly correlated with task success, is a practical contribution aimed at simplifying evaluation while maintaining alignment with human judgment and execution success.
The experiments are rigorous and comprehensive. The dataset covers 50 manipulation tasks across four regimes with 5,000 episodes and 25,000 multi-view videos. The authors evaluate four representative video world models (Cosmos 3, Wan 2.2, CogVideoX, RoboDreamer) and multiple reconstruction methods. Key findings include that Cosmos 3 achieves the highest RoboPhyscore (0.6330), but state- and execution-grounded metrics reveal substantial failures that perceptual metrics miss. The correlation analysis shows strong agreement between RoboPhyscore and human evaluation (Pearson r = 0.9761), validating the metric selection. The ablation on prompt specificity and IDM choice further demonstrates the sensitivity of the benchmark to conditioning and action decoding, providing actionable insights for practitioners.
The paper provides detailed supplementary information on dataset statistics, model configurations, and the specific prompts used for captioning. The normalization strategies for the 50 metrics are explicitly defined, including the bounds used for affine mapping. The human study protocol is well-documented, including participant recruitment, interface details, and aggregation methods. However, the code and dataset are not explicitly linked in the provided text (no URLs found), which may hinder immediate reproducibility unless released separately. The reliance on specific simulator (RoboTwin 2.0) and reconstruction pipelines requires access to these tools.
The benchmark is heavily dependent on the quality of the 3D reconstruction pipeline; if the reconstruction method fails, the evaluation metrics may be skewed, although the paper attempts to mitigate this by comparing reconstruction-induced vs. generation-induced errors. The evaluation is limited to the RoboTwin 2.0 simulator environment, which may not fully generalize to real-world robotic manipulation with different dynamics or sensor noise. The use of VLMs (Qwen3-VL) for some metrics introduces potential biases inherent to the VLM's training data. The computational cost of running 50 metrics across multiple models and reconstruction methods is likely high, potentially limiting adoption for rapid prototyping.
This paper addresses a critical gap in the evaluation of video world models for robotics. By providing a 3D-grounded protocol, it enables more accurate assessment of whether generated videos represent physically plausible and executable states. The findings that perceptual metrics often fail to capture execution-level failures will likely influence how the community designs future EWM benchmarks. The high correlation with human judgment suggests that RoboPhyscore could become a standard proxy for human evaluation in this domain. The work bridges the gap between computer vision (video generation) and robotics (action planning), promoting more integrated evaluation frameworks. The paper presents a rigorous 3D-grounded benchmark for evaluating embodied world models, effectively disentangling generation errors from reconstruction artifacts. By introducing a comprehensive metric suite and a task-aligned score (RoboPhyscore) that strongly correlates with human judgment and execution success, it provides a valuable tool for advancing the reliability of video-based robotic planning.
Caching is widely used across the system stack to improve performance and efficiency, with eviction algorithms at its core. Existing cache eviction policies fall into two broad categories: static heuristics (e.g., 2Q, S3-FIFO) and smart algorithms (e.g., ARC, LRB). Smart caches can adapt to workloads and have the potential to achieve higher efficiency and robustness than static heuristics. However, we find that existing smart caches suffer from objective mismatches and instability. We introduce Learning-Augmented Heuristics (LAH), a framework that learns the cache-level parameters of static heuristics. By decoupling the data and control planes, LAH supports simple, high-speed data reads and writes on the data plane, while performing occasional asynchronous learning on the control plane using cache-level features. We demonstrate the effectiveness of LAH through S4-FIFO, a Smart S3-FIFO cache eviction algorithm. We pre-train a single model on 4,140 production traces and embed it in S4-FIFO to learn optimal cache parameters. On 1,035 evaluation traces, S4-FIFO improves the mean efficiency by 26% compared to S3-FIFO and by 8% compared to 3L-Cache, the best state-of-the-art algorithm. S4-FIFO is also robust---increasing miss ratio over FIFO by 0.8% on the worst trace, whereas 3L-Cache increases FIFO's miss ratio by 8.8%. Finally, S4-FIFO's decisions are also interpretable: a language model can provide a rationale for why a particular configuration was chosen.
Primary: Harvard University
All Institutions: Harvard University, University of Illinois Urbana-Champaign (UIUC), University of Chicago, Institut Teknologi Bandung, Meta
The paper introduces a robust and interpretable framework for learning-augmented cache eviction that significantly outperforms state-of-the-art methods in both efficiency and worst-case stability. By decoupling the data and control planes, it enables the practical deployment of ML in latency-sensitive systems, offering a compelling alternative to complex adaptive algorithms.
The paper proposes the Learning-Augmented Heuristics (LAH) framework, which addresses the trade-off between the simplicity of static cache eviction heuristics (like S3-FIFO) and the adaptability of smart algorithms (like ARC). The core innovation is the decoupling of the data plane (fast, simple heuristic execution) from the control plane (asynchronous learning). Instead of learning complex per-item policies, the model learns to tune a small set of semantic parameters (knobs) of a static heuristic based on cache-level features. This approach is clever because it preserves the low-latency guarantees of static heuristics while gaining the adaptivity of ML. The instantiation, S4-FIFO, uses a pre-trained model on 4,140 production traces to predict optimal parameter configurations. The use of a Language Model (LM) for interpretability is a novel addition, allowing the system to explain *why* a specific configuration was chosen, which is rare in systems ML literature.
The evaluation is extensive, utilizing 1,035 evaluation traces. The results show a 26% improvement in mean efficiency over S3-FIFO and an 8% improvement over 3L-Cache (state-of-the-art). Crucially, the paper demonstrates robustness: S4-FIFO increases the miss ratio over FIFO by only 0.8% on the worst trace, whereas 3L-Cache increases it by 8.8%. This robustness metric is critical for production systems where worst-case performance matters as much as average performance. The throughput matches that of pure heuristics, validating the data/control plane decoupling.
The paper mentions pre-training on 4,140 production traces and evaluation on 1,035 traces. While the specific dataset names are not listed in the provided text snippet, the scale suggests the use of standard public benchmarks (e.g., Web, Cloud, or specific industry datasets often shared in OSDI papers). The lack of explicit code links in the provided text is a minor negative, but the detailed description of the framework and the standard nature of the baselines (S3-FIFO, ARC, 3L-Cache) make reproduction feasible for a skilled systems researcher.
The primary limitation is the dependency on the quality and diversity of the pre-training traces. If the production traces do not cover the distribution of the target workload, the learned parameters may be suboptimal. Additionally, the "interpretability" via LM is a post-hoc explanation rather than an intrinsic property of the decision process, which may limit its utility in safety-critical contexts. The framework is specific to parameter tuning of heuristics and may not generalize to entirely new eviction logic.
This work bridges the gap between ML and high-performance systems by showing that ML does not need to replace heuristics but can augment them. This paradigm is likely to be adopted in other system components (e.g., scheduling, memory management) where latency is critical. The robustness results are particularly significant for cloud providers and data centers, where cache efficiency directly impacts cost and performance. The paper introduces a robust and interpretable framework for learning-augmented cache eviction that significantly outperforms state-of-the-art methods in both efficiency and worst-case stability. By decoupling the data and control planes, it enables the practical deployment of ML in latency-sensitive systems, offering a compelling alternative to complex adaptive algorithms.