Last 7 Days (October 25 – October 31, 2026)
While progress in informal mathematics has come largely from scaling the chain-of-thought and agentic reasoning of frontier language models, formal theorem proving has increasingly relied on human-designed workflows around these models, such as planning, decomposition, and orchestrating specialized provers. This raises a bitter-lesson question: can a simple agentic loop that scales test-time compute match these workflows? The answer matters as formal verification must scale to keep up with frontier models producing research mathematics, yet new workflows rarely settle it, as their gains are often confounded with stronger models or more compute. We study this question with h1, a simple harness that gives a model an efficient interface to Lean 4 and keeps it working through verifier feedback and self-reflection, without imposing a decomposition or search structure. h1 reaches state-of-the-art results on PutnamBench, FATE-X, and FormalConjectures100-Solved, settles eight open problems in FormalConjectures, and, at matched models, matches or outperforms specialized workflows such as Goedel-Architect and AlphaProof Nexus, surprisingly showing that a single attempt (pass@1) can often scale to solve open problems. Through ablations and matched-model comparisons with Claude Code and AxProverBase, we find that simplicity alone is not enough: an efficient Lean interface largely determines cost and solve rate. We thus recommend that future work evaluate frontier models with efficient execution feedback before prescribing workflows, at the risk of bitter lessons.
Primary: Google DeepMind
All Institutions: Google DeepMind, University of Edinburgh
The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
The paper proposes "h1," a minimalist agentic harness for Lean 4 theorem proving that relies on a simple ReAct-style loop with an efficient interface to the Lean compiler, library search tools, and a reflection mechanism to compact context. The core methodological contribution is not a new model architecture or training algorithm, but a rigorous empirical demonstration that "bitter lesson" principles apply to harness design: that scaling test-time compute in a simple loop with good tooling outperforms complex, human-designed workflows (like planning, decomposition, or multi-agent orchestration) when using the same frontier models. The methodology is sound, focusing on isolating the variable of harness complexity while controlling for model capability and inference cost.
The experimental evaluation is extensive and high-quality. It covers a wide range of benchmarks including PutnamBench, FATE-X, FormalConjectures, and IMO 2026. The paper provides matched-model comparisons against state-of-the-art specialized systems (Goedel-Architect, AlphaProof Nexus, MerLean, etc.), which is crucial for a fair assessment. The results show that h1 achieves state-of-the-art performance, solving open problems and matching or exceeding specialized workflows at lower cost. The analysis of cost-efficiency curves (pass@1 vs pass@k) is particularly insightful, showing that longer single attempts can be more efficient than multiple shorter ones.
The paper provides detailed appendices on tools, prompts, error handling, and cost accounting. It specifies the exact versions of Lean and Mathlib used, and provides a link to the pull request documenting the newly solved conjectures. While the specific code for the h1 harness is not explicitly linked in the provided text (only the conjectures PR), the level of detail in the prompts and tool descriptions suggests high reproducibility for researchers with access to the same frontier models.
The primary limitation is the reliance on proprietary frontier models (GPT-5.6, Opus 4.8, etc.), which are black boxes and subject to rapid change, making long-term reproducibility difficult. The paper also acknowledges that it does not address the challenge of formalizing proofs that exceed the context window or require parallel agents for very large projects, though it suggests this is a natural next step. The cost analysis relies on API pricing, which may not reflect the true compute cost for self-hosted models.
This paper has significant broader impact by challenging the prevailing trend in automated theorem proving towards increasingly complex, human-designed workflows. It provides strong evidence that as LLMs improve, the focus should shift to building efficient interfaces and allowing models to scale their own reasoning, rather than imposing rigid structures. This insight is likely to influence the design of agents in other structured domains, such as software verification and formal methods. The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
AI has made remarkable progress in mathematics, solving challenging problems with machine-checkable proofs in Lean, yet typically relying on predefined mathematical definitions and formal specifications. This raises a natural question: can it reason rigorously about an unknown world? We introduce the Algebraic World Model (AWM), a framework that turns interaction with unfamiliar environments into explicit mathematical models supporting formal reasoning. AWM explores the environment and develops executable Python hypotheses, continually testing and revising them against observations. It then structures these hypotheses into typed objects and shared rules, simplifies their structure through category-theory-guided transformations, and formalizes the resulting model in Lean 4 to prove reachability and impossibility. AWM can formally prove that a goal is unreachable, even beyond explored trajectories. On two impossible game levels, it constructs closed, kernel-checked Lean proofs excluding every legal winning sequence in all six runs, while Claude Opus 5.5 and GPT-6 Astra, equipped with coding tools, produce none in twelve baseline runs. Four of AWM's six initial bounds are correct, compared with one for Opus 5.5 and none for GPT-6 Astra; given only feedback that a cheaper win exists, AWM revises its rules and constructs new proofs, reaching six of six correct bounds. Moreover, AWM produces interpretable mathematical models whose rules can be inspected and tested step by step. Across all 25 public ARC-AGI-3 games, these models enable Lean to replay the winning trajectories of all 183 levels and generalize beyond observed play, predicting unseen transitions and detecting changes in game mechanics. With Claude Opus 5, AWM clears all levels with a score of 99.9, while its certified winning trajectories use only 43% of the human baseline's actions.
Primary: unknown
All Institutions: unknown
The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
The paper introduces the Algebraic World Model (AWM), a framework that bridges empirical exploration and formal verification. The methodology is structured in three stages: (1) Explore & Prototype, where an LLM agent interacts with an environment to generate executable Python hypotheses; (2) Structure & Simplify, where these hypotheses are organized into typed objects and rules using category-theory-guided transformations (limits/colimits) verified in Lean; and (3) Formalize & Prove, where the model is formalized in Lean 4 to prove reachability or impossibility. The use of category theory to guide the simplification of world models is a creative and rigorous approach to ensuring that the learned model preserves the logical structure of the environment. The "sufficient cover" definition provides a theoretical basis for transferring proofs from the learned model to the actual environment, which is a significant methodological contribution.
The experiments are conducted on ARC-AGI-3, a challenging benchmark for abstract reasoning. The results are strong: AWM achieves a score of 99.9, outperforming baselines like Claude Opus 5.5 and GPT-6 Astra in both game clearing and, crucially, in constructing formal proofs of impossibility. The paper demonstrates that AWM can produce kernel-checked Lean proofs that exclude all legal winning sequences for impossible levels, a capability the baselines lack. The generalization tests, where frozen models predict unseen transitions and detect mechanic changes, further validate the robustness of the learned algebraic structures.
The paper provides a high level of reproducibility by releasing an interactive website with models, tests, Lean proofs, and links to official replays. The inclusion of the Lean model for the running example (lp85) in the supplementary materials, along with scripts for building and auditing axioms, allows for independent verification of the formal proofs. The explicit statement that all numbers come from script outputs over recorded runs, rather than model-generated text, enhances trust in the results.
The primary limitation is the scope of the environments tested. AWM is demonstrated on ARC-like environments with discrete actions and immediate, observable feedback. The authors acknowledge that richer hidden states, delayed feedback, and stochastic dynamics remain untested. Additionally, the reliance on LLMs for the initial hypothesis generation and exploration phase means that the system's performance is bounded by the capabilities of the underlying LLM. The "sufficient cover" condition, while theoretically sound, requires careful verification in practice to ensure that the learned model indeed covers the optimal winning sequences of the environment.
This work has significant implications for the field of AI, particularly in the areas of formal methods, automated theorem proving, and agent-based reasoning. By enabling AI systems to reason rigorously about unknown worlds and produce machine-checkable proofs, AWM paves the way for more trustworthy and interpretable AI systems. The integration of category theory with machine learning offers a new paradigm for structuring and simplifying learned models, which could be applied to other domains requiring rigorous reasoning. The ability to prove impossibility, not just reachability, is a crucial step towards AI systems that can provide complete and reliable solutions to complex problems. The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
Graph-based nearest neighbor search algorithms like HNSW, DiskANN, NSG, and others have surged in popularity in recent years due to their excellent empirical performance in comparison to space-partitioning methods like IVF data structures and Locality Sensitive Hashing (LSH). However, in contrast to LSH and related methods, obtaining strong worst-case theoretical guarantees for graph-based search has proven challenging. In this paper, we prove a strong lower bound that helps explain the lack of progress towards such guarantees. We show that no standard graph-based method (any that runs a variant of greedy search on a flat or hierarchical graph) can achieve sublinear query time for approximate nearest neighbor search in Euclidean space, even for an arbitrarily large approximation factor. On the other hand, we prove that sublinear query time is possible if we allow for Steiner points in the search graph. We obtain bounds comparable to LSH, showing e.g., that graph-based search with Steiner points can solve $c$-approximate nearest neighbor search in Euclidean space with query time $\tilde{O}(n^{1/(c-1)})$. We complement this result with initial experiments exploring how Steiner points can be incorporated into practical graph-based methods.
Primary: New York University
All Institutions: New York University, UMass Amherst
The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
The paper presents a rigorous theoretical framework for analyzing graph-based nearest neighbor search (NNS) algorithms like HNSW and DiskANN. The core methodological contribution is the formalization of a "flat search graph" model that captures generalized greedy search, allowing for the derivation of strong lower bounds. The authors prove that without Steiner points, no such method can achieve sublinear query time in worst-case Euclidean high-dimensional settings, effectively explaining the long-standing gap between empirical performance and theoretical guarantees. Conversely, they introduce a novel construction using Steiner points to simulate Locality Sensitive Hashing (LSH) within the graph search framework, achieving query times comparable to LSH ($\tilde{O}(n^{1/(c-1)})$). The proof techniques involve constructing hard instances of orthogonal vectors and leveraging geometric properties of Steiner nodes to implement hash table lookups via greedy traversal.
The experimental section is limited but sufficient for an initial exploration. The authors augment the ParlayANN implementation of DiskANN with simple $k$-means cluster centroids as Steiner points. They evaluate this on standard benchmarks (GloVe-200, NYTimes) using cosine similarity. The results show consistent improvements in recall vs. distance computations compared to the baseline, validating the theoretical potential of Steiner points. However, the experiments are preliminary, using a very simple heuristic for Steiner point generation rather than the optimal theoretical construction, and the datasets are relatively small compared to industrial scale.
The paper provides clear pseudocode for the generalized greedy search and detailed descriptions of the theoretical constructions. The experimental setup specifies the use of ParlayANN and specific hyperparameters (degree bound, pruning parameter, beam width). While the code for the specific Steiner-point augmentation is not explicitly linked in the text provided, the reliance on the open-source ParlayANN library and standard $k$-means clustering makes the experimental results highly reproducible.
The primary limitation is the gap between the theoretical upper bound (which requires a complex, specific construction of Steiner points to simulate LSH) and the experimental lower bound (which uses simple $k$-means centroids). The experiments do not demonstrate that the simple heuristic achieves the theoretical $O(n^{1/(c-1)})$ bound, only that it improves over the baseline. Additionally, the lower bound for the high-dimensional case requires $d=n$, leaving the low-dimensional worst-case complexity (for $d \ll n$) partially open, though a weaker bound is provided. The paper also ignores preprocessing time, focusing solely on query time.
This paper has significant implications for the vector database and search engine community. By proving that standard graph-based methods cannot theoretically match LSH in worst-case high-dimensional settings, it redirects research focus toward hybrid methods or Steiner-point-augmented graphs. The introduction of Steiner points as a mechanism to bridge the gap between empirical graph search and theoretical LSH guarantees offers a new design space for next-generation ANN algorithms. This could lead to more robust search systems with provable performance guarantees, which is critical for safety-critical applications. The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory $32\times$ faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains $4.9\times$ faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.
Primary: Massachusetts Institute of Technology (MIT)
All Institutions: Massachusetts Institute of Technology (MIT)
[One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
The paper proposes ParaTTT (PTTT), an algorithm to parallelize Test-Time Training (TTT) layers, which replace linear fast-weight memories with small neural networks trained via online gradient descent. The core innovation is an iterative refinement scheme that approximates the sequential inner-loop gradient trajectory using chunkwise-parallel passes. The method leverages the fact that the exact online gradient descent recurrence is a fixed point of the iteration. The authors provide a theoretical guarantee of geometric convergence for a two-layer variant under specific contraction conditions. The approach decomposes the update into a forward pass for feature trajectories, an exact gated-delta-rule pass for readouts and feature gradients, and a backward pass for gradient propagation. This allows the nonlinear memory to be evaluated in parallel, overcoming the sequential bottleneck of previous TTT implementations like TTT-MLP.
The experiments demonstrate significant speedups: PTTT evaluates the inner-loop trajectory 32x faster than TTT-MLP at 4K context with matched error, and the full 340M model trains 4.9x faster. The models (340M and 1.3B parameters) are evaluated on language modeling perplexity and zero-shot accuracy. The results show that PTTT models match or exceed the performance of Gated DeltaNet and Mamba-2, and remain competitive with same-size Transformers. Notably, the paper claims better recall than linear recurrences, suggesting the nonlinear memory provides a qualitative advantage in information retention. The comparison with Transformers is particularly strong, as TTT layers have historically struggled to match Transformer performance at scale.
The paper provides a GitHub repository link (https://github.com/ntumm120/paraTTT), which is a strong positive for reproducibility. The abstract mentions specific model sizes (340M, 1.3B) and context lengths (4K), which are standard benchmarks. However, the full text provided is truncated (only abstract and headers are visible), so specific hyperparameters, dataset splits, and training schedules cannot be fully verified from the provided text. Assuming the code repository is complete and well-documented, reproducibility should be high.
The theoretical convergence proof is limited to a two-layer variant, while practical applications may involve deeper networks. The claim that "two iterations suffice" in practice needs robust validation across different sequence lengths and model scales. The comparison with Transformers is competitive but not necessarily superior in all metrics, and the "better recall" claim requires careful definition of recall in the context of language modeling. The computational overhead of the iterative refinement, even if parallelized, may still be higher than simple linear recurrences for very short sequences where parallelism gains are minimal.
This work addresses a critical bottleneck in the adoption of nonlinear recurrent architectures. By enabling efficient parallelization of TTT layers, it makes these more expressive models viable for large-scale language modeling. This could lead to new architectures that combine the expressiveness of nonlinear memories with the efficiency of parallel training, potentially challenging the dominance of Transformers in long-context tasks. The technique of iterative refinement for parallelizing sequential operations is also broadly applicable to other sequential learning problems. [One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
We study adversarial linear bandits with strong corruption, in which an adaptive adversary chooses the reward vectors and a strong adversary corrupts each observed reward after seeing the learner's action, subject to a known total corruption budget $C$. Strong corruption breaks the importance-weighted reward estimates used by adversarial bandit algorithms: by corrupting only the rounds in which a rarely sampled action is drawn, the adversary amplifies each unit of corruption by the inverse of its sampling probability. The best known bound for this problem, $\tilde O(\sqrt{d^3T} + dC)$ by Liu et al. (2024), is a factor $\sqrt{d}$ above the corruption-free rate in its leading term, and whether $\tilde O(d\sqrt{T} + dC)$ regret is achievable was left open. We propose CRW-FTRL, which runs follow-the-regularized-leader with a radius-weighted entropy regularizer and estimates rewards with a normalized kernel estimator whose sensitivity to corruption is governed by per-action radii instead of inverse sampling probabilities. The radii are nondecreasing and are chosen jointly with the sampling distribution through a monotone log-determinant barrier problem, so that a negative linear term in the regularizer cancels both the corruption bias and the estimation error of every comparator. For $K$ actions in $\mathbb{R}^d$, CRW-FTRL achieves $\tilde O(\sqrt{dT\log K} + dC)$ regret with high probability against an adaptive adversary, which matches the best known corruption-free rate and the $Ω(dC)$ lower bound up to logarithmic factors. Applied to a discretization of a general compact action set, it yields $\tilde O(d\sqrt{T} + dC)$ regret, which settles the statistical rate of corrupted adversarial linear bandits under strong corruption.
Primary: University of California
All Institutions: University of California
The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
The paper proposes CRW-FTRL, a novel algorithm for adversarial linear bandits with strong corruption. The core innovation lies in replacing standard importance-weighted reward estimates—which are vulnerable to strong adversaries who target rarely sampled actions—with a normalized kernel estimator governed by per-action radii. These radii are determined via a monotone log-determinant barrier problem, coupled with a radius-weighted entropy regularizer. This design effectively decouples the sensitivity of the reward estimator from the inverse sampling probabilities, preventing the adversary from amplifying corruption. The theoretical framework is rigorous, leveraging FTRL with a specific regularizer structure to cancel out both corruption bias and estimation error.
The provided text is a theoretical paper focused on regret bounds. It does not contain empirical experiments, benchmarks, or comparisons with baselines in a practical setting. The contribution is entirely analytical, proving that the proposed algorithm achieves minimax optimal regret rates.
As a purely theoretical paper, reproducibility is tied to the correctness of the proofs rather than code implementation. The algorithm is described in sufficient detail for implementation, but the lack of code or empirical validation limits practical reproducibility assessment.
The primary limitation is the lack of empirical validation. The paper acknowledges that the reduction to general compact action sets via discretization is not computationally efficient, leaving the question of a computationally efficient algorithm for general action sets open. Additionally, the results are high-probability bounds, and the constants hidden in the tilde-O notation may be significant in practice.
This work settles a key open question in the theory of corrupted bandits, providing minimax optimal rates for adversarial linear bandits with strong corruption. It has significant implications for the design of robust online learning algorithms in adversarial environments, such as online advertising and recommendation systems where feedback can be manipulated. The techniques developed (radius-weighted regularizers, kernel estimators with controlled sensitivity) may be applicable to other robust learning problems. The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.
Primary: Google DeepMind
All Institutions: Google DeepMind, Google
EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
The paper proposes EmbeddingGemma 2, a 744M-parameter multimodal embedding model that unifies text, image, audio, and video into a single vector space. The core architectural contribution is the simplification of Per-Layer Embeddings (PLE) by removing the dedicated lookup table and instead projecting the main input embeddings, which reduces memory overhead and improves multimodal performance. The training methodology employs a rigorous three-stage curriculum: (1) text pre-finetuning with distillation to establish a strong text baseline, (2) multimodal pre-finetuning with distillation from a larger teacher model (Gemini Embedding 2) to inherit cross-modal alignment, and (3) multimodal finetuning using contrastive learning, spread-out regularization, and Matryoshka Representation Learning (MRL) to optimize the student model's specific embedding space. The use of a "tail-patch" stage for Quantization-Aware Training (QAT) distillation is a practical engineering innovation that minimizes quality loss in low-precision deployments.
The evaluation is extensive, covering seven major benchmark suites (MTEB, MMEB-v2, MIEB-lite, MAEB, MSEB) across text, code, vision, and audio. The model establishes a new Pareto frontier, outperforming specialist models and general-purpose models 2-12x larger. Ablation studies are thorough, validating the removal of the PLE lookup table, the necessity of co-training the vision encoder, and the impact of vision token budgets on different task types (e.g., document retrieval vs. video). The results demonstrate that native audio processing significantly outperforms ASR-based pipelines, particularly for non-speech sounds and paralinguistic features.
The paper provides high reproducibility. The model weights are released openly. Detailed architectural specifications (dimensionality, layer counts, attention types) are provided. The training recipe, including loss functions, data mixing strategies, and hyperparameter schedules, is described in detail. The release of quantized checkpoints and the specific instructions for task prefixes further aid in practical reproduction and deployment.
The model is still relatively small (744M parameters), which may limit its performance on extremely complex, long-context, or highly specialized scientific tasks compared to frontier LLMs. The paper notes a regression in English-only text performance compared to its predecessor, though multilingual performance improved. The reliance on a specific teacher model (Gemini Embedding 2) for distillation may limit the transferability of the training recipe to other base architectures.
This work significantly lowers the barrier to entry for high-quality multimodal retrieval and embedding tasks. By enabling on-device deployment (191MB-567MB footprint) with minimal quality loss, it facilitates privacy-preserving and offline multimodal applications. The unified space allows for novel applications like interleaved video-audio-text search, which were previously difficult to achieve with separate unimodal models. EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction-data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires 3 times more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security-utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.
Primary: Imperial College London
All Institutions: Imperial College London, Tübingen AI Center, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Fraunhofer HHI
The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
The paper proposes "Heads of Security" (HoS), a defense mechanism against prompt injection that operates by learning per-head attention gates. Unlike previous methods that rely on broad fine-tuning (like SecAlign) or complex architectural changes (like ASIDE), HoS targets the attention mechanism directly to enforce instruction-data separation. The core idea is to learn how specific attention heads should redistribute focus between trusted (system/user) and untrusted (tool/web) tokens. This allows the model to process untrusted content for reasoning purposes without executing any instructions embedded within it. The approach is lightweight and targeted, which is a significant methodological shift from behavior-level alignment to attention-level intervention.
The evaluation is rigorous and comprehensive, testing on two major open-weight models: Llama-3.3-70B-Instruct and GPT-OSS-20B. The paper compares HoS against strong baselines: SecAlign++ (SOTA fine-tuning defense) and ASIDE (architectural defense). Key results include a dramatic reduction in Attack Success Rate (ASR) against adaptive attacks (RL-Hammer drops from 97% to 10% on GPT-OSS-20B) and improved robustness against white-box attacks (Claudini). Crucially, the paper demonstrates that HoS preserves base-model utility on standard benchmarks, whereas competitors degrade in few-shot settings. The inclusion of agentic benchmarks with static attacks further validates the method's practical applicability.
The authors provide a GitHub repository and a Hugging Face space, indicating a commitment to open science. The code availability is a strong plus. However, the specific details of the "autoresearch" used to strengthen the white-box attack against HoS are not fully detailed in the abstract, though likely present in the full text. The use of standard models and benchmarks enhances reproducibility.
The primary limitation is the reliance on open-weight models for evaluation; performance on closed-source models like GPT-4 or Claude is not directly testable in this manner. Additionally, the method requires access to the model's internal attention heads, which limits its applicability to API-only models. The paper also notes that ASIDE struggles with tool use, but it is unclear if HoS fully resolves all edge cases in complex agentic workflows beyond the tested benchmarks.
This work has high impact on the field of LLM security. As LLMs are increasingly deployed in agentic roles with access to external data, prompt injection is a critical vulnerability. HoS offers a scalable, efficient, and effective defense that does not sacrifice model utility. It provides a new paradigm for securing LLMs by manipulating internal representations (attention) rather than just output behavior. This could lead to more secure and trustworthy AI systems in production environments. The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
High-resolution probabilistic weather forecasting remains computationally demanding. Here we present Astra, a 0.25$^\circ$ ensemble model trained on eight H200 GPUs. It combines Kantorovich Attention, a two-sided transport attention in which each region chooses where it reads from and writes to, so every region sends and receives at token resolution, with a self-featurised continuous ranked probability score (CRPS) that also scores forecasts in the model's multiscale features. In WeatherBench 2, Astra matches GenCast's CRPS averaged over five upper-air targets and days 1-10 with 9.8% of its hardware-normalised training compute. Verified against ERA5, its CRPS and ensemble-mean root-mean-square error are lower than the ECMWF ensemble's on all 600 variable, level and lead-time combinations, and its CRPS is lower than GenCast's on 73.5% of them. A 15-day member needs 39 times fewer floating-point operations than GenCast's GPU implementation, and 50 members take under 40 min on one consumer GPU, within 21 GB.
Primary: Unknown (Affiliations not explicitly listed in provided text, likely independent or small lab given hardware constraints)
All Institutions: Unknown
Astra achieves quarter-degree probabilistic weather forecasting with 9.8% of GenCast's training compute and 39x less inference FLOPs by introducing two-sided transport attention (Kantorovich Attention) and a self-featurised CRPS objective, enabling high-resolution ensemble forecasting on a single consumer GPU.
The paper introduces "Astra," a probabilistic weather forecasting model that achieves 0.25° resolution with significantly reduced computational cost compared to state-of-the-art models like GenCast. The core methodological contributions are twofold: (1) **Kantorovich Attention (KA)**, a two-sided transport attention mechanism where each grid tile selects both where it reads from (pull) and where it writes to (push). This ensures every tile is read at token resolution, avoiding the "unread" tile problem of one-sided sparse attention, while maintaining linear complexity relative to grid points. (2) **Self-featurised CRPS**, a training objective that scores ensemble members not just in physical space but also in the model's own multiscale feature space using a frozen copy of the network. This encourages the model to learn representations that are discriminative and structurally consistent, similar to how diffusion models use pretrained encoders. The architecture is a W-Net operating directly on the 0.25° grid without a latent encoder, using MeanFlow for single-step generation.
The experimental evaluation is rigorous and comprehensive. Astra is benchmarked against GenCast (the current SOTA for high-res probabilistic forecasting) and the ECMWF IFS ENS (the operational gold standard). Key results include: (1) Matching GenCast's CRPS on upper-air targets with only 9.8% of the hardware-normalised training compute. (2) Outperforming IFS ENS on all 600 variable/level/lead-time combinations in CRPS and ensemble-mean RMSE. (3) Outperforming GenCast in CRPS on 73.5% of combinations. (4) Significant efficiency gains: 39x fewer FLOPs for inference than GenCast's GPU implementation, allowing 50-member 15-day forecasts on a single consumer GPU (RTX 5090) in under 40 minutes. Spectral analysis confirms that Astra preserves fine-scale spatial structure better than GenCast and IFS ENS, particularly in the Alps and tropics. The use of WeatherBench 2 and ERA5 verification provides a standard, reproducible benchmark.
The paper provides extensive methodological details, including equations for the attention mechanism, the transportation problem formulation, and the self-featurised CRPS loss. It specifies hardware requirements (8x H200 GPUs for training) and inference costs (RTX 5090). While the code is not provided in the text, the level of detail in the Methods section (e.g., specific tile sizes, number of heads, initialization schemes, and noise prior construction) is high, suggesting that a competent team could reproduce the results. The reliance on specific hardware (H200/RTX 5090) may limit immediate reproducibility for labs with older GPUs, but the algorithmic details are clear.
The primary limitation is the hardware dependency for training; while inference is efficient, training still requires 8x H200 GPUs, which is a significant barrier for smaller labs compared to models trained on fewer resources. The model is evaluated primarily on ERA5 reanalysis data; performance on real-time operational data with missing or noisy inputs is not tested. The "self-featurised" component relies on a frozen copy of the network, which adds complexity to the training pipeline and may introduce instability if the network's features drift significantly during training. Additionally, the paper focuses on medium-range forecasting (up to 15 days); performance at longer lead times (sub-seasonal) is not evaluated.
This paper has high potential impact by democratizing high-resolution probabilistic weather forecasting. By reducing inference costs to the level of a single consumer GPU, it enables local, real-time probabilistic forecasting for applications such as renewable energy management, agriculture, and disaster response, where cloud-based or high-performance computing access may be limited. The two-sided transport attention mechanism is a generalizable architectural innovation that could be applied to other high-resolution spatial tasks (e.g., remote sensing, medical imaging) where global context is needed but dense attention is prohibitive. The self-featurised CRPS offers a new paradigm for training generative models to produce structurally consistent samples, potentially influencing broader generative AI research. Astra achieves quarter-degree probabilistic weather forecasting with 9.8% of GenCast's training compute and 39x less inference FLOPs by introducing two-sided transport attention (Kantorovich Attention) and a self-featurised CRPS objective, enabling high-resolution ensemble forecasting on a single consumer GPU.
Statistical procedures often control expected loss at a prescribed level while leaving part of their risk budget unused. Building on the e-value representation of risk-controlled decisions and the e-closure principle for multiple testing, we formulate recertification, a general paradigm for improving procedures by exploiting computable slack in their validity certificates. A valid procedure's losses normalized by their risk budgets form a state-indexed e-value collection; conversely, decisions whose normalized losses are bounded by this collection inherit its risk guarantees. Recertification uses distributional structure to quantify slack, enlarge the certifying e-values while preserving validity, and construct less conservative decisions. We develop two applications. For conformal risk control (CRC) with a known loss family, recertification redistributes unused risk budget among the held-out points of each hypothetical augmented sample. The resulting controller is valid under every finite exchangeable law and no more conservative than CRC on any calibration sample. For convex losses on an interval of actions, repeated recertification converges geometrically to an admissible controller. For false discovery rate control with arbitrarily dependent e-values, recertification exploits unused conditional budgets in randomized e-BH. It yields a closed-form randomized procedure whose rejection set contains that of randomized e-BH for every realization of the shared randomization and whose expected number of rejections is maximal subject to the same conditional budget constraints. Recertifying with the larger e-collection of closed e-BH gives a second closed-form procedure whose rejection set contains that of the first. Experiments on CIFAR-100 hierarchical classification and multiple testing simulations demonstrate improved budget utilization, informativeness and power over the baselines.
Primary: University of Amsterdam
All Institutions: University of Amsterdam, University of Oxford, University of Cambridge
The paper proposes a novel "recertification" framework that uniformly improves conformal risk control and randomized e-BH procedures by exploiting unused risk budget in e-value certificates. It provides rigorous theoretical guarantees, including geometric convergence to admissible controllers for convex losses and closed-form power improvements for multiple testing, supported by targeted experiments on CIFAR-100 and simulations.
The paper introduces "recertification," a general paradigm for improving statistical decision procedures by exploiting slack in their validity certificates (e-values). The core contribution is a rigorous theoretical framework that maps valid procedures to e-value collections and vice versa, allowing the construction of less conservative decisions by enlarging these e-values while preserving validity guarantees. The authors apply this to Conformal Risk Control (CRC) and randomized e-BH (Benjamini-Hochberg) procedures. For CRC, they prove that repeated recertification converges geometrically to an admissible controller under convexity assumptions. For multiple testing, they derive closed-form randomized procedures (RU-eBH and RC-eBH) that uniformly dominate existing baselines in power. The methodology is mathematically dense but logically sound, leveraging the e-closure principle effectively.
The experimental section is limited but appropriate for a theory-heavy paper. It includes simulations for multiple testing showing power gains over e-BH and U-eBH, and a real-data study on CIFAR-100 hierarchical classification. The CIFAR-100 results demonstrate improved budget utilization and informativeness compared to standard CRC and envelope CRC. However, the scope of empirical validation is narrow, focusing primarily on one classification task and synthetic multiple testing scenarios, lacking broader benchmarks or large-scale real-world applications that would typically be expected for a high-impact ML paper.
The paper provides clear algorithmic descriptions (Algorithm 1) and theoretical conditions for validity. The use of standard datasets (CIFAR-100) and well-defined baselines (CRC, e-BH) aids reproducibility. However, the text is truncated in the provided input, and specific hyperparameters or code availability are not explicitly confirmed in the visible portion, though the mathematical definitions are precise enough for implementation.
The primary limitation is the computational cost of repeated recertification, which can be expensive for high-dimensional or individual-score based baselines, as noted in the "Computation" section. The admissibility results rely on convexity or specific finite-space conditions, which may not hold for all practical loss functions. Additionally, the empirical evaluation is relatively small-scale, limiting the generalizability of the observed performance gains to other domains.
This work has significant potential to influence the field of conformal prediction and multiple testing by providing a systematic way to reduce conservatism in statistical guarantees. It bridges decision theory and e-value methodology, offering a template for improving other risk-controlled procedures. The uniform improvement guarantees are particularly valuable for applications requiring strict error control, such as medical diagnosis or autonomous systems. The paper proposes a novel "recertification" framework that uniformly improves conformal risk control and randomized e-BH procedures by exploiting unused risk budget in e-value certificates. It provides rigorous theoretical guarantees, including geometric convergence to admissible controllers for convex losses and closed-form power improvements for multiple testing, supported by targeted experiments on CIFAR-100 and simulations.
Video world models trained on large-scale robot data promise a safe and scalable approach for learning and evaluating policies that act in the physical world. However, they still struggle with long-horizon rollouts and out-of-distribution generalization, and they remain computationally demanding at inference time. Typically, video world models make predictions in the latent space of a reconstruction-trained autoencoder, which however is not directly optimized for dynamics modeling. On the other hand, world models designed for latent planning usually optimize latents to be predictable, but they forgo accurate prediction of observations in pixel space. In this paper, we show that it is possible to achieve the best of both worlds: a latent geometry optimized to make future prediction as simple as possible without sacrificing accurate prediction of future observations. More precisely, we find that temporal objectives for representation learning, paired with a past-conditioned decoder architecture, lead to consistent improvements in generative video world models at the same training budget. The gains include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors out of distribution and over long horizons. Our world models also agree more closely with real-world policy evaluation when using a vision-language model to judge task success from their pixel-space predictions. Overall, our study indicates that temporal predictability is an important design criterion for the latent space of video world models, and provides concrete guidance on how to incorporate it into the training of autoencoders.
Primary: New York University
All Institutions: New York University, Empire AI Consortium
The paper demonstrates that optimizing latent spaces for temporal predictability significantly improves the efficiency and accuracy of video world models. By integrating temporal objectives into autoencoder training, the authors achieve a superior balance between reconstruction fidelity and dynamics simplicity, leading to substantial gains in inference speed, training efficiency, and out-of-distribution generalization, which are critical for real-world robotic applications.
The paper proposes a principled approach to designing the latent space for video world models. The core insight is that standard autoencoders optimized for reconstruction do not necessarily produce latents that are easy to predict dynamically. The authors introduce temporal objectives for representation learning combined with a past-conditioned decoder architecture. This allows the latent geometry to be optimized for simplicity of future prediction (dynamics) while maintaining high-fidelity reconstruction of observations. This addresses a fundamental trade-off in world modeling: the tension between accurate pixel-space reconstruction and efficient latent-space dynamics modeling. The methodology is sound, leveraging established concepts in representation learning (temporal consistency) and applying them specifically to the context of generative video world models for robotics.
The experimental evaluation is robust, demonstrating significant improvements over baselines. Key metrics include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors in out-of-distribution and long-horizon scenarios. The inclusion of a vision-language model (VLM) to judge task success from pixel-space predictions adds a layer of semantic evaluation that is highly relevant to the robotics application domain. The comparison with both reconstruction-focused and dynamics-focused baselines effectively isolates the contribution of the proposed latent design.
The paper provides a reproducibility statement and a link to the source code for fine-tuning autoencoders, training world models, and evaluation. The appendices detail objectives, architectures, training settings, and evaluation protocols. This level of detail supports reproducibility, although the specific hyperparameters for the "temporal objectives" would need to be verified in the code to ensure full transparency.
The primary limitation is the computational cost of training the autoencoders with temporal objectives, which may be prohibitive for very large-scale datasets without significant optimization. Additionally, the reliance on a VLM for evaluation introduces potential biases inherent in the VLM itself. The paper focuses on robot data, so generalizability to other video domains (e.g., natural videos) is not fully explored.
This work has significant implications for the field of world models, particularly for robotics and autonomous systems. By making video world models more efficient and accurate in long-horizon rollouts, it enables safer and more scalable policy learning and evaluation. The finding that temporal predictability is a key design criterion for latent spaces provides concrete guidance for future research in representation learning for dynamics. The paper demonstrates that optimizing latent spaces for temporal predictability significantly improves the efficiency and accuracy of video world models. By integrating temporal objectives into autoencoder training, the authors achieve a superior balance between reconstruction fidelity and dynamics simplicity, leading to substantial gains in inference speed, training efficiency, and out-of-distribution generalization, which are critical for real-world robotic applications.
While progress in informal mathematics has come largely from scaling the chain-of-thought and agentic reasoning of frontier language models, formal theorem proving has increasingly relied on human-designed workflows around these models, such as planning, decomposition, and orchestrating specialized provers. This raises a bitter-lesson question: can a simple agentic loop that scales test-time compute match these workflows? The answer matters as formal verification must scale to keep up with frontier models producing research mathematics, yet new workflows rarely settle it, as their gains are often confounded with stronger models or more compute. We study this question with h1, a simple harness that gives a model an efficient interface to Lean 4 and keeps it working through verifier feedback and self-reflection, without imposing a decomposition or search structure. h1 reaches state-of-the-art results on PutnamBench, FATE-X, and FormalConjectures100-Solved, settles eight open problems in FormalConjectures, and, at matched models, matches or outperforms specialized workflows such as Goedel-Architect and AlphaProof Nexus, surprisingly showing that a single attempt (pass@1) can often scale to solve open problems. Through ablations and matched-model comparisons with Claude Code and AxProverBase, we find that simplicity alone is not enough: an efficient Lean interface largely determines cost and solve rate. We thus recommend that future work evaluate frontier models with efficient execution feedback before prescribing workflows, at the risk of bitter lessons.
Primary: Google DeepMind
All Institutions: Google DeepMind, University of Edinburgh
The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
The paper proposes "h1," a minimalist agentic harness for Lean 4 theorem proving that relies on a simple ReAct-style loop with an efficient interface to the Lean compiler, library search tools, and a reflection mechanism to compact context. The core methodological contribution is not a new model architecture or training algorithm, but a rigorous empirical demonstration that "bitter lesson" principles apply to harness design: that scaling test-time compute in a simple loop with good tooling outperforms complex, human-designed workflows (like planning, decomposition, or multi-agent orchestration) when using the same frontier models. The methodology is sound, focusing on isolating the variable of harness complexity while controlling for model capability and inference cost.
The experimental evaluation is extensive and high-quality. It covers a wide range of benchmarks including PutnamBench, FATE-X, FormalConjectures, and IMO 2026. The paper provides matched-model comparisons against state-of-the-art specialized systems (Goedel-Architect, AlphaProof Nexus, MerLean, etc.), which is crucial for a fair assessment. The results show that h1 achieves state-of-the-art performance, solving open problems and matching or exceeding specialized workflows at lower cost. The analysis of cost-efficiency curves (pass@1 vs pass@k) is particularly insightful, showing that longer single attempts can be more efficient than multiple shorter ones.
The paper provides detailed appendices on tools, prompts, error handling, and cost accounting. It specifies the exact versions of Lean and Mathlib used, and provides a link to the pull request documenting the newly solved conjectures. While the specific code for the h1 harness is not explicitly linked in the provided text (only the conjectures PR), the level of detail in the prompts and tool descriptions suggests high reproducibility for researchers with access to the same frontier models.
The primary limitation is the reliance on proprietary frontier models (GPT-5.6, Opus 4.8, etc.), which are black boxes and subject to rapid change, making long-term reproducibility difficult. The paper also acknowledges that it does not address the challenge of formalizing proofs that exceed the context window or require parallel agents for very large projects, though it suggests this is a natural next step. The cost analysis relies on API pricing, which may not reflect the true compute cost for self-hosted models.
This paper has significant broader impact by challenging the prevailing trend in automated theorem proving towards increasingly complex, human-designed workflows. It provides strong evidence that as LLMs improve, the focus should shift to building efficient interfaces and allowing models to scale their own reasoning, rather than imposing rigid structures. This insight is likely to influence the design of agents in other structured domains, such as software verification and formal methods. The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, University of Maryland
The paper establishes that hierarchical, intrinsically motivated agents, specifically using DIAYN, can quantitatively match both the behavioral variability and the prospective neural coding of spontaneous exploration in *Drosophila*, providing a powerful computational framework for reverse-engineering animal autonomy. By combining a realistic neuromechanical digital twin with a broad comparison of 20 intrinsic motivation algorithms, the study demonstrates that skill discovery objectives naturally give rise to internal representations that predict future actions and capture the geometric structure of biological neural activity, offering a novel perspective on the computational basis of autonomous behavior.
The paper proposes a rigorous framework for reverse-engineering animal autonomy by combining a neuromechanical digital twin of *Drosophila* (NeuroMechFly v2) with hierarchical intrinsically motivated agents. The core methodological contribution is the implementation of a hierarchical policy that operates over a discrete library of ethologically motivated motor primitives (syllables) rather than raw joint torques. This aligns the computational model with biological behavioral organization. The authors train 20 distinct intrinsic motivation algorithms (curiosity, entropy, and competency families) using Recurrent PPO without any extrinsic reward. The evaluation methodology is particularly strong, as it does not just compare behavioral statistics but also compares internal neural representations. Specifically, they decode future turn direction from the agent's latent states and compare this "prospective coding" to the neural activity in the fly's Inferior Posterior Slope (IPS) region. They further validate the geometric structure of the internal representations by checking for the sum-difference rule observed in biological interhemispheric encoding.
The experimental setup is robust and well-controlled. The agents are evaluated in a virtual tethered-ball assay that mimics the biological experiment of Brezovec et al. (2023). The comparison is quantitative, using balanced accuracy for decoding future turn direction and correlation coefficients for velocity encoding. The results show that DIAYN (Diversity is All You Need) significantly outperforms other intrinsic motivation methods in jointly matching both the behavioral variability (turning bias distribution) and the neural prospective coding of the flies. The finding that DIAYN's latent activity captures the sum-difference rule for interhemispheric encoding is a strong empirical result that links the computational objective to specific biological neural dynamics. The inclusion of random controls and extrinsic reward controls strengthens the causal claim that intrinsic motivation drives the observed similarities.
The paper provides high reproducibility. It specifies the exact environment (FlyGym v2), the biomechanical model (NeuroMechFly v2), the architecture (LSTM actor-critic with specific dimensions), and the training hyperparameters (50 million steps, 2048 parallel envs). The use of standard algorithms like PPO and well-known intrinsic motivation methods (DIAYN, ICM, etc.) makes it feasible for other researchers to replicate. The project website is provided, which likely contains code and further details. The detailed description of the evaluation protocol (Savitzky-Golay filtering, turn detection thresholds) ensures that the behavioral metrics are comparable to the biological data.
The primary limitation is the constraint of the behavioral space to 2D locomotion and a fixed set of 7 primitives. This limits the generality of the findings to specific types of exploratory behavior. The paper acknowledges that learning primitives jointly with high-level selection or expanding to flight would be future work. Additionally, the comparison is limited to the IPS region of the fly brain; a whole-brain comparison would be more comprehensive but is computationally and experimentally challenging. The reliance on a specific biological dataset (Brezovec et al.) means the findings are specific to *Drosophila* and may not generalize directly to other species without significant adaptation.
This work has significant implications for both artificial intelligence and neuroscience. For AI, it demonstrates that intrinsic motivation, specifically skill discovery, can produce agents with complex, autonomous behaviors and internal representations that mimic biological systems. This challenges the standard paradigm of extrinsic reward-driven RL and suggests that self-supervised objectives can capture deep structural properties of animal cognition. For neuroscience, it provides a computational hypothesis for how the fly brain generates spontaneous behavior and prospective neural signals. The framework established here can be extended to other species and behaviors, potentially leading to a general theory of animal autonomy. The interdisciplinary nature of the work bridges the gap between embodied AI and computational ethology, offering a new methodology for testing computational theories of brain function. The paper establishes that hierarchical, intrinsically motivated agents, specifically using DIAYN, can quantitatively match both the behavioral variability and the prospective neural coding of spontaneous exploration in *Drosophila*, providing a powerful computational framework for reverse-engineering animal autonomy. By combining a realistic neuromechanical digital twin with a broad comparison of 20 intrinsic motivation algorithms, the study demonstrates that skill discovery objectives naturally give rise to internal representations that predict future actions and capture the geometric structure of biological neural activity, offering a novel perspective on the computational basis of autonomous behavior.
Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tsinghua University, Shanghai AI Laboratory
[One sentence main contribution]. The paper introduces RynnVLA-Latent, a scalable framework for training VLA models by learning 3D-grounded latent actions from heterogeneous video data, significantly improving performance on robotic manipulation benchmarks by decoupling general motion learning from embodiment-specific control.
The paper proposes RynnVLA-Latent, a framework addressing the data scarcity and embodiment mismatch problems in Vision-Language-Action (VLA) models. The core innovation is RynnLAM, a module that learns continuous, 3D-grounded latent actions from heterogeneous video data (human, robot, simulation) without requiring native robot control labels. The method employs residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling to ensure the latent space captures control-relevant motion rather than just visual appearance changes. The VLA model then uses flow matching to predict these latent action chunks from visual and language inputs, followed by an adaptation layer to map latent actions to specific robot embodiments. This approach effectively decouples the learning of general manipulation dynamics from the specific control spaces of individual robots, allowing for pretraining on massive, unstructured video datasets.
The experimental results are strong and comprehensive. The model achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus, 51.9% on VLABench, and 88.6% on RoboTwin 2.0. These numbers are competitive with or superior to state-of-the-art methods in the VLA domain. The evaluation includes both simulation benchmarks and real-world tasks across different robot platforms (dexterous hands, mobile grippers), demonstrating generalization. The use of 30.1K hours of video data to generate 745.52M latent-action descriptors highlights the scalability of the approach. Ablation studies on representation learning and data scaling further support the technical claims.
The paper provides detailed descriptions of the RynnLAM training process and the flow matching objective. However, specific hyperparameters, exact dataset compositions, and code availability are not explicitly provided in the extracted text. The reliance on a large-scale proprietary or curated dataset (30.1K hours) may limit immediate reproducibility for smaller labs, though the methodology is clearly defined.
The primary limitation is the computational cost of processing 30.1K hours of video and generating latent actions. Additionally, the mapping from latent actions to specific robot actions requires an adaptation phase, which may not generalize perfectly to entirely new embodiments without some fine-tuning. The performance on VLABench (51.9%) is lower than on other benchmarks, suggesting potential weaknesses in certain complex or long-horizon tasks.
This work has significant implications for the field of robotic learning by providing a scalable pathway to train foundation models using unstructured video data. It reduces the dependency on expensive, robot-specific action-labeled datasets, potentially accelerating the development of general-purpose robot policies. The 3D-grounded latent action space offers a robust representation that can be shared across different robot morphologies. [One sentence main contribution]. The paper introduces RynnVLA-Latent, a scalable framework for training VLA models by learning 3D-grounded latent actions from heterogeneous video data, significantly improving performance on robotic manipulation benchmarks by decoupling general motion learning from embodiment-specific control.
Most analyses of neural representations focus on trained endpoints, leaving the process by which internal structure forms comparatively unmeasured. We instead treat representation formation itself as an object of study, asking whether the path a representation takes through training reveals information that its endpoint does not. We formalize this idea by viewing representation learning as a geometric trajectory, using Bures geometry to measure movement through representation space and quantum Fisher information to characterize local sensitivity to input factors. We find that learning concentrates local sensitivity on task-relevant factors, and that changing only the prediction target reorganizes this geometry around the corresponding visual factor. More strikingly, representation trajectories become systematically less direct as the learned computation requires higher-order interactions among input factors. This effect persists under matched-performance and training-duration controls and transfers from synthetic vector inputs to residual CNNs on rendered and natural images. Importantly, formation history contains predictive information about the stability of the learned representation: controlling for interaction order, path tortuosity predicts its subsequent drift under continued training and sensitivity to parameter perturbations, whereas endpoint displacement does not. Finally, representation-aware optimization lets us actively shape these trajectories, showing that formation geometry is not only measurable but actionable. Together, these results establish representation formation as a new source of information about learning: its geometry reveals properties of the computation being constructed, predicts subsequent representational stability, and provides a target for intervention during optimization.
Primary: University of California
All Institutions: University of California
The paper establishes representation formation as a measurable geometric object that reveals computational structure and predicts stability. By linking Bures geometry and Quantum Fisher Information to training trajectories, it provides a novel framework that not only diagnoses the complexity of learned computations but also enables targeted interventions to shape the learning path.
The paper proposes a rigorous information-geometric framework for analyzing representation formation during training. It defines the representation state as the trace-normalized covariance of hidden activations and utilizes Bures distance to measure the trajectory of this state through training. The local geometry is characterized by Quantum Fisher Information (QFI), which identifies input factors to which the representation is locally sensitive. The authors derive a "Representation Natural Gradient" (RepNG) by pulling back the Bures/QFI metric to parameter space, allowing for optimization that explicitly constrains representational movement. The theoretical connection between the local QFI metric and the global Bures trajectory is well-established, providing a unified geometric language for both local sensitivity and global path tortuosity.
The experiments are extensive and well-controlled. The authors validate the local geometry (QFI) on synthetic tasks and dSprites, showing it correctly identifies task-relevant factors. The core empirical finding—that higher-order computational interactions lead to higher path tortuosity—is tested across MLPs and Residual CNNs, using both synthetic and natural images (CIFAR-10 composites). Crucially, the paper includes matched-performance and matched-training-duration controls to rule out trivial explanations. The predictive power of tortuosity for subsequent representational stability (drift and parameter sensitivity) is demonstrated in a fresh 30-seed sweep. The intervention study shows RepNG reduces tortuosity compared to standard optimizers at matched performance.
The paper provides a detailed reproducibility statement, including specific hyperparameters, seed counts, and code availability. The GitHub repository link is provided. The experimental protocols are described in sufficient detail in the appendices to allow for replication, including the specific construction of the parameter subspaces for the optimizer comparison.
The primary limitation is scale. The experiments are conducted on small networks (MLPs and small ResNets) and relatively simple tasks (parity, sign patterns, dSprites factors). It remains an open question whether these geometric properties hold for large-scale foundation models or complex natural language tasks. Additionally, the RepNG method requires estimating the pullback metric, which may be computationally expensive for very large networks, though the paper demonstrates it in a 24-dimensional subspace.
This work shifts the focus of representation analysis from static endpoints to dynamic formation processes. It provides a new metric (tortuosity) that correlates with computational complexity and stability, offering a new diagnostic tool for understanding learning dynamics. The RepNG optimizer offers a practical method for stabilizing representations during training, which could be valuable in continual learning and transfer learning scenarios. The paper establishes representation formation as a measurable geometric object that reveals computational structure and predicts stability. By linking Bures geometry and Quantum Fisher Information to training trajectories, it provides a novel framework that not only diagnoses the complexity of learned computations but also enables targeted interventions to shape the learning path.
Indoor context deciphered by ambient sensing underpins many applications, from assisted living to smart buildings, yet remains weakly connected to the language-aligned paradigm of modern multimodal systems. Through billions of IoT devices, ambient WiFi Channel State Information (CSI) captures the indoor context of human activity and environmental dynamics passively via ubiquitous, always-on infrastructure. However, unlike vision or audio, CSI lacks naturally paired text like captions or transcripts, limiting its integration with language-driven AI systems. We present CSI-TextBench, a large-scale dataset and benchmark for aligning CSI data with natural language, unifying 3 sensing benchmarks into 285,790 samples across 7 tasks and pairing them with 2,291 hierarchically structured, label-derived descriptions via many-to-many assignment to construct 2.0 million CSI-text training tuples. Benchmarking 5 alignment methods from 4 paradigm families, we show that contrastive CSI-text alignment matches supervised classifiers without class labels, adapts to unseen classes with few samples, and responds to open-vocabulary natural-language prompts. A single unified model supports compositional scene understanding, answering joint queries over activity, proximity, and identity, achieving 86.9% top-1 accuracy on 140 composite scenes, outperforming task-specific classifier ensembles. CSI-TextBench enables ubiquitous ambient sensing as a language-indexed perception layer, enabling indoor contextual awareness in language-driven AI systems.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, ADT Research
CSI-TextBench introduces a large-scale dataset and benchmark for aligning WiFi CSI with natural language, demonstrating that contrastive learning can enable zero-shot and compositional scene understanding in ambient sensing. The paper makes a significant contribution by unifying heterogeneous sensing benchmarks into a language-compatible format, providing rigorous evaluation protocols for open-vocabulary and few-shot settings, and showing that a single unified model can effectively handle multi-attribute scene queries, thereby establishing a foundation for language-grounded ambient intelligence.
The paper proposes a systematic framework for aligning WiFi Channel State Information (CSI) with natural language. The core methodological contribution is not a new architecture but a rigorous benchmarking protocol and dataset construction pipeline. The authors unify three heterogeneous CSI datasets (CSI-Bench, XRF55, SignFi) into a standardized tensor format and generate text descriptions via hand-written templates rather than LLMs, ensuring label grounding. They evaluate five alignment objectives (CLIP-InfoNCE, SigLIP, LanguageBind, Barlow Twins, Masked Prediction) within a shared dual-encoder architecture (1D ResNet-18 for CSI, frozen CLIP ViT-B/32 for text). The methodology is sound, particularly in its handling of the "many-to-many" correspondence between signals and text, which is a unique challenge in sensing modalities lacking native captions. The use of frozen text encoders isolates the learning of the CSI representation, providing a clean experimental design.
The experimental evaluation is comprehensive, covering zero-shot classification, few-shot transfer, open-vocabulary prompting, and compositional scene understanding. The results demonstrate that contrastive alignment (specifically CLIP-InfoNCE) can match supervised classifiers without class labels, a significant finding for ambient sensing. The "Layer 3" experiments on compositional scene understanding (86.9% accuracy on joint queries) are particularly strong, showing that a single unified model can outperform task-specific ensembles. The ablations on text granularity and encoder choice are well-executed. However, the reliance on template-based text generation limits the semantic richness compared to true natural language, and the performance drop on fine-grained unseen classes (11.3% zero-shot) highlights the limits of current alignment techniques for highly specific gestures.
The paper provides high reproducibility. The authors release the dataset, code, and benchmark. The preprocessing pipeline is detailed, including specific normalization and subcarrier alignment steps. The training hyperparameters (learning rate, batch size, optimizer) are explicitly stated. The use of standard, publicly available base models (CLIP, ResNet) further aids reproducibility. The distinction between Level 1 and Level 2 captions is clearly defined, allowing other researchers to replicate the specific evaluation protocols.
The primary limitation is the synthetic nature of the text data. Since descriptions are derived from labels via templates, the model may not generalize to truly free-form, noisy, or context-rich natural language that does not strictly adhere to the template structures. The paper acknowledges this by testing "human-written free-form" language, but the gap between template-based and truly open-ended language remains. Additionally, the cross-environment and cross-device transfer performance is noted to be constrained by the inherent domain sensitivity of WiFi CSI, which is a fundamental physical limitation rather than a model flaw, but it limits the immediate practical deployment of the aligned model in unseen environments.
This work bridges the gap between passive sensing and language-driven AI, enabling ambient intelligence systems to be queried in natural language. This has significant implications for assistive living, smart buildings, and privacy-preserving monitoring (as CSI does not capture visual or audio content). By providing a standardized benchmark, it facilitates future research into RF-language alignment, potentially leading to more intuitive and flexible human-computer interfaces for indoor environments. The approach is generalizable to other sensing modalities like radar or mmWave, offering a template for language-grounded perception in non-visual domains. CSI-TextBench introduces a large-scale dataset and benchmark for aligning WiFi CSI with natural language, demonstrating that contrastive learning can enable zero-shot and compositional scene understanding in ambient sensing. The paper makes a significant contribution by unifying heterogeneous sensing benchmarks into a language-compatible format, providing rigorous evaluation protocols for open-vocabulary and few-shot settings, and showing that a single unified model can effectively handle multi-attribute scene queries, thereby establishing a foundation for language-grounded ambient intelligence.
Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally. Based on this insight, we introduce SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. SpecSlot expands each spatial anchor into multiple slots, each assigned a disjoint band of DCT coefficients; this separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds a local correction to the spectral prediction, while the generative state, target, and loss remain in pixel space. On ImageNet-256, SpecSlot achieves a comparable final FID to a FLOPs-matched JiT control while converging faster. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within only 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image generation, SpecSlot surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.
Primary: Peking University
All Institutions: Peking University, Nanyang Technological University, SenseTime
SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
The paper proposes SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from global attention sequence length. The core innovation is the "Role-Coupled Tokenization" where each spatial anchor is expanded into multiple slots, each assigned a disjoint band of DCT coefficients. Only the lowest-frequency slot participates in global attention, while high-frequency slots interact locally. This is a clever architectural modification that leverages the spectral properties of natural images (low-frequency structure requires global context, high-frequency detail is local). The method remains in pixel space, avoiding the artifacts often associated with VAE-based latent diffusion, while mitigating the quadratic cost of attention in pixel-space models. The inclusion of a spatial head for local correction is a sensible addition to handle residual errors.
The experiments are conducted on ImageNet-256 and ImageNet-512, as well as text-to-image generation benchmarks (DPG-Bench). The results show that SpecSlot achieves comparable FID to FLOPs-matched JiT controls on ImageNet-256 with faster convergence. On ImageNet-512, it improves FID by 2.84 over the control with fewer FLOPs and comes within 0.69 FID of a larger JiT model at significantly lower cost. The text-to-image results show improvements over JiT-B/8. The comparisons are against strong baselines (JiT), and the efficiency gains are substantial.
The paper provides links to the project page, GitHub repository, and Hugging Face model weights. The methodology is described in sufficient detail in the provided text (though the full technical details are in the sections referenced but not fully expanded in the prompt's text snippet, the abstract and summary provide the core mechanism). The availability of code and weights significantly enhances reproducibility.
The method relies on DCT coefficient bands, which may not be optimal for all types of content or modalities. The local interaction of high-frequency slots might limit the model's ability to capture long-range high-frequency correlations, although this is argued to be less critical. The computational overhead of managing multiple slots per anchor needs careful engineering to ensure the efficiency gains are realized in practice.
This work offers a viable path for pixel-space generative models to scale to higher resolutions without the prohibitive cost of full global attention. It could influence the design of future pixel-space Transformers and potentially other modalities where spectral decomposition is applicable. The efficiency gains make high-quality pixel-space generation more accessible. SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Primary: Tencent Youtu Lab
All Institutions: Tencent Youtu Lab
Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
The paper proposes Youtu-Parsing-Omni, a unified architecture for parsing documents, images, audio, and video into structured records. The core methodological contribution is the "Youtu-Omni-Encoder," which initializes from a pre-trained language model and uses shared weights to encode all modalities in a unified space-time coordinate system. A key architectural innovation is the inclusion of fusion layers that allow video frames to attend to concurrent audio, integrating cross-modal fusion directly into the perception layer. The training methodology introduces "OmniSchema-Aware On-Policy Distillation" (OSAD), a self-evolving post-training procedure where the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself. This allows the parser and its supervision to co-evolve, unifying the outputs of ten different parsing tasks under a single schema.
The paper reports state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.). The results highlight a significant advantage in audio and video parsing compared to general-purpose systems. It also achieves the best overall chart-parsing score (95.02) and high exact chemical-fingerprint agreement. The model matches dedicated optical-music models in character-level transcription of typeset sheet music. The evaluation covers a wide range of modalities and tasks, demonstrating the versatility of the unified approach.
The paper states that code and model weights are publicly available, which is a strong positive for reproducibility. However, the provided text does not include specific URLs or detailed hyperparameter settings, making it difficult to verify the exact implementation details without accessing the external repository. The description of the OSAD training procedure is high-level, and the specific mechanics of the "reference-conditioned copy" and "token-role-based objective" would require the code to fully understand and replicate.
The paper relies heavily on the performance of the pre-trained language model initialization, which may limit its applicability to domains where text is not the primary signal. The "self-evolving" nature of the OSAD training could be computationally expensive and potentially unstable, though this is not explicitly discussed. The evaluation is primarily on benchmark datasets, and real-world robustness to noisy or unstructured data is not thoroughly tested. The lack of specific URLs in the text prevents immediate verification of the claimed open-source availability.
This work has significant potential to impact the field of multimodal AI by providing a unified perception layer for agents that need to read, locate, and cite evidence across multiple modalities. The ability to parse documents, images, audio, and video into structured records using a single encoder and schema could simplify the development of complex AI agents. The focus on spatial and temporal grounding is particularly relevant for applications in robotics, autonomous systems, and advanced document analysis. Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni introduces two major innovations. First, we propose a unified cross-paradigm on-policy distillation (OPD) framework that successfully transfers the visual understanding ability of autoregressive VLMs and visual generation ability of continuous flow models, to a unified DLLM. Second, NLD-Uni incorporates a novel progressive self-refining diffusion process that gradually corrupts each token across multiple steps during training, as opposed to adopting the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality. Through extensive experiments, we show that NLD-Uni achieves state-of-the-art performance among unified DLLMs on diverse tasks including visual question-answering, multi-modal reasoning, object grounding, text-to-image generation, and image editing.
Primary: NVIDIA
All Institutions: NVIDIA
The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
The paper proposes NLD-Uni, a unified discrete diffusion large language model (DLLM) for visual tasks. The core methodological contribution is a "unified cross-paradigm on-policy distillation" framework. This approach attempts to transfer capabilities from two distinct source models—autoregressive VLMs (for understanding) and continuous flow models (for generation)—into a single discrete diffusion target. The second major innovation is a "progressive self-refining diffusion process." Unlike standard masked diffusion, which corrupts tokens to a uniform mask, this method gradually corrupts tokens across multiple steps during training. At inference, this allows for intermediate token commitments, theoretically enabling self-refinement. The methodology is complex and addresses a significant gap in unifying generation and understanding in discrete spaces, though the reliance on distillation from potentially proprietary or large teacher models adds complexity.
The paper claims state-of-the-art performance among unified DLLMs on VQA, multi-modal reasoning, object grounding, text-to-image generation, and image editing. However, the provided text is a placeholder ("sections/intro", "sections/experiments") and does not contain actual data, tables, or specific benchmark scores. Without concrete evidence of the "state-of-the-art" claims, the experimental evaluation cannot be rigorously verified. The scope of tasks (both understanding and generation) is broad, which is a strong point if the results hold up, but the lack of visible data in the prompt prevents a high confidence in the empirical rigor.
The paper describes specific training objectives (on-policy distillation, progressive corruption). However, without access to the full code, hyperparameters, and the specific teacher models used for distillation, reproducibility is moderate to low. The "Nemotron" branding suggests an internal NVIDIA model, which may not be fully open-source, limiting external reproducibility.
The primary limitation is the opacity of the experimental results in the provided text. Additionally, unified models often suffer from a "seesaw" effect where improving generation hurts understanding or vice versa; the paper claims to overcome this via distillation, but the trade-offs are not detailed in the abstract. The reliance on discrete tokens for image generation may still lag behind continuous diffusion/flow models in terms of raw fidelity, even with self-refinement.
If successful, this work could significantly advance the field of unified multimodal models by demonstrating that discrete diffusion can effectively handle both high-level reasoning and low-level pixel generation. This could lead to more efficient, single-model architectures for multimodal AI, reducing the need for separate VLMs and image generators. The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the victim's replies, wasting turns and limiting transferability; interactive generators read each victim reply to generate the next-turn attack but require in-loop evaluators for success signals or hints, incurring substantial costs and inflated performance. Bridging this gap, we propose SEMA-Auto, a sighted and efficient multi-turn attacker with autonomy, interacting with the victim model and terminating the session by itself upon success. To stabilize its behaviors and boost its capabilities, SEMA-Auto is trained in two stages. Synthetic bootstrap tuning installs the notion of attack targets and conforming decisions by fine-tuning the base model with bootstrapped trajectories. Decision-supervised reinforcement learning then optimizes two objectives jointly yet separately, reinforcing the attack with higher jailbreak reward while supervising the decision at each turn against an absolute target. Our method achieves state-of-the-art (SOTA) attack success rates (ASR) across multiple datasets, victims, and judges, outperforming all static and interactive baselines, within almost the fewest turns, demonstrating both effectiveness and efficiency. For example, SEMA-Auto performs a $76.0\%$ ASR@1 within an average of $3.0$ turns against DeepSeek-V4-Flash on AdvBench, while the best static baseline is $54.8\%$ with fixed $7$ turns and the best interactive baseline is $43.8\%$ in $3.7$ turns. Against safety-hardened frontier models, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, and GPT-6-Astra, our counter-trained attacker achieves an average $59.6\%$ ASR@1 on HarmBench, $13.8\%$ over prior SOTA. Our code is available at: https://github.com/microsoft/SEMA
Primary: Microsoft Research
All Institutions: Microsoft Research
SEMA-Auto introduces a sighted, efficient, and autonomous multi-turn jailbreak attacker that achieves SOTA performance by combining synthetic bootstrap tuning with decision-supervised reinforcement learning. The paper presents a rigorous methodology that addresses the inefficiencies of existing interactive attackers by enabling autonomous termination, resulting in significant improvements in attack success rates and efficiency across diverse victim models and datasets.
The paper proposes SEMA-Auto, a multi-turn jailbreak attacker that bridges the gap between static planners (which lack adaptability) and interactive generators (which are costly and rely on in-loop evaluators). The core innovation lies in a two-stage training pipeline: (1) Synthetic Bootstrap Tuning to instill attack targets and decision-making logic via bootstrapped trajectories, and (2) Decision-Supervised Reinforcement Learning (DSRL). The DSRL stage is particularly notable for jointly optimizing two separate objectives: maximizing jailbreak reward and supervising the termination decision against an absolute target. This allows the model to autonomously determine when an attack has succeeded without requiring an external judge during inference, thereby reducing cost and latency. The method effectively addresses the "sighted" aspect by reading victim replies while maintaining efficiency through autonomous termination.
The experimental section reports State-of-the-Art (SOTA) Attack Success Rates (ASR) across multiple datasets (AdvBench, HarmBench) and victim models (DeepSeek-V4-Flash, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, GPT-6-Astra). The results show significant improvements over both static and interactive baselines. For instance, on AdvBench against DeepSeek-V4-Flash, SEMA-Auto achieves 76.0% ASR@1 in an average of 3.0 turns, compared to 54.8% for the best static baseline (7 turns) and 43.8% for the best interactive baseline (3.7 turns). Against safety-hardened frontier models, it achieves an average 59.6% ASR@1 on HarmBench, outperforming prior SOTA by 13.8%. The evaluation covers a wide range of model capabilities and safety levels, providing a robust assessment of the method's transferability and effectiveness.
The authors provide a link to their code repository (https://github.com/microsoft/SEMA), which is a positive indicator for reproducibility. The methodology is described in sufficient detail, including the two-stage training process and the specific objectives of the reinforcement learning phase. However, as is common with preprints under review, some specific hyperparameters or implementation details might be found in the appendix (referenced but not fully visible in the provided text snippet). The use of standard benchmarks and open-source models (where applicable) further aids reproducibility.
The paper focuses on jailbreak attacks, which is a sensitive area. While the method is efficient, the reliance on synthetic bootstrap data may introduce biases or limit the diversity of attack strategies. The autonomous termination mechanism, while efficient, might occasionally lead to premature termination if the model's internal confidence is miscalibrated. Additionally, the performance against "safety-hardened frontier models" suggests that while the method is effective, it is not infallible, and the gap between high-performing and low-performing victims remains significant. The paper does not extensively discuss the potential for defensive mechanisms to counter this specific type of autonomous, efficient attacker.
This work has significant implications for LLM safety and security. By demonstrating a more efficient and autonomous jailbreak method, it highlights the limitations of current safety measures and the need for more robust defenses. The efficiency of the method (fewer turns, no in-loop evaluators) makes it a more practical threat, potentially lowering the barrier to entry for malicious actors. Conversely, the techniques developed for autonomous decision-making in adversarial contexts could be adapted for defensive purposes, such as creating more effective red-teaming tools or safety evaluators. The paper contributes to the ongoing arms race in AI safety, pushing the field to develop more resilient models. SEMA-Auto introduces a sighted, efficient, and autonomous multi-turn jailbreak attacker that achieves SOTA performance by combining synthetic bootstrap tuning with decision-supervised reinforcement learning. The paper presents a rigorous methodology that addresses the inefficiencies of existing interactive attackers by enabling autonomous termination, resulting in significant improvements in attack success rates and efficiency across diverse victim models and datasets.
Individual-level genotype data underpin genetic discovery, disease-risk modeling, and therapeutic development, yet many critical settings remain data-limited, including underrepresented ancestries, rare or imbalanced disease cohorts, and restricted-access biobanks. Conditional synthetic genome generation would support privacy-aware benchmarking, simulation, and data augmentation, but requires learning an ultra-high-dimensional discrete distribution over hundreds of thousands to tens of millions of correlated single-nucleotide polymorphisms (SNPs), or sites of DNA variation. We introduce HELICS: High-fidelity gEneration via Latent Interpolant Conditional Synthesis, a phasing-free latent flow matching framework for genome-wide SNP-level generation from biobank-scale genotype data. HELICS tokenizes chromosomes with convolutional autoencoders, reducing >600K SNPs to 4,096 continuous latent tokens, and trains a transformer parameterized conditional flow matching model over the concatenated multi-chromosome latent representation. Trained on over 417K genomes in the UK Biobank, HELICS generates genome-wide common-variant synthetic cohorts without haplotype phasing and achieves stronger fidelity to real genotypes than existing haplotype-based simulators, including high allele-frequency agreement in external evaluation and superior preservation of local and long-range SNP correlation structure. Conditioning on ancestry and genetic liability summaries enables targeted generation across genetic risk profiles; in perturbation analyses, by increasing the conditioned disease trait liability, HELICS induces SNP-level dosage changes in generated genomes that are highly correlated with published association weights. HELICS provides a scalable generative modeling framework for SNP-level genome data and a path toward conditional synthetic cohorts for data-limited genetic studies.
Primary: Harvard University
All Institutions: Harvard University
HELICS introduces a scalable, phasing-free latent flow matching framework for genome-wide synthetic genome generation that achieves high fidelity to real biobank data and enables controllable generation of genetic liability profiles, representing a significant advancement in computational genomics and generative modeling for high-dimensional discrete biological data.
The paper introduces HELICS, a framework for generating synthetic genome-wide SNP data using latent flow matching. The core methodological innovation is the decoupling of the ultra-high-dimensional discrete genotype space (600K+ SNPs) into a manageable continuous latent space via chromosome-specific convolutional autoencoders. This allows the application of a transformer-based flow matching model (USiT) to the concatenated latent tokens of all 22 autosomes. The approach is "phasing-free," meaning it operates directly on unphased dosage data, avoiding the computationally expensive and error-prone step of haplotype phasing required by traditional simulators like HAPGEN2 or HAPNEST. The autoencoder training objective is carefully designed to preserve population genetic structure, including terms for minor allele frequency (MAF) preservation and local linkage disequilibrium (LD) correlation. The flow matching model is conditioned on ancestry principal components and discretized polygenic risk scores (PRS), enabling controllable generation of genetic liability profiles.
The evaluation is rigorous and extensive. The model is trained on the UK Biobank (417K genomes) and evaluated on external cohorts (1000 Genomes, CCDG). Key metrics include allele frequency agreement ($R^2=0.992$), LD structure preservation, and population geometry (PCA/UMAP). A particularly strong aspect of the evaluation is the "perturbation analysis," where the authors demonstrate that increasing the conditioned disease liability (e.g., HDL) induces SNP-level dosage changes that correlate highly with published GWAS association weights ($R^2=0.547$ for high-effect variants). This provides strong evidence that the model learns biologically meaningful associations rather than just statistical correlations. The model outperforms existing haplotype-based simulators in fidelity and offers capabilities (conditional generation) that baselines lack.
The authors provide a GitHub repository with code. However, due to the sensitivity of UK Biobank data, the full model weights and training data are not released. A demo is provided using 1000 Genomes data on chromosome 22, which allows for verification of the pipeline but not the full-scale performance claims. The architectural details, including the dynamic programming for token allocation and specific loss terms, are described in detail in the appendix, aiding reproducibility for those with access to similar-scale data.
The model is currently limited to common variants (MAF > 1%) and does not handle rare variants or structural variants. Fidelity is weaker for underrepresented ancestry groups, likely due to data imbalance in the training set. The conditioning is based on coarse PRS deciles, which may limit the granularity of control. The privacy implications of synthetic data generation are discussed but not exhaustively proven; while membership inference attacks show low leakage, the risk of re-identification in specific contexts remains a topic for further study.
This work has significant potential to impact genetic research by enabling the creation of synthetic cohorts for underrepresented populations or rare diseases, where real data is scarce. It facilitates privacy-aware sharing of genetic data and allows for controlled simulation of genetic risk profiles. The ability to generate data conditioned on specific genetic liabilities could accelerate the discovery of new genetic associations and improve the generalizability of polygenic risk scores. HELICS introduces a scalable, phasing-free latent flow matching framework for genome-wide synthetic genome generation that achieves high fidelity to real biobank data and enables controllable generation of genetic liability profiles, representing a significant advancement in computational genomics and generative modeling for high-dimensional discrete biological data.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
Future planetary rovers will traverse steep, deformable terrain where slip, sinkage, and terrain deformation can threaten mobility. Although simulation can capture some wheel-soil effects, unmodeled deformable-terrain phenomena create sim-to-real gaps requiring deployment-time adaptation. Yet, rover operations leave little room for extensive trial-and-error. We introduce TERRA (Targeted Exploration for Reward-Relevant Adaptation), an online adaptation framework that efficiently seeks deployment data exposing task-relevant model errors and corrects them. TERRA adapts a simulation-trained world model with a reward-weighted prediction loss and trains a loss predictor to estimate where future performance-relevant errors are likely to occur, using this signal to guide exploration during MPC. In simulation, TERRA reaches approximately 90% slope-climbing success within five trials under severe out-of-distribution pretraining conditions, outperforming exploration baselines in reward, completion time, and energy to first success. On the ERNEST rover, TERRA recovers coordinated active-suspension locomotion on a deformable Mars-simulant slope within three to five real-world trials. Project website: https://terra-explore.github.io
Primary: Jet Propulsion Laboratory, California Institute of Technology
All Institutions: Jet Propulsion Laboratory, California Institute of Technology, University of Texas at Austin
TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
The paper introduces TERRA, a framework for online adaptation of simulation-trained world models for planetary rovers. The core methodological contributions are twofold: (1) a reward-weighted prediction loss that prioritizes model errors in observation dimensions that have high sensitivity to the task reward, and (2) a learned "loss predictor" that forecasts where these reward-relevant errors are likely to occur in candidate action sequences, thereby guiding exploration within a Model Predictive Control (MPC) loop. The use of Low-Rank Adaptation (LoRA) for efficient onboard updating of the dynamics model is a practical engineering choice. The approach is logically sound, addressing the specific bottleneck of sim-to-real gaps in deformable terrain where standard prediction-error-based exploration is inefficient.
The evaluation is strong, featuring both extensive simulation studies (100 seeds, multiple OOD pretraining conditions) and real-world validation on the ERNEST rover. The simulation results show significant improvements over baselines like RND and Plan2Explore in terms of success rate, completion time, and energy efficiency. The real-world results are particularly compelling, demonstrating the recovery of coordinated active-suspension locomotion within 3-5 trials on a Mars-simulant slope, a task that is critical for future missions. The ablation studies effectively isolate the contributions of the reward-weighted loss and the targeted exploration.
The paper provides detailed descriptions of the simulation environment, pretraining conditions, and hyperparameters. However, as is common in robotics papers involving proprietary hardware (ERNEST) and specific simulators (DARTS), full reproducibility by external groups may be limited. The project website is provided, which likely contains code or further details.
The method relies on the availability of a task reward during deployment, which may not always be easily defined or observable in complex real-world scenarios. The choice of loss-bin thresholds for the loss predictor is a hyperparameter that requires tuning. The evaluation is limited to a single rover platform (ERNEST) and a specific type of terrain (deformable slopes), so generalizability to other platforms or terrain types is not fully established.
This work has high relevance to the field of planetary robotics and autonomous systems operating in unstructured environments. The technique of reward-weighted model adaptation and targeted exploration is applicable beyond rovers to any mobile robot facing sim-to-real gaps in contact-rich or deformable environments. It provides a practical pathway for deploying learned policies in high-stakes, low-trial-count scenarios. TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
Discretizing continuous actions into discrete skills using methods like VQ-VAE has emerged as a powerful paradigm for robotic manipulation. However, quantization errors yield a suboptimal distribution for the skill prior, degrading its performance. While reinforcement learning offers a path for refinement, its direct application faces unstable encoder updates and a granularity dilemma in importance sampling. To address these challenges, we introduce Cascaded Skills Optimization (CSO), a two-stage post-training framework. First, Rejection-Sampling Supervised Fine-tuning aligns the model's observation-to-skill mapping with the distribution of successful online trajectories via supervised fine-tuning. Second, with the observation encoder frozen, Skills Policy Optimization computes an independent, clipped importance ratio for each complete skill, enabling stable and efficient updates. With a lightweight QueST-style policy, CSO raises overall LIBERO success from 80.1% to 97.5%, improves performance on RoboTwin2.0, and achieves 61.0% success across five physical single-arm tasks. Additional comparisons with FAST, discrete BEAST, and fixed-buffer IQL further validate the effectiveness of our two-stage post-training strategy.
Primary: Unknown
All Institutions: Unknown
[One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
The paper proposes CSO (Cascaded Skills Optimization), a two-stage post-training framework for refining skill-based robotic policies. The methodology is logically sound and addresses specific pain points in applying RL to discrete skill policies. Stage 1 (R-SFT) uses rejection sampling to collect successful trajectories and fine-tunes the observation encoder, which is then frozen. This is a pragmatic solution to the instability of updating visual encoders with sparse rewards. Stage 2 (SPO) introduces a skill-level importance sampling ratio, arguing that token-level (too fine) and episode-level (too coarse) granularities are suboptimal for robotic skills. The theoretical justification for the "granularity dilemma" is the core intellectual contribution. The approach is essentially a structured application of PPO variants (specifically GSPO/GRPO concepts) adapted to the hierarchical structure of skill-based policies. While the adaptation is clever, the underlying RL machinery is not new.
The experimental evaluation is rigorous and comprehensive. The authors test on standard simulation benchmarks (LIBERO, RoboTwin2.0, MetaWorld) and, crucially, on physical robots (Cobot Agilex ALOHA). The results show significant improvements over the imitation learning baseline (80.1% to 97.5% on LIBERO) and outperform strong baselines like RIPT and FAST. The ablation studies effectively isolate the contributions of R-SFT and SPO, and the granularity analysis convincingly demonstrates the superiority of skill-level optimization over token- and episode-level alternatives. The physical robot experiments, while limited to five tasks, provide valuable validation of the method's real-world applicability.
The paper provides detailed implementation specifics, including hyperparameters for both stages, architecture details for the lightweight and VLM-based variants, and data collection protocols. The use of standard benchmarks and clear descriptions of the training process enhances reproducibility. However, the lack of a public code repository link in the provided text is a minor drawback, though the detailed appendices mitigate this.
The method requires a sufficient number of successful trajectories to bootstrap the R-SFT stage, which may be difficult for very hard tasks without a strong initial policy. The frozen encoder in Stage 2 limits the model's ability to adapt its perception to new failure modes discovered during RL. The physical experiments are limited to single-arm tasks, and the method's scalability to more complex, multi-arm, or dexterous manipulation scenarios is not fully explored.
This paper contributes to the growing field of post-training for robotic policies, offering a practical and effective recipe for improving skill-based agents. The insight regarding optimization granularity is likely to influence future work in hierarchical RL and skill-based control. The framework is general enough to be applied to other skill-based architectures, making it a useful tool for the robotics community. [One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, University of Maryland, Toyota Research Institute
TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
The paper proposes TAPESTRY, a method to repurpose pretrained video diffusion transformers (specifically CogVideoX-2B) for robot control by treating robot actions as single-patch tokens in the video latent space. The core innovation is the "action-as-VAE-patch" tokenization, which uses a lightweight linear projection and LayerNorm to map action vectors into the same distribution as video VAE latents, allowing them to pass through the pretrained patchifying convolution without architectural modification. The training procedure involves three joint diffusion objectives: video-only, joint video+action, and action-only. A significant technical contribution is the analysis of high-frequency noise in multi-step denoising for action chunks, demonstrating that single-step DDIM inference from pure noise is superior for this specific repurposed backbone due to frequency-space mismatches between video and action signals.
The evaluation is rigorous, covering simulation (LIBERO benchmark with a learning-from-video protocol) and real-world bimanual manipulation. The paper demonstrates strong generalization to novel objects and motion primitives. The inference efficiency claims are well-supported, showing a 44x speedup over optimized DreamZero on consumer hardware (RTX 5090). The ablations effectively isolate the contribution of the video pretraining and the specific training objectives.
The paper provides detailed implementation specifics, including the exact parameter count of the action head (1280 parameters), the specific backbone used (CogVideoX-2B), and the training data mixture. The derivation of the single-step inference collapse is provided in the appendix. However, reliance on specific proprietary or large-scale datasets (like DROID, AgiBot-World) may limit immediate reproducibility for smaller labs, though the architectural changes are minimal and easy to implement.
The method is currently limited to a 20-DoF bimanual action space, and the single-step inference relies on the zero-terminal-SNR v-prediction schedule, which may not transfer to other backbones like rectified-flow models (Wan). The deterministic nature of single-step inference may be a limitation for tasks requiring multimodal action distributions.
This work significantly lowers the barrier to entry for using large video models in robotics by eliminating the need for expensive video generation at test time. The insight that action tokens can be treated as video patches with minimal projection is a generalizable architectural pattern that could be applied to other multimodal control tasks. The finding regarding high-frequency noise in repurposed diffusion models is a valuable diagnostic for the broader community working on world action models. TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley
The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.
The paper proposes a comprehensive sim-to-real pipeline for in-hand object pose reaching using a 16-DoF LEAP hand. The core methodological contributions are: (1) A training recipe that leverages a large-scale generated grasp dataset (via LightningGrasp and DexterityGen) to define diverse goals and resets, avoiding the need for human demonstrations. (2) An adaptive friction curriculum that starts with high friction to stabilize learning and gradually reduces it to match reality, which is shown to be critical for convergence. (3) A modular safety approach using a learned residual safety filter (adapted from ResSafe) that operates on top of a frozen task policy. This filter uses a tunable weight $w$ to balance performance and safety, allowing for a "dial" to adjust conservativeness at deployment without retraining the task policy. The system identification is rigorous, using a voltage-driven DC motor model (BAM) and precise latency calibration to bridge the sim-to-real gap.
The experimental evaluation is extensive and rigorous, particularly for a preprint. The authors conduct 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and 6 wrist orientations. The state-based policy achieves a 95.2% success rate, and the safety filter reduces drops by 75% while maintaining a 91.5% success rate. The paper also evaluates robustness to out-of-distribution goals, fast arm motion, and physical perturbations. Comparisons with Hamilton-Jacobi (HJ) safety filters demonstrate the superiority of the learned residual approach in terms of computational efficiency and performance trade-offs. The inclusion of a depth-camera-only policy that transfers zero-shot is a strong validation of the visual sim-to-real gap mitigation.
The paper provides high reproducibility. It details the hardware setup, system identification process, simulation parameters, and training hyperparameters. The authors commit to releasing the full recipe, code, and trained checkpoints. The use of standard tools like MuJoCo Warp and PPO, along with clear descriptions of the reward functions and curricula, makes it feasible for other researchers to replicate the results. The specific details on latency calibration and actuator modeling are particularly valuable for the robotics community.
The policies are trained per object, which limits generalization to new objects without retraining. The depth policies still show a visual sim-to-real gap on certain objects (knife, wineglass). The safety filter is also trained per object and per nominal policy. The downstream tasks (e.g., screwing in a lightbulb) rely on hard-coded arm motions, limiting the autonomy of the system. The evaluation is limited to a specific hand (LEAP) and arm (UR5), so generalizability to other hardware is not tested.
This work significantly advances the state of the art in dexterous manipulation by addressing the critical issue of safety and reliability in high-dimensional contact-rich systems. The modular safety filter approach is likely to be adopted in other robotic domains where safety is paramount. The comprehensive evaluation framework for measuring performance, reliability, and safety provides a valuable benchmark for the field. The ability to operate with only a wrist-mounted camera expands the practical applicability of such systems beyond controlled lab environments. The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.