Last 7 Days (October 26 – November 01, 2026)
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
AI has made remarkable progress in mathematics, solving challenging problems with machine-checkable proofs in Lean, yet typically relying on predefined mathematical definitions and formal specifications. This raises a natural question: can it reason rigorously about an unknown world? We introduce the Algebraic World Model (AWM), a framework that turns interaction with unfamiliar environments into explicit mathematical models supporting formal reasoning. AWM explores the environment and develops executable Python hypotheses, continually testing and revising them against observations. It then structures these hypotheses into typed objects and shared rules, simplifies their structure through category-theory-guided transformations, and formalizes the resulting model in Lean 4 to prove reachability and impossibility. AWM can formally prove that a goal is unreachable, even beyond explored trajectories. On two impossible game levels, it constructs closed, kernel-checked Lean proofs excluding every legal winning sequence in all six runs, while Claude Opus 5.5 and GPT-6 Astra, equipped with coding tools, produce none in twelve baseline runs. Four of AWM's six initial bounds are correct, compared with one for Opus 5.5 and none for GPT-6 Astra; given only feedback that a cheaper win exists, AWM revises its rules and constructs new proofs, reaching six of six correct bounds. Moreover, AWM produces interpretable mathematical models whose rules can be inspected and tested step by step. Across all 25 public ARC-AGI-3 games, these models enable Lean to replay the winning trajectories of all 183 levels and generalize beyond observed play, predicting unseen transitions and detecting changes in game mechanics. With Claude Opus 5, AWM clears all levels with a score of 99.9, while its certified winning trajectories use only 43% of the human baseline's actions.
Primary: unknown
All Institutions: unknown
The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
The paper introduces the Algebraic World Model (AWM), a framework that bridges empirical exploration and formal verification. The methodology is structured in three stages: (1) Explore & Prototype, where an LLM agent interacts with an environment to generate executable Python hypotheses; (2) Structure & Simplify, where these hypotheses are organized into typed objects and rules using category-theory-guided transformations (limits/colimits) verified in Lean; and (3) Formalize & Prove, where the model is formalized in Lean 4 to prove reachability or impossibility. The use of category theory to guide the simplification of world models is a creative and rigorous approach to ensuring that the learned model preserves the logical structure of the environment. The "sufficient cover" definition provides a theoretical basis for transferring proofs from the learned model to the actual environment, which is a significant methodological contribution.
The experiments are conducted on ARC-AGI-3, a challenging benchmark for abstract reasoning. The results are strong: AWM achieves a score of 99.9, outperforming baselines like Claude Opus 5.5 and GPT-6 Astra in both game clearing and, crucially, in constructing formal proofs of impossibility. The paper demonstrates that AWM can produce kernel-checked Lean proofs that exclude all legal winning sequences for impossible levels, a capability the baselines lack. The generalization tests, where frozen models predict unseen transitions and detect mechanic changes, further validate the robustness of the learned algebraic structures.
The paper provides a high level of reproducibility by releasing an interactive website with models, tests, Lean proofs, and links to official replays. The inclusion of the Lean model for the running example (lp85) in the supplementary materials, along with scripts for building and auditing axioms, allows for independent verification of the formal proofs. The explicit statement that all numbers come from script outputs over recorded runs, rather than model-generated text, enhances trust in the results.
The primary limitation is the scope of the environments tested. AWM is demonstrated on ARC-like environments with discrete actions and immediate, observable feedback. The authors acknowledge that richer hidden states, delayed feedback, and stochastic dynamics remain untested. Additionally, the reliance on LLMs for the initial hypothesis generation and exploration phase means that the system's performance is bounded by the capabilities of the underlying LLM. The "sufficient cover" condition, while theoretically sound, requires careful verification in practice to ensure that the learned model indeed covers the optimal winning sequences of the environment.
This work has significant implications for the field of AI, particularly in the areas of formal methods, automated theorem proving, and agent-based reasoning. By enabling AI systems to reason rigorously about unknown worlds and produce machine-checkable proofs, AWM paves the way for more trustworthy and interpretable AI systems. The integration of category theory with machine learning offers a new paradigm for structuring and simplifying learned models, which could be applied to other domains requiring rigorous reasoning. The ability to prove impossibility, not just reachability, is a crucial step towards AI systems that can provide complete and reliable solutions to complex problems. The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
AI has made remarkable progress in mathematics, solving challenging problems with machine-checkable proofs in Lean, yet typically relying on predefined mathematical definitions and formal specifications. This raises a natural question: can it reason rigorously about an unknown world? We introduce the Algebraic World Model (AWM), a framework that turns interaction with unfamiliar environments into explicit mathematical models supporting formal reasoning. AWM explores the environment and develops executable Python hypotheses, continually testing and revising them against observations. It then structures these hypotheses into typed objects and shared rules, simplifies their structure through category-theory-guided transformations, and formalizes the resulting model in Lean 4 to prove reachability and impossibility. AWM can formally prove that a goal is unreachable, even beyond explored trajectories. On two impossible game levels, it constructs closed, kernel-checked Lean proofs excluding every legal winning sequence in all six runs, while Claude Opus 5.5 and GPT-6 Astra, equipped with coding tools, produce none in twelve baseline runs. Four of AWM's six initial bounds are correct, compared with one for Opus 5.5 and none for GPT-6 Astra; given only feedback that a cheaper win exists, AWM revises its rules and constructs new proofs, reaching six of six correct bounds. Moreover, AWM produces interpretable mathematical models whose rules can be inspected and tested step by step. Across all 25 public ARC-AGI-3 games, these models enable Lean to replay the winning trajectories of all 183 levels and generalize beyond observed play, predicting unseen transitions and detecting changes in game mechanics. With Claude Opus 5, AWM clears all levels with a score of 99.9, while its certified winning trajectories use only 43% of the human baseline's actions.
Primary: unknown
All Institutions: unknown
The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
The paper introduces the Algebraic World Model (AWM), a framework that bridges empirical exploration and formal verification. The methodology is structured in three stages: (1) Explore & Prototype, where an LLM agent interacts with an environment to generate executable Python hypotheses; (2) Structure & Simplify, where these hypotheses are organized into typed objects and rules using category-theory-guided transformations (limits/colimits) verified in Lean; and (3) Formalize & Prove, where the model is formalized in Lean 4 to prove reachability or impossibility. The use of category theory to guide the simplification of world models is a creative and rigorous approach to ensuring that the learned model preserves the logical structure of the environment. The "sufficient cover" definition provides a theoretical basis for transferring proofs from the learned model to the actual environment, which is a significant methodological contribution.
The experiments are conducted on ARC-AGI-3, a challenging benchmark for abstract reasoning. The results are strong: AWM achieves a score of 99.9, outperforming baselines like Claude Opus 5.5 and GPT-6 Astra in both game clearing and, crucially, in constructing formal proofs of impossibility. The paper demonstrates that AWM can produce kernel-checked Lean proofs that exclude all legal winning sequences for impossible levels, a capability the baselines lack. The generalization tests, where frozen models predict unseen transitions and detect mechanic changes, further validate the robustness of the learned algebraic structures.
The paper provides a high level of reproducibility by releasing an interactive website with models, tests, Lean proofs, and links to official replays. The inclusion of the Lean model for the running example (lp85) in the supplementary materials, along with scripts for building and auditing axioms, allows for independent verification of the formal proofs. The explicit statement that all numbers come from script outputs over recorded runs, rather than model-generated text, enhances trust in the results.
The primary limitation is the scope of the environments tested. AWM is demonstrated on ARC-like environments with discrete actions and immediate, observable feedback. The authors acknowledge that richer hidden states, delayed feedback, and stochastic dynamics remain untested. Additionally, the reliance on LLMs for the initial hypothesis generation and exploration phase means that the system's performance is bounded by the capabilities of the underlying LLM. The "sufficient cover" condition, while theoretically sound, requires careful verification in practice to ensure that the learned model indeed covers the optimal winning sequences of the environment.
This work has significant implications for the field of AI, particularly in the areas of formal methods, automated theorem proving, and agent-based reasoning. By enabling AI systems to reason rigorously about unknown worlds and produce machine-checkable proofs, AWM paves the way for more trustworthy and interpretable AI systems. The integration of category theory with machine learning offers a new paradigm for structuring and simplifying learned models, which could be applied to other domains requiring rigorous reasoning. The ability to prove impossibility, not just reachability, is a crucial step towards AI systems that can provide complete and reliable solutions to complex problems. The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
Graph-based nearest neighbor search algorithms like HNSW, DiskANN, NSG, and others have surged in popularity in recent years due to their excellent empirical performance in comparison to space-partitioning methods like IVF data structures and Locality Sensitive Hashing (LSH). However, in contrast to LSH and related methods, obtaining strong worst-case theoretical guarantees for graph-based search has proven challenging. In this paper, we prove a strong lower bound that helps explain the lack of progress towards such guarantees. We show that no standard graph-based method (any that runs a variant of greedy search on a flat or hierarchical graph) can achieve sublinear query time for approximate nearest neighbor search in Euclidean space, even for an arbitrarily large approximation factor. On the other hand, we prove that sublinear query time is possible if we allow for Steiner points in the search graph. We obtain bounds comparable to LSH, showing e.g., that graph-based search with Steiner points can solve $c$-approximate nearest neighbor search in Euclidean space with query time $\tilde{O}(n^{1/(c-1)})$. We complement this result with initial experiments exploring how Steiner points can be incorporated into practical graph-based methods.
Primary: New York University
All Institutions: New York University, UMass Amherst
The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
The paper presents a rigorous theoretical framework for analyzing graph-based nearest neighbor search (NNS) algorithms like HNSW and DiskANN. The core methodological contribution is the formalization of a "flat search graph" model that captures generalized greedy search, allowing for the derivation of strong lower bounds. The authors prove that without Steiner points, no such method can achieve sublinear query time in worst-case Euclidean high-dimensional settings, effectively explaining the long-standing gap between empirical performance and theoretical guarantees. Conversely, they introduce a novel construction using Steiner points to simulate Locality Sensitive Hashing (LSH) within the graph search framework, achieving query times comparable to LSH ($\tilde{O}(n^{1/(c-1)})$). The proof techniques involve constructing hard instances of orthogonal vectors and leveraging geometric properties of Steiner nodes to implement hash table lookups via greedy traversal.
The experimental section is limited but sufficient for an initial exploration. The authors augment the ParlayANN implementation of DiskANN with simple $k$-means cluster centroids as Steiner points. They evaluate this on standard benchmarks (GloVe-200, NYTimes) using cosine similarity. The results show consistent improvements in recall vs. distance computations compared to the baseline, validating the theoretical potential of Steiner points. However, the experiments are preliminary, using a very simple heuristic for Steiner point generation rather than the optimal theoretical construction, and the datasets are relatively small compared to industrial scale.
The paper provides clear pseudocode for the generalized greedy search and detailed descriptions of the theoretical constructions. The experimental setup specifies the use of ParlayANN and specific hyperparameters (degree bound, pruning parameter, beam width). While the code for the specific Steiner-point augmentation is not explicitly linked in the text provided, the reliance on the open-source ParlayANN library and standard $k$-means clustering makes the experimental results highly reproducible.
The primary limitation is the gap between the theoretical upper bound (which requires a complex, specific construction of Steiner points to simulate LSH) and the experimental lower bound (which uses simple $k$-means centroids). The experiments do not demonstrate that the simple heuristic achieves the theoretical $O(n^{1/(c-1)})$ bound, only that it improves over the baseline. Additionally, the lower bound for the high-dimensional case requires $d=n$, leaving the low-dimensional worst-case complexity (for $d \ll n$) partially open, though a weaker bound is provided. The paper also ignores preprocessing time, focusing solely on query time.
This paper has significant implications for the vector database and search engine community. By proving that standard graph-based methods cannot theoretically match LSH in worst-case high-dimensional settings, it redirects research focus toward hybrid methods or Steiner-point-augmented graphs. The introduction of Steiner points as a mechanism to bridge the gap between empirical graph search and theoretical LSH guarantees offers a new design space for next-generation ANN algorithms. This could lead to more robust search systems with provable performance guarantees, which is critical for safety-critical applications. The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory $32\times$ faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains $4.9\times$ faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.
Primary: Massachusetts Institute of Technology (MIT)
All Institutions: Massachusetts Institute of Technology (MIT)
[One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
The paper proposes ParaTTT (PTTT), an algorithm to parallelize Test-Time Training (TTT) layers, which replace linear fast-weight memories with small neural networks trained via online gradient descent. The core innovation is an iterative refinement scheme that approximates the sequential inner-loop gradient trajectory using chunkwise-parallel passes. The method leverages the fact that the exact online gradient descent recurrence is a fixed point of the iteration. The authors provide a theoretical guarantee of geometric convergence for a two-layer variant under specific contraction conditions. The approach decomposes the update into a forward pass for feature trajectories, an exact gated-delta-rule pass for readouts and feature gradients, and a backward pass for gradient propagation. This allows the nonlinear memory to be evaluated in parallel, overcoming the sequential bottleneck of previous TTT implementations like TTT-MLP.
The experiments demonstrate significant speedups: PTTT evaluates the inner-loop trajectory 32x faster than TTT-MLP at 4K context with matched error, and the full 340M model trains 4.9x faster. The models (340M and 1.3B parameters) are evaluated on language modeling perplexity and zero-shot accuracy. The results show that PTTT models match or exceed the performance of Gated DeltaNet and Mamba-2, and remain competitive with same-size Transformers. Notably, the paper claims better recall than linear recurrences, suggesting the nonlinear memory provides a qualitative advantage in information retention. The comparison with Transformers is particularly strong, as TTT layers have historically struggled to match Transformer performance at scale.
The paper provides a GitHub repository link (https://github.com/ntumm120/paraTTT), which is a strong positive for reproducibility. The abstract mentions specific model sizes (340M, 1.3B) and context lengths (4K), which are standard benchmarks. However, the full text provided is truncated (only abstract and headers are visible), so specific hyperparameters, dataset splits, and training schedules cannot be fully verified from the provided text. Assuming the code repository is complete and well-documented, reproducibility should be high.
The theoretical convergence proof is limited to a two-layer variant, while practical applications may involve deeper networks. The claim that "two iterations suffice" in practice needs robust validation across different sequence lengths and model scales. The comparison with Transformers is competitive but not necessarily superior in all metrics, and the "better recall" claim requires careful definition of recall in the context of language modeling. The computational overhead of the iterative refinement, even if parallelized, may still be higher than simple linear recurrences for very short sequences where parallelism gains are minimal.
This work addresses a critical bottleneck in the adoption of nonlinear recurrent architectures. By enabling efficient parallelization of TTT layers, it makes these more expressive models viable for large-scale language modeling. This could lead to new architectures that combine the expressiveness of nonlinear memories with the efficiency of parallel training, potentially challenging the dominance of Transformers in long-context tasks. The technique of iterative refinement for parallelizing sequential operations is also broadly applicable to other sequential learning problems. [One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
We study adversarial linear bandits with strong corruption, in which an adaptive adversary chooses the reward vectors and a strong adversary corrupts each observed reward after seeing the learner's action, subject to a known total corruption budget $C$. Strong corruption breaks the importance-weighted reward estimates used by adversarial bandit algorithms: by corrupting only the rounds in which a rarely sampled action is drawn, the adversary amplifies each unit of corruption by the inverse of its sampling probability. The best known bound for this problem, $\tilde O(\sqrt{d^3T} + dC)$ by Liu et al. (2024), is a factor $\sqrt{d}$ above the corruption-free rate in its leading term, and whether $\tilde O(d\sqrt{T} + dC)$ regret is achievable was left open. We propose CRW-FTRL, which runs follow-the-regularized-leader with a radius-weighted entropy regularizer and estimates rewards with a normalized kernel estimator whose sensitivity to corruption is governed by per-action radii instead of inverse sampling probabilities. The radii are nondecreasing and are chosen jointly with the sampling distribution through a monotone log-determinant barrier problem, so that a negative linear term in the regularizer cancels both the corruption bias and the estimation error of every comparator. For $K$ actions in $\mathbb{R}^d$, CRW-FTRL achieves $\tilde O(\sqrt{dT\log K} + dC)$ regret with high probability against an adaptive adversary, which matches the best known corruption-free rate and the $Ω(dC)$ lower bound up to logarithmic factors. Applied to a discretization of a general compact action set, it yields $\tilde O(d\sqrt{T} + dC)$ regret, which settles the statistical rate of corrupted adversarial linear bandits under strong corruption.
Primary: University of California
All Institutions: University of California
The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
The paper proposes CRW-FTRL, a novel algorithm for adversarial linear bandits with strong corruption. The core innovation lies in replacing standard importance-weighted reward estimates—which are vulnerable to strong adversaries who target rarely sampled actions—with a normalized kernel estimator governed by per-action radii. These radii are determined via a monotone log-determinant barrier problem, coupled with a radius-weighted entropy regularizer. This design effectively decouples the sensitivity of the reward estimator from the inverse sampling probabilities, preventing the adversary from amplifying corruption. The theoretical framework is rigorous, leveraging FTRL with a specific regularizer structure to cancel out both corruption bias and estimation error.
The provided text is a theoretical paper focused on regret bounds. It does not contain empirical experiments, benchmarks, or comparisons with baselines in a practical setting. The contribution is entirely analytical, proving that the proposed algorithm achieves minimax optimal regret rates.
As a purely theoretical paper, reproducibility is tied to the correctness of the proofs rather than code implementation. The algorithm is described in sufficient detail for implementation, but the lack of code or empirical validation limits practical reproducibility assessment.
The primary limitation is the lack of empirical validation. The paper acknowledges that the reduction to general compact action sets via discretization is not computationally efficient, leaving the question of a computationally efficient algorithm for general action sets open. Additionally, the results are high-probability bounds, and the constants hidden in the tilde-O notation may be significant in practice.
This work settles a key open question in the theory of corrupted bandits, providing minimax optimal rates for adversarial linear bandits with strong corruption. It has significant implications for the design of robust online learning algorithms in adversarial environments, such as online advertising and recommendation systems where feedback can be manipulated. The techniques developed (radius-weighted regularizers, kernel estimators with controlled sensitivity) may be applicable to other robust learning problems. The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.
Primary: Google DeepMind
All Institutions: Google DeepMind, Google
EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
The paper proposes EmbeddingGemma 2, a 744M-parameter multimodal embedding model that unifies text, image, audio, and video into a single vector space. The core architectural contribution is the simplification of Per-Layer Embeddings (PLE) by removing the dedicated lookup table and instead projecting the main input embeddings, which reduces memory overhead and improves multimodal performance. The training methodology employs a rigorous three-stage curriculum: (1) text pre-finetuning with distillation to establish a strong text baseline, (2) multimodal pre-finetuning with distillation from a larger teacher model (Gemini Embedding 2) to inherit cross-modal alignment, and (3) multimodal finetuning using contrastive learning, spread-out regularization, and Matryoshka Representation Learning (MRL) to optimize the student model's specific embedding space. The use of a "tail-patch" stage for Quantization-Aware Training (QAT) distillation is a practical engineering innovation that minimizes quality loss in low-precision deployments.
The evaluation is extensive, covering seven major benchmark suites (MTEB, MMEB-v2, MIEB-lite, MAEB, MSEB) across text, code, vision, and audio. The model establishes a new Pareto frontier, outperforming specialist models and general-purpose models 2-12x larger. Ablation studies are thorough, validating the removal of the PLE lookup table, the necessity of co-training the vision encoder, and the impact of vision token budgets on different task types (e.g., document retrieval vs. video). The results demonstrate that native audio processing significantly outperforms ASR-based pipelines, particularly for non-speech sounds and paralinguistic features.
The paper provides high reproducibility. The model weights are released openly. Detailed architectural specifications (dimensionality, layer counts, attention types) are provided. The training recipe, including loss functions, data mixing strategies, and hyperparameter schedules, is described in detail. The release of quantized checkpoints and the specific instructions for task prefixes further aid in practical reproduction and deployment.
The model is still relatively small (744M parameters), which may limit its performance on extremely complex, long-context, or highly specialized scientific tasks compared to frontier LLMs. The paper notes a regression in English-only text performance compared to its predecessor, though multilingual performance improved. The reliance on a specific teacher model (Gemini Embedding 2) for distillation may limit the transferability of the training recipe to other base architectures.
This work significantly lowers the barrier to entry for high-quality multimodal retrieval and embedding tasks. By enabling on-device deployment (191MB-567MB footprint) with minimal quality loss, it facilitates privacy-preserving and offline multimodal applications. The unified space allows for novel applications like interleaved video-audio-text search, which were previously difficult to achieve with separate unimodal models. EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction-data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires 3 times more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security-utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.
Primary: Imperial College London
All Institutions: Imperial College London, Tübingen AI Center, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Fraunhofer HHI
The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
The paper proposes "Heads of Security" (HoS), a defense mechanism against prompt injection that operates by learning per-head attention gates. Unlike previous methods that rely on broad fine-tuning (like SecAlign) or complex architectural changes (like ASIDE), HoS targets the attention mechanism directly to enforce instruction-data separation. The core idea is to learn how specific attention heads should redistribute focus between trusted (system/user) and untrusted (tool/web) tokens. This allows the model to process untrusted content for reasoning purposes without executing any instructions embedded within it. The approach is lightweight and targeted, which is a significant methodological shift from behavior-level alignment to attention-level intervention.
The evaluation is rigorous and comprehensive, testing on two major open-weight models: Llama-3.3-70B-Instruct and GPT-OSS-20B. The paper compares HoS against strong baselines: SecAlign++ (SOTA fine-tuning defense) and ASIDE (architectural defense). Key results include a dramatic reduction in Attack Success Rate (ASR) against adaptive attacks (RL-Hammer drops from 97% to 10% on GPT-OSS-20B) and improved robustness against white-box attacks (Claudini). Crucially, the paper demonstrates that HoS preserves base-model utility on standard benchmarks, whereas competitors degrade in few-shot settings. The inclusion of agentic benchmarks with static attacks further validates the method's practical applicability.
The authors provide a GitHub repository and a Hugging Face space, indicating a commitment to open science. The code availability is a strong plus. However, the specific details of the "autoresearch" used to strengthen the white-box attack against HoS are not fully detailed in the abstract, though likely present in the full text. The use of standard models and benchmarks enhances reproducibility.
The primary limitation is the reliance on open-weight models for evaluation; performance on closed-source models like GPT-4 or Claude is not directly testable in this manner. Additionally, the method requires access to the model's internal attention heads, which limits its applicability to API-only models. The paper also notes that ASIDE struggles with tool use, but it is unclear if HoS fully resolves all edge cases in complex agentic workflows beyond the tested benchmarks.
This work has high impact on the field of LLM security. As LLMs are increasingly deployed in agentic roles with access to external data, prompt injection is a critical vulnerability. HoS offers a scalable, efficient, and effective defense that does not sacrifice model utility. It provides a new paradigm for securing LLMs by manipulating internal representations (attention) rather than just output behavior. This could lead to more secure and trustworthy AI systems in production environments. The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
High-resolution probabilistic weather forecasting remains computationally demanding. Here we present Astra, a 0.25$^\circ$ ensemble model trained on eight H200 GPUs. It combines Kantorovich Attention, a two-sided transport attention in which each region chooses where it reads from and writes to, so every region sends and receives at token resolution, with a self-featurised continuous ranked probability score (CRPS) that also scores forecasts in the model's multiscale features. In WeatherBench 2, Astra matches GenCast's CRPS averaged over five upper-air targets and days 1-10 with 9.8% of its hardware-normalised training compute. Verified against ERA5, its CRPS and ensemble-mean root-mean-square error are lower than the ECMWF ensemble's on all 600 variable, level and lead-time combinations, and its CRPS is lower than GenCast's on 73.5% of them. A 15-day member needs 39 times fewer floating-point operations than GenCast's GPU implementation, and 50 members take under 40 min on one consumer GPU, within 21 GB.
Primary: Unknown (Affiliations not explicitly listed in provided text, likely independent or small lab given hardware constraints)
All Institutions: Unknown
Astra achieves quarter-degree probabilistic weather forecasting with 9.8% of GenCast's training compute and 39x less inference FLOPs by introducing two-sided transport attention (Kantorovich Attention) and a self-featurised CRPS objective, enabling high-resolution ensemble forecasting on a single consumer GPU.
The paper introduces "Astra," a probabilistic weather forecasting model that achieves 0.25° resolution with significantly reduced computational cost compared to state-of-the-art models like GenCast. The core methodological contributions are twofold: (1) **Kantorovich Attention (KA)**, a two-sided transport attention mechanism where each grid tile selects both where it reads from (pull) and where it writes to (push). This ensures every tile is read at token resolution, avoiding the "unread" tile problem of one-sided sparse attention, while maintaining linear complexity relative to grid points. (2) **Self-featurised CRPS**, a training objective that scores ensemble members not just in physical space but also in the model's own multiscale feature space using a frozen copy of the network. This encourages the model to learn representations that are discriminative and structurally consistent, similar to how diffusion models use pretrained encoders. The architecture is a W-Net operating directly on the 0.25° grid without a latent encoder, using MeanFlow for single-step generation.
The experimental evaluation is rigorous and comprehensive. Astra is benchmarked against GenCast (the current SOTA for high-res probabilistic forecasting) and the ECMWF IFS ENS (the operational gold standard). Key results include: (1) Matching GenCast's CRPS on upper-air targets with only 9.8% of the hardware-normalised training compute. (2) Outperforming IFS ENS on all 600 variable/level/lead-time combinations in CRPS and ensemble-mean RMSE. (3) Outperforming GenCast in CRPS on 73.5% of combinations. (4) Significant efficiency gains: 39x fewer FLOPs for inference than GenCast's GPU implementation, allowing 50-member 15-day forecasts on a single consumer GPU (RTX 5090) in under 40 minutes. Spectral analysis confirms that Astra preserves fine-scale spatial structure better than GenCast and IFS ENS, particularly in the Alps and tropics. The use of WeatherBench 2 and ERA5 verification provides a standard, reproducible benchmark.
The paper provides extensive methodological details, including equations for the attention mechanism, the transportation problem formulation, and the self-featurised CRPS loss. It specifies hardware requirements (8x H200 GPUs for training) and inference costs (RTX 5090). While the code is not provided in the text, the level of detail in the Methods section (e.g., specific tile sizes, number of heads, initialization schemes, and noise prior construction) is high, suggesting that a competent team could reproduce the results. The reliance on specific hardware (H200/RTX 5090) may limit immediate reproducibility for labs with older GPUs, but the algorithmic details are clear.
The primary limitation is the hardware dependency for training; while inference is efficient, training still requires 8x H200 GPUs, which is a significant barrier for smaller labs compared to models trained on fewer resources. The model is evaluated primarily on ERA5 reanalysis data; performance on real-time operational data with missing or noisy inputs is not tested. The "self-featurised" component relies on a frozen copy of the network, which adds complexity to the training pipeline and may introduce instability if the network's features drift significantly during training. Additionally, the paper focuses on medium-range forecasting (up to 15 days); performance at longer lead times (sub-seasonal) is not evaluated.
This paper has high potential impact by democratizing high-resolution probabilistic weather forecasting. By reducing inference costs to the level of a single consumer GPU, it enables local, real-time probabilistic forecasting for applications such as renewable energy management, agriculture, and disaster response, where cloud-based or high-performance computing access may be limited. The two-sided transport attention mechanism is a generalizable architectural innovation that could be applied to other high-resolution spatial tasks (e.g., remote sensing, medical imaging) where global context is needed but dense attention is prohibitive. The self-featurised CRPS offers a new paradigm for training generative models to produce structurally consistent samples, potentially influencing broader generative AI research. Astra achieves quarter-degree probabilistic weather forecasting with 9.8% of GenCast's training compute and 39x less inference FLOPs by introducing two-sided transport attention (Kantorovich Attention) and a self-featurised CRPS objective, enabling high-resolution ensemble forecasting on a single consumer GPU.
Statistical procedures often control expected loss at a prescribed level while leaving part of their risk budget unused. Building on the e-value representation of risk-controlled decisions and the e-closure principle for multiple testing, we formulate recertification, a general paradigm for improving procedures by exploiting computable slack in their validity certificates. A valid procedure's losses normalized by their risk budgets form a state-indexed e-value collection; conversely, decisions whose normalized losses are bounded by this collection inherit its risk guarantees. Recertification uses distributional structure to quantify slack, enlarge the certifying e-values while preserving validity, and construct less conservative decisions. We develop two applications. For conformal risk control (CRC) with a known loss family, recertification redistributes unused risk budget among the held-out points of each hypothetical augmented sample. The resulting controller is valid under every finite exchangeable law and no more conservative than CRC on any calibration sample. For convex losses on an interval of actions, repeated recertification converges geometrically to an admissible controller. For false discovery rate control with arbitrarily dependent e-values, recertification exploits unused conditional budgets in randomized e-BH. It yields a closed-form randomized procedure whose rejection set contains that of randomized e-BH for every realization of the shared randomization and whose expected number of rejections is maximal subject to the same conditional budget constraints. Recertifying with the larger e-collection of closed e-BH gives a second closed-form procedure whose rejection set contains that of the first. Experiments on CIFAR-100 hierarchical classification and multiple testing simulations demonstrate improved budget utilization, informativeness and power over the baselines.
Primary: University of Amsterdam
All Institutions: University of Amsterdam, University of Oxford, University of Cambridge
The paper proposes a novel "recertification" framework that uniformly improves conformal risk control and randomized e-BH procedures by exploiting unused risk budget in e-value certificates. It provides rigorous theoretical guarantees, including geometric convergence to admissible controllers for convex losses and closed-form power improvements for multiple testing, supported by targeted experiments on CIFAR-100 and simulations.
The paper introduces "recertification," a general paradigm for improving statistical decision procedures by exploiting slack in their validity certificates (e-values). The core contribution is a rigorous theoretical framework that maps valid procedures to e-value collections and vice versa, allowing the construction of less conservative decisions by enlarging these e-values while preserving validity guarantees. The authors apply this to Conformal Risk Control (CRC) and randomized e-BH (Benjamini-Hochberg) procedures. For CRC, they prove that repeated recertification converges geometrically to an admissible controller under convexity assumptions. For multiple testing, they derive closed-form randomized procedures (RU-eBH and RC-eBH) that uniformly dominate existing baselines in power. The methodology is mathematically dense but logically sound, leveraging the e-closure principle effectively.
The experimental section is limited but appropriate for a theory-heavy paper. It includes simulations for multiple testing showing power gains over e-BH and U-eBH, and a real-data study on CIFAR-100 hierarchical classification. The CIFAR-100 results demonstrate improved budget utilization and informativeness compared to standard CRC and envelope CRC. However, the scope of empirical validation is narrow, focusing primarily on one classification task and synthetic multiple testing scenarios, lacking broader benchmarks or large-scale real-world applications that would typically be expected for a high-impact ML paper.
The paper provides clear algorithmic descriptions (Algorithm 1) and theoretical conditions for validity. The use of standard datasets (CIFAR-100) and well-defined baselines (CRC, e-BH) aids reproducibility. However, the text is truncated in the provided input, and specific hyperparameters or code availability are not explicitly confirmed in the visible portion, though the mathematical definitions are precise enough for implementation.
The primary limitation is the computational cost of repeated recertification, which can be expensive for high-dimensional or individual-score based baselines, as noted in the "Computation" section. The admissibility results rely on convexity or specific finite-space conditions, which may not hold for all practical loss functions. Additionally, the empirical evaluation is relatively small-scale, limiting the generalizability of the observed performance gains to other domains.
This work has significant potential to influence the field of conformal prediction and multiple testing by providing a systematic way to reduce conservatism in statistical guarantees. It bridges decision theory and e-value methodology, offering a template for improving other risk-controlled procedures. The uniform improvement guarantees are particularly valuable for applications requiring strict error control, such as medical diagnosis or autonomous systems. The paper proposes a novel "recertification" framework that uniformly improves conformal risk control and randomized e-BH procedures by exploiting unused risk budget in e-value certificates. It provides rigorous theoretical guarantees, including geometric convergence to admissible controllers for convex losses and closed-form power improvements for multiple testing, supported by targeted experiments on CIFAR-100 and simulations.
Video world models trained on large-scale robot data promise a safe and scalable approach for learning and evaluating policies that act in the physical world. However, they still struggle with long-horizon rollouts and out-of-distribution generalization, and they remain computationally demanding at inference time. Typically, video world models make predictions in the latent space of a reconstruction-trained autoencoder, which however is not directly optimized for dynamics modeling. On the other hand, world models designed for latent planning usually optimize latents to be predictable, but they forgo accurate prediction of observations in pixel space. In this paper, we show that it is possible to achieve the best of both worlds: a latent geometry optimized to make future prediction as simple as possible without sacrificing accurate prediction of future observations. More precisely, we find that temporal objectives for representation learning, paired with a past-conditioned decoder architecture, lead to consistent improvements in generative video world models at the same training budget. The gains include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors out of distribution and over long horizons. Our world models also agree more closely with real-world policy evaluation when using a vision-language model to judge task success from their pixel-space predictions. Overall, our study indicates that temporal predictability is an important design criterion for the latent space of video world models, and provides concrete guidance on how to incorporate it into the training of autoencoders.
Primary: New York University
All Institutions: New York University, Empire AI Consortium
The paper demonstrates that optimizing latent spaces for temporal predictability significantly improves the efficiency and accuracy of video world models. By integrating temporal objectives into autoencoder training, the authors achieve a superior balance between reconstruction fidelity and dynamics simplicity, leading to substantial gains in inference speed, training efficiency, and out-of-distribution generalization, which are critical for real-world robotic applications.
The paper proposes a principled approach to designing the latent space for video world models. The core insight is that standard autoencoders optimized for reconstruction do not necessarily produce latents that are easy to predict dynamically. The authors introduce temporal objectives for representation learning combined with a past-conditioned decoder architecture. This allows the latent geometry to be optimized for simplicity of future prediction (dynamics) while maintaining high-fidelity reconstruction of observations. This addresses a fundamental trade-off in world modeling: the tension between accurate pixel-space reconstruction and efficient latent-space dynamics modeling. The methodology is sound, leveraging established concepts in representation learning (temporal consistency) and applying them specifically to the context of generative video world models for robotics.
The experimental evaluation is robust, demonstrating significant improvements over baselines. Key metrics include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors in out-of-distribution and long-horizon scenarios. The inclusion of a vision-language model (VLM) to judge task success from pixel-space predictions adds a layer of semantic evaluation that is highly relevant to the robotics application domain. The comparison with both reconstruction-focused and dynamics-focused baselines effectively isolates the contribution of the proposed latent design.
The paper provides a reproducibility statement and a link to the source code for fine-tuning autoencoders, training world models, and evaluation. The appendices detail objectives, architectures, training settings, and evaluation protocols. This level of detail supports reproducibility, although the specific hyperparameters for the "temporal objectives" would need to be verified in the code to ensure full transparency.
The primary limitation is the computational cost of training the autoencoders with temporal objectives, which may be prohibitive for very large-scale datasets without significant optimization. Additionally, the reliance on a VLM for evaluation introduces potential biases inherent in the VLM itself. The paper focuses on robot data, so generalizability to other video domains (e.g., natural videos) is not fully explored.
This work has significant implications for the field of world models, particularly for robotics and autonomous systems. By making video world models more efficient and accurate in long-horizon rollouts, it enables safer and more scalable policy learning and evaluation. The finding that temporal predictability is a key design criterion for latent spaces provides concrete guidance for future research in representation learning for dynamics. The paper demonstrates that optimizing latent spaces for temporal predictability significantly improves the efficiency and accuracy of video world models. By integrating temporal objectives into autoencoder training, the authors achieve a superior balance between reconstruction fidelity and dynamics simplicity, leading to substantial gains in inference speed, training efficiency, and out-of-distribution generalization, which are critical for real-world robotic applications.
Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally. Based on this insight, we introduce SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. SpecSlot expands each spatial anchor into multiple slots, each assigned a disjoint band of DCT coefficients; this separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds a local correction to the spectral prediction, while the generative state, target, and loss remain in pixel space. On ImageNet-256, SpecSlot achieves a comparable final FID to a FLOPs-matched JiT control while converging faster. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within only 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image generation, SpecSlot surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.
Primary: Peking University
All Institutions: Peking University, Nanyang Technological University, SenseTime
SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
The paper proposes SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from global attention sequence length. The core innovation is the "Role-Coupled Tokenization" where each spatial anchor is expanded into multiple slots, each assigned a disjoint band of DCT coefficients. Only the lowest-frequency slot participates in global attention, while high-frequency slots interact locally. This is a clever architectural modification that leverages the spectral properties of natural images (low-frequency structure requires global context, high-frequency detail is local). The method remains in pixel space, avoiding the artifacts often associated with VAE-based latent diffusion, while mitigating the quadratic cost of attention in pixel-space models. The inclusion of a spatial head for local correction is a sensible addition to handle residual errors.
The experiments are conducted on ImageNet-256 and ImageNet-512, as well as text-to-image generation benchmarks (DPG-Bench). The results show that SpecSlot achieves comparable FID to FLOPs-matched JiT controls on ImageNet-256 with faster convergence. On ImageNet-512, it improves FID by 2.84 over the control with fewer FLOPs and comes within 0.69 FID of a larger JiT model at significantly lower cost. The text-to-image results show improvements over JiT-B/8. The comparisons are against strong baselines (JiT), and the efficiency gains are substantial.
The paper provides links to the project page, GitHub repository, and Hugging Face model weights. The methodology is described in sufficient detail in the provided text (though the full technical details are in the sections referenced but not fully expanded in the prompt's text snippet, the abstract and summary provide the core mechanism). The availability of code and weights significantly enhances reproducibility.
The method relies on DCT coefficient bands, which may not be optimal for all types of content or modalities. The local interaction of high-frequency slots might limit the model's ability to capture long-range high-frequency correlations, although this is argued to be less critical. The computational overhead of managing multiple slots per anchor needs careful engineering to ensure the efficiency gains are realized in practice.
This work offers a viable path for pixel-space generative models to scale to higher resolutions without the prohibitive cost of full global attention. It could influence the design of future pixel-space Transformers and potentially other modalities where spectral decomposition is applicable. The efficiency gains make high-quality pixel-space generation more accessible. SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Primary: Tencent Youtu Lab
All Institutions: Tencent Youtu Lab
Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
The paper proposes Youtu-Parsing-Omni, a unified architecture for parsing documents, images, audio, and video into structured records. The core methodological contribution is the "Youtu-Omni-Encoder," which initializes from a pre-trained language model and uses shared weights to encode all modalities in a unified space-time coordinate system. A key architectural innovation is the inclusion of fusion layers that allow video frames to attend to concurrent audio, integrating cross-modal fusion directly into the perception layer. The training methodology introduces "OmniSchema-Aware On-Policy Distillation" (OSAD), a self-evolving post-training procedure where the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself. This allows the parser and its supervision to co-evolve, unifying the outputs of ten different parsing tasks under a single schema.
The paper reports state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.). The results highlight a significant advantage in audio and video parsing compared to general-purpose systems. It also achieves the best overall chart-parsing score (95.02) and high exact chemical-fingerprint agreement. The model matches dedicated optical-music models in character-level transcription of typeset sheet music. The evaluation covers a wide range of modalities and tasks, demonstrating the versatility of the unified approach.
The paper states that code and model weights are publicly available, which is a strong positive for reproducibility. However, the provided text does not include specific URLs or detailed hyperparameter settings, making it difficult to verify the exact implementation details without accessing the external repository. The description of the OSAD training procedure is high-level, and the specific mechanics of the "reference-conditioned copy" and "token-role-based objective" would require the code to fully understand and replicate.
The paper relies heavily on the performance of the pre-trained language model initialization, which may limit its applicability to domains where text is not the primary signal. The "self-evolving" nature of the OSAD training could be computationally expensive and potentially unstable, though this is not explicitly discussed. The evaluation is primarily on benchmark datasets, and real-world robustness to noisy or unstructured data is not thoroughly tested. The lack of specific URLs in the text prevents immediate verification of the claimed open-source availability.
This work has significant potential to impact the field of multimodal AI by providing a unified perception layer for agents that need to read, locate, and cite evidence across multiple modalities. The ability to parse documents, images, audio, and video into structured records using a single encoder and schema could simplify the development of complex AI agents. The focus on spatial and temporal grounding is particularly relevant for applications in robotics, autonomous systems, and advanced document analysis. Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni introduces two major innovations. First, we propose a unified cross-paradigm on-policy distillation (OPD) framework that successfully transfers the visual understanding ability of autoregressive VLMs and visual generation ability of continuous flow models, to a unified DLLM. Second, NLD-Uni incorporates a novel progressive self-refining diffusion process that gradually corrupts each token across multiple steps during training, as opposed to adopting the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality. Through extensive experiments, we show that NLD-Uni achieves state-of-the-art performance among unified DLLMs on diverse tasks including visual question-answering, multi-modal reasoning, object grounding, text-to-image generation, and image editing.
Primary: NVIDIA
All Institutions: NVIDIA
The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
The paper proposes NLD-Uni, a unified discrete diffusion large language model (DLLM) for visual tasks. The core methodological contribution is a "unified cross-paradigm on-policy distillation" framework. This approach attempts to transfer capabilities from two distinct source models—autoregressive VLMs (for understanding) and continuous flow models (for generation)—into a single discrete diffusion target. The second major innovation is a "progressive self-refining diffusion process." Unlike standard masked diffusion, which corrupts tokens to a uniform mask, this method gradually corrupts tokens across multiple steps during training. At inference, this allows for intermediate token commitments, theoretically enabling self-refinement. The methodology is complex and addresses a significant gap in unifying generation and understanding in discrete spaces, though the reliance on distillation from potentially proprietary or large teacher models adds complexity.
The paper claims state-of-the-art performance among unified DLLMs on VQA, multi-modal reasoning, object grounding, text-to-image generation, and image editing. However, the provided text is a placeholder ("sections/intro", "sections/experiments") and does not contain actual data, tables, or specific benchmark scores. Without concrete evidence of the "state-of-the-art" claims, the experimental evaluation cannot be rigorously verified. The scope of tasks (both understanding and generation) is broad, which is a strong point if the results hold up, but the lack of visible data in the prompt prevents a high confidence in the empirical rigor.
The paper describes specific training objectives (on-policy distillation, progressive corruption). However, without access to the full code, hyperparameters, and the specific teacher models used for distillation, reproducibility is moderate to low. The "Nemotron" branding suggests an internal NVIDIA model, which may not be fully open-source, limiting external reproducibility.
The primary limitation is the opacity of the experimental results in the provided text. Additionally, unified models often suffer from a "seesaw" effect where improving generation hurts understanding or vice versa; the paper claims to overcome this via distillation, but the trade-offs are not detailed in the abstract. The reliance on discrete tokens for image generation may still lag behind continuous diffusion/flow models in terms of raw fidelity, even with self-refinement.
If successful, this work could significantly advance the field of unified multimodal models by demonstrating that discrete diffusion can effectively handle both high-level reasoning and low-level pixel generation. This could lead to more efficient, single-model architectures for multimodal AI, reducing the need for separate VLMs and image generators. The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the victim's replies, wasting turns and limiting transferability; interactive generators read each victim reply to generate the next-turn attack but require in-loop evaluators for success signals or hints, incurring substantial costs and inflated performance. Bridging this gap, we propose SEMA-Auto, a sighted and efficient multi-turn attacker with autonomy, interacting with the victim model and terminating the session by itself upon success. To stabilize its behaviors and boost its capabilities, SEMA-Auto is trained in two stages. Synthetic bootstrap tuning installs the notion of attack targets and conforming decisions by fine-tuning the base model with bootstrapped trajectories. Decision-supervised reinforcement learning then optimizes two objectives jointly yet separately, reinforcing the attack with higher jailbreak reward while supervising the decision at each turn against an absolute target. Our method achieves state-of-the-art (SOTA) attack success rates (ASR) across multiple datasets, victims, and judges, outperforming all static and interactive baselines, within almost the fewest turns, demonstrating both effectiveness and efficiency. For example, SEMA-Auto performs a $76.0\%$ ASR@1 within an average of $3.0$ turns against DeepSeek-V4-Flash on AdvBench, while the best static baseline is $54.8\%$ with fixed $7$ turns and the best interactive baseline is $43.8\%$ in $3.7$ turns. Against safety-hardened frontier models, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, and GPT-6-Astra, our counter-trained attacker achieves an average $59.6\%$ ASR@1 on HarmBench, $13.8\%$ over prior SOTA. Our code is available at: https://github.com/microsoft/SEMA
Primary: Microsoft Research
All Institutions: Microsoft Research
SEMA-Auto introduces a sighted, efficient, and autonomous multi-turn jailbreak attacker that achieves SOTA performance by combining synthetic bootstrap tuning with decision-supervised reinforcement learning. The paper presents a rigorous methodology that addresses the inefficiencies of existing interactive attackers by enabling autonomous termination, resulting in significant improvements in attack success rates and efficiency across diverse victim models and datasets.
The paper proposes SEMA-Auto, a multi-turn jailbreak attacker that bridges the gap between static planners (which lack adaptability) and interactive generators (which are costly and rely on in-loop evaluators). The core innovation lies in a two-stage training pipeline: (1) Synthetic Bootstrap Tuning to instill attack targets and decision-making logic via bootstrapped trajectories, and (2) Decision-Supervised Reinforcement Learning (DSRL). The DSRL stage is particularly notable for jointly optimizing two separate objectives: maximizing jailbreak reward and supervising the termination decision against an absolute target. This allows the model to autonomously determine when an attack has succeeded without requiring an external judge during inference, thereby reducing cost and latency. The method effectively addresses the "sighted" aspect by reading victim replies while maintaining efficiency through autonomous termination.
The experimental section reports State-of-the-Art (SOTA) Attack Success Rates (ASR) across multiple datasets (AdvBench, HarmBench) and victim models (DeepSeek-V4-Flash, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, GPT-6-Astra). The results show significant improvements over both static and interactive baselines. For instance, on AdvBench against DeepSeek-V4-Flash, SEMA-Auto achieves 76.0% ASR@1 in an average of 3.0 turns, compared to 54.8% for the best static baseline (7 turns) and 43.8% for the best interactive baseline (3.7 turns). Against safety-hardened frontier models, it achieves an average 59.6% ASR@1 on HarmBench, outperforming prior SOTA by 13.8%. The evaluation covers a wide range of model capabilities and safety levels, providing a robust assessment of the method's transferability and effectiveness.
The authors provide a link to their code repository (https://github.com/microsoft/SEMA), which is a positive indicator for reproducibility. The methodology is described in sufficient detail, including the two-stage training process and the specific objectives of the reinforcement learning phase. However, as is common with preprints under review, some specific hyperparameters or implementation details might be found in the appendix (referenced but not fully visible in the provided text snippet). The use of standard benchmarks and open-source models (where applicable) further aids reproducibility.
The paper focuses on jailbreak attacks, which is a sensitive area. While the method is efficient, the reliance on synthetic bootstrap data may introduce biases or limit the diversity of attack strategies. The autonomous termination mechanism, while efficient, might occasionally lead to premature termination if the model's internal confidence is miscalibrated. Additionally, the performance against "safety-hardened frontier models" suggests that while the method is effective, it is not infallible, and the gap between high-performing and low-performing victims remains significant. The paper does not extensively discuss the potential for defensive mechanisms to counter this specific type of autonomous, efficient attacker.
This work has significant implications for LLM safety and security. By demonstrating a more efficient and autonomous jailbreak method, it highlights the limitations of current safety measures and the need for more robust defenses. The efficiency of the method (fewer turns, no in-loop evaluators) makes it a more practical threat, potentially lowering the barrier to entry for malicious actors. Conversely, the techniques developed for autonomous decision-making in adversarial contexts could be adapted for defensive purposes, such as creating more effective red-teaming tools or safety evaluators. The paper contributes to the ongoing arms race in AI safety, pushing the field to develop more resilient models. SEMA-Auto introduces a sighted, efficient, and autonomous multi-turn jailbreak attacker that achieves SOTA performance by combining synthetic bootstrap tuning with decision-supervised reinforcement learning. The paper presents a rigorous methodology that addresses the inefficiencies of existing interactive attackers by enabling autonomous termination, resulting in significant improvements in attack success rates and efficiency across diverse victim models and datasets.
Individual-level genotype data underpin genetic discovery, disease-risk modeling, and therapeutic development, yet many critical settings remain data-limited, including underrepresented ancestries, rare or imbalanced disease cohorts, and restricted-access biobanks. Conditional synthetic genome generation would support privacy-aware benchmarking, simulation, and data augmentation, but requires learning an ultra-high-dimensional discrete distribution over hundreds of thousands to tens of millions of correlated single-nucleotide polymorphisms (SNPs), or sites of DNA variation. We introduce HELICS: High-fidelity gEneration via Latent Interpolant Conditional Synthesis, a phasing-free latent flow matching framework for genome-wide SNP-level generation from biobank-scale genotype data. HELICS tokenizes chromosomes with convolutional autoencoders, reducing >600K SNPs to 4,096 continuous latent tokens, and trains a transformer parameterized conditional flow matching model over the concatenated multi-chromosome latent representation. Trained on over 417K genomes in the UK Biobank, HELICS generates genome-wide common-variant synthetic cohorts without haplotype phasing and achieves stronger fidelity to real genotypes than existing haplotype-based simulators, including high allele-frequency agreement in external evaluation and superior preservation of local and long-range SNP correlation structure. Conditioning on ancestry and genetic liability summaries enables targeted generation across genetic risk profiles; in perturbation analyses, by increasing the conditioned disease trait liability, HELICS induces SNP-level dosage changes in generated genomes that are highly correlated with published association weights. HELICS provides a scalable generative modeling framework for SNP-level genome data and a path toward conditional synthetic cohorts for data-limited genetic studies.
Primary: Harvard University
All Institutions: Harvard University
HELICS introduces a scalable, phasing-free latent flow matching framework for genome-wide synthetic genome generation that achieves high fidelity to real biobank data and enables controllable generation of genetic liability profiles, representing a significant advancement in computational genomics and generative modeling for high-dimensional discrete biological data.
The paper introduces HELICS, a framework for generating synthetic genome-wide SNP data using latent flow matching. The core methodological innovation is the decoupling of the ultra-high-dimensional discrete genotype space (600K+ SNPs) into a manageable continuous latent space via chromosome-specific convolutional autoencoders. This allows the application of a transformer-based flow matching model (USiT) to the concatenated latent tokens of all 22 autosomes. The approach is "phasing-free," meaning it operates directly on unphased dosage data, avoiding the computationally expensive and error-prone step of haplotype phasing required by traditional simulators like HAPGEN2 or HAPNEST. The autoencoder training objective is carefully designed to preserve population genetic structure, including terms for minor allele frequency (MAF) preservation and local linkage disequilibrium (LD) correlation. The flow matching model is conditioned on ancestry principal components and discretized polygenic risk scores (PRS), enabling controllable generation of genetic liability profiles.
The evaluation is rigorous and extensive. The model is trained on the UK Biobank (417K genomes) and evaluated on external cohorts (1000 Genomes, CCDG). Key metrics include allele frequency agreement ($R^2=0.992$), LD structure preservation, and population geometry (PCA/UMAP). A particularly strong aspect of the evaluation is the "perturbation analysis," where the authors demonstrate that increasing the conditioned disease liability (e.g., HDL) induces SNP-level dosage changes that correlate highly with published GWAS association weights ($R^2=0.547$ for high-effect variants). This provides strong evidence that the model learns biologically meaningful associations rather than just statistical correlations. The model outperforms existing haplotype-based simulators in fidelity and offers capabilities (conditional generation) that baselines lack.
The authors provide a GitHub repository with code. However, due to the sensitivity of UK Biobank data, the full model weights and training data are not released. A demo is provided using 1000 Genomes data on chromosome 22, which allows for verification of the pipeline but not the full-scale performance claims. The architectural details, including the dynamic programming for token allocation and specific loss terms, are described in detail in the appendix, aiding reproducibility for those with access to similar-scale data.
The model is currently limited to common variants (MAF > 1%) and does not handle rare variants or structural variants. Fidelity is weaker for underrepresented ancestry groups, likely due to data imbalance in the training set. The conditioning is based on coarse PRS deciles, which may limit the granularity of control. The privacy implications of synthetic data generation are discussed but not exhaustively proven; while membership inference attacks show low leakage, the risk of re-identification in specific contexts remains a topic for further study.
This work has significant potential to impact genetic research by enabling the creation of synthetic cohorts for underrepresented populations or rare diseases, where real data is scarce. It facilitates privacy-aware sharing of genetic data and allows for controlled simulation of genetic risk profiles. The ability to generate data conditioned on specific genetic liabilities could accelerate the discovery of new genetic associations and improve the generalizability of polygenic risk scores. HELICS introduces a scalable, phasing-free latent flow matching framework for genome-wide synthetic genome generation that achieves high fidelity to real biobank data and enables controllable generation of genetic liability profiles, representing a significant advancement in computational genomics and generative modeling for high-dimensional discrete biological data.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
Future planetary rovers will traverse steep, deformable terrain where slip, sinkage, and terrain deformation can threaten mobility. Although simulation can capture some wheel-soil effects, unmodeled deformable-terrain phenomena create sim-to-real gaps requiring deployment-time adaptation. Yet, rover operations leave little room for extensive trial-and-error. We introduce TERRA (Targeted Exploration for Reward-Relevant Adaptation), an online adaptation framework that efficiently seeks deployment data exposing task-relevant model errors and corrects them. TERRA adapts a simulation-trained world model with a reward-weighted prediction loss and trains a loss predictor to estimate where future performance-relevant errors are likely to occur, using this signal to guide exploration during MPC. In simulation, TERRA reaches approximately 90% slope-climbing success within five trials under severe out-of-distribution pretraining conditions, outperforming exploration baselines in reward, completion time, and energy to first success. On the ERNEST rover, TERRA recovers coordinated active-suspension locomotion on a deformable Mars-simulant slope within three to five real-world trials. Project website: https://terra-explore.github.io
Primary: Jet Propulsion Laboratory, California Institute of Technology
All Institutions: Jet Propulsion Laboratory, California Institute of Technology, University of Texas at Austin
TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
The paper introduces TERRA, a framework for online adaptation of simulation-trained world models for planetary rovers. The core methodological contributions are twofold: (1) a reward-weighted prediction loss that prioritizes model errors in observation dimensions that have high sensitivity to the task reward, and (2) a learned "loss predictor" that forecasts where these reward-relevant errors are likely to occur in candidate action sequences, thereby guiding exploration within a Model Predictive Control (MPC) loop. The use of Low-Rank Adaptation (LoRA) for efficient onboard updating of the dynamics model is a practical engineering choice. The approach is logically sound, addressing the specific bottleneck of sim-to-real gaps in deformable terrain where standard prediction-error-based exploration is inefficient.
The evaluation is strong, featuring both extensive simulation studies (100 seeds, multiple OOD pretraining conditions) and real-world validation on the ERNEST rover. The simulation results show significant improvements over baselines like RND and Plan2Explore in terms of success rate, completion time, and energy efficiency. The real-world results are particularly compelling, demonstrating the recovery of coordinated active-suspension locomotion within 3-5 trials on a Mars-simulant slope, a task that is critical for future missions. The ablation studies effectively isolate the contributions of the reward-weighted loss and the targeted exploration.
The paper provides detailed descriptions of the simulation environment, pretraining conditions, and hyperparameters. However, as is common in robotics papers involving proprietary hardware (ERNEST) and specific simulators (DARTS), full reproducibility by external groups may be limited. The project website is provided, which likely contains code or further details.
The method relies on the availability of a task reward during deployment, which may not always be easily defined or observable in complex real-world scenarios. The choice of loss-bin thresholds for the loss predictor is a hyperparameter that requires tuning. The evaluation is limited to a single rover platform (ERNEST) and a specific type of terrain (deformable slopes), so generalizability to other platforms or terrain types is not fully established.
This work has high relevance to the field of planetary robotics and autonomous systems operating in unstructured environments. The technique of reward-weighted model adaptation and targeted exploration is applicable beyond rovers to any mobile robot facing sim-to-real gaps in contact-rich or deformable environments. It provides a practical pathway for deploying learned policies in high-stakes, low-trial-count scenarios. TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
Discretizing continuous actions into discrete skills using methods like VQ-VAE has emerged as a powerful paradigm for robotic manipulation. However, quantization errors yield a suboptimal distribution for the skill prior, degrading its performance. While reinforcement learning offers a path for refinement, its direct application faces unstable encoder updates and a granularity dilemma in importance sampling. To address these challenges, we introduce Cascaded Skills Optimization (CSO), a two-stage post-training framework. First, Rejection-Sampling Supervised Fine-tuning aligns the model's observation-to-skill mapping with the distribution of successful online trajectories via supervised fine-tuning. Second, with the observation encoder frozen, Skills Policy Optimization computes an independent, clipped importance ratio for each complete skill, enabling stable and efficient updates. With a lightweight QueST-style policy, CSO raises overall LIBERO success from 80.1% to 97.5%, improves performance on RoboTwin2.0, and achieves 61.0% success across five physical single-arm tasks. Additional comparisons with FAST, discrete BEAST, and fixed-buffer IQL further validate the effectiveness of our two-stage post-training strategy.
Primary: Unknown
All Institutions: Unknown
[One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
The paper proposes CSO (Cascaded Skills Optimization), a two-stage post-training framework for refining skill-based robotic policies. The methodology is logically sound and addresses specific pain points in applying RL to discrete skill policies. Stage 1 (R-SFT) uses rejection sampling to collect successful trajectories and fine-tunes the observation encoder, which is then frozen. This is a pragmatic solution to the instability of updating visual encoders with sparse rewards. Stage 2 (SPO) introduces a skill-level importance sampling ratio, arguing that token-level (too fine) and episode-level (too coarse) granularities are suboptimal for robotic skills. The theoretical justification for the "granularity dilemma" is the core intellectual contribution. The approach is essentially a structured application of PPO variants (specifically GSPO/GRPO concepts) adapted to the hierarchical structure of skill-based policies. While the adaptation is clever, the underlying RL machinery is not new.
The experimental evaluation is rigorous and comprehensive. The authors test on standard simulation benchmarks (LIBERO, RoboTwin2.0, MetaWorld) and, crucially, on physical robots (Cobot Agilex ALOHA). The results show significant improvements over the imitation learning baseline (80.1% to 97.5% on LIBERO) and outperform strong baselines like RIPT and FAST. The ablation studies effectively isolate the contributions of R-SFT and SPO, and the granularity analysis convincingly demonstrates the superiority of skill-level optimization over token- and episode-level alternatives. The physical robot experiments, while limited to five tasks, provide valuable validation of the method's real-world applicability.
The paper provides detailed implementation specifics, including hyperparameters for both stages, architecture details for the lightweight and VLM-based variants, and data collection protocols. The use of standard benchmarks and clear descriptions of the training process enhances reproducibility. However, the lack of a public code repository link in the provided text is a minor drawback, though the detailed appendices mitigate this.
The method requires a sufficient number of successful trajectories to bootstrap the R-SFT stage, which may be difficult for very hard tasks without a strong initial policy. The frozen encoder in Stage 2 limits the model's ability to adapt its perception to new failure modes discovered during RL. The physical experiments are limited to single-arm tasks, and the method's scalability to more complex, multi-arm, or dexterous manipulation scenarios is not fully explored.
This paper contributes to the growing field of post-training for robotic policies, offering a practical and effective recipe for improving skill-based agents. The insight regarding optimization granularity is likely to influence future work in hierarchical RL and skill-based control. The framework is general enough to be applied to other skill-based architectures, making it a useful tool for the robotics community. [One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, University of Maryland, Toyota Research Institute
TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
The paper proposes TAPESTRY, a method to repurpose pretrained video diffusion transformers (specifically CogVideoX-2B) for robot control by treating robot actions as single-patch tokens in the video latent space. The core innovation is the "action-as-VAE-patch" tokenization, which uses a lightweight linear projection and LayerNorm to map action vectors into the same distribution as video VAE latents, allowing them to pass through the pretrained patchifying convolution without architectural modification. The training procedure involves three joint diffusion objectives: video-only, joint video+action, and action-only. A significant technical contribution is the analysis of high-frequency noise in multi-step denoising for action chunks, demonstrating that single-step DDIM inference from pure noise is superior for this specific repurposed backbone due to frequency-space mismatches between video and action signals.
The evaluation is rigorous, covering simulation (LIBERO benchmark with a learning-from-video protocol) and real-world bimanual manipulation. The paper demonstrates strong generalization to novel objects and motion primitives. The inference efficiency claims are well-supported, showing a 44x speedup over optimized DreamZero on consumer hardware (RTX 5090). The ablations effectively isolate the contribution of the video pretraining and the specific training objectives.
The paper provides detailed implementation specifics, including the exact parameter count of the action head (1280 parameters), the specific backbone used (CogVideoX-2B), and the training data mixture. The derivation of the single-step inference collapse is provided in the appendix. However, reliance on specific proprietary or large-scale datasets (like DROID, AgiBot-World) may limit immediate reproducibility for smaller labs, though the architectural changes are minimal and easy to implement.
The method is currently limited to a 20-DoF bimanual action space, and the single-step inference relies on the zero-terminal-SNR v-prediction schedule, which may not transfer to other backbones like rectified-flow models (Wan). The deterministic nature of single-step inference may be a limitation for tasks requiring multimodal action distributions.
This work significantly lowers the barrier to entry for using large video models in robotics by eliminating the need for expensive video generation at test time. The insight that action tokens can be treated as video patches with minimal projection is a generalizable architectural pattern that could be applied to other multimodal control tasks. The finding regarding high-frequency noise in repurposed diffusion models is a valuable diagnostic for the broader community working on world action models. TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley
The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.
The paper proposes a comprehensive sim-to-real pipeline for in-hand object pose reaching using a 16-DoF LEAP hand. The core methodological contributions are: (1) A training recipe that leverages a large-scale generated grasp dataset (via LightningGrasp and DexterityGen) to define diverse goals and resets, avoiding the need for human demonstrations. (2) An adaptive friction curriculum that starts with high friction to stabilize learning and gradually reduces it to match reality, which is shown to be critical for convergence. (3) A modular safety approach using a learned residual safety filter (adapted from ResSafe) that operates on top of a frozen task policy. This filter uses a tunable weight $w$ to balance performance and safety, allowing for a "dial" to adjust conservativeness at deployment without retraining the task policy. The system identification is rigorous, using a voltage-driven DC motor model (BAM) and precise latency calibration to bridge the sim-to-real gap.
The experimental evaluation is extensive and rigorous, particularly for a preprint. The authors conduct 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and 6 wrist orientations. The state-based policy achieves a 95.2% success rate, and the safety filter reduces drops by 75% while maintaining a 91.5% success rate. The paper also evaluates robustness to out-of-distribution goals, fast arm motion, and physical perturbations. Comparisons with Hamilton-Jacobi (HJ) safety filters demonstrate the superiority of the learned residual approach in terms of computational efficiency and performance trade-offs. The inclusion of a depth-camera-only policy that transfers zero-shot is a strong validation of the visual sim-to-real gap mitigation.
The paper provides high reproducibility. It details the hardware setup, system identification process, simulation parameters, and training hyperparameters. The authors commit to releasing the full recipe, code, and trained checkpoints. The use of standard tools like MuJoCo Warp and PPO, along with clear descriptions of the reward functions and curricula, makes it feasible for other researchers to replicate the results. The specific details on latency calibration and actuator modeling are particularly valuable for the robotics community.
The policies are trained per object, which limits generalization to new objects without retraining. The depth policies still show a visual sim-to-real gap on certain objects (knife, wineglass). The safety filter is also trained per object and per nominal policy. The downstream tasks (e.g., screwing in a lightbulb) rely on hard-coded arm motions, limiting the autonomy of the system. The evaluation is limited to a specific hand (LEAP) and arm (UR5), so generalizability to other hardware is not tested.
This work significantly advances the state of the art in dexterous manipulation by addressing the critical issue of safety and reliability in high-dimensional contact-rich systems. The modular safety filter approach is likely to be adopted in other robotic domains where safety is paramount. The comprehensive evaluation framework for measuring performance, reliability, and safety provides a valuable benchmark for the field. The ability to operate with only a wrist-mounted camera expands the practical applicability of such systems beyond controlled lab environments. The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.