Last 7 Days (October 27 – November 02, 2026)
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
AI has made remarkable progress in mathematics, solving challenging problems with machine-checkable proofs in Lean, yet typically relying on predefined mathematical definitions and formal specifications. This raises a natural question: can it reason rigorously about an unknown world? We introduce the Algebraic World Model (AWM), a framework that turns interaction with unfamiliar environments into explicit mathematical models supporting formal reasoning. AWM explores the environment and develops executable Python hypotheses, continually testing and revising them against observations. It then structures these hypotheses into typed objects and shared rules, simplifies their structure through category-theory-guided transformations, and formalizes the resulting model in Lean 4 to prove reachability and impossibility. AWM can formally prove that a goal is unreachable, even beyond explored trajectories. On two impossible game levels, it constructs closed, kernel-checked Lean proofs excluding every legal winning sequence in all six runs, while Claude Opus 5.5 and GPT-6 Astra, equipped with coding tools, produce none in twelve baseline runs. Four of AWM's six initial bounds are correct, compared with one for Opus 5.5 and none for GPT-6 Astra; given only feedback that a cheaper win exists, AWM revises its rules and constructs new proofs, reaching six of six correct bounds. Moreover, AWM produces interpretable mathematical models whose rules can be inspected and tested step by step. Across all 25 public ARC-AGI-3 games, these models enable Lean to replay the winning trajectories of all 183 levels and generalize beyond observed play, predicting unseen transitions and detecting changes in game mechanics. With Claude Opus 5, AWM clears all levels with a score of 99.9, while its certified winning trajectories use only 43% of the human baseline's actions.
Primary: unknown
All Institutions: unknown
The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
The paper introduces the Algebraic World Model (AWM), a framework that bridges empirical exploration and formal verification. The methodology is structured in three stages: (1) Explore & Prototype, where an LLM agent interacts with an environment to generate executable Python hypotheses; (2) Structure & Simplify, where these hypotheses are organized into typed objects and rules using category-theory-guided transformations (limits/colimits) verified in Lean; and (3) Formalize & Prove, where the model is formalized in Lean 4 to prove reachability or impossibility. The use of category theory to guide the simplification of world models is a creative and rigorous approach to ensuring that the learned model preserves the logical structure of the environment. The "sufficient cover" definition provides a theoretical basis for transferring proofs from the learned model to the actual environment, which is a significant methodological contribution.
The experiments are conducted on ARC-AGI-3, a challenging benchmark for abstract reasoning. The results are strong: AWM achieves a score of 99.9, outperforming baselines like Claude Opus 5.5 and GPT-6 Astra in both game clearing and, crucially, in constructing formal proofs of impossibility. The paper demonstrates that AWM can produce kernel-checked Lean proofs that exclude all legal winning sequences for impossible levels, a capability the baselines lack. The generalization tests, where frozen models predict unseen transitions and detect mechanic changes, further validate the robustness of the learned algebraic structures.
The paper provides a high level of reproducibility by releasing an interactive website with models, tests, Lean proofs, and links to official replays. The inclusion of the Lean model for the running example (lp85) in the supplementary materials, along with scripts for building and auditing axioms, allows for independent verification of the formal proofs. The explicit statement that all numbers come from script outputs over recorded runs, rather than model-generated text, enhances trust in the results.
The primary limitation is the scope of the environments tested. AWM is demonstrated on ARC-like environments with discrete actions and immediate, observable feedback. The authors acknowledge that richer hidden states, delayed feedback, and stochastic dynamics remain untested. Additionally, the reliance on LLMs for the initial hypothesis generation and exploration phase means that the system's performance is bounded by the capabilities of the underlying LLM. The "sufficient cover" condition, while theoretically sound, requires careful verification in practice to ensure that the learned model indeed covers the optimal winning sequences of the environment.
This work has significant implications for the field of AI, particularly in the areas of formal methods, automated theorem proving, and agent-based reasoning. By enabling AI systems to reason rigorously about unknown worlds and produce machine-checkable proofs, AWM paves the way for more trustworthy and interpretable AI systems. The integration of category theory with machine learning offers a new paradigm for structuring and simplifying learned models, which could be applied to other domains requiring rigorous reasoning. The ability to prove impossibility, not just reachability, is a crucial step towards AI systems that can provide complete and reliable solutions to complex problems. The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
Data-driven learning for the solution operator of a differential equation has broad scientific importance, but observations from a target system are often scarce and costly to collect. Related source systems may provide abundant observations under similar physical conditions, whereas even small changes in the governing equation can induce complex changes in the solution operator, making naive data pooling ineffective. We propose a structured transfer learning method that accounts for unknown sparse differences in the governing equations from a known library of candidate terms. Using the resolvent identity and a whitened instrumental variable approach, we construct unbiased source estimating equations for the target operator. Building on these equations, we develop a three-stage estimator that consists of a target-only ridge pilot, source-wise sparse calibration, and a projected refit designed to remove the leading calibration error inherited from the pilot. An iterative recalibration scheme further reduces the remaining error. We establish non-asymptotic prediction bounds showing when transfer achieves the pooled all-source rate, together with minimax lower bounds that characterize both rate optimality and fundamental identifiability limits under common shifts. Simulations and an electronic band-pass filter application demonstrate clear gains over target-only learning, naive pooling and existing transfer methods.
Primary: National University of Singapore
All Institutions: National University of Singapore, Tongji University
The paper introduces a principled transfer learning framework for PDE solution operators that exploits sparse differences in governing equations via instrumental variables and projected refitting, achieving minimax optimal rates under specific identifiability conditions. This represents a significant methodological advance in statistical operator learning, offering a rigorous alternative to heuristic neural operator fine-tuning approaches by providing clear theoretical guarantees on when and how source data improves target estimation.
The paper proposes a rigorous statistical framework for transfer learning of solution operators of partial differential equations (PDEs). The core innovation is the use of the resolvent identity to relate source systems (with known sparse perturbations in their governing equations) to the target system. By treating the source forcing as an instrumental variable and applying a whitening transformation, the authors construct unbiased estimating equations for the target operator. A key technical contribution is the "projected refit" stage, which uses orthogonal projection to remove the leading-order calibration error inherited from the initial target-only pilot estimator. This allows the method to leverage large source datasets without being biased by the small target sample size used for initial calibration. The theoretical analysis is extensive, providing non-asymptotic prediction bounds and minimax lower bounds that characterize the conditions under which transfer learning achieves the optimal all-source rate.
The experimental section includes simulations on functional regression and advection-diffusion equations, as well as a real-world application to an electronic band-pass filter. The results demonstrate clear improvements over target-only learning, naive pooling, and existing transfer learning methods (such as those by Jiao-McKeague, Lin-Reimherr, and Hu-Lin). The reduction in spectral prediction error for the filter application (approx. 36.3%) is a strong practical result. However, the scope of experiments is limited to linear PDEs and specific physical benchmarks, which may not fully generalize to the complex, nonlinear PDEs common in modern scientific machine learning.
The paper provides detailed algorithmic descriptions and theoretical conditions. However, as an arXiv preprint, code availability is not explicitly confirmed in the text provided. The tuning parameters and specific implementation details for the baselines are referenced in appendices, which are not fully visible in the truncated text, but the methodology is described with sufficient precision for a skilled statistician to implement.
The method assumes that the differences between source and target governing equations are sparse combinations of known candidate terms. This "library" assumption may not hold for all physical systems where the governing laws are unknown or highly complex. Additionally, the method is designed for linear operators; extending it to nonlinear PDEs would require significant additional work. The computational cost of the iterative recalibration and the projection steps could be high for very high-dimensional function spaces, although the convex nature of the sub-problems helps.
This work bridges the gap between statistical transfer learning and scientific computing/PDE operator learning. It provides a principled way to utilize historical data from related physical systems, which is a major bottleneck in data-driven simulation. The rigorous statistical guarantees (minimax optimality) are valuable for applications where reliability is critical, such as safety-critical engineering or medical imaging. It sets a new standard for how structural knowledge (sparse equation differences) can be exploited in operator learning. The paper introduces a principled transfer learning framework for PDE solution operators that exploits sparse differences in governing equations via instrumental variables and projected refitting, achieving minimax optimal rates under specific identifiability conditions. This represents a significant methodological advance in statistical operator learning, offering a rigorous alternative to heuristic neural operator fine-tuning approaches by providing clear theoretical guarantees on when and how source data improves target estimation.
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
AI has made remarkable progress in mathematics, solving challenging problems with machine-checkable proofs in Lean, yet typically relying on predefined mathematical definitions and formal specifications. This raises a natural question: can it reason rigorously about an unknown world? We introduce the Algebraic World Model (AWM), a framework that turns interaction with unfamiliar environments into explicit mathematical models supporting formal reasoning. AWM explores the environment and develops executable Python hypotheses, continually testing and revising them against observations. It then structures these hypotheses into typed objects and shared rules, simplifies their structure through category-theory-guided transformations, and formalizes the resulting model in Lean 4 to prove reachability and impossibility. AWM can formally prove that a goal is unreachable, even beyond explored trajectories. On two impossible game levels, it constructs closed, kernel-checked Lean proofs excluding every legal winning sequence in all six runs, while Claude Opus 5.5 and GPT-6 Astra, equipped with coding tools, produce none in twelve baseline runs. Four of AWM's six initial bounds are correct, compared with one for Opus 5.5 and none for GPT-6 Astra; given only feedback that a cheaper win exists, AWM revises its rules and constructs new proofs, reaching six of six correct bounds. Moreover, AWM produces interpretable mathematical models whose rules can be inspected and tested step by step. Across all 25 public ARC-AGI-3 games, these models enable Lean to replay the winning trajectories of all 183 levels and generalize beyond observed play, predicting unseen transitions and detecting changes in game mechanics. With Claude Opus 5, AWM clears all levels with a score of 99.9, while its certified winning trajectories use only 43% of the human baseline's actions.
Primary: unknown
All Institutions: unknown
The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
The paper introduces the Algebraic World Model (AWM), a framework that bridges empirical exploration and formal verification. The methodology is structured in three stages: (1) Explore & Prototype, where an LLM agent interacts with an environment to generate executable Python hypotheses; (2) Structure & Simplify, where these hypotheses are organized into typed objects and rules using category-theory-guided transformations (limits/colimits) verified in Lean; and (3) Formalize & Prove, where the model is formalized in Lean 4 to prove reachability or impossibility. The use of category theory to guide the simplification of world models is a creative and rigorous approach to ensuring that the learned model preserves the logical structure of the environment. The "sufficient cover" definition provides a theoretical basis for transferring proofs from the learned model to the actual environment, which is a significant methodological contribution.
The experiments are conducted on ARC-AGI-3, a challenging benchmark for abstract reasoning. The results are strong: AWM achieves a score of 99.9, outperforming baselines like Claude Opus 5.5 and GPT-6 Astra in both game clearing and, crucially, in constructing formal proofs of impossibility. The paper demonstrates that AWM can produce kernel-checked Lean proofs that exclude all legal winning sequences for impossible levels, a capability the baselines lack. The generalization tests, where frozen models predict unseen transitions and detect mechanic changes, further validate the robustness of the learned algebraic structures.
The paper provides a high level of reproducibility by releasing an interactive website with models, tests, Lean proofs, and links to official replays. The inclusion of the Lean model for the running example (lp85) in the supplementary materials, along with scripts for building and auditing axioms, allows for independent verification of the formal proofs. The explicit statement that all numbers come from script outputs over recorded runs, rather than model-generated text, enhances trust in the results.
The primary limitation is the scope of the environments tested. AWM is demonstrated on ARC-like environments with discrete actions and immediate, observable feedback. The authors acknowledge that richer hidden states, delayed feedback, and stochastic dynamics remain untested. Additionally, the reliance on LLMs for the initial hypothesis generation and exploration phase means that the system's performance is bounded by the capabilities of the underlying LLM. The "sufficient cover" condition, while theoretically sound, requires careful verification in practice to ensure that the learned model indeed covers the optimal winning sequences of the environment.
This work has significant implications for the field of AI, particularly in the areas of formal methods, automated theorem proving, and agent-based reasoning. By enabling AI systems to reason rigorously about unknown worlds and produce machine-checkable proofs, AWM paves the way for more trustworthy and interpretable AI systems. The integration of category theory with machine learning offers a new paradigm for structuring and simplifying learned models, which could be applied to other domains requiring rigorous reasoning. The ability to prove impossibility, not just reachability, is a crucial step towards AI systems that can provide complete and reliable solutions to complex problems. The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
Graph-based nearest neighbor search algorithms like HNSW, DiskANN, NSG, and others have surged in popularity in recent years due to their excellent empirical performance in comparison to space-partitioning methods like IVF data structures and Locality Sensitive Hashing (LSH). However, in contrast to LSH and related methods, obtaining strong worst-case theoretical guarantees for graph-based search has proven challenging. In this paper, we prove a strong lower bound that helps explain the lack of progress towards such guarantees. We show that no standard graph-based method (any that runs a variant of greedy search on a flat or hierarchical graph) can achieve sublinear query time for approximate nearest neighbor search in Euclidean space, even for an arbitrarily large approximation factor. On the other hand, we prove that sublinear query time is possible if we allow for Steiner points in the search graph. We obtain bounds comparable to LSH, showing e.g., that graph-based search with Steiner points can solve $c$-approximate nearest neighbor search in Euclidean space with query time $\tilde{O}(n^{1/(c-1)})$. We complement this result with initial experiments exploring how Steiner points can be incorporated into practical graph-based methods.
Primary: New York University
All Institutions: New York University, UMass Amherst
The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
The paper presents a rigorous theoretical framework for analyzing graph-based nearest neighbor search (NNS) algorithms like HNSW and DiskANN. The core methodological contribution is the formalization of a "flat search graph" model that captures generalized greedy search, allowing for the derivation of strong lower bounds. The authors prove that without Steiner points, no such method can achieve sublinear query time in worst-case Euclidean high-dimensional settings, effectively explaining the long-standing gap between empirical performance and theoretical guarantees. Conversely, they introduce a novel construction using Steiner points to simulate Locality Sensitive Hashing (LSH) within the graph search framework, achieving query times comparable to LSH ($\tilde{O}(n^{1/(c-1)})$). The proof techniques involve constructing hard instances of orthogonal vectors and leveraging geometric properties of Steiner nodes to implement hash table lookups via greedy traversal.
The experimental section is limited but sufficient for an initial exploration. The authors augment the ParlayANN implementation of DiskANN with simple $k$-means cluster centroids as Steiner points. They evaluate this on standard benchmarks (GloVe-200, NYTimes) using cosine similarity. The results show consistent improvements in recall vs. distance computations compared to the baseline, validating the theoretical potential of Steiner points. However, the experiments are preliminary, using a very simple heuristic for Steiner point generation rather than the optimal theoretical construction, and the datasets are relatively small compared to industrial scale.
The paper provides clear pseudocode for the generalized greedy search and detailed descriptions of the theoretical constructions. The experimental setup specifies the use of ParlayANN and specific hyperparameters (degree bound, pruning parameter, beam width). While the code for the specific Steiner-point augmentation is not explicitly linked in the text provided, the reliance on the open-source ParlayANN library and standard $k$-means clustering makes the experimental results highly reproducible.
The primary limitation is the gap between the theoretical upper bound (which requires a complex, specific construction of Steiner points to simulate LSH) and the experimental lower bound (which uses simple $k$-means centroids). The experiments do not demonstrate that the simple heuristic achieves the theoretical $O(n^{1/(c-1)})$ bound, only that it improves over the baseline. Additionally, the lower bound for the high-dimensional case requires $d=n$, leaving the low-dimensional worst-case complexity (for $d \ll n$) partially open, though a weaker bound is provided. The paper also ignores preprocessing time, focusing solely on query time.
This paper has significant implications for the vector database and search engine community. By proving that standard graph-based methods cannot theoretically match LSH in worst-case high-dimensional settings, it redirects research focus toward hybrid methods or Steiner-point-augmented graphs. The introduction of Steiner points as a mechanism to bridge the gap between empirical graph search and theoretical LSH guarantees offers a new design space for next-generation ANN algorithms. This could lead to more robust search systems with provable performance guarantees, which is critical for safety-critical applications. The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory $32\times$ faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains $4.9\times$ faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.
Primary: Massachusetts Institute of Technology (MIT)
All Institutions: Massachusetts Institute of Technology (MIT)
[One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
The paper proposes ParaTTT (PTTT), an algorithm to parallelize Test-Time Training (TTT) layers, which replace linear fast-weight memories with small neural networks trained via online gradient descent. The core innovation is an iterative refinement scheme that approximates the sequential inner-loop gradient trajectory using chunkwise-parallel passes. The method leverages the fact that the exact online gradient descent recurrence is a fixed point of the iteration. The authors provide a theoretical guarantee of geometric convergence for a two-layer variant under specific contraction conditions. The approach decomposes the update into a forward pass for feature trajectories, an exact gated-delta-rule pass for readouts and feature gradients, and a backward pass for gradient propagation. This allows the nonlinear memory to be evaluated in parallel, overcoming the sequential bottleneck of previous TTT implementations like TTT-MLP.
The experiments demonstrate significant speedups: PTTT evaluates the inner-loop trajectory 32x faster than TTT-MLP at 4K context with matched error, and the full 340M model trains 4.9x faster. The models (340M and 1.3B parameters) are evaluated on language modeling perplexity and zero-shot accuracy. The results show that PTTT models match or exceed the performance of Gated DeltaNet and Mamba-2, and remain competitive with same-size Transformers. Notably, the paper claims better recall than linear recurrences, suggesting the nonlinear memory provides a qualitative advantage in information retention. The comparison with Transformers is particularly strong, as TTT layers have historically struggled to match Transformer performance at scale.
The paper provides a GitHub repository link (https://github.com/ntumm120/paraTTT), which is a strong positive for reproducibility. The abstract mentions specific model sizes (340M, 1.3B) and context lengths (4K), which are standard benchmarks. However, the full text provided is truncated (only abstract and headers are visible), so specific hyperparameters, dataset splits, and training schedules cannot be fully verified from the provided text. Assuming the code repository is complete and well-documented, reproducibility should be high.
The theoretical convergence proof is limited to a two-layer variant, while practical applications may involve deeper networks. The claim that "two iterations suffice" in practice needs robust validation across different sequence lengths and model scales. The comparison with Transformers is competitive but not necessarily superior in all metrics, and the "better recall" claim requires careful definition of recall in the context of language modeling. The computational overhead of the iterative refinement, even if parallelized, may still be higher than simple linear recurrences for very short sequences where parallelism gains are minimal.
This work addresses a critical bottleneck in the adoption of nonlinear recurrent architectures. By enabling efficient parallelization of TTT layers, it makes these more expressive models viable for large-scale language modeling. This could lead to new architectures that combine the expressiveness of nonlinear memories with the efficiency of parallel training, potentially challenging the dominance of Transformers in long-context tasks. The technique of iterative refinement for parallelizing sequential operations is also broadly applicable to other sequential learning problems. [One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
We study adversarial linear bandits with strong corruption, in which an adaptive adversary chooses the reward vectors and a strong adversary corrupts each observed reward after seeing the learner's action, subject to a known total corruption budget $C$. Strong corruption breaks the importance-weighted reward estimates used by adversarial bandit algorithms: by corrupting only the rounds in which a rarely sampled action is drawn, the adversary amplifies each unit of corruption by the inverse of its sampling probability. The best known bound for this problem, $\tilde O(\sqrt{d^3T} + dC)$ by Liu et al. (2024), is a factor $\sqrt{d}$ above the corruption-free rate in its leading term, and whether $\tilde O(d\sqrt{T} + dC)$ regret is achievable was left open. We propose CRW-FTRL, which runs follow-the-regularized-leader with a radius-weighted entropy regularizer and estimates rewards with a normalized kernel estimator whose sensitivity to corruption is governed by per-action radii instead of inverse sampling probabilities. The radii are nondecreasing and are chosen jointly with the sampling distribution through a monotone log-determinant barrier problem, so that a negative linear term in the regularizer cancels both the corruption bias and the estimation error of every comparator. For $K$ actions in $\mathbb{R}^d$, CRW-FTRL achieves $\tilde O(\sqrt{dT\log K} + dC)$ regret with high probability against an adaptive adversary, which matches the best known corruption-free rate and the $Ω(dC)$ lower bound up to logarithmic factors. Applied to a discretization of a general compact action set, it yields $\tilde O(d\sqrt{T} + dC)$ regret, which settles the statistical rate of corrupted adversarial linear bandits under strong corruption.
Primary: University of California
All Institutions: University of California
The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
The paper proposes CRW-FTRL, a novel algorithm for adversarial linear bandits with strong corruption. The core innovation lies in replacing standard importance-weighted reward estimates—which are vulnerable to strong adversaries who target rarely sampled actions—with a normalized kernel estimator governed by per-action radii. These radii are determined via a monotone log-determinant barrier problem, coupled with a radius-weighted entropy regularizer. This design effectively decouples the sensitivity of the reward estimator from the inverse sampling probabilities, preventing the adversary from amplifying corruption. The theoretical framework is rigorous, leveraging FTRL with a specific regularizer structure to cancel out both corruption bias and estimation error.
The provided text is a theoretical paper focused on regret bounds. It does not contain empirical experiments, benchmarks, or comparisons with baselines in a practical setting. The contribution is entirely analytical, proving that the proposed algorithm achieves minimax optimal regret rates.
As a purely theoretical paper, reproducibility is tied to the correctness of the proofs rather than code implementation. The algorithm is described in sufficient detail for implementation, but the lack of code or empirical validation limits practical reproducibility assessment.
The primary limitation is the lack of empirical validation. The paper acknowledges that the reduction to general compact action sets via discretization is not computationally efficient, leaving the question of a computationally efficient algorithm for general action sets open. Additionally, the results are high-probability bounds, and the constants hidden in the tilde-O notation may be significant in practice.
This work settles a key open question in the theory of corrupted bandits, providing minimax optimal rates for adversarial linear bandits with strong corruption. It has significant implications for the design of robust online learning algorithms in adversarial environments, such as online advertising and recommendation systems where feedback can be manipulated. The techniques developed (radius-weighted regularizers, kernel estimators with controlled sensitivity) may be applicable to other robust learning problems. The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.
Primary: Google DeepMind
All Institutions: Google DeepMind, Google
EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
The paper proposes EmbeddingGemma 2, a 744M-parameter multimodal embedding model that unifies text, image, audio, and video into a single vector space. The core architectural contribution is the simplification of Per-Layer Embeddings (PLE) by removing the dedicated lookup table and instead projecting the main input embeddings, which reduces memory overhead and improves multimodal performance. The training methodology employs a rigorous three-stage curriculum: (1) text pre-finetuning with distillation to establish a strong text baseline, (2) multimodal pre-finetuning with distillation from a larger teacher model (Gemini Embedding 2) to inherit cross-modal alignment, and (3) multimodal finetuning using contrastive learning, spread-out regularization, and Matryoshka Representation Learning (MRL) to optimize the student model's specific embedding space. The use of a "tail-patch" stage for Quantization-Aware Training (QAT) distillation is a practical engineering innovation that minimizes quality loss in low-precision deployments.
The evaluation is extensive, covering seven major benchmark suites (MTEB, MMEB-v2, MIEB-lite, MAEB, MSEB) across text, code, vision, and audio. The model establishes a new Pareto frontier, outperforming specialist models and general-purpose models 2-12x larger. Ablation studies are thorough, validating the removal of the PLE lookup table, the necessity of co-training the vision encoder, and the impact of vision token budgets on different task types (e.g., document retrieval vs. video). The results demonstrate that native audio processing significantly outperforms ASR-based pipelines, particularly for non-speech sounds and paralinguistic features.
The paper provides high reproducibility. The model weights are released openly. Detailed architectural specifications (dimensionality, layer counts, attention types) are provided. The training recipe, including loss functions, data mixing strategies, and hyperparameter schedules, is described in detail. The release of quantized checkpoints and the specific instructions for task prefixes further aid in practical reproduction and deployment.
The model is still relatively small (744M parameters), which may limit its performance on extremely complex, long-context, or highly specialized scientific tasks compared to frontier LLMs. The paper notes a regression in English-only text performance compared to its predecessor, though multilingual performance improved. The reliance on a specific teacher model (Gemini Embedding 2) for distillation may limit the transferability of the training recipe to other base architectures.
This work significantly lowers the barrier to entry for high-quality multimodal retrieval and embedding tasks. By enabling on-device deployment (191MB-567MB footprint) with minimal quality loss, it facilitates privacy-preserving and offline multimodal applications. The unified space allows for novel applications like interleaved video-audio-text search, which were previously difficult to achieve with separate unimodal models. EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction-data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires 3 times more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security-utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.
Primary: Imperial College London
All Institutions: Imperial College London, Tübingen AI Center, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Fraunhofer HHI
The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
The paper proposes "Heads of Security" (HoS), a defense mechanism against prompt injection that operates by learning per-head attention gates. Unlike previous methods that rely on broad fine-tuning (like SecAlign) or complex architectural changes (like ASIDE), HoS targets the attention mechanism directly to enforce instruction-data separation. The core idea is to learn how specific attention heads should redistribute focus between trusted (system/user) and untrusted (tool/web) tokens. This allows the model to process untrusted content for reasoning purposes without executing any instructions embedded within it. The approach is lightweight and targeted, which is a significant methodological shift from behavior-level alignment to attention-level intervention.
The evaluation is rigorous and comprehensive, testing on two major open-weight models: Llama-3.3-70B-Instruct and GPT-OSS-20B. The paper compares HoS against strong baselines: SecAlign++ (SOTA fine-tuning defense) and ASIDE (architectural defense). Key results include a dramatic reduction in Attack Success Rate (ASR) against adaptive attacks (RL-Hammer drops from 97% to 10% on GPT-OSS-20B) and improved robustness against white-box attacks (Claudini). Crucially, the paper demonstrates that HoS preserves base-model utility on standard benchmarks, whereas competitors degrade in few-shot settings. The inclusion of agentic benchmarks with static attacks further validates the method's practical applicability.
The authors provide a GitHub repository and a Hugging Face space, indicating a commitment to open science. The code availability is a strong plus. However, the specific details of the "autoresearch" used to strengthen the white-box attack against HoS are not fully detailed in the abstract, though likely present in the full text. The use of standard models and benchmarks enhances reproducibility.
The primary limitation is the reliance on open-weight models for evaluation; performance on closed-source models like GPT-4 or Claude is not directly testable in this manner. Additionally, the method requires access to the model's internal attention heads, which limits its applicability to API-only models. The paper also notes that ASIDE struggles with tool use, but it is unclear if HoS fully resolves all edge cases in complex agentic workflows beyond the tested benchmarks.
This work has high impact on the field of LLM security. As LLMs are increasingly deployed in agentic roles with access to external data, prompt injection is a critical vulnerability. HoS offers a scalable, efficient, and effective defense that does not sacrifice model utility. It provides a new paradigm for securing LLMs by manipulating internal representations (attention) rather than just output behavior. This could lead to more secure and trustworthy AI systems in production environments. The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.
Primary: Massachusetts Institute of Technology
All Institutions: Massachusetts Institute of Technology, Carnegie Mellon University, University of Maryland
The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
The paper introduces a novel cross-modal pre-training framework that aligns egocentric video with surface electromyography (EMG) signals. The core methodological contribution is the use of EMG as a "privileged supervisory signal" in a CLIP-style contrastive objective. By jointly training a video encoder and an EMG encoder, the authors create a shared embedding space where visual representations are shaped by the underlying motor intent, which is then discarded at inference time. This approach is technically sound, leveraging the dense, occlusion-free nature of EMG to provide a stronger inductive bias for action-centric vision tasks than standard self-supervised video learning. The inclusion of a VQ-VAE to learn discrete "motor codes" for conditioning video diffusion models is a creative extension, though the reliance on the Video2LoRA framework limits the novelty of the generative component itself.
The experimental evaluation is extensive and rigorous. The authors demonstrate consistent improvements over strong baselines (SimCLR, MoCo, BYOL, and ImageNet/MAE initialization) across four distinct task families: action recognition (EPIC-Kitchens, SSv2, Ego-Exo4D), 3D hand reconstruction (ARCTIC, H2O, HOT3D), 3D contact reconstruction, and video generation. The gains are particularly notable in the frozen-backbone setting for action recognition, suggesting the pre-training captures robust, transferable features. The comparison against RGB-only self-supervised methods trained on the same data effectively isolates the contribution of the EMG signal. The qualitative results for video diffusion show clear improvements in fine-grained hand-object interaction fidelity compared to language or video-history conditioning.
The paper provides high reproducibility. It details the specific hardware (MindRove Armband, Meta Aria glasses), data collection protocols, preprocessing steps (filtering, normalization), and model architectures. The release of the 100-hour dataset is a significant asset. However, the specific implementation of the "Video2LoRA" and "HyperNetwork" components relies on external citations, and the code for the VQ-VAE motor code discovery is described but not fully detailed in the main text, potentially requiring access to the supplementary materials or code repository (not explicitly linked in the provided text, though a project page is mentioned) for full replication.
The primary limitation is the scale of the dataset (100 hours) compared to massive video datasets like Kinetics or Ego4D. The EMG hardware used is consumer-grade (8 channels), which may limit the granularity of muscle activity captured compared to research-grade high-density EMG. The video diffusion experiments are limited to a specific backbone (CogVideoX-Fun) and a lightweight LoRA approach, which may not generalize to all state-of-the-art video generation models. Additionally, the dataset is focused on kitchen/household manipulation, which may limit generalizability to other domains.
This work has significant potential impact on embodied AI, robotics, and VR/AR. By providing a dataset and method that links visual perception with motor intent, it offers a pathway to more intuitive human-robot interaction and better understanding of human actions from video. The motor-code conditioning for video generation could enable new tools for content creation and simulation. The dataset itself becomes a valuable resource for the community, addressing a gap in paired video-EMG data. The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally. Based on this insight, we introduce SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. SpecSlot expands each spatial anchor into multiple slots, each assigned a disjoint band of DCT coefficients; this separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds a local correction to the spectral prediction, while the generative state, target, and loss remain in pixel space. On ImageNet-256, SpecSlot achieves a comparable final FID to a FLOPs-matched JiT control while converging faster. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within only 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image generation, SpecSlot surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.
Primary: Peking University
All Institutions: Peking University, Nanyang Technological University, SenseTime
SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
The paper proposes SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from global attention sequence length. The core innovation is the "Role-Coupled Tokenization" where each spatial anchor is expanded into multiple slots, each assigned a disjoint band of DCT coefficients. Only the lowest-frequency slot participates in global attention, while high-frequency slots interact locally. This is a clever architectural modification that leverages the spectral properties of natural images (low-frequency structure requires global context, high-frequency detail is local). The method remains in pixel space, avoiding the artifacts often associated with VAE-based latent diffusion, while mitigating the quadratic cost of attention in pixel-space models. The inclusion of a spatial head for local correction is a sensible addition to handle residual errors.
The experiments are conducted on ImageNet-256 and ImageNet-512, as well as text-to-image generation benchmarks (DPG-Bench). The results show that SpecSlot achieves comparable FID to FLOPs-matched JiT controls on ImageNet-256 with faster convergence. On ImageNet-512, it improves FID by 2.84 over the control with fewer FLOPs and comes within 0.69 FID of a larger JiT model at significantly lower cost. The text-to-image results show improvements over JiT-B/8. The comparisons are against strong baselines (JiT), and the efficiency gains are substantial.
The paper provides links to the project page, GitHub repository, and Hugging Face model weights. The methodology is described in sufficient detail in the provided text (though the full technical details are in the sections referenced but not fully expanded in the prompt's text snippet, the abstract and summary provide the core mechanism). The availability of code and weights significantly enhances reproducibility.
The method relies on DCT coefficient bands, which may not be optimal for all types of content or modalities. The local interaction of high-frequency slots might limit the model's ability to capture long-range high-frequency correlations, although this is argued to be less critical. The computational overhead of managing multiple slots per anchor needs careful engineering to ensure the efficiency gains are realized in practice.
This work offers a viable path for pixel-space generative models to scale to higher resolutions without the prohibitive cost of full global attention. It could influence the design of future pixel-space Transformers and potentially other modalities where spectral decomposition is applicable. The efficiency gains make high-quality pixel-space generation more accessible. SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Primary: Tencent Youtu Lab
All Institutions: Tencent Youtu Lab
Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
The paper proposes Youtu-Parsing-Omni, a unified architecture for parsing documents, images, audio, and video into structured records. The core methodological contribution is the "Youtu-Omni-Encoder," which initializes from a pre-trained language model and uses shared weights to encode all modalities in a unified space-time coordinate system. A key architectural innovation is the inclusion of fusion layers that allow video frames to attend to concurrent audio, integrating cross-modal fusion directly into the perception layer. The training methodology introduces "OmniSchema-Aware On-Policy Distillation" (OSAD), a self-evolving post-training procedure where the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself. This allows the parser and its supervision to co-evolve, unifying the outputs of ten different parsing tasks under a single schema.
The paper reports state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.). The results highlight a significant advantage in audio and video parsing compared to general-purpose systems. It also achieves the best overall chart-parsing score (95.02) and high exact chemical-fingerprint agreement. The model matches dedicated optical-music models in character-level transcription of typeset sheet music. The evaluation covers a wide range of modalities and tasks, demonstrating the versatility of the unified approach.
The paper states that code and model weights are publicly available, which is a strong positive for reproducibility. However, the provided text does not include specific URLs or detailed hyperparameter settings, making it difficult to verify the exact implementation details without accessing the external repository. The description of the OSAD training procedure is high-level, and the specific mechanics of the "reference-conditioned copy" and "token-role-based objective" would require the code to fully understand and replicate.
The paper relies heavily on the performance of the pre-trained language model initialization, which may limit its applicability to domains where text is not the primary signal. The "self-evolving" nature of the OSAD training could be computationally expensive and potentially unstable, though this is not explicitly discussed. The evaluation is primarily on benchmark datasets, and real-world robustness to noisy or unstructured data is not thoroughly tested. The lack of specific URLs in the text prevents immediate verification of the claimed open-source availability.
This work has significant potential to impact the field of multimodal AI by providing a unified perception layer for agents that need to read, locate, and cite evidence across multiple modalities. The ability to parse documents, images, audio, and video into structured records using a single encoder and schema could simplify the development of complex AI agents. The focus on spatial and temporal grounding is particularly relevant for applications in robotics, autonomous systems, and advanced document analysis. Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni introduces two major innovations. First, we propose a unified cross-paradigm on-policy distillation (OPD) framework that successfully transfers the visual understanding ability of autoregressive VLMs and visual generation ability of continuous flow models, to a unified DLLM. Second, NLD-Uni incorporates a novel progressive self-refining diffusion process that gradually corrupts each token across multiple steps during training, as opposed to adopting the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality. Through extensive experiments, we show that NLD-Uni achieves state-of-the-art performance among unified DLLMs on diverse tasks including visual question-answering, multi-modal reasoning, object grounding, text-to-image generation, and image editing.
Primary: NVIDIA
All Institutions: NVIDIA
The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
The paper proposes NLD-Uni, a unified discrete diffusion large language model (DLLM) for visual tasks. The core methodological contribution is a "unified cross-paradigm on-policy distillation" framework. This approach attempts to transfer capabilities from two distinct source models—autoregressive VLMs (for understanding) and continuous flow models (for generation)—into a single discrete diffusion target. The second major innovation is a "progressive self-refining diffusion process." Unlike standard masked diffusion, which corrupts tokens to a uniform mask, this method gradually corrupts tokens across multiple steps during training. At inference, this allows for intermediate token commitments, theoretically enabling self-refinement. The methodology is complex and addresses a significant gap in unifying generation and understanding in discrete spaces, though the reliance on distillation from potentially proprietary or large teacher models adds complexity.
The paper claims state-of-the-art performance among unified DLLMs on VQA, multi-modal reasoning, object grounding, text-to-image generation, and image editing. However, the provided text is a placeholder ("sections/intro", "sections/experiments") and does not contain actual data, tables, or specific benchmark scores. Without concrete evidence of the "state-of-the-art" claims, the experimental evaluation cannot be rigorously verified. The scope of tasks (both understanding and generation) is broad, which is a strong point if the results hold up, but the lack of visible data in the prompt prevents a high confidence in the empirical rigor.
The paper describes specific training objectives (on-policy distillation, progressive corruption). However, without access to the full code, hyperparameters, and the specific teacher models used for distillation, reproducibility is moderate to low. The "Nemotron" branding suggests an internal NVIDIA model, which may not be fully open-source, limiting external reproducibility.
The primary limitation is the opacity of the experimental results in the provided text. Additionally, unified models often suffer from a "seesaw" effect where improving generation hurts understanding or vice versa; the paper claims to overcome this via distillation, but the trade-offs are not detailed in the abstract. The reliance on discrete tokens for image generation may still lag behind continuous diffusion/flow models in terms of raw fidelity, even with self-refinement.
If successful, this work could significantly advance the field of unified multimodal models by demonstrating that discrete diffusion can effectively handle both high-level reasoning and low-level pixel generation. This could lead to more efficient, single-model architectures for multimodal AI, reducing the need for separate VLMs and image generators. The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
Real-world robotic manipulation requires reasoning about cluttered scenes, partially observable or hidden objects, and semantically appropriate actions, in addition to low-level execution within those scenes. However, existing simulation benchmarks primarily test the latter and remain dominated by comparatively clean environments with limited clutter and narrowly scoped task structure, leaving many forms of commonsense reasoning untested. We address this by introducing a taxonomy of reasoning processes for robotic manipulation grounded in developmental cognitive science. We operationalize this taxonomy through a new benchmark of robot tasks, MolmoSpaces-CommonSense, that requires reasoning about occlusion, containment, support relations, semantically correct grasp points, spatial affordances, and hidden objects. These evaluations require policies not only to recognize relevant scene structure but also to execute correct actions in them. We evaluate several zero-shot generalist manipulation policies. Across model types and architectures, we observe consistent failures in commonsense reasoning: success rates in cluttered scenes drop by over 50% as distractors increase, policies act on incorrect objects in cluttered settings 30% of the time, and affordance-sensitive grasping remains unreliable even with highly specific language prompts, with correct object part selection happening only about 50% of the time. Further, VLM-based success estimation struggles to reliably evaluate rollouts in these settings, suggesting that the challenge extends beyond control to the broader perception-reasoning-action stack. Our results suggest that existing benchmarks may substantially overestimate zero-shot real-world readiness. Code and policy leaderboard link at https://github.com/allenai/molmospaces.
Primary: Allen Institute for AI
All Institutions: Allen Institute for AI
The paper introduces a new benchmark for evaluating commonsense reasoning in robotic manipulation policies. It provides a rigorous evaluation framework that highlights significant gaps in current zero-shot generalist policies, suggesting that existing benchmarks may overestimate real-world readiness.
The paper introduces a taxonomy of commonsense reasoning processes for robotic manipulation, grounded in developmental cognitive science. It operationalizes this taxonomy into a benchmark, MolmoSpaces-CommonSense, which tests policies on tasks requiring reasoning about occlusion, containment, support relations, and spatial affordances. The methodology shifts focus from low-level execution in clean environments to high-level reasoning in cluttered, partially observable scenes. The approach is sound, leveraging existing simulation environments but augmenting them with specific task structures that isolate and test commonsense capabilities.
The experiments evaluate several zero-shot generalist manipulation policies. The results reveal consistent failures in commonsense reasoning, with success rates dropping by over 50% as distractors increase. Policies frequently act on incorrect objects in cluttered settings (30% error rate), and affordance-sensitive grasping remains unreliable. The evaluation also highlights limitations in VLM-based success estimation, suggesting that the challenge extends to the perception-reasoning-action stack. The experiments are rigorous in demonstrating the gap between current capabilities and the requirements of real-world manipulation.
The paper provides a link to the code and policy leaderboard on GitHub, which enhances reproducibility. The use of standard simulation environments and clear task definitions should allow other researchers to replicate the results. However, the specific configurations of the policies and the exact setup of the cluttered scenes would need to be detailed in the code repository for full reproducibility.
The benchmark is limited to simulation, which may not fully capture the complexities of real-world robotic manipulation. The taxonomy, while grounded in cognitive science, may not cover all forms of commonsense reasoning. Additionally, the evaluation is focused on zero-shot generalist policies, and the findings may not generalize to specialized or fine-tuned policies.
The paper has significant implications for the field of robotic manipulation, highlighting the need for benchmarks that test high-level reasoning in addition to low-level execution. The findings suggest that existing benchmarks may overestimate the readiness of current policies for real-world deployment. This work could drive future research towards developing more robust and commonsense-aware manipulation policies. The paper introduces a new benchmark for evaluating commonsense reasoning in robotic manipulation policies. It provides a rigorous evaluation framework that highlights significant gaps in current zero-shot generalist policies, suggesting that existing benchmarks may overestimate real-world readiness.
Robot agents built on vision-language models depend on both the planner and its harness: the tools, feedback, memory, and recovery logic around it. Can improvements to a large teacher's harness be carried into a compact agent with lower inference compute cost? We present TinyAstral, a two-stage approach. We first freeze an open-weight teacher and improve its harness through recursive self-improvement (RSI): frontier LLM agents read the teacher's trajectories and propose harness edits, and each edit is accepted only if it passes a paired statistical test. We then freeze the final harness, collect teacher trajectories with reasoning and failure recovery, and distill a small student that replaces the teacher in the same harness. On RoboTwin, harness evolution raises the teacher's success from 33.4% to 63.6%; the final 4B student reaches 62.8%. Across four harness versions, distilled 4B students score close to their corresponding teachers, and later versions yield stronger students, showing that harness evolution gains carry into compact models. The harness gains also transfer to other frozen planners without retraining. On identical episodes, the final 4B student outperforms open 4B agents, the open 27B harness-RSI model SenseNova-RoboRSI, and our own teacher model placed in three competitors' harnesses. It stays close to its teacher on all three training benchmarks, and on three benchmarks never used for RSI or training it scores close to a 552B planner running the same harness. On the measured deployment episodes, it uses about one fifth of its teacher's accelerator time.
Primary: Unknown
All Institutions: Unknown
The paper introduces a novel two-stage distillation method that evolves the agent's "harness" via recursive self-improvement before transferring capabilities to a compact student model, achieving near-teacher performance at a fraction of the computational cost. While the results on RoboTwin are promising and the methodology offers a fresh perspective on agent optimization, the limited scope of experimental validation and the opacity of the harness implementation prevent it from reaching the threshold of a field-defining contribution.
The paper proposes "TinyAstral," a two-stage framework for transferring capabilities from large Vision-Language Models (VLMs) to smaller agents via "harness" evolution. The core novelty lies in decoupling the model weights from the "harness" (tools, memory, recovery logic) and applying Recursive Self-Improvement (RSI) to the harness using frontier LLMs to propose edits, validated by statistical tests. The second stage involves distilling a small student model (4B) that operates within this evolved harness. The methodology is conceptually sound, leveraging the idea that structural improvements in the agent's environment can be more transferable and cost-effective than weight-level fine-tuning for complex manipulation tasks.
The experiments are conducted on the RoboTwin benchmark. The reported results show a significant improvement in the teacher model's success rate (33.4% to 63.6%) due to harness evolution. The distilled 4B student achieves 62.8% success, closely matching the teacher. The paper claims the student outperforms other open 4B agents and a 27B model (SenseNova-RoboRSI). However, the evaluation is limited to a single benchmark suite (RoboTwin) and specific tasks. The comparison against a 552B planner on unseen benchmarks is a strong claim, but the lack of broader generalization testing across different robot embodiments or environments limits the robustness of the findings.
The paper mentions specific models (4B student, 27B SenseNova-RoboRSI) and benchmarks (RoboTwin), which aids reproducibility. However, the "harness" components (tools, feedback loops, memory structures) are complex and likely proprietary or difficult to replicate without detailed code. The statistical tests used for accepting harness edits are mentioned but not fully detailed in the abstract. Without access to the specific harness code and the RSI prompts, full reproduction is challenging.
The primary limitation is the reliance on a specific benchmark (RoboTwin) and potentially specific robot hardware. The "harness" is a black box in the summary, and its complexity may make it hard to generalize to other domains without significant re-engineering. Additionally, the RSI process relies on frontier LLMs, which may introduce biases or costs that are not fully accounted for in the "deployment" cost analysis. The gap between the teacher and student is small, but the absolute performance (62.8%) may still be insufficient for real-world deployment in unstructured environments.
This work contributes to the field of efficient robotics by demonstrating that agent-level improvements (harness) can be distilled into smaller models, reducing inference costs. It offers a pathway for deploying VLM-based agents on edge devices with limited compute. The concept of "harness evolution" could be applied to other AI agent domains beyond robotics, such as software engineering or web navigation, where tool use and recovery logic are critical. The paper introduces a novel two-stage distillation method that evolves the agent's "harness" via recursive self-improvement before transferring capabilities to a compact student model, achieving near-teacher performance at a fraction of the computational cost. While the results on RoboTwin are promising and the methodology offers a fresh perspective on agent optimization, the limited scope of experimental validation and the opacity of the harness implementation prevent it from reaching the threshold of a field-defining contribution.
Robot policy learning is dominated by zeroth-order methods such as PPO, which treat the simulator as a black box and pay for it with enormous sample counts. First-order policy learning differentiates through the dynamics, but on contact-rich whole-body control it has rarely worked: its gradients explode. A recent world-model method showed that derivative information can make humanoid motion tracking far more efficient, using surrogate gradients of a learned model. We show that exact gradients of the simulator itself do better, once two obstacles are removed. Exact gradients contain rare unbounded samples, because contact dynamics is not differentiable everywhere and its derivative diverges near some events; and they are expensive, because the simulator's adjoint has to replay the contact solve. $\nabla$Mimic bounds these samples with median-relative per-sample gradient clipping at every step of the backward pass, and reuses each costly simulated window by multistep regression onto gradient-derived action targets, which bounds every action target independently of the gradient scale. NablaEngine, a differentiable GPU simulation engine for the Unitree G1, removes the cost: it computes MuJoCo's full contact dynamics with fused per-world kernels and the adjoint of a whole control step as a single megakernel, with analytic gradients that match float64 finite differences, $11.5\times$ faster than the available MuJoCo Warp adjoint. On whole-body tracking of all 40 LAFAN1 clips with one policy on one GPU, $\nabla$Mimic reaches the score reported by the state-of-the-art world-model method, 0.82, in 20 minutes instead of 1.2 h, and 0.874 in 1.8 h. SHAC, the standard first-order baseline, stops on exploding gradients within minutes. We release NablaEngine and $\nabla$Mimic as open-source infrastructure for first-order humanoid policy learning.
Primary: unknown
All Institutions: unknown
The paper presents a highly effective and efficient method for first-order policy learning in humanoid control, combining novel gradient stabilization techniques with a high-performance differentiable simulation engine to achieve state-of-the-art results in a fraction of the time.
The paper proposes a rigorous solution to the long-standing problem of gradient explosion in differentiable simulation for contact-rich robotics. The core methodological contribution is the combination of per-sample median-relative gradient clipping during the backward pass and multistep regression onto gradient-derived action targets. This approach effectively bounds the influence of rare, unbounded gradients caused by non-differentiable contact events without introducing the systematic bias of a learned world model. The engineering contribution, NablaEngine, is equally significant, implementing a hand-derived adjoint for MuJoCo dynamics as a single GPU megakernel, achieving exact gradients at a cost lower than an additional forward step.
The experiments are highly compelling. The paper demonstrates that the proposed method reaches the state-of-the-art score (0.82) on the LAFAN1 benchmark in 20 minutes, compared to 1.2 hours for the best world-model baseline (SuperMimic) and hours for PPO. The ablation studies convincingly show that both the clipping and the multistep regression are necessary for stability and performance. The comparison against SHAC clearly illustrates the failure mode of previous first-order methods. The results are consistent across seeds and provide a clear speedup advantage.
The authors state that they release NablaEngine and the method as open-source infrastructure. The paper provides detailed hyperparameters, reward structures, and engine specifications. The use of a specific robot (Unitree G1) and a standard benchmark (LAFAN1) enhances reproducibility. However, the lack of a public URL in the provided text limits immediate verification, though the promise of open-source release is a strong positive.
The method is currently specialized for the Unitree G1 and flat ground. Extending it to other robots requires re-deriving the adjoint kernels. The paper acknowledges that the gradient holds discrete contact decisions fixed, which is an approximation of the true dynamics at contact transitions. There is no hardware deployment in this paper, limiting the assessment of sim-to-real transfer.
This work has the potential to significantly accelerate the development of humanoid foundation models by reducing the training cost of motion tracking policies from hours to minutes. By making first-order methods viable for contact-rich tasks, it opens the door to more efficient and precise policy learning in robotics, potentially enabling the training of policies on much larger and more diverse motion datasets. The paper presents a highly effective and efficient method for first-order policy learning in humanoid control, combining novel gradient stabilization techniques with a high-performance differentiable simulation engine to achieve state-of-the-art results in a fraction of the time.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
Future planetary rovers will traverse steep, deformable terrain where slip, sinkage, and terrain deformation can threaten mobility. Although simulation can capture some wheel-soil effects, unmodeled deformable-terrain phenomena create sim-to-real gaps requiring deployment-time adaptation. Yet, rover operations leave little room for extensive trial-and-error. We introduce TERRA (Targeted Exploration for Reward-Relevant Adaptation), an online adaptation framework that efficiently seeks deployment data exposing task-relevant model errors and corrects them. TERRA adapts a simulation-trained world model with a reward-weighted prediction loss and trains a loss predictor to estimate where future performance-relevant errors are likely to occur, using this signal to guide exploration during MPC. In simulation, TERRA reaches approximately 90% slope-climbing success within five trials under severe out-of-distribution pretraining conditions, outperforming exploration baselines in reward, completion time, and energy to first success. On the ERNEST rover, TERRA recovers coordinated active-suspension locomotion on a deformable Mars-simulant slope within three to five real-world trials. Project website: https://terra-explore.github.io
Primary: Jet Propulsion Laboratory, California Institute of Technology
All Institutions: Jet Propulsion Laboratory, California Institute of Technology, University of Texas at Austin
TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
The paper introduces TERRA, a framework for online adaptation of simulation-trained world models for planetary rovers. The core methodological contributions are twofold: (1) a reward-weighted prediction loss that prioritizes model errors in observation dimensions that have high sensitivity to the task reward, and (2) a learned "loss predictor" that forecasts where these reward-relevant errors are likely to occur in candidate action sequences, thereby guiding exploration within a Model Predictive Control (MPC) loop. The use of Low-Rank Adaptation (LoRA) for efficient onboard updating of the dynamics model is a practical engineering choice. The approach is logically sound, addressing the specific bottleneck of sim-to-real gaps in deformable terrain where standard prediction-error-based exploration is inefficient.
The evaluation is strong, featuring both extensive simulation studies (100 seeds, multiple OOD pretraining conditions) and real-world validation on the ERNEST rover. The simulation results show significant improvements over baselines like RND and Plan2Explore in terms of success rate, completion time, and energy efficiency. The real-world results are particularly compelling, demonstrating the recovery of coordinated active-suspension locomotion within 3-5 trials on a Mars-simulant slope, a task that is critical for future missions. The ablation studies effectively isolate the contributions of the reward-weighted loss and the targeted exploration.
The paper provides detailed descriptions of the simulation environment, pretraining conditions, and hyperparameters. However, as is common in robotics papers involving proprietary hardware (ERNEST) and specific simulators (DARTS), full reproducibility by external groups may be limited. The project website is provided, which likely contains code or further details.
The method relies on the availability of a task reward during deployment, which may not always be easily defined or observable in complex real-world scenarios. The choice of loss-bin thresholds for the loss predictor is a hyperparameter that requires tuning. The evaluation is limited to a single rover platform (ERNEST) and a specific type of terrain (deformable slopes), so generalizability to other platforms or terrain types is not fully established.
This work has high relevance to the field of planetary robotics and autonomous systems operating in unstructured environments. The technique of reward-weighted model adaptation and targeted exploration is applicable beyond rovers to any mobile robot facing sim-to-real gaps in contact-rich or deformable environments. It provides a practical pathway for deploying learned policies in high-stakes, low-trial-count scenarios. TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
Discretizing continuous actions into discrete skills using methods like VQ-VAE has emerged as a powerful paradigm for robotic manipulation. However, quantization errors yield a suboptimal distribution for the skill prior, degrading its performance. While reinforcement learning offers a path for refinement, its direct application faces unstable encoder updates and a granularity dilemma in importance sampling. To address these challenges, we introduce Cascaded Skills Optimization (CSO), a two-stage post-training framework. First, Rejection-Sampling Supervised Fine-tuning aligns the model's observation-to-skill mapping with the distribution of successful online trajectories via supervised fine-tuning. Second, with the observation encoder frozen, Skills Policy Optimization computes an independent, clipped importance ratio for each complete skill, enabling stable and efficient updates. With a lightweight QueST-style policy, CSO raises overall LIBERO success from 80.1% to 97.5%, improves performance on RoboTwin2.0, and achieves 61.0% success across five physical single-arm tasks. Additional comparisons with FAST, discrete BEAST, and fixed-buffer IQL further validate the effectiveness of our two-stage post-training strategy.
Primary: Unknown
All Institutions: Unknown
[One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
The paper proposes CSO (Cascaded Skills Optimization), a two-stage post-training framework for refining skill-based robotic policies. The methodology is logically sound and addresses specific pain points in applying RL to discrete skill policies. Stage 1 (R-SFT) uses rejection sampling to collect successful trajectories and fine-tunes the observation encoder, which is then frozen. This is a pragmatic solution to the instability of updating visual encoders with sparse rewards. Stage 2 (SPO) introduces a skill-level importance sampling ratio, arguing that token-level (too fine) and episode-level (too coarse) granularities are suboptimal for robotic skills. The theoretical justification for the "granularity dilemma" is the core intellectual contribution. The approach is essentially a structured application of PPO variants (specifically GSPO/GRPO concepts) adapted to the hierarchical structure of skill-based policies. While the adaptation is clever, the underlying RL machinery is not new.
The experimental evaluation is rigorous and comprehensive. The authors test on standard simulation benchmarks (LIBERO, RoboTwin2.0, MetaWorld) and, crucially, on physical robots (Cobot Agilex ALOHA). The results show significant improvements over the imitation learning baseline (80.1% to 97.5% on LIBERO) and outperform strong baselines like RIPT and FAST. The ablation studies effectively isolate the contributions of R-SFT and SPO, and the granularity analysis convincingly demonstrates the superiority of skill-level optimization over token- and episode-level alternatives. The physical robot experiments, while limited to five tasks, provide valuable validation of the method's real-world applicability.
The paper provides detailed implementation specifics, including hyperparameters for both stages, architecture details for the lightweight and VLM-based variants, and data collection protocols. The use of standard benchmarks and clear descriptions of the training process enhances reproducibility. However, the lack of a public code repository link in the provided text is a minor drawback, though the detailed appendices mitigate this.
The method requires a sufficient number of successful trajectories to bootstrap the R-SFT stage, which may be difficult for very hard tasks without a strong initial policy. The frozen encoder in Stage 2 limits the model's ability to adapt its perception to new failure modes discovered during RL. The physical experiments are limited to single-arm tasks, and the method's scalability to more complex, multi-arm, or dexterous manipulation scenarios is not fully explored.
This paper contributes to the growing field of post-training for robotic policies, offering a practical and effective recipe for improving skill-based agents. The insight regarding optimization granularity is likely to influence future work in hierarchical RL and skill-based control. The framework is general enough to be applied to other skill-based architectures, making it a useful tool for the robotics community. [One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, University of Maryland, Toyota Research Institute
TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
The paper proposes TAPESTRY, a method to repurpose pretrained video diffusion transformers (specifically CogVideoX-2B) for robot control by treating robot actions as single-patch tokens in the video latent space. The core innovation is the "action-as-VAE-patch" tokenization, which uses a lightweight linear projection and LayerNorm to map action vectors into the same distribution as video VAE latents, allowing them to pass through the pretrained patchifying convolution without architectural modification. The training procedure involves three joint diffusion objectives: video-only, joint video+action, and action-only. A significant technical contribution is the analysis of high-frequency noise in multi-step denoising for action chunks, demonstrating that single-step DDIM inference from pure noise is superior for this specific repurposed backbone due to frequency-space mismatches between video and action signals.
The evaluation is rigorous, covering simulation (LIBERO benchmark with a learning-from-video protocol) and real-world bimanual manipulation. The paper demonstrates strong generalization to novel objects and motion primitives. The inference efficiency claims are well-supported, showing a 44x speedup over optimized DreamZero on consumer hardware (RTX 5090). The ablations effectively isolate the contribution of the video pretraining and the specific training objectives.
The paper provides detailed implementation specifics, including the exact parameter count of the action head (1280 parameters), the specific backbone used (CogVideoX-2B), and the training data mixture. The derivation of the single-step inference collapse is provided in the appendix. However, reliance on specific proprietary or large-scale datasets (like DROID, AgiBot-World) may limit immediate reproducibility for smaller labs, though the architectural changes are minimal and easy to implement.
The method is currently limited to a 20-DoF bimanual action space, and the single-step inference relies on the zero-terminal-SNR v-prediction schedule, which may not transfer to other backbones like rectified-flow models (Wan). The deterministic nature of single-step inference may be a limitation for tasks requiring multimodal action distributions.
This work significantly lowers the barrier to entry for using large video models in robotics by eliminating the need for expensive video generation at test time. The insight that action tokens can be treated as video patches with minimal projection is a generalizable architectural pattern that could be applied to other multimodal control tasks. The finding regarding high-frequency noise in repurposed diffusion models is a valuable diagnostic for the broader community working on world action models. TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley
The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.
The paper proposes a comprehensive sim-to-real pipeline for in-hand object pose reaching using a 16-DoF LEAP hand. The core methodological contributions are: (1) A training recipe that leverages a large-scale generated grasp dataset (via LightningGrasp and DexterityGen) to define diverse goals and resets, avoiding the need for human demonstrations. (2) An adaptive friction curriculum that starts with high friction to stabilize learning and gradually reduces it to match reality, which is shown to be critical for convergence. (3) A modular safety approach using a learned residual safety filter (adapted from ResSafe) that operates on top of a frozen task policy. This filter uses a tunable weight $w$ to balance performance and safety, allowing for a "dial" to adjust conservativeness at deployment without retraining the task policy. The system identification is rigorous, using a voltage-driven DC motor model (BAM) and precise latency calibration to bridge the sim-to-real gap.
The experimental evaluation is extensive and rigorous, particularly for a preprint. The authors conduct 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and 6 wrist orientations. The state-based policy achieves a 95.2% success rate, and the safety filter reduces drops by 75% while maintaining a 91.5% success rate. The paper also evaluates robustness to out-of-distribution goals, fast arm motion, and physical perturbations. Comparisons with Hamilton-Jacobi (HJ) safety filters demonstrate the superiority of the learned residual approach in terms of computational efficiency and performance trade-offs. The inclusion of a depth-camera-only policy that transfers zero-shot is a strong validation of the visual sim-to-real gap mitigation.
The paper provides high reproducibility. It details the hardware setup, system identification process, simulation parameters, and training hyperparameters. The authors commit to releasing the full recipe, code, and trained checkpoints. The use of standard tools like MuJoCo Warp and PPO, along with clear descriptions of the reward functions and curricula, makes it feasible for other researchers to replicate the results. The specific details on latency calibration and actuator modeling are particularly valuable for the robotics community.
The policies are trained per object, which limits generalization to new objects without retraining. The depth policies still show a visual sim-to-real gap on certain objects (knife, wineglass). The safety filter is also trained per object and per nominal policy. The downstream tasks (e.g., screwing in a lightbulb) rely on hard-coded arm motions, limiting the autonomy of the system. The evaluation is limited to a specific hand (LEAP) and arm (UR5), so generalizability to other hardware is not tested.
This work significantly advances the state of the art in dexterous manipulation by addressing the critical issue of safety and reliability in high-dimensional contact-rich systems. The modular safety filter approach is likely to be adopted in other robotic domains where safety is paramount. The comprehensive evaluation framework for measuring performance, reliability, and safety provides a valuable benchmark for the field. The ability to operate with only a wrist-mounted camera expands the practical applicability of such systems beyond controlled lab environments. The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.