Last 7 Days (October 28 – November 03, 2026)
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
Accurate forecasting with uncertainty quantification is critical in a variety of scientific applications ranging from epidemiology and economics to energy systems and earth sciences. Yet most forecasting models still need to be developed and trained separately for each domain-specific dataset. Time series foundation models (TSFMs) aim to remove this limitation and enable the application of the same model across domains through large-scale pretraining. However, most existing TSFMs treat each time series independently and therefore cannot model interactions among time series and external drivers. Here, we introduce Chronos-2, a foundation model for zero-shot forecasting that models the interactions between multivariate time series and heterogeneous covariates, which commonly arise in scientific applications. The model infers relationships among time series directly from historical context and generates probabilistic forecasts without task-specific treatment. Across various benchmarks, Chronos-2 achieves state-of-the-art accuracy on average by improving skill scores by 5.2 percentage points relative to the strongest baselines and increasing win rates by 10 percentage points. Chronos-2 is also computationally efficient, generating forecasts for 100 time series in under one second on a single mid-range GPU, 50x and 1000x faster end-to-end than the best task-specific point and probabilistic forecasters, respectively. In addition, we demonstrate its effectiveness in diverse scientific settings -- including infectious-disease forecasting, macroeconomic modeling, carbon-flux prediction, and runoff simulation -- where the model matches or surpasses specialized methods developed for each domain. These results show that a single pretrained model can capture structure shared across temporal systems.
Primary: Amazon
All Institutions: Amazon, AWS
Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
The paper introduces Chronos-2, a 120M parameter encoder-only Transformer foundation model for time series forecasting. The core methodological contribution is the "group attention" mechanism, which allows the model to infer dependencies among multivariate time series and heterogeneous covariates directly from historical context (in-context learning) without task-specific training. The architecture alternates between temporal attention (within a series) and group attention (across related series). A significant portion of the methodology focuses on the synthetic data generation pipeline, which creates diverse multivariate dynamics by coupling univariate generators (Gaussian Processes, AR, ETS) through linear (VAR, mixing matrices) and non-linear (MLP, RNN) transformations. This approach addresses the scarcity of large-scale multivariate time series data for pretraining. The model uses a quantile regression head to provide probabilistic forecasts, supporting uncertainty quantification.
The evaluation is extensive and rigorous, covering both standard benchmarks and domain-specific case studies. On the FEV benchmark (100 tasks), Chronos-2 achieves state-of-the-art results, outperforming strong baselines like TiRex, TimesFM, and Moirai. It demonstrates significant gains in win rate and skill score over univariate inference, validating the group attention mechanism. The paper includes four detailed case studies: epidemiology (ECDC RespiCast), macroeconomics (German CPI), earth science (FLUXNET carbon flux), and hydrology (CAMELS-DE rainfall-runoff). In each domain, the model matches or surpasses specialized, task-specific models (e.g., HBV, LSTM ensembles, XGBoost) in zero-shot settings. The inclusion of hierarchical forecasting experiments further demonstrates the model's ability to capture cross-level dependencies without explicit hierarchy information. The computational efficiency is also highlighted, with inference times under 1 second for 100 series on a mid-range GPU.
The paper provides detailed architectural specifications, training hyperparameters, and data generation procedures. The model size (120M parameters) and training setup (4x A100 GPUs) are clearly stated. The synthetic data generation process is described in detail, allowing for potential replication of the pretraining corpus. However, the specific code repository or model weights are not explicitly linked in the provided text (though likely available given the Amazon affiliation and prior Chronos releases). The evaluation protocols for the case studies are well-defined, referencing standard domain metrics (WIS, NSE, etc.).
The model is limited to 120M parameters, which is smaller than some competing foundation models (e.g., Moirai, TimesFM), potentially limiting its capacity for extremely complex dynamics. The reliance on synthetic data for multivariate pretraining may introduce a domain gap if real-world interactions are more complex than the synthesized ones. The paper does not extensively discuss the interpretability of the inferred relationships, which is a known challenge in foundation models. Additionally, the model does not natively support static metadata (e.g., soil type, demographics), which is a limitation for certain scientific applications like ungauged basin prediction, though the authors acknowledge this as future work.
This paper has high potential impact by democratizing advanced forecasting capabilities. By providing a single, efficient, zero-shot model that handles multivariate interactions, it reduces the barrier to entry for applying state-of-the-art forecasting in scientific and industrial domains. The demonstrated success in diverse fields (epidemiology, economics, earth sciences) suggests broad applicability. The efficiency of the model (running on consumer GPUs/CPU) makes it accessible for edge deployment or resource-constrained environments. It shifts the paradigm from task-specific model fitting to contextual inference, which could streamline workflows in data science teams. Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
Reasoning-oriented post-training enables large language models (LLMs) to solve complex tasks via multi-step inference. Supervised fine-tuning (SFT) on high-quality reasoning traces is a particularly efficient approach, but its effectiveness depends critically on careful data curation. Existing pipelines rely on a costly generate-then-filter paradigm: large pools of long reasoning traces are produced by multiple teacher models and subsequently filtered for difficulty and diversity using additional LLMs, often requiring weeks of computation. In this work, we propose SFTProbe, a fundamentally different approach that bypasses this pipeline by directly predicting the quality of reasoning data. We show that a brief LoRA-based adaptation, combined with evaluating loss on only 5% of each reasoning trace, suffices to estimate both difficulty and diversity. SFTProbe is grounded in a geometric view of post-training, where pretrained LLMs lie in a low-rank, anisotropic loss basin. By probing this basin via low-rank perturbations, we construct compact loss signatures whose clustering captures diversity, while loss at curvature inflection points provides a robust proxy for difficulty. We also provide a simple criterion for teacher selection. Empirically, our approach reduces data curation time from weeks to hours while maintaining or improving performance. Remarkably, on mathematical reasoning, 5k generated examples match the highly filtered 89k-example in OpenThoughts-Math dataset, and 7.5k generated examples outperforms it by 2.5% in accuracy. We further introduce a high-quality physics reasoning dataset, and use it to train a model that outperforms an IPhO 2025 silver-medal-winning model. The code and the dataset are released at https://bigml-cs-ucla.github.io/SFTProbe-page/.
Primary: University of California, Los Angeles (UCLA)
All Institutions: University of California, Los Angeles (UCLA)
SFTProbe introduces a principled, geometric approach to SFT data curation that significantly reduces computational costs while maintaining high performance in reasoning tasks. By leveraging low-rank perturbations to estimate data difficulty and diversity, the method offers a scalable alternative to expensive LLM-based filtering, with strong empirical validation on mathematical and physics reasoning benchmarks.
The paper proposes SFTProbe, a method to bypass the standard "generate-then-filter" paradigm for SFT data curation. Instead of using LLMs to judge quality, it utilizes a brief LoRA-based adaptation on a small subset of data to predict the quality of the full dataset. The core theoretical contribution is a geometric view of the loss landscape, arguing that pretrained LLMs reside in a low-rank, anisotropic basin. By probing this basin with low-rank perturbations, the authors construct "loss signatures" where clustering indicates diversity and loss at curvature inflection points indicates difficulty. This is a novel shift from heuristic filtering to a principled, optimization-based estimation of data utility.
The empirical results are strong and directly address the efficiency bottleneck of current pipelines. The claim that 5k generated examples match the performance of 89k highly filtered examples from OpenThoughts-Math is significant, suggesting a massive reduction in compute requirements for data generation. Furthermore, the introduction of a high-quality physics reasoning dataset and the training of a model that outperforms an IPhO 2025 silver-medal-winning model provides concrete evidence of the method's effectiveness in complex reasoning domains. The reduction of curation time from weeks to hours is a practical win for the community.
The paper states that code and datasets are released at the provided URL. The methodology relies on standard techniques (LoRA, loss evaluation) combined with specific probing strategies. While the geometric theory is abstract, the practical implementation appears straightforward enough for reproduction given the released artifacts. The specific hyperparameters for the "brief LoRA-based adaptation" and the definition of "5% of each reasoning trace" are critical details that must be present in the code release for full reproducibility.
The primary limitation is the reliance on the geometric assumption of a "low-rank, anisotropic loss basin." If this assumption fails for certain model architectures or domains, the probing method may lose efficacy. Additionally, the comparison is primarily against the OpenThoughts-Math dataset; broader validation across diverse reasoning tasks (e.g., coding, general QA) would strengthen the claim. The "weeks to hours" claim depends heavily on the scale of the original pipeline being compared against, which may vary across labs.
This work has high potential for adoption in the LLM post-training community. As reasoning models become the standard for complex tasks, the cost of data curation is a major bottleneck. A method that reduces this cost by an order of magnitude while maintaining or improving quality is highly valuable. The geometric perspective on loss landscapes for data selection could also inspire further research into the interplay between model optimization and data quality. SFTProbe introduces a principled, geometric approach to SFT data curation that significantly reduces computational costs while maintaining high performance in reasoning tasks. By leveraging low-rank perturbations to estimate data difficulty and diversity, the method offers a scalable alternative to expensive LLM-based filtering, with strong empirical validation on mathematical and physics reasoning benchmarks.
LLMs are increasingly used to design heuristics for computer systems. However, most synthesis frameworks evaluate candidates by their average performance, which can mask severe regressions on specific workloads. Even adding adversarial training inputs doesn't eliminate these regressions, and percentile objectives (e.g., optimizing for the 95th percentile) overlook how severe regressions can be beyond the selected threshold. We present TAILOR, a framework that guides LLM synthesis using the Conditional Value at Risk (CVaR) of the cost difference between each candidate and an existing baseline on the same workload. This paired CVaR measures the average difference over the worst fraction of workloads. We show that a negative paired CVaR guarantees improvements in both average performance and tail optimality gap relative to the unknown optimum, without ever computing that optimum. We find that TAILOR achieves the best mean--tail trade-off among the synthesis objectives we evaluate across three case studies: Azure GPU batch admission, Azure VM packing, and server placement. Tested across seven diverse optimization benchmarks, TAILOR produces heuristics that are certified with 99% confidence to improve both average performance and tail optimality gaps over the baseline in six of them. On these six benchmarks, TAILOR also outperforms known robust, human-designed alternatives in the tail while achieving better or comparable average-performance improvement over the baseline.
Primary: MIT
All Institutions: MIT, University of Massachusetts Amherst, Microsoft, The University of Texas at Austin
The paper proposes TAILOR, a framework using Conditional Value at Risk (CVaR) to guide LLM-based heuristic synthesis, ensuring robustness against tail workloads. It demonstrates that optimizing for paired CVaR yields heuristics with certified improvements in both average and tail performance over baselines across diverse Azure system benchmarks, outperforming human-designed robust alternatives.
The paper introduces TAILOR, a framework that shifts the objective function for LLM-based heuristic synthesis from average performance to Conditional Value at Risk (CVaR). Specifically, it uses a "paired CVaR" metric measuring the average cost difference between the generated candidate and a baseline over the worst-performing fraction of workloads. This is a significant methodological improvement over standard mean-based objectives or simple percentile objectives, as it directly targets tail risk and guarantees improvements in both average performance and tail optimality gap relative to the unknown optimum without needing to compute the optimum itself. The approach is mathematically grounded and addresses a critical flaw in current LLM-for-optimization pipelines: the masking of severe regressions on specific, albeit rare, workloads.
The evaluation is rigorous and diverse, covering three distinct case studies: Azure GPU batch admission, Azure VM packing, and server placement. The authors test across seven diverse optimization benchmarks. The results show that TAILOR achieves the best mean-tail trade-off compared to other synthesis objectives. Crucially, the paper provides certification: the generated heuristics are certified with 99% confidence to improve both average performance and tail optimality gaps over the baseline in six of the seven benchmarks. The comparison against known robust, human-designed alternatives further strengthens the empirical claims, showing TAILOR outperforms them in the tail while maintaining competitive average performance.
The paper is from a top-tier systems venue (MLSys) with authors from major institutions (MIT, Microsoft, etc.), suggesting high standards for reproducibility. While the provided text is a LaTeX skeleton, the detailed description of the CVaR formulation and the specific benchmarks (Azure workloads) implies that the methodology is well-defined. The use of standard optimization benchmarks and clear metric definitions (paired CVaR) enhances reproducibility. However, without access to the code repository (not provided in the text), full reproducibility of the LLM prompts and specific implementation details of the synthesis loop cannot be verified from the text alone.
The primary limitation is the dependency on the quality of the baseline heuristic; the method optimizes relative to an existing baseline, so if the baseline is poor, the "improvement" might still be suboptimal in absolute terms. Additionally, the computational cost of evaluating CVaR over multiple workloads during the synthesis loop could be high, potentially slowing down the LLM generation process compared to simpler mean-based objectives. The paper also notes that the method requires access to workload data to compute the CVaR, which may not always be available in real-time scenarios.
This work has significant implications for the field of AI-driven systems optimization. By providing a principled way to ensure robustness in LLM-generated heuristics, it addresses a major barrier to deploying LLMs in safety-critical or high-stakes infrastructure management. The certification aspect is particularly valuable for industrial adoption, as it provides statistical guarantees on performance improvements. This could lead to wider adoption of LLMs in designing schedulers, allocators, and other system components where tail performance is critical. The paper proposes TAILOR, a framework using Conditional Value at Risk (CVaR) to guide LLM-based heuristic synthesis, ensuring robustness against tail workloads. It demonstrates that optimizing for paired CVaR yields heuristics with certified improvements in both average and tail performance over baselines across diverse Azure system benchmarks, outperforming human-designed robust alternatives.
Accurate forecasting with uncertainty quantification is critical in a variety of scientific applications ranging from epidemiology and economics to energy systems and earth sciences. Yet most forecasting models still need to be developed and trained separately for each domain-specific dataset. Time series foundation models (TSFMs) aim to remove this limitation and enable the application of the same model across domains through large-scale pretraining. However, most existing TSFMs treat each time series independently and therefore cannot model interactions among time series and external drivers. Here, we introduce Chronos-2, a foundation model for zero-shot forecasting that models the interactions between multivariate time series and heterogeneous covariates, which commonly arise in scientific applications. The model infers relationships among time series directly from historical context and generates probabilistic forecasts without task-specific treatment. Across various benchmarks, Chronos-2 achieves state-of-the-art accuracy on average by improving skill scores by 5.2 percentage points relative to the strongest baselines and increasing win rates by 10 percentage points. Chronos-2 is also computationally efficient, generating forecasts for 100 time series in under one second on a single mid-range GPU, 50x and 1000x faster end-to-end than the best task-specific point and probabilistic forecasters, respectively. In addition, we demonstrate its effectiveness in diverse scientific settings -- including infectious-disease forecasting, macroeconomic modeling, carbon-flux prediction, and runoff simulation -- where the model matches or surpasses specialized methods developed for each domain. These results show that a single pretrained model can capture structure shared across temporal systems.
Primary: Amazon
All Institutions: Amazon, AWS
Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
The paper introduces Chronos-2, a 120M parameter encoder-only Transformer foundation model for time series forecasting. The core methodological contribution is the "group attention" mechanism, which allows the model to infer dependencies among multivariate time series and heterogeneous covariates directly from historical context (in-context learning) without task-specific training. The architecture alternates between temporal attention (within a series) and group attention (across related series). A significant portion of the methodology focuses on the synthetic data generation pipeline, which creates diverse multivariate dynamics by coupling univariate generators (Gaussian Processes, AR, ETS) through linear (VAR, mixing matrices) and non-linear (MLP, RNN) transformations. This approach addresses the scarcity of large-scale multivariate time series data for pretraining. The model uses a quantile regression head to provide probabilistic forecasts, supporting uncertainty quantification.
The evaluation is extensive and rigorous, covering both standard benchmarks and domain-specific case studies. On the FEV benchmark (100 tasks), Chronos-2 achieves state-of-the-art results, outperforming strong baselines like TiRex, TimesFM, and Moirai. It demonstrates significant gains in win rate and skill score over univariate inference, validating the group attention mechanism. The paper includes four detailed case studies: epidemiology (ECDC RespiCast), macroeconomics (German CPI), earth science (FLUXNET carbon flux), and hydrology (CAMELS-DE rainfall-runoff). In each domain, the model matches or surpasses specialized, task-specific models (e.g., HBV, LSTM ensembles, XGBoost) in zero-shot settings. The inclusion of hierarchical forecasting experiments further demonstrates the model's ability to capture cross-level dependencies without explicit hierarchy information. The computational efficiency is also highlighted, with inference times under 1 second for 100 series on a mid-range GPU.
The paper provides detailed architectural specifications, training hyperparameters, and data generation procedures. The model size (120M parameters) and training setup (4x A100 GPUs) are clearly stated. The synthetic data generation process is described in detail, allowing for potential replication of the pretraining corpus. However, the specific code repository or model weights are not explicitly linked in the provided text (though likely available given the Amazon affiliation and prior Chronos releases). The evaluation protocols for the case studies are well-defined, referencing standard domain metrics (WIS, NSE, etc.).
The model is limited to 120M parameters, which is smaller than some competing foundation models (e.g., Moirai, TimesFM), potentially limiting its capacity for extremely complex dynamics. The reliance on synthetic data for multivariate pretraining may introduce a domain gap if real-world interactions are more complex than the synthesized ones. The paper does not extensively discuss the interpretability of the inferred relationships, which is a known challenge in foundation models. Additionally, the model does not natively support static metadata (e.g., soil type, demographics), which is a limitation for certain scientific applications like ungauged basin prediction, though the authors acknowledge this as future work.
This paper has high potential impact by democratizing advanced forecasting capabilities. By providing a single, efficient, zero-shot model that handles multivariate interactions, it reduces the barrier to entry for applying state-of-the-art forecasting in scientific and industrial domains. The demonstrated success in diverse fields (epidemiology, economics, earth sciences) suggests broad applicability. The efficiency of the model (running on consumer GPUs/CPU) makes it accessible for edge deployment or resource-constrained environments. It shifts the paradigm from task-specific model fitting to contextual inference, which could streamline workflows in data science teams. Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
We develop a new algorithmic framework for learning Boolean concept classes under logconcave distributions. Our framework applies to broad families of multi-index models and yields significantly faster robust learners than were previously known in these settings. It extends to partial classification and testable learning. Consequently, we obtain improved learners for several fundamental concept classes, including functions of halfspaces and multiclass linear classifiers, under logconcave marginals. Our starting point is a new boosting framework based on weak partial learners. Classical boosting approaches require often-intractable distribution-free weak learners or yield weak approximation guarantees. We show instead that weak learners for partial classification, where labels may be undefined on part of the domain, suffice for strong agnostic learning. The key challenge is that boosting requires a weak learner that succeeds under conditioned distributions that are not guaranteed to be logconcave. To address this, we develop a recursive moment-based partitioning procedure: given a logconcave distribution $X$ and a conditional version $X'$, we partition the space into polynomially many convex regions so that, on most regions, the low-degree moments of $X$ and $X'$ approximately agree. On each good region, a well-behaved multi-index model correlates with a low-degree polynomial, giving a weak hypothesis. Combining these hypotheses yields the required weak learner. A key ingredient of our approach is an efficient method for detecting moment discrepancy. Specifically, we show how to find a low-dimensional subspace that captures any non-trivial discrepancy between $X$ and $X'$, if one exists. Our approach combines spectral analysis of moment tensors with recent progress on isoperimetry for logconcave measures. This discrepancy-finding procedure may be of independent interest.
Primary: University of Wisconsin-Madison
All Institutions: University of Wisconsin-Madison, University of California, San Diego
The paper develops a new algorithmic framework for robust learning of multi-index models under logconcave distributions using weak partial learners, yielding significantly faster robust learners for fundamental concept classes like halfspaces and multiclass linear classifiers. It introduces a novel boosting approach that overcomes the challenge of non-logconcave conditioned distributions via a recursive moment-based partitioning procedure and an efficient discrepancy finder, providing substantial theoretical improvements over prior state-of-the-art methods.
The paper introduces a novel algorithmic framework for robust learning of multi-index models (MIMs) under logconcave distributions. The core innovation is a boosting framework based on "weak partial learners," which allows for strong agnostic learning guarantees without requiring intractable distribution-free weak learners. A key technical contribution is a recursive moment-based partitioning procedure that handles the challenge of boosting under conditioned distributions that are not logconcave. The authors develop an efficient discrepancy finder using spectral analysis of moment tensors and isoperimetry for logconcave measures to identify low-dimensional subspaces capturing moment discrepancies. This enables the construction of weak hypotheses on convex regions where moments approximately match, which are then combined via boosting. The approach is theoretically rigorous, providing sample and computational complexity bounds that significantly improve upon prior state-of-the-art for learning halfspaces, functions of halfspaces, and multiclass linear classifiers under logconcave marginals.
As a theoretical computer science paper, it does not contain empirical experiments on real-world datasets. The "evaluation" consists of rigorous mathematical proofs establishing the correctness and complexity bounds of the proposed algorithms. The results are validated by comparing the derived bounds against known lower bounds and previous upper bounds in the literature, demonstrating significant improvements in efficiency and robustness.
The paper provides detailed algorithmic descriptions and theoretical proofs. While no code is provided, the algorithms are described in sufficient detail for theoretical replication. The reliance on standard statistical query models and polynomial regression techniques makes the theoretical framework reproducible in a mathematical sense.
The primary limitation is the theoretical nature of the work; there is no empirical validation on real-world data to confirm that the theoretical gains translate to practical performance improvements. The algorithms may have high constant factors or practical overheads not captured by the asymptotic analysis. The focus on logconcave distributions, while broad, excludes other common distributional assumptions.
This work contributes to the foundational understanding of efficient learning under distributional assumptions. It provides new tools (weak partial learners, moment discrepancy finders) that could be applicable to other learning problems beyond MIMs. The results advance the state of the art in robust learning, which is crucial for real-world applications where data is often noisy and distributional assumptions are only approximately true. The paper develops a new algorithmic framework for robust learning of multi-index models under logconcave distributions using weak partial learners, yielding significantly faster robust learners for fundamental concept classes like halfspaces and multiclass linear classifiers. It introduces a novel boosting approach that overcomes the challenge of non-logconcave conditioned distributions via a recursive moment-based partitioning procedure and an efficient discrepancy finder, providing substantial theoretical improvements over prior state-of-the-art methods.
Understanding the training dynamics of deep neural networks in large-scale limits is a question both of foundational and of practical importance. Prior works have obtained such descriptions through dynamical mean field theory (DMFT), yielding a system of non-Markovian dynamics that are challenging to analyze and that obscure the optimization structure of the original dynamics. In this paper, we obtain instead a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the $1/\text{depth}$ branch scaling that allows maximal (local) feature updates. For a model trained with gradient descent (GD) from a random initialization, the limit dynamics is simply GD on a Neural Mean ODE with Gaussian initial weights. Our analysis covers population or empirical risk GD in the joint limit of large width, depth and embedding, input, and output dimensions. We obtain approximation scaling laws that depend on the shape of the ResNet and the covariance spectrum of the data. We also illustrate the flexibility of our approach by discussing Mixture of Expert models, where likewise the limit admits an intuitive (nested) Mean ODE description.
Primary: EPFL
All Institutions: EPFL, University of Oxford
The paper establishes that the large-scale training dynamics of ResNets converge to Gradient Descent on a Gaussian Mean ODE, providing a tractable and structurally faithful limit model. This represents a significant advance in the theoretical analysis of deep learning by simplifying the complex non-Markovian dynamics of DMFT into a standard optimization problem, enabling sharper convergence rates and deeper insights into the role of architecture shape and data covariance.
The paper proposes a rigorous theoretical framework for analyzing the training dynamics of Residual Neural Networks (ResNets) in the joint limit of large depth, width, and embedding dimension. The core innovation is deriving a "Gaussian Mean ODE" limit, which simplifies the complex non-Markovian dynamics typically found in Dynamical Mean Field Theory (DMFT) into a standard Gradient Descent (GD) process on a Neural Mean ODE. The methodology relies on a two-stage limit: first, depth divergence leads to a Mean ODE, and second, embedding dimension divergence with Gaussian initialization leads to a dimension-free Gaussian Mean ODE. The authors provide quantitative convergence rates and prove that the limit dynamics preserve the optimization structure of the original network, a significant improvement over prior DMFT approaches that obscure this structure. The use of Hilbert space formalism allows for generalization to infinite-dimensional input/output spaces.
As a theoretical paper, experimental evaluation is limited to numerical validations of the derived convergence rates and the behavior of the limit model. The paper includes plots demonstrating that the empirical errors match the theoretical bounds (e.g., $O(1/L + D/ML)$) and illustrates the "quasi-smoothness" of the limit objective. While not providing large-scale empirical benchmarks, the numerical experiments effectively validate the theoretical claims and the tightness of the derived bounds.
The paper is highly reproducible in a theoretical sense, with clear definitions, lemmas, and theorems. The code for numerical experiments is likely available (though not explicitly linked in the provided text snippet, standard practice for such papers from this group implies code availability). The mathematical derivations are detailed, allowing other researchers to verify the proofs. The reliance on AI for specific proof steps (GPT-6 Astra) is disclosed, which is a novel aspect of reproducibility/transparency in the research process itself.
The analysis is restricted to ResNets with two-layer perceptron blocks and specific scaling regimes (MLU/1-depth). It does not cover attention mechanisms or more complex architectures directly, although it mentions MoE. The results are for Gradient Descent; extensions to SGD or other optimizers are not covered. The assumption of Gaussian initialization for the sharpest results is a limitation, though they do discuss non-Gaussian cases with weaker bounds.
This work provides a crucial bridge between the practical architecture of ResNets and the theoretical understanding of their training dynamics. By showing that the limit is simply GD on a Mean ODE, it makes the analysis of deep network training significantly more tractable. This could lead to better hyperparameter tuning strategies, understanding of generalization in deep networks, and potentially new optimization algorithms designed specifically for the Mean ODE limit. It contributes to the foundational understanding of why deep networks train well. The paper establishes that the large-scale training dynamics of ResNets converge to Gradient Descent on a Gaussian Mean ODE, providing a tractable and structurally faithful limit model. This represents a significant advance in the theoretical analysis of deep learning by simplifying the complex non-Markovian dynamics of DMFT into a standard optimization problem, enabling sharper convergence rates and deeper insights into the role of architecture shape and data covariance.
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
AI has made remarkable progress in mathematics, solving challenging problems with machine-checkable proofs in Lean, yet typically relying on predefined mathematical definitions and formal specifications. This raises a natural question: can it reason rigorously about an unknown world? We introduce the Algebraic World Model (AWM), a framework that turns interaction with unfamiliar environments into explicit mathematical models supporting formal reasoning. AWM explores the environment and develops executable Python hypotheses, continually testing and revising them against observations. It then structures these hypotheses into typed objects and shared rules, simplifies their structure through category-theory-guided transformations, and formalizes the resulting model in Lean 4 to prove reachability and impossibility. AWM can formally prove that a goal is unreachable, even beyond explored trajectories. On two impossible game levels, it constructs closed, kernel-checked Lean proofs excluding every legal winning sequence in all six runs, while Claude Opus 5.5 and GPT-6 Astra, equipped with coding tools, produce none in twelve baseline runs. Four of AWM's six initial bounds are correct, compared with one for Opus 5.5 and none for GPT-6 Astra; given only feedback that a cheaper win exists, AWM revises its rules and constructs new proofs, reaching six of six correct bounds. Moreover, AWM produces interpretable mathematical models whose rules can be inspected and tested step by step. Across all 25 public ARC-AGI-3 games, these models enable Lean to replay the winning trajectories of all 183 levels and generalize beyond observed play, predicting unseen transitions and detecting changes in game mechanics. With Claude Opus 5, AWM clears all levels with a score of 99.9, while its certified winning trajectories use only 43% of the human baseline's actions.
Primary: unknown
All Institutions: unknown
The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
The paper introduces the Algebraic World Model (AWM), a framework that bridges empirical exploration and formal verification. The methodology is structured in three stages: (1) Explore & Prototype, where an LLM agent interacts with an environment to generate executable Python hypotheses; (2) Structure & Simplify, where these hypotheses are organized into typed objects and rules using category-theory-guided transformations (limits/colimits) verified in Lean; and (3) Formalize & Prove, where the model is formalized in Lean 4 to prove reachability or impossibility. The use of category theory to guide the simplification of world models is a creative and rigorous approach to ensuring that the learned model preserves the logical structure of the environment. The "sufficient cover" definition provides a theoretical basis for transferring proofs from the learned model to the actual environment, which is a significant methodological contribution.
The experiments are conducted on ARC-AGI-3, a challenging benchmark for abstract reasoning. The results are strong: AWM achieves a score of 99.9, outperforming baselines like Claude Opus 5.5 and GPT-6 Astra in both game clearing and, crucially, in constructing formal proofs of impossibility. The paper demonstrates that AWM can produce kernel-checked Lean proofs that exclude all legal winning sequences for impossible levels, a capability the baselines lack. The generalization tests, where frozen models predict unseen transitions and detect mechanic changes, further validate the robustness of the learned algebraic structures.
The paper provides a high level of reproducibility by releasing an interactive website with models, tests, Lean proofs, and links to official replays. The inclusion of the Lean model for the running example (lp85) in the supplementary materials, along with scripts for building and auditing axioms, allows for independent verification of the formal proofs. The explicit statement that all numbers come from script outputs over recorded runs, rather than model-generated text, enhances trust in the results.
The primary limitation is the scope of the environments tested. AWM is demonstrated on ARC-like environments with discrete actions and immediate, observable feedback. The authors acknowledge that richer hidden states, delayed feedback, and stochastic dynamics remain untested. Additionally, the reliance on LLMs for the initial hypothesis generation and exploration phase means that the system's performance is bounded by the capabilities of the underlying LLM. The "sufficient cover" condition, while theoretically sound, requires careful verification in practice to ensure that the learned model indeed covers the optimal winning sequences of the environment.
This work has significant implications for the field of AI, particularly in the areas of formal methods, automated theorem proving, and agent-based reasoning. By enabling AI systems to reason rigorously about unknown worlds and produce machine-checkable proofs, AWM paves the way for more trustworthy and interpretable AI systems. The integration of category theory with machine learning offers a new paradigm for structuring and simplifying learned models, which could be applied to other domains requiring rigorous reasoning. The ability to prove impossibility, not just reachability, is a crucial step towards AI systems that can provide complete and reliable solutions to complex problems. The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
Graph-based nearest neighbor search algorithms like HNSW, DiskANN, NSG, and others have surged in popularity in recent years due to their excellent empirical performance in comparison to space-partitioning methods like IVF data structures and Locality Sensitive Hashing (LSH). However, in contrast to LSH and related methods, obtaining strong worst-case theoretical guarantees for graph-based search has proven challenging. In this paper, we prove a strong lower bound that helps explain the lack of progress towards such guarantees. We show that no standard graph-based method (any that runs a variant of greedy search on a flat or hierarchical graph) can achieve sublinear query time for approximate nearest neighbor search in Euclidean space, even for an arbitrarily large approximation factor. On the other hand, we prove that sublinear query time is possible if we allow for Steiner points in the search graph. We obtain bounds comparable to LSH, showing e.g., that graph-based search with Steiner points can solve $c$-approximate nearest neighbor search in Euclidean space with query time $\tilde{O}(n^{1/(c-1)})$. We complement this result with initial experiments exploring how Steiner points can be incorporated into practical graph-based methods.
Primary: New York University
All Institutions: New York University, UMass Amherst
The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
The paper presents a rigorous theoretical framework for analyzing graph-based nearest neighbor search (NNS) algorithms like HNSW and DiskANN. The core methodological contribution is the formalization of a "flat search graph" model that captures generalized greedy search, allowing for the derivation of strong lower bounds. The authors prove that without Steiner points, no such method can achieve sublinear query time in worst-case Euclidean high-dimensional settings, effectively explaining the long-standing gap between empirical performance and theoretical guarantees. Conversely, they introduce a novel construction using Steiner points to simulate Locality Sensitive Hashing (LSH) within the graph search framework, achieving query times comparable to LSH ($\tilde{O}(n^{1/(c-1)})$). The proof techniques involve constructing hard instances of orthogonal vectors and leveraging geometric properties of Steiner nodes to implement hash table lookups via greedy traversal.
The experimental section is limited but sufficient for an initial exploration. The authors augment the ParlayANN implementation of DiskANN with simple $k$-means cluster centroids as Steiner points. They evaluate this on standard benchmarks (GloVe-200, NYTimes) using cosine similarity. The results show consistent improvements in recall vs. distance computations compared to the baseline, validating the theoretical potential of Steiner points. However, the experiments are preliminary, using a very simple heuristic for Steiner point generation rather than the optimal theoretical construction, and the datasets are relatively small compared to industrial scale.
The paper provides clear pseudocode for the generalized greedy search and detailed descriptions of the theoretical constructions. The experimental setup specifies the use of ParlayANN and specific hyperparameters (degree bound, pruning parameter, beam width). While the code for the specific Steiner-point augmentation is not explicitly linked in the text provided, the reliance on the open-source ParlayANN library and standard $k$-means clustering makes the experimental results highly reproducible.
The primary limitation is the gap between the theoretical upper bound (which requires a complex, specific construction of Steiner points to simulate LSH) and the experimental lower bound (which uses simple $k$-means centroids). The experiments do not demonstrate that the simple heuristic achieves the theoretical $O(n^{1/(c-1)})$ bound, only that it improves over the baseline. Additionally, the lower bound for the high-dimensional case requires $d=n$, leaving the low-dimensional worst-case complexity (for $d \ll n$) partially open, though a weaker bound is provided. The paper also ignores preprocessing time, focusing solely on query time.
This paper has significant implications for the vector database and search engine community. By proving that standard graph-based methods cannot theoretically match LSH in worst-case high-dimensional settings, it redirects research focus toward hybrid methods or Steiner-point-augmented graphs. The introduction of Steiner points as a mechanism to bridge the gap between empirical graph search and theoretical LSH guarantees offers a new design space for next-generation ANN algorithms. This could lead to more robust search systems with provable performance guarantees, which is critical for safety-critical applications. The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory $32\times$ faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains $4.9\times$ faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.
Primary: Massachusetts Institute of Technology (MIT)
All Institutions: Massachusetts Institute of Technology (MIT)
[One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
The paper proposes ParaTTT (PTTT), an algorithm to parallelize Test-Time Training (TTT) layers, which replace linear fast-weight memories with small neural networks trained via online gradient descent. The core innovation is an iterative refinement scheme that approximates the sequential inner-loop gradient trajectory using chunkwise-parallel passes. The method leverages the fact that the exact online gradient descent recurrence is a fixed point of the iteration. The authors provide a theoretical guarantee of geometric convergence for a two-layer variant under specific contraction conditions. The approach decomposes the update into a forward pass for feature trajectories, an exact gated-delta-rule pass for readouts and feature gradients, and a backward pass for gradient propagation. This allows the nonlinear memory to be evaluated in parallel, overcoming the sequential bottleneck of previous TTT implementations like TTT-MLP.
The experiments demonstrate significant speedups: PTTT evaluates the inner-loop trajectory 32x faster than TTT-MLP at 4K context with matched error, and the full 340M model trains 4.9x faster. The models (340M and 1.3B parameters) are evaluated on language modeling perplexity and zero-shot accuracy. The results show that PTTT models match or exceed the performance of Gated DeltaNet and Mamba-2, and remain competitive with same-size Transformers. Notably, the paper claims better recall than linear recurrences, suggesting the nonlinear memory provides a qualitative advantage in information retention. The comparison with Transformers is particularly strong, as TTT layers have historically struggled to match Transformer performance at scale.
The paper provides a GitHub repository link (https://github.com/ntumm120/paraTTT), which is a strong positive for reproducibility. The abstract mentions specific model sizes (340M, 1.3B) and context lengths (4K), which are standard benchmarks. However, the full text provided is truncated (only abstract and headers are visible), so specific hyperparameters, dataset splits, and training schedules cannot be fully verified from the provided text. Assuming the code repository is complete and well-documented, reproducibility should be high.
The theoretical convergence proof is limited to a two-layer variant, while practical applications may involve deeper networks. The claim that "two iterations suffice" in practice needs robust validation across different sequence lengths and model scales. The comparison with Transformers is competitive but not necessarily superior in all metrics, and the "better recall" claim requires careful definition of recall in the context of language modeling. The computational overhead of the iterative refinement, even if parallelized, may still be higher than simple linear recurrences for very short sequences where parallelism gains are minimal.
This work addresses a critical bottleneck in the adoption of nonlinear recurrent architectures. By enabling efficient parallelization of TTT layers, it makes these more expressive models viable for large-scale language modeling. This could lead to new architectures that combine the expressiveness of nonlinear memories with the efficiency of parallel training, potentially challenging the dominance of Transformers in long-context tasks. The technique of iterative refinement for parallelizing sequential operations is also broadly applicable to other sequential learning problems. [One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
We study adversarial linear bandits with strong corruption, in which an adaptive adversary chooses the reward vectors and a strong adversary corrupts each observed reward after seeing the learner's action, subject to a known total corruption budget $C$. Strong corruption breaks the importance-weighted reward estimates used by adversarial bandit algorithms: by corrupting only the rounds in which a rarely sampled action is drawn, the adversary amplifies each unit of corruption by the inverse of its sampling probability. The best known bound for this problem, $\tilde O(\sqrt{d^3T} + dC)$ by Liu et al. (2024), is a factor $\sqrt{d}$ above the corruption-free rate in its leading term, and whether $\tilde O(d\sqrt{T} + dC)$ regret is achievable was left open. We propose CRW-FTRL, which runs follow-the-regularized-leader with a radius-weighted entropy regularizer and estimates rewards with a normalized kernel estimator whose sensitivity to corruption is governed by per-action radii instead of inverse sampling probabilities. The radii are nondecreasing and are chosen jointly with the sampling distribution through a monotone log-determinant barrier problem, so that a negative linear term in the regularizer cancels both the corruption bias and the estimation error of every comparator. For $K$ actions in $\mathbb{R}^d$, CRW-FTRL achieves $\tilde O(\sqrt{dT\log K} + dC)$ regret with high probability against an adaptive adversary, which matches the best known corruption-free rate and the $Ω(dC)$ lower bound up to logarithmic factors. Applied to a discretization of a general compact action set, it yields $\tilde O(d\sqrt{T} + dC)$ regret, which settles the statistical rate of corrupted adversarial linear bandits under strong corruption.
Primary: University of California
All Institutions: University of California
The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
The paper proposes CRW-FTRL, a novel algorithm for adversarial linear bandits with strong corruption. The core innovation lies in replacing standard importance-weighted reward estimates—which are vulnerable to strong adversaries who target rarely sampled actions—with a normalized kernel estimator governed by per-action radii. These radii are determined via a monotone log-determinant barrier problem, coupled with a radius-weighted entropy regularizer. This design effectively decouples the sensitivity of the reward estimator from the inverse sampling probabilities, preventing the adversary from amplifying corruption. The theoretical framework is rigorous, leveraging FTRL with a specific regularizer structure to cancel out both corruption bias and estimation error.
The provided text is a theoretical paper focused on regret bounds. It does not contain empirical experiments, benchmarks, or comparisons with baselines in a practical setting. The contribution is entirely analytical, proving that the proposed algorithm achieves minimax optimal regret rates.
As a purely theoretical paper, reproducibility is tied to the correctness of the proofs rather than code implementation. The algorithm is described in sufficient detail for implementation, but the lack of code or empirical validation limits practical reproducibility assessment.
The primary limitation is the lack of empirical validation. The paper acknowledges that the reduction to general compact action sets via discretization is not computationally efficient, leaving the question of a computationally efficient algorithm for general action sets open. Additionally, the results are high-probability bounds, and the constants hidden in the tilde-O notation may be significant in practice.
This work settles a key open question in the theory of corrupted bandits, providing minimax optimal rates for adversarial linear bandits with strong corruption. It has significant implications for the design of robust online learning algorithms in adversarial environments, such as online advertising and recommendation systems where feedback can be manipulated. The techniques developed (radius-weighted regularizers, kernel estimators with controlled sensitivity) may be applicable to other robust learning problems. The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.
Primary: Massachusetts Institute of Technology
All Institutions: Massachusetts Institute of Technology, Carnegie Mellon University, University of Maryland
The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
The paper introduces a novel cross-modal pre-training framework that aligns egocentric video with surface electromyography (EMG) signals. The core methodological contribution is the use of EMG as a "privileged supervisory signal" in a CLIP-style contrastive objective. By jointly training a video encoder and an EMG encoder, the authors create a shared embedding space where visual representations are shaped by the underlying motor intent, which is then discarded at inference time. This approach is technically sound, leveraging the dense, occlusion-free nature of EMG to provide a stronger inductive bias for action-centric vision tasks than standard self-supervised video learning. The inclusion of a VQ-VAE to learn discrete "motor codes" for conditioning video diffusion models is a creative extension, though the reliance on the Video2LoRA framework limits the novelty of the generative component itself.
The experimental evaluation is extensive and rigorous. The authors demonstrate consistent improvements over strong baselines (SimCLR, MoCo, BYOL, and ImageNet/MAE initialization) across four distinct task families: action recognition (EPIC-Kitchens, SSv2, Ego-Exo4D), 3D hand reconstruction (ARCTIC, H2O, HOT3D), 3D contact reconstruction, and video generation. The gains are particularly notable in the frozen-backbone setting for action recognition, suggesting the pre-training captures robust, transferable features. The comparison against RGB-only self-supervised methods trained on the same data effectively isolates the contribution of the EMG signal. The qualitative results for video diffusion show clear improvements in fine-grained hand-object interaction fidelity compared to language or video-history conditioning.
The paper provides high reproducibility. It details the specific hardware (MindRove Armband, Meta Aria glasses), data collection protocols, preprocessing steps (filtering, normalization), and model architectures. The release of the 100-hour dataset is a significant asset. However, the specific implementation of the "Video2LoRA" and "HyperNetwork" components relies on external citations, and the code for the VQ-VAE motor code discovery is described but not fully detailed in the main text, potentially requiring access to the supplementary materials or code repository (not explicitly linked in the provided text, though a project page is mentioned) for full replication.
The primary limitation is the scale of the dataset (100 hours) compared to massive video datasets like Kinetics or Ego4D. The EMG hardware used is consumer-grade (8 channels), which may limit the granularity of muscle activity captured compared to research-grade high-density EMG. The video diffusion experiments are limited to a specific backbone (CogVideoX-Fun) and a lightweight LoRA approach, which may not generalize to all state-of-the-art video generation models. Additionally, the dataset is focused on kitchen/household manipulation, which may limit generalizability to other domains.
This work has significant potential impact on embodied AI, robotics, and VR/AR. By providing a dataset and method that links visual perception with motor intent, it offers a pathway to more intuitive human-robot interaction and better understanding of human actions from video. The motor-code conditioning for video generation could enable new tools for content creation and simulation. The dataset itself becomes a valuable resource for the community, addressing a gap in paired video-EMG data. The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally. Based on this insight, we introduce SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. SpecSlot expands each spatial anchor into multiple slots, each assigned a disjoint band of DCT coefficients; this separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds a local correction to the spectral prediction, while the generative state, target, and loss remain in pixel space. On ImageNet-256, SpecSlot achieves a comparable final FID to a FLOPs-matched JiT control while converging faster. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within only 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image generation, SpecSlot surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.
Primary: Peking University
All Institutions: Peking University, Nanyang Technological University, SenseTime
SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
The paper proposes SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from global attention sequence length. The core innovation is the "Role-Coupled Tokenization" where each spatial anchor is expanded into multiple slots, each assigned a disjoint band of DCT coefficients. Only the lowest-frequency slot participates in global attention, while high-frequency slots interact locally. This is a clever architectural modification that leverages the spectral properties of natural images (low-frequency structure requires global context, high-frequency detail is local). The method remains in pixel space, avoiding the artifacts often associated with VAE-based latent diffusion, while mitigating the quadratic cost of attention in pixel-space models. The inclusion of a spatial head for local correction is a sensible addition to handle residual errors.
The experiments are conducted on ImageNet-256 and ImageNet-512, as well as text-to-image generation benchmarks (DPG-Bench). The results show that SpecSlot achieves comparable FID to FLOPs-matched JiT controls on ImageNet-256 with faster convergence. On ImageNet-512, it improves FID by 2.84 over the control with fewer FLOPs and comes within 0.69 FID of a larger JiT model at significantly lower cost. The text-to-image results show improvements over JiT-B/8. The comparisons are against strong baselines (JiT), and the efficiency gains are substantial.
The paper provides links to the project page, GitHub repository, and Hugging Face model weights. The methodology is described in sufficient detail in the provided text (though the full technical details are in the sections referenced but not fully expanded in the prompt's text snippet, the abstract and summary provide the core mechanism). The availability of code and weights significantly enhances reproducibility.
The method relies on DCT coefficient bands, which may not be optimal for all types of content or modalities. The local interaction of high-frequency slots might limit the model's ability to capture long-range high-frequency correlations, although this is argued to be less critical. The computational overhead of managing multiple slots per anchor needs careful engineering to ensure the efficiency gains are realized in practice.
This work offers a viable path for pixel-space generative models to scale to higher resolutions without the prohibitive cost of full global attention. It could influence the design of future pixel-space Transformers and potentially other modalities where spectral decomposition is applicable. The efficiency gains make high-quality pixel-space generation more accessible. SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Primary: Tencent Youtu Lab
All Institutions: Tencent Youtu Lab
Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
The paper proposes Youtu-Parsing-Omni, a unified architecture for parsing documents, images, audio, and video into structured records. The core methodological contribution is the "Youtu-Omni-Encoder," which initializes from a pre-trained language model and uses shared weights to encode all modalities in a unified space-time coordinate system. A key architectural innovation is the inclusion of fusion layers that allow video frames to attend to concurrent audio, integrating cross-modal fusion directly into the perception layer. The training methodology introduces "OmniSchema-Aware On-Policy Distillation" (OSAD), a self-evolving post-training procedure where the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself. This allows the parser and its supervision to co-evolve, unifying the outputs of ten different parsing tasks under a single schema.
The paper reports state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.). The results highlight a significant advantage in audio and video parsing compared to general-purpose systems. It also achieves the best overall chart-parsing score (95.02) and high exact chemical-fingerprint agreement. The model matches dedicated optical-music models in character-level transcription of typeset sheet music. The evaluation covers a wide range of modalities and tasks, demonstrating the versatility of the unified approach.
The paper states that code and model weights are publicly available, which is a strong positive for reproducibility. However, the provided text does not include specific URLs or detailed hyperparameter settings, making it difficult to verify the exact implementation details without accessing the external repository. The description of the OSAD training procedure is high-level, and the specific mechanics of the "reference-conditioned copy" and "token-role-based objective" would require the code to fully understand and replicate.
The paper relies heavily on the performance of the pre-trained language model initialization, which may limit its applicability to domains where text is not the primary signal. The "self-evolving" nature of the OSAD training could be computationally expensive and potentially unstable, though this is not explicitly discussed. The evaluation is primarily on benchmark datasets, and real-world robustness to noisy or unstructured data is not thoroughly tested. The lack of specific URLs in the text prevents immediate verification of the claimed open-source availability.
This work has significant potential to impact the field of multimodal AI by providing a unified perception layer for agents that need to read, locate, and cite evidence across multiple modalities. The ability to parse documents, images, audio, and video into structured records using a single encoder and schema could simplify the development of complex AI agents. The focus on spatial and temporal grounding is particularly relevant for applications in robotics, autonomous systems, and advanced document analysis. Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.
Primary: Unknown
All Institutions: Unknown
The paper introduces a novel "WorldSketch" paradigm where VLMs construct minimal kinematic world models for robotic planning, achieving state-of-the-art zero-shot success rates in manipulation and navigation tasks while significantly reducing token usage compared to code-generation agents. By decoupling semantic scene understanding from geometric motion planning, the method demonstrates that explicit, verifiable world models built by VLMs can outperform end-to-end action generation, offering a scalable and interpretable approach to general-purpose robotics.
The paper proposes "WorldSketch," a paradigm where a Vision-Language Model (VLM) constructs a minimal, kinematic world model (sketch) rather than generating actions directly. The VLM acts as an agent querying perception tools (depth lifting, point marking) to build a model of task-relevant entities (joints, links, parts). A geometric planner then optimizes contact and whole-body motion on this explicit model. The method includes a self-improving loop where execution feedback verifies outcomes and updates joint hypotheses via Bayes' rule. The methodology is sound, leveraging the VLM's semantic understanding for scene abstraction and the planner's precision for geometry. The separation of concerns (VLM for "what matters," Planner for "how to move") is a strong architectural choice that addresses the limitations of end-to-end VLA models and code-as-policy agents.
The evaluation is extensive, covering RoboCasa (kitchen manipulation), LIBERO-PRO (tabletop), and R2R-CE (navigation). The results are impressive, with 91% success on RoboCasa zero-shot, significantly outperforming baselines like CaP-X (33%) and Code as Policies (9%). The ablation studies are rigorous, isolating the impact of grounding methods (points vs. masks), overlay checks, and hypothesis testing. The comparison against trained VLAs (GR00T, pi0.5) demonstrates the superiority of the zero-shot agentic approach in generalization. The token efficiency analysis (10x fewer tokens than CaP-X) is a valuable practical insight.
The paper provides detailed descriptions of the sketch language, tool interfaces, and planner components. However, the system relies on proprietary VLMs (GPT-6.1-Sol, GPT-6-Astra) and specific simulation environments (RoboCasa365, MuJoCo). The exact prompts and backend logic are described but not fully open-sourced in the text provided. The reliance on specific VLM versions may limit immediate reproducibility for those without access to these models, but the architectural principles are clearly defined.
The system is evaluated primarily in simulation. Real-world sensor noise (depth errors >1cm) degrades performance. The current sketches cover only rigid parts with single joints, excluding deformable objects and complex multi-joint chains. The VLM latency (15s median) adds significant time to the loop. Contact-rich execution on small/thin parts remains a challenge, with failures often due to gripper limitations rather than planning errors.
This work offers a compelling alternative to the dominant VLA trend, suggesting that explicit world modeling by VLMs can achieve superior zero-shot generalization with lower computational cost. It bridges the gap between high-level semantic reasoning and low-level geometric planning. The paradigm of "sketching" the world is intuitive and could be extended to other robotic domains, potentially influencing how future agentic systems are designed to interact with physical environments. The paper introduces a novel "WorldSketch" paradigm where VLMs construct minimal kinematic world models for robotic planning, achieving state-of-the-art zero-shot success rates in manipulation and navigation tasks while significantly reducing token usage compared to code-generation agents. By decoupling semantic scene understanding from geometric motion planning, the method demonstrates that explicit, verifiable world models built by VLMs can outperform end-to-end action generation, offering a scalable and interpretable approach to general-purpose robotics.
A robot's next action can depend on recent motion or earlier visual evidence absent from its current observation. A recurrent visual-memory architecture can retain this history in a visual state, but its usefulness depends on what the state represents and how it is updated with new observations. We introduce Prediction-Regulated Memory (PREM), a framework that uses action-conditioned future prediction to guide both memory learning and updating in vision-language-action policies. During training, predicting future visual features provides temporal supervision for memory learning. During execution, PREM compares current visual features with the prediction made at the preceding decision and uses the spatially normalized prediction discrepancy to regulate updates to the visual state at each spatial location. The updated visual state supplies temporal context for action generation alongside current observations. Built on $π_{0.5}$, PREM improves average success rate from 9.63% to 20.74% on DOMINO dynamic manipulation tasks and from 14.4% to 83.8% on the RMBench M(1) memory-dependent setting. Further evaluations on LIBERO and LIBERO-Pro show improvements in general manipulation and robustness to spatial perturbations. PREM incurs a mean inference overhead of 7.7 ms on a single NVIDIA H800 GPU.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces a novel memory regulation mechanism for VLA policies that leverages action-conditioned future prediction to improve temporal context retention, demonstrating significant performance gains on memory-dependent and dynamic manipulation benchmarks with minimal computational overhead.
The paper proposes PREM (Prediction-Regulated Memory), a framework that integrates action-conditioned future prediction into the memory mechanism of Vision-Language-Action (VLA) policies. The core innovation is twofold: during training, future visual feature prediction serves as a temporal supervision signal for the memory state; during inference, the discrepancy between the predicted future visual state and the actual current observation is used to spatially regulate the update of the recurrent memory. This approach addresses the common issue in robotic manipulation where the current observation lacks sufficient context about recent dynamics or hidden states. By leveraging prediction errors to modulate memory updates, the method aims to retain relevant historical information while discarding noise. The method is built upon the $\pi_{0.5}$ architecture, a strong baseline in the field.
The experimental evaluation is robust, covering multiple benchmarks: DOMINO (dynamic manipulation), RMBench (memory-dependent tasks), LIBERO, and LIBERO-Pro. The reported improvements are substantial, particularly on RMBench M(1), where success rate jumps from 14.4% to 83.8%, and on DOMINO, from 9.63% to 20.74%. These results suggest the method effectively handles tasks requiring temporal context. The inclusion of LIBERO-Pro evaluations for robustness to spatial perturbations adds depth to the generalization claims. The inference overhead of 7.7 ms on an H800 GPU is reported as negligible, which is a practical advantage for real-time control.
The paper includes a detailed reproducibility statement, promising code and configuration files in the supplementary material. It specifies architectural dimensions, hyperparameters, and training schedules in the appendix. The use of standard benchmarks (LIBERO, DOMINO, RMBench) facilitates independent verification. The AI use statement is transparent, clarifying that AI was used only for editing and formatting, not for data generation or hypothesis formulation.
The primary limitation is the reliance on the $\pi_{0.5}$ backbone; it is unclear how well the PREM module generalizes to other VLA architectures or non-recurrent memory structures. The evaluation is limited to simulation and specific real-world setups (implied by the benchmarks), and long-horizon real-world deployment challenges (e.g., lighting changes, object deformations) are not extensively explored. The "prediction" mechanism assumes a certain degree of predictability in the environment, which may not hold in highly stochastic or adversarial settings.
This work contributes to the broader goal of creating robots that can learn from experience and maintain context over time. The technique of using prediction errors to regulate memory updates is a concept that could be adapted to other sequential decision-making tasks beyond robotics, such as autonomous driving or interactive agents. It addresses a fundamental challenge in embodied AI: how to efficiently compress and update historical information for future action planning. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces a novel memory regulation mechanism for VLA policies that leverages action-conditioned future prediction to improve temporal context retention, demonstrating significant performance gains on memory-dependent and dynamic manipulation benchmarks with minimal computational overhead.
Long-horizon embodied manipulation requires policies to act on information that is no longer visible. Modified objects may leave the field of view, while completed subtasks continue to constrain future actions. Existing approaches extend temporal context through recurrent state updates, test-time training, or storage and retrieval of past observations. As horizons grow, these mechanisms can still leave the policy to infer task state from long, redundant, and viewpoint-dependent visual experience. We argue that long-horizon agents benefit not merely from remembering more, but from remembering task-relevant state changes and progress. We propose Memory of Object States and Task Progress (MOST), a policy-internal memory framework that compresses multi-view observations into temporal tokens. Historical and predictive future tokens are explicitly supervised with object states and subtask progress, structuring memory around what has changed in the world and what has already been accomplished. The resulting memory directly conditions action generation within a single end-to-end policy, without an external planner or retrieval stage. We further introduce RoboCasa-Long, a benchmark of multi-minute, multi-stage mobile manipulation tasks. Across RoboCasa365, RoboCasa-Long, and real-world robot experiments, MOST consistently outperforms Xiaomi-Robotics-1 (XR-1), the previous state-of-the-art model. It raises average success from 65.9% to 72.1% on RoboCasa365, average maximum task progress from 20.83% to 42.22% on RoboCasa-Long, and real-world success from 6.66% to 43.33%. MOST also outperforms a matched visual-history memory variant, showing that effective long-horizon memory depends not only on how much experience is retained, but on whether it preserves decision-relevant progress and world-state changes.
Primary: Unknown
All Institutions: Unknown
[One sentence main contribution]. The paper introduces MOST, a policy-internal memory framework that explicitly supervises temporal tokens with object states and task progress, significantly improving long-horizon mobile manipulation performance across simulation and real-world benchmarks. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The technical contribution is substantial, offering a principled approach to memory in embodied AI that shifts focus from raw data retention to semantic state tracking. The methodology is sound, combining architectural innovation with rigorous evaluation on both existing and new benchmarks. The significance to the field is high, as it addresses a critical bottleneck in long-horizon task execution and provides a new evaluation standard (RoboCasa-Long) that will likely influence future research directions in robotics and embodied learning.
The paper proposes MOST (Memory of Object States and Task Progress), a policy-internal memory framework for long-horizon mobile manipulation. The core innovation is the explicit supervision of temporal tokens with object states and subtask progress, rather than relying solely on recurrent state updates or raw visual history retrieval. The method compresses multi-view observations into tokens that are structured around "what has changed" and "what has been accomplished." This is a logical and well-motivated extension of current embodied AI techniques, moving from passive memory to active, task-relevant state tracking. The integration within a single end-to-end policy without external planners is a significant architectural contribution that simplifies the deployment pipeline compared to hierarchical approaches.
The experimental setup is robust, utilizing the RoboCasa365 benchmark, a new proposed benchmark (RoboCasa-Long), and real-world robot experiments. The results show consistent improvements over the previous state-of-the-art (XR-1), with notable gains in real-world success rates (6.66% to 43.33%). The introduction of RoboCasa-Long, which focuses on multi-minute, multi-stage tasks, is a valuable contribution to the field, addressing the gap in long-horizon evaluation. The comparison against a "matched visual-history memory variant" effectively isolates the benefit of the proposed state/progress supervision, providing strong causal evidence for the method's effectiveness.
The paper includes a reproducibility statement indicating that implementation details, benchmark construction, and evaluation protocols are in the appendix. However, the provided text does not contain the actual appendix content or code links. The reliance on a new benchmark (RoboCasa-Long) may limit immediate reproducibility if the assets are not widely available, though the use of RoboCasa365 provides a standard baseline.
The primary limitation is the lack of explicit code or dataset links in the provided text, which hinders immediate verification. Additionally, the method's reliance on explicit supervision for object states and subtask progress may require careful annotation or labeling strategies that could be costly to scale to new domains. The performance on real-world tasks, while improved, is still relatively low (43.33%), suggesting significant challenges remain in sim-to-real transfer for long-horizon tasks.
This work has significant implications for the development of autonomous mobile manipulators in unstructured environments. By improving the agent's ability to track task progress and world state changes, it moves closer to general-purpose robotic assistants. The new benchmark will likely become a standard for evaluating long-horizon manipulation capabilities, driving future research in memory mechanisms for embodied agents. [One sentence main contribution]. The paper introduces MOST, a policy-internal memory framework that explicitly supervises temporal tokens with object states and task progress, significantly improving long-horizon mobile manipulation performance across simulation and real-world benchmarks. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The technical contribution is substantial, offering a principled approach to memory in embodied AI that shifts focus from raw data retention to semantic state tracking. The methodology is sound, combining architectural innovation with rigorous evaluation on both existing and new benchmarks. The significance to the field is high, as it addresses a critical bottleneck in long-horizon task execution and provides a new evaluation standard (RoboCasa-Long) that will likely influence future research directions in robotics and embodied learning.
Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy's training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.
Primary: Stanford University
All Institutions: Stanford University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces RoboArchitect, a method that automates the design of agent harnesses around frozen robot policies using iterative coding LLM optimization, demonstrating that substantial performance gains in robotic manipulation can be achieved without weight updates by evolving control logic and persistent memory, effectively turning general policies into specialists for specific setups.
The paper proposes "RoboArchitect," a framework that automates the design of an "agent harness" surrounding a frozen Vision-Language-Action (VLA) policy. The core methodology involves an iterative optimization loop where a coding LLM (Claude Opus 5) observes robot rollouts, diagnoses failure modes, and rewrites the control code (harness) to mediate between the environment and the frozen policy. The harness is not a fixed architecture but evolves to include specific control routines (e.g., jam recovery, force rescaling) and persistent memory structures. The approach is distinct from standard fine-tuning as it updates no model weights, relying instead on programmatic logic and external tool calls (VLMs, detectors). The novelty lies in treating the agent's control logic as an evolvable software artifact rather than a static wrapper, allowing the system to dynamically decide when to trust the policy vs. direct control.
The experiments are extensive, covering multiple benchmarks (LIBERO-Pro, RoboCerebra, VLABench, RoboCasa, SimplerEnv) and multiple policies (OpenVLA, Pi-0, MolmoAct-2, etc.). The evaluation protocols are rigorous, distinguishing between "unseen scenes" (same tasks, new layouts) and "unseen tasks" (new tasks, shared skills). The results show significant improvements (e.g., 9.7% to 43.3% on unseen scenes), and notably, the evolved harness often outperforms the fine-tuned version of the same policy in out-of-domain settings. The ablation on iterative refinement vs. one-shot design confirms the value of the self-improvement loop. The emergence of persistent memory (e.g., storing grasp depths) is a strong qualitative and quantitative finding.
The paper provides a GitHub link and detailed implementation sections. It specifies the exact coding model (Claude Opus 5), auxiliary tools (Qwen3, Molmo2, Grounding-DINO, SAM 3), and iteration limits. The cost analysis (<$150, <6 hours) is a strong point for practical reproducibility. However, the heavy reliance on a specific, proprietary coding model (Claude) may limit reproducibility for labs without access to that specific model version or API.
The primary limitation is that all evaluations are in simulation. The paper acknowledges this and suggests the interface should transfer to real robots, but no real-world validation is provided. Additionally, the method is capped at 5 iterations, and the saturation point is not fully explored. The complexity of the harness code could become a maintenance burden, though the paper argues it is interpretable. The reliance on a powerful coding LLM makes the method expensive in terms of API costs, though the paper argues this is cheaper than collecting new demonstrations.
This work has high potential impact by offering an alternative to the data-hungry fine-tuning paradigm in robotics. If the "harness optimization" approach scales to real-world robots, it could drastically reduce the cost of adapting robot policies to new environments. It bridges the gap between code-as-policy and learned VLA policies, suggesting a hybrid future where LLMs write the control logic around frozen neural networks. The concept of "evolving" the agent's software interface is a significant shift in how robotic agents are deployed. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces RoboArchitect, a method that automates the design of agent harnesses around frozen robot policies using iterative coding LLM optimization, demonstrating that substantial performance gains in robotic manipulation can be achieved without weight updates by evolving control logic and persistent memory, effectively turning general policies into specialists for specific setups.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
Future planetary rovers will traverse steep, deformable terrain where slip, sinkage, and terrain deformation can threaten mobility. Although simulation can capture some wheel-soil effects, unmodeled deformable-terrain phenomena create sim-to-real gaps requiring deployment-time adaptation. Yet, rover operations leave little room for extensive trial-and-error. We introduce TERRA (Targeted Exploration for Reward-Relevant Adaptation), an online adaptation framework that efficiently seeks deployment data exposing task-relevant model errors and corrects them. TERRA adapts a simulation-trained world model with a reward-weighted prediction loss and trains a loss predictor to estimate where future performance-relevant errors are likely to occur, using this signal to guide exploration during MPC. In simulation, TERRA reaches approximately 90% slope-climbing success within five trials under severe out-of-distribution pretraining conditions, outperforming exploration baselines in reward, completion time, and energy to first success. On the ERNEST rover, TERRA recovers coordinated active-suspension locomotion on a deformable Mars-simulant slope within three to five real-world trials. Project website: https://terra-explore.github.io
Primary: Jet Propulsion Laboratory, California Institute of Technology
All Institutions: Jet Propulsion Laboratory, California Institute of Technology, University of Texas at Austin
TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
The paper introduces TERRA, a framework for online adaptation of simulation-trained world models for planetary rovers. The core methodological contributions are twofold: (1) a reward-weighted prediction loss that prioritizes model errors in observation dimensions that have high sensitivity to the task reward, and (2) a learned "loss predictor" that forecasts where these reward-relevant errors are likely to occur in candidate action sequences, thereby guiding exploration within a Model Predictive Control (MPC) loop. The use of Low-Rank Adaptation (LoRA) for efficient onboard updating of the dynamics model is a practical engineering choice. The approach is logically sound, addressing the specific bottleneck of sim-to-real gaps in deformable terrain where standard prediction-error-based exploration is inefficient.
The evaluation is strong, featuring both extensive simulation studies (100 seeds, multiple OOD pretraining conditions) and real-world validation on the ERNEST rover. The simulation results show significant improvements over baselines like RND and Plan2Explore in terms of success rate, completion time, and energy efficiency. The real-world results are particularly compelling, demonstrating the recovery of coordinated active-suspension locomotion within 3-5 trials on a Mars-simulant slope, a task that is critical for future missions. The ablation studies effectively isolate the contributions of the reward-weighted loss and the targeted exploration.
The paper provides detailed descriptions of the simulation environment, pretraining conditions, and hyperparameters. However, as is common in robotics papers involving proprietary hardware (ERNEST) and specific simulators (DARTS), full reproducibility by external groups may be limited. The project website is provided, which likely contains code or further details.
The method relies on the availability of a task reward during deployment, which may not always be easily defined or observable in complex real-world scenarios. The choice of loss-bin thresholds for the loss predictor is a hyperparameter that requires tuning. The evaluation is limited to a single rover platform (ERNEST) and a specific type of terrain (deformable slopes), so generalizability to other platforms or terrain types is not fully established.
This work has high relevance to the field of planetary robotics and autonomous systems operating in unstructured environments. The technique of reward-weighted model adaptation and targeted exploration is applicable beyond rovers to any mobile robot facing sim-to-real gaps in contact-rich or deformable environments. It provides a practical pathway for deploying learned policies in high-stakes, low-trial-count scenarios. TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
Discretizing continuous actions into discrete skills using methods like VQ-VAE has emerged as a powerful paradigm for robotic manipulation. However, quantization errors yield a suboptimal distribution for the skill prior, degrading its performance. While reinforcement learning offers a path for refinement, its direct application faces unstable encoder updates and a granularity dilemma in importance sampling. To address these challenges, we introduce Cascaded Skills Optimization (CSO), a two-stage post-training framework. First, Rejection-Sampling Supervised Fine-tuning aligns the model's observation-to-skill mapping with the distribution of successful online trajectories via supervised fine-tuning. Second, with the observation encoder frozen, Skills Policy Optimization computes an independent, clipped importance ratio for each complete skill, enabling stable and efficient updates. With a lightweight QueST-style policy, CSO raises overall LIBERO success from 80.1% to 97.5%, improves performance on RoboTwin2.0, and achieves 61.0% success across five physical single-arm tasks. Additional comparisons with FAST, discrete BEAST, and fixed-buffer IQL further validate the effectiveness of our two-stage post-training strategy.
Primary: Unknown
All Institutions: Unknown
[One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
The paper proposes CSO (Cascaded Skills Optimization), a two-stage post-training framework for refining skill-based robotic policies. The methodology is logically sound and addresses specific pain points in applying RL to discrete skill policies. Stage 1 (R-SFT) uses rejection sampling to collect successful trajectories and fine-tunes the observation encoder, which is then frozen. This is a pragmatic solution to the instability of updating visual encoders with sparse rewards. Stage 2 (SPO) introduces a skill-level importance sampling ratio, arguing that token-level (too fine) and episode-level (too coarse) granularities are suboptimal for robotic skills. The theoretical justification for the "granularity dilemma" is the core intellectual contribution. The approach is essentially a structured application of PPO variants (specifically GSPO/GRPO concepts) adapted to the hierarchical structure of skill-based policies. While the adaptation is clever, the underlying RL machinery is not new.
The experimental evaluation is rigorous and comprehensive. The authors test on standard simulation benchmarks (LIBERO, RoboTwin2.0, MetaWorld) and, crucially, on physical robots (Cobot Agilex ALOHA). The results show significant improvements over the imitation learning baseline (80.1% to 97.5% on LIBERO) and outperform strong baselines like RIPT and FAST. The ablation studies effectively isolate the contributions of R-SFT and SPO, and the granularity analysis convincingly demonstrates the superiority of skill-level optimization over token- and episode-level alternatives. The physical robot experiments, while limited to five tasks, provide valuable validation of the method's real-world applicability.
The paper provides detailed implementation specifics, including hyperparameters for both stages, architecture details for the lightweight and VLM-based variants, and data collection protocols. The use of standard benchmarks and clear descriptions of the training process enhances reproducibility. However, the lack of a public code repository link in the provided text is a minor drawback, though the detailed appendices mitigate this.
The method requires a sufficient number of successful trajectories to bootstrap the R-SFT stage, which may be difficult for very hard tasks without a strong initial policy. The frozen encoder in Stage 2 limits the model's ability to adapt its perception to new failure modes discovered during RL. The physical experiments are limited to single-arm tasks, and the method's scalability to more complex, multi-arm, or dexterous manipulation scenarios is not fully explored.
This paper contributes to the growing field of post-training for robotic policies, offering a practical and effective recipe for improving skill-based agents. The insight regarding optimization granularity is likely to influence future work in hierarchical RL and skill-based control. The framework is general enough to be applied to other skill-based architectures, making it a useful tool for the robotics community. [One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.