Week of October 25 – November 01, 2026
While progress in informal mathematics has come largely from scaling the chain-of-thought and agentic reasoning of frontier language models, formal theorem proving has increasingly relied on human-designed workflows around these models, such as planning, decomposition, and orchestrating specialized provers. This raises a bitter-lesson question: can a simple agentic loop that scales test-time compute match these workflows? The answer matters as formal verification must scale to keep up with frontier models producing research mathematics, yet new workflows rarely settle it, as their gains are often confounded with stronger models or more compute. We study this question with h1, a simple harness that gives a model an efficient interface to Lean 4 and keeps it working through verifier feedback and self-reflection, without imposing a decomposition or search structure. h1 reaches state-of-the-art results on PutnamBench, FATE-X, and FormalConjectures100-Solved, settles eight open problems in FormalConjectures, and, at matched models, matches or outperforms specialized workflows such as Goedel-Architect and AlphaProof Nexus, surprisingly showing that a single attempt (pass@1) can often scale to solve open problems. Through ablations and matched-model comparisons with Claude Code and AxProverBase, we find that simplicity alone is not enough: an efficient Lean interface largely determines cost and solve rate. We thus recommend that future work evaluate frontier models with efficient execution feedback before prescribing workflows, at the risk of bitter lessons.
Primary: Google DeepMind
All Institutions: Google DeepMind, University of Edinburgh
The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
The paper proposes "h1," a minimalist agentic harness for Lean 4 theorem proving that relies on a simple ReAct-style loop with an efficient interface to the Lean compiler, library search tools, and a reflection mechanism to compact context. The core methodological contribution is not a new model architecture or training algorithm, but a rigorous empirical demonstration that "bitter lesson" principles apply to harness design: that scaling test-time compute in a simple loop with good tooling outperforms complex, human-designed workflows (like planning, decomposition, or multi-agent orchestration) when using the same frontier models. The methodology is sound, focusing on isolating the variable of harness complexity while controlling for model capability and inference cost.
The experimental evaluation is extensive and high-quality. It covers a wide range of benchmarks including PutnamBench, FATE-X, FormalConjectures, and IMO 2026. The paper provides matched-model comparisons against state-of-the-art specialized systems (Goedel-Architect, AlphaProof Nexus, MerLean, etc.), which is crucial for a fair assessment. The results show that h1 achieves state-of-the-art performance, solving open problems and matching or exceeding specialized workflows at lower cost. The analysis of cost-efficiency curves (pass@1 vs pass@k) is particularly insightful, showing that longer single attempts can be more efficient than multiple shorter ones.
The paper provides detailed appendices on tools, prompts, error handling, and cost accounting. It specifies the exact versions of Lean and Mathlib used, and provides a link to the pull request documenting the newly solved conjectures. While the specific code for the h1 harness is not explicitly linked in the provided text (only the conjectures PR), the level of detail in the prompts and tool descriptions suggests high reproducibility for researchers with access to the same frontier models.
The primary limitation is the reliance on proprietary frontier models (GPT-5.6, Opus 4.8, etc.), which are black boxes and subject to rapid change, making long-term reproducibility difficult. The paper also acknowledges that it does not address the challenge of formalizing proofs that exceed the context window or require parallel agents for very large projects, though it suggests this is a natural next step. The cost analysis relies on API pricing, which may not reflect the true compute cost for self-hosted models.
This paper has significant broader impact by challenging the prevailing trend in automated theorem proving towards increasingly complex, human-designed workflows. It provides strong evidence that as LLMs improve, the focus should shift to building efficient interfaces and allowing models to scale their own reasoning, rather than imposing rigid structures. This insight is likely to influence the design of agents in other structured domains, such as software verification and formal methods. The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
Accurate forecasting with uncertainty quantification is critical in a variety of scientific applications ranging from epidemiology and economics to energy systems and earth sciences. Yet most forecasting models still need to be developed and trained separately for each domain-specific dataset. Time series foundation models (TSFMs) aim to remove this limitation and enable the application of the same model across domains through large-scale pretraining. However, most existing TSFMs treat each time series independently and therefore cannot model interactions among time series and external drivers. Here, we introduce Chronos-2, a foundation model for zero-shot forecasting that models the interactions between multivariate time series and heterogeneous covariates, which commonly arise in scientific applications. The model infers relationships among time series directly from historical context and generates probabilistic forecasts without task-specific treatment. Across various benchmarks, Chronos-2 achieves state-of-the-art accuracy on average by improving skill scores by 5.2 percentage points relative to the strongest baselines and increasing win rates by 10 percentage points. Chronos-2 is also computationally efficient, generating forecasts for 100 time series in under one second on a single mid-range GPU, 50x and 1000x faster end-to-end than the best task-specific point and probabilistic forecasters, respectively. In addition, we demonstrate its effectiveness in diverse scientific settings -- including infectious-disease forecasting, macroeconomic modeling, carbon-flux prediction, and runoff simulation -- where the model matches or surpasses specialized methods developed for each domain. These results show that a single pretrained model can capture structure shared across temporal systems.
Primary: Amazon
All Institutions: Amazon, AWS
Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
The paper introduces Chronos-2, a 120M parameter encoder-only Transformer foundation model for time series forecasting. The core methodological contribution is the "group attention" mechanism, which allows the model to infer dependencies among multivariate time series and heterogeneous covariates directly from historical context (in-context learning) without task-specific training. The architecture alternates between temporal attention (within a series) and group attention (across related series). A significant portion of the methodology focuses on the synthetic data generation pipeline, which creates diverse multivariate dynamics by coupling univariate generators (Gaussian Processes, AR, ETS) through linear (VAR, mixing matrices) and non-linear (MLP, RNN) transformations. This approach addresses the scarcity of large-scale multivariate time series data for pretraining. The model uses a quantile regression head to provide probabilistic forecasts, supporting uncertainty quantification.
The evaluation is extensive and rigorous, covering both standard benchmarks and domain-specific case studies. On the FEV benchmark (100 tasks), Chronos-2 achieves state-of-the-art results, outperforming strong baselines like TiRex, TimesFM, and Moirai. It demonstrates significant gains in win rate and skill score over univariate inference, validating the group attention mechanism. The paper includes four detailed case studies: epidemiology (ECDC RespiCast), macroeconomics (German CPI), earth science (FLUXNET carbon flux), and hydrology (CAMELS-DE rainfall-runoff). In each domain, the model matches or surpasses specialized, task-specific models (e.g., HBV, LSTM ensembles, XGBoost) in zero-shot settings. The inclusion of hierarchical forecasting experiments further demonstrates the model's ability to capture cross-level dependencies without explicit hierarchy information. The computational efficiency is also highlighted, with inference times under 1 second for 100 series on a mid-range GPU.
The paper provides detailed architectural specifications, training hyperparameters, and data generation procedures. The model size (120M parameters) and training setup (4x A100 GPUs) are clearly stated. The synthetic data generation process is described in detail, allowing for potential replication of the pretraining corpus. However, the specific code repository or model weights are not explicitly linked in the provided text (though likely available given the Amazon affiliation and prior Chronos releases). The evaluation protocols for the case studies are well-defined, referencing standard domain metrics (WIS, NSE, etc.).
The model is limited to 120M parameters, which is smaller than some competing foundation models (e.g., Moirai, TimesFM), potentially limiting its capacity for extremely complex dynamics. The reliance on synthetic data for multivariate pretraining may introduce a domain gap if real-world interactions are more complex than the synthesized ones. The paper does not extensively discuss the interpretability of the inferred relationships, which is a known challenge in foundation models. Additionally, the model does not natively support static metadata (e.g., soil type, demographics), which is a limitation for certain scientific applications like ungauged basin prediction, though the authors acknowledge this as future work.
This paper has high potential impact by democratizing advanced forecasting capabilities. By providing a single, efficient, zero-shot model that handles multivariate interactions, it reduces the barrier to entry for applying state-of-the-art forecasting in scientific and industrial domains. The demonstrated success in diverse fields (epidemiology, economics, earth sciences) suggests broad applicability. The efficiency of the model (running on consumer GPUs/CPU) makes it accessible for edge deployment or resource-constrained environments. It shifts the paradigm from task-specific model fitting to contextual inference, which could streamline workflows in data science teams. Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
We study probabilistic forecasting of dynamical systems such as fluids and waves from coarse observations. Since the unseen fine scales influence the dynamics, the same coarse initial state can evolve into a whole distribution of futures. Generative forecasters may produce visually plausible samples, but it is hard to tell whether they sample the right conditional distribution, since trajectory data contain a single future for each initial state. We address this problem by generating reference data in a hierarchical way: for each coarse initial state, we generate many different fine-scale components and numerically evolve the full states. Using these conditional ensembles, we show that standard diffusion models and stochastic interpolants make substantial distributional errors. We then show that cascading shorter forecasts greatly reduces these errors for the turbulent flows and waves we study, but we also find and characterize cases in which it hurts. Since the coarse process is not Markov, cascading discards information about the unresolved fine scales even with exact transitions. To understand this, we propose a solvable analytic model which shows that the optimal cascade depth decreases as the coupling between the unresolved and observed scales gets stronger, and increases as the dependence of the forecast on the observation becomes more oscillatory. Consistent with our experiments, cascading is detrimental when the direct forecast is ``easy'' to learn, for example Gaussian as in the passive scalar case. We release our datasets as a first benchmark for probabilistic forecasting with ground-truth conditional ensembles.
Primary: University of Basel
All Institutions: University of Basel, PSL University
The paper provides a rigorous theoretical and experimental analysis of when cascading generative forecasters improves probabilistic accuracy, introducing a novel benchmark for evaluating conditional distributions in dynamical systems.
The paper introduces a rigorous experimental framework for evaluating probabilistic forecasters by generating "reference ensembles" via hierarchical data generation. This allows for the direct comparison of the learned conditional distribution $P(X_T|X_0)$ against the true conditional distribution, addressing a critical gap where standard trajectory data only provides single futures. The theoretical contribution is significant, proposing a solvable analytic model (rotation driven by a hidden Ornstein-Uhlenbeck process) that isolates the trade-off between approximation error (favoring shorter transitions) and memory loss (favoring direct forecasts due to non-Markovian coarse-graining). The authors prove that an optimal finite cascade depth exists under specific conditions, providing a theoretical foundation for why cascading helps in some systems (turbulent flows) but hurts in others (passive scalars).
The experiments are extensive, covering five distinct physical systems (Euler, SQG, Heterogeneous Waves, Klein-Gordon, Passive Scalar) to test the generality of the findings. The use of spatial FID and conditional energy distance on large reference ensembles (10,000 samples per test observation) provides statistically robust evaluation. The results clearly demonstrate that direct diffusion models and stochastic interpolants fail to capture the correct conditional distribution despite producing visually plausible samples, while cascaded SIs recover the distribution in nonlinear systems. The finding that cascading is detrimental for the passive scalar case is a strong, counter-intuitive result supported by the theory.
The paper provides detailed descriptions of the numerical solvers, initial conditions, and model architectures. The release of the datasets as a benchmark is a major contribution to reproducibility. However, the text provided does not explicitly list a GitHub URL, though it mentions releasing datasets. The hyperparameters and training protocols are described in sufficient detail for replication.
The study is limited to 2D periodic domains and specific PDEs. The "fixed budget" constraint (equalizing the number of supervised pairs) is a harsh constraint that may not reflect all practical scenarios where intermediate data is abundant. The theory, while elegant, is based on a simplified linear model and may not fully capture the complexities of highly chaotic 3D turbulence. The inference cost of cascading increases with depth, which is a practical limitation not fully explored in terms of compute-time trade-offs.
This work has high impact on the field of scientific machine learning and probabilistic forecasting. It provides a new benchmark for evaluating generative forecasters and a theoretical understanding of when autoregressive/cascaded approaches are beneficial. This is crucial for weather forecasting and fluid dynamics simulation, where capturing the correct uncertainty distribution is as important as the mean forecast. The insights into the non-Markovian nature of coarse-grained processes will influence how future models handle partial observations. The paper provides a rigorous theoretical and experimental analysis of when cascading generative forecasters improves probabilistic accuracy, introducing a novel benchmark for evaluating conditional distributions in dynamical systems.
Reasoning-oriented post-training enables large language models (LLMs) to solve complex tasks via multi-step inference. Supervised fine-tuning (SFT) on high-quality reasoning traces is a particularly efficient approach, but its effectiveness depends critically on careful data curation. Existing pipelines rely on a costly generate-then-filter paradigm: large pools of long reasoning traces are produced by multiple teacher models and subsequently filtered for difficulty and diversity using additional LLMs, often requiring weeks of computation. In this work, we propose SFTProbe, a fundamentally different approach that bypasses this pipeline by directly predicting the quality of reasoning data. We show that a brief LoRA-based adaptation, combined with evaluating loss on only 5% of each reasoning trace, suffices to estimate both difficulty and diversity. SFTProbe is grounded in a geometric view of post-training, where pretrained LLMs lie in a low-rank, anisotropic loss basin. By probing this basin via low-rank perturbations, we construct compact loss signatures whose clustering captures diversity, while loss at curvature inflection points provides a robust proxy for difficulty. We also provide a simple criterion for teacher selection. Empirically, our approach reduces data curation time from weeks to hours while maintaining or improving performance. Remarkably, on mathematical reasoning, 5k generated examples match the highly filtered 89k-example in OpenThoughts-Math dataset, and 7.5k generated examples outperforms it by 2.5% in accuracy. We further introduce a high-quality physics reasoning dataset, and use it to train a model that outperforms an IPhO 2025 silver-medal-winning model. The code and the dataset are released at https://bigml-cs-ucla.github.io/SFTProbe-page/.
Primary: University of California, Los Angeles (UCLA)
All Institutions: University of California, Los Angeles (UCLA)
SFTProbe introduces a principled, geometric approach to SFT data curation that significantly reduces computational costs while maintaining high performance in reasoning tasks. By leveraging low-rank perturbations to estimate data difficulty and diversity, the method offers a scalable alternative to expensive LLM-based filtering, with strong empirical validation on mathematical and physics reasoning benchmarks.
The paper proposes SFTProbe, a method to bypass the standard "generate-then-filter" paradigm for SFT data curation. Instead of using LLMs to judge quality, it utilizes a brief LoRA-based adaptation on a small subset of data to predict the quality of the full dataset. The core theoretical contribution is a geometric view of the loss landscape, arguing that pretrained LLMs reside in a low-rank, anisotropic basin. By probing this basin with low-rank perturbations, the authors construct "loss signatures" where clustering indicates diversity and loss at curvature inflection points indicates difficulty. This is a novel shift from heuristic filtering to a principled, optimization-based estimation of data utility.
The empirical results are strong and directly address the efficiency bottleneck of current pipelines. The claim that 5k generated examples match the performance of 89k highly filtered examples from OpenThoughts-Math is significant, suggesting a massive reduction in compute requirements for data generation. Furthermore, the introduction of a high-quality physics reasoning dataset and the training of a model that outperforms an IPhO 2025 silver-medal-winning model provides concrete evidence of the method's effectiveness in complex reasoning domains. The reduction of curation time from weeks to hours is a practical win for the community.
The paper states that code and datasets are released at the provided URL. The methodology relies on standard techniques (LoRA, loss evaluation) combined with specific probing strategies. While the geometric theory is abstract, the practical implementation appears straightforward enough for reproduction given the released artifacts. The specific hyperparameters for the "brief LoRA-based adaptation" and the definition of "5% of each reasoning trace" are critical details that must be present in the code release for full reproducibility.
The primary limitation is the reliance on the geometric assumption of a "low-rank, anisotropic loss basin." If this assumption fails for certain model architectures or domains, the probing method may lose efficacy. Additionally, the comparison is primarily against the OpenThoughts-Math dataset; broader validation across diverse reasoning tasks (e.g., coding, general QA) would strengthen the claim. The "weeks to hours" claim depends heavily on the scale of the original pipeline being compared against, which may vary across labs.
This work has high potential for adoption in the LLM post-training community. As reasoning models become the standard for complex tasks, the cost of data curation is a major bottleneck. A method that reduces this cost by an order of magnitude while maintaining or improving quality is highly valuable. The geometric perspective on loss landscapes for data selection could also inspire further research into the interplay between model optimization and data quality. SFTProbe introduces a principled, geometric approach to SFT data curation that significantly reduces computational costs while maintaining high performance in reasoning tasks. By leveraging low-rank perturbations to estimate data difficulty and diversity, the method offers a scalable alternative to expensive LLM-based filtering, with strong empirical validation on mathematical and physics reasoning benchmarks.
Accurate forecasting with uncertainty quantification is critical in a variety of scientific applications ranging from epidemiology and economics to energy systems and earth sciences. Yet most forecasting models still need to be developed and trained separately for each domain-specific dataset. Time series foundation models (TSFMs) aim to remove this limitation and enable the application of the same model across domains through large-scale pretraining. However, most existing TSFMs treat each time series independently and therefore cannot model interactions among time series and external drivers. Here, we introduce Chronos-2, a foundation model for zero-shot forecasting that models the interactions between multivariate time series and heterogeneous covariates, which commonly arise in scientific applications. The model infers relationships among time series directly from historical context and generates probabilistic forecasts without task-specific treatment. Across various benchmarks, Chronos-2 achieves state-of-the-art accuracy on average by improving skill scores by 5.2 percentage points relative to the strongest baselines and increasing win rates by 10 percentage points. Chronos-2 is also computationally efficient, generating forecasts for 100 time series in under one second on a single mid-range GPU, 50x and 1000x faster end-to-end than the best task-specific point and probabilistic forecasters, respectively. In addition, we demonstrate its effectiveness in diverse scientific settings -- including infectious-disease forecasting, macroeconomic modeling, carbon-flux prediction, and runoff simulation -- where the model matches or surpasses specialized methods developed for each domain. These results show that a single pretrained model can capture structure shared across temporal systems.
Primary: Amazon
All Institutions: Amazon, AWS
Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
The paper introduces Chronos-2, a 120M parameter encoder-only Transformer foundation model for time series forecasting. The core methodological contribution is the "group attention" mechanism, which allows the model to infer dependencies among multivariate time series and heterogeneous covariates directly from historical context (in-context learning) without task-specific training. The architecture alternates between temporal attention (within a series) and group attention (across related series). A significant portion of the methodology focuses on the synthetic data generation pipeline, which creates diverse multivariate dynamics by coupling univariate generators (Gaussian Processes, AR, ETS) through linear (VAR, mixing matrices) and non-linear (MLP, RNN) transformations. This approach addresses the scarcity of large-scale multivariate time series data for pretraining. The model uses a quantile regression head to provide probabilistic forecasts, supporting uncertainty quantification.
The evaluation is extensive and rigorous, covering both standard benchmarks and domain-specific case studies. On the FEV benchmark (100 tasks), Chronos-2 achieves state-of-the-art results, outperforming strong baselines like TiRex, TimesFM, and Moirai. It demonstrates significant gains in win rate and skill score over univariate inference, validating the group attention mechanism. The paper includes four detailed case studies: epidemiology (ECDC RespiCast), macroeconomics (German CPI), earth science (FLUXNET carbon flux), and hydrology (CAMELS-DE rainfall-runoff). In each domain, the model matches or surpasses specialized, task-specific models (e.g., HBV, LSTM ensembles, XGBoost) in zero-shot settings. The inclusion of hierarchical forecasting experiments further demonstrates the model's ability to capture cross-level dependencies without explicit hierarchy information. The computational efficiency is also highlighted, with inference times under 1 second for 100 series on a mid-range GPU.
The paper provides detailed architectural specifications, training hyperparameters, and data generation procedures. The model size (120M parameters) and training setup (4x A100 GPUs) are clearly stated. The synthetic data generation process is described in detail, allowing for potential replication of the pretraining corpus. However, the specific code repository or model weights are not explicitly linked in the provided text (though likely available given the Amazon affiliation and prior Chronos releases). The evaluation protocols for the case studies are well-defined, referencing standard domain metrics (WIS, NSE, etc.).
The model is limited to 120M parameters, which is smaller than some competing foundation models (e.g., Moirai, TimesFM), potentially limiting its capacity for extremely complex dynamics. The reliance on synthetic data for multivariate pretraining may introduce a domain gap if real-world interactions are more complex than the synthesized ones. The paper does not extensively discuss the interpretability of the inferred relationships, which is a known challenge in foundation models. Additionally, the model does not natively support static metadata (e.g., soil type, demographics), which is a limitation for certain scientific applications like ungauged basin prediction, though the authors acknowledge this as future work.
This paper has high potential impact by democratizing advanced forecasting capabilities. By providing a single, efficient, zero-shot model that handles multivariate interactions, it reduces the barrier to entry for applying state-of-the-art forecasting in scientific and industrial domains. The demonstrated success in diverse fields (epidemiology, economics, earth sciences) suggests broad applicability. The efficiency of the model (running on consumer GPUs/CPU) makes it accessible for edge deployment or resource-constrained environments. It shifts the paradigm from task-specific model fitting to contextual inference, which could streamline workflows in data science teams. Chronos-2 introduces a multivariate time series foundation model leveraging group attention and synthetic pretraining to achieve state-of-the-art zero-shot forecasting across diverse scientific domains. The paper demonstrates that a single pretrained model can effectively capture complex multivariate interactions and external drivers without task-specific training, outperforming specialized baselines in epidemiology, macroeconomics, earth science, and hydrology, while maintaining high computational efficiency.
We develop a new algorithmic framework for learning Boolean concept classes under logconcave distributions. Our framework applies to broad families of multi-index models and yields significantly faster robust learners than were previously known in these settings. It extends to partial classification and testable learning. Consequently, we obtain improved learners for several fundamental concept classes, including functions of halfspaces and multiclass linear classifiers, under logconcave marginals. Our starting point is a new boosting framework based on weak partial learners. Classical boosting approaches require often-intractable distribution-free weak learners or yield weak approximation guarantees. We show instead that weak learners for partial classification, where labels may be undefined on part of the domain, suffice for strong agnostic learning. The key challenge is that boosting requires a weak learner that succeeds under conditioned distributions that are not guaranteed to be logconcave. To address this, we develop a recursive moment-based partitioning procedure: given a logconcave distribution $X$ and a conditional version $X'$, we partition the space into polynomially many convex regions so that, on most regions, the low-degree moments of $X$ and $X'$ approximately agree. On each good region, a well-behaved multi-index model correlates with a low-degree polynomial, giving a weak hypothesis. Combining these hypotheses yields the required weak learner. A key ingredient of our approach is an efficient method for detecting moment discrepancy. Specifically, we show how to find a low-dimensional subspace that captures any non-trivial discrepancy between $X$ and $X'$, if one exists. Our approach combines spectral analysis of moment tensors with recent progress on isoperimetry for logconcave measures. This discrepancy-finding procedure may be of independent interest.
Primary: University of Wisconsin-Madison
All Institutions: University of Wisconsin-Madison, University of California, San Diego
The paper develops a new algorithmic framework for robust learning of multi-index models under logconcave distributions using weak partial learners, yielding significantly faster robust learners for fundamental concept classes like halfspaces and multiclass linear classifiers. It introduces a novel boosting approach that overcomes the challenge of non-logconcave conditioned distributions via a recursive moment-based partitioning procedure and an efficient discrepancy finder, providing substantial theoretical improvements over prior state-of-the-art methods.
The paper introduces a novel algorithmic framework for robust learning of multi-index models (MIMs) under logconcave distributions. The core innovation is a boosting framework based on "weak partial learners," which allows for strong agnostic learning guarantees without requiring intractable distribution-free weak learners. A key technical contribution is a recursive moment-based partitioning procedure that handles the challenge of boosting under conditioned distributions that are not logconcave. The authors develop an efficient discrepancy finder using spectral analysis of moment tensors and isoperimetry for logconcave measures to identify low-dimensional subspaces capturing moment discrepancies. This enables the construction of weak hypotheses on convex regions where moments approximately match, which are then combined via boosting. The approach is theoretically rigorous, providing sample and computational complexity bounds that significantly improve upon prior state-of-the-art for learning halfspaces, functions of halfspaces, and multiclass linear classifiers under logconcave marginals.
As a theoretical computer science paper, it does not contain empirical experiments on real-world datasets. The "evaluation" consists of rigorous mathematical proofs establishing the correctness and complexity bounds of the proposed algorithms. The results are validated by comparing the derived bounds against known lower bounds and previous upper bounds in the literature, demonstrating significant improvements in efficiency and robustness.
The paper provides detailed algorithmic descriptions and theoretical proofs. While no code is provided, the algorithms are described in sufficient detail for theoretical replication. The reliance on standard statistical query models and polynomial regression techniques makes the theoretical framework reproducible in a mathematical sense.
The primary limitation is the theoretical nature of the work; there is no empirical validation on real-world data to confirm that the theoretical gains translate to practical performance improvements. The algorithms may have high constant factors or practical overheads not captured by the asymptotic analysis. The focus on logconcave distributions, while broad, excludes other common distributional assumptions.
This work contributes to the foundational understanding of efficient learning under distributional assumptions. It provides new tools (weak partial learners, moment discrepancy finders) that could be applicable to other learning problems beyond MIMs. The results advance the state of the art in robust learning, which is crucial for real-world applications where data is often noisy and distributional assumptions are only approximately true. The paper develops a new algorithmic framework for robust learning of multi-index models under logconcave distributions using weak partial learners, yielding significantly faster robust learners for fundamental concept classes like halfspaces and multiclass linear classifiers. It introduces a novel boosting approach that overcomes the challenge of non-logconcave conditioned distributions via a recursive moment-based partitioning procedure and an efficient discrepancy finder, providing substantial theoretical improvements over prior state-of-the-art methods.
Understanding the training dynamics of deep neural networks in large-scale limits is a question both of foundational and of practical importance. Prior works have obtained such descriptions through dynamical mean field theory (DMFT), yielding a system of non-Markovian dynamics that are challenging to analyze and that obscure the optimization structure of the original dynamics. In this paper, we obtain instead a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the $1/\text{depth}$ branch scaling that allows maximal (local) feature updates. For a model trained with gradient descent (GD) from a random initialization, the limit dynamics is simply GD on a Neural Mean ODE with Gaussian initial weights. Our analysis covers population or empirical risk GD in the joint limit of large width, depth and embedding, input, and output dimensions. We obtain approximation scaling laws that depend on the shape of the ResNet and the covariance spectrum of the data. We also illustrate the flexibility of our approach by discussing Mixture of Expert models, where likewise the limit admits an intuitive (nested) Mean ODE description.
Primary: EPFL
All Institutions: EPFL, University of Oxford
The paper establishes that the large-scale training dynamics of ResNets converge to Gradient Descent on a Gaussian Mean ODE, providing a tractable and structurally faithful limit model. This represents a significant advance in the theoretical analysis of deep learning by simplifying the complex non-Markovian dynamics of DMFT into a standard optimization problem, enabling sharper convergence rates and deeper insights into the role of architecture shape and data covariance.
The paper proposes a rigorous theoretical framework for analyzing the training dynamics of Residual Neural Networks (ResNets) in the joint limit of large depth, width, and embedding dimension. The core innovation is deriving a "Gaussian Mean ODE" limit, which simplifies the complex non-Markovian dynamics typically found in Dynamical Mean Field Theory (DMFT) into a standard Gradient Descent (GD) process on a Neural Mean ODE. The methodology relies on a two-stage limit: first, depth divergence leads to a Mean ODE, and second, embedding dimension divergence with Gaussian initialization leads to a dimension-free Gaussian Mean ODE. The authors provide quantitative convergence rates and prove that the limit dynamics preserve the optimization structure of the original network, a significant improvement over prior DMFT approaches that obscure this structure. The use of Hilbert space formalism allows for generalization to infinite-dimensional input/output spaces.
As a theoretical paper, experimental evaluation is limited to numerical validations of the derived convergence rates and the behavior of the limit model. The paper includes plots demonstrating that the empirical errors match the theoretical bounds (e.g., $O(1/L + D/ML)$) and illustrates the "quasi-smoothness" of the limit objective. While not providing large-scale empirical benchmarks, the numerical experiments effectively validate the theoretical claims and the tightness of the derived bounds.
The paper is highly reproducible in a theoretical sense, with clear definitions, lemmas, and theorems. The code for numerical experiments is likely available (though not explicitly linked in the provided text snippet, standard practice for such papers from this group implies code availability). The mathematical derivations are detailed, allowing other researchers to verify the proofs. The reliance on AI for specific proof steps (GPT-6 Astra) is disclosed, which is a novel aspect of reproducibility/transparency in the research process itself.
The analysis is restricted to ResNets with two-layer perceptron blocks and specific scaling regimes (MLU/1-depth). It does not cover attention mechanisms or more complex architectures directly, although it mentions MoE. The results are for Gradient Descent; extensions to SGD or other optimizers are not covered. The assumption of Gaussian initialization for the sharpest results is a limitation, though they do discuss non-Gaussian cases with weaker bounds.
This work provides a crucial bridge between the practical architecture of ResNets and the theoretical understanding of their training dynamics. By showing that the limit is simply GD on a Mean ODE, it makes the analysis of deep network training significantly more tractable. This could lead to better hyperparameter tuning strategies, understanding of generalization in deep networks, and potentially new optimization algorithms designed specifically for the Mean ODE limit. It contributes to the foundational understanding of why deep networks train well. The paper establishes that the large-scale training dynamics of ResNets converge to Gradient Descent on a Gaussian Mean ODE, providing a tractable and structurally faithful limit model. This represents a significant advance in the theoretical analysis of deep learning by simplifying the complex non-Markovian dynamics of DMFT into a standard optimization problem, enabling sharper convergence rates and deeper insights into the role of architecture shape and data covariance.
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.
Primary: Unknown (Not explicitly stated in provided text)
All Institutions: Unknown
The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
The paper presents a rigorous theoretical framework for the Platonic Representation Hypothesis (PRH) by proving that Joint-Embedding Predictive Architectures (JEPAs) trained on "good views" (multiple observations of the same scene) converge to the same representation up to rotation. The core novelty lies in distinguishing "good views" (cross-observation positives) from "bad views" (corruptions of a single observation) and formally linking next-token prediction to the latter. The theoretical proofs rely on spectral analysis of the positive-pair kernel, establishing conditions under which cross-modal alignment is guaranteed. The methodology is sound, leveraging established tools from contrastive learning theory (Laplacian eigenmaps, spectral gaps) but applying them in a novel cross-modal context. The extension to show that next-token prediction is a "bad-view" JEPA is a clever and insightful reformulation that explains the inefficiency of LLMs in achieving cross-modal alignment compared to vision models.
The experimental validation is extensive and convincing. The authors validate the theory on Flickr30k, COCO, and large-scale web image datasets (Recap-DataComp). Key results include: (1) Good-view JEPAs achieve significantly higher cross-modal alignment (mutual 10-NN) than bad-view JEPAs and next-token prediction on identical data. (2) A 109M-parameter text JEPA trained on good views outperforms language models up to 72B parameters in alignment with DINOv2 and CLIP image encoders. (3) The paper demonstrates a 42x sample efficiency gain for good views over next-token prediction to reach equivalent alignment. (4) A practical application is shown where a single rotation matrix aligns frozen DINOv2 and text JEPA encoders to enable zero-shot image captioning (CIDEr .838 on COCO) and zero-shot classification (64% on ImageNet-100), outperforming larger LLMs. The scaling law derived for alignment is predictive of unseen runs within seed noise, adding significant empirical weight to the theoretical claims.
The paper provides detailed hyperparameters, dataset splits, and training protocols. The use of standard architectures (ViT, BERT) and objectives (LeJEPA) enhances reproducibility. The code for the rotation fitting and captioning demo is implied to be available, though specific URLs are not present in the text. The scaling law parameters are provided, allowing others to predict alignment for their own setups.
The theoretical proofs assume a finite world and injective renderings, which are idealizations. The "good view" definition requires multiple distinct observations of the same scene, which can be data-hungry for images (e.g., video frames) compared to text (multiple captions). The paper acknowledges that image JEPAs trained on good views (video frames) are currently weaker than DINOv2 due to data scale, leaving the image-side scaling of good views as future work. The captioning results rely on a pretrained LLM decoder, so the "zero-shot" claim applies to the alignment/rotation, not the language generation capability itself.
This paper has high potential impact by providing a theoretical recipe for improving cross-modal alignment. It suggests that current practices of scaling LLMs for multimodal tasks are inefficient and that focusing on "good views" (multi-observation positives) is a more effective lever. The demonstration of alignment via simple rotations opens new avenues for lightweight multimodal integration, potentially reducing the need for massive paired datasets or joint training. The insights into why LLMs align poorly with vision models (bad views) are valuable for the design of future multimodal foundation models. The paper proves the Platonic Representation Hypothesis for JEPAs with good views, demonstrating that cross-modal alignment is a property of the positive-pair distribution rather than just scale, and validates this with a 109M model outperforming 72B LLMs in alignment and enabling efficient zero-shot captioning.
AI has made remarkable progress in mathematics, solving challenging problems with machine-checkable proofs in Lean, yet typically relying on predefined mathematical definitions and formal specifications. This raises a natural question: can it reason rigorously about an unknown world? We introduce the Algebraic World Model (AWM), a framework that turns interaction with unfamiliar environments into explicit mathematical models supporting formal reasoning. AWM explores the environment and develops executable Python hypotheses, continually testing and revising them against observations. It then structures these hypotheses into typed objects and shared rules, simplifies their structure through category-theory-guided transformations, and formalizes the resulting model in Lean 4 to prove reachability and impossibility. AWM can formally prove that a goal is unreachable, even beyond explored trajectories. On two impossible game levels, it constructs closed, kernel-checked Lean proofs excluding every legal winning sequence in all six runs, while Claude Opus 5.5 and GPT-6 Astra, equipped with coding tools, produce none in twelve baseline runs. Four of AWM's six initial bounds are correct, compared with one for Opus 5.5 and none for GPT-6 Astra; given only feedback that a cheaper win exists, AWM revises its rules and constructs new proofs, reaching six of six correct bounds. Moreover, AWM produces interpretable mathematical models whose rules can be inspected and tested step by step. Across all 25 public ARC-AGI-3 games, these models enable Lean to replay the winning trajectories of all 183 levels and generalize beyond observed play, predicting unseen transitions and detecting changes in game mechanics. With Claude Opus 5, AWM clears all levels with a score of 99.9, while its certified winning trajectories use only 43% of the human baseline's actions.
Primary: unknown
All Institutions: unknown
The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
The paper introduces the Algebraic World Model (AWM), a framework that bridges empirical exploration and formal verification. The methodology is structured in three stages: (1) Explore & Prototype, where an LLM agent interacts with an environment to generate executable Python hypotheses; (2) Structure & Simplify, where these hypotheses are organized into typed objects and rules using category-theory-guided transformations (limits/colimits) verified in Lean; and (3) Formalize & Prove, where the model is formalized in Lean 4 to prove reachability or impossibility. The use of category theory to guide the simplification of world models is a creative and rigorous approach to ensuring that the learned model preserves the logical structure of the environment. The "sufficient cover" definition provides a theoretical basis for transferring proofs from the learned model to the actual environment, which is a significant methodological contribution.
The experiments are conducted on ARC-AGI-3, a challenging benchmark for abstract reasoning. The results are strong: AWM achieves a score of 99.9, outperforming baselines like Claude Opus 5.5 and GPT-6 Astra in both game clearing and, crucially, in constructing formal proofs of impossibility. The paper demonstrates that AWM can produce kernel-checked Lean proofs that exclude all legal winning sequences for impossible levels, a capability the baselines lack. The generalization tests, where frozen models predict unseen transitions and detect mechanic changes, further validate the robustness of the learned algebraic structures.
The paper provides a high level of reproducibility by releasing an interactive website with models, tests, Lean proofs, and links to official replays. The inclusion of the Lean model for the running example (lp85) in the supplementary materials, along with scripts for building and auditing axioms, allows for independent verification of the formal proofs. The explicit statement that all numbers come from script outputs over recorded runs, rather than model-generated text, enhances trust in the results.
The primary limitation is the scope of the environments tested. AWM is demonstrated on ARC-like environments with discrete actions and immediate, observable feedback. The authors acknowledge that richer hidden states, delayed feedback, and stochastic dynamics remain untested. Additionally, the reliance on LLMs for the initial hypothesis generation and exploration phase means that the system's performance is bounded by the capabilities of the underlying LLM. The "sufficient cover" condition, while theoretically sound, requires careful verification in practice to ensure that the learned model indeed covers the optimal winning sequences of the environment.
This work has significant implications for the field of AI, particularly in the areas of formal methods, automated theorem proving, and agent-based reasoning. By enabling AI systems to reason rigorously about unknown worlds and produce machine-checkable proofs, AWM paves the way for more trustworthy and interpretable AI systems. The integration of category theory with machine learning offers a new paradigm for structuring and simplifying learned models, which could be applied to other domains requiring rigorous reasoning. The ability to prove impossibility, not just reachability, is a crucial step towards AI systems that can provide complete and reliable solutions to complex problems. The paper presents a novel framework, AWM, that enables AI agents to learn algebraic world models from interaction and use them to construct formal proofs of reachability and impossibility in Lean 4. By combining empirical exploration with category-theory-guided model simplification and kernel-checked formal verification, AWM achieves state-of-the-art performance on ARC-AGI-3 and demonstrates a unique capability to provide rigorous, interpretable reasoning about unknown environments, marking a significant advancement in the integration of machine learning and formal methods.
Graph-based nearest neighbor search algorithms like HNSW, DiskANN, NSG, and others have surged in popularity in recent years due to their excellent empirical performance in comparison to space-partitioning methods like IVF data structures and Locality Sensitive Hashing (LSH). However, in contrast to LSH and related methods, obtaining strong worst-case theoretical guarantees for graph-based search has proven challenging. In this paper, we prove a strong lower bound that helps explain the lack of progress towards such guarantees. We show that no standard graph-based method (any that runs a variant of greedy search on a flat or hierarchical graph) can achieve sublinear query time for approximate nearest neighbor search in Euclidean space, even for an arbitrarily large approximation factor. On the other hand, we prove that sublinear query time is possible if we allow for Steiner points in the search graph. We obtain bounds comparable to LSH, showing e.g., that graph-based search with Steiner points can solve $c$-approximate nearest neighbor search in Euclidean space with query time $\tilde{O}(n^{1/(c-1)})$. We complement this result with initial experiments exploring how Steiner points can be incorporated into practical graph-based methods.
Primary: New York University
All Institutions: New York University, UMass Amherst
The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
The paper presents a rigorous theoretical framework for analyzing graph-based nearest neighbor search (NNS) algorithms like HNSW and DiskANN. The core methodological contribution is the formalization of a "flat search graph" model that captures generalized greedy search, allowing for the derivation of strong lower bounds. The authors prove that without Steiner points, no such method can achieve sublinear query time in worst-case Euclidean high-dimensional settings, effectively explaining the long-standing gap between empirical performance and theoretical guarantees. Conversely, they introduce a novel construction using Steiner points to simulate Locality Sensitive Hashing (LSH) within the graph search framework, achieving query times comparable to LSH ($\tilde{O}(n^{1/(c-1)})$). The proof techniques involve constructing hard instances of orthogonal vectors and leveraging geometric properties of Steiner nodes to implement hash table lookups via greedy traversal.
The experimental section is limited but sufficient for an initial exploration. The authors augment the ParlayANN implementation of DiskANN with simple $k$-means cluster centroids as Steiner points. They evaluate this on standard benchmarks (GloVe-200, NYTimes) using cosine similarity. The results show consistent improvements in recall vs. distance computations compared to the baseline, validating the theoretical potential of Steiner points. However, the experiments are preliminary, using a very simple heuristic for Steiner point generation rather than the optimal theoretical construction, and the datasets are relatively small compared to industrial scale.
The paper provides clear pseudocode for the generalized greedy search and detailed descriptions of the theoretical constructions. The experimental setup specifies the use of ParlayANN and specific hyperparameters (degree bound, pruning parameter, beam width). While the code for the specific Steiner-point augmentation is not explicitly linked in the text provided, the reliance on the open-source ParlayANN library and standard $k$-means clustering makes the experimental results highly reproducible.
The primary limitation is the gap between the theoretical upper bound (which requires a complex, specific construction of Steiner points to simulate LSH) and the experimental lower bound (which uses simple $k$-means centroids). The experiments do not demonstrate that the simple heuristic achieves the theoretical $O(n^{1/(c-1)})$ bound, only that it improves over the baseline. Additionally, the lower bound for the high-dimensional case requires $d=n$, leaving the low-dimensional worst-case complexity (for $d \ll n$) partially open, though a weaker bound is provided. The paper also ignores preprocessing time, focusing solely on query time.
This paper has significant implications for the vector database and search engine community. By proving that standard graph-based methods cannot theoretically match LSH in worst-case high-dimensional settings, it redirects research focus toward hybrid methods or Steiner-point-augmented graphs. The introduction of Steiner points as a mechanism to bridge the gap between empirical graph search and theoretical LSH guarantees offers a new design space for next-generation ANN algorithms. This could lead to more robust search systems with provable performance guarantees, which is critical for safety-critical applications. The paper establishes a fundamental lower bound for graph-based nearest neighbor search and demonstrates that Steiner points can restore sublinear query time guarantees comparable to LSH. It rigorously explains the theoretical limitations of current popular algorithms like HNSW and DiskANN while providing a constructive path forward through graph augmentation, bridging a significant gap between empirical success and theoretical understanding in high-dimensional search.
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory $32\times$ faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains $4.9\times$ faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.
Primary: Massachusetts Institute of Technology (MIT)
All Institutions: Massachusetts Institute of Technology (MIT)
[One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
The paper proposes ParaTTT (PTTT), an algorithm to parallelize Test-Time Training (TTT) layers, which replace linear fast-weight memories with small neural networks trained via online gradient descent. The core innovation is an iterative refinement scheme that approximates the sequential inner-loop gradient trajectory using chunkwise-parallel passes. The method leverages the fact that the exact online gradient descent recurrence is a fixed point of the iteration. The authors provide a theoretical guarantee of geometric convergence for a two-layer variant under specific contraction conditions. The approach decomposes the update into a forward pass for feature trajectories, an exact gated-delta-rule pass for readouts and feature gradients, and a backward pass for gradient propagation. This allows the nonlinear memory to be evaluated in parallel, overcoming the sequential bottleneck of previous TTT implementations like TTT-MLP.
The experiments demonstrate significant speedups: PTTT evaluates the inner-loop trajectory 32x faster than TTT-MLP at 4K context with matched error, and the full 340M model trains 4.9x faster. The models (340M and 1.3B parameters) are evaluated on language modeling perplexity and zero-shot accuracy. The results show that PTTT models match or exceed the performance of Gated DeltaNet and Mamba-2, and remain competitive with same-size Transformers. Notably, the paper claims better recall than linear recurrences, suggesting the nonlinear memory provides a qualitative advantage in information retention. The comparison with Transformers is particularly strong, as TTT layers have historically struggled to match Transformer performance at scale.
The paper provides a GitHub repository link (https://github.com/ntumm120/paraTTT), which is a strong positive for reproducibility. The abstract mentions specific model sizes (340M, 1.3B) and context lengths (4K), which are standard benchmarks. However, the full text provided is truncated (only abstract and headers are visible), so specific hyperparameters, dataset splits, and training schedules cannot be fully verified from the provided text. Assuming the code repository is complete and well-documented, reproducibility should be high.
The theoretical convergence proof is limited to a two-layer variant, while practical applications may involve deeper networks. The claim that "two iterations suffice" in practice needs robust validation across different sequence lengths and model scales. The comparison with Transformers is competitive but not necessarily superior in all metrics, and the "better recall" claim requires careful definition of recall in the context of language modeling. The computational overhead of the iterative refinement, even if parallelized, may still be higher than simple linear recurrences for very short sequences where parallelism gains are minimal.
This work addresses a critical bottleneck in the adoption of nonlinear recurrent architectures. By enabling efficient parallelization of TTT layers, it makes these more expressive models viable for large-scale language modeling. This could lead to new architectures that combine the expressiveness of nonlinear memories with the efficiency of parallel training, potentially challenging the dominance of Transformers in long-context tasks. The technique of iterative refinement for parallelizing sequential operations is also broadly applicable to other sequential learning problems. [One sentence main contribution]. The paper introduces ParaTTT, an algorithm that parallelizes Test-Time Training layers by iteratively refining gradient trajectory estimates, achieving significant speedups while maintaining or improving performance over linear recurrent architectures and matching Transformers.
We study adversarial linear bandits with strong corruption, in which an adaptive adversary chooses the reward vectors and a strong adversary corrupts each observed reward after seeing the learner's action, subject to a known total corruption budget $C$. Strong corruption breaks the importance-weighted reward estimates used by adversarial bandit algorithms: by corrupting only the rounds in which a rarely sampled action is drawn, the adversary amplifies each unit of corruption by the inverse of its sampling probability. The best known bound for this problem, $\tilde O(\sqrt{d^3T} + dC)$ by Liu et al. (2024), is a factor $\sqrt{d}$ above the corruption-free rate in its leading term, and whether $\tilde O(d\sqrt{T} + dC)$ regret is achievable was left open. We propose CRW-FTRL, which runs follow-the-regularized-leader with a radius-weighted entropy regularizer and estimates rewards with a normalized kernel estimator whose sensitivity to corruption is governed by per-action radii instead of inverse sampling probabilities. The radii are nondecreasing and are chosen jointly with the sampling distribution through a monotone log-determinant barrier problem, so that a negative linear term in the regularizer cancels both the corruption bias and the estimation error of every comparator. For $K$ actions in $\mathbb{R}^d$, CRW-FTRL achieves $\tilde O(\sqrt{dT\log K} + dC)$ regret with high probability against an adaptive adversary, which matches the best known corruption-free rate and the $Ω(dC)$ lower bound up to logarithmic factors. Applied to a discretization of a general compact action set, it yields $\tilde O(d\sqrt{T} + dC)$ regret, which settles the statistical rate of corrupted adversarial linear bandits under strong corruption.
Primary: University of California
All Institutions: University of California
The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
The paper proposes CRW-FTRL, a novel algorithm for adversarial linear bandits with strong corruption. The core innovation lies in replacing standard importance-weighted reward estimates—which are vulnerable to strong adversaries who target rarely sampled actions—with a normalized kernel estimator governed by per-action radii. These radii are determined via a monotone log-determinant barrier problem, coupled with a radius-weighted entropy regularizer. This design effectively decouples the sensitivity of the reward estimator from the inverse sampling probabilities, preventing the adversary from amplifying corruption. The theoretical framework is rigorous, leveraging FTRL with a specific regularizer structure to cancel out both corruption bias and estimation error.
The provided text is a theoretical paper focused on regret bounds. It does not contain empirical experiments, benchmarks, or comparisons with baselines in a practical setting. The contribution is entirely analytical, proving that the proposed algorithm achieves minimax optimal regret rates.
As a purely theoretical paper, reproducibility is tied to the correctness of the proofs rather than code implementation. The algorithm is described in sufficient detail for implementation, but the lack of code or empirical validation limits practical reproducibility assessment.
The primary limitation is the lack of empirical validation. The paper acknowledges that the reduction to general compact action sets via discretization is not computationally efficient, leaving the question of a computationally efficient algorithm for general action sets open. Additionally, the results are high-probability bounds, and the constants hidden in the tilde-O notation may be significant in practice.
This work settles a key open question in the theory of corrupted bandits, providing minimax optimal rates for adversarial linear bandits with strong corruption. It has significant implications for the design of robust online learning algorithms in adversarial environments, such as online advertising and recommendation systems where feedback can be manipulated. The techniques developed (radius-weighted regularizers, kernel estimators with controlled sensitivity) may be applicable to other robust learning problems. The paper resolves an open problem in adversarial linear bandits by proposing CRW-FTRL, which achieves minimax optimal regret under strong corruption. The technical contribution is significant, introducing a novel reward estimation and regularization scheme that effectively neutralizes the advantage of strong adversaries, thereby establishing tight theoretical bounds for this challenging setting.
We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.
Primary: Google DeepMind
All Institutions: Google DeepMind, Google
EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
The paper proposes EmbeddingGemma 2, a 744M-parameter multimodal embedding model that unifies text, image, audio, and video into a single vector space. The core architectural contribution is the simplification of Per-Layer Embeddings (PLE) by removing the dedicated lookup table and instead projecting the main input embeddings, which reduces memory overhead and improves multimodal performance. The training methodology employs a rigorous three-stage curriculum: (1) text pre-finetuning with distillation to establish a strong text baseline, (2) multimodal pre-finetuning with distillation from a larger teacher model (Gemini Embedding 2) to inherit cross-modal alignment, and (3) multimodal finetuning using contrastive learning, spread-out regularization, and Matryoshka Representation Learning (MRL) to optimize the student model's specific embedding space. The use of a "tail-patch" stage for Quantization-Aware Training (QAT) distillation is a practical engineering innovation that minimizes quality loss in low-precision deployments.
The evaluation is extensive, covering seven major benchmark suites (MTEB, MMEB-v2, MIEB-lite, MAEB, MSEB) across text, code, vision, and audio. The model establishes a new Pareto frontier, outperforming specialist models and general-purpose models 2-12x larger. Ablation studies are thorough, validating the removal of the PLE lookup table, the necessity of co-training the vision encoder, and the impact of vision token budgets on different task types (e.g., document retrieval vs. video). The results demonstrate that native audio processing significantly outperforms ASR-based pipelines, particularly for non-speech sounds and paralinguistic features.
The paper provides high reproducibility. The model weights are released openly. Detailed architectural specifications (dimensionality, layer counts, attention types) are provided. The training recipe, including loss functions, data mixing strategies, and hyperparameter schedules, is described in detail. The release of quantized checkpoints and the specific instructions for task prefixes further aid in practical reproduction and deployment.
The model is still relatively small (744M parameters), which may limit its performance on extremely complex, long-context, or highly specialized scientific tasks compared to frontier LLMs. The paper notes a regression in English-only text performance compared to its predecessor, though multilingual performance improved. The reliance on a specific teacher model (Gemini Embedding 2) for distillation may limit the transferability of the training recipe to other base architectures.
This work significantly lowers the barrier to entry for high-quality multimodal retrieval and embedding tasks. By enabling on-device deployment (191MB-567MB footprint) with minimal quality loss, it facilitates privacy-preserving and offline multimodal applications. The unified space allows for novel applications like interleaved video-audio-text search, which were previously difficult to achieve with separate unimodal models. EmbeddingGemma 2 introduces a compact, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks through a simplified per-layer embedding architecture and a multi-stage distillation-to-contrastive training curriculum. The paper demonstrates that a sub-1B parameter model can rival or exceed much larger specialists by leveraging native multimodal processing and rigorous regularization, setting a new standard for efficient, unified representation learning.
We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction-data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires 3 times more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security-utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.
Primary: Imperial College London
All Institutions: Imperial College London, Tübingen AI Center, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Fraunhofer HHI
The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
The paper proposes "Heads of Security" (HoS), a defense mechanism against prompt injection that operates by learning per-head attention gates. Unlike previous methods that rely on broad fine-tuning (like SecAlign) or complex architectural changes (like ASIDE), HoS targets the attention mechanism directly to enforce instruction-data separation. The core idea is to learn how specific attention heads should redistribute focus between trusted (system/user) and untrusted (tool/web) tokens. This allows the model to process untrusted content for reasoning purposes without executing any instructions embedded within it. The approach is lightweight and targeted, which is a significant methodological shift from behavior-level alignment to attention-level intervention.
The evaluation is rigorous and comprehensive, testing on two major open-weight models: Llama-3.3-70B-Instruct and GPT-OSS-20B. The paper compares HoS against strong baselines: SecAlign++ (SOTA fine-tuning defense) and ASIDE (architectural defense). Key results include a dramatic reduction in Attack Success Rate (ASR) against adaptive attacks (RL-Hammer drops from 97% to 10% on GPT-OSS-20B) and improved robustness against white-box attacks (Claudini). Crucially, the paper demonstrates that HoS preserves base-model utility on standard benchmarks, whereas competitors degrade in few-shot settings. The inclusion of agentic benchmarks with static attacks further validates the method's practical applicability.
The authors provide a GitHub repository and a Hugging Face space, indicating a commitment to open science. The code availability is a strong plus. However, the specific details of the "autoresearch" used to strengthen the white-box attack against HoS are not fully detailed in the abstract, though likely present in the full text. The use of standard models and benchmarks enhances reproducibility.
The primary limitation is the reliance on open-weight models for evaluation; performance on closed-source models like GPT-4 or Claude is not directly testable in this manner. Additionally, the method requires access to the model's internal attention heads, which limits its applicability to API-only models. The paper also notes that ASIDE struggles with tool use, but it is unclear if HoS fully resolves all edge cases in complex agentic workflows beyond the tested benchmarks.
This work has high impact on the field of LLM security. As LLMs are increasingly deployed in agentic roles with access to external data, prompt injection is a critical vulnerability. HoS offers a scalable, efficient, and effective defense that does not sacrifice model utility. It provides a new paradigm for securing LLMs by manipulating internal representations (attention) rather than just output behavior. This could lead to more secure and trustworthy AI systems in production environments. The paper introduces Heads of Security (HoS), a novel prompt injection defense that enforces instruction-data separation via learned per-head attention gates. By targeting the attention mechanism directly, HoS achieves superior robustness against adaptive and white-box attacks compared to state-of-the-art fine-tuning and architectural baselines, while preserving the model's general utility and few-shot performance, marking a significant advancement in secure LLM deployment.
High-resolution probabilistic weather forecasting remains computationally demanding. Here we present Astra, a 0.25$^\circ$ ensemble model trained on eight H200 GPUs. It combines Kantorovich Attention, a two-sided transport attention in which each region chooses where it reads from and writes to, so every region sends and receives at token resolution, with a self-featurised continuous ranked probability score (CRPS) that also scores forecasts in the model's multiscale features. In WeatherBench 2, Astra matches GenCast's CRPS averaged over five upper-air targets and days 1-10 with 9.8% of its hardware-normalised training compute. Verified against ERA5, its CRPS and ensemble-mean root-mean-square error are lower than the ECMWF ensemble's on all 600 variable, level and lead-time combinations, and its CRPS is lower than GenCast's on 73.5% of them. A 15-day member needs 39 times fewer floating-point operations than GenCast's GPU implementation, and 50 members take under 40 min on one consumer GPU, within 21 GB.
Primary: Unknown (Affiliations not explicitly listed in provided text, likely independent or small lab given hardware constraints)
All Institutions: Unknown
Astra achieves quarter-degree probabilistic weather forecasting with 9.8% of GenCast's training compute and 39x less inference FLOPs by introducing two-sided transport attention (Kantorovich Attention) and a self-featurised CRPS objective, enabling high-resolution ensemble forecasting on a single consumer GPU.
The paper introduces "Astra," a probabilistic weather forecasting model that achieves 0.25° resolution with significantly reduced computational cost compared to state-of-the-art models like GenCast. The core methodological contributions are twofold: (1) **Kantorovich Attention (KA)**, a two-sided transport attention mechanism where each grid tile selects both where it reads from (pull) and where it writes to (push). This ensures every tile is read at token resolution, avoiding the "unread" tile problem of one-sided sparse attention, while maintaining linear complexity relative to grid points. (2) **Self-featurised CRPS**, a training objective that scores ensemble members not just in physical space but also in the model's own multiscale feature space using a frozen copy of the network. This encourages the model to learn representations that are discriminative and structurally consistent, similar to how diffusion models use pretrained encoders. The architecture is a W-Net operating directly on the 0.25° grid without a latent encoder, using MeanFlow for single-step generation.
The experimental evaluation is rigorous and comprehensive. Astra is benchmarked against GenCast (the current SOTA for high-res probabilistic forecasting) and the ECMWF IFS ENS (the operational gold standard). Key results include: (1) Matching GenCast's CRPS on upper-air targets with only 9.8% of the hardware-normalised training compute. (2) Outperforming IFS ENS on all 600 variable/level/lead-time combinations in CRPS and ensemble-mean RMSE. (3) Outperforming GenCast in CRPS on 73.5% of combinations. (4) Significant efficiency gains: 39x fewer FLOPs for inference than GenCast's GPU implementation, allowing 50-member 15-day forecasts on a single consumer GPU (RTX 5090) in under 40 minutes. Spectral analysis confirms that Astra preserves fine-scale spatial structure better than GenCast and IFS ENS, particularly in the Alps and tropics. The use of WeatherBench 2 and ERA5 verification provides a standard, reproducible benchmark.
The paper provides extensive methodological details, including equations for the attention mechanism, the transportation problem formulation, and the self-featurised CRPS loss. It specifies hardware requirements (8x H200 GPUs for training) and inference costs (RTX 5090). While the code is not provided in the text, the level of detail in the Methods section (e.g., specific tile sizes, number of heads, initialization schemes, and noise prior construction) is high, suggesting that a competent team could reproduce the results. The reliance on specific hardware (H200/RTX 5090) may limit immediate reproducibility for labs with older GPUs, but the algorithmic details are clear.
The primary limitation is the hardware dependency for training; while inference is efficient, training still requires 8x H200 GPUs, which is a significant barrier for smaller labs compared to models trained on fewer resources. The model is evaluated primarily on ERA5 reanalysis data; performance on real-time operational data with missing or noisy inputs is not tested. The "self-featurised" component relies on a frozen copy of the network, which adds complexity to the training pipeline and may introduce instability if the network's features drift significantly during training. Additionally, the paper focuses on medium-range forecasting (up to 15 days); performance at longer lead times (sub-seasonal) is not evaluated.
This paper has high potential impact by democratizing high-resolution probabilistic weather forecasting. By reducing inference costs to the level of a single consumer GPU, it enables local, real-time probabilistic forecasting for applications such as renewable energy management, agriculture, and disaster response, where cloud-based or high-performance computing access may be limited. The two-sided transport attention mechanism is a generalizable architectural innovation that could be applied to other high-resolution spatial tasks (e.g., remote sensing, medical imaging) where global context is needed but dense attention is prohibitive. The self-featurised CRPS offers a new paradigm for training generative models to produce structurally consistent samples, potentially influencing broader generative AI research. Astra achieves quarter-degree probabilistic weather forecasting with 9.8% of GenCast's training compute and 39x less inference FLOPs by introducing two-sided transport attention (Kantorovich Attention) and a self-featurised CRPS objective, enabling high-resolution ensemble forecasting on a single consumer GPU.
Statistical procedures often control expected loss at a prescribed level while leaving part of their risk budget unused. Building on the e-value representation of risk-controlled decisions and the e-closure principle for multiple testing, we formulate recertification, a general paradigm for improving procedures by exploiting computable slack in their validity certificates. A valid procedure's losses normalized by their risk budgets form a state-indexed e-value collection; conversely, decisions whose normalized losses are bounded by this collection inherit its risk guarantees. Recertification uses distributional structure to quantify slack, enlarge the certifying e-values while preserving validity, and construct less conservative decisions. We develop two applications. For conformal risk control (CRC) with a known loss family, recertification redistributes unused risk budget among the held-out points of each hypothetical augmented sample. The resulting controller is valid under every finite exchangeable law and no more conservative than CRC on any calibration sample. For convex losses on an interval of actions, repeated recertification converges geometrically to an admissible controller. For false discovery rate control with arbitrarily dependent e-values, recertification exploits unused conditional budgets in randomized e-BH. It yields a closed-form randomized procedure whose rejection set contains that of randomized e-BH for every realization of the shared randomization and whose expected number of rejections is maximal subject to the same conditional budget constraints. Recertifying with the larger e-collection of closed e-BH gives a second closed-form procedure whose rejection set contains that of the first. Experiments on CIFAR-100 hierarchical classification and multiple testing simulations demonstrate improved budget utilization, informativeness and power over the baselines.
Primary: University of Amsterdam
All Institutions: University of Amsterdam, University of Oxford, University of Cambridge
The paper proposes a novel "recertification" framework that uniformly improves conformal risk control and randomized e-BH procedures by exploiting unused risk budget in e-value certificates. It provides rigorous theoretical guarantees, including geometric convergence to admissible controllers for convex losses and closed-form power improvements for multiple testing, supported by targeted experiments on CIFAR-100 and simulations.
The paper introduces "recertification," a general paradigm for improving statistical decision procedures by exploiting slack in their validity certificates (e-values). The core contribution is a rigorous theoretical framework that maps valid procedures to e-value collections and vice versa, allowing the construction of less conservative decisions by enlarging these e-values while preserving validity guarantees. The authors apply this to Conformal Risk Control (CRC) and randomized e-BH (Benjamini-Hochberg) procedures. For CRC, they prove that repeated recertification converges geometrically to an admissible controller under convexity assumptions. For multiple testing, they derive closed-form randomized procedures (RU-eBH and RC-eBH) that uniformly dominate existing baselines in power. The methodology is mathematically dense but logically sound, leveraging the e-closure principle effectively.
The experimental section is limited but appropriate for a theory-heavy paper. It includes simulations for multiple testing showing power gains over e-BH and U-eBH, and a real-data study on CIFAR-100 hierarchical classification. The CIFAR-100 results demonstrate improved budget utilization and informativeness compared to standard CRC and envelope CRC. However, the scope of empirical validation is narrow, focusing primarily on one classification task and synthetic multiple testing scenarios, lacking broader benchmarks or large-scale real-world applications that would typically be expected for a high-impact ML paper.
The paper provides clear algorithmic descriptions (Algorithm 1) and theoretical conditions for validity. The use of standard datasets (CIFAR-100) and well-defined baselines (CRC, e-BH) aids reproducibility. However, the text is truncated in the provided input, and specific hyperparameters or code availability are not explicitly confirmed in the visible portion, though the mathematical definitions are precise enough for implementation.
The primary limitation is the computational cost of repeated recertification, which can be expensive for high-dimensional or individual-score based baselines, as noted in the "Computation" section. The admissibility results rely on convexity or specific finite-space conditions, which may not hold for all practical loss functions. Additionally, the empirical evaluation is relatively small-scale, limiting the generalizability of the observed performance gains to other domains.
This work has significant potential to influence the field of conformal prediction and multiple testing by providing a systematic way to reduce conservatism in statistical guarantees. It bridges decision theory and e-value methodology, offering a template for improving other risk-controlled procedures. The uniform improvement guarantees are particularly valuable for applications requiring strict error control, such as medical diagnosis or autonomous systems. The paper proposes a novel "recertification" framework that uniformly improves conformal risk control and randomized e-BH procedures by exploiting unused risk budget in e-value certificates. It provides rigorous theoretical guarantees, including geometric convergence to admissible controllers for convex losses and closed-form power improvements for multiple testing, supported by targeted experiments on CIFAR-100 and simulations.
Video world models trained on large-scale robot data promise a safe and scalable approach for learning and evaluating policies that act in the physical world. However, they still struggle with long-horizon rollouts and out-of-distribution generalization, and they remain computationally demanding at inference time. Typically, video world models make predictions in the latent space of a reconstruction-trained autoencoder, which however is not directly optimized for dynamics modeling. On the other hand, world models designed for latent planning usually optimize latents to be predictable, but they forgo accurate prediction of observations in pixel space. In this paper, we show that it is possible to achieve the best of both worlds: a latent geometry optimized to make future prediction as simple as possible without sacrificing accurate prediction of future observations. More precisely, we find that temporal objectives for representation learning, paired with a past-conditioned decoder architecture, lead to consistent improvements in generative video world models at the same training budget. The gains include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors out of distribution and over long horizons. Our world models also agree more closely with real-world policy evaluation when using a vision-language model to judge task success from their pixel-space predictions. Overall, our study indicates that temporal predictability is an important design criterion for the latent space of video world models, and provides concrete guidance on how to incorporate it into the training of autoencoders.
Primary: New York University
All Institutions: New York University, Empire AI Consortium
The paper demonstrates that optimizing latent spaces for temporal predictability significantly improves the efficiency and accuracy of video world models. By integrating temporal objectives into autoencoder training, the authors achieve a superior balance between reconstruction fidelity and dynamics simplicity, leading to substantial gains in inference speed, training efficiency, and out-of-distribution generalization, which are critical for real-world robotic applications.
The paper proposes a principled approach to designing the latent space for video world models. The core insight is that standard autoencoders optimized for reconstruction do not necessarily produce latents that are easy to predict dynamically. The authors introduce temporal objectives for representation learning combined with a past-conditioned decoder architecture. This allows the latent geometry to be optimized for simplicity of future prediction (dynamics) while maintaining high-fidelity reconstruction of observations. This addresses a fundamental trade-off in world modeling: the tension between accurate pixel-space reconstruction and efficient latent-space dynamics modeling. The methodology is sound, leveraging established concepts in representation learning (temporal consistency) and applying them specifically to the context of generative video world models for robotics.
The experimental evaluation is robust, demonstrating significant improvements over baselines. Key metrics include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors in out-of-distribution and long-horizon scenarios. The inclusion of a vision-language model (VLM) to judge task success from pixel-space predictions adds a layer of semantic evaluation that is highly relevant to the robotics application domain. The comparison with both reconstruction-focused and dynamics-focused baselines effectively isolates the contribution of the proposed latent design.
The paper provides a reproducibility statement and a link to the source code for fine-tuning autoencoders, training world models, and evaluation. The appendices detail objectives, architectures, training settings, and evaluation protocols. This level of detail supports reproducibility, although the specific hyperparameters for the "temporal objectives" would need to be verified in the code to ensure full transparency.
The primary limitation is the computational cost of training the autoencoders with temporal objectives, which may be prohibitive for very large-scale datasets without significant optimization. Additionally, the reliance on a VLM for evaluation introduces potential biases inherent in the VLM itself. The paper focuses on robot data, so generalizability to other video domains (e.g., natural videos) is not fully explored.
This work has significant implications for the field of world models, particularly for robotics and autonomous systems. By making video world models more efficient and accurate in long-horizon rollouts, it enables safer and more scalable policy learning and evaluation. The finding that temporal predictability is a key design criterion for latent spaces provides concrete guidance for future research in representation learning for dynamics. The paper demonstrates that optimizing latent spaces for temporal predictability significantly improves the efficiency and accuracy of video world models. By integrating temporal objectives into autoencoder training, the authors achieve a superior balance between reconstruction fidelity and dynamics simplicity, leading to substantial gains in inference speed, training efficiency, and out-of-distribution generalization, which are critical for real-world robotic applications.
While progress in informal mathematics has come largely from scaling the chain-of-thought and agentic reasoning of frontier language models, formal theorem proving has increasingly relied on human-designed workflows around these models, such as planning, decomposition, and orchestrating specialized provers. This raises a bitter-lesson question: can a simple agentic loop that scales test-time compute match these workflows? The answer matters as formal verification must scale to keep up with frontier models producing research mathematics, yet new workflows rarely settle it, as their gains are often confounded with stronger models or more compute. We study this question with h1, a simple harness that gives a model an efficient interface to Lean 4 and keeps it working through verifier feedback and self-reflection, without imposing a decomposition or search structure. h1 reaches state-of-the-art results on PutnamBench, FATE-X, and FormalConjectures100-Solved, settles eight open problems in FormalConjectures, and, at matched models, matches or outperforms specialized workflows such as Goedel-Architect and AlphaProof Nexus, surprisingly showing that a single attempt (pass@1) can often scale to solve open problems. Through ablations and matched-model comparisons with Claude Code and AxProverBase, we find that simplicity alone is not enough: an efficient Lean interface largely determines cost and solve rate. We thus recommend that future work evaluate frontier models with efficient execution feedback before prescribing workflows, at the risk of bitter lessons.
Primary: Google DeepMind
All Institutions: Google DeepMind, University of Edinburgh
The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
The paper proposes "h1," a minimalist agentic harness for Lean 4 theorem proving that relies on a simple ReAct-style loop with an efficient interface to the Lean compiler, library search tools, and a reflection mechanism to compact context. The core methodological contribution is not a new model architecture or training algorithm, but a rigorous empirical demonstration that "bitter lesson" principles apply to harness design: that scaling test-time compute in a simple loop with good tooling outperforms complex, human-designed workflows (like planning, decomposition, or multi-agent orchestration) when using the same frontier models. The methodology is sound, focusing on isolating the variable of harness complexity while controlling for model capability and inference cost.
The experimental evaluation is extensive and high-quality. It covers a wide range of benchmarks including PutnamBench, FATE-X, FormalConjectures, and IMO 2026. The paper provides matched-model comparisons against state-of-the-art specialized systems (Goedel-Architect, AlphaProof Nexus, MerLean, etc.), which is crucial for a fair assessment. The results show that h1 achieves state-of-the-art performance, solving open problems and matching or exceeding specialized workflows at lower cost. The analysis of cost-efficiency curves (pass@1 vs pass@k) is particularly insightful, showing that longer single attempts can be more efficient than multiple shorter ones.
The paper provides detailed appendices on tools, prompts, error handling, and cost accounting. It specifies the exact versions of Lean and Mathlib used, and provides a link to the pull request documenting the newly solved conjectures. While the specific code for the h1 harness is not explicitly linked in the provided text (only the conjectures PR), the level of detail in the prompts and tool descriptions suggests high reproducibility for researchers with access to the same frontier models.
The primary limitation is the reliance on proprietary frontier models (GPT-5.6, Opus 4.8, etc.), which are black boxes and subject to rapid change, making long-term reproducibility difficult. The paper also acknowledges that it does not address the challenge of formalizing proofs that exceed the context window or require parallel agents for very large projects, though it suggests this is a natural next step. The cost analysis relies on API pricing, which may not reflect the true compute cost for self-hosted models.
This paper has significant broader impact by challenging the prevailing trend in automated theorem proving towards increasingly complex, human-designed workflows. It provides strong evidence that as LLMs improve, the focus should shift to building efficient interfaces and allowing models to scale their own reasoning, rather than imposing rigid structures. This insight is likely to influence the design of agents in other structured domains, such as software verification and formal methods. The paper demonstrates that a simple agentic loop with an efficient Lean interface and test-time scaling can match or outperform complex, human-designed workflows in automated theorem proving, settling open mathematical problems and achieving state-of-the-art results on major benchmarks. This work provides a compelling empirical case for the "bitter lesson" in the context of agent harness design, suggesting that future progress will come from scaling model capabilities and interaction efficiency rather than from increasingly elaborate orchestration strategies.
Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, University of Maryland
The paper establishes that hierarchical, intrinsically motivated agents, specifically using DIAYN, can quantitatively match both the behavioral variability and the prospective neural coding of spontaneous exploration in *Drosophila*, providing a powerful computational framework for reverse-engineering animal autonomy. By combining a realistic neuromechanical digital twin with a broad comparison of 20 intrinsic motivation algorithms, the study demonstrates that skill discovery objectives naturally give rise to internal representations that predict future actions and capture the geometric structure of biological neural activity, offering a novel perspective on the computational basis of autonomous behavior.
The paper proposes a rigorous framework for reverse-engineering animal autonomy by combining a neuromechanical digital twin of *Drosophila* (NeuroMechFly v2) with hierarchical intrinsically motivated agents. The core methodological contribution is the implementation of a hierarchical policy that operates over a discrete library of ethologically motivated motor primitives (syllables) rather than raw joint torques. This aligns the computational model with biological behavioral organization. The authors train 20 distinct intrinsic motivation algorithms (curiosity, entropy, and competency families) using Recurrent PPO without any extrinsic reward. The evaluation methodology is particularly strong, as it does not just compare behavioral statistics but also compares internal neural representations. Specifically, they decode future turn direction from the agent's latent states and compare this "prospective coding" to the neural activity in the fly's Inferior Posterior Slope (IPS) region. They further validate the geometric structure of the internal representations by checking for the sum-difference rule observed in biological interhemispheric encoding.
The experimental setup is robust and well-controlled. The agents are evaluated in a virtual tethered-ball assay that mimics the biological experiment of Brezovec et al. (2023). The comparison is quantitative, using balanced accuracy for decoding future turn direction and correlation coefficients for velocity encoding. The results show that DIAYN (Diversity is All You Need) significantly outperforms other intrinsic motivation methods in jointly matching both the behavioral variability (turning bias distribution) and the neural prospective coding of the flies. The finding that DIAYN's latent activity captures the sum-difference rule for interhemispheric encoding is a strong empirical result that links the computational objective to specific biological neural dynamics. The inclusion of random controls and extrinsic reward controls strengthens the causal claim that intrinsic motivation drives the observed similarities.
The paper provides high reproducibility. It specifies the exact environment (FlyGym v2), the biomechanical model (NeuroMechFly v2), the architecture (LSTM actor-critic with specific dimensions), and the training hyperparameters (50 million steps, 2048 parallel envs). The use of standard algorithms like PPO and well-known intrinsic motivation methods (DIAYN, ICM, etc.) makes it feasible for other researchers to replicate. The project website is provided, which likely contains code and further details. The detailed description of the evaluation protocol (Savitzky-Golay filtering, turn detection thresholds) ensures that the behavioral metrics are comparable to the biological data.
The primary limitation is the constraint of the behavioral space to 2D locomotion and a fixed set of 7 primitives. This limits the generality of the findings to specific types of exploratory behavior. The paper acknowledges that learning primitives jointly with high-level selection or expanding to flight would be future work. Additionally, the comparison is limited to the IPS region of the fly brain; a whole-brain comparison would be more comprehensive but is computationally and experimentally challenging. The reliance on a specific biological dataset (Brezovec et al.) means the findings are specific to *Drosophila* and may not generalize directly to other species without significant adaptation.
This work has significant implications for both artificial intelligence and neuroscience. For AI, it demonstrates that intrinsic motivation, specifically skill discovery, can produce agents with complex, autonomous behaviors and internal representations that mimic biological systems. This challenges the standard paradigm of extrinsic reward-driven RL and suggests that self-supervised objectives can capture deep structural properties of animal cognition. For neuroscience, it provides a computational hypothesis for how the fly brain generates spontaneous behavior and prospective neural signals. The framework established here can be extended to other species and behaviors, potentially leading to a general theory of animal autonomy. The interdisciplinary nature of the work bridges the gap between embodied AI and computational ethology, offering a new methodology for testing computational theories of brain function. The paper establishes that hierarchical, intrinsically motivated agents, specifically using DIAYN, can quantitatively match both the behavioral variability and the prospective neural coding of spontaneous exploration in *Drosophila*, providing a powerful computational framework for reverse-engineering animal autonomy. By combining a realistic neuromechanical digital twin with a broad comparison of 20 intrinsic motivation algorithms, the study demonstrates that skill discovery objectives naturally give rise to internal representations that predict future actions and capture the geometric structure of biological neural activity, offering a novel perspective on the computational basis of autonomous behavior.
Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tsinghua University, Shanghai AI Laboratory
[One sentence main contribution]. The paper introduces RynnVLA-Latent, a scalable framework for training VLA models by learning 3D-grounded latent actions from heterogeneous video data, significantly improving performance on robotic manipulation benchmarks by decoupling general motion learning from embodiment-specific control.
The paper proposes RynnVLA-Latent, a framework addressing the data scarcity and embodiment mismatch problems in Vision-Language-Action (VLA) models. The core innovation is RynnLAM, a module that learns continuous, 3D-grounded latent actions from heterogeneous video data (human, robot, simulation) without requiring native robot control labels. The method employs residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling to ensure the latent space captures control-relevant motion rather than just visual appearance changes. The VLA model then uses flow matching to predict these latent action chunks from visual and language inputs, followed by an adaptation layer to map latent actions to specific robot embodiments. This approach effectively decouples the learning of general manipulation dynamics from the specific control spaces of individual robots, allowing for pretraining on massive, unstructured video datasets.
The experimental results are strong and comprehensive. The model achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus, 51.9% on VLABench, and 88.6% on RoboTwin 2.0. These numbers are competitive with or superior to state-of-the-art methods in the VLA domain. The evaluation includes both simulation benchmarks and real-world tasks across different robot platforms (dexterous hands, mobile grippers), demonstrating generalization. The use of 30.1K hours of video data to generate 745.52M latent-action descriptors highlights the scalability of the approach. Ablation studies on representation learning and data scaling further support the technical claims.
The paper provides detailed descriptions of the RynnLAM training process and the flow matching objective. However, specific hyperparameters, exact dataset compositions, and code availability are not explicitly provided in the extracted text. The reliance on a large-scale proprietary or curated dataset (30.1K hours) may limit immediate reproducibility for smaller labs, though the methodology is clearly defined.
The primary limitation is the computational cost of processing 30.1K hours of video and generating latent actions. Additionally, the mapping from latent actions to specific robot actions requires an adaptation phase, which may not generalize perfectly to entirely new embodiments without some fine-tuning. The performance on VLABench (51.9%) is lower than on other benchmarks, suggesting potential weaknesses in certain complex or long-horizon tasks.
This work has significant implications for the field of robotic learning by providing a scalable pathway to train foundation models using unstructured video data. It reduces the dependency on expensive, robot-specific action-labeled datasets, potentially accelerating the development of general-purpose robot policies. The 3D-grounded latent action space offers a robust representation that can be shared across different robot morphologies. [One sentence main contribution]. The paper introduces RynnVLA-Latent, a scalable framework for training VLA models by learning 3D-grounded latent actions from heterogeneous video data, significantly improving performance on robotic manipulation benchmarks by decoupling general motion learning from embodiment-specific control.
Most analyses of neural representations focus on trained endpoints, leaving the process by which internal structure forms comparatively unmeasured. We instead treat representation formation itself as an object of study, asking whether the path a representation takes through training reveals information that its endpoint does not. We formalize this idea by viewing representation learning as a geometric trajectory, using Bures geometry to measure movement through representation space and quantum Fisher information to characterize local sensitivity to input factors. We find that learning concentrates local sensitivity on task-relevant factors, and that changing only the prediction target reorganizes this geometry around the corresponding visual factor. More strikingly, representation trajectories become systematically less direct as the learned computation requires higher-order interactions among input factors. This effect persists under matched-performance and training-duration controls and transfers from synthetic vector inputs to residual CNNs on rendered and natural images. Importantly, formation history contains predictive information about the stability of the learned representation: controlling for interaction order, path tortuosity predicts its subsequent drift under continued training and sensitivity to parameter perturbations, whereas endpoint displacement does not. Finally, representation-aware optimization lets us actively shape these trajectories, showing that formation geometry is not only measurable but actionable. Together, these results establish representation formation as a new source of information about learning: its geometry reveals properties of the computation being constructed, predicts subsequent representational stability, and provides a target for intervention during optimization.
Primary: University of California
All Institutions: University of California
The paper establishes representation formation as a measurable geometric object that reveals computational structure and predicts stability. By linking Bures geometry and Quantum Fisher Information to training trajectories, it provides a novel framework that not only diagnoses the complexity of learned computations but also enables targeted interventions to shape the learning path.
The paper proposes a rigorous information-geometric framework for analyzing representation formation during training. It defines the representation state as the trace-normalized covariance of hidden activations and utilizes Bures distance to measure the trajectory of this state through training. The local geometry is characterized by Quantum Fisher Information (QFI), which identifies input factors to which the representation is locally sensitive. The authors derive a "Representation Natural Gradient" (RepNG) by pulling back the Bures/QFI metric to parameter space, allowing for optimization that explicitly constrains representational movement. The theoretical connection between the local QFI metric and the global Bures trajectory is well-established, providing a unified geometric language for both local sensitivity and global path tortuosity.
The experiments are extensive and well-controlled. The authors validate the local geometry (QFI) on synthetic tasks and dSprites, showing it correctly identifies task-relevant factors. The core empirical finding—that higher-order computational interactions lead to higher path tortuosity—is tested across MLPs and Residual CNNs, using both synthetic and natural images (CIFAR-10 composites). Crucially, the paper includes matched-performance and matched-training-duration controls to rule out trivial explanations. The predictive power of tortuosity for subsequent representational stability (drift and parameter sensitivity) is demonstrated in a fresh 30-seed sweep. The intervention study shows RepNG reduces tortuosity compared to standard optimizers at matched performance.
The paper provides a detailed reproducibility statement, including specific hyperparameters, seed counts, and code availability. The GitHub repository link is provided. The experimental protocols are described in sufficient detail in the appendices to allow for replication, including the specific construction of the parameter subspaces for the optimizer comparison.
The primary limitation is scale. The experiments are conducted on small networks (MLPs and small ResNets) and relatively simple tasks (parity, sign patterns, dSprites factors). It remains an open question whether these geometric properties hold for large-scale foundation models or complex natural language tasks. Additionally, the RepNG method requires estimating the pullback metric, which may be computationally expensive for very large networks, though the paper demonstrates it in a 24-dimensional subspace.
This work shifts the focus of representation analysis from static endpoints to dynamic formation processes. It provides a new metric (tortuosity) that correlates with computational complexity and stability, offering a new diagnostic tool for understanding learning dynamics. The RepNG optimizer offers a practical method for stabilizing representations during training, which could be valuable in continual learning and transfer learning scenarios. The paper establishes representation formation as a measurable geometric object that reveals computational structure and predicts stability. By linking Bures geometry and Quantum Fisher Information to training trajectories, it provides a novel framework that not only diagnoses the complexity of learned computations but also enables targeted interventions to shape the learning path.
Indoor context deciphered by ambient sensing underpins many applications, from assisted living to smart buildings, yet remains weakly connected to the language-aligned paradigm of modern multimodal systems. Through billions of IoT devices, ambient WiFi Channel State Information (CSI) captures the indoor context of human activity and environmental dynamics passively via ubiquitous, always-on infrastructure. However, unlike vision or audio, CSI lacks naturally paired text like captions or transcripts, limiting its integration with language-driven AI systems. We present CSI-TextBench, a large-scale dataset and benchmark for aligning CSI data with natural language, unifying 3 sensing benchmarks into 285,790 samples across 7 tasks and pairing them with 2,291 hierarchically structured, label-derived descriptions via many-to-many assignment to construct 2.0 million CSI-text training tuples. Benchmarking 5 alignment methods from 4 paradigm families, we show that contrastive CSI-text alignment matches supervised classifiers without class labels, adapts to unseen classes with few samples, and responds to open-vocabulary natural-language prompts. A single unified model supports compositional scene understanding, answering joint queries over activity, proximity, and identity, achieving 86.9% top-1 accuracy on 140 composite scenes, outperforming task-specific classifier ensembles. CSI-TextBench enables ubiquitous ambient sensing as a language-indexed perception layer, enabling indoor contextual awareness in language-driven AI systems.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign, ADT Research
CSI-TextBench introduces a large-scale dataset and benchmark for aligning WiFi CSI with natural language, demonstrating that contrastive learning can enable zero-shot and compositional scene understanding in ambient sensing. The paper makes a significant contribution by unifying heterogeneous sensing benchmarks into a language-compatible format, providing rigorous evaluation protocols for open-vocabulary and few-shot settings, and showing that a single unified model can effectively handle multi-attribute scene queries, thereby establishing a foundation for language-grounded ambient intelligence.
The paper proposes a systematic framework for aligning WiFi Channel State Information (CSI) with natural language. The core methodological contribution is not a new architecture but a rigorous benchmarking protocol and dataset construction pipeline. The authors unify three heterogeneous CSI datasets (CSI-Bench, XRF55, SignFi) into a standardized tensor format and generate text descriptions via hand-written templates rather than LLMs, ensuring label grounding. They evaluate five alignment objectives (CLIP-InfoNCE, SigLIP, LanguageBind, Barlow Twins, Masked Prediction) within a shared dual-encoder architecture (1D ResNet-18 for CSI, frozen CLIP ViT-B/32 for text). The methodology is sound, particularly in its handling of the "many-to-many" correspondence between signals and text, which is a unique challenge in sensing modalities lacking native captions. The use of frozen text encoders isolates the learning of the CSI representation, providing a clean experimental design.
The experimental evaluation is comprehensive, covering zero-shot classification, few-shot transfer, open-vocabulary prompting, and compositional scene understanding. The results demonstrate that contrastive alignment (specifically CLIP-InfoNCE) can match supervised classifiers without class labels, a significant finding for ambient sensing. The "Layer 3" experiments on compositional scene understanding (86.9% accuracy on joint queries) are particularly strong, showing that a single unified model can outperform task-specific ensembles. The ablations on text granularity and encoder choice are well-executed. However, the reliance on template-based text generation limits the semantic richness compared to true natural language, and the performance drop on fine-grained unseen classes (11.3% zero-shot) highlights the limits of current alignment techniques for highly specific gestures.
The paper provides high reproducibility. The authors release the dataset, code, and benchmark. The preprocessing pipeline is detailed, including specific normalization and subcarrier alignment steps. The training hyperparameters (learning rate, batch size, optimizer) are explicitly stated. The use of standard, publicly available base models (CLIP, ResNet) further aids reproducibility. The distinction between Level 1 and Level 2 captions is clearly defined, allowing other researchers to replicate the specific evaluation protocols.
The primary limitation is the synthetic nature of the text data. Since descriptions are derived from labels via templates, the model may not generalize to truly free-form, noisy, or context-rich natural language that does not strictly adhere to the template structures. The paper acknowledges this by testing "human-written free-form" language, but the gap between template-based and truly open-ended language remains. Additionally, the cross-environment and cross-device transfer performance is noted to be constrained by the inherent domain sensitivity of WiFi CSI, which is a fundamental physical limitation rather than a model flaw, but it limits the immediate practical deployment of the aligned model in unseen environments.
This work bridges the gap between passive sensing and language-driven AI, enabling ambient intelligence systems to be queried in natural language. This has significant implications for assistive living, smart buildings, and privacy-preserving monitoring (as CSI does not capture visual or audio content). By providing a standardized benchmark, it facilitates future research into RF-language alignment, potentially leading to more intuitive and flexible human-computer interfaces for indoor environments. The approach is generalizable to other sensing modalities like radar or mmWave, offering a template for language-grounded perception in non-visual domains. CSI-TextBench introduces a large-scale dataset and benchmark for aligning WiFi CSI with natural language, demonstrating that contrastive learning can enable zero-shot and compositional scene understanding in ambient sensing. The paper makes a significant contribution by unifying heterogeneous sensing benchmarks into a language-compatible format, providing rigorous evaluation protocols for open-vocabulary and few-shot settings, and showing that a single unified model can effectively handle multi-attribute scene queries, thereby establishing a foundation for language-grounded ambient intelligence.
Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.
Primary: Massachusetts Institute of Technology
All Institutions: Massachusetts Institute of Technology, Carnegie Mellon University, University of Maryland
The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
The paper introduces a novel cross-modal pre-training framework that aligns egocentric video with surface electromyography (EMG) signals. The core methodological contribution is the use of EMG as a "privileged supervisory signal" in a CLIP-style contrastive objective. By jointly training a video encoder and an EMG encoder, the authors create a shared embedding space where visual representations are shaped by the underlying motor intent, which is then discarded at inference time. This approach is technically sound, leveraging the dense, occlusion-free nature of EMG to provide a stronger inductive bias for action-centric vision tasks than standard self-supervised video learning. The inclusion of a VQ-VAE to learn discrete "motor codes" for conditioning video diffusion models is a creative extension, though the reliance on the Video2LoRA framework limits the novelty of the generative component itself.
The experimental evaluation is extensive and rigorous. The authors demonstrate consistent improvements over strong baselines (SimCLR, MoCo, BYOL, and ImageNet/MAE initialization) across four distinct task families: action recognition (EPIC-Kitchens, SSv2, Ego-Exo4D), 3D hand reconstruction (ARCTIC, H2O, HOT3D), 3D contact reconstruction, and video generation. The gains are particularly notable in the frozen-backbone setting for action recognition, suggesting the pre-training captures robust, transferable features. The comparison against RGB-only self-supervised methods trained on the same data effectively isolates the contribution of the EMG signal. The qualitative results for video diffusion show clear improvements in fine-grained hand-object interaction fidelity compared to language or video-history conditioning.
The paper provides high reproducibility. It details the specific hardware (MindRove Armband, Meta Aria glasses), data collection protocols, preprocessing steps (filtering, normalization), and model architectures. The release of the 100-hour dataset is a significant asset. However, the specific implementation of the "Video2LoRA" and "HyperNetwork" components relies on external citations, and the code for the VQ-VAE motor code discovery is described but not fully detailed in the main text, potentially requiring access to the supplementary materials or code repository (not explicitly linked in the provided text, though a project page is mentioned) for full replication.
The primary limitation is the scale of the dataset (100 hours) compared to massive video datasets like Kinetics or Ego4D. The EMG hardware used is consumer-grade (8 channels), which may limit the granularity of muscle activity captured compared to research-grade high-density EMG. The video diffusion experiments are limited to a specific backbone (CogVideoX-Fun) and a lightweight LoRA approach, which may not generalize to all state-of-the-art video generation models. Additionally, the dataset is focused on kitchen/household manipulation, which may limit generalizability to other domains.
This work has significant potential impact on embodied AI, robotics, and VR/AR. By providing a dataset and method that links visual perception with motor intent, it offers a pathway to more intuitive human-robot interaction and better understanding of human actions from video. The motor-code conditioning for video generation could enable new tools for content creation and simulation. The dataset itself becomes a valuable resource for the community, addressing a gap in paired video-EMG data. The paper introduces MotorSense, a large-scale paired egocentric video and EMG dataset, and demonstrates that cross-modal sensorimotor alignment significantly enhances action-centric computer vision tasks. By leveraging EMG as a privileged supervisory signal in a CLIP-style pre-training framework, the authors achieve consistent state-of-the-art results in action recognition, 3D hand reconstruction, and contact prediction, while also enabling fine-grained control of video diffusion models through learned motor codes.
Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally. Based on this insight, we introduce SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. SpecSlot expands each spatial anchor into multiple slots, each assigned a disjoint band of DCT coefficients; this separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds a local correction to the spectral prediction, while the generative state, target, and loss remain in pixel space. On ImageNet-256, SpecSlot achieves a comparable final FID to a FLOPs-matched JiT control while converging faster. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within only 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image generation, SpecSlot surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.
Primary: Peking University
All Institutions: Peking University, Nanyang Technological University, SenseTime
SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
The paper proposes SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from global attention sequence length. The core innovation is the "Role-Coupled Tokenization" where each spatial anchor is expanded into multiple slots, each assigned a disjoint band of DCT coefficients. Only the lowest-frequency slot participates in global attention, while high-frequency slots interact locally. This is a clever architectural modification that leverages the spectral properties of natural images (low-frequency structure requires global context, high-frequency detail is local). The method remains in pixel space, avoiding the artifacts often associated with VAE-based latent diffusion, while mitigating the quadratic cost of attention in pixel-space models. The inclusion of a spatial head for local correction is a sensible addition to handle residual errors.
The experiments are conducted on ImageNet-256 and ImageNet-512, as well as text-to-image generation benchmarks (DPG-Bench). The results show that SpecSlot achieves comparable FID to FLOPs-matched JiT controls on ImageNet-256 with faster convergence. On ImageNet-512, it improves FID by 2.84 over the control with fewer FLOPs and comes within 0.69 FID of a larger JiT model at significantly lower cost. The text-to-image results show improvements over JiT-B/8. The comparisons are against strong baselines (JiT), and the efficiency gains are substantial.
The paper provides links to the project page, GitHub repository, and Hugging Face model weights. The methodology is described in sufficient detail in the provided text (though the full technical details are in the sections referenced but not fully expanded in the prompt's text snippet, the abstract and summary provide the core mechanism). The availability of code and weights significantly enhances reproducibility.
The method relies on DCT coefficient bands, which may not be optimal for all types of content or modalities. The local interaction of high-frequency slots might limit the model's ability to capture long-range high-frequency correlations, although this is argued to be less critical. The computational overhead of managing multiple slots per anchor needs careful engineering to ensure the efficiency gains are realized in practice.
This work offers a viable path for pixel-space generative models to scale to higher resolutions without the prohibitive cost of full global attention. It could influence the design of future pixel-space Transformers and potentially other modalities where spectral decomposition is applicable. The efficiency gains make high-quality pixel-space generation more accessible. SpecSlot introduces a role-coupled tokenization scheme for pixel-space generative Transformers that decouples spectral detail from global attention length, achieving significant efficiency gains and competitive performance on ImageNet and text-to-image tasks. The paper presents a well-motivated architectural innovation that addresses a key bottleneck in pixel-space generation, with strong empirical results demonstrating its effectiveness and efficiency compared to state-of-the-art baselines like JiT.
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Primary: Tencent Youtu Lab
All Institutions: Tencent Youtu Lab
Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
The paper proposes Youtu-Parsing-Omni, a unified architecture for parsing documents, images, audio, and video into structured records. The core methodological contribution is the "Youtu-Omni-Encoder," which initializes from a pre-trained language model and uses shared weights to encode all modalities in a unified space-time coordinate system. A key architectural innovation is the inclusion of fusion layers that allow video frames to attend to concurrent audio, integrating cross-modal fusion directly into the perception layer. The training methodology introduces "OmniSchema-Aware On-Policy Distillation" (OSAD), a self-evolving post-training procedure where the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself. This allows the parser and its supervision to co-evolve, unifying the outputs of ten different parsing tasks under a single schema.
The paper reports state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.). The results highlight a significant advantage in audio and video parsing compared to general-purpose systems. It also achieves the best overall chart-parsing score (95.02) and high exact chemical-fingerprint agreement. The model matches dedicated optical-music models in character-level transcription of typeset sheet music. The evaluation covers a wide range of modalities and tasks, demonstrating the versatility of the unified approach.
The paper states that code and model weights are publicly available, which is a strong positive for reproducibility. However, the provided text does not include specific URLs or detailed hyperparameter settings, making it difficult to verify the exact implementation details without accessing the external repository. The description of the OSAD training procedure is high-level, and the specific mechanics of the "reference-conditioned copy" and "token-role-based objective" would require the code to fully understand and replicate.
The paper relies heavily on the performance of the pre-trained language model initialization, which may limit its applicability to domains where text is not the primary signal. The "self-evolving" nature of the OSAD training could be computationally expensive and potentially unstable, though this is not explicitly discussed. The evaluation is primarily on benchmark datasets, and real-world robustness to noisy or unstructured data is not thoroughly tested. The lack of specific URLs in the text prevents immediate verification of the claimed open-source availability.
This work has significant potential to impact the field of multimodal AI by providing a unified perception layer for agents that need to read, locate, and cite evidence across multiple modalities. The ability to parse documents, images, audio, and video into structured records using a single encoder and schema could simplify the development of complex AI agents. The focus on spatial and temporal grounding is particularly relevant for applications in robotics, autonomous systems, and advanced document analysis. Youtu-Parsing-Omni introduces a unified encoder and schema for omni-modal parsing, achieving state-of-the-art results on multiple benchmarks. The paper presents a novel architecture that integrates cross-modal fusion into the perception layer and a self-evolving training procedure, demonstrating strong performance across documents, images, audio, and video.
We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni introduces two major innovations. First, we propose a unified cross-paradigm on-policy distillation (OPD) framework that successfully transfers the visual understanding ability of autoregressive VLMs and visual generation ability of continuous flow models, to a unified DLLM. Second, NLD-Uni incorporates a novel progressive self-refining diffusion process that gradually corrupts each token across multiple steps during training, as opposed to adopting the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality. Through extensive experiments, we show that NLD-Uni achieves state-of-the-art performance among unified DLLMs on diverse tasks including visual question-answering, multi-modal reasoning, object grounding, text-to-image generation, and image editing.
Primary: NVIDIA
All Institutions: NVIDIA
The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
The paper proposes NLD-Uni, a unified discrete diffusion large language model (DLLM) for visual tasks. The core methodological contribution is a "unified cross-paradigm on-policy distillation" framework. This approach attempts to transfer capabilities from two distinct source models—autoregressive VLMs (for understanding) and continuous flow models (for generation)—into a single discrete diffusion target. The second major innovation is a "progressive self-refining diffusion process." Unlike standard masked diffusion, which corrupts tokens to a uniform mask, this method gradually corrupts tokens across multiple steps during training. At inference, this allows for intermediate token commitments, theoretically enabling self-refinement. The methodology is complex and addresses a significant gap in unifying generation and understanding in discrete spaces, though the reliance on distillation from potentially proprietary or large teacher models adds complexity.
The paper claims state-of-the-art performance among unified DLLMs on VQA, multi-modal reasoning, object grounding, text-to-image generation, and image editing. However, the provided text is a placeholder ("sections/intro", "sections/experiments") and does not contain actual data, tables, or specific benchmark scores. Without concrete evidence of the "state-of-the-art" claims, the experimental evaluation cannot be rigorously verified. The scope of tasks (both understanding and generation) is broad, which is a strong point if the results hold up, but the lack of visible data in the prompt prevents a high confidence in the empirical rigor.
The paper describes specific training objectives (on-policy distillation, progressive corruption). However, without access to the full code, hyperparameters, and the specific teacher models used for distillation, reproducibility is moderate to low. The "Nemotron" branding suggests an internal NVIDIA model, which may not be fully open-source, limiting external reproducibility.
The primary limitation is the opacity of the experimental results in the provided text. Additionally, unified models often suffer from a "seesaw" effect where improving generation hurts understanding or vice versa; the paper claims to overcome this via distillation, but the trade-offs are not detailed in the abstract. The reliance on discrete tokens for image generation may still lag behind continuous diffusion/flow models in terms of raw fidelity, even with self-refinement.
If successful, this work could significantly advance the field of unified multimodal models by demonstrating that discrete diffusion can effectively handle both high-level reasoning and low-level pixel generation. This could lead to more efficient, single-model architectures for multimodal AI, reducing the need for separate VLMs and image generators. The paper introduces a unified discrete diffusion LLM that leverages cross-paradigm on-policy distillation and progressive self-refinement to achieve competitive performance in both visual understanding and generation tasks. While the methodological approach is innovative and addresses a key challenge in unifying autoregressive and flow-based paradigms, the lack of visible experimental data in the provided text limits the ability to fully verify the claimed state-of-the-art results, resulting in a high but cautious score.
Generative spoken language models (SLMs) offer a route to learning linguistic structure purely from speech, yet they lag far behind text-based models in both capability and data scaling efficiency. We identify two bottlenecks underlying this gap: the limited linguistic quality of existing speech units and the reliance on a single stream of discrete units to represent speech. Multiple codebooks are widely used to encode acoustic features but rarely to represent speech semantics. We present GSLM2, which learns and models complementary properties of speech at multiple levels of abstraction. Its encoder, SpidR 2.0, uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A multi-stream SLM then jointly models these units using a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. Across training data sizes and model scales, up to 6M hours and 6.5B parameters, GSLM2 consistently improves linguistic competence while scaling more efficiently with data. As a result, it substantially narrows the gap between spoken and text-based LMs; across read and conversational speech domains, we shrink the gap by about two orders of magnitude on sWuggy, sBLIMP, and tSC, while also outperforming competitive speech-text systems on all three tasks. We open-source the training code and model checkpoints at https://github.com/facebookresearch/gslm2.
Primary: Meta Superintelligence Labs
All Institutions: Meta Superintelligence Labs
GSLM2 introduces a multi-stream spoken language modeling framework that learns complementary discrete units at multiple levels of abstraction via differentiable self-supervised pretraining, significantly narrowing the data-efficiency gap between speech and text language models. The paper presents a rigorous and scalable approach to textless language modeling, demonstrating that leveraging multi-level representations from self-supervised encoders allows SLMs to achieve linguistic competence comparable to text LMs with far less data, offering a promising path for speech-native AI applications.
The paper introduces GSLM2, a framework that addresses the data-efficiency gap between spoken language models (SLMs) and text-based LMs. The core methodological contribution is twofold: (1) SpidR 2.0, an encoder that uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels of abstraction (layers) rather than relying on post-hoc k-means on a single layer; and (2) a multi-stream SLM architecture that jointly models these complementary unit streams using a shared temporal backbone, stream-specific predictors, and an autoregressive generative head. The use of entropy regularization to stabilize the code distributions across layers is a significant technical improvement over prior work like SpidR. The approach effectively decouples the representation of fine-grained phonetic details from higher-level semantic regularities, allowing the model to leverage both for efficient language modeling.
The experiments are extensive and rigorous. The authors scale models up to 6.5B parameters and 6M hours of data, which is a substantial computational effort. They demonstrate that GSLM2 shrinks the gap to text-based LMs by about two orders of magnitude on key benchmarks (sWuggy, sBLIMP, tSC). The comparison against previous SLMs (including Moshi) shows clear superiority in parameter and data efficiency. The generative evaluation, while relying on ASR and LLM judges (which introduces noise), still shows consistent improvements in coherence and perplexity. The ablation studies on layer-wise representations provide strong evidence for the multi-level hypothesis.
The paper is highly reproducible. The authors explicitly state they open-source the training code and model checkpoints. The reproducibility statement is detailed, covering datasets, preprocessing, model configurations, and evaluation protocols. The use of standard datasets (Libri-Light, LibriSpeech) and public benchmarks further enhances reproducibility.
The paper acknowledges that it does not explicitly preserve acoustic properties like prosody, which is crucial for speech resynthesis. Multi-stream modeling increases computational cost. The evaluation of generation quality relies on indirect metrics (ASR-PPL, LLMJ) which may conflate language modeling errors with TTS/ASR errors. The analysis of how individual streams contribute to the final performance is limited to empirical probing.
This work has significant implications for the development of speech-first AI systems. By demonstrating that linguistic structure can be learned efficiently from speech alone using multi-level representations, it challenges the prevailing reliance on text supervision for speech models. This could lead to more robust, language-agnostic speech models that do not require written language resources. The open-sourcing of the code and models will likely accelerate research in this area. GSLM2 introduces a multi-stream spoken language modeling framework that learns complementary discrete units at multiple levels of abstraction via differentiable self-supervised pretraining, significantly narrowing the data-efficiency gap between speech and text language models. The paper presents a rigorous and scalable approach to textless language modeling, demonstrating that leveraging multi-level representations from self-supervised encoders allows SLMs to achieve linguistic competence comparable to text LMs with far less data, offering a promising path for speech-native AI applications.
Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the victim's replies, wasting turns and limiting transferability; interactive generators read each victim reply to generate the next-turn attack but require in-loop evaluators for success signals or hints, incurring substantial costs and inflated performance. Bridging this gap, we propose SEMA-Auto, a sighted and efficient multi-turn attacker with autonomy, interacting with the victim model and terminating the session by itself upon success. To stabilize its behaviors and boost its capabilities, SEMA-Auto is trained in two stages. Synthetic bootstrap tuning installs the notion of attack targets and conforming decisions by fine-tuning the base model with bootstrapped trajectories. Decision-supervised reinforcement learning then optimizes two objectives jointly yet separately, reinforcing the attack with higher jailbreak reward while supervising the decision at each turn against an absolute target. Our method achieves state-of-the-art (SOTA) attack success rates (ASR) across multiple datasets, victims, and judges, outperforming all static and interactive baselines, within almost the fewest turns, demonstrating both effectiveness and efficiency. For example, SEMA-Auto performs a $76.0\%$ ASR@1 within an average of $3.0$ turns against DeepSeek-V4-Flash on AdvBench, while the best static baseline is $54.8\%$ with fixed $7$ turns and the best interactive baseline is $43.8\%$ in $3.7$ turns. Against safety-hardened frontier models, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, and GPT-6-Astra, our counter-trained attacker achieves an average $59.6\%$ ASR@1 on HarmBench, $13.8\%$ over prior SOTA. Our code is available at: https://github.com/microsoft/SEMA
Primary: Microsoft Research
All Institutions: Microsoft Research
SEMA-Auto introduces a sighted, efficient, and autonomous multi-turn jailbreak attacker that achieves SOTA performance by combining synthetic bootstrap tuning with decision-supervised reinforcement learning. The paper presents a rigorous methodology that addresses the inefficiencies of existing interactive attackers by enabling autonomous termination, resulting in significant improvements in attack success rates and efficiency across diverse victim models and datasets.
The paper proposes SEMA-Auto, a multi-turn jailbreak attacker that bridges the gap between static planners (which lack adaptability) and interactive generators (which are costly and rely on in-loop evaluators). The core innovation lies in a two-stage training pipeline: (1) Synthetic Bootstrap Tuning to instill attack targets and decision-making logic via bootstrapped trajectories, and (2) Decision-Supervised Reinforcement Learning (DSRL). The DSRL stage is particularly notable for jointly optimizing two separate objectives: maximizing jailbreak reward and supervising the termination decision against an absolute target. This allows the model to autonomously determine when an attack has succeeded without requiring an external judge during inference, thereby reducing cost and latency. The method effectively addresses the "sighted" aspect by reading victim replies while maintaining efficiency through autonomous termination.
The experimental section reports State-of-the-Art (SOTA) Attack Success Rates (ASR) across multiple datasets (AdvBench, HarmBench) and victim models (DeepSeek-V4-Flash, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, GPT-6-Astra). The results show significant improvements over both static and interactive baselines. For instance, on AdvBench against DeepSeek-V4-Flash, SEMA-Auto achieves 76.0% ASR@1 in an average of 3.0 turns, compared to 54.8% for the best static baseline (7 turns) and 43.8% for the best interactive baseline (3.7 turns). Against safety-hardened frontier models, it achieves an average 59.6% ASR@1 on HarmBench, outperforming prior SOTA by 13.8%. The evaluation covers a wide range of model capabilities and safety levels, providing a robust assessment of the method's transferability and effectiveness.
The authors provide a link to their code repository (https://github.com/microsoft/SEMA), which is a positive indicator for reproducibility. The methodology is described in sufficient detail, including the two-stage training process and the specific objectives of the reinforcement learning phase. However, as is common with preprints under review, some specific hyperparameters or implementation details might be found in the appendix (referenced but not fully visible in the provided text snippet). The use of standard benchmarks and open-source models (where applicable) further aids reproducibility.
The paper focuses on jailbreak attacks, which is a sensitive area. While the method is efficient, the reliance on synthetic bootstrap data may introduce biases or limit the diversity of attack strategies. The autonomous termination mechanism, while efficient, might occasionally lead to premature termination if the model's internal confidence is miscalibrated. Additionally, the performance against "safety-hardened frontier models" suggests that while the method is effective, it is not infallible, and the gap between high-performing and low-performing victims remains significant. The paper does not extensively discuss the potential for defensive mechanisms to counter this specific type of autonomous, efficient attacker.
This work has significant implications for LLM safety and security. By demonstrating a more efficient and autonomous jailbreak method, it highlights the limitations of current safety measures and the need for more robust defenses. The efficiency of the method (fewer turns, no in-loop evaluators) makes it a more practical threat, potentially lowering the barrier to entry for malicious actors. Conversely, the techniques developed for autonomous decision-making in adversarial contexts could be adapted for defensive purposes, such as creating more effective red-teaming tools or safety evaluators. The paper contributes to the ongoing arms race in AI safety, pushing the field to develop more resilient models. SEMA-Auto introduces a sighted, efficient, and autonomous multi-turn jailbreak attacker that achieves SOTA performance by combining synthetic bootstrap tuning with decision-supervised reinforcement learning. The paper presents a rigorous methodology that addresses the inefficiencies of existing interactive attackers by enabling autonomous termination, resulting in significant improvements in attack success rates and efficiency across diverse victim models and datasets.
Individual-level genotype data underpin genetic discovery, disease-risk modeling, and therapeutic development, yet many critical settings remain data-limited, including underrepresented ancestries, rare or imbalanced disease cohorts, and restricted-access biobanks. Conditional synthetic genome generation would support privacy-aware benchmarking, simulation, and data augmentation, but requires learning an ultra-high-dimensional discrete distribution over hundreds of thousands to tens of millions of correlated single-nucleotide polymorphisms (SNPs), or sites of DNA variation. We introduce HELICS: High-fidelity gEneration via Latent Interpolant Conditional Synthesis, a phasing-free latent flow matching framework for genome-wide SNP-level generation from biobank-scale genotype data. HELICS tokenizes chromosomes with convolutional autoencoders, reducing >600K SNPs to 4,096 continuous latent tokens, and trains a transformer parameterized conditional flow matching model over the concatenated multi-chromosome latent representation. Trained on over 417K genomes in the UK Biobank, HELICS generates genome-wide common-variant synthetic cohorts without haplotype phasing and achieves stronger fidelity to real genotypes than existing haplotype-based simulators, including high allele-frequency agreement in external evaluation and superior preservation of local and long-range SNP correlation structure. Conditioning on ancestry and genetic liability summaries enables targeted generation across genetic risk profiles; in perturbation analyses, by increasing the conditioned disease trait liability, HELICS induces SNP-level dosage changes in generated genomes that are highly correlated with published association weights. HELICS provides a scalable generative modeling framework for SNP-level genome data and a path toward conditional synthetic cohorts for data-limited genetic studies.
Primary: Harvard University
All Institutions: Harvard University
HELICS introduces a scalable, phasing-free latent flow matching framework for genome-wide synthetic genome generation that achieves high fidelity to real biobank data and enables controllable generation of genetic liability profiles, representing a significant advancement in computational genomics and generative modeling for high-dimensional discrete biological data.
The paper introduces HELICS, a framework for generating synthetic genome-wide SNP data using latent flow matching. The core methodological innovation is the decoupling of the ultra-high-dimensional discrete genotype space (600K+ SNPs) into a manageable continuous latent space via chromosome-specific convolutional autoencoders. This allows the application of a transformer-based flow matching model (USiT) to the concatenated latent tokens of all 22 autosomes. The approach is "phasing-free," meaning it operates directly on unphased dosage data, avoiding the computationally expensive and error-prone step of haplotype phasing required by traditional simulators like HAPGEN2 or HAPNEST. The autoencoder training objective is carefully designed to preserve population genetic structure, including terms for minor allele frequency (MAF) preservation and local linkage disequilibrium (LD) correlation. The flow matching model is conditioned on ancestry principal components and discretized polygenic risk scores (PRS), enabling controllable generation of genetic liability profiles.
The evaluation is rigorous and extensive. The model is trained on the UK Biobank (417K genomes) and evaluated on external cohorts (1000 Genomes, CCDG). Key metrics include allele frequency agreement ($R^2=0.992$), LD structure preservation, and population geometry (PCA/UMAP). A particularly strong aspect of the evaluation is the "perturbation analysis," where the authors demonstrate that increasing the conditioned disease liability (e.g., HDL) induces SNP-level dosage changes that correlate highly with published GWAS association weights ($R^2=0.547$ for high-effect variants). This provides strong evidence that the model learns biologically meaningful associations rather than just statistical correlations. The model outperforms existing haplotype-based simulators in fidelity and offers capabilities (conditional generation) that baselines lack.
The authors provide a GitHub repository with code. However, due to the sensitivity of UK Biobank data, the full model weights and training data are not released. A demo is provided using 1000 Genomes data on chromosome 22, which allows for verification of the pipeline but not the full-scale performance claims. The architectural details, including the dynamic programming for token allocation and specific loss terms, are described in detail in the appendix, aiding reproducibility for those with access to similar-scale data.
The model is currently limited to common variants (MAF > 1%) and does not handle rare variants or structural variants. Fidelity is weaker for underrepresented ancestry groups, likely due to data imbalance in the training set. The conditioning is based on coarse PRS deciles, which may limit the granularity of control. The privacy implications of synthetic data generation are discussed but not exhaustively proven; while membership inference attacks show low leakage, the risk of re-identification in specific contexts remains a topic for further study.
This work has significant potential to impact genetic research by enabling the creation of synthetic cohorts for underrepresented populations or rare diseases, where real data is scarce. It facilitates privacy-aware sharing of genetic data and allows for controlled simulation of genetic risk profiles. The ability to generate data conditioned on specific genetic liabilities could accelerate the discovery of new genetic associations and improve the generalizability of polygenic risk scores. HELICS introduces a scalable, phasing-free latent flow matching framework for genome-wide synthetic genome generation that achieves high fidelity to real biobank data and enables controllable generation of genetic liability profiles, representing a significant advancement in computational genomics and generative modeling for high-dimensional discrete biological data.
Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.
Primary: Unknown
All Institutions: Unknown
The paper introduces a novel "WorldSketch" paradigm where VLMs construct minimal kinematic world models for robotic planning, achieving state-of-the-art zero-shot success rates in manipulation and navigation tasks while significantly reducing token usage compared to code-generation agents. By decoupling semantic scene understanding from geometric motion planning, the method demonstrates that explicit, verifiable world models built by VLMs can outperform end-to-end action generation, offering a scalable and interpretable approach to general-purpose robotics.
The paper proposes "WorldSketch," a paradigm where a Vision-Language Model (VLM) constructs a minimal, kinematic world model (sketch) rather than generating actions directly. The VLM acts as an agent querying perception tools (depth lifting, point marking) to build a model of task-relevant entities (joints, links, parts). A geometric planner then optimizes contact and whole-body motion on this explicit model. The method includes a self-improving loop where execution feedback verifies outcomes and updates joint hypotheses via Bayes' rule. The methodology is sound, leveraging the VLM's semantic understanding for scene abstraction and the planner's precision for geometry. The separation of concerns (VLM for "what matters," Planner for "how to move") is a strong architectural choice that addresses the limitations of end-to-end VLA models and code-as-policy agents.
The evaluation is extensive, covering RoboCasa (kitchen manipulation), LIBERO-PRO (tabletop), and R2R-CE (navigation). The results are impressive, with 91% success on RoboCasa zero-shot, significantly outperforming baselines like CaP-X (33%) and Code as Policies (9%). The ablation studies are rigorous, isolating the impact of grounding methods (points vs. masks), overlay checks, and hypothesis testing. The comparison against trained VLAs (GR00T, pi0.5) demonstrates the superiority of the zero-shot agentic approach in generalization. The token efficiency analysis (10x fewer tokens than CaP-X) is a valuable practical insight.
The paper provides detailed descriptions of the sketch language, tool interfaces, and planner components. However, the system relies on proprietary VLMs (GPT-6.1-Sol, GPT-6-Astra) and specific simulation environments (RoboCasa365, MuJoCo). The exact prompts and backend logic are described but not fully open-sourced in the text provided. The reliance on specific VLM versions may limit immediate reproducibility for those without access to these models, but the architectural principles are clearly defined.
The system is evaluated primarily in simulation. Real-world sensor noise (depth errors >1cm) degrades performance. The current sketches cover only rigid parts with single joints, excluding deformable objects and complex multi-joint chains. The VLM latency (15s median) adds significant time to the loop. Contact-rich execution on small/thin parts remains a challenge, with failures often due to gripper limitations rather than planning errors.
This work offers a compelling alternative to the dominant VLA trend, suggesting that explicit world modeling by VLMs can achieve superior zero-shot generalization with lower computational cost. It bridges the gap between high-level semantic reasoning and low-level geometric planning. The paradigm of "sketching" the world is intuitive and could be extended to other robotic domains, potentially influencing how future agentic systems are designed to interact with physical environments. The paper introduces a novel "WorldSketch" paradigm where VLMs construct minimal kinematic world models for robotic planning, achieving state-of-the-art zero-shot success rates in manipulation and navigation tasks while significantly reducing token usage compared to code-generation agents. By decoupling semantic scene understanding from geometric motion planning, the method demonstrates that explicit, verifiable world models built by VLMs can outperform end-to-end action generation, offering a scalable and interpretable approach to general-purpose robotics.
A robot's next action can depend on recent motion or earlier visual evidence absent from its current observation. A recurrent visual-memory architecture can retain this history in a visual state, but its usefulness depends on what the state represents and how it is updated with new observations. We introduce Prediction-Regulated Memory (PREM), a framework that uses action-conditioned future prediction to guide both memory learning and updating in vision-language-action policies. During training, predicting future visual features provides temporal supervision for memory learning. During execution, PREM compares current visual features with the prediction made at the preceding decision and uses the spatially normalized prediction discrepancy to regulate updates to the visual state at each spatial location. The updated visual state supplies temporal context for action generation alongside current observations. Built on $π_{0.5}$, PREM improves average success rate from 9.63% to 20.74% on DOMINO dynamic manipulation tasks and from 14.4% to 83.8% on the RMBench M(1) memory-dependent setting. Further evaluations on LIBERO and LIBERO-Pro show improvements in general manipulation and robustness to spatial perturbations. PREM incurs a mean inference overhead of 7.7 ms on a single NVIDIA H800 GPU.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces a novel memory regulation mechanism for VLA policies that leverages action-conditioned future prediction to improve temporal context retention, demonstrating significant performance gains on memory-dependent and dynamic manipulation benchmarks with minimal computational overhead.
The paper proposes PREM (Prediction-Regulated Memory), a framework that integrates action-conditioned future prediction into the memory mechanism of Vision-Language-Action (VLA) policies. The core innovation is twofold: during training, future visual feature prediction serves as a temporal supervision signal for the memory state; during inference, the discrepancy between the predicted future visual state and the actual current observation is used to spatially regulate the update of the recurrent memory. This approach addresses the common issue in robotic manipulation where the current observation lacks sufficient context about recent dynamics or hidden states. By leveraging prediction errors to modulate memory updates, the method aims to retain relevant historical information while discarding noise. The method is built upon the $\pi_{0.5}$ architecture, a strong baseline in the field.
The experimental evaluation is robust, covering multiple benchmarks: DOMINO (dynamic manipulation), RMBench (memory-dependent tasks), LIBERO, and LIBERO-Pro. The reported improvements are substantial, particularly on RMBench M(1), where success rate jumps from 14.4% to 83.8%, and on DOMINO, from 9.63% to 20.74%. These results suggest the method effectively handles tasks requiring temporal context. The inclusion of LIBERO-Pro evaluations for robustness to spatial perturbations adds depth to the generalization claims. The inference overhead of 7.7 ms on an H800 GPU is reported as negligible, which is a practical advantage for real-time control.
The paper includes a detailed reproducibility statement, promising code and configuration files in the supplementary material. It specifies architectural dimensions, hyperparameters, and training schedules in the appendix. The use of standard benchmarks (LIBERO, DOMINO, RMBench) facilitates independent verification. The AI use statement is transparent, clarifying that AI was used only for editing and formatting, not for data generation or hypothesis formulation.
The primary limitation is the reliance on the $\pi_{0.5}$ backbone; it is unclear how well the PREM module generalizes to other VLA architectures or non-recurrent memory structures. The evaluation is limited to simulation and specific real-world setups (implied by the benchmarks), and long-horizon real-world deployment challenges (e.g., lighting changes, object deformations) are not extensively explored. The "prediction" mechanism assumes a certain degree of predictability in the environment, which may not hold in highly stochastic or adversarial settings.
This work contributes to the broader goal of creating robots that can learn from experience and maintain context over time. The technique of using prediction errors to regulate memory updates is a concept that could be adapted to other sequential decision-making tasks beyond robotics, such as autonomous driving or interactive agents. It addresses a fundamental challenge in embodied AI: how to efficiently compress and update historical information for future action planning. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces a novel memory regulation mechanism for VLA policies that leverages action-conditioned future prediction to improve temporal context retention, demonstrating significant performance gains on memory-dependent and dynamic manipulation benchmarks with minimal computational overhead.
Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy's training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.
Primary: Stanford University
All Institutions: Stanford University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces RoboArchitect, a method that automates the design of agent harnesses around frozen robot policies using iterative coding LLM optimization, demonstrating that substantial performance gains in robotic manipulation can be achieved without weight updates by evolving control logic and persistent memory, effectively turning general policies into specialists for specific setups.
The paper proposes "RoboArchitect," a framework that automates the design of an "agent harness" surrounding a frozen Vision-Language-Action (VLA) policy. The core methodology involves an iterative optimization loop where a coding LLM (Claude Opus 5) observes robot rollouts, diagnoses failure modes, and rewrites the control code (harness) to mediate between the environment and the frozen policy. The harness is not a fixed architecture but evolves to include specific control routines (e.g., jam recovery, force rescaling) and persistent memory structures. The approach is distinct from standard fine-tuning as it updates no model weights, relying instead on programmatic logic and external tool calls (VLMs, detectors). The novelty lies in treating the agent's control logic as an evolvable software artifact rather than a static wrapper, allowing the system to dynamically decide when to trust the policy vs. direct control.
The experiments are extensive, covering multiple benchmarks (LIBERO-Pro, RoboCerebra, VLABench, RoboCasa, SimplerEnv) and multiple policies (OpenVLA, Pi-0, MolmoAct-2, etc.). The evaluation protocols are rigorous, distinguishing between "unseen scenes" (same tasks, new layouts) and "unseen tasks" (new tasks, shared skills). The results show significant improvements (e.g., 9.7% to 43.3% on unseen scenes), and notably, the evolved harness often outperforms the fine-tuned version of the same policy in out-of-domain settings. The ablation on iterative refinement vs. one-shot design confirms the value of the self-improvement loop. The emergence of persistent memory (e.g., storing grasp depths) is a strong qualitative and quantitative finding.
The paper provides a GitHub link and detailed implementation sections. It specifies the exact coding model (Claude Opus 5), auxiliary tools (Qwen3, Molmo2, Grounding-DINO, SAM 3), and iteration limits. The cost analysis (<$150, <6 hours) is a strong point for practical reproducibility. However, the heavy reliance on a specific, proprietary coding model (Claude) may limit reproducibility for labs without access to that specific model version or API.
The primary limitation is that all evaluations are in simulation. The paper acknowledges this and suggests the interface should transfer to real robots, but no real-world validation is provided. Additionally, the method is capped at 5 iterations, and the saturation point is not fully explored. The complexity of the harness code could become a maintenance burden, though the paper argues it is interpretable. The reliance on a powerful coding LLM makes the method expensive in terms of API costs, though the paper argues this is cheaper than collecting new demonstrations.
This work has high potential impact by offering an alternative to the data-hungry fine-tuning paradigm in robotics. If the "harness optimization" approach scales to real-world robots, it could drastically reduce the cost of adapting robot policies to new environments. It bridges the gap between code-as-policy and learned VLA policies, suggesting a hybrid future where LLMs write the control logic around frozen neural networks. The concept of "evolving" the agent's software interface is a significant shift in how robotic agents are deployed. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces RoboArchitect, a method that automates the design of agent harnesses around frozen robot policies using iterative coding LLM optimization, demonstrating that substantial performance gains in robotic manipulation can be achieved without weight updates by evolving control logic and persistent memory, effectively turning general policies into specialists for specific setups.
Realistic, fast robotics simulations could enable large-scale training of embodied AI models that transfer to the real world. However, despite decades of progress, available simulations cover only a narrow range of real-world tasks and physical phenomena. The core bottleneck is that creating better simulations requires significant expert effort and therefore does not scale. To address this, we introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations. Our core insight is that simulation creation is a coding agent task, so improvements in coding agents translate directly into richer and more faithful simulations. We select simulations based on U.S. GDP data to cover economically valuable physical work, define what makes a good simulation and a good task, and develop an adversarial creation-audit loop that checks each simulation and task against these definitions. Using this pipeline, we build RoboWorld, a set of 25 simulations with 25 test and 175 training RL environments of varying difficulty. Even the best frontier models fail on every RoboWorld test task, achieving an average reward of only 0.06. Building on RoboWorld, we introduce several new ways for frontier models to improve, including in-context learning across repeated attempts, a privileged teacher that reads the simulator state and teaches a student in context, self-assisted learning from the model's own privileged experience, and code policies that generate demonstrations across varied scenes to train vision-language-action models (VLAs) such as $π_{0.5}$. Each of these substantially improves performance. We release our pipeline, RL environments, and benchmark to facilitate future research. Code and data are available at https://gym-anything.github.io/robotics
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University
The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
The paper introduces "Gym-Anything for Robotics," a framework leveraging coding agents to automatically generate realistic robotics simulations. The core methodological innovation is the "adversarial creation-audit loop," where an agent creates a simulation and a separate verifier checks it against rigorous physical and task definitions. The paper demonstrates this by building "RoboWorld," a suite of 25 simulations covering economically valuable physical tasks (e.g., welding, sewing, fluid mixing, forklift operation). The simulations are notably detailed, incorporating specific physical laws (e.g., Hertz/Hunt-Crossley contact, Drucker-Prager powder mechanics, Bingham thin-film lubrication) and real-world object parameters (datasheet dimensions, material properties). The learning component explores several novel training strategies, including in-context learning across attempts, privileged teacher-student setups, and code policies for generating demonstrations for Vision-Language-Action (VLA) models.
The experimental evaluation is extensive in scope, covering 25 distinct simulation domains. The paper reports that even frontier models fail on the test tasks, achieving an average reward of only 0.06, which highlights the difficulty and realism of the benchmark. The authors demonstrate that their proposed learning methods (in-context learning, privileged teaching, code policies) substantially improve performance over baselines. The comparison table with prior simulations is detailed, showing that RoboWorld simulations include significantly more physics checks and adversarial evaluator cases than prior work. However, the evaluation is limited to simulation performance; there is no sim-to-real transfer evaluation, which is a critical gap for a robotics paper claiming to enable real-world training.
The authors state that the pipeline, RL environments, and benchmark are released. The project URL is provided. The level of detail in the simulation descriptions (e.g., specific solver parameters, material properties, robot configurations) suggests high reproducibility for the simulation setup. However, the reliance on coding agents for generation introduces variability, and the exact prompts or agent configurations used to generate the 25 simulations are not fully detailed in the provided text, which may limit exact reproduction of the generation process.
The primary limitation is the lack of sim-to-real validation. The paper argues that realistic simulations enable transfer, but does not demonstrate this. The benchmark is extremely difficult (average reward 0.06), which may be a barrier for adoption if the learning curves are not well-characterized. The reliance on coding agents for simulation creation may introduce subtle errors or biases that are not fully captured by the adversarial audit loop. The paper is also very long and complex, which may hinder accessibility.
This work has high potential impact by addressing a fundamental bottleneck in robotics research: the scalability of simulation creation. If the framework can be extended to generate thousands of high-fidelity simulations, it could enable large-scale pre-training of embodied AI models. The focus on economically valuable tasks (GDP-based selection) aligns with practical industrial applications. The release of the benchmark and pipeline will likely stimulate research in simulation-based RL and embodied AI. The paper presents a novel framework for scaling robotics simulation creation using coding agents, resulting in a high-fidelity benchmark (RoboWorld) that exposes the limitations of current frontier models. While the lack of sim-to-real transfer is a significant gap, the technical depth of the simulations and the innovative use of LLMs for simulation generation represent a substantial contribution to the field.
Future planetary rovers will traverse steep, deformable terrain where slip, sinkage, and terrain deformation can threaten mobility. Although simulation can capture some wheel-soil effects, unmodeled deformable-terrain phenomena create sim-to-real gaps requiring deployment-time adaptation. Yet, rover operations leave little room for extensive trial-and-error. We introduce TERRA (Targeted Exploration for Reward-Relevant Adaptation), an online adaptation framework that efficiently seeks deployment data exposing task-relevant model errors and corrects them. TERRA adapts a simulation-trained world model with a reward-weighted prediction loss and trains a loss predictor to estimate where future performance-relevant errors are likely to occur, using this signal to guide exploration during MPC. In simulation, TERRA reaches approximately 90% slope-climbing success within five trials under severe out-of-distribution pretraining conditions, outperforming exploration baselines in reward, completion time, and energy to first success. On the ERNEST rover, TERRA recovers coordinated active-suspension locomotion on a deformable Mars-simulant slope within three to five real-world trials. Project website: https://terra-explore.github.io
Primary: Jet Propulsion Laboratory, California Institute of Technology
All Institutions: Jet Propulsion Laboratory, California Institute of Technology, University of Texas at Austin
TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
The paper introduces TERRA, a framework for online adaptation of simulation-trained world models for planetary rovers. The core methodological contributions are twofold: (1) a reward-weighted prediction loss that prioritizes model errors in observation dimensions that have high sensitivity to the task reward, and (2) a learned "loss predictor" that forecasts where these reward-relevant errors are likely to occur in candidate action sequences, thereby guiding exploration within a Model Predictive Control (MPC) loop. The use of Low-Rank Adaptation (LoRA) for efficient onboard updating of the dynamics model is a practical engineering choice. The approach is logically sound, addressing the specific bottleneck of sim-to-real gaps in deformable terrain where standard prediction-error-based exploration is inefficient.
The evaluation is strong, featuring both extensive simulation studies (100 seeds, multiple OOD pretraining conditions) and real-world validation on the ERNEST rover. The simulation results show significant improvements over baselines like RND and Plan2Explore in terms of success rate, completion time, and energy efficiency. The real-world results are particularly compelling, demonstrating the recovery of coordinated active-suspension locomotion within 3-5 trials on a Mars-simulant slope, a task that is critical for future missions. The ablation studies effectively isolate the contributions of the reward-weighted loss and the targeted exploration.
The paper provides detailed descriptions of the simulation environment, pretraining conditions, and hyperparameters. However, as is common in robotics papers involving proprietary hardware (ERNEST) and specific simulators (DARTS), full reproducibility by external groups may be limited. The project website is provided, which likely contains code or further details.
The method relies on the availability of a task reward during deployment, which may not always be easily defined or observable in complex real-world scenarios. The choice of loss-bin thresholds for the loss predictor is a hyperparameter that requires tuning. The evaluation is limited to a single rover platform (ERNEST) and a specific type of terrain (deformable slopes), so generalizability to other platforms or terrain types is not fully established.
This work has high relevance to the field of planetary robotics and autonomous systems operating in unstructured environments. The technique of reward-weighted model adaptation and targeted exploration is applicable beyond rovers to any mobile robot facing sim-to-real gaps in contact-rich or deformable environments. It provides a practical pathway for deploying learned policies in high-stakes, low-trial-count scenarios. TERRA introduces a targeted exploration framework for efficient online adaptation of rover world models. The paper presents a rigorous and impactful solution to the sim-to-real gap in planetary rover mobility, combining reward-weighted dynamics learning with a loss-predictor-guided exploration strategy that demonstrates significant efficiency gains in both simulation and real-world hardware trials.
Discretizing continuous actions into discrete skills using methods like VQ-VAE has emerged as a powerful paradigm for robotic manipulation. However, quantization errors yield a suboptimal distribution for the skill prior, degrading its performance. While reinforcement learning offers a path for refinement, its direct application faces unstable encoder updates and a granularity dilemma in importance sampling. To address these challenges, we introduce Cascaded Skills Optimization (CSO), a two-stage post-training framework. First, Rejection-Sampling Supervised Fine-tuning aligns the model's observation-to-skill mapping with the distribution of successful online trajectories via supervised fine-tuning. Second, with the observation encoder frozen, Skills Policy Optimization computes an independent, clipped importance ratio for each complete skill, enabling stable and efficient updates. With a lightweight QueST-style policy, CSO raises overall LIBERO success from 80.1% to 97.5%, improves performance on RoboTwin2.0, and achieves 61.0% success across five physical single-arm tasks. Additional comparisons with FAST, discrete BEAST, and fixed-buffer IQL further validate the effectiveness of our two-stage post-training strategy.
Primary: Unknown
All Institutions: Unknown
[One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
The paper proposes CSO (Cascaded Skills Optimization), a two-stage post-training framework for refining skill-based robotic policies. The methodology is logically sound and addresses specific pain points in applying RL to discrete skill policies. Stage 1 (R-SFT) uses rejection sampling to collect successful trajectories and fine-tunes the observation encoder, which is then frozen. This is a pragmatic solution to the instability of updating visual encoders with sparse rewards. Stage 2 (SPO) introduces a skill-level importance sampling ratio, arguing that token-level (too fine) and episode-level (too coarse) granularities are suboptimal for robotic skills. The theoretical justification for the "granularity dilemma" is the core intellectual contribution. The approach is essentially a structured application of PPO variants (specifically GSPO/GRPO concepts) adapted to the hierarchical structure of skill-based policies. While the adaptation is clever, the underlying RL machinery is not new.
The experimental evaluation is rigorous and comprehensive. The authors test on standard simulation benchmarks (LIBERO, RoboTwin2.0, MetaWorld) and, crucially, on physical robots (Cobot Agilex ALOHA). The results show significant improvements over the imitation learning baseline (80.1% to 97.5% on LIBERO) and outperform strong baselines like RIPT and FAST. The ablation studies effectively isolate the contributions of R-SFT and SPO, and the granularity analysis convincingly demonstrates the superiority of skill-level optimization over token- and episode-level alternatives. The physical robot experiments, while limited to five tasks, provide valuable validation of the method's real-world applicability.
The paper provides detailed implementation specifics, including hyperparameters for both stages, architecture details for the lightweight and VLM-based variants, and data collection protocols. The use of standard benchmarks and clear descriptions of the training process enhances reproducibility. However, the lack of a public code repository link in the provided text is a minor drawback, though the detailed appendices mitigate this.
The method requires a sufficient number of successful trajectories to bootstrap the R-SFT stage, which may be difficult for very hard tasks without a strong initial policy. The frozen encoder in Stage 2 limits the model's ability to adapt its perception to new failure modes discovered during RL. The physical experiments are limited to single-arm tasks, and the method's scalability to more complex, multi-arm, or dexterous manipulation scenarios is not fully explored.
This paper contributes to the growing field of post-training for robotic policies, offering a practical and effective recipe for improving skill-based agents. The insight regarding optimization granularity is likely to influence future work in hierarchical RL and skill-based control. The framework is general enough to be applied to other skill-based architectures, making it a useful tool for the robotics community. [One sentence main contribution]. The paper introduces a two-stage post-training framework that stabilizes skill-based robotic policies via supervised fine-tuning on successful trajectories and resolves optimization granularity issues through skill-level policy optimization, achieving significant performance gains in simulation and real-world tasks.
Pretrained video generation models encode rich priors about physical interactions, but leveraging them for robot control typically requires either resource-intensive video generation at test time or action heads trained from scratch that do not fully utilize the pretrained backbone. We introduce Tokenizing Actions as Video Patches in a Shared Transformer Policy (TAPESTRY), a world action model that repurposes a pretrained video diffusion transformer into a robotic policy without requiring video generation at inference. Our key idea is to treat robot actions as single-patch tokens in the video model's latent space. A lightweight linear projection maps each action timestep to the patch dimensions of the pretrained VAE, and the resulting tokens are processed by the same patchifying layers and transformer backbone used for video. The shared backbone is optimized through three diffusion objectives sampled per minibatch element: Video diffusion uses VAE latents from robot and human videos to establish dynamics priors. Joint video+action diffusion trains on robot demonstrations using a coupled diffusion timestep and an asymmetric attention mask. Action-only diffusion denoises actions conditioned on clean image latents. At test time, the model generates a compact action token sequence in a single DDIM step from pure noise, producing smooth action chunks at 84 ms per action chunk on consumer hardware and avoiding the cost of full video generation. Across simulation benchmarks and real-robot experiments, TAPESTRY achieves strong performance with substantially lower inference cost than comparable joint video--action policies. More information can be found at https://tapestry-wam.github.io.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, University of Maryland, Toyota Research Institute
TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
The paper proposes TAPESTRY, a method to repurpose pretrained video diffusion transformers (specifically CogVideoX-2B) for robot control by treating robot actions as single-patch tokens in the video latent space. The core innovation is the "action-as-VAE-patch" tokenization, which uses a lightweight linear projection and LayerNorm to map action vectors into the same distribution as video VAE latents, allowing them to pass through the pretrained patchifying convolution without architectural modification. The training procedure involves three joint diffusion objectives: video-only, joint video+action, and action-only. A significant technical contribution is the analysis of high-frequency noise in multi-step denoising for action chunks, demonstrating that single-step DDIM inference from pure noise is superior for this specific repurposed backbone due to frequency-space mismatches between video and action signals.
The evaluation is rigorous, covering simulation (LIBERO benchmark with a learning-from-video protocol) and real-world bimanual manipulation. The paper demonstrates strong generalization to novel objects and motion primitives. The inference efficiency claims are well-supported, showing a 44x speedup over optimized DreamZero on consumer hardware (RTX 5090). The ablations effectively isolate the contribution of the video pretraining and the specific training objectives.
The paper provides detailed implementation specifics, including the exact parameter count of the action head (1280 parameters), the specific backbone used (CogVideoX-2B), and the training data mixture. The derivation of the single-step inference collapse is provided in the appendix. However, reliance on specific proprietary or large-scale datasets (like DROID, AgiBot-World) may limit immediate reproducibility for smaller labs, though the architectural changes are minimal and easy to implement.
The method is currently limited to a 20-DoF bimanual action space, and the single-step inference relies on the zero-terminal-SNR v-prediction schedule, which may not transfer to other backbones like rectified-flow models (Wan). The deterministic nature of single-step inference may be a limitation for tasks requiring multimodal action distributions.
This work significantly lowers the barrier to entry for using large video models in robotics by eliminating the need for expensive video generation at test time. The insight that action tokens can be treated as video patches with minimal projection is a generalizable architectural pattern that could be applied to other multimodal control tasks. The finding regarding high-frequency noise in repurposed diffusion models is a valuable diagnostic for the broader community working on world action models. TAPESTRY introduces a novel action tokenization scheme that allows pretrained video diffusion models to serve as efficient robot policies by treating actions as video patches, achieving state-of-the-art performance with 44x faster inference than comparable joint video-action models.
Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley
The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.
The paper proposes a comprehensive sim-to-real pipeline for in-hand object pose reaching using a 16-DoF LEAP hand. The core methodological contributions are: (1) A training recipe that leverages a large-scale generated grasp dataset (via LightningGrasp and DexterityGen) to define diverse goals and resets, avoiding the need for human demonstrations. (2) An adaptive friction curriculum that starts with high friction to stabilize learning and gradually reduces it to match reality, which is shown to be critical for convergence. (3) A modular safety approach using a learned residual safety filter (adapted from ResSafe) that operates on top of a frozen task policy. This filter uses a tunable weight $w$ to balance performance and safety, allowing for a "dial" to adjust conservativeness at deployment without retraining the task policy. The system identification is rigorous, using a voltage-driven DC motor model (BAM) and precise latency calibration to bridge the sim-to-real gap.
The experimental evaluation is extensive and rigorous, particularly for a preprint. The authors conduct 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and 6 wrist orientations. The state-based policy achieves a 95.2% success rate, and the safety filter reduces drops by 75% while maintaining a 91.5% success rate. The paper also evaluates robustness to out-of-distribution goals, fast arm motion, and physical perturbations. Comparisons with Hamilton-Jacobi (HJ) safety filters demonstrate the superiority of the learned residual approach in terms of computational efficiency and performance trade-offs. The inclusion of a depth-camera-only policy that transfers zero-shot is a strong validation of the visual sim-to-real gap mitigation.
The paper provides high reproducibility. It details the hardware setup, system identification process, simulation parameters, and training hyperparameters. The authors commit to releasing the full recipe, code, and trained checkpoints. The use of standard tools like MuJoCo Warp and PPO, along with clear descriptions of the reward functions and curricula, makes it feasible for other researchers to replicate the results. The specific details on latency calibration and actuator modeling are particularly valuable for the robotics community.
The policies are trained per object, which limits generalization to new objects without retraining. The depth policies still show a visual sim-to-real gap on certain objects (knife, wineglass). The safety filter is also trained per object and per nominal policy. The downstream tasks (e.g., screwing in a lightbulb) rely on hard-coded arm motions, limiting the autonomy of the system. The evaluation is limited to a specific hand (LEAP) and arm (UR5), so generalizability to other hardware is not tested.
This work significantly advances the state of the art in dexterous manipulation by addressing the critical issue of safety and reliability in high-dimensional contact-rich systems. The modular safety filter approach is likely to be adopted in other robotic domains where safety is paramount. The comprehensive evaluation framework for measuring performance, reliability, and safety provides a valuable benchmark for the field. The ability to operate with only a wrist-mounted camera expands the practical applicability of such systems beyond controlled lab environments. The paper presents a robust sim-to-real recipe for dexterous in-hand pose reaching that achieves high success rates and significantly improves safety through a modular, tunable residual safety filter, setting a new standard for reliability in high-dimensional robotic manipulation.