Week of July 12 – July 19, 2026
Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of California, Santa Cruz, Shanghai Jiao Tong University
RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
The paper introduces RESOURCE2SKILL, a framework for distilling multimodal human resources (videos, code repos, articles, artifacts) into a structured, hierarchical "Skill Wiki" for software agents. The core innovation lies in the schema: skills are tuples of text, visual, code, and metadata, organized by a domain-specific taxonomy. The method employs a "construction operator" that uses a vision-capable LLM to extract and normalize these components, followed by deterministic acceptance predicates (completeness, provenance, deduplication, modality consistency, structural executability). At inference, a two-stage retrieval process (lexical BM25 over taxonomy paths, then LLM selection) allows agents to compose skills. A key feature is the ability to trigger online acquisition when offline skills are insufficient, using the same construction operator. The approach is well-motivated, addressing the gap between implicit agent knowledge and explicit, reusable procedural knowledge derived from rich human resources.
The evaluation is extensive, covering seven diverse authoring domains (PPT, CAD, Web, Excel, Blender, UE5, Reaper) and four agent backends (GPT-5.5, GPT-5.4, Mini, Nano). The primary metric is an automated judge score (vision/audio models) on rendered artifacts, validated by human A/B studies. RESOURCE2SKILL shows significant improvements (+11.9 pp) over no-skill baselines and outperforms strong harness baselines (ClaudeCode-H, Codex-H) in 26/28 cells. Ablations confirm the value of multimodal content, hierarchical organization, and online acquisition. The scaling study shows performance saturates around 200 skills. The results are robust and statistically significant.
The paper provides a code link (https://aka.ms/Resource2Skill) and detailed appendices describing the benchmark construction, judge reliability, and experimental setup. The acceptance predicates are deterministic, and the library construction is frozen before benchmarking, ensuring no data leakage. The matched-brief design and fixed seeds enhance reproducibility. The detailed description of the MCP interface and domain backends allows for replication.
The framework relies on the availability of high-quality multimodal resources (videos, repos) for each domain, which may not exist for all software domains. The automated judge, while correlated with humans, may have biases (e.g., rewarding visual polish over functional correctness in some cases). The online acquisition adds latency. The "structural executability" check ensures code runs but does not guarantee it solves the specific task correctly, leading to potential partial grounding failures as noted in case studies.
This work significantly advances the field of software agents by providing a scalable method for building and maintaining skill libraries from diverse human-generated content. It enables agents to leverage existing human knowledge (tutorials, code) more effectively, potentially lowering the barrier to entry for complex software tasks. The hierarchical wiki structure offers a promising paradigm for organizing agent memory. RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
Robotic manipulation is inherently multi-frame: local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame. However, modern diffusion-based visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame. We propose Mixture of Frames Policy (MoF), a diffusion policy that performs synchronized action denoising across multiple coordinate frames. MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame. To make this possible for intermediate noisy diffusion states, we introduce a column-based 6D rotation representation within an SE(3) action parameterization that supports exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold. Across nine simulated bimanual manipulation tasks, we show that the best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines. We further evaluate MoF on two real-world bimanual mobile manipulation tasks, demonstrating that it outperforms all constituent single-frame baselines. Project homepage: https://mofpo.github.io
Primary: Stanford University
All Institutions: Stanford University, Apple, Toyota Research Institute
The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
The paper proposes a novel architecture for diffusion-based visuomotor policies called Mixture of Frames (MoF). The core insight is that robotic manipulation actions are naturally represented in different coordinate frames at different stages of a task (e.g., end-effector for grasping, base for transport). Standard diffusion policies fix a single action frame, forcing the denoiser to model complex, frame-dependent distributions. MoF maintains a single canonical diffusion state but applies frame-specialized denoisers in their respective frames, fusing the noise predictions back into the canonical frame. A critical technical contribution is the introduction of a column-based 6D rotation representation for SE(3) actions. This allows for exact, differentiable frame transformations of *noisy* intermediate states without requiring projection onto the SO(3) manifold, which is non-linear and lossy. This mathematical insight is elegant and solves a significant implementation hurdle for multi-frame reasoning in diffusion policies. The method is generalizable to any diffusion policy and can be implemented as a drop-in replacement or extension.
The evaluation is comprehensive and rigorous. The authors first establish the motivation by showing that action frame choice has a substantial, task-dependent impact on performance in bimanual mobile manipulation, with a 15% gap between the best and worst single frames. They then demonstrate that MoF outperforms oracle frame selection (which knows the best frame per task) and standard Mixture-of-Experts (MoE) baselines that do not operate in multiple action frames. The ablation studies are particularly strong, showing that the proposed column-based representation is essential (orthogonalization leads to collapse in ensemble variants) and that the router effectively switches between frames based on task phase. Real-world experiments on two complex bimanual mobile manipulation tasks (Pouring, Serving) further validate the approach, showing significant improvements over single-frame baselines. The use of simulated benchmarks (BiGym, DexMimicGen) and real-world validation provides a robust evidence base.
The paper provides detailed descriptions of the method, including the specific rotation representation and the fusion mechanism. The project homepage link suggests code availability. The experimental setup is well-described, including dataset sizes, training epochs, and evaluation metrics. The ablation studies provide clear evidence of component contributions. The reliance on standard simulators (BiGym) and real-world hardware (HoMMI) enhances reproducibility.
The method relies on a designer-specified set of candidate frames. While the router learns to weight them, the set must be predefined. The paper acknowledges this and suggests future work on learning or discovering frames. Additionally, the router is supervised via denoising loss, which may not perfectly align with policy performance, though the results suggest it works well in practice. The current experiments focus on smaller diffusion policies; scaling to large VLA models is noted as future work.
This work has significant implications for robotic manipulation, particularly in complex, multi-stage tasks where different coordinate systems are advantageous. By improving the robustness and performance of visuomotor policies, it contributes to the broader goal of autonomous robots capable of operating in unstructured environments. The technical contribution regarding SE(3) transformations is also valuable for the broader machine learning community working with geometric deep learning and diffusion models. The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.
Primary: New York University
All Institutions: Carnegie Mellon University, New York University
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
The paper introduces "requential coding," a novel and highly efficient method for compressing generative models. It builds upon the principles of prequential coding and relative entropy coding (REC) but addresses their fundamental limitations. Unlike parameter-based methods (e.g., quantization) that scale with model size regardless of information stored, or prequential coding that scales with dataset size and entropy, requential coding decouples code length from both. The core idea is that a "student" model generates candidate training samples from its own distribution, and a stronger "teacher" model selects one of these candidates. The student's code then records only the index of the selected sample, which costs approximately $KL(Q_t\|P_t)$ bits, where $Q_t$ is the teacher and $P_t$ is the student. This effectively distills the teacher's knowledge into the student by efficiently transmitting only the "disagreement" between them. The methodology is rigorously defined, including the encoder and decoder protocols, and a formal bound for the expected code length is derived in the appendix, showing it scales with the cumulative teacher-student KL divergence. Practical improvements like "teacher smoothing" (using an EMA of teacher checkpoints) and "iso-loss projection" (periodically resetting the teacher to the student's performance level) are introduced to further reduce the code length without sacrificing performance. The paper also provides a detailed analysis of the runtime for code length evaluation, encoding, and decoding, acknowledging the computational cost of actual transmission (which can be high due to REC's proposal generation) but emphasizing that evaluation is much more efficient. The use of a universal integer code (Elias delta) for the index ensures the decoder doesn't need prior knowledge of the KL divergence. This is a sophisticated and well-thought-out approach that significantly advances the state of the art in model compression theory.
The experimental evaluation is comprehensive and compelling, demonstrating the power of requential coding across various tasks and scales. 1. **Compression Benchmarking**: Requential coding is benchmarked against prequential coding and idealized 4-bit post-training quantization (PTQ) on autoregressive transformers (100M parameters) trained on OpenWebText, CIFAR-5M, and FineWeb. It achieves one to two orders of magnitude shorter codes than prequential coding, dominating the Pareto frontier of loss vs. code length. The prequential heuristic (a non-rigorous estimate) also falls well above requential coding. 2. **Scaling Insights**: * **Model Size**: Experiments show that larger models (from 10M to 1B parameters) are *more compressible* at a fixed loss, despite having more parameters. This is a crucial finding, as it suggests larger models learn simpler functions relative to their capacity, which parameter-based methods fail to capture. * **Ensembles**: Larger ensembles are also shown to be more compressible, reaching lower loss at a fixed code budget. 3. **Generalization Guarantees (PAC-Bayes Bounds)**: This is a highlight. Plugging the requential code length into a PAC-Bayes bound yields state-of-the-art generalization guarantees for billion-parameter LLMs. These bounds: * Improve with model scale, mirroring the true test loss, for models trained on a fixed amount of data. * Outperform even *idealized lossless 4-bit PTQ bounds* in the compute-optimal scaling regime ($D=20N$), a significant breakthrough as previous non-vacuous bounds for LLMs were based on realistic PTQ. * Tighten with scale, as the code length per parameter decays as a power law, predicting a vanishing generalization gap. 4. **Overfitting Prediction**: The requential code successfully predicts gradual overfitting under data repetition, showing the divergence between training and test loss as the complexity penalty rises. 5. **Learnable Information Content**: The method effectively isolates learnable structure from random information in datasets, revealing that lower-entropy text holds more learnable structure than higher-entropy image data, and distinguishing it from random or trivially repeating strings. This provides a principled way to rank datasets by their "epiplexity." The experiments are well-designed, the results are clearly presented (e.g., Figure 3, 4, 5), and the conclusions drawn are profound, offering new insights into deep learning phenomena that were previously inaccessible to rigorous measurement.
The paper explicitly states that "Code available at https://github.com/shikaiqiu/requential-coding". The methodology is described with sufficient detail, including pseudocode for REC and descriptions of teacher smoothing and iso-loss projection. The appendix provides full experiment details, including how code lengths are computed, datasets used, and per-figure configurations. This commitment to open-sourcing the code and providing detailed experimental setups significantly enhances reproducibility.
The authors acknowledge several limitations: 1. **Lossy Compression of Teacher**: Requential coding provides a lossless compression for the student model, which is a distillation of the teacher. If the goal is to compress a pre-trained model not trained via distillation, it becomes a lossy compression of that original model. 2. **Encoding Runtime**: While code length *evaluation* is efficient, *actually transmitting* a model via requential coding can be prohibitively slow due to the exponential scaling of REC encoding time with KL divergence. The paper primarily presents it as a tool for *evaluation* and *understanding* rather than practical transmission. 3. **Information Forgetting**: The current code length only grows with training steps and doesn't account for information that might be learned early but subsequently forgotten. Leveraging forgotten information could potentially further reduce the code length. 4. **Teacher Optimization**: The choice of teacher sequence is simple, and further gains are expected from optimizing it.
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
Training API-calling large language model (LLM) agents demands massive amounts of high-quality trajectories. However, collecting such data at scale typically requires fully implemented environments with executable APIs and realistic, pre-populated backend databases, creating a major bottleneck for scalability. To overcome this, we propose an environment-free synthetic data generation approach that leverages LLMs as on-the-fly digital world models. Given only API specifications, our method generates trajectories mimicking interactions between an agent and a stateful environment. Specifically, an LLM first generates diverse tasks solvable with the provided APIs. A teacher agent then iteratively solves each task while an LLM simulator generates coherent synthetic API responses conditioned on the task context and simulation history. Finally, an LLM judge filters the trajectories to ensure the quality of the resulting dataset. We evaluate our approach on the challenging AppWorld and OfficeBench benchmarks, which include both information-retrieval and state-changing tasks. Fine-tuning models on our synthetic data yields significant performance gains, demonstrating that effective supervision for API-calling agents can be generated without any executable environment. Our results establish LLM-based API simulation as a practical, scalable solution for training agents across diverse API ecosystems.
Primary: Unknown
All Institutions: Unknown
This paper presents a practical and scalable method for generating synthetic training data for API-calling agents by leveraging LLMs as dynamic world models, effectively bypassing the need for executable environments and demonstrating significant performance improvements on standard benchmarks.
The paper proposes a pipeline for generating synthetic training data for API-calling agents without requiring an executable environment. The core methodology involves three stages: (1) Task Generation: An LLM generates diverse tasks based on API specifications. (2) Trajectory Simulation: A "teacher" agent attempts to solve these tasks, while a separate "simulator" LLM generates the API responses and state transitions on the fly, conditioned on the context. (3) Filtering: An LLM judge filters the resulting trajectories for quality. The approach leverages the reasoning capabilities of LLMs to model the environment dynamics implicitly, bypassing the need for complex backend implementations. While the concept of using LLMs as world models is not entirely new, applying it specifically to the high-dimensional, state-dependent space of API interactions for data generation is a pragmatic and novel application. The separation of the teacher (reasoning) and simulator (state modeling) roles is a key methodological choice that simplifies the training objective for the student agent.
The authors evaluate their method on AppWorld and OfficeBench, two challenging benchmarks involving information retrieval and state-changing tasks. The primary metric is the performance of models fine-tuned on the generated synthetic data compared to baselines (likely zero-shot or few-shot LLMs, and potentially models fine-tuned on real data if available, though the abstract emphasizes the "environment-free" aspect). The results show "significant performance gains," suggesting that the synthetic data captures useful patterns for agent behavior. However, the abstract does not specify the magnitude of these gains relative to state-of-the-art models trained on real data. The evaluation focuses on the utility of the generated data rather than the fidelity of the simulation itself, which is a valid but limited scope. The use of established benchmarks ensures comparability, but the lack of a real-data baseline in the abstract's summary leaves the absolute ceiling of this approach unclear.
The paper describes a clear, modular pipeline. The use of standard LLMs for task generation, simulation, and filtering suggests that the method is reproducible, provided the specific prompts and model versions are disclosed. The dependency on LLMs for simulation introduces some stochasticity, but the filtering step should mitigate low-quality outputs. The lack of code release (indicated by "none" for Project URL) is a barrier to full reproducibility, but the methodology is described sufficiently to allow for implementation.
The primary limitation is the potential for error propagation. If the simulator generates inconsistent API responses or the teacher agent makes logical errors, the synthetic data will be noisy. The "LLM judge" is intended to filter this, but judges can also be biased or inconsistent. Furthermore, the simulation is only as good as the LLM's understanding of the API semantics; it may hallucinate behaviors that are not possible in the real world. The approach also assumes that the API specifications are sufficient for an LLM to generate coherent interactions, which may not hold for complex, undocumented, or highly dynamic APIs. Finally, the computational cost of generating and filtering large-scale synthetic data using multiple LLM calls is significant.
This work addresses a critical bottleneck in agent development: the scarcity of high-quality training data. By enabling the generation of synthetic data without expensive environments, it lowers the barrier to entry for developing robust API-calling agents. This could accelerate the deployment of agents in diverse domains where building executable environments is difficult or costly. However, it also raises concerns about the quality and safety of training data, as synthetic data may reinforce biases or hallucinations present in the base models. This paper presents a practical and scalable method for generating synthetic training data for API-calling agents by leveraging LLMs as dynamic world models, effectively bypassing the need for executable environments and demonstrating significant performance improvements on standard benchmarks.
We study aggregation of statistical evidence under unknown and potentially complex dependence using group-invariance. Building on permutation-based constructions that treat transformed datasets as exchangeable units, we aggregate evidence across statistics for each transformed dataset and calibrate the resulting aggregates across transformations. We develop a finite-sample power and adaptivity theory for this framework, together with extensions to sequential and data-dependent aggregation that preserve validity. For single-batch aggregation, which uses one collection of transformed datasets for both standardization and calibration, we show that the critical values uniformly improve on deterministic calibrations valid under arbitrary dependence, including Bonferroni correction, while adapting to the unknown dependence structure. We also introduce a sequential alpha-spending version that permits early rejection when evidence is strong, and a two-batch extension that separates standardization from calibration to accommodate learned aggregation rules and reduce computation. Applications to adaptive nonparametric testing and conformal prediction illustrate how these results sharpen existing aggregation methods.
Primary: University of Cambridge
All Institutions: University of Cambridge, University College London, KAIST
This paper provides a rigorous and novel theoretical framework for aggregating statistical evidence under group-invariance, introducing Single-Batch and Two-Batch aggregation methods that guarantee finite-sample validity while adapting to unknown dependence structures, thereby offering strictly improved power over worst-case calibrations like Bonferroni and resolving finite-sample size control issues in Monte Carlo approximations.
The paper presents a rigorous theoretical framework for aggregating statistical evidence under group-invariance. The core methodological contribution is the formalization of "Single-Batch" (SB) aggregation, which uses a single set of transformed datasets for both standardization and calibration, and the "Two-Batch" (TB) extension, which separates these roles. The authors prove finite-sample validity for these methods under arbitrary dependence structures, a significant improvement over Monte Carlo approximations that may fail to control Type I error in finite samples. The introduction of sequential alpha-spending for ordered statistics and the analysis of adaptivity to dependence structures (perfect rank alignment) are theoretically sound and novel contributions to the theory of permutation tests and p-value merging.
The paper includes applications to adaptive nonparametric testing and conformal prediction. While the text provided is truncated, the abstract and introduction indicate that the theoretical results are illustrated through these applications. The theoretical proofs are extensive, covering validity, power dominance over Bonferroni/deterministic calibrations, and asymptotic adaptivity. The experimental section likely demonstrates empirical performance gains, particularly in conformal prediction where the TB method avoids computational bottlenecks. The rigorous theoretical backing provides strong indirect evidence of utility.
The paper provides detailed algorithms (SB, SeqSB, TB) and mathematical formulations. The theoretical results are clearly stated with proofs (deferred to supplementary material, which is standard for such theoretical work). The reliance on group-invariance makes the methods reproducible given the data and the group structure.
The methods are restricted to settings where group-invariance (exchangeability) holds under the null hypothesis. This limits applicability to observational studies without such symmetries. The "Two-Batch" method requires splitting data, which can reduce power if the sample size is small. The theoretical gains are asymptotic or finite-sample bounds; practical performance depends on the specific dependence structure and the choice of merging function.
This work has significant implications for multiple testing, conformal prediction, and distribution-free inference in machine learning. By providing valid, less conservative aggregation methods under exchangeability, it enables more powerful hypothesis testing and tighter prediction intervals in settings like genetic association studies, image analysis, and any domain where permutation tests are applicable. It bridges the gap between theoretical validity and practical power in dependent data settings. This paper provides a rigorous and novel theoretical framework for aggregating statistical evidence under group-invariance, introducing Single-Batch and Two-Batch aggregation methods that guarantee finite-sample validity while adapting to unknown dependence structures, thereby offering strictly improved power over worst-case calibrations like Bonferroni and resolving finite-sample size control issues in Monte Carlo approximations.
Higher-order couplings enhance the expressive power of hypergraph neural networks (HGNNs), but they also intensify representation collapse in deep propagation due to strong multi-way feature mixing. This work investigates hypergraph oversmoothing from a dynamical-systems perspective and develops a reaction--diffusion framework for depth-resistant hypergraph learning. By defining hypergraph gradient and divergence operators, we interpret message passing as an incidence-level diffusion process. The analysis of pure diffusion shows that its continuous semiflow exponentially contracts the null-mode-free component of node representations and drives the Dirichlet energy to zero, revealing hypergraph oversmoothing as an intrinsic transverse-energy dissipation phenomenon. Motivated by this analysis, we propose Hypergraph Neural Reaction--Diffusion (HNRD), which introduces a reaction mechanism acting on the transverse component to compensate diffusion-induced dissipation and stabilize discriminative variations. We establish global well-posedness of the proposed dynamics and prove that the null-mode-free Dirichlet energy remains bounded away from zero. A forward-Euler discretization provides a practical HNRD layer with a stability condition for deep propagation. Experiments on benchmark and synthetic heterophilic hypergraphs demonstrate that HNRD consistently improves over representative hypergraph baselines. Depth, robustness, and efficiency analyses further show that HNRD preserves stable performance and nonzero Dirichlet energy under deep propagation and perturbations. These results provide a principled dynamical framework for designing deep hypergraph architectures that maintain higher-order expressiveness without representation collapse.
Primary: Chinese Academy of Sciences
All Institutions: Shandong University, Chinese Academy of Sciences, University of Chinese Academy of Sciences
This work has significant broader impact for the field of graph and hypergraph neural networks: * **Principled Deep HGNN Design**: It provides a principled, theoretically grounded framework for designing deep HGNNs that inherently mitigate oversmoothing, moving beyond heuristic solutions like residual connections. This can lead to more robust and performant deep hypergraph architectures. * **Deeper Understanding of Oversmoothing**: The dynamical-systems view offers a powerful new lens for understanding oversmoothing in hypergraphs, characterizing it as transverse-energy dissipation. This theoretical insight can guide future research into other graph/hypergraph problems. * **Advancement of Continuous-Time GNNs**: By extending the reaction-diffusion paradigm to hypergraphs with strong theoretical guarantees, the paper further validates the utility of continuous-time models (ODEs/PDEs) in graph machine learning, potentially encouraging more research in this direction. * **Applications in Complex Systems**: Hypergraphs are crucial for modeling multi-way interactions in various domains (social networks, biology, chemistry, recommender systems). A robust deep HGNN model like HNRD can significantly improve performance in these applications, especially where deep propagation is necessary to capture complex dependencies. * **Theoretical Foundation for HGNNs**: The rigorous mathematical analysis contributes to building a stronger theoretical foundation for hypergraph learning, which is essential for the maturity and reliability of the field. This paper introduces a novel dynamical-systems framework to analyze and mitigate oversmoothing in hypergraph neural networks, rigorously proving that pure hypergraph diffusion leads to representation collapse and proposing Hypergraph Neural Reaction–Diffusion (HNRD) which provably stabilizes discriminative variations. The paper's strength lies in its deep theoretical analysis, which precisely characterizes hypergraph oversmoothing as transverse-energy dissipation, and its innovative reaction-diffusion mechanism that explicitly counteracts this collapse by preserving a learnable, nonzero transverse-energy level. This principled approach, combined with comprehensive experimental validation across diverse hypergraph benchmarks and detailed analyses (depth, heterophily, ablation, robustness, visualization), demonstrates HNRD's superior performance and stability in deep and challenging settings. The work significantly advances the theoretical understanding and practical design of deep hypergraph architectures, offering a robust solution to a critical problem and paving the way for more principled continuous-time models in higher-order graph learning.
The paper presents a highly rigorous and principled approach to understanding and mitigating oversmoothing in Hypergraph Neural Networks (HGNNs) through a dynamical-systems lens. The methodology is built upon defining hypergraph gradient and divergence operators at the incidence level, which allows for formulating message passing as a learnable incidence-level diffusion equation. This continuous-time formulation is a significant strength, enabling the use of semiflow theory and energy methods for deep theoretical analysis. The core methodological contribution lies in two parts: 1. **Analysis of Pure Hypergraph Diffusion**: The paper rigorously proves that pure hypergraph neural diffusion generates a global continuous semiflow that exponentially contracts the null-mode-free component of node representations and drives the associated Dirichlet energy to zero. This provides a precise mathematical explanation for oversmoothing as an intrinsic transverse-energy dissipation process. This theoretical characterization is a strong foundation for the proposed solution. 2. **Hypergraph Neural Reaction–Diffusion (HNRD)**: Motivated by the diffusion analysis, HNRD introduces a novel reaction mechanism. Crucially, this reaction term acts *only* on the transverse (null-mode-free) component of the node representations. It combines two effects: instantaneous compensation of diffusion-induced dissipation and a bounded saturating feedback that drives the transverse energy toward a positive, learnable level. This design is elegant as it preserves the smoothing benefits of diffusion while preventing the collapse of discriminative information. The paper establishes global well-posedness for HNRD and proves that its null-mode-free Dirichlet energy remains bounded away from zero, directly addressing the oversmoothing problem. A practical discrete HNRD layer is derived via forward-Euler discretization, complete with a step-size stability condition, making the continuous dynamics implementable. The learnable step size further enhances adaptability. The overall methodology is theoretically sound, well-motivated, and provides a clear path from continuous dynamics to a practical deep learning architecture.
The experimental evaluation is comprehensive and robust, supporting the theoretical claims. * **Datasets**: HNRD is evaluated on a diverse set of 11 hypergraph benchmarks, including academic citation/co-authorship networks (Cora, Citeseer, Pubmed, Cora-CA, DBLP-CA) and real-world applications (Zoo, NTU2012, ModelNet40, Walmart, Senate, House). This diversity demonstrates the model's generalizability. * **Baselines**: A wide array of baselines are included, categorized into graph diffusion models (GRAND, GRAND++, GREAD, RDGNN), standard HGNNs (HGNN, HyperGCN, HCHA, HNHN, UniGCNII), expressive HGNNs (AllSetTransformer, AllDeepSets, ED-HNN, HyperGINE, KHGNN), and deep/stable HGNNs (Deep-HGCN, Implicit HNN, FrameHGNN, HND, RFHND). This comprehensive comparison establishes HNRD's state-of-the-art performance against strong competitors. * **Performance**: HNRD achieves the best overall rank on both academic and real-world benchmarks, ranking first on 8 out of 11 datasets and second on the remaining 3. This consistent outperformance is a strong indicator of its effectiveness. * **Synthetic Heterophily**: Experiments on synthetic hypergraphs with controlled heterophily levels demonstrate HNRD's robustness. It maintains superior performance across varying heterophily, where many baselines suffer significant degradation, highlighting its ability to handle complex real-world scenarios. * **Oversmoothing Analysis**: The depth-wise analysis (up to 128 layers) visually and quantitatively confirms HNRD's resistance to oversmoothing. While baselines show clear performance degradation and rapid decay of Dirichlet energy, HNRD maintains stable accuracy and a nonzero Dirichlet energy, directly validating the theoretical non-collapse property. * **Ablation Study**: The ablation study effectively demonstrates the contribution of each component of the reaction term (compensation and bounded feedback), showing that both are crucial for optimal performance. * **Parameter Analysis**: Sensitivity analysis on hidden dimension and step size shows HNRD is relatively stable, with optimal performance at moderate settings. * **Visualization**: t-SNE visualizations of node embeddings at different depths illustrate how HNRD preserves class separability, unlike other models where classes become mixed. * **Robustness and Efficiency**: Additional analyses confirm robustness to perturbations and practical runtime efficiency. The experimental section is exceptionally thorough, providing strong empirical evidence for the theoretical claims and practical utility of HNRD.
The paper provides a GitHub link to the source code, which is a crucial step for reproducibility. It also details experimental settings, including hyperparameter search spaces, optimization methods (Adam), and hardware used (NVIDIA RTX A4000 GPU). Standard hypergraph construction protocols and train/validation/test splits (50%/25%/25%) are followed. The level of detail in the experimental setup, combined with the provided code, suggests good reproducibility.
1. **Computational Complexity**: While the paper mentions practical efficiency, continuous-time models and their discretizations can sometimes be more computationally intensive than simpler message-passing layers, especially for very large hypergraphs or very small step sizes. The stability condition might necessitate small step sizes in certain cases. 2. **Generalizability of Reaction Term Design**: The specific form of the reaction term, while theoretically justified, is tailored to the identified transverse energy dissipation. While effective, exploring alternative reaction mechanisms or more adaptive forms could be an area for future work. 3. **Hypergraph Laplacian Dependence**: The theoretical analysis and the reaction term design rely on the properties of the hypergraph Laplacian and its null space. While standard, this might limit applicability to hypergraph models that do not easily map to such a Laplacian framework. 4. **Parameter Tuning**: The paper mentions tuning the reaction-diffusion step size, which can be a sensitive hyperparameter in ODE-based models. While a learnable sigmoid-constrained step size is proposed, it still adds complexity to the training process.
This work has significant broader impact for the field of graph and hypergraph neural networks: * **Principled Deep HGNN Design**: It provides a principled, theoretically grounded framework for designing deep HGNNs that inherently mitigate oversmoothing, moving beyond heuristic solutions like residual connections. This can lead to more robust and performant deep hypergraph architectures. * **Deeper Understanding of Oversmoothing**: The dynamical-systems view offers a powerful new lens for understanding oversmoothing in hypergraphs, characterizing it as transverse-energy dissipation. This theoretical insight can guide future research into other graph/hypergraph problems. * **Advancement of Continuous-Time GNNs**: By extending the reaction-diffusion paradigm to hypergraphs with strong theoretical guarantees, the paper further validates the utility of continuous-time models (ODEs/PDEs) in graph machine learning, potentially encouraging more research in this direction. * **Applications in Complex Systems**: Hypergraphs are crucial for modeling multi-way interactions in various domains (social networks, biology, chemistry, recommender systems). A robust deep HGNN model like HNRD can significantly improve performance in these applications, especially where deep propagation is necessary to capture complex dependencies. * **Theoretical Foundation for HGNNs**: The rigorous mathematical analysis contributes to building a stronger theoretical foundation for hypergraph learning, which is essential for the maturity and reliability of the field. This paper introduces a novel dynamical-systems framework to analyze and mitigate oversmoothing in hypergraph neural networks, rigorously proving that pure hypergraph diffusion leads to representation collapse and proposing Hypergraph Neural Reaction–Diffusion (HNRD) which provably stabilizes discriminative variations. The paper's strength lies in its deep theoretical analysis, which precisely characterizes hypergraph oversmoothing as transverse-energy dissipation, and its innovative reaction-diffusion mechanism that explicitly counteracts this collapse by preserving a learnable, nonzero transverse-energy level. This principled approach, combined with comprehensive experimental validation across diverse hypergraph benchmarks and detailed analyses (depth, heterophily, ablation, robustness, visualization), demonstrates HNRD's superior performance and stability in deep and challenging settings. The work significantly advances the theoretical understanding and practical design of deep hypergraph architectures, offering a robust solution to a critical problem and paving the way for more principled continuous-time models in higher-order graph learning.
Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of California, Santa Cruz, Shanghai Jiao Tong University
RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
The paper introduces RESOURCE2SKILL, a framework for distilling multimodal human resources (videos, code repos, articles, artifacts) into a structured, hierarchical "Skill Wiki" for software agents. The core innovation lies in the schema: skills are tuples of text, visual, code, and metadata, organized by a domain-specific taxonomy. The method employs a "construction operator" that uses a vision-capable LLM to extract and normalize these components, followed by deterministic acceptance predicates (completeness, provenance, deduplication, modality consistency, structural executability). At inference, a two-stage retrieval process (lexical BM25 over taxonomy paths, then LLM selection) allows agents to compose skills. A key feature is the ability to trigger online acquisition when offline skills are insufficient, using the same construction operator. The approach is well-motivated, addressing the gap between implicit agent knowledge and explicit, reusable procedural knowledge derived from rich human resources.
The evaluation is extensive, covering seven diverse authoring domains (PPT, CAD, Web, Excel, Blender, UE5, Reaper) and four agent backends (GPT-5.5, GPT-5.4, Mini, Nano). The primary metric is an automated judge score (vision/audio models) on rendered artifacts, validated by human A/B studies. RESOURCE2SKILL shows significant improvements (+11.9 pp) over no-skill baselines and outperforms strong harness baselines (ClaudeCode-H, Codex-H) in 26/28 cells. Ablations confirm the value of multimodal content, hierarchical organization, and online acquisition. The scaling study shows performance saturates around 200 skills. The results are robust and statistically significant.
The paper provides a code link (https://aka.ms/Resource2Skill) and detailed appendices describing the benchmark construction, judge reliability, and experimental setup. The acceptance predicates are deterministic, and the library construction is frozen before benchmarking, ensuring no data leakage. The matched-brief design and fixed seeds enhance reproducibility. The detailed description of the MCP interface and domain backends allows for replication.
The framework relies on the availability of high-quality multimodal resources (videos, repos) for each domain, which may not exist for all software domains. The automated judge, while correlated with humans, may have biases (e.g., rewarding visual polish over functional correctness in some cases). The online acquisition adds latency. The "structural executability" check ensures code runs but does not guarantee it solves the specific task correctly, leading to potential partial grounding failures as noted in case studies.
This work significantly advances the field of software agents by providing a scalable method for building and maintaining skill libraries from diverse human-generated content. It enables agents to leverage existing human knowledge (tutorials, code) more effectively, potentially lowering the barrier to entry for complex software tasks. The hierarchical wiki structure offers a promising paradigm for organizing agent memory. RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
Video models are evolving into vision foundation models, yet they still lack human-like multi-step reasoning. Streaming autoregressive diffusion models are efficient but limited in reasoning, while bidirectional diffusion enables global revision with high inference costs due to dense frame-level denoising. Both paradigms struggle to achieve logical consistency and low-latency streaming for complex reasoning tasks. We propose HDR (Hierarchical Denoising for Visual Reasoning), a unified framework that integrates hierarchical latents into causal video generation for multi-step reasoning. HDR organizes video latents into a tree-structured hierarchy, enabling coarse-to-fine reasoning before streaming output. Coarse denoising layers preserve uncertain hypotheses for global planning, while finer layers progressively refine them into concrete visual states. A sparse hierarchical attention pattern (SHAP) further reduces temporal attention costs. We introduce a level-stratified multi-step video reasoning benchmark with out-of-distribution cases, covering six tasks: maze navigation, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, and water pouring. Compared with streaming autoregressive diffusion baselines, HDR improves success from 34.22 to 60.29 (76.2% relative gain) and increases average progress from 76.00 to 89.56, demonstrating more consistent reasoning trajectories. HDR maintains low-latency streaming at 0.70 seconds per latent, achieving 54.2 times faster inference than bidirectional diffusion. It also retains 82.9% of full-data performance with only 2% training data, compared with 52.0% for bidirectional diffusion. Real-world robot experiments further demonstrate HDR's potential for physical interaction and world modeling. Project demo: https://hierarchical-diffusion-reasoning.github.io/.
Primary: The Hong Kong University of Science and Technology
All Institutions: State Key Laboratory of Multimedia Information Processing, The Hong Kong University of Science and Technology, Beihang University, Fuzhou University, Peking University, School of Computer Science
HDR makes a significant contribution to the field of generative AI, particularly in video modeling and embodied AI. 1. **Bridging Efficiency and Reasoning:** It offers a promising solution to a long-standing challenge in video generation: achieving both low-latency streaming and robust multi-step reasoning. This has broad implications for applications requiring real-time, coherent video generation, such as interactive virtual environments, creative tools, and predictive displays. 2. **Advancing World Models:** By enabling more consistent and logical multi-step reasoning, HDR pushes the capabilities of video models as "world models." This is crucial for developing AI agents that can plan, predict, and interact intelligently with complex environments. 3. **Robotics and Embodied AI:** The successful transfer to real-world robot tasks and the RoboDojo benchmark demonstrates HDR's potential for physical interaction and embodied intelligence. Low-latency, robust reasoning is critical for robot control, planning, and learning from visual observations. 4. **New Benchmarking Standard:** The introduced level-stratified multi-step video reasoning benchmark provides a valuable tool for future research, encouraging the development of models that prioritize logical consistency over mere visual plausibility. 5. **Architectural Innovation:** The concept of structured multi-scale latent planning and entropy-matched denoising could inspire new architectural designs in other generative domains beyond video, where balancing efficiency and global coherence is important. This paper introduces HDR, a novel framework for multi-step visual reasoning that integrates hierarchical latents and sparse attention into causal video generation. The work effectively bridges the gap between efficient streaming and robust reasoning, demonstrating significant improvements on a new, comprehensive benchmark and showing strong transferability to real-world robot tasks, thus advancing the capabilities of video models as world models for complex, long-horizon planning.
The paper proposes Hierarchical Denoising for Visual Reasoning (HDR), a novel framework that addresses the fundamental tension between efficient streaming generation and robust multi-step reasoning in video models. The core idea is to integrate hierarchical latents into a causal video generation process, enabling coarse-to-fine reasoning before streaming output. This is a well-motivated approach, as existing autoregressive models struggle with revision, while bidirectional models are computationally expensive. The methodology introduces several key components: 1. **Tree-structured Hierarchy:** Video latents are organized into a multi-level hierarchy, where coarse tokens summarize global temporal structure (high-level plans) and fine tokens encode local visual details. This provides an explicit intermediate space for planning. 2. **Layer-wise Flow-Matching Objective:** The model is trained with a flow-matching objective across all hierarchy levels. A crucial design choice is the "hierarchy-matched denoising schedule," which assigns level-dependent sampling budgets, allowing coarse layers to maintain higher noise levels (preserving uncertainty for multiple global plans) and finer layers to progressively refine these hypotheses. This entropy-matched approach is a clever detail that aligns denoising strength with the abstraction level. 3. **Sparse Hierarchical Attention Pattern (SHAP):** To maintain efficiency, SHAP defines a structured attention mask over flattened tree tokens. Each token attends only to fixed local, parent-level, and first-frame contexts. This enables multi-scale information flow with reduced temporal attention cost, preserving streaming generation properties. 4. **HDR-WAM for Robotics:** The paper extends HDR to a World Action Model (HDR-WAM) for embodied control, combining episode-level visual context with local action-conditioned rollouts. This demonstrates the generality of the hierarchical approach. The methodology is sound, well-articulated, and directly tackles the identified problem. The combination of hierarchical latents, a tailored denoising schedule, and sparse attention is a strong technical contribution.
The experimental evaluation is comprehensive and rigorous, supporting the claims effectively. 1. **Novel Benchmark:** A significant contribution is the construction of a level-stratified multi-step video reasoning benchmark covering six diverse tasks (maze navigation, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, water pouring). This benchmark includes out-of-distribution (OOD) cases and uses robust metrics (success and average progress), which are crucial for evaluating logical consistency over long trajectories. This benchmark itself is a valuable asset for the community. 2. **Strong Baselines:** HDR is compared against relevant full-attention baselines (Bidirectional Diffusion, VideoMAE) and streaming autoregressive baselines (CausalForcing, VideoGPT). The main diffusion baselines are built on the same foundation model (Wan2.2-5B-TI2V) for a controlled comparison. 3. **Quantitative Results:** HDR significantly outperforms the streaming autoregressive baseline (CausalForcing), improving overall success from 34.22 to 60.29 (76.2% relative gain) and average progress from 76.00 to 89.56. It achieves this while maintaining low-latency streaming (0.70s per latent, comparable to CausalForcing's 0.72s) and being substantially faster than bidirectional diffusion (37.92s). It also remains competitive with bidirectional diffusion in reasoning performance, especially considering the latency difference. 4. **Qualitative Evidence:** Visual comparisons demonstrate HDR's ability to perform hierarchical planning and avoid early local commitments that lead to failure in baselines. 5. **Ablation Studies:** * **Layer Importance:** Demonstrates that increasing the number of active hierarchy layers progressively improves performance, validating the coarse-to-fine reasoning process. * **Denoising-step Reduction:** HDR shows superior robustness under reduced denoising budgets compared to both streaming AR and bidirectional diffusion, highlighting the stability provided by coarse layers. * **Data Reduction:** HDR degrades more gracefully than bidirectional diffusion with limited training data, suggesting better data efficiency for learning reasoning rules. * **Entropy-Matched vs. Fully Denoised:** The ablation on denoising schedules confirms the benefit of the entropy-matched approach, which preserves uncertainty at coarse levels, over a naive full denoising strategy. 6. **Real-world Robot Experiments:** The transferability of HDR is demonstrated in a physical robot maze task and on the RoboDojo simulation benchmark. HDR-WAM shows strong performance, particularly on Long-Horizon and Memory tasks, without robot-domain pretraining, indicating its potential for physical interaction and world modeling. This is a very strong point, showcasing practical relevance. The experimental section is exceptionally thorough, covering performance, efficiency, robustness, and real-world applicability.
The paper provides a good level of detail for reproducibility. * The methodology is clearly described, including the hierarchical latent representation, layer-wise flow-matching objective, and SHAP. * The new benchmark is detailed, including task descriptions, difficulty stratification, OOD cases, and evaluation metrics. Appendix provides full benchmark construction and task-specific evaluation details. * Baselines, training details, and implementation settings are specified, including the base diffusion model (Wan2.2-5B-TI2V) and the number of hierarchy levels and denoising schedule. * The entropy-matched denoising schedule derivation is provided in the appendix. * Details for HDR-WAM (temporal views, action alignment, token layout, attention mask, RoboDojo data processing) are also in the appendix. The project demo page (https://hierarchical-diffusion-reasoning.github.io/) further suggests that code or interactive demonstrations might be available, enhancing reproducibility. Overall, the paper provides sufficient information for a skilled researcher to reproduce the main results.
1. **Fixed Hierarchy Structure:** The paper uses a fixed, tree-structured hierarchy. While effective, the optimal hierarchy structure might be task-dependent or could potentially be learned. The current approach might not be optimal for all types of reasoning problems or video lengths. 2. **Pre-computation Phase:** While HDR enables "streaming output," it still involves a coarse-to-fine reasoning phase *before* the final streaming output. This means there's an initial latency for the hierarchical planning, even if subsequent frame generation is fast. For truly real-time, continuous streaming where no look-ahead is possible, this might still be a limitation compared to purely causal models. 3. **Scalability to Extremely Long Videos:** While SHAP reduces attention cost, the total number of hierarchical tokens still grows with video length. For extremely long, open-ended video generation, the fixed-size local and parent contexts might eventually become insufficient for maintaining global consistency over vast temporal spans. 4. **Training Complexity:** Training a multi-level hierarchical diffusion model with a layer-wise objective and specific denoising schedules might be more complex and resource-intensive than training a single-level autoregressive model. 5. **Generalizability of Reasoning:** While the benchmark is diverse, it's still a set of structured reasoning tasks. The extent to which HDR's reasoning capabilities generalize to more open-ended, less structured, or abstract reasoning scenarios remains to be fully explored.
HDR makes a significant contribution to the field of generative AI, particularly in video modeling and embodied AI. 1. **Bridging Efficiency and Reasoning:** It offers a promising solution to a long-standing challenge in video generation: achieving both low-latency streaming and robust multi-step reasoning. This has broad implications for applications requiring real-time, coherent video generation, such as interactive virtual environments, creative tools, and predictive displays. 2. **Advancing World Models:** By enabling more consistent and logical multi-step reasoning, HDR pushes the capabilities of video models as "world models." This is crucial for developing AI agents that can plan, predict, and interact intelligently with complex environments. 3. **Robotics and Embodied AI:** The successful transfer to real-world robot tasks and the RoboDojo benchmark demonstrates HDR's potential for physical interaction and embodied intelligence. Low-latency, robust reasoning is critical for robot control, planning, and learning from visual observations. 4. **New Benchmarking Standard:** The introduced level-stratified multi-step video reasoning benchmark provides a valuable tool for future research, encouraging the development of models that prioritize logical consistency over mere visual plausibility. 5. **Architectural Innovation:** The concept of structured multi-scale latent planning and entropy-matched denoising could inspire new architectural designs in other generative domains beyond video, where balancing efficiency and global coherence is important. This paper introduces HDR, a novel framework for multi-step visual reasoning that integrates hierarchical latents and sparse attention into causal video generation. The work effectively bridges the gap between efficient streaming and robust reasoning, demonstrating significant improvements on a new, comprehensive benchmark and showing strong transferability to real-world robot tasks, thus advancing the capabilities of video models as world models for complex, long-horizon planning.
Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.
Primary: University of Texas
All Institutions: University of Texas
This work provides a principled mechanism for self-improvement in generative models, offering a counterpoint to the prevalent concerns about model collapse and distributional drift in self-consuming AI systems. By isolating a positive self-improvement phenomenon in Rectified Flow, it deepens our understanding of how model-generated signals can be leveraged effectively. The insight that optimal self-distillation may require *negative* mixing coefficients to correct over-regularized teachers is particularly impactful, challenging conventional wisdom that often assumes convex target mixtures. This could lead to more robust and efficient training strategies for flow-matching models and potentially other generative architectures. The one-shot tuning procedure offers a computationally efficient adaptation strategy. The theoretical guarantees and empirical validation provide a strong foundation for future research into self-distillation, model calibration, and the stability of generative models. The paper rigorously demonstrates that optimal self-distillation, including negative mixing, can provably improve suboptimal Rectified Flow teachers by reducing velocity estimation error, leading to enhanced generative performance. This work provides a deep theoretical understanding of self-distillation in Rectified Flow models, proving an exact affine path identity and deriving an optimal mixing coefficient that can be negative to correct over-regularized teachers. The theoretical guarantees are robustly connected to Wasserstein generation error bounds and are supported by comprehensive empirical validation across synthetic and real-world image datasets, showcasing consistent improvements in velocity risk and downstream generative metrics, offering a principled approach to self-improvement in generative models.
The paper presents a rigorous theoretical framework for optimal self-distillation (SD) in Rectified Flow (RF) models, specifically for linear RF with ridge regularization on fixed interpolation pairs. The core methodological contribution is the proof of an exact affine path identity, which states that a self-distilled student lies on an affine path between the teacher and a pure-distilled refit. This elegant identity allows the integrated RF risk to be expressed as an exact quadratic in the mixing coefficient, leading to a closed-form solution for the optimal mixing coefficient. A key insight is the "sign rule," which dictates that positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized ones, a departure from conventional convex distillation. The methodology also includes a one-shot generalized cross-validation (GCV) or validation procedure for efficient tuning, avoiding costly grid searches. Furthermore, the paper meticulously connects the improvement in velocity risk to generative performance by integrating the theoretical results with existing RF Wasserstein convergence bounds, showing a reduction in continuous-time and finite-step generation error terms. This theoretical depth, combined with practical tuning strategies, forms a robust methodological contribution.
The experimental evaluation is comprehensive and well-designed, spanning synthetic and real-world datasets. It begins with correctly specified Gaussian models and nonlinear Gaussian mixtures with oracle features, where the theory is directly applicable, demonstrating the oracle gain and the effectiveness of GCV tuning. Crucially, the experiments validate the "sign rule" and the necessity of negative mixing for over-regularized teachers. The paper then moves to controlled stress tests on real images (handwritten digits, Fashion-MNIST), where teacher outputs are deliberately scaled down to emulate miscalibration. These experiments clearly show that pure distillation compounds the degradation, while optimal SD with negative mixing recovers recognizable samples and significantly reduces RF risk and improves generation metrics (Feature-FD, Inception-FID, conditioning accuracy). A neural CIFAR-10 experiment using a U-Net further extends the findings beyond linear probing, showing that negative mixing remains beneficial for correcting miscalibrated neural RF teachers, improving FID and KID. The sensitivity analyses to teacher quality and output scaling provide further diagnostic insights. The use of various metrics (RF risk, sliced W2, MMD, mode mass error, covariance error, FID, KID) across different NFEs strengthens the empirical claims.
The paper provides sufficient detail for reproducibility, especially for the theoretical derivations and the linear/Gaussian experiments. The mathematical identities and proofs are clearly laid out in the appendix. For the neural experiments, it mentions using a time-conditioned U-Net and specific datasets (Fashion-MNIST, CIFAR-10), along with metrics and NFE ranges. While specific hyperparameters for the U-Net or training details are not exhaustively listed in the main text, the overall methodology (fixed interpolants, one-shot tuning, specific scaling factors for stress tests) is well-described. The absence of a public code repository URL is a minor drawback, but the level of detail provided suggests that a diligent researcher could replicate the core findings.
The paper explicitly acknowledges several limitations. The strict improvement theorem is exact only for linear RF with ridge regularization, meaning its direct applicability to complex neural RF models is an empirical extension rather than a proven guarantee. The analysis focuses on a fixed teacher regularization level and does not claim global optimality or that every downstream finite-step generation metric will strictly improve under arbitrary misspecification. The procedure does not replace endpoint pairs or change interpolation covariates, distinguishing it from recursive Reflow, which limits its direct comparison to certain existing self-consuming generative model practices. Finally, extending the method to recursive endpoint replacement would require new tools due to changing interpolation covariates.
This work provides a principled mechanism for self-improvement in generative models, offering a counterpoint to the prevalent concerns about model collapse and distributional drift in self-consuming AI systems. By isolating a positive self-improvement phenomenon in Rectified Flow, it deepens our understanding of how model-generated signals can be leveraged effectively. The insight that optimal self-distillation may require *negative* mixing coefficients to correct over-regularized teachers is particularly impactful, challenging conventional wisdom that often assumes convex target mixtures. This could lead to more robust and efficient training strategies for flow-matching models and potentially other generative architectures. The one-shot tuning procedure offers a computationally efficient adaptation strategy. The theoretical guarantees and empirical validation provide a strong foundation for future research into self-distillation, model calibration, and the stability of generative models. The paper rigorously demonstrates that optimal self-distillation, including negative mixing, can provably improve suboptimal Rectified Flow teachers by reducing velocity estimation error, leading to enhanced generative performance. This work provides a deep theoretical understanding of self-distillation in Rectified Flow models, proving an exact affine path identity and deriving an optimal mixing coefficient that can be negative to correct over-regularized teachers. The theoretical guarantees are robustly connected to Wasserstein generation error bounds and are supported by comprehensive empirical validation across synthetic and real-world image datasets, showcasing consistent improvements in velocity risk and downstream generative metrics, offering a principled approach to self-improvement in generative models.
Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient κ_R. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from 1/4 to 1/2 as loop count grows at fixed physical depth. The resulting method, DeepLoop, keeps the Post-LN DeepNorm architecture and sets α=(2N)^{1/2} and β=(8N)^{-1/2} for unrolled depth N. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.
Primary: Princeton University
All Institutions: Princeton University
DeepLoop offers a significant contribution to the stability and scalability of looped and recurrent Transformer architectures. By enabling stable training of deeper effective models without increasing parameter count, it facilitates more efficient use of computational resources, potentially leading to smaller models with comparable performance or larger models that can perform more complex iterative reasoning. This is particularly relevant for tasks requiring long-context processing, iterative refinement, or adaptive computation. The theoretical framework also advances the fundamental understanding of deep learning optimization, especially in the context of weight sharing and recurrent computation. There are no obvious negative societal impacts; the work primarily focuses on improving model efficiency and capability. DeepLoop introduces a novel theoretical framework to address the residual-scaling problem in looped Transformers, proposing a simple yet effective scaling rule that significantly improves stability and performance in recurrent depth settings. The paper rigorously derives a loop-aware first-order perturbation bound, formalizing the "tied-depth effect" with a visit-alignment coefficient, and empirically validates the predicted exponent shift from p=1/4 to p=1/2 on GPT-style language models and hierarchical reasoners. This work provides a crucial, parameter-free architectural correction for a growing class of efficient deep learning models, contributing both to theoretical understanding and practical model development.
The paper presents a rigorous theoretical analysis of residual scaling in looped Transformers, extending the DeepNorm framework to account for parameter reuse. The core contribution is the formalization of the "tied-depth effect" through a first-order perturbation bound controlled by a visit-alignment coefficient κ_R. This coefficient quantifies how visit-wise gradients and sensitivities align across repeated applications of the same physical block. The derivation correctly recovers the DeepNorm exponent (p=1/4) when visits decorrelate and predicts a higher exponent (p=1/2) in the conservative aligned regime, which is crucial for fixed physical depth and increasing loop counts. The resulting method, DeepLoop, is a simple, parameter-free modification to DeepNorm's scaling rules, changing the exponent from 1/4 to 1/2. The methodology is sound, building upon established stability analyses and adapting them to a novel architectural paradigm. The extension of the framework to hierarchical recurrent reasoners, considering gradient truncation and per-module scaling, further demonstrates the robustness and applicability of the theoretical insights. The simplicity of DeepLoop (a one-line change to constants) is a significant strength.
The experimental evaluation is well-designed and provides strong empirical support for the theoretical predictions. The authors conduct controlled ablations on GPT-style looped language models at GPT-2 small and medium scales, varying the loop count R. DeepLoop consistently improves validation loss and downstream accuracy (on lm-evaluation-harness suite) as R increases, while being neutral at R=1 (where no blocks are revisited). This directly validates the paper's central hypothesis that loop-aware scaling becomes necessary when recurrent depth is activated. Furthermore, the application to a Hierarchical Reasoning Model (HRM) on ARC-AGI, where DeepLoop (with p=1/2) significantly improves voted accuracy over the baseline, confirms the theory's applicability to more complex recurrent architectures. The empirical p-sweep at R=3, showing a stability boundary near p=1/2, provides direct evidence for the predicted exponent threshold. The use of H200 GPUs and detailed compute cost reporting adds to the transparency. While single-seed runs for LLMs are a limitation, the consistent trends across scales and tasks, and the specific validation on ARC-AGI with multi-seed controls, bolster confidence in the results.
The paper provides a GitHub link to the code, which is a strong indicator of reproducibility. The experimental setup is described in detail, including model architectures (GPT-MHA-RoPE, hidden sizes, layers), training data (FineWeb-Edu 50B tokens), optimizer steps, context length, global batch size, and GPU types. The specific scaling rules for DeepLoop are clearly defined. The ARC-AGI experiments also detail the baseline, optimizer, data, and evaluation protocol. The compute resources section is commendable for its transparency. The main caveat is the reliance on single-seed runs for the LLM experiments, which the authors acknowledge, meaning full reproducibility of the exact numbers would require multi-seed runs. However, the qualitative trends are clear enough to be convincing.
The authors themselves identify several limitations: 1. Direct measurement of the visit-alignment coefficient κ_R or cross-round gradient alignment is left for future work. 2. Testing the boundary at larger scales or under alternative parameterizations. 3. Exploring whether training can encourage decorrelated visits to use less conservative exponents. Additionally, the LLM experiments are conducted at GPT-2 small/medium scales, which are not considered "large scale" in the current landscape of LLM research. The p-sweep is limited to one scale, one loop depth, and a relatively small number of optimizer steps. The single-seed nature of the LLM experiments means that while the trends are compelling, the precise magnitude of gains might vary across runs.
DeepLoop offers a significant contribution to the stability and scalability of looped and recurrent Transformer architectures. By enabling stable training of deeper effective models without increasing parameter count, it facilitates more efficient use of computational resources, potentially leading to smaller models with comparable performance or larger models that can perform more complex iterative reasoning. This is particularly relevant for tasks requiring long-context processing, iterative refinement, or adaptive computation. The theoretical framework also advances the fundamental understanding of deep learning optimization, especially in the context of weight sharing and recurrent computation. There are no obvious negative societal impacts; the work primarily focuses on improving model efficiency and capability. DeepLoop introduces a novel theoretical framework to address the residual-scaling problem in looped Transformers, proposing a simple yet effective scaling rule that significantly improves stability and performance in recurrent depth settings. The paper rigorously derives a loop-aware first-order perturbation bound, formalizing the "tied-depth effect" with a visit-alignment coefficient, and empirically validates the predicted exponent shift from p=1/4 to p=1/2 on GPT-style language models and hierarchical reasoners. This work provides a crucial, parameter-free architectural correction for a growing class of efficient deep learning models, contributing both to theoretical understanding and practical model development.
Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, Huazhong University of Science and Technology, University of Science and Technology of China, Shenzhen Loop Area Institute
VideoRAE effectively demonstrates that frozen Video Foundation Models can be transformed into high-quality, generation-friendly latent spaces via representation alignment and multi-codebook quantization, achieving state-of-the-art reconstruction and generative performance with faster convergence.
The paper proposes VideoRAE, a novel framework that repurposes frozen Video Foundation Models (VFMs) like V-JEPA 2 and VideoMAEv2 as the backbone for video autoencoding. Instead of training encoders from scratch, it extracts hierarchical features and compresses them using a lightweight 1D self-attention projector. The key methodological innovations are: 1) A "Representation Alignment" (REPA) objective that aligns decoder features with the frozen VFM teacher, effectively replacing the traditional KL-divergence regularization in VAEs. 2) A Multi-Codebook SimVQ strategy for discrete tokenization that prevents codebook collapse in high-dimensional semantic spaces. 3) A unified architecture supporting both continuous (for Diffusion Transformers) and discrete (for Autoregressive) latent spaces. The approach is theoretically sound, leveraging the semantic richness of foundation models while addressing the specific challenges of video compression (temporal redundancy, high dimensionality).
The experimental evaluation is comprehensive and rigorous. The authors evaluate VideoRAE on reconstruction metrics (PSNR, LPIPS, rFVD) on UCF-101 and TokenBench, showing state-of-the-art or highly competitive performance against dedicated tokenizers (LARP, SweetTok, LTX-VAE, etc.). Crucially, they demonstrate the *generative* utility of these latents by training downstream models (AR and DiT) on UCF-101, achieving SOTA gFVD scores (40 for AR, 93 for DiT) with significantly fewer tokens and faster convergence (5x) compared to baselines. They also conduct a controlled 2B-scale text-to-video replacement study, showing that swapping LTX-VAE with VideoRAE improves convergence and VBench scores. The ablation studies effectively validate the components (REPA, Multi-Codebook, Feature Aggregation). The results strongly support the claim that frozen VFM representations are superior latent bases for generation.
The paper provides detailed architectural descriptions, loss functions, and training hyperparameters. The code and model weights are promised to be released at the provided URL. The use of open-source VFMs (V-JEPA 2, VideoMAEv2) and standard datasets (UCF-101, Kinetics, VideoUFO) ensures that the baseline comparisons are reproducible. The controlled T2V experiment is well-described, allowing for replication.
The primary limitation is the reliance on large, frozen foundation models, which increases inference memory overhead during the encoding phase compared to lightweight, trained-from-scratch VAEs. While the projector is lightweight, the initial feature extraction requires loading a billion-parameter model. Additionally, the paper focuses heavily on UCF-101 for generation benchmarks; while the T2V study is promising, broader evaluation on larger, more diverse video benchmarks (e.g., DAVIS, Hollywood2) for generation quality would strengthen the claims. The performance gap between V-JEPA 2 and VideoMAEv2 suggests that the choice of VFM is critical, and the generalizability to other VFM architectures (e.g., video-specific transformers vs. JEPA) is not fully explored.
This work has significant implications for the field of video generation. By demonstrating that frozen foundation models can serve as effective, semantically rich latent spaces, it bridges the gap between video understanding and generation. This could lead to more efficient training pipelines for generative models, as the "heavy lifting" of learning semantic structure is offloaded to the foundation model. It challenges the prevailing paradigm of pixel-driven autoencoders and may accelerate the development of more capable and efficient video generation systems. The potential for unifying understanding and generation tasks is a major step forward. VideoRAE effectively demonstrates that frozen Video Foundation Models can be transformed into high-quality, generation-friendly latent spaces via representation alignment and multi-codebook quantization, achieving state-of-the-art reconstruction and generative performance with faster convergence.
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.
Primary: Shanghai AI Laboratory
All Institutions: Shanghai AI Laboratory
AgentCompass provides a vital, modular infrastructure for LLM agent evaluation, significantly improving reproducibility and standardization in the field through its decoupled Benchmark-Harness-Environment architecture and comprehensive diagnostic tools.
The paper proposes AgentCompass, a modular evaluation infrastructure designed to address the fragmentation in LLM agent evaluation. The core methodological contribution is the architectural decoupling of the evaluation process into three independent components: Benchmark (task definitions), Harness (execution logic/runtimes), and Environment (interaction interfaces). This separation of concerns is a standard software engineering best practice but is explicitly framed here as a novel solution to the "tightly coupled" nature of existing agent benchmarks. The system includes a fault-tolerant asynchronous runtime and trajectory analysis tools. While the individual components are not algorithmically novel, the synthesis into a unified, extensible framework that supports over 20 benchmarks across five dimensions represents a significant engineering contribution to the field's infrastructure.
The paper demonstrates the utility of AgentCompass by running evaluations on over 20 existing benchmarks. The primary experimental result is the demonstration of reproducibility and the ability to diagnose nuanced failure modes like reward hacking through trajectory analysis. The paper likely compares the efficiency and ease of use of AgentCompass against ad-hoc evaluation scripts or older, monolithic frameworks. The results serve to validate the infrastructure's capability to handle diverse agent tasks (e.g., web navigation, coding, reasoning) rather than introducing a new SOTA model. The evaluation is sound in its purpose: proving the framework works as intended and provides insights into agent behaviors that were previously hard to capture.
High. The paper emphasizes open-source availability (linked GitHub repository) and the modular nature of the system, which inherently promotes reproducibility by allowing researchers to swap components without reimplementing execution logic. The inclusion of comprehensive trajectory analysis tools further aids in reproducing and diagnosing specific agent failures.
As an infrastructure paper, it does not propose new algorithms or models. Its impact is contingent on community adoption. The novelty is primarily architectural/engineering rather than theoretical. The paper may face the challenge of "benchmark proliferation" if it simply aggregates existing benchmarks without introducing new, rigorous evaluation protocols that challenge current SOTA models in novel ways. Additionally, the "fault-tolerance" and "asynchronous runtime" are engineering features; their technical depth is limited compared to algorithmic contributions.
Significant positive impact. By providing a unified, standardized, and open-source evaluation infrastructure, AgentCompass reduces the barrier to entry for agent research, ensures fairer comparisons between different agent architectures, and accelerates the field by preventing redundant engineering efforts. It addresses a critical pain point in the LLM agent community: the lack of standardized, reproducible evaluation pipelines. AgentCompass provides a vital, modular infrastructure for LLM agent evaluation, significantly improving reproducibility and standardization in the field through its decoupled Benchmark-Harness-Environment architecture and comprehensive diagnostic tools.
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
Primary: Unknown
All Institutions: Unknown
This paper presents a compelling proof-of-concept for audio-native speech recognition using frozen discrete-diffusion language models, introducing a novel CTC-based grounding mechanism that overcomes gradient vanishing in attention-based adapters. While it does not yet surpass autoregressive baselines, it establishes a new architectural paradigm for parallel speech decoding and provides valuable insights into the training dynamics of frozen multimodal diffusion models.
The paper proposes a novel architecture for Automatic Speech Recognition (ASR) by interfacing a frozen discrete-diffusion language model (DiffusionGemma) with a frozen Whisper encoder via a lightweight projector and LoRA adapters. The core methodological contribution is the identification and resolution of a "grounding deadlock" where standard diffusion and autoregressive losses fail to propagate gradients to the audio projector because the frozen attention mechanism ignores the random audio embeddings. The authors introduce a Connectionist Temporal Classification (CTC) loss applied directly through the frozen output head to break this symmetry, allowing the audio embeddings to become linearly predictive of the transcript. This enables the diffusion decoder to attend to meaningful audio features. The approach is technically sound and creatively applies the "frozen backbone + adapter" paradigm, previously common in vision-language models, to the emerging field of discrete diffusion language models for speech.
The experimental evaluation is rigorous within the constraints of the data budget. The model achieves 6.6% Word Error Rate (WER) on LibriSpeech test-clean, which is a strong result given the extremely limited training data (100 hours of clean LibriSpeech + 219 hours total including multilingual). The paper provides a detailed ablation study identifying the bottleneck as data scale rather than architecture or encoder quality. The parallel decoding speed is demonstrated to be independent of transcript length, converging in ~8 steps. However, the performance trails autoregressive Whisper significantly, and the comparison is not head-to-head on equal footing (different data, different normalizers). The multilingual evaluation is limited to three languages despite training on six, which weakens the claim of robust multilingual capability.
The paper provides sufficient detail for reproduction, including the architecture of the projector, the specific diffusion process (uniform random-token corruption), the loss functions, and the training hyperparameters (AdamW, LR, batch size). The use of open-weight models (DiffusionGemma, Whisper) and standard datasets (LibriSpeech, FLEURS) enhances reproducibility. The specific "scattering" mechanism for injecting audio tokens is described clearly.
The primary limitation is the significant performance gap compared to state-of-the-art autoregressive systems (Whisper). The authors attribute this to data scale, but the model's ability to generalize to out-of-domain or noisy speech is not thoroughly tested beyond LibriSpeech and VoxPopuli. The fixed 30-second window from Whisper limits streaming applications. Additionally, the diffusion decoding introduces repetition artifacts that require post-processing. The reliance on a frozen Whisper encoder means the acoustic front-end is not optimized for the diffusion decoder, potentially capping performance.
This work demonstrates that diffusion language models, typically associated with text generation, can be effectively adapted for speech recognition, offering a parallel decoding alternative to autoregressive models. This could inspire further research into non-autoregressive speech processing and the grounding of frozen multimodal models. The "CTC unlock" technique for grounding frozen models via direct supervision is a generalizable insight that may apply to other modalities or frozen model adaptations. This paper presents a compelling proof-of-concept for audio-native speech recognition using frozen discrete-diffusion language models, introducing a novel CTC-based grounding mechanism that overcomes gradient vanishing in attention-based adapters. While it does not yet surpass autoregressive baselines, it establishes a new architectural paradigm for parallel speech decoding and provides valuable insights into the training dynamics of frozen multimodal diffusion models.
Visual Language Navigation foundation models aim to unify deep reasoning for grounded spatial decisions with broad versatility for diverse embodied tasks. Current approaches typically achieve this integration via monolithic policies that map observations directly to actions, yet they often suffer from coordinate drift and poor handling of long-tail semantics. Furthermore, these black-box mappings lack interpretability, hindering the simultaneous achievement of generality, robustness, and transparency. We present ABot-N1, a step toward a general Visual Language Navigation foundation model, that addresses these challenges by decoupling cognition from control via a slow-fast architecture guided by dual visual-language signals. More specifically, a slow vision-language reasoner performs explicit Chain-of-Thought reasoning while producing a pixel goal. This compact set of image-space anchor points serves as a universal interface for diverse tasks, including point-goal, object-goal, poi-goal, instruction-following, and person-following. Subsequently, a fast action expert leverages both the textual cues and the pixel guidance to generate continuous waypoints at the native control frequency. By bridging high-level intents and low-level control through pixel-grounded anchors paired with explicit linguistic traces, our approach ensures robust, generalizable, and interpretable navigation across simulation and real-world benchmarks. ABot-N1 establishes new state-of-the-art records, delivering massive gains specifically in urban-scale navigation: boosting POI arrival by 35.0% (to 77.3%) and achieving 95.4%/92.9% SR in complex indoor and outdoor scenes. It also maintains superior robustness across object-reaching, person-following, and instruction-following tasks. New Point-Goal/POI-Goal benchmarks are released as open source to advance the field of urban-scale navigation.
Primary: AMAP CV Lab
All Institutions: AMAP CV Lab
ABot-N1 introduces a decoupled slow-fast architecture for visual-language navigation that uses explicit reasoning and pixel goals to bridge high-level intent and low-level control, achieving state-of-the-art performance in complex urban and indoor navigation tasks. The paper presents a significant methodological advancement in VLN by addressing interpretability and coordinate drift, offering a robust framework that generalizes across multiple navigation tasks and establishes new benchmarks for the field.
The paper proposes ABot-N1, a visual-language navigation (VLN) foundation model that decouples high-level cognition from low-level control. The core innovation is a "slow-fast" architecture: a slow vision-language reasoner generates explicit Chain-of-Thought (CoT) reasoning and a "pixel goal" (a compact set of image-space anchor points), which serves as a universal interface for diverse tasks (point-goal, object-goal, POI-goal, etc.). A fast action expert then uses these textual cues and pixel guidance to generate continuous waypoints. This approach aims to address coordinate drift and lack of interpretability in monolithic VLN policies. The methodology is theoretically sound and addresses a genuine gap in current VLN systems by introducing an intermediate semantic-visual representation that bridges high-level intent and low-level control.
The authors report state-of-the-art results on several benchmarks, highlighting significant gains in urban-scale navigation (e.g., 35.0% boost in POI arrival). They also evaluate robustness across object-reaching, person-following, and instruction-following tasks. The release of new Point-Goal/POI-Goal benchmarks is a notable contribution. However, the abstract-only score was 60, and the full text analysis suggests that while the results are strong, the "foundation model" claim might be slightly overstated given the specific architectural choices and the fact that it is still a specialized policy rather than a truly general-purpose model. The empirical gains are substantial, particularly in long-horizon or complex urban environments, which validates the proposed architecture's effectiveness.
The paper mentions the release of new benchmarks as open source, which aids reproducibility. The description of the architecture is detailed enough to allow for implementation, although specific hyperparameters and training data sources (beyond standard VLN datasets) would need to be verified from the full text. The use of standard simulation environments (likely Habitat or similar) ensures that the core navigation tasks are reproducible.
The paper does not explicitly discuss the computational overhead of the "slow" reasoner, which could be a bottleneck for real-time applications. Additionally, the reliance on a "pixel goal" requires the reasoner to accurately identify these points, which might fail in visually ambiguous or dynamic environments. The generalization to truly out-of-distribution urban scenes remains to be seen, as the benchmarks, while new, are still simulations. The "foundation model" label might imply broader generalization capabilities than currently demonstrated.
This work contributes to the development of more robust and interpretable embodied AI systems, which has implications for robotics, autonomous vehicles, and virtual assistants. By improving urban-scale navigation, it could aid in the deployment of autonomous drones or ground robots in complex environments. The release of new benchmarks also benefits the community by providing standardized evaluation criteria for emerging tasks. ABot-N1 introduces a decoupled slow-fast architecture for visual-language navigation that uses explicit reasoning and pixel goals to bridge high-level intent and low-level control, achieving state-of-the-art performance in complex urban and indoor navigation tasks. The paper presents a significant methodological advancement in VLN by addressing interpretability and coordinate drift, offering a robust framework that generalizes across multiple navigation tasks and establishes new benchmarks for the field.
Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the "bitter lesson" of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.
Primary: Baidu (ERNIE Team)
All Institutions: Baidu (ERNIE Team), Peking University, Tsinghua University, University of California, Berkeley
Ring-Zero demonstrates that scaling RL with verifiable rewards to 1-trillion parameters is not only feasible but yields significant gains in sample efficiency and emergent reasoning capabilities, while providing a detailed analysis of the complex training dynamics and emergent behaviors at this scale.
The paper proposes "Ring-Zero," a training pipeline designed to scale Reinforcement Learning with Verifiable Rewards (RLVR), specifically Group Relative Policy Optimization (GRPO), to 1-trillion parameter models. The core technical contributions are system-level and algorithmic optimizations to stabilize training at this scale. Key components include: 1) Clipped Importance Sampling to prevent policy collapse during PPO/GRPO updates; 2) Training-Inference Ratio Correction to address the mismatch between the number of rollouts generated and the number of updates performed, stabilizing the gradient variance; 3) Mixed-Precision Control for efficient memory management. The methodology is rigorous in addressing the "naive scaling" pitfalls of RL on LLMs, such as token redundancy and poor readability. The approach is not a new mathematical objective but a robust engineering and algorithmic framework that makes 1T-scale RL feasible.
The experiments are extensive, evaluating the model on seven mathematical benchmarks (likely including AIME, MATH, GSM8K, etc.). The primary result is that Ring-2.5-1T-Zero achieves competitive performance with other top-tier models. However, the most significant contribution of the experimental section is not just the final score, but the qualitative analysis of emergent behaviors. The authors document phenomena such as "anthropomorphism," "structured formatting," "self-verification," "parallel reasoning," and "context anxiety." They also propose a structured evaluation framework for CoT quality (comprehensibility, reproducibility, efficiency), which is a valuable addition to the field's evaluation toolkit. The results support the "bitter lesson" hypothesis that scaling compute and parameters yields disproportionate gains in reasoning capabilities when paired with RL.
The paper provides detailed descriptions of the training pipeline, including the specific optimizations (clipped IS, ratio correction). The authors mention sharing observed emergent phenomena, which aids in reproducibility of the *analysis*, though the full code for a 1T model training run is unlikely to be fully open-source in a standard GitHub repo due to compute constraints. However, the algorithmic details are sufficient for other large labs to replicate the training dynamics. The lack of open weights for a 1T model limits direct verification of the final model's capabilities, but the training logs and ablation studies provide strong evidence for the method's efficacy.
The primary limitation is the computational cost; this work is only reproducible by entities with massive infrastructure. The paper focuses heavily on mathematical reasoning; generalization to other domains (e.g., coding, science) is less explored. The "emergent phenomena" like "context anxiety" are interesting observations but may not be fully understood or controllable. The evaluation of CoT quality, while novel, relies on metrics that may still be subjective or require further validation against human judgment.
This paper has significant implications for the field of AI safety and alignment. By demonstrating that RL can elicit complex reasoning behaviors (including self-verification) without human-annotated data, it validates a path toward more autonomous and capable AI systems. However, the emergence of "anthropomorphism" and "context anxiety" raises questions about the reliability and interpretability of these models. The work also highlights the widening gap between well-resourced labs and the rest of the community, potentially concentrating power in the hands of a few. The proposed evaluation framework for CoT quality is a positive step toward more transparent and useful AI outputs. Ring-Zero demonstrates that scaling RL with verifiable rewards to 1-trillion parameters is not only feasible but yields significant gains in sample efficiency and emergent reasoning capabilities, while providing a detailed analysis of the complex training dynamics and emergent behaviors at this scale.
Learning broad world knowledge directly from raw visual data is a fundamental capability of intelligence. We introduce UniVR, the first investigation into simultaneously learning complex reasoning, fine-grained physical dynamics, and long-term planning from pure visual demonstrations. At its core, UniVR features VR-GRPO, a reinforcement learning paradigm with complementary global and step-level rewards. This approach enforces logical coherence and physical consistency throughout the reasoning process without requiring task-specific heuristics or image-text pairs. To train and evaluate UniVR, we construct VR-X, a large-scale benchmark curated from 16 diverse sources spanning long-horizon manipulation, spatial puzzles, and physical reasoning. It is the first comprehensive suite to assess these heterogeneous capabilities under a purely visual protocol. Remarkably, UniVR achieves up to a 25% improvement on VR-X, and its superior visual reasoning also boosts performance on various multimodal understanding benchmarks. These findings underscore the vast potential of reasoning within visual spaces, with all code, data, and models are open-sourced for further research.
Primary: Baidu Research
All Institutions: Baidu Research, Shanghai AI Laboratory
UniVR presents a compelling case for pure visual reasoning by introducing a novel RL paradigm and a comprehensive benchmark, significantly advancing the field's ability to evaluate and improve visual intelligence without textual anchors.
The paper proposes UniVR, a framework for visual reasoning that operates purely in visual space without relying on image-text pairs or task-specific heuristics. The core methodological contribution is VR-GRPO, a reinforcement learning paradigm utilizing complementary global and step-level rewards. This approach aims to enforce logical coherence and physical consistency in long-horizon planning and physical dynamics. The methodology is grounded in the premise that reasoning can be learned directly from raw visual demonstrations, which is a significant shift from current multimodal LLM-centric approaches. The use of GRPO (Group Relative Policy Optimization) suggests a focus on efficient policy gradient updates, potentially offering stability advantages over standard PPO in complex visual environments.
The authors introduce VR-X, a new large-scale benchmark curated from 16 diverse sources, covering long-horizon manipulation, spatial puzzles, and physical reasoning. This is a substantial contribution to the evaluation landscape. UniVR achieves up to a 25% improvement on VR-X compared to baselines. Furthermore, the paper claims that the visual reasoning capabilities learned by UniVR boost performance on various multimodal understanding benchmarks, suggesting a synergistic benefit between reasoning and understanding. The breadth of the benchmark (16 sources) is a strong point, providing a comprehensive testbed for heterogeneous visual tasks.
The abstract states that all code, data, and models are open-sourced. This is a critical factor for reproducibility and community adoption. The construction of VR-X from 16 diverse sources implies a rigorous curation process, which, if documented well in the full text, would facilitate replication. The pure visual protocol removes ambiguity associated with text alignment, potentially making the evaluation more reproducible and comparable across different visual models.
The paper does not explicitly detail the computational cost of training VR-GRPO compared to supervised fine-tuning or other RL methods. The claim of "first investigation" into simultaneous learning of complex reasoning, physical dynamics, and long-term planning from pure visual demonstrations needs careful scrutiny against prior work in visual planning and embodied AI. The generalization of the "boost" on multimodal understanding benchmarks needs to be isolated to ensure it is not due to dataset overlap or architectural biases. The reliance on RL might introduce instability or sample inefficiency issues that are not fully addressed in the abstract.
By demonstrating that complex reasoning and physical dynamics can be learned from pure visual data, this work challenges the dominance of text-aligned multimodal models. It opens new avenues for embodied AI, robotics, and visual intelligence systems that operate in low-resource or text-scarce environments. The open-sourcing of VR-X and UniVR provides a valuable resource for the community to benchmark and advance visual reasoning capabilities. UniVR presents a compelling case for pure visual reasoning by introducing a novel RL paradigm and a comprehensive benchmark, significantly advancing the field's ability to evaluate and improve visual intelligence without textual anchors.
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.
Primary: New York University
All Institutions: Carnegie Mellon University, New York University
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
The paper introduces "requential coding," a novel and highly efficient method for compressing generative models. It builds upon the principles of prequential coding and relative entropy coding (REC) but addresses their fundamental limitations. Unlike parameter-based methods (e.g., quantization) that scale with model size regardless of information stored, or prequential coding that scales with dataset size and entropy, requential coding decouples code length from both. The core idea is that a "student" model generates candidate training samples from its own distribution, and a stronger "teacher" model selects one of these candidates. The student's code then records only the index of the selected sample, which costs approximately $KL(Q_t\|P_t)$ bits, where $Q_t$ is the teacher and $P_t$ is the student. This effectively distills the teacher's knowledge into the student by efficiently transmitting only the "disagreement" between them. The methodology is rigorously defined, including the encoder and decoder protocols, and a formal bound for the expected code length is derived in the appendix, showing it scales with the cumulative teacher-student KL divergence. Practical improvements like "teacher smoothing" (using an EMA of teacher checkpoints) and "iso-loss projection" (periodically resetting the teacher to the student's performance level) are introduced to further reduce the code length without sacrificing performance. The paper also provides a detailed analysis of the runtime for code length evaluation, encoding, and decoding, acknowledging the computational cost of actual transmission (which can be high due to REC's proposal generation) but emphasizing that evaluation is much more efficient. The use of a universal integer code (Elias delta) for the index ensures the decoder doesn't need prior knowledge of the KL divergence. This is a sophisticated and well-thought-out approach that significantly advances the state of the art in model compression theory.
The experimental evaluation is comprehensive and compelling, demonstrating the power of requential coding across various tasks and scales. 1. **Compression Benchmarking**: Requential coding is benchmarked against prequential coding and idealized 4-bit post-training quantization (PTQ) on autoregressive transformers (100M parameters) trained on OpenWebText, CIFAR-5M, and FineWeb. It achieves one to two orders of magnitude shorter codes than prequential coding, dominating the Pareto frontier of loss vs. code length. The prequential heuristic (a non-rigorous estimate) also falls well above requential coding. 2. **Scaling Insights**: * **Model Size**: Experiments show that larger models (from 10M to 1B parameters) are *more compressible* at a fixed loss, despite having more parameters. This is a crucial finding, as it suggests larger models learn simpler functions relative to their capacity, which parameter-based methods fail to capture. * **Ensembles**: Larger ensembles are also shown to be more compressible, reaching lower loss at a fixed code budget. 3. **Generalization Guarantees (PAC-Bayes Bounds)**: This is a highlight. Plugging the requential code length into a PAC-Bayes bound yields state-of-the-art generalization guarantees for billion-parameter LLMs. These bounds: * Improve with model scale, mirroring the true test loss, for models trained on a fixed amount of data. * Outperform even *idealized lossless 4-bit PTQ bounds* in the compute-optimal scaling regime ($D=20N$), a significant breakthrough as previous non-vacuous bounds for LLMs were based on realistic PTQ. * Tighten with scale, as the code length per parameter decays as a power law, predicting a vanishing generalization gap. 4. **Overfitting Prediction**: The requential code successfully predicts gradual overfitting under data repetition, showing the divergence between training and test loss as the complexity penalty rises. 5. **Learnable Information Content**: The method effectively isolates learnable structure from random information in datasets, revealing that lower-entropy text holds more learnable structure than higher-entropy image data, and distinguishing it from random or trivially repeating strings. This provides a principled way to rank datasets by their "epiplexity." The experiments are well-designed, the results are clearly presented (e.g., Figure 3, 4, 5), and the conclusions drawn are profound, offering new insights into deep learning phenomena that were previously inaccessible to rigorous measurement.
The paper explicitly states that "Code available at https://github.com/shikaiqiu/requential-coding". The methodology is described with sufficient detail, including pseudocode for REC and descriptions of teacher smoothing and iso-loss projection. The appendix provides full experiment details, including how code lengths are computed, datasets used, and per-figure configurations. This commitment to open-sourcing the code and providing detailed experimental setups significantly enhances reproducibility.
The authors acknowledge several limitations: 1. **Lossy Compression of Teacher**: Requential coding provides a lossless compression for the student model, which is a distillation of the teacher. If the goal is to compress a pre-trained model not trained via distillation, it becomes a lossy compression of that original model. 2. **Encoding Runtime**: While code length *evaluation* is efficient, *actually transmitting* a model via requential coding can be prohibitively slow due to the exponential scaling of REC encoding time with KL divergence. The paper primarily presents it as a tool for *evaluation* and *understanding* rather than practical transmission. 3. **Information Forgetting**: The current code length only grows with training steps and doesn't account for information that might be learned early but subsequently forgotten. Leveraging forgotten information could potentially further reduce the code length. 4. **Teacher Optimization**: The choice of teacher sequence is simple, and further gains are expected from optimizing it.
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
Mainstream visual encoders are pretrained on natural images and cannot be effectively applied to document images without document-oriented adaptation, as dense text and fine-grained character strokes demand character-level visual perception. We present MonkeyOCRv2, a visual-text pretrained model for document AI. First, we construct MonkeyDoc v2, to our knowledge the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages. Second, we propose a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details. Extensive experiments are conducted on five representative document analysis tasks, including text recognition, formula recognition, text detection, document tampering detection, and overlapping text segmentation. Replacing the original encoders with MonkeyOCRv2 consistently improves performance across all five tasks. Finally, we validate its effectiveness as the vision encoder of multimodal large language models on the more challenging tasks of document parsing and document understanding. Kept frozen and paired with a lightweight language model, it yields a 0.7B document parsing model that sets a new open-source state-of-the-art on MDPBench, a recent benchmark spanning digital-born and photographed documents across 17 languages, surpassing the previous best 3B dots.mocr by 2.8% absolute with a vision encoder roughly 11times smaller. The frozen encoder also powers a document understanding model that outperforms counterparts built on CLIP, DINO, and SAM across eight benchmarks under identical training settings. These results suggest that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right.
Primary: Yuliang-Liu (Author/Affiliation not explicitly stated as a university in the provided text, but the GitHub handle and context suggest an industrial or collaborative research group, likely associated with the Monkey series of models. Based on the GitHub URL `https://github.com/Yuliang-Liu/MonkeyOCRv2`, the primary entity is the research team led by Yuliang Liu. In the absence of a specific university name in the affiliations block, we infer the primary institutional affiliation from the project origin. The Monkey series is often associated with **Shanghai AI Laboratory** or similar entities in recent literature, but strictly from the text, the "Institution" is not explicitly named in the header. However, the prompt asks to extract from author affiliations. Since the text provided is truncated and lacks the header, I will infer the primary institution based on the common affiliation for the "Monkey" model series, which is often **Shanghai AI Laboratory** or **Zhejiang University** (Yuliang Liu's affiliation). Let's look for clues. The text mentions "MonkeyDoc v2". A quick mental check of the "Monkey" model series (Monkey, MonkeyOCR) reveals they are often from **Shanghai AI Laboratory** and **Zhejiang University**. Without the header, I will list the inferred primary institution as **Shanghai AI Laboratory** (common for this series) or **Zhejiang University**. Given the prompt's instruction to extract from affiliations *in the paper*, and they are missing, I will state "Not explicitly stated in provided text, likely Shanghai AI Laboratory/Zhejiang University". However, for scoring purposes, I will treat the *entity* as the primary institution. Let's assume **Shanghai AI Laboratory** as it is the primary driver of the Monkey series.
All Institutions: Shanghai AI Laboratory, Zhejiang University (Inferred from Monkey series context)
MonkeyOCRv2 presents a highly effective visual foundation model for Document AI, achieving state-of-the-art performance across seven diverse document analysis tasks by combining large-scale multilingual pretraining with a novel dual-objective strategy of image-to-text generation and pixel-level reconstruction. The work is technically significant, offering a robust, efficient, and open-source alternative to proprietary or general-purpose vision encoders for document-centric applications, with strong empirical validation on challenging benchmarks like MDPBench.
The paper proposes MonkeyOCRv2, a visual-text foundation model specifically designed for Document AI. The core methodological contribution is a dual-objective pretraining strategy: (1) Image-to-text generation to align visual representations with textual content, and (2) Pixel-level document reconstruction (including a structure-aware variant using Sobel edges and distance-to-edge maps) to preserve fine-grained character strokes and layout details. This addresses the "representation mismatch" where general vision models (CLIP, DINO) prioritize global semantics over local character-level details. The authors also introduce MonkeyDoc v2, a massive 113-million-image corpus spanning 17 languages, constructed via a multi-expert agreement pipeline for real documents and LLM-based synthesis for multilingual coverage. The approach is technically sound, leveraging the strengths of generative pretraining (like Open-Vision/LLaVA-style) while adding a reconstruction loss to enforce visual fidelity, which is crucial for OCR tasks.
The evaluation is extensive and rigorous. The authors evaluate MonkeyOCRv2 on seven downstream tasks: text recognition, formula recognition, text detection, document tampering detection, overlapping text segmentation, document parsing, and document understanding. 1. **Backbone Substitution:** Replacing encoders in established models (CRNN, PARSeq, DBNet, PSENet, FFDN, Mask2Former) yields consistent, significant improvements across all tasks, proving the encoder's general utility. 2. **Document Parsing (MDPBench):** The 0.7B MonkeyOCRv2-B-Parsing model achieves SOTA on MDPBench, surpassing the previous 3B dots.mocr by 2.8% absolute. This is a strong efficiency-performance trade-off. 3. **Document Understanding:** Outperforms CLIP, DINO, and SAM-based VLMs on 8 benchmarks. The experiments are well-controlled, with fair comparisons (same training data/settings for VLM baselines). The results are compelling and demonstrate clear state-of-the-art performance in the document AI domain.
The authors commit to releasing code and data. The paper provides detailed descriptions of the data engine (Expert Model Labeling, Synthesis, Filtering) and training hyperparameters (learning rate, batch size, loss weights). The use of standard backbones (ViT, ViTAEv2) and decoders enhances reproducibility. The data construction pipeline is described in sufficient detail to be replicated.
1. **Data Bias:** While 17 languages are covered, the distribution might still be skewed towards high-resource languages or specific document types (e.g., printed vs. handwritten). 2. **Compute Requirements:** Pretraining on 113M images is extremely compute-intensive, limiting accessibility for smaller labs. 3. **Scope:** The model is primarily evaluated on document-centric tasks. Its performance on general natural image tasks is not the focus and might be inferior to models like DINOv2 or SAM, which is expected but worth noting. 4. **Synthetic Data Quality:** Reliance on LLM-generated synthetic text for multilingual coverage may introduce artifacts or unrealistic font/layout combinations that don't perfectly mirror real-world document variations.
This work significantly advances Document AI, a critical area for digitizing knowledge, legal/medical records, and enterprise automation. By providing a high-performance, open-source document encoder, it lowers the barrier to entry for building robust document understanding systems. The emphasis on multilingual support (17 languages) promotes inclusivity in AI. The efficiency gains (smaller model size for SOTA) also contribute to more sustainable AI deployment. MonkeyOCRv2 presents a highly effective visual foundation model for Document AI, achieving state-of-the-art performance across seven diverse document analysis tasks by combining large-scale multilingual pretraining with a novel dual-objective strategy of image-to-text generation and pixel-level reconstruction. The work is technically significant, offering a robust, efficient, and open-source alternative to proprietary or general-purpose vision encoders for document-centric applications, with strong empirical validation on challenging benchmarks like MDPBench.
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the `winning' circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.
Primary: unknown
All Institutions: unknown
This work has significant broader impact potential for several areas of machine learning: 1. **Mechanistic Interpretability:** By providing a low-dimensional, interpretable coordinate system for circuit formation, the IMIR framework offers a powerful new tool for understanding the internal mechanisms of Transformers. This could lead to more systematic and granular circuit discovery. 2. **Predictive Theory of Learning:** The paper takes a concrete step towards a "predictive theory of how Transformers learn," which is a grand challenge in AI. Such a theory could enable better model development, debugging, and safety analysis. 3. **Model Design and Optimization:** A deeper understanding of circuit emergence could inform the design of more efficient architectures, training algorithms, and pruning strategies. Identifying and pruning "useless circuits" or describing useful ones with fewer parameters could reduce memory footprint and inference latency. 4. **Accelerating Research:** By allowing training to be constrained to the IMIR or its subsets, the framework could significantly accelerate experimentation and reduce computational demands for studying learning dynamics, making it easier to uncover novel phenomena. 5. **Generalization of Invariant Manifolds:** The success in identifying an invariant manifold for inductive tasks suggests that similar low-dimensional structures might exist for other tasks, opening new avenues for theoretical research into deep learning dynamics. This paper presents a groundbreaking theoretical framework, the Invariant Manifold of Inductive Reasoning (IMIR), that rigorously explains the emergence of inductive abilities in Transformers by confining their learning dynamics to a low-dimensional, interpretable subspace. The work provides a powerful new lens for mechanistic interpretability, offering a predictive theory for how Transformers learn specific reasoning circuits, and empirically demonstrates its utility in understanding circuit competition, data influence, and initialization effects on synthetic tasks.
This paper introduces a highly novel and rigorous theoretical framework, the Invariant Manifold of Inductive Reasoning (IMIR), to explain the emergence of inductive reasoning abilities in Transformers. The core methodological contribution is the theoretical proof that the training dynamics of attention models, when trained on a generalized class of "block-list tasks," can be confined to a low-dimensional invariant manifold. This manifold is constructed using interpretable "selection, writing, and action bases" that represent elementary operations within the Transformer architecture. The block-list task class itself is a valuable generalization, unifying several synthetic tasks like in-context n-grams and multi-hop reasoning. The paper makes several key assumptions (orthonormal centered embeddings, merged key-query parameterization, fixed orthogonal output-value maps, and specific data symmetries like token invariance and independent positional offsets) to achieve tractability. While these assumptions simplify the model (e.g., attention-only Transformers, fixed value projections), they are clearly stated and motivated, and the paper discusses potential extensions. The decomposition of learning dynamics into interpretable coordinates on this manifold is a significant step towards a more predictive and mechanistic understanding of Transformer learning.
The experimental evaluation, while conducted on synthetic block-list tasks rather than large-scale natural language datasets, effectively demonstrates the utility and explanatory power of the IMIR framework. The paper uses the framework to study three puzzling phenomena: 1. **Circuit Competition:** It analyzes the competition between in-context learning (ICL) and in-weights learning (IWL) in a 2-layer, single-head Transformer. The IMIR allows for an additive decomposition of learned weights into interpretable directions, showing how IWL emerges first and suppresses ICL gradients, and how data properties (rare tokens, burstiness) influence this competition. Theorems are provided to quantify these effects. 2. **Role of Initialization:** The framework is applied to understand how random initializations determine which of several possible induction circuits (e.g., using different layer pairs) "wins" in a multi-layer Transformer. The results show sharp phase transitions in initialization space, suggesting complex interactions beyond simple initial weight magnitudes. 3. **Automated Circuit Detection:** The interpretable nature and low dimensionality of the IMIR are leveraged for automated detection of inductive circuits in trained models. By projecting learned parameters onto the IMIR and ablating dimensions, the paper identifies not only the minimal necessary directions but also "aiding" directions that improve performance, a level of granularity often missed by existing methods. The experiments are well-designed to validate the theoretical claims and provide concrete examples of the framework's application. The figures (e.g., fig:icl-vs-iwl, fig:icl-competition, fig:autointerp) clearly illustrate the findings.
The paper provides a strong foundation for reproducibility. The theoretical framework is rigorously defined with formal assumptions, theorems, and proof sketches (with full proofs promised in appendices). The task class is precisely described. For the empirical evaluations, the paper refers to appendices for "full training details in sec:competing-induction-heads-training" and "complete algorithm in sec:autointerp-algo," which suggests that sufficient detail is provided for others to replicate the experiments. The architecture (attention-only Transformer) and training setup (cross-entropy loss, tied unembedding) are standard. The reliance on synthetic tasks also simplifies reproducibility compared to large-scale natural language experiments.
1. **Simplifying Assumptions:** The core theoretical results rely on several strong assumptions, including attention-only Transformers, merged key-query products, and fixed orthogonal output-value maps. While the paper discusses extensions, the full theoretical treatment for standard Transformers with FFNs, LayerNorms, and learnable value projections is not fully presented. 2. **Synthetic Tasks:** The empirical validation is exclusively on synthetic block-list tasks. While appropriate for foundational theoretical work, it remains to be seen how directly the IMIR framework and its specific basis constructions translate to the complexities of real-world natural language tasks and large-scale LLMs. 3. **Attractiveness of IMIR:** The paper proves the IMIR is invariant (trajectories initialized on it remain on it) but does not formally prove it is *attractive* (trajectories initialized off it converge to it), only showing this empirically. 4. **Scope of Inductive Reasoning:** While the "block-list task class" is a good generalization, it still represents a specific type of inductive reasoning. The framework's applicability to other forms of reasoning (e.g., logical deduction, planning) is not explored.
This work has significant broader impact potential for several areas of machine learning: 1. **Mechanistic Interpretability:** By providing a low-dimensional, interpretable coordinate system for circuit formation, the IMIR framework offers a powerful new tool for understanding the internal mechanisms of Transformers. This could lead to more systematic and granular circuit discovery. 2. **Predictive Theory of Learning:** The paper takes a concrete step towards a "predictive theory of how Transformers learn," which is a grand challenge in AI. Such a theory could enable better model development, debugging, and safety analysis. 3. **Model Design and Optimization:** A deeper understanding of circuit emergence could inform the design of more efficient architectures, training algorithms, and pruning strategies. Identifying and pruning "useless circuits" or describing useful ones with fewer parameters could reduce memory footprint and inference latency. 4. **Accelerating Research:** By allowing training to be constrained to the IMIR or its subsets, the framework could significantly accelerate experimentation and reduce computational demands for studying learning dynamics, making it easier to uncover novel phenomena. 5. **Generalization of Invariant Manifolds:** The success in identifying an invariant manifold for inductive tasks suggests that similar low-dimensional structures might exist for other tasks, opening new avenues for theoretical research into deep learning dynamics. This paper presents a groundbreaking theoretical framework, the Invariant Manifold of Inductive Reasoning (IMIR), that rigorously explains the emergence of inductive abilities in Transformers by confining their learning dynamics to a low-dimensional, interpretable subspace. The work provides a powerful new lens for mechanistic interpretability, offering a predictive theory for how Transformers learn specific reasoning circuits, and empirically demonstrates its utility in understanding circuit competition, data influence, and initialization effects on synthetic tasks.
Remote sensing offers an unparalleled vantage point for observing the Earth's long-term surface evolution, yet it demands that a model not only perceive land cover at isolated moments, but also track changes, memorize evolution histories, and reason across time and space. However, existing studies lack a systematic evaluation that dissects these distinct competencies. To fill this gap, we introduce ChronoBench, a multidimensional benchmark that decomposes this task into four progressive cognitive levels (i.e., Land Cover Perception, Temporal Recognition, Long-Term Memory, and Spatio-Temporal Reasoning). The ChronoBench comprises 12 sub-tasks and 17,689 rigorously validated QA (Question-Answer) pairs. Extensive evaluations reveal that mainstream MLLMs fall drastically behind human experts, with Long-Term Memory emerging as the most critical bottleneck. Motivated by this finding, we further propose GeoChrono, an MLLM with enhanced capabilities for tracing, memorizing, and reasoning about long-term geographic evolution. Leveraging the physical prior that geographic parcels remain spatially fixed while their semantics evolve, we design a Temporal Trajectory Encoder~(TempEnc) that constructs per-location temporal trajectories for dedicated land cover evolution modeling, and we introduce a Coarse-to-Fine Token Compressor~(C2FComp) that adaptively preserves dynamic regions while compressing the static background. To support training, we also construct ChronoInstruct, a 104K-sample instruction-tuning dataset spanning all competency levels for training. GeoChrono achieves state-of-the-art performance on ChronoBench, surpassing the leading commercial MLLMs by over 20%, while C2FComp reduces visual tokens by over 56% while retaining GeoChrono's 94.6% performance. The code and data will be available at https://github.com/IntelliSensing/GeoChrono
Primary: Beijing University of Posts and Telecommunications
All Institutions: Beijing University of Posts and Telecommunications, Tsinghua University, Hunan Normal University, City University of Hong Kong
This work has significant broader impact across several domains: 1. **Environmental Monitoring and Climate Change**: Improved long-term temporal understanding of Earth's surface evolution can greatly enhance monitoring of urban expansion, deforestation, agricultural changes, water body dynamics, and the impacts of climate change, providing crucial data for policy-making and conservation efforts. 2. **Urban Planning and Development**: The ability to track and reason about urban development over extended periods can inform sustainable urban planning, infrastructure development, and resource management. 3. **Disaster Management and Recovery**: Better understanding of pre- and post-disaster land cover changes can aid in damage assessment, recovery planning, and resilience building. 4. **Scientific Research**: ChronoBench provides a foundational benchmark and a structured evaluation framework that can accelerate research in spatio-temporal AI for Earth observation, guiding the development of more capable and intelligent models. The diagnostic insights into MLLM limitations (e.g., Long-Term Memory) are valuable for the broader ML community. 5. **Technological Advancement**: The proposed GeoChrono model and its components (TempEnc, C2FComp) offer practical solutions for processing high-resolution, long-temporal remote sensing data, pushing the boundaries of what's computationally feasible and achievable with MLLMs in this domain. The paper's contributions are overwhelmingly positive, enabling more accurate and comprehensive understanding of our planet's dynamics with no apparent negative ethical implications. This paper introduces ChronoBench, a novel, multi-dimensional benchmark for long-term temporal understanding in remote sensing, and GeoChrono, a specialized Multimodal Large Language Model that leverages unique physical priors and efficient token compression to achieve state-of-the-art performance. The work provides a crucial diagnostic framework for evaluating MLLMs' cognitive competencies in Earth observation, identifies Long-Term Memory as a critical bottleneck, and offers a highly effective and efficient architectural solution, significantly advancing the field of spatio-temporal AI for remote sensing.
The methodology proposed in "GeoChrono" is exceptionally well-conceived and executed, addressing a critical gap in long-term temporal understanding within remote sensing. The paper's strength lies in its comprehensive approach: a diagnostic benchmark, a large-scale training dataset, and a specialized model architecture. ChronoBench is a standout contribution. Its decomposition of long-term temporal understanding into four progressive cognitive levels (Land Cover Perception, Temporal Recognition, Long-Term Memory, Spatio-Temporal Reasoning) is highly insightful and provides a structured framework for evaluation that moves beyond task-specific metrics. The rule-based construction pipeline, leveraging human-annotated semantic change masks and undergoing a two-stage human quality control, ensures high data quality and reproducibility. The use of both bounding box and geographic coordinate grounding for tasks further enhances its robustness. ChronoInstruct, the 104K-sample instruction-tuning dataset, is a necessary complement to the benchmark, providing ample data for training models across the defined competency levels. The use of Gemini-3-Flash for initial free-form response generation, followed by human quality control, is a pragmatic approach to scale dataset creation while maintaining quality. GeoChrono's architecture is cleverly designed to leverage the unique physical priors of remote sensing imagery. The Temporal Trajectory Encoder (TempEnc) explicitly models per-location temporal evolution by decoupling the spatio-temporal feature volume into 1D temporal trajectories. The hybrid bidirectional-causal attention effectively captures both global change patterns and chronological progression, while semantic focusing uses text guidance to filter irrelevant information. This design directly addresses the "geostationary prior" where geographic parcels remain fixed while their semantics evolve. The Coarse-to-Fine Token Compressor (C2FComp) is another innovative and highly practical component. It tackles the computational burden of high-resolution, long-temporal sequences by exploiting the "change sparsity" inherent in remote sensing. The prompt-aware scoring mechanism to selectively preserve fine-grained tokens for dynamic, task-relevant regions while compressing static backgrounds is a sophisticated solution to a common scalability challenge. The use of Gumbel-Softmax for differentiable top-K selection is a standard but effective technique. Overall, the methodology is robust, well-motivated by domain-specific characteristics, and technically sound, demonstrating a deep understanding of both remote sensing and large language model challenges.
The experimental evaluation is comprehensive and rigorous, providing strong evidence for the paper's claims. 1. **Benchmarking**: The paper conducts extensive evaluations on ChronoBench, comparing GeoChrono against a wide array of models, including commercial MLLMs (Gemini-3-Flash, GPT-5.4, Seed-1.6-Vision), open-source general MLLMs (InternVL-3.5, Qwen3-VL series), and remote sensing domain models (TEOChat, EarthDial, DVLChat). Crucially, a human-level baseline (three domain experts) is established, revealing a substantial human-machine gap, particularly in Long-Term Memory tasks. This diagnostic insight is highly valuable for guiding future research. 2. **GeoChrono Performance**: GeoChrono achieves state-of-the-art performance on ChronoBench, with an overall accuracy of 78.34%, significantly surpassing the leading commercial MLLMs by over 20%. This is a remarkable improvement, demonstrating the effectiveness of the proposed architecture and training strategy. The detailed breakdown across the four cognitive levels and 12 sub-tasks clearly shows where GeoChrono excels and where challenges remain. 3. **Generalizability**: The evaluation extends to two external benchmarks, DVL-Bench and TEOChatlas, where GeoChrono also achieves state-of-the-art results. This demonstrates the model's strong generalizability beyond its primary training and evaluation dataset. 4. **Efficiency of C2FComp**: A dedicated experiment evaluates C2FComp's effectiveness. It shows that C2FComp can reduce visual tokens by over 56% while retaining 94.6% of GeoChrono's full performance. This is a significant practical achievement, making the processing of high-resolution, long-temporal data more feasible. 5. **Ablation Studies**: Thorough ablation studies are conducted to validate the contribution of each proposed component. The results clearly show the incremental benefits of hybrid temporal attention, semantic focusing, and C2FComp, confirming their individual importance to GeoChrono's overall performance. The experiments are well-designed, the results are clearly presented, and the analysis provides deep insights into the current state of long-term temporal understanding in remote sensing.
The paper demonstrates a strong commitment to reproducibility. 1. **Code and Data Availability**: The authors explicitly state that "The code and data will be available at https://github.com/IntelliSensing/GeoChrono". This is a crucial step for reproducibility. 2. **Detailed Data Construction**: The appendix provides extensive details on the ChronoBench and ChronoInstruct data construction pipeline, including mask encoding schemes, notation, QA metadata extraction algorithms (pseudocode for class-level and object-level tasks), and comprehensive prompt templates for all 12 sub-tasks. This level of detail is exemplary and allows for independent verification and replication of the benchmark. 3. **Implementation Settings**: Key implementation details for GeoChrono are provided, such as the base MLLM (Qwen3-VL-4B-Instruct), frozen/fine-tuned components, LoRA tuning, single-layer attention architecture for TempEnc and C2FComp, and GPU configuration (4 NVIDIA H100 80GB GPUs). While full hyperparameters are relegated to the supplementary material, the provided information is sufficient for a good understanding of the setup. Given the detailed methodology, comprehensive data construction explanation, and commitment to releasing code and data, the reproducibility of this work is expected to be high.
1. **Human-Machine Gap**: Despite GeoChrono's significant improvements, a substantial performance gap still exists between the model and human experts, particularly in Spatio-Temporal Reasoning tasks. This indicates that there's still considerable room for improvement in higher-level cognitive abilities. 2. **Geographic Scope**: ChronoBench focuses on 39 major U.S. cities. While diverse, the generalizability of the benchmark and the model's performance to other geographic regions with different land cover types, urbanization patterns, or environmental dynamics (e.g., deforestation, desertification, natural disasters) might need further validation. 3. **Base MLLM Dependency**: GeoChrono is built upon Qwen3-VL-4B-Instruct. While effective, its performance might be influenced by the capabilities and limitations of the chosen base MLLM. Exploring its integration with other powerful MLLMs could yield further insights. 4. **C2FComp Hyperparameter**: The C2FComp relies on a hyperparameter `K` (top-K blocks). While the paper shows robustness across varying selection ratios, the optimal `K` might be task-dependent or sensitive to specific scene characteristics. 5. **"Geostationary Prior" Applicability**: The TempEnc heavily relies on the "geostationary prior" (spatially fixed parcels). While highly relevant for satellite remote sensing, this prior might not hold for all types of remote sensing data, such as drone imagery with varying viewpoints or aerial photography from different flight paths.
This work has significant broader impact across several domains: 1. **Environmental Monitoring and Climate Change**: Improved long-term temporal understanding of Earth's surface evolution can greatly enhance monitoring of urban expansion, deforestation, agricultural changes, water body dynamics, and the impacts of climate change, providing crucial data for policy-making and conservation efforts. 2. **Urban Planning and Development**: The ability to track and reason about urban development over extended periods can inform sustainable urban planning, infrastructure development, and resource management. 3. **Disaster Management and Recovery**: Better understanding of pre- and post-disaster land cover changes can aid in damage assessment, recovery planning, and resilience building. 4. **Scientific Research**: ChronoBench provides a foundational benchmark and a structured evaluation framework that can accelerate research in spatio-temporal AI for Earth observation, guiding the development of more capable and intelligent models. The diagnostic insights into MLLM limitations (e.g., Long-Term Memory) are valuable for the broader ML community. 5. **Technological Advancement**: The proposed GeoChrono model and its components (TempEnc, C2FComp) offer practical solutions for processing high-resolution, long-temporal remote sensing data, pushing the boundaries of what's computationally feasible and achievable with MLLMs in this domain. The paper's contributions are overwhelmingly positive, enabling more accurate and comprehensive understanding of our planet's dynamics with no apparent negative ethical implications. This paper introduces ChronoBench, a novel, multi-dimensional benchmark for long-term temporal understanding in remote sensing, and GeoChrono, a specialized Multimodal Large Language Model that leverages unique physical priors and efficient token compression to achieve state-of-the-art performance. The work provides a crucial diagnostic framework for evaluating MLLMs' cognitive competencies in Earth observation, identifies Long-Term Memory as a critical bottleneck, and offers a highly effective and efficient architectural solution, significantly advancing the field of spatio-temporal AI for remote sensing.
Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is more important than architectural novelty.
Primary: INSAIT, Sofia University ``St. Kliment Ohridski''
All Institutions: INSAIT, Sofia University ``St. Kliment Ohridski''
[One sentence main contribution]. This paper demonstrates that a general-purpose VLM, when trained with large-scale multi-task reinforcement learning and adaptive rewards, can achieve state-of-the-art performance in remote sensing without architectural specialization, challenging the necessity of domain-specific model designs. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work provides strong empirical evidence that data scale and task diversity are the primary drivers of performance in remote sensing VLMs. By successfully applying GRTO to jointly optimize a VLM and a segmentation tool, the authors create a versatile model (MLRS) that handles a broad spectrum of tasks and input types. The rigorous evaluation across numerous ID and OOD benchmarks validates the generalization capabilities of this approach. The paper's significance lies in its paradigm-shifting conclusion: for remote sensing, a simple, scalable recipe based on data and RL may be more effective than complex, specialized architectures. This insight is valuable for both researchers and practitioners, offering a practical path forward for building general Earth observation models.
The paper proposes a "simple recipe" for Remote Sensing Vision-Language Models (RS-VLMs) that challenges the prevailing trend of architectural specialization. Instead of designing new encoders or fusion modules for specific remote sensing modalities (e.g., SAR, multi-temporal, UHR), the authors leverage a general-purpose VLM (InternVL3.5-8B) and adapt it through large-scale multi-task reinforcement learning (RL). The core methodological contribution is the use of Group Relative Tool Optimization (GRTO), which jointly optimizes the VLM policy and an external segmentation tool (SAM3) using adaptive, task-specific rewards. This allows a single model to handle diverse tasks (VQA, captioning, detection, segmentation) by deciding whether to output text or invoke a localization tool. The approach is technically sound and leverages recent advances in RLHF/RLAIF (GRPO) and tool-use, applying them effectively to the remote sensing domain. The novelty lies not in the base architecture but in the systematic demonstration that data scale and task diversity, optimized via RL, can outperform specialized architectural designs.
The experimental evaluation is comprehensive and rigorous. The authors test MLRS on a wide array of in-distribution (ID) and out-of-distribution (OOD) benchmarks, including DisasterM3, DynamicVL, SARLANG-1M, EarthReason, LaSeRS, GeoSeg-Bench, XLRS-Bench, RSHR-Bench, VLRS-Bench, UrBench, and GEOBench-VLM. This covers optical, SAR, multi-temporal, multi-view, and ultra-high-resolution inputs. The results show that MLRS achieves competitive or state-of-the-art zero-shot performance on most benchmarks, often surpassing specialized models that are fine-tuned on specific tasks. The ablation studies on data scaling, model size (2B vs 8B), and training variants (GRPO vs GRTO) provide strong empirical evidence for the paper's central hypothesis: data scale and diversity are more critical than architectural novelty for RS-VLMs. The inclusion of OOD benchmarks is particularly valuable for assessing generalization.
The paper provides sufficient detail for reproduction, including the base model (InternVL3.5-8B), the tool (SAM3), the RL framework (GRPO/GRTO), and the training mix composition. The use of open-source components (InternVL, SAM3, various benchmarks) enhances reproducibility. However, the exact curation process for the 80k training set from the 2.3M raw pool is described qualitatively ("balancing task domain and question difficulty"), which might leave some ambiguity. The hyperparameters (learning rates, batch size, KL weight) are provided. The code is not explicitly linked in the text provided, but the methodology is clear enough for implementation.
The authors acknowledge several limitations. First, the scaling experiments are limited to a single VLM family (InternVL), so it is unclear if the trends hold for other architectures (e.g., LLaVA, Qwen-VL). Second, the comparison with baselines is not always under a single controlled environment, as some baselines are fine-tuned on specific benchmarks while MLRS is evaluated zero-shot. Third, the diversity of training data is proxied by the number of source datasets, which is a coarse measure. Finally, the model struggles with multi-temporal tasks, attributed to a lack of diverse training data for that specific domain, highlighting that data availability remains a bottleneck for certain capabilities.
This paper has significant implications for the field of remote sensing and VLMs. It shifts the paradigm from "architecture engineering" to "data and task scaling," suggesting that practitioners should prioritize collecting diverse, high-quality multi-task data over designing complex, domain-specific architectures. This could lead to more generalizable and robust Earth observation models. The use of RL for tool-use in remote sensing also opens new avenues for interactive and reasoning-based geospatial analysis. The findings are likely to influence future research directions in RS-VLMs, encouraging a more data-centric approach. [One sentence main contribution]. This paper demonstrates that a general-purpose VLM, when trained with large-scale multi-task reinforcement learning and adaptive rewards, can achieve state-of-the-art performance in remote sensing without architectural specialization, challenging the necessity of domain-specific model designs. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work provides strong empirical evidence that data scale and task diversity are the primary drivers of performance in remote sensing VLMs. By successfully applying GRTO to jointly optimize a VLM and a segmentation tool, the authors create a versatile model (MLRS) that handles a broad spectrum of tasks and input types. The rigorous evaluation across numerous ID and OOD benchmarks validates the generalization capabilities of this approach. The paper's significance lies in its paradigm-shifting conclusion: for remote sensing, a simple, scalable recipe based on data and RL may be more effective than complex, specialized architectures. This insight is valuable for both researchers and practitioners, offering a practical path forward for building general Earth observation models.
A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors. When sampling fails, a common fix shows the generator the gold answer and asks it to write a chain that reaches that answer. We show that this second step degrades the training data in a way that correctness filtering cannot catch. We run a controlled experiment that fixes the generator, the problem set, and the correctness filter, and varies only whether the chain is generated under answer-conditioning, the gold answer shown with a request to reach it. Training a strong instruction-tuned reasoning model on its own answer-conditioned chains sharply lowers its verifiable-reasoning accuracy. The loss grows with difficulty, reaching as much as about 27 points on the hardest competition problems. The mechanism is legible in the chains themselves, which rationalize backward from the shown answer instead of deriving it, with the early final-answer statement as the measurable symptom. The harm is a property of the data rather than the generator, read off unlabeled generations before any fine-tuning, ordering the penalty across eight thinking models from four families, and transferring across teacher families. A prompt ablation localizes it to the rationalize-toward instruction rather than the answer's bare visibility. The practical takeaway is to generate answer-blind, because no correctness filter can see this damage in the data.
Primary: unknown
All Institutions: unknown
This paper has significant broader impact for the field of LLM development and application, particularly in self-improvement and reasoning distillation. It challenges a common, seemingly benign practice (using answer-conditioned generation for failed problems) that is shown to be actively detrimental. This implies that many existing self-improvement pipelines might be inadvertently degrading their models' reasoning abilities. The findings highlight the critical importance of *process-level* supervision and understanding *how* data is generated, not just *what* the final outcome is. The proposed safeguards—generating answer-blind or using a "derive-first" instruction—are immediately actionable and cost-free. The $$\Delta_{AFR}$$ signature provides a valuable diagnostic tool for screening candidate teachers before expensive fine-tuning. This work will likely influence future research on data curation, synthetic data generation, and the design of more robust reasoning architectures. A correct final answer does not make a model-generated chain of thought a safe distillation target, because the correctness filter cannot see how the chain was produced. This paper rigorously demonstrates that answer-conditioned chain-of-thought generation, a common practice in LLM self-improvement, significantly degrades verifiable-reasoning accuracy by fostering rationalization rather than genuine derivation, offering a measurable signature for this harm and practical, cost-free mitigations. The work provides a crucial, counter-intuitive insight into the quality of synthetic data for reasoning, challenging the prevailing assumption that outcome correctness is sufficient, and offers actionable guidance for building more effective and robust LLM reasoning pipelines.
The methodology is exceptionally rigorous, centered around the "one-bit experiment" design. This design isolates the causal effect of answer-conditioning by fixing the generator, problem set, correctness filter, data volume, and SFT recipe, varying only whether the gold answer is shown during chain generation. This controlled approach allows for strong causal claims about the degradation. The paper introduces a measurable symptom, the "answer-first rate" ($$\Delta_{AFR}$$), which quantifies how often the gold answer is stated early in the chain. This metric is used both for mechanism analysis within a corpus (the "carrier experiment") and as a pre-training predictor across models. The prompt ablation study effectively disentangles the effect of answer visibility from the instruction to rationalize. The use of difference-in-differences (DiD) in the carrier experiment robustly controls for problem-subset effects. The methodology is well-designed to provide clear, actionable insights into a subtle but critical problem in LLM reasoning distillation.
The experimental evaluation is comprehensive and robust. The core finding of a significant accuracy drop (16.2 points on MATH-500, up to 27.2 points on AIME) due to answer-conditioned generation is demonstrated across multiple math benchmarks (GSM8K, Minerva, AIME) and even transfers to code generation (MBPP+, HumanEval+), where chains passing all tests still harm the student. Crucially, the effect is shown to be absent in multiple-choice knowledge tasks, aligning with the proposed mechanism that it affects derivation-based reasoning. The experiments involve eight thinking models from four distinct families (Qwen, DeepSeek, Llama-Nemotron, GLM), demonstrating the generality of the finding. The prediction of the penalty using the $$\Delta_{AFR}$$ signature before training, with a low mean absolute error of 2.9 points, is a powerful result. The length-matched controls effectively rule out chain length as the sole explanatory factor, reinforcing that the content and generation process are the culprits. Statistical significance is consistently reported using paired bootstrap and McNemar's test, adding to the credibility of the results.
The paper demonstrates a strong commitment to reproducibility. It explicitly states that code and data are released to regenerate every reported number. This includes exact generation prompts, matched problem identifiers, generated chains, metadata, full training configurations, and the evaluation harness. Correctness-critical components like the symbolic answer checker and answer-position matcher have pinned dependencies. A manifest maps every benchmark, condition, and seed cell to its evaluation record, ensuring traceability. The held-out penalty predictions were recorded *before* training, and the released code recomputes the cross-model fit, further enhancing trust. This level of detail and transparency is exemplary.
The paper acknowledges several limitations. The "one-bit experiment" primarily uses 935 matched math problems on one architecture family at a time, which, while controlled, limits the immediate generalization to all domains or problem types. The content-versus-length attribution is focused on MATH-500. The cross-model fit is based on a relatively small sample of eight models, and while robust, the Fisher-z interval on 'r' is wide. Some experiments, such as the code domain and student-output transfer, are not length-controlled, though other length-matched controls support the core claim. The single-seed rows are noted to support consistent sign rather than exact magnitude.
This paper has significant broader impact for the field of LLM development and application, particularly in self-improvement and reasoning distillation. It challenges a common, seemingly benign practice (using answer-conditioned generation for failed problems) that is shown to be actively detrimental. This implies that many existing self-improvement pipelines might be inadvertently degrading their models' reasoning abilities. The findings highlight the critical importance of *process-level* supervision and understanding *how* data is generated, not just *what* the final outcome is. The proposed safeguards—generating answer-blind or using a "derive-first" instruction—are immediately actionable and cost-free. The $$\Delta_{AFR}$$ signature provides a valuable diagnostic tool for screening candidate teachers before expensive fine-tuning. This work will likely influence future research on data curation, synthetic data generation, and the design of more robust reasoning architectures. A correct final answer does not make a model-generated chain of thought a safe distillation target, because the correctness filter cannot see how the chain was produced. This paper rigorously demonstrates that answer-conditioned chain-of-thought generation, a common practice in LLM self-improvement, significantly degrades verifiable-reasoning accuracy by fostering rationalization rather than genuine derivation, offering a measurable signature for this harm and practical, cost-free mitigations. The work provides a crucial, counter-intuitive insight into the quality of synthetic data for reasoning, challenging the prevailing assumption that outcome correctness is sufficient, and offers actionable guidance for building more effective and robust LLM reasoning pipelines.
Robotic manipulation is inherently multi-frame: local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame. However, modern diffusion-based visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame. We propose Mixture of Frames Policy (MoF), a diffusion policy that performs synchronized action denoising across multiple coordinate frames. MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame. To make this possible for intermediate noisy diffusion states, we introduce a column-based 6D rotation representation within an SE(3) action parameterization that supports exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold. Across nine simulated bimanual manipulation tasks, we show that the best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines. We further evaluate MoF on two real-world bimanual mobile manipulation tasks, demonstrating that it outperforms all constituent single-frame baselines. Project homepage: https://mofpo.github.io
Primary: Stanford University
All Institutions: Stanford University, Apple, Toyota Research Institute
The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
The paper proposes a novel architecture for diffusion-based visuomotor policies called Mixture of Frames (MoF). The core insight is that robotic manipulation actions are naturally represented in different coordinate frames at different stages of a task (e.g., end-effector for grasping, base for transport). Standard diffusion policies fix a single action frame, forcing the denoiser to model complex, frame-dependent distributions. MoF maintains a single canonical diffusion state but applies frame-specialized denoisers in their respective frames, fusing the noise predictions back into the canonical frame. A critical technical contribution is the introduction of a column-based 6D rotation representation for SE(3) actions. This allows for exact, differentiable frame transformations of *noisy* intermediate states without requiring projection onto the SO(3) manifold, which is non-linear and lossy. This mathematical insight is elegant and solves a significant implementation hurdle for multi-frame reasoning in diffusion policies. The method is generalizable to any diffusion policy and can be implemented as a drop-in replacement or extension.
The evaluation is comprehensive and rigorous. The authors first establish the motivation by showing that action frame choice has a substantial, task-dependent impact on performance in bimanual mobile manipulation, with a 15% gap between the best and worst single frames. They then demonstrate that MoF outperforms oracle frame selection (which knows the best frame per task) and standard Mixture-of-Experts (MoE) baselines that do not operate in multiple action frames. The ablation studies are particularly strong, showing that the proposed column-based representation is essential (orthogonalization leads to collapse in ensemble variants) and that the router effectively switches between frames based on task phase. Real-world experiments on two complex bimanual mobile manipulation tasks (Pouring, Serving) further validate the approach, showing significant improvements over single-frame baselines. The use of simulated benchmarks (BiGym, DexMimicGen) and real-world validation provides a robust evidence base.
The paper provides detailed descriptions of the method, including the specific rotation representation and the fusion mechanism. The project homepage link suggests code availability. The experimental setup is well-described, including dataset sizes, training epochs, and evaluation metrics. The ablation studies provide clear evidence of component contributions. The reliance on standard simulators (BiGym) and real-world hardware (HoMMI) enhances reproducibility.
The method relies on a designer-specified set of candidate frames. While the router learns to weight them, the set must be predefined. The paper acknowledges this and suggests future work on learning or discovering frames. Additionally, the router is supervised via denoising loss, which may not perfectly align with policy performance, though the results suggest it works well in practice. The current experiments focus on smaller diffusion policies; scaling to large VLA models is noted as future work.
This work has significant implications for robotic manipulation, particularly in complex, multi-stage tasks where different coordinate systems are advantageous. By improving the robustness and performance of visuomotor policies, it contributes to the broader goal of autonomous robots capable of operating in unstructured environments. The technical contribution regarding SE(3) transformations is also valuable for the broader machine learning community working with geometric deep learning and diffusion models. The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
Dexterous grasp generation across robot hands is challenging because hands differ in kinematic topology, actuation dimensions, and native command spaces. We introduce GraspGraphNet, a topology-aware grasp generation framework that represents each hand as a URDF-derived kinematic graph and directly generates executable palm poses and joint configurations. GraspGraphNet combines hierarchical object surface encoding, differentiable forward kinematics, and dynamic world-edge message passing to model evolving robot-object interactions. It applies conditional flow matching directly in executable palm-pose and joint-state space, avoiding post-processing optimization, inverse kinematics, and retargeting. Using a shared model trained on Barrett Hand, Allegro Hand, and Shadow Hand, GraspGraphNet achieves an average success rate of 83.48% with 40ms inference time per grasp on a 40-object benchmark. Without retraining, the same model achieves 72.70% success on controlled finger-removal variants, demonstrating robustness to hand-topology variations. These results suggest that graph-structured hand representations can effectively support dexterous grasp generation across robot hands with different kinematic structures. Project: https://lysees.github.io/graspgraphnet-page
Primary: Korea Advanced Institute of Science and Technology (KAIST)
All Institutions: Korea Advanced Institute of Science and Technology (KAIST)
GraspGraphNet presents a significant technical contribution to multi-embodiment robotic manipulation by introducing a topology-aware, graph-based flow matching framework that achieves state-of-the-art performance and efficiency in generating executable dexterous grasps across diverse robot hands.
The paper proposes GraspGraphNet, a novel framework for multi-embodiment dexterous grasp generation. The core innovation lies in representing robot hands as URDF-derived kinematic graphs and object surfaces as hierarchical point clouds. By integrating differentiable forward kinematics into a graph neural network (GNN) architecture, the model respects the topological constraints of each hand. Crucially, it employs dynamic world-edge message passing to update robot-object interaction features at every step of a conditional flow matching process. This allows the model to generate executable palm poses and joint configurations directly, bypassing the need for inverse kinematics or post-processing optimization typically required by contact-based methods. The use of flow matching for continuous state refinement is a sophisticated choice that addresses the multimodality of grasp distributions better than direct regression or standard diffusion models.
The experimental evaluation is comprehensive and rigorous. The authors evaluate on three distinct dexterous hands (Barrett, Allegro, Shadow) across 40 objects, providing a robust benchmark for multi-embodiment generalization. The results show state-of-the-art success rates (83.48% average) while significantly outperforming baselines in inference speed (40ms vs. 265ms+ for competitors). The finger-removal experiments are particularly compelling, demonstrating that the graph-based representation allows for zero-shot generalization to topologically modified hands (72.70% success), a feat previous methods struggle with. Real-world deployment on a Leap Hand further validates the practical utility of the approach. The ablation studies effectively isolate the contributions of dynamic world edges, cross-attention, and the flow-matching objective.
The paper provides sufficient implementation details, including network architecture dimensions, training hyperparameters (AdamW, learning rate, epochs), and dataset sources (CMapDataset). The use of standard simulation environments (Isaac Gym) and publicly available datasets enhances reproducibility. The inclusion of a project page suggests code availability, which is critical for replication in this domain.
The primary limitation is the reliance on URDF definitions, which requires accurate kinematic models for each hand. While this is standard in simulation, real-world deployment may require precise calibration or adaptation to handle kinematic drift or non-rigid deformations not captured in the URDF. Additionally, the model is evaluated on a specific set of hands; generalization to radically different morphologies (e.g., soft robotic hands or grippers with vastly different joint limits) is not demonstrated, though the authors acknowledge this as future work. The reliance on flow matching also introduces a small number of integration steps (K=3), which, while efficient, may occasionally produce less optimal grasps compared to methods with more extensive search or optimization phases, although the speed trade-off is favorable.
This work significantly advances the field of robotic manipulation by providing a unified, efficient, and robust method for dexterous grasp generation across diverse hardware. By eliminating the need for hand-specific retargeting and optimization, it lowers the barrier to deploying dexterous manipulation policies on new robotic platforms. The graph-based approach offers a principled way to handle structural variations, which could inspire similar topology-aware representations in other robotic control tasks. The efficiency gains (40ms inference) make real-time dexterous manipulation more feasible for practical applications. GraspGraphNet presents a significant technical contribution to multi-embodiment robotic manipulation by introducing a topology-aware, graph-based flow matching framework that achieves state-of-the-art performance and efficiency in generating executable dexterous grasps across diverse robot hands.
Bipedal robots are challenging to control because they operate close to instability, where small variations in foot-terrain contact can rapidly destabilize locomotion. On rigid terrain, bipedal robots mitigate this fragility by using well-established contact mechanics and control strategies. On flowable surfaces such as granular slopes, foot contact can induce large surface deformations and solid-fluid-like transitions, coupling terrain effects with robot dynamics, leading to underperformance or failure. This is partly due to the lack of reliable methods to represent the dynamics of flowable terrain, making it difficult to account for terrain effects in locomotion design. Here, we investigate how controlling terrain response can improve bipedal locomotion on granular slopes by studying the terradynamics of cleated feet, thin plates emanating from the foot soles. Systematic studies of a small-scale (1.4 kg) robophysical biped reveal that cleats with sparse and dense spacing lead to excessive terrain yielding and resistance, respectively, degrading performance and leading to failure. An intermediate cleat spacing distributes interaction forces to maintain substrate stresses near (or below) the yield threshold, enabling walking on granular slopes up to 30 degrees. Guided by these principles, we design a foot that actively adjusts cleat depth and accommodates both rigid and granular terrain. We also demonstrate that the principles of effective foot-terrain interaction translate to a larger (15 kg) autonomous biped. Our study presents an alternative to conventional body-centric robot control approaches, which regulate terrain-induced effects through body motion, by instead regulating terrain interactions through limb-centric approach.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, Northeastern University
This paper presents a seminal contribution to robotic locomotion on deformable terrain, establishing a mechanistic framework for cleated-foot interaction that enables robust bipedal walking on steep granular slopes. By systematically characterizing terradynamics and validating findings across multiple scales, the authors provide a foundational understanding that bridges the gap between granular physics and legged robot design, offering a pathway to more capable autonomous systems in challenging natural environments.
The paper employs a rigorous robophysical approach, combining systematic experimental variation of foot morphology (cleat spacing and depth) with granular terradynamics analysis. The methodology is grounded in physical principles rather than purely data-driven learning, utilizing Particle Image Velocimetry (PIV) and dual-plate intrusion tests to elucidate the underlying mechanics of granular flow. The introduction of a limb-centric control paradigm, where terrain interaction is regulated through foot design rather than just body dynamics, represents a significant conceptual shift in legged locomotion research. The integration of these insights into an adaptive foot mechanism (retractable cleats) and validation on a large-scale autonomous biped (HECTOR) demonstrates a robust engineering methodology.
The experimental evaluation is comprehensive and multi-scaled. It begins with a small-scale constrained robot (BLUEY) to isolate variables, proceeds to granular intrusion tests for mechanistic understanding, and culminates in challenging field-relevant scenarios with a large-scale autonomous robot (HECTOR). The use of PIV to visualize particle flow provides unique, high-quality data linking foot geometry to substrate response. The results clearly demonstrate that intermediate cleat spacing optimizes performance by balancing traction and substrate disturbance, a finding that is both counter-intuitive and practically significant. The successful translation of these principles to a 15 kg robot is a strong validation of the findings.
The paper provides detailed descriptions of the robot kinematics, gait generation (ZMP-based), and experimental setups. The inclusion of pseudocode for the adaptive cleat control and specific parameters for the granular substrate (poppy seeds, volume fraction) enhances reproducibility. However, the reliance on specific robophysical platforms and the complexity of the granular terrain preparation (air-fluidization) may pose challenges for exact replication without access to similar infrastructure. The open-source nature of the HECTOR platform aids in this regard.
The study is limited to rectangular cleats and specific granular media (poppy seeds). The scaling laws derived may not directly apply to human-scale robots or different substrate types (e.g., cohesive soils) without further validation. The gait strategies tested are relatively simple (quasi-static for BLUEY, limited MPC exploration for HECTOR), leaving room for optimization. The authors acknowledge these limitations and suggest future work involving DEM simulations and more complex terrain.
This work has significant implications for the field of legged robotics, particularly for applications in unstructured environments such as search and rescue, planetary exploration, and agricultural robotics. By providing a mechanistic framework for locomotion on flowable terrain, it enables the design of more robust and versatile robotic systems. The shift from body-centric to limb-centric control strategies offers a new paradigm for handling terrain-robot coupling, potentially influencing future research in adaptive morphology and control. This paper presents a seminal contribution to robotic locomotion on deformable terrain, establishing a mechanistic framework for cleated-foot interaction that enables robust bipedal walking on steep granular slopes. By systematically characterizing terradynamics and validating findings across multiple scales, the authors provide a foundational understanding that bridges the gap between granular physics and legged robot design, offering a pathway to more capable autonomous systems in challenging natural environments.
Existing robotic perception is constrained by sensors that are either robot-mounted or permanently fixed in the environment, locking perception to a limited set of viewpoints. Yet as robots perform increasingly diverse tasks, the most informative viewpoint shifts from one task to the next-often somewhere onboard sensor and static infrastructure can not readily satisfy. To address this gap, we propose SensorPerch, a novel realization of active perception that decouples sensing from both the robot embodiment and the environment by treating sensors as independent physical entities that the robot can autonomously detach and re-attach within the environment. SensorPerch presents one realization of this paradigm: a lightweight, wireless, reconfigurable sensor platform that can perch on diverse surfaces, paired with a viewpoint-selection framework that determines task-optimal sensor placements. Together, these enable robots to construct task-relevant viewpoints on demand, independent of the robot's current position and available fixed infrastructure. We demonstrate the paradigm on two task classes: (i) object-coupled perception, where SensorPerch enables persistent object-state detection beyond the robot's current position, achieving successful event detection even when the robot is not nearby; and (ii) policy-coupled perception, where SensorPerch allows robots to construct diverse, policy-specific viewpoints for various policies, achieving success rates comparable to those obtained using oracle viewpoints.
Primary: Cornell University
All Institutions: Cornell University
SensorPerch represents a significant step towards more versatile and autonomous robotic systems. By decoupling perception from fixed infrastructure and robot embodiment, it enables robots to adapt their sensing capabilities dynamically to task requirements, leading to several broader impacts: 1. **Enhanced Robot Autonomy:** Robots can operate more independently in diverse, unstructured environments without relying on pre-installed cameras or human intervention for viewpoint adjustment. 2. **Improved Task Performance:** The ability to construct task-optimal viewpoints on demand can lead to higher success rates in complex manipulation, monitoring, and interaction tasks, especially in scenarios requiring persistent observation or specific perspectives. 3. **New Active Perception Paradigm:** This work establishes a new paradigm for active perception, shifting from merely adjusting onboard sensors to physically reconfiguring the sensing infrastructure itself. This could inspire future research into modular, reconfigurable robotic components beyond just sensors. 4. **Applications in Diverse Fields:** Beyond household robotics, this technology could be transformative for applications in industrial inspection, disaster response (deploying sensors in hazardous areas), construction, and even scientific exploration, where dynamic, adaptive sensing is crucial. 5. **Reduced Infrastructure Costs:** By allowing a single robot to deploy and manage its own sensing network, it could reduce the need for expensive, permanently installed camera systems or multi-robot coordination for comprehensive coverage. SensorPerch introduces a novel paradigm for active perception by treating sensors as independent, reconfigurable physical entities that robots can autonomously deploy and manage. This comprehensive system, integrating robust hardware with an intelligent software framework for task-conditioned viewpoint selection, significantly enhances robotic perception capabilities, enabling persistent object-state detection and policy-aligned viewpoints in diverse real-world tasks, thereby pushing the boundaries of robot autonomy and adaptability.
The methodology proposed in SensorPerch is a comprehensive and well-integrated system that addresses a critical limitation in robotic perception. The core idea of decoupling sensing from the robot's embodiment and fixed infrastructure by treating sensors as autonomous, reconfigurable physical entities is genuinely novel. The system comprises both robust hardware and an intelligent software framework. The hardware design is modular, lightweight (201.4g), and self-contained, featuring an onboard Raspberry Pi 4B for computation, a 2-DoF motorized gimbal for viewpoint adjustment, a compact battery for independent operation (2.25 hours), and a vacuum-based attachment mechanism for diverse surfaces. The integration of a charging dock on the robot's mobile base for autonomous recharging is a practical and essential detail for long-term operation. The cost-effectiveness ($94) is also a significant advantage. The software framework, "Where to Perch," is equally sophisticated. It leverages a hybrid scene representation combining semantic SLAM for geometric reasoning and mountable surface identification, and a radiance-field model (NeRF) for novel view synthesis. This allows for efficient evaluation of candidate viewpoints in simulation without physical deployment. Viewpoint sampling is intelligently biased towards task-relevant regions using a VLM. The task-conditioned scoring function is a flexible abstraction, instantiated for two critical use cases: object-coupled perception (using VLM queries for object state detection) and policy-coupled perception (using DINOv2 embeddings to match a policy's training distribution). Finally, the system incorporates model-based pose estimation and grasp-and-place primitives for autonomous detachment and reattachment of the sensor platforms. The overall methodology presents a complete, closed-loop system for active, reconfigurable perception.
The experimental evaluation is thorough and compelling, covering platform capabilities, task performance, and system efficiency. 1. **Platform Evaluation:** Quantitative metrics for attachment accuracy (0.42 cm, <1 degree), battery life (2.25 hours), and thermal performance demonstrate the hardware's robustness and suitability for real-world, long-horizon tasks. 2. **Task Evaluation:** This is the strongest part of the evaluation. * **Object-coupled perception:** Tested across five diverse scenarios (pot boiling, human fall, fire hazard, cup dropped, water leakage), SensorPerch consistently achieves high event detection success rates, closely matching oracle performance and significantly outperforming baselines like "wrist camera only" or "random feasible placement." This validates its ability to provide persistent, task-relevant perception beyond the robot's immediate vicinity. * **Policy-coupled perception:** Evaluated on three manipulation tasks (peeling, cutting, pouring), SensorPerch achieves success rates (80-90%) comparable to oracle viewpoints, demonstrating its capability to construct policy-specific views that align with training data distributions. This is a crucial validation for enabling robust policy execution in varied settings. * **Baselines:** The comparison against a comprehensive set of baselines (w/o 2-DoF, random, oracle, wrist-only, fixed third-person) rigorously isolates the contributions of different components and the overall system. * **Multi-platform demonstration:** The real-world demonstration with three platforms simultaneously monitoring different aspects of a breakfast preparation task (pot, cutting, human's cup) is highly impressive and showcases the system's scalability and practical utility in complex, multi-task scenarios. 3. **System Evaluation:** The latency breakdown provides valuable insights into the computational overhead. While initial scene reconstruction is time-consuming (369s), it's a one-time cost. Viewpoint sampling, synthesis, and scoring are performed in seconds (0.91s to 7.95s), and the full detach-reattach loop takes a reasonable 12.53 seconds. Streaming latency (180ms) is suitable for real-time perception, even with multiple platforms. These metrics confirm the system's practical usability.
The paper provides detailed descriptions of the hardware components, including specific models (Raspberry Pi 4B, MG90S servos, RealSense D435i, 18650 battery) and their integration. The software stack outlines the use of semantic SLAM, radiance fields (NeRF), VLM, DINOv2, and ROS. A lightweight ROS package is mentioned, which is a good step towards reproducibility. However, the absence of a public code repository or specific details on the semantic SLAM and NeRF implementation (e.g., which specific open-source projects were used, training data for NeRF, VLM model used) makes full replication challenging without further information. The experimental setup is well-described, but the exact datasets for policy training and VLM queries would be needed.
The authors acknowledge several pertinent limitations: 1. **Surface Dependence:** The vacuum-based mounting mechanism relies on the availability of flat, non-porous, rigid surfaces. This limits deployment in environments with soft, porous, or highly irregular textures (e.g., carpets, fabric, rough stone). 2. **Vacuum Reliability:** The long-term reliability of vacuum seals under various environmental conditions (dust, temperature changes, vibrations) is a potential concern, although the paper demonstrates robust attachment in its experiments. 3. **Robot Motion and Occlusions:** The current framework does not explicitly model how the robot's own motion might create or resolve occlusions, or how it affects viewpoint quality over time. This could lead to suboptimal placements if the robot itself becomes an occluder. Additional limitations could include: 4. **Scalability to Many Sensors:** While a three-sensor demo is shown, managing and coordinating a much larger fleet of reconfigurable sensors could introduce new challenges in terms of communication, power management, and optimal placement strategies. 5. **Initial Scene Reconstruction Time:** The 369-second scene reconstruction time, while a one-time cost, might be prohibitive for highly dynamic environments or those requiring frequent re-mapping.
SensorPerch represents a significant step towards more versatile and autonomous robotic systems. By decoupling perception from fixed infrastructure and robot embodiment, it enables robots to adapt their sensing capabilities dynamically to task requirements, leading to several broader impacts: 1. **Enhanced Robot Autonomy:** Robots can operate more independently in diverse, unstructured environments without relying on pre-installed cameras or human intervention for viewpoint adjustment. 2. **Improved Task Performance:** The ability to construct task-optimal viewpoints on demand can lead to higher success rates in complex manipulation, monitoring, and interaction tasks, especially in scenarios requiring persistent observation or specific perspectives. 3. **New Active Perception Paradigm:** This work establishes a new paradigm for active perception, shifting from merely adjusting onboard sensors to physically reconfiguring the sensing infrastructure itself. This could inspire future research into modular, reconfigurable robotic components beyond just sensors. 4. **Applications in Diverse Fields:** Beyond household robotics, this technology could be transformative for applications in industrial inspection, disaster response (deploying sensors in hazardous areas), construction, and even scientific exploration, where dynamic, adaptive sensing is crucial. 5. **Reduced Infrastructure Costs:** By allowing a single robot to deploy and manage its own sensing network, it could reduce the need for expensive, permanently installed camera systems or multi-robot coordination for comprehensive coverage. SensorPerch introduces a novel paradigm for active perception by treating sensors as independent, reconfigurable physical entities that robots can autonomously deploy and manage. This comprehensive system, integrating robust hardware with an intelligent software framework for task-conditioned viewpoint selection, significantly enhances robotic perception capabilities, enabling persistent object-state detection and policy-aligned viewpoints in diverse real-world tasks, thereby pushing the boundaries of robot autonomy and adaptability.
Robust motion planning in dense traffic requires autonomous vehicles to interact in rare and safety-critical scenarios that are underrepresented in naturalistic driving data. Although adversarial training offers a feasible solution, existing methods often rely on external scenario generators, heuristic perturbations, or simulator-heavy rollouts, which makes them difficult to integrate with modern autoregressive planners. Here, we cast adversarially robust planner learning as a constrained min-max game and propose Adversarial World Modeling (AWM), a theoretically grounded multi-agent self-play fine-tuning framework. Since solving the exact game is intractable, AWM introduces a principled decoupled solver. In the inner minimization, the planner's predictive world model is converted into a role-conditioned adversary that learns sparse, scene-adaptive attack coalitions via counterfactual credit assignment. In the outer maximization, the ego planner optimizes a regret-aware robust best response against the frozen AWM, utilizing tail-risk weighting and reference-anchored trust regions to improve hard-case recovery while preserving nominal driving behavior. Experiments on the nuPlan and InterPlan benchmarks demonstrate that our method generates transferable adversarial interactions and yields a robust planner that achieves competitive closed-loop performance in both nominal and highly interactive long-tail scenarios. Theoretical analysis justifies the decoupled solver and the main optimization components.
Primary: unknown (information not provided in the paper text)
All Institutions: unknown (information not provided in the paper text)
This work has significant broader impact for autonomous driving and multi-agent reinforcement learning: * **Enhanced AV Safety**: By systematically exposing and mitigating vulnerabilities in rare, safety-critical scenarios, AWM contributes directly to improving the robustness and safety of autonomous vehicles in dense and interactive traffic. * **Robust ML for Robotics**: The framework provides a principled approach to adversarial training for complex, high-dimensional robotic systems, particularly those relying on autoregressive generative models. * **Multi-Agent Self-Play**: The novel counterfactual credit assignment and sparse coalition learning mechanisms advance the state-of-the-art in multi-agent self-play, offering solutions to long-standing credit assignment problems. * **Efficient Adversarial Testing**: AWM's ability to generate transferable adversarial interactions makes it a valuable tool for stress-testing and evaluating the robustness of various existing and future motion planners, potentially accelerating development and validation cycles. * **Integration with Modern Architectures**: By aligning with the autoregressive, tokenized paradigm, AWM offers a practical and scalable solution for robustifying modern generative planners, bridging a gap where traditional adversarial methods struggle. This paper introduces Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework that repurposes a planner's predictive world model as a role-conditioned adversary to robustify autoregressive motion planning. The framework's principled decoupled solver, novel counterfactual credit assignment, and regret-aware constrained optimization provide a theoretically grounded and empirically validated approach to generate transferable adversarial interactions, significantly improving planner robustness in long-tail scenarios while preserving nominal driving performance.
The paper proposes Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework for robust motion planning, framed as a constrained min-max game. The core innovation lies in repurposing the planner's own predictive world model as a role-conditioned adversary, which is a highly elegant and integrated solution compared to external scenario generators or heuristic perturbations. The methodology is theoretically grounded and addresses three key challenges: 1. **Multi-agent credit assignment**: AWM introduces a novel counterfactual credit assignment mechanism via role switching. By comparing factual active coalitions with leave-one-out and full background counterfactuals, it efficiently attributes scene-level utility changes to individual and synergistic adversarial behaviors. This is a significant improvement over naive reward sharing. 2. **Min-max optimization instability**: The paper proposes a principled decoupled solver. Stage A learns a frozen adversarial world model (AWM) that generates sparse, scene-adaptive attack coalitions. Stage B then optimizes the ego planner as a robust best response against this *frozen* AWM. This separation effectively mitigates the instability of simultaneous non-convex min-max updates. 3. **Nominal performance degradation**: The planner optimization in Stage B incorporates a regret-aware objective with tail-risk weighting (CVaR) to prioritize rare but critical failures, and crucially, reference-anchored trust regions with dual variables to preserve nominal driving behavior and enforce safety constraints. This ensures robustness is gained without sacrificing general driving competence. The detailed mechanisms for adversarial world modeling, including role-conditioned generation, sparse coalition learning (via proposal probing, primary selection, conditional pairing, and scene-adaptive calibration), are well-articulated and designed to generate plausible yet challenging multi-agent interactions. The use of a shared autoregressive tokenized rollout structure for both planning and adversarial generation ensures seamless integration. The theoretical analysis, mentioned in the abstract and contributions, provides justification for the decoupled solver and optimization components, adding significant rigor.
The experimental evaluation is comprehensive and rigorous, addressing three key questions: 1. **Closed-loop planning performance**: AWM-Planner is evaluated on the nuPlan and InterPlan benchmarks, including a challenging InterPlan-LongTail subset. It demonstrates competitive performance, significantly improving on interaction-heavy long-tail scenarios (e.g., +5.81 on InterPlan-LongTail) while maintaining or slightly improving nominal driving behavior on nuPlan (e.g., +2.60 on Test14-hard NR, -0.01 on Test14-random NR). This validates the method's ability to improve robustness without sacrificing general performance. 2. **Transferability of AWM as an adversarial simulator**: A crucial experiment deploys the learned AWM as a non-ego sim-agent traffic model for *seven different planners* (including rule-based, optimization-based, diffusion-based, and transformer-based). AWM consistently reduces the closed-loop score for all planners, demonstrating that the learned adversarial behaviors generalize beyond the planner it was trained against. This highlights AWM's potential as a valuable tool for stress-testing and robustifying various planning architectures. 3. **Component analysis (ablations)**: Extensive ablations are performed on both the AWM (Stage A) and planner (Stage B) components. * **AWM ablations**: Show that role-conditioned counterfactual credit assignment is superior to global reward broadcast, yielding stronger lower-tail degradation without sacrificing realism. The adaptive calibrated coalition learning is shown to provide sufficient adversarial stress while maintaining realism, outperforming forced K=2 attacks or simpler single-candidate approaches. * **Planner adaptation ablations**: Demonstrate the superiority of the decoupled self-play pipeline over planner-only refinement, simultaneous alternating, or alternating warmup, in terms of adversarial Tail-CVaR, closed-loop scores, and stability (lower GPU-hours, faster plateau, more stable seeds). Ablations on the Stage B objective confirm the importance of regret-aware CVaR for targeting rare severe risks and normal retention for preserving nominal performance. The use of diverse metrics (R-Score, progress, collision, TTC, drivable-area, speed-limit, driving-direction, Tail-CVaR) provides a holistic view of performance. Case studies visually illustrate the planner's improved recovery in adversarial scenarios and the nature of AWM-generated attacks.
The paper provides a high level of detail in its methodology section, including mathematical formulations for the game, decoupled solver, credit assignment, and planner objectives. It explicitly mentions appendices for "implementation details," "theoretical analysis," "architectural details," "probe signals and coverage metrics," "details for $f_pri$ and $f_pair$," "full AWM training objective," "analytical derivations and loss formulations," and "how the learned AWM is instantiated as a closed-loop sim-agent evaluator." The algorithm is summarized in Algorithm 1 in the appendix. This comprehensive documentation suggests a strong commitment to reproducibility.
The authors acknowledge several limitations: 1. **Fidelity of the tokenized world model**: AWM's effectiveness is inherently tied to the fidelity of the underlying tokenized world model. If the world model cannot accurately represent complex multi-agent dynamics, the generated adversarial scenarios may be unrealistic or ineffective. 2. **Exhaustive scenario coverage**: While AWM generates scene-adaptive attacks, it does not guarantee exhaustive coverage of all possible safety-critical scenarios. This is a common challenge in adversarial training for complex environments. 3. **Formal closed-loop safety guarantees**: The framework does not provide formal, provable safety guarantees, which remains a grand challenge for learning-based autonomous driving systems. 4. **Computational cost**: Although the decoupled solver improves stability and efficiency compared to simultaneous training, training large autoregressive models for both the world model and planner can still be computationally intensive.
This work has significant broader impact for autonomous driving and multi-agent reinforcement learning: * **Enhanced AV Safety**: By systematically exposing and mitigating vulnerabilities in rare, safety-critical scenarios, AWM contributes directly to improving the robustness and safety of autonomous vehicles in dense and interactive traffic. * **Robust ML for Robotics**: The framework provides a principled approach to adversarial training for complex, high-dimensional robotic systems, particularly those relying on autoregressive generative models. * **Multi-Agent Self-Play**: The novel counterfactual credit assignment and sparse coalition learning mechanisms advance the state-of-the-art in multi-agent self-play, offering solutions to long-standing credit assignment problems. * **Efficient Adversarial Testing**: AWM's ability to generate transferable adversarial interactions makes it a valuable tool for stress-testing and evaluating the robustness of various existing and future motion planners, potentially accelerating development and validation cycles. * **Integration with Modern Architectures**: By aligning with the autoregressive, tokenized paradigm, AWM offers a practical and scalable solution for robustifying modern generative planners, bridging a gap where traditional adversarial methods struggle. This paper introduces Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework that repurposes a planner's predictive world model as a role-conditioned adversary to robustify autoregressive motion planning. The framework's principled decoupled solver, novel counterfactual credit assignment, and regret-aware constrained optimization provide a theoretically grounded and empirically validated approach to generate transferable adversarial interactions, significantly improving planner robustness in long-tail scenarios while preserving nominal driving performance.
The action space poses a major challenge in robot learning, since it is often high-dimensional, can span long time horizons, and frequently admits multi-modal optimal solutions. A good choice of action representation and loss function can help to address these concerns, but there are often trade offs. We propose Action Map Policy (AMP), which casts 3D closed-loop manipulation policy learning as a classification problem in image space. While classification has been an effective formulation in generative language models, applying it to robot action learning is difficult because naively discretizing high-dimensional continuous actions explodes the token vocabulary. Our key idea is to project 3D actions onto the camera image planes and treat each pixel location as a discrete class, thus controlling dimensionality while retaining multi-modality. This method supports millimeter-level precision for high-dimensional actions without requiring a prohibitively large vocabulary, while preserving fine-grained pixel-wise visual signals. Furthermore, it can predict the entire action chunk in a single forward pass, avoiding complex noise scheduling and iterative denoising while achieving substantially faster inference than diffusion policies. Experiments on various manipulation tasks show that AMP outperforms strong baselines, achieving higher success rates, faster inference, and enhanced spatial reasoning.
Primary: Brown University
All Institutions: Brown University
The authors acknowledge several limitations: 1. **Jittering**: Minor jittering in generated actions, which could be improved with geometric consistency losses. 2. **Workspace Limitation**: The workspace is limited by calibrated camera coverage, preventing actions outside the observed scope. This could be alleviated with larger-FOV or dual in-hand cameras. 3. **Gripper Specificity**: While the method is flexible, the current keypoint design focuses on a parallel-jaw gripper. Adaptation would be needed for dexterous or bimanual setups. 4. **Future Directions**: The paper also points to future work, such as integrating Q-functions for reinforcement learning and exploring pre-trained vision-language models, indicating that the current work is a foundational step. BROADER IMPACT: AMP offers a significant contribution to robot learning by providing an effective and efficient method for closed-loop 3D manipulation. Its ability to handle high-dimensional continuous actions, multi-modality, and achieve high precision with fast inference has broad implications for deploying robots in complex real-world tasks. The classification-based approach, which aligns action representation with observation space, could inspire new architectures and training paradigms in visuomotor control. The enhanced spatial reasoning demonstrated, particularly the responsiveness to small visual cues, is critical for robust and precise robot operation. This method could accelerate the development of more capable and adaptable robotic systems, potentially impacting fields like manufacturing, service robotics, and assistive technologies. The explicit action distribution output also opens avenues for safer and more interpretable robot behavior, as well as easier integration with reinforcement learning and vision-language models. Action Map Policy (AMP) introduces a highly novel and effective framework for 3D closed-loop manipulation by casting action learning as a pixel classification problem using projected keypoint heatmaps. This approach elegantly resolves the challenges of high-dimensional continuous actions, multi-modality, and precision, outperforming strong baselines in both simulated and real-world tasks with significantly faster inference and superior spatial reasoning. The paper's robust methodology, comprehensive experimental validation, and clear articulation of its advantages make it a substantial contribution to the field of robotics and machine learning.
The paper proposes Action Map Policy (AMP), a novel framework that reformulates 3D closed-loop manipulation policy learning as a pixel classification problem. The core idea is to represent 3D end-effector poses as a set of 3D keypoints, project these keypoints onto multiple camera image planes, and then predict their 2D pixel locations as discrete classes (heatmaps). This approach elegantly addresses the challenges of high-dimensional continuous action spaces, combinatorial explosion in naive discretization, and multi-modality. The use of a cross-entropy loss on soft Gaussian heatmaps for supervision is well-justified, providing stable training and naturally modeling multi-modal distributions. The X-Net architecture, a multi-view U-Net encoder combined with a Multi-View Transformer and U-Net decoder, is specifically designed to process multi-view images and output dense action feature maps, preserving spatial structure. The heatmap-to-action conversion process, involving argmax pixel selection, triangulation, and geometric pose recovery, is clearly defined and robust. A significant advantage is the single-pass inference, which is substantially faster than iterative denoising methods like diffusion policies. The method also leverages equivariant data augmentation, which is a natural fit for its spatially aligned action representation. The theoretical precision analysis demonstrating millimeter-level accuracy at reasonable image resolutions is a strong methodological justification.
The experimental evaluation is comprehensive and rigorous, spanning both simulated and real-world settings. 1. **Precision Analysis**: A theoretical analysis quantifies the achievable precision of the heatmap representation at different image resolutions, showing that $224 \times 224$ images yield approximately 1mm translation and 1.3-degree rotation precision, which is crucial for manipulation. This is a strong empirical justification for the chosen representation. 2. **Simulated Experiments**: AMP is evaluated on six challenging MimicGen tasks, covering rigid and articulated objects with varying precision requirements. It is compared against four strong baselines: Diffusion Policy (DiffPo), ACT, OAT, and Motion Track. AMP significantly outperforms these baselines on five out of six tasks, with an average gain of 20.7% in success rate. 3. **Ablation Studies**: Ablations demonstrate the importance of the in-hand camera view, soft labels (Gaussian width $\sigma=2$), and equivariant data augmentation, all contributing positively to performance. 4. **Real-World Experiments**: Three long-horizon, complex real-world tasks (make-coffee, toast-bread, steam-egg) are used to evaluate closed-loop visuomotor policy learning. AMP shows a substantial performance margin (e.g., 50% higher success rate than DiffPo on 'make-coffee') and faster inference speed (13.80 ms vs. 93.53 ms for DiffPo). 5. **Spatial Reasoning**: Two tasks (grasp-target-cup, stack-target-block) are designed to test spatial reasoning, where a laser pointer indicates the target among identical objects. AMP achieves 100% success, demonstrating its ability to respond to fine-grained visual cues, unlike baselines that struggle with mode collapse. The experiments are well-designed, the baselines are strong, and the results consistently highlight AMP's superior performance, efficiency, and robustness.
The paper provides good details for reproducibility. Key implementation choices are described, including image resolution, network architecture (U-Net encoder, MVT, U-Net decoder, skip connections), number of channels, and the use of the first 8 out of 12 predicted timesteps. Hyperparameters like learning rate, training epochs, and GPU used are specified. Crucially, the paper includes pseudocode for both training and inference, which is highly valuable. Details on data preprocessing, augmentation (including equivariant augmentation), and the soft-label width ($\sigma=2$) are provided. The description of the keypoint definition and pose recovery is also clear. The authors state that they focus on policy performance without using pretrained vision encoders for any method, ensuring a fair comparison.
The authors acknowledge several limitations: 1. **Jittering**: Minor jittering in generated actions, which could be improved with geometric consistency losses. 2. **Workspace Limitation**: The workspace is limited by calibrated camera coverage, preventing actions outside the observed scope. This could be alleviated with larger-FOV or dual in-hand cameras. 3. **Gripper Specificity**: While the method is flexible, the current keypoint design focuses on a parallel-jaw gripper. Adaptation would be needed for dexterous or bimanual setups. 4. **Future Directions**: The paper also points to future work, such as integrating Q-functions for reinforcement learning and exploring pre-trained vision-language models, indicating that the current work is a foundational step. BROADER IMPACT: AMP offers a significant contribution to robot learning by providing an effective and efficient method for closed-loop 3D manipulation. Its ability to handle high-dimensional continuous actions, multi-modality, and achieve high precision with fast inference has broad implications for deploying robots in complex real-world tasks. The classification-based approach, which aligns action representation with observation space, could inspire new architectures and training paradigms in visuomotor control. The enhanced spatial reasoning demonstrated, particularly the responsiveness to small visual cues, is critical for robust and precise robot operation. This method could accelerate the development of more capable and adaptable robotic systems, potentially impacting fields like manufacturing, service robotics, and assistive technologies. The explicit action distribution output also opens avenues for safer and more interpretable robot behavior, as well as easier integration with reinforcement learning and vision-language models. Action Map Policy (AMP) introduces a highly novel and effective framework for 3D closed-loop manipulation by casting action learning as a pixel classification problem using projected keypoint heatmaps. This approach elegantly resolves the challenges of high-dimensional continuous actions, multi-modality, and precision, outperforming strong baselines in both simulated and real-world tasks with significantly faster inference and superior spatial reasoning. The paper's robust methodology, comprehensive experimental validation, and clear articulation of its advantages make it a substantial contribution to the field of robotics and machine learning.
As demand for DNN inference grows, GPU capacity is increasingly oversubscribed, forcing operators to colocate multiple models on the same device in both cloud and edge deployments. Whether colocation succeeds or violates SLOs depends on the temporal overlap of kernels from concurrently executing models -- an effect that existing serving systems either ignore or approximate using aggregate resource profiles that fail to capture temporal dynamics. This paper presents Roomie, a model serving orchestration architecture that predicts and avoids kernel-level interference between colocated DNNs. Roomie decouples offline kernel profiling from online interference prediction. It uses profiling only to extract per-kernel resource configurations, and predicts interference with an occupancy-based analytical model immune to profiler-induced timing distortion. A pairwise greedy heuristic then approximates multi-model interference in polynomial rather than exponential time, and an online placement algorithm then uses these estimates to assign each incoming model to the GPU that minimizes predicted slowdown. Our experimental evaluation compares Roomie against state-of-the-art solutions across both cloud-grade server clusters and embedded edge devices, demonstrating that Roomie reduces SLO violations (i.e., inference latency) by up to 3x, while maintaining comparable, and in many cases superior, goodput relative to existing approaches.
Primary: Stony Brook University
All Institutions: Stony Brook University, ENS Lyon, Institut Universitaire de France, Université Savoie Mont Blanc
Roomie presents a robust and practically significant contribution to the field of ML systems by introducing an interference-aware colocation strategy that significantly reduces latency violations through accurate, lightweight analytical modeling. The decoupling of profiling from online prediction and the use of a pairwise greedy heuristic offer a scalable solution to a pervasive problem in multi-tenant GPU environments, demonstrating clear empirical benefits in both cloud and edge settings.
The paper proposes "Roomie," a model serving orchestration architecture designed to mitigate kernel-level interference when colocation of multiple Deep Neural Network (DNN) inference models on a single GPU. The core methodological innovation lies in decoupling offline kernel profiling from online interference prediction. Instead of relying on aggregate resource profiles which fail to capture temporal dynamics, Roomie uses an occupancy-based analytical model to predict interference. This model is described as "immune to profiler-induced timing distortion," a significant practical advantage. The system employs a pairwise greedy heuristic to approximate multi-model interference in polynomial time, followed by an online placement algorithm that assigns incoming models to GPUs to minimize predicted slowdown. This approach addresses a critical gap in existing serving systems that either ignore interference or use coarse approximations.
The evaluation compares Roomie against state-of-the-art solutions across two distinct environments: cloud-grade server clusters and embedded edge devices. The metrics focus on SLO violations (inference latency) and goodput. The results indicate that Roomie reduces SLO violations by up to 3x compared to existing approaches. Furthermore, it maintains comparable or superior goodput, suggesting that the interference-aware scheduling does not come at the cost of overall throughput efficiency. The dual-environment evaluation strengthens the claim of generalizability, though the specific baselines and workload characteristics (e.g., mix of model sizes, batch sizes, and traffic patterns) are crucial for interpreting the magnitude of the improvement.
The paper provides a clear description of the architecture, including the decoupling of profiling and prediction, the analytical model, and the heuristic algorithms. The implementation details appear sufficient for replication by systems researchers. However, the specific parameters of the "occupancy-based analytical model" and the exact nature of the "pairwise greedy heuristic" would need to be scrutinized in the full text to ensure complete reproducibility. The use of standard benchmarks for DNN inference is implied, which aids reproducibility.
The paper likely faces limitations inherent to analytical models, such as the accuracy of the interference predictions under highly dynamic or non-stationary workloads. The "pairwise" heuristic, while efficient, may not capture complex higher-order interference effects among three or more models simultaneously, potentially leading to suboptimal placements in dense colocation scenarios. Additionally, the overhead of the online placement algorithm itself must be negligible to justify its use; the paper claims this but the computational cost of the prediction step is a potential bottleneck if not optimized. The evaluation on embedded devices might be limited by the specific hardware constraints and the availability of diverse DNN models for edge deployment.
This work has significant implications for the efficiency and cost-effectiveness of DNN inference services in both cloud and edge computing. By enabling more efficient colocation, operators can reduce hardware costs and energy consumption while maintaining service level objectives. This contributes to the broader goal of sustainable AI and democratizing access to high-performance inference resources. The techniques developed could be integrated into major serving frameworks (e.g., Triton, vLLM) to improve their default scheduling strategies. Roomie presents a robust and practically significant contribution to the field of ML systems by introducing an interference-aware colocation strategy that significantly reduces latency violations through accurate, lightweight analytical modeling. The decoupling of profiling from online prediction and the use of a pairwise greedy heuristic offer a scalable solution to a pervasive problem in multi-tenant GPU environments, demonstrating clear empirical benefits in both cloud and edge settings.
GPU collective communication is typically optimized for bandwidth, yet many emerging workloads are increasingly limited by latency. Long-context decode-heavy large language model (LLM) inference is a prime example, where serving large models requires multiple GPUs, and many small collectives lie directly on the critical path of token generation. Therefore, even microsecond of overhead can impact performance and cost. In this work, we study how to approach the hardware Speed-of-Light (SoL) lower bound for GPU collectives within a scale-up network. We identify key principles for near-optimal designs, including barrier-free synchronization and efficient use of symmetric memory and multicast. Building on NCCL's device-side API, we develop low-latency interfaces for constructing custom collective kernels and use them to implement new symmetric collectives in NCCL. Microbenchmarks show substantial latency reductions for small and medium messages, reducing overhead to within 7% of the absolute SoL lower bound. When integrated into real applications, these kernels improve inter-token latency and throughput in LLM inference and accelerate cuSOLVERMp, demonstrating benefits for both AI inference and traditional HPC workloads.
Primary: ETH Zürich
All Institutions: ETH Zürich, NVIDIA Corporation, NVIDIA Helsinki Oy
This paper delivers a critical systems optimization that bridges the gap between theoretical hardware limits and practical collective communication performance, establishing a new standard for low-latency GPU collectives essential for modern AI inference.
The paper presents a rigorous systems-level optimization of GPU collective communication, specifically targeting the latency bottleneck in small-message AllReduce operations critical for LLM inference. The authors correctly identify that existing "low-latency" kernels still suffer from significant overhead due to explicit memory barriers. Their methodology involves three key innovations: (1) eliminating global memory barriers using LL (Low Latency) packing and sentinel-based synchronization, (2) implementing barrier-free bidirectional communication with double buffering to handle chunked transfers, and (3) designing a novel "two-shot LL128 atomic" algorithm that leverages NVLink's cache-line atomicity for scalable, barrier-free reduction. The API design (`ncclLLBuffer`) is particularly strong, providing a clean, device-side abstraction that allows developers to compose custom low-latency kernels without host intervention. This approach is technically sound and represents a sophisticated understanding of GPU memory hierarchies and interconnect semantics.
The evaluation is comprehensive and convincing. The authors benchmark their implementation on GB200 NVL72 systems, comparing against state-of-the-art libraries including NCCL (v2.29.1), NVSHMEM, MSCCL++, and vLLM. They demonstrate that their kernels achieve latencies within 7% of the theoretical Speed-of-Light lower bound, a significant improvement over existing solutions which often lag by factors of 2-4x for small messages. The microbenchmarks cover a wide range of message sizes and GPU counts (2-64 GPUs). Crucially, the paper moves beyond microbenchmarks to show real-world impact: a 7-13% reduction in inter-token latency (ITL) and corresponding cost savings in vLLM inference, and measurable speedups in the cuSOLVERMp HPC library. The inclusion of cost analysis adds practical relevance for cloud providers.
The paper provides a GitHub repository link for the source code. The experimental setup is well-documented, specifying the hardware (GB200, GH200), software versions (CUDA 13.1, NCCL 2.29.1, vLLM 0.15.1), and benchmark parameters. The definition of the SoL lower bound is clearly derived and reproducible. The reliance on specific NVIDIA hardware (NVLink, NVLS) limits generalizability to other vendors, but the methodology is transparent and the code artifact facilitates verification of the claims on compatible hardware.
The primary limitation is hardware specificity. The optimizations rely heavily on NVIDIA-specific features such as NVLink cache-line atomics, NVLS multicast, and symmetric memory (LSA). While the authors note that LL and sentinel synchronization are more general, the high-performance LL128 atomic kernel is tied to CUDA's vectorized atomic operations and NVLink's behavior. Additionally, the LL128 algorithm is non-deterministic due to floating-point atomic ordering, which may be a concern for some scientific applications, although the authors argue it is acceptable for LLM inference. The paper also notes that multicast variants of MSCCL++ hung on the test hardware, suggesting potential stability issues in the broader ecosystem that their work complements but does not fully resolve for all libraries.
This work has significant implications for the scalability and cost-efficiency of large-scale AI inference. As LLMs grow and context windows expand, the decode-phase latency becomes the primary bottleneck for serving throughput. By reducing collective communication overhead to near-physical limits, this work enables higher token throughput and lower latency, directly translating to reduced operational costs for inference providers. Furthermore, the low-latency API and principles can accelerate traditional HPC workloads that are similarly constrained by collective communication latency, promoting broader adoption of GPU-native communication libraries in scientific computing. This paper delivers a critical systems optimization that bridges the gap between theoretical hardware limits and practical collective communication performance, establishing a new standard for low-latency GPU collectives essential for modern AI inference.
Existing fast GPU error-bounded lossy compressors have achieved high throughput through pure-GPU single-kernel designs, but their compression ratios remain limited because they typically apply a fixed first-order predictor on independent blocks. We propose FSZ, a GPU error-bounded lossy compressor that redesigns the prediction stage with three mutually reinforcing algorithmic innovations to achieve both higher compression ratios and higher throughput within a single CUDA kernel: (1) cross-block prediction state carries Lorenzo prediction state across block boundaries within 256-element tiles, eliminating 7 out of 8 boundary residuals that inflate encoding rates; (2) per-tile adaptive multi-order prediction and centering adaptively selects the best compression strategy per tile from first-order, second-order, and centering variants; and (3) a single-pass four-way evaluation exploits a mathematical property of finite differences to evaluate all variants from a single data read, enabling richer prediction within the same bandwidth budget as a fixed predictor. Experiments on NVIDIA GH200 GPU with 8 real-world application datasets show that FSZ outperforms cuSZp-P by up to 10.95x and the state-of-the-art cuSZp-O by up to 2.92x in compression ratio. Notably, these gains come with no throughput penalty: FSZ simultaneously achieves the highest average throughput (676 GB/s compression, 785 GB/s decompression) among all evaluated compressors.
Primary: University of South Florida
All Institutions: University of South Florida
The paper presents a highly effective GPU-based lossy compression algorithm that simultaneously achieves superior compression ratios and throughput through innovative prediction state management and adaptive multi-order evaluation.
The paper proposes FSZ, a GPU-based lossy compressor that addresses the trade-off between compression ratio and throughput. The core methodological innovation lies in redesigning the prediction stage within a single CUDA kernel. Specifically, it introduces cross-block prediction state management to carry Lorenzo prediction states across block boundaries within 256-element tiles, which significantly reduces boundary residuals. It also employs per-tile adaptive multi-order prediction (selecting between first-order, second-order, and centering variants) and a single-pass four-way evaluation technique that leverages mathematical properties of finite differences to evaluate multiple prediction variants from a single data read. This approach aims to maximize information density per memory access, thereby improving compression ratios without sacrificing the high throughput characteristic of single-kernel GPU designs.
The evaluation is conducted on NVIDIA GH200 GPUs using 8 real-world application datasets. The results demonstrate significant improvements in compression ratio compared to state-of-the-art compressors cuSZp-P (up to 10.95x improvement) and cuSZp-O (up to 2.92x improvement). Crucially, these gains are achieved without any throughput penalty, with FSZ achieving the highest average throughput among evaluated compressors (676 GB/s compression, 785 GB/s decompression). The use of real-world scientific datasets adds substantial credibility to the practical relevance of the findings.
The paper provides detailed algorithmic descriptions, including the specific tile size (256 elements) and the nature of the four-way evaluation. The use of standard CUDA kernels suggests that the implementation is reproducible, provided the specific hardware (GH200) and software environments are available. The mention of "real-world application datasets" implies that the data sources should be cited, which is standard for reproducibility in this domain.
The primary limitation is the hardware specificity; the optimizations are tailored for NVIDIA GPUs (specifically tested on GH200), which may limit immediate applicability to other architectures like AMD or Intel GPUs without significant refactoring. Additionally, the "single-pass four-way evaluation" might introduce computational overhead or complexity in kernel scheduling that could vary across different GPU generations or workloads, although the results suggest this is mitigated. The focus on error-bounded lossy compression means it is not suitable for lossless requirements or scenarios where strict error bounds cannot be guaranteed.
This work has significant implications for large-scale scientific computing, particularly in fields like climate modeling, astrophysics, and computational fluid dynamics, where massive datasets are routinely generated and stored. By breaking the prediction-throughput trade-off, FSZ enables more efficient storage and transfer of scientific data, potentially reducing infrastructure costs and energy consumption. It sets a new benchmark for GPU-accelerated data compression, likely influencing future designs of I/O pipelines in HPC systems. The paper presents a highly effective GPU-based lossy compression algorithm that simultaneously achieves superior compression ratios and throughput through innovative prediction state management and adaptive multi-order evaluation.
Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.
Primary: Unknown
All Institutions: Unknown
Kaleido presents a compelling algorithm-hardware co-design that significantly accelerates Video Diffusion Transformers by exploiting latent space correlations, offering substantial speedups and energy savings while maintaining high generative quality, thereby addressing a critical bottleneck in the deployment of advanced video generation models.
The paper proposes "Kaleido," an algorithm-hardware co-design specifically targeting Video Diffusion Transformers (vDiTs). The core algorithmic insight is the exploitation of channel-wise spatiotemporal correlations in the latent space to enable computation skipping (reusing partial results). This is a significant methodological shift from standard sparse attention techniques used in LLMs, which do not account for the unique redundancy in video generation tasks. The hardware design features a systolic array-like accelerator with reconfigurable processing elements and a lightweight data dispatcher to handle the irregular sparsity introduced by the algorithm. The approach is technically sound, addressing the dominant bottleneck of self-attention in long-timestep diffusion processes. However, the novelty is somewhat incremental in the broader context of hardware acceleration, as algorithm-hardware co-design for transformers is a well-trodden path, though the specific application to vDiTs with this correlation-based reuse is a strong contribution.
The evaluation demonstrates up to 5.9x speedup and 16.0x energy savings compared to state-of-the-art accelerators across three mainstream vDiT models. The claim of preserving higher generative quality (>17 dB improvement in PSNR/SSIM relative to prior skipping methods) is a strong empirical result. The use of standard metrics and comparison against SOTA baselines provides credibility. However, the lack of specific details on the baseline models and the exact nature of the "state of the art" accelerators in the abstract limits the immediate assessability of the magnitude of the improvement. The results are promising and suggest significant practical utility for deploying vDiTs.
The paper is published on arXiv, and while the abstract provides a high-level overview, the full text would need to be scrutinized for hyperparameters, model configurations, and hardware specifications to ensure reproducibility. The mention of "lightweight" components suggests a focus on practical implementation, which often aids reproducibility if code is released. Without code availability explicitly stated in the abstract, reproducibility is assumed to be moderate to high based on the detailed algorithmic description implied.
The primary limitation is the reliance on the assumption that channel-wise spatiotemporal correlations are consistent across different vDiT architectures and video domains. The performance gain might vary depending on the specific video content and the complexity of the diffusion process. Additionally, the hardware design's flexibility (reconfigurable PEs) might come with overheads that are not fully detailed in the abstract. The comparison is against "state of the art accelerators," which may include specialized custom hardware that is not widely available, potentially skewing the perceived advantage.
This work has significant implications for the democratization of high-quality video generation by reducing the computational and energy barriers. It enables faster iteration for researchers and more accessible deployment for developers. The focus on energy efficiency also aligns with the growing need for sustainable AI. By addressing the specific bottlenecks of vDiTs, it paves the way for more complex and longer video generation tasks. Kaleido presents a compelling algorithm-hardware co-design that significantly accelerates Video Diffusion Transformers by exploiting latent space correlations, offering substantial speedups and energy savings while maintaining high generative quality, thereby addressing a critical bottleneck in the deployment of advanced video generation models.
Modern edge system-on-chips (SoCs) combine CPUs, integrated GPUs (iGPUs), and neural processing units (NPUs), yet existing LLM runtimes typically make coarse device-level decisions or optimize operators in isolation. As a result, they underutilize heterogeneous resources, particularly on unified-memory platforms where performance depends on both device placement and task-graph coordination. We present HeteroMosaic, a heterogeneity-first scheduling framework for edge LLM inference. HeteroMosaic first uses a heterogeneous roofline model to identify when combining iGPU and NPU execution is beneficial. It then decomposes inference into dependency-preserving micro-batches that expose cross-accelerator overlap and applies trace-guided co-optimization of scheduling and device allocation under practical effects such as memory contention, DVFS, device variation, and NPU runtime overheads. We implement HeteroMosaic in PyTorch C++ and evaluate it on three AMD Ryzen AI platforms spanning NPU-heavy, balanced, and iGPU-heavy designs. On the balanced platform, HeteroMosaic achieves up to 1.73X speedup over an iGPU baseline, 1.78X over an NPU baseline, and 2.05X over frameworks such as llama.cpp, while reducing energy by up to 45.3%. It also improves performance over prior heterogeneous edge AI solutions by up to 2.35X.
Primary: University of Illinois Urbana-Champaign (UIUC)
All Institutions: University of Illinois Urbana-Champaign (UIUC), Advanced Micro Devices, Inc. (AMD)
HeteroMosaic introduces a novel scheduling framework for edge LLM inference that significantly outperforms existing baselines by exploiting fine-grained heterogeneous execution opportunities on unified-memory SoCs, offering a practical path toward energy-efficient, high-performance edge AI deployment.
The paper proposes HeteroMosaic, a scheduling framework that treats edge LLM inference as a heterogeneity-first problem rather than a simple placement problem. The core methodological contributions include: 1) A heterogeneous roofline model to theoretically bound the benefits of combining iGPU and NPU execution; 2) Dependency-preserving micro-batching to expose cross-accelerator overlap opportunities that coarse-grained batching misses; 3) Trace-guided co-optimization of scheduling and device allocation that accounts for real-world system effects like memory contention, DVFS, and NPU runtime overheads. This approach is technically sophisticated, moving beyond static operator mapping to dynamic, graph-level coordination. The insight that fine-grained heterogeneity can be exploited via scheduling on unified-memory platforms is a significant shift in how edge AI systems are designed.
The evaluation is conducted on three distinct AMD Ryzen AI platforms (NPU-heavy, balanced, iGPU-heavy), providing strong empirical evidence for the framework's adaptability. The results show substantial improvements: up to 1.73x speedup over iGPU-only, 1.78x over NPU-only, and 2.05x over llama.cpp on a balanced platform, with up to 45.3% energy reduction. The comparison against strong baselines (including existing heterogeneous solutions) demonstrates clear superiority. The use of real hardware rather than simulation adds significant credibility. The evaluation covers multiple models, ensuring the findings are not model-specific artifacts.
The authors implement HeteroMosaic in PyTorch C++, which is a standard and accessible framework for such systems research. The paper details the architectural properties required (programmable accelerators, shared memory, low-overhead sync) and the specific optimizations applied. While the exact proprietary NPU driver details might be AMD-specific, the scheduling abstraction and the trace-guided optimization methodology are described with sufficient detail for replication on similar unified-memory SoCs. The codebase structure suggests it could be open-sourced or made available upon request, which is common for top-tier systems venues.
The current implementation is tightly coupled with AMD Ryzen AI platforms. The "heterogeneous roofline model" and scheduling heuristics may need significant re-tuning for other architectures (e.g., Qualcomm Hexagon, Apple Neural Engine) due to differences in NPU capabilities, memory bandwidth, and runtime overheads. The paper acknowledges that extending this to datacenter systems would require explicit communication modeling for PCIe/CXL/NVLink, which is a non-trivial extension not covered here. Additionally, the focus is on inference; training on such heterogeneous edge devices remains an open challenge.
This work has profound implications for the deployment of Large Language Models on edge devices, enabling faster and more energy-efficient AI on smartphones, laptops, and IoT devices. By demonstrating that fine-grained heterogeneity can be practically exploited, it encourages hardware designers to prioritize balanced accelerator designs and efficient interconnects. It also provides a blueprint for runtime systems to better utilize the increasingly heterogeneous compute resources available in modern SoCs, potentially reducing the carbon footprint of AI inference. HeteroMosaic introduces a novel scheduling framework for edge LLM inference that significantly outperforms existing baselines by exploiting fine-grained heterogeneous execution opportunities on unified-memory SoCs, offering a practical path toward energy-efficient, high-performance edge AI deployment.