Last 7 Days (July 15 – July 21, 2026)
Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of California, Santa Cruz, Shanghai Jiao Tong University
RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
The paper introduces RESOURCE2SKILL, a framework for distilling multimodal human resources (videos, code repos, articles, artifacts) into a structured, hierarchical "Skill Wiki" for software agents. The core innovation lies in the schema: skills are tuples of text, visual, code, and metadata, organized by a domain-specific taxonomy. The method employs a "construction operator" that uses a vision-capable LLM to extract and normalize these components, followed by deterministic acceptance predicates (completeness, provenance, deduplication, modality consistency, structural executability). At inference, a two-stage retrieval process (lexical BM25 over taxonomy paths, then LLM selection) allows agents to compose skills. A key feature is the ability to trigger online acquisition when offline skills are insufficient, using the same construction operator. The approach is well-motivated, addressing the gap between implicit agent knowledge and explicit, reusable procedural knowledge derived from rich human resources.
The evaluation is extensive, covering seven diverse authoring domains (PPT, CAD, Web, Excel, Blender, UE5, Reaper) and four agent backends (GPT-5.5, GPT-5.4, Mini, Nano). The primary metric is an automated judge score (vision/audio models) on rendered artifacts, validated by human A/B studies. RESOURCE2SKILL shows significant improvements (+11.9 pp) over no-skill baselines and outperforms strong harness baselines (ClaudeCode-H, Codex-H) in 26/28 cells. Ablations confirm the value of multimodal content, hierarchical organization, and online acquisition. The scaling study shows performance saturates around 200 skills. The results are robust and statistically significant.
The paper provides a code link (https://aka.ms/Resource2Skill) and detailed appendices describing the benchmark construction, judge reliability, and experimental setup. The acceptance predicates are deterministic, and the library construction is frozen before benchmarking, ensuring no data leakage. The matched-brief design and fixed seeds enhance reproducibility. The detailed description of the MCP interface and domain backends allows for replication.
The framework relies on the availability of high-quality multimodal resources (videos, repos) for each domain, which may not exist for all software domains. The automated judge, while correlated with humans, may have biases (e.g., rewarding visual polish over functional correctness in some cases). The online acquisition adds latency. The "structural executability" check ensures code runs but does not guarantee it solves the specific task correctly, leading to potential partial grounding failures as noted in case studies.
This work significantly advances the field of software agents by providing a scalable method for building and maintaining skill libraries from diverse human-generated content. It enables agents to leverage existing human knowledge (tutorials, code) more effectively, potentially lowering the barrier to entry for complex software tasks. The hierarchical wiki structure offers a promising paradigm for organizing agent memory. RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
Video models are evolving into vision foundation models, yet they still lack human-like multi-step reasoning. Streaming autoregressive diffusion models are efficient but limited in reasoning, while bidirectional diffusion enables global revision with high inference costs due to dense frame-level denoising. Both paradigms struggle to achieve logical consistency and low-latency streaming for complex reasoning tasks. We propose HDR (Hierarchical Denoising for Visual Reasoning), a unified framework that integrates hierarchical latents into causal video generation for multi-step reasoning. HDR organizes video latents into a tree-structured hierarchy, enabling coarse-to-fine reasoning before streaming output. Coarse denoising layers preserve uncertain hypotheses for global planning, while finer layers progressively refine them into concrete visual states. A sparse hierarchical attention pattern (SHAP) further reduces temporal attention costs. We introduce a level-stratified multi-step video reasoning benchmark with out-of-distribution cases, covering six tasks: maze navigation, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, and water pouring. Compared with streaming autoregressive diffusion baselines, HDR improves success from 34.22 to 60.29 (76.2% relative gain) and increases average progress from 76.00 to 89.56, demonstrating more consistent reasoning trajectories. HDR maintains low-latency streaming at 0.70 seconds per latent, achieving 54.2 times faster inference than bidirectional diffusion. It also retains 82.9% of full-data performance with only 2% training data, compared with 52.0% for bidirectional diffusion. Real-world robot experiments further demonstrate HDR's potential for physical interaction and world modeling. Project demo: https://hierarchical-diffusion-reasoning.github.io/.
Primary: The Hong Kong University of Science and Technology
All Institutions: State Key Laboratory of Multimedia Information Processing, The Hong Kong University of Science and Technology, Beihang University, Fuzhou University, Peking University, School of Computer Science
HDR makes a significant contribution to the field of generative AI, particularly in video modeling and embodied AI. 1. **Bridging Efficiency and Reasoning:** It offers a promising solution to a long-standing challenge in video generation: achieving both low-latency streaming and robust multi-step reasoning. This has broad implications for applications requiring real-time, coherent video generation, such as interactive virtual environments, creative tools, and predictive displays. 2. **Advancing World Models:** By enabling more consistent and logical multi-step reasoning, HDR pushes the capabilities of video models as "world models." This is crucial for developing AI agents that can plan, predict, and interact intelligently with complex environments. 3. **Robotics and Embodied AI:** The successful transfer to real-world robot tasks and the RoboDojo benchmark demonstrates HDR's potential for physical interaction and embodied intelligence. Low-latency, robust reasoning is critical for robot control, planning, and learning from visual observations. 4. **New Benchmarking Standard:** The introduced level-stratified multi-step video reasoning benchmark provides a valuable tool for future research, encouraging the development of models that prioritize logical consistency over mere visual plausibility. 5. **Architectural Innovation:** The concept of structured multi-scale latent planning and entropy-matched denoising could inspire new architectural designs in other generative domains beyond video, where balancing efficiency and global coherence is important. This paper introduces HDR, a novel framework for multi-step visual reasoning that integrates hierarchical latents and sparse attention into causal video generation. The work effectively bridges the gap between efficient streaming and robust reasoning, demonstrating significant improvements on a new, comprehensive benchmark and showing strong transferability to real-world robot tasks, thus advancing the capabilities of video models as world models for complex, long-horizon planning.
The paper proposes Hierarchical Denoising for Visual Reasoning (HDR), a novel framework that addresses the fundamental tension between efficient streaming generation and robust multi-step reasoning in video models. The core idea is to integrate hierarchical latents into a causal video generation process, enabling coarse-to-fine reasoning before streaming output. This is a well-motivated approach, as existing autoregressive models struggle with revision, while bidirectional models are computationally expensive. The methodology introduces several key components: 1. **Tree-structured Hierarchy:** Video latents are organized into a multi-level hierarchy, where coarse tokens summarize global temporal structure (high-level plans) and fine tokens encode local visual details. This provides an explicit intermediate space for planning. 2. **Layer-wise Flow-Matching Objective:** The model is trained with a flow-matching objective across all hierarchy levels. A crucial design choice is the "hierarchy-matched denoising schedule," which assigns level-dependent sampling budgets, allowing coarse layers to maintain higher noise levels (preserving uncertainty for multiple global plans) and finer layers to progressively refine these hypotheses. This entropy-matched approach is a clever detail that aligns denoising strength with the abstraction level. 3. **Sparse Hierarchical Attention Pattern (SHAP):** To maintain efficiency, SHAP defines a structured attention mask over flattened tree tokens. Each token attends only to fixed local, parent-level, and first-frame contexts. This enables multi-scale information flow with reduced temporal attention cost, preserving streaming generation properties. 4. **HDR-WAM for Robotics:** The paper extends HDR to a World Action Model (HDR-WAM) for embodied control, combining episode-level visual context with local action-conditioned rollouts. This demonstrates the generality of the hierarchical approach. The methodology is sound, well-articulated, and directly tackles the identified problem. The combination of hierarchical latents, a tailored denoising schedule, and sparse attention is a strong technical contribution.
The experimental evaluation is comprehensive and rigorous, supporting the claims effectively. 1. **Novel Benchmark:** A significant contribution is the construction of a level-stratified multi-step video reasoning benchmark covering six diverse tasks (maze navigation, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, water pouring). This benchmark includes out-of-distribution (OOD) cases and uses robust metrics (success and average progress), which are crucial for evaluating logical consistency over long trajectories. This benchmark itself is a valuable asset for the community. 2. **Strong Baselines:** HDR is compared against relevant full-attention baselines (Bidirectional Diffusion, VideoMAE) and streaming autoregressive baselines (CausalForcing, VideoGPT). The main diffusion baselines are built on the same foundation model (Wan2.2-5B-TI2V) for a controlled comparison. 3. **Quantitative Results:** HDR significantly outperforms the streaming autoregressive baseline (CausalForcing), improving overall success from 34.22 to 60.29 (76.2% relative gain) and average progress from 76.00 to 89.56. It achieves this while maintaining low-latency streaming (0.70s per latent, comparable to CausalForcing's 0.72s) and being substantially faster than bidirectional diffusion (37.92s). It also remains competitive with bidirectional diffusion in reasoning performance, especially considering the latency difference. 4. **Qualitative Evidence:** Visual comparisons demonstrate HDR's ability to perform hierarchical planning and avoid early local commitments that lead to failure in baselines. 5. **Ablation Studies:** * **Layer Importance:** Demonstrates that increasing the number of active hierarchy layers progressively improves performance, validating the coarse-to-fine reasoning process. * **Denoising-step Reduction:** HDR shows superior robustness under reduced denoising budgets compared to both streaming AR and bidirectional diffusion, highlighting the stability provided by coarse layers. * **Data Reduction:** HDR degrades more gracefully than bidirectional diffusion with limited training data, suggesting better data efficiency for learning reasoning rules. * **Entropy-Matched vs. Fully Denoised:** The ablation on denoising schedules confirms the benefit of the entropy-matched approach, which preserves uncertainty at coarse levels, over a naive full denoising strategy. 6. **Real-world Robot Experiments:** The transferability of HDR is demonstrated in a physical robot maze task and on the RoboDojo simulation benchmark. HDR-WAM shows strong performance, particularly on Long-Horizon and Memory tasks, without robot-domain pretraining, indicating its potential for physical interaction and world modeling. This is a very strong point, showcasing practical relevance. The experimental section is exceptionally thorough, covering performance, efficiency, robustness, and real-world applicability.
The paper provides a good level of detail for reproducibility. * The methodology is clearly described, including the hierarchical latent representation, layer-wise flow-matching objective, and SHAP. * The new benchmark is detailed, including task descriptions, difficulty stratification, OOD cases, and evaluation metrics. Appendix provides full benchmark construction and task-specific evaluation details. * Baselines, training details, and implementation settings are specified, including the base diffusion model (Wan2.2-5B-TI2V) and the number of hierarchy levels and denoising schedule. * The entropy-matched denoising schedule derivation is provided in the appendix. * Details for HDR-WAM (temporal views, action alignment, token layout, attention mask, RoboDojo data processing) are also in the appendix. The project demo page (https://hierarchical-diffusion-reasoning.github.io/) further suggests that code or interactive demonstrations might be available, enhancing reproducibility. Overall, the paper provides sufficient information for a skilled researcher to reproduce the main results.
1. **Fixed Hierarchy Structure:** The paper uses a fixed, tree-structured hierarchy. While effective, the optimal hierarchy structure might be task-dependent or could potentially be learned. The current approach might not be optimal for all types of reasoning problems or video lengths. 2. **Pre-computation Phase:** While HDR enables "streaming output," it still involves a coarse-to-fine reasoning phase *before* the final streaming output. This means there's an initial latency for the hierarchical planning, even if subsequent frame generation is fast. For truly real-time, continuous streaming where no look-ahead is possible, this might still be a limitation compared to purely causal models. 3. **Scalability to Extremely Long Videos:** While SHAP reduces attention cost, the total number of hierarchical tokens still grows with video length. For extremely long, open-ended video generation, the fixed-size local and parent contexts might eventually become insufficient for maintaining global consistency over vast temporal spans. 4. **Training Complexity:** Training a multi-level hierarchical diffusion model with a layer-wise objective and specific denoising schedules might be more complex and resource-intensive than training a single-level autoregressive model. 5. **Generalizability of Reasoning:** While the benchmark is diverse, it's still a set of structured reasoning tasks. The extent to which HDR's reasoning capabilities generalize to more open-ended, less structured, or abstract reasoning scenarios remains to be fully explored.
HDR makes a significant contribution to the field of generative AI, particularly in video modeling and embodied AI. 1. **Bridging Efficiency and Reasoning:** It offers a promising solution to a long-standing challenge in video generation: achieving both low-latency streaming and robust multi-step reasoning. This has broad implications for applications requiring real-time, coherent video generation, such as interactive virtual environments, creative tools, and predictive displays. 2. **Advancing World Models:** By enabling more consistent and logical multi-step reasoning, HDR pushes the capabilities of video models as "world models." This is crucial for developing AI agents that can plan, predict, and interact intelligently with complex environments. 3. **Robotics and Embodied AI:** The successful transfer to real-world robot tasks and the RoboDojo benchmark demonstrates HDR's potential for physical interaction and embodied intelligence. Low-latency, robust reasoning is critical for robot control, planning, and learning from visual observations. 4. **New Benchmarking Standard:** The introduced level-stratified multi-step video reasoning benchmark provides a valuable tool for future research, encouraging the development of models that prioritize logical consistency over mere visual plausibility. 5. **Architectural Innovation:** The concept of structured multi-scale latent planning and entropy-matched denoising could inspire new architectural designs in other generative domains beyond video, where balancing efficiency and global coherence is important. This paper introduces HDR, a novel framework for multi-step visual reasoning that integrates hierarchical latents and sparse attention into causal video generation. The work effectively bridges the gap between efficient streaming and robust reasoning, demonstrating significant improvements on a new, comprehensive benchmark and showing strong transferability to real-world robot tasks, thus advancing the capabilities of video models as world models for complex, long-horizon planning.
We study aggregation of statistical evidence under unknown and potentially complex dependence using group-invariance. Building on permutation-based constructions that treat transformed datasets as exchangeable units, we aggregate evidence across statistics for each transformed dataset and calibrate the resulting aggregates across transformations. We develop a finite-sample power and adaptivity theory for this framework, together with extensions to sequential and data-dependent aggregation that preserve validity. For single-batch aggregation, which uses one collection of transformed datasets for both standardization and calibration, we show that the critical values uniformly improve on deterministic calibrations valid under arbitrary dependence, including Bonferroni correction, while adapting to the unknown dependence structure. We also introduce a sequential alpha-spending version that permits early rejection when evidence is strong, and a two-batch extension that separates standardization from calibration to accommodate learned aggregation rules and reduce computation. Applications to adaptive nonparametric testing and conformal prediction illustrate how these results sharpen existing aggregation methods.
Primary: University of Cambridge
All Institutions: University of Cambridge, University College London, KAIST
This paper provides a rigorous and novel theoretical framework for aggregating statistical evidence under group-invariance, introducing Single-Batch and Two-Batch aggregation methods that guarantee finite-sample validity while adapting to unknown dependence structures, thereby offering strictly improved power over worst-case calibrations like Bonferroni and resolving finite-sample size control issues in Monte Carlo approximations.
The paper presents a rigorous theoretical framework for aggregating statistical evidence under group-invariance. The core methodological contribution is the formalization of "Single-Batch" (SB) aggregation, which uses a single set of transformed datasets for both standardization and calibration, and the "Two-Batch" (TB) extension, which separates these roles. The authors prove finite-sample validity for these methods under arbitrary dependence structures, a significant improvement over Monte Carlo approximations that may fail to control Type I error in finite samples. The introduction of sequential alpha-spending for ordered statistics and the analysis of adaptivity to dependence structures (perfect rank alignment) are theoretically sound and novel contributions to the theory of permutation tests and p-value merging.
The paper includes applications to adaptive nonparametric testing and conformal prediction. While the text provided is truncated, the abstract and introduction indicate that the theoretical results are illustrated through these applications. The theoretical proofs are extensive, covering validity, power dominance over Bonferroni/deterministic calibrations, and asymptotic adaptivity. The experimental section likely demonstrates empirical performance gains, particularly in conformal prediction where the TB method avoids computational bottlenecks. The rigorous theoretical backing provides strong indirect evidence of utility.
The paper provides detailed algorithms (SB, SeqSB, TB) and mathematical formulations. The theoretical results are clearly stated with proofs (deferred to supplementary material, which is standard for such theoretical work). The reliance on group-invariance makes the methods reproducible given the data and the group structure.
The methods are restricted to settings where group-invariance (exchangeability) holds under the null hypothesis. This limits applicability to observational studies without such symmetries. The "Two-Batch" method requires splitting data, which can reduce power if the sample size is small. The theoretical gains are asymptotic or finite-sample bounds; practical performance depends on the specific dependence structure and the choice of merging function.
This work has significant implications for multiple testing, conformal prediction, and distribution-free inference in machine learning. By providing valid, less conservative aggregation methods under exchangeability, it enables more powerful hypothesis testing and tighter prediction intervals in settings like genetic association studies, image analysis, and any domain where permutation tests are applicable. It bridges the gap between theoretical validity and practical power in dependent data settings. This paper provides a rigorous and novel theoretical framework for aggregating statistical evidence under group-invariance, introducing Single-Batch and Two-Batch aggregation methods that guarantee finite-sample validity while adapting to unknown dependence structures, thereby offering strictly improved power over worst-case calibrations like Bonferroni and resolving finite-sample size control issues in Monte Carlo approximations.
We study aggregation of statistical evidence under unknown and potentially complex dependence using group-invariance. Building on permutation-based constructions that treat transformed datasets as exchangeable units, we aggregate evidence across statistics for each transformed dataset and calibrate the resulting aggregates across transformations. We develop a finite-sample power and adaptivity theory for this framework, together with extensions to sequential and data-dependent aggregation that preserve validity. For single-batch aggregation, which uses one collection of transformed datasets for both standardization and calibration, we show that the critical values uniformly improve on deterministic calibrations valid under arbitrary dependence, including Bonferroni correction, while adapting to the unknown dependence structure. We also introduce a sequential alpha-spending version that permits early rejection when evidence is strong, and a two-batch extension that separates standardization from calibration to accommodate learned aggregation rules and reduce computation. Applications to adaptive nonparametric testing and conformal prediction illustrate how these results sharpen existing aggregation methods.
Primary: University of Cambridge
All Institutions: University of Cambridge, University College London, KAIST
This paper provides a rigorous and novel theoretical framework for aggregating statistical evidence under group-invariance, introducing Single-Batch and Two-Batch aggregation methods that guarantee finite-sample validity while adapting to unknown dependence structures, thereby offering strictly improved power over worst-case calibrations like Bonferroni and resolving finite-sample size control issues in Monte Carlo approximations.
The paper presents a rigorous theoretical framework for aggregating statistical evidence under group-invariance. The core methodological contribution is the formalization of "Single-Batch" (SB) aggregation, which uses a single set of transformed datasets for both standardization and calibration, and the "Two-Batch" (TB) extension, which separates these roles. The authors prove finite-sample validity for these methods under arbitrary dependence structures, a significant improvement over Monte Carlo approximations that may fail to control Type I error in finite samples. The introduction of sequential alpha-spending for ordered statistics and the analysis of adaptivity to dependence structures (perfect rank alignment) are theoretically sound and novel contributions to the theory of permutation tests and p-value merging.
The paper includes applications to adaptive nonparametric testing and conformal prediction. While the text provided is truncated, the abstract and introduction indicate that the theoretical results are illustrated through these applications. The theoretical proofs are extensive, covering validity, power dominance over Bonferroni/deterministic calibrations, and asymptotic adaptivity. The experimental section likely demonstrates empirical performance gains, particularly in conformal prediction where the TB method avoids computational bottlenecks. The rigorous theoretical backing provides strong indirect evidence of utility.
The paper provides detailed algorithms (SB, SeqSB, TB) and mathematical formulations. The theoretical results are clearly stated with proofs (deferred to supplementary material, which is standard for such theoretical work). The reliance on group-invariance makes the methods reproducible given the data and the group structure.
The methods are restricted to settings where group-invariance (exchangeability) holds under the null hypothesis. This limits applicability to observational studies without such symmetries. The "Two-Batch" method requires splitting data, which can reduce power if the sample size is small. The theoretical gains are asymptotic or finite-sample bounds; practical performance depends on the specific dependence structure and the choice of merging function.
This work has significant implications for multiple testing, conformal prediction, and distribution-free inference in machine learning. By providing valid, less conservative aggregation methods under exchangeability, it enables more powerful hypothesis testing and tighter prediction intervals in settings like genetic association studies, image analysis, and any domain where permutation tests are applicable. It bridges the gap between theoretical validity and practical power in dependent data settings. This paper provides a rigorous and novel theoretical framework for aggregating statistical evidence under group-invariance, introducing Single-Batch and Two-Batch aggregation methods that guarantee finite-sample validity while adapting to unknown dependence structures, thereby offering strictly improved power over worst-case calibrations like Bonferroni and resolving finite-sample size control issues in Monte Carlo approximations.
Higher-order couplings enhance the expressive power of hypergraph neural networks (HGNNs), but they also intensify representation collapse in deep propagation due to strong multi-way feature mixing. This work investigates hypergraph oversmoothing from a dynamical-systems perspective and develops a reaction--diffusion framework for depth-resistant hypergraph learning. By defining hypergraph gradient and divergence operators, we interpret message passing as an incidence-level diffusion process. The analysis of pure diffusion shows that its continuous semiflow exponentially contracts the null-mode-free component of node representations and drives the Dirichlet energy to zero, revealing hypergraph oversmoothing as an intrinsic transverse-energy dissipation phenomenon. Motivated by this analysis, we propose Hypergraph Neural Reaction--Diffusion (HNRD), which introduces a reaction mechanism acting on the transverse component to compensate diffusion-induced dissipation and stabilize discriminative variations. We establish global well-posedness of the proposed dynamics and prove that the null-mode-free Dirichlet energy remains bounded away from zero. A forward-Euler discretization provides a practical HNRD layer with a stability condition for deep propagation. Experiments on benchmark and synthetic heterophilic hypergraphs demonstrate that HNRD consistently improves over representative hypergraph baselines. Depth, robustness, and efficiency analyses further show that HNRD preserves stable performance and nonzero Dirichlet energy under deep propagation and perturbations. These results provide a principled dynamical framework for designing deep hypergraph architectures that maintain higher-order expressiveness without representation collapse.
Primary: Chinese Academy of Sciences
All Institutions: Shandong University, Chinese Academy of Sciences, University of Chinese Academy of Sciences
This work has significant broader impact for the field of graph and hypergraph neural networks: * **Principled Deep HGNN Design**: It provides a principled, theoretically grounded framework for designing deep HGNNs that inherently mitigate oversmoothing, moving beyond heuristic solutions like residual connections. This can lead to more robust and performant deep hypergraph architectures. * **Deeper Understanding of Oversmoothing**: The dynamical-systems view offers a powerful new lens for understanding oversmoothing in hypergraphs, characterizing it as transverse-energy dissipation. This theoretical insight can guide future research into other graph/hypergraph problems. * **Advancement of Continuous-Time GNNs**: By extending the reaction-diffusion paradigm to hypergraphs with strong theoretical guarantees, the paper further validates the utility of continuous-time models (ODEs/PDEs) in graph machine learning, potentially encouraging more research in this direction. * **Applications in Complex Systems**: Hypergraphs are crucial for modeling multi-way interactions in various domains (social networks, biology, chemistry, recommender systems). A robust deep HGNN model like HNRD can significantly improve performance in these applications, especially where deep propagation is necessary to capture complex dependencies. * **Theoretical Foundation for HGNNs**: The rigorous mathematical analysis contributes to building a stronger theoretical foundation for hypergraph learning, which is essential for the maturity and reliability of the field. This paper introduces a novel dynamical-systems framework to analyze and mitigate oversmoothing in hypergraph neural networks, rigorously proving that pure hypergraph diffusion leads to representation collapse and proposing Hypergraph Neural Reaction–Diffusion (HNRD) which provably stabilizes discriminative variations. The paper's strength lies in its deep theoretical analysis, which precisely characterizes hypergraph oversmoothing as transverse-energy dissipation, and its innovative reaction-diffusion mechanism that explicitly counteracts this collapse by preserving a learnable, nonzero transverse-energy level. This principled approach, combined with comprehensive experimental validation across diverse hypergraph benchmarks and detailed analyses (depth, heterophily, ablation, robustness, visualization), demonstrates HNRD's superior performance and stability in deep and challenging settings. The work significantly advances the theoretical understanding and practical design of deep hypergraph architectures, offering a robust solution to a critical problem and paving the way for more principled continuous-time models in higher-order graph learning.
The paper presents a highly rigorous and principled approach to understanding and mitigating oversmoothing in Hypergraph Neural Networks (HGNNs) through a dynamical-systems lens. The methodology is built upon defining hypergraph gradient and divergence operators at the incidence level, which allows for formulating message passing as a learnable incidence-level diffusion equation. This continuous-time formulation is a significant strength, enabling the use of semiflow theory and energy methods for deep theoretical analysis. The core methodological contribution lies in two parts: 1. **Analysis of Pure Hypergraph Diffusion**: The paper rigorously proves that pure hypergraph neural diffusion generates a global continuous semiflow that exponentially contracts the null-mode-free component of node representations and drives the associated Dirichlet energy to zero. This provides a precise mathematical explanation for oversmoothing as an intrinsic transverse-energy dissipation process. This theoretical characterization is a strong foundation for the proposed solution. 2. **Hypergraph Neural Reaction–Diffusion (HNRD)**: Motivated by the diffusion analysis, HNRD introduces a novel reaction mechanism. Crucially, this reaction term acts *only* on the transverse (null-mode-free) component of the node representations. It combines two effects: instantaneous compensation of diffusion-induced dissipation and a bounded saturating feedback that drives the transverse energy toward a positive, learnable level. This design is elegant as it preserves the smoothing benefits of diffusion while preventing the collapse of discriminative information. The paper establishes global well-posedness for HNRD and proves that its null-mode-free Dirichlet energy remains bounded away from zero, directly addressing the oversmoothing problem. A practical discrete HNRD layer is derived via forward-Euler discretization, complete with a step-size stability condition, making the continuous dynamics implementable. The learnable step size further enhances adaptability. The overall methodology is theoretically sound, well-motivated, and provides a clear path from continuous dynamics to a practical deep learning architecture.
The experimental evaluation is comprehensive and robust, supporting the theoretical claims. * **Datasets**: HNRD is evaluated on a diverse set of 11 hypergraph benchmarks, including academic citation/co-authorship networks (Cora, Citeseer, Pubmed, Cora-CA, DBLP-CA) and real-world applications (Zoo, NTU2012, ModelNet40, Walmart, Senate, House). This diversity demonstrates the model's generalizability. * **Baselines**: A wide array of baselines are included, categorized into graph diffusion models (GRAND, GRAND++, GREAD, RDGNN), standard HGNNs (HGNN, HyperGCN, HCHA, HNHN, UniGCNII), expressive HGNNs (AllSetTransformer, AllDeepSets, ED-HNN, HyperGINE, KHGNN), and deep/stable HGNNs (Deep-HGCN, Implicit HNN, FrameHGNN, HND, RFHND). This comprehensive comparison establishes HNRD's state-of-the-art performance against strong competitors. * **Performance**: HNRD achieves the best overall rank on both academic and real-world benchmarks, ranking first on 8 out of 11 datasets and second on the remaining 3. This consistent outperformance is a strong indicator of its effectiveness. * **Synthetic Heterophily**: Experiments on synthetic hypergraphs with controlled heterophily levels demonstrate HNRD's robustness. It maintains superior performance across varying heterophily, where many baselines suffer significant degradation, highlighting its ability to handle complex real-world scenarios. * **Oversmoothing Analysis**: The depth-wise analysis (up to 128 layers) visually and quantitatively confirms HNRD's resistance to oversmoothing. While baselines show clear performance degradation and rapid decay of Dirichlet energy, HNRD maintains stable accuracy and a nonzero Dirichlet energy, directly validating the theoretical non-collapse property. * **Ablation Study**: The ablation study effectively demonstrates the contribution of each component of the reaction term (compensation and bounded feedback), showing that both are crucial for optimal performance. * **Parameter Analysis**: Sensitivity analysis on hidden dimension and step size shows HNRD is relatively stable, with optimal performance at moderate settings. * **Visualization**: t-SNE visualizations of node embeddings at different depths illustrate how HNRD preserves class separability, unlike other models where classes become mixed. * **Robustness and Efficiency**: Additional analyses confirm robustness to perturbations and practical runtime efficiency. The experimental section is exceptionally thorough, providing strong empirical evidence for the theoretical claims and practical utility of HNRD.
The paper provides a GitHub link to the source code, which is a crucial step for reproducibility. It also details experimental settings, including hyperparameter search spaces, optimization methods (Adam), and hardware used (NVIDIA RTX A4000 GPU). Standard hypergraph construction protocols and train/validation/test splits (50%/25%/25%) are followed. The level of detail in the experimental setup, combined with the provided code, suggests good reproducibility.
1. **Computational Complexity**: While the paper mentions practical efficiency, continuous-time models and their discretizations can sometimes be more computationally intensive than simpler message-passing layers, especially for very large hypergraphs or very small step sizes. The stability condition might necessitate small step sizes in certain cases. 2. **Generalizability of Reaction Term Design**: The specific form of the reaction term, while theoretically justified, is tailored to the identified transverse energy dissipation. While effective, exploring alternative reaction mechanisms or more adaptive forms could be an area for future work. 3. **Hypergraph Laplacian Dependence**: The theoretical analysis and the reaction term design rely on the properties of the hypergraph Laplacian and its null space. While standard, this might limit applicability to hypergraph models that do not easily map to such a Laplacian framework. 4. **Parameter Tuning**: The paper mentions tuning the reaction-diffusion step size, which can be a sensitive hyperparameter in ODE-based models. While a learnable sigmoid-constrained step size is proposed, it still adds complexity to the training process.
This work has significant broader impact for the field of graph and hypergraph neural networks: * **Principled Deep HGNN Design**: It provides a principled, theoretically grounded framework for designing deep HGNNs that inherently mitigate oversmoothing, moving beyond heuristic solutions like residual connections. This can lead to more robust and performant deep hypergraph architectures. * **Deeper Understanding of Oversmoothing**: The dynamical-systems view offers a powerful new lens for understanding oversmoothing in hypergraphs, characterizing it as transverse-energy dissipation. This theoretical insight can guide future research into other graph/hypergraph problems. * **Advancement of Continuous-Time GNNs**: By extending the reaction-diffusion paradigm to hypergraphs with strong theoretical guarantees, the paper further validates the utility of continuous-time models (ODEs/PDEs) in graph machine learning, potentially encouraging more research in this direction. * **Applications in Complex Systems**: Hypergraphs are crucial for modeling multi-way interactions in various domains (social networks, biology, chemistry, recommender systems). A robust deep HGNN model like HNRD can significantly improve performance in these applications, especially where deep propagation is necessary to capture complex dependencies. * **Theoretical Foundation for HGNNs**: The rigorous mathematical analysis contributes to building a stronger theoretical foundation for hypergraph learning, which is essential for the maturity and reliability of the field. This paper introduces a novel dynamical-systems framework to analyze and mitigate oversmoothing in hypergraph neural networks, rigorously proving that pure hypergraph diffusion leads to representation collapse and proposing Hypergraph Neural Reaction–Diffusion (HNRD) which provably stabilizes discriminative variations. The paper's strength lies in its deep theoretical analysis, which precisely characterizes hypergraph oversmoothing as transverse-energy dissipation, and its innovative reaction-diffusion mechanism that explicitly counteracts this collapse by preserving a learnable, nonzero transverse-energy level. This principled approach, combined with comprehensive experimental validation across diverse hypergraph benchmarks and detailed analyses (depth, heterophily, ablation, robustness, visualization), demonstrates HNRD's superior performance and stability in deep and challenging settings. The work significantly advances the theoretical understanding and practical design of deep hypergraph architectures, offering a robust solution to a critical problem and paving the way for more principled continuous-time models in higher-order graph learning.
Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.
Primary: Microsoft Research
All Institutions: Microsoft Research, University of California, Santa Cruz, Shanghai Jiao Tong University
RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
The paper introduces RESOURCE2SKILL, a framework for distilling multimodal human resources (videos, code repos, articles, artifacts) into a structured, hierarchical "Skill Wiki" for software agents. The core innovation lies in the schema: skills are tuples of text, visual, code, and metadata, organized by a domain-specific taxonomy. The method employs a "construction operator" that uses a vision-capable LLM to extract and normalize these components, followed by deterministic acceptance predicates (completeness, provenance, deduplication, modality consistency, structural executability). At inference, a two-stage retrieval process (lexical BM25 over taxonomy paths, then LLM selection) allows agents to compose skills. A key feature is the ability to trigger online acquisition when offline skills are insufficient, using the same construction operator. The approach is well-motivated, addressing the gap between implicit agent knowledge and explicit, reusable procedural knowledge derived from rich human resources.
The evaluation is extensive, covering seven diverse authoring domains (PPT, CAD, Web, Excel, Blender, UE5, Reaper) and four agent backends (GPT-5.5, GPT-5.4, Mini, Nano). The primary metric is an automated judge score (vision/audio models) on rendered artifacts, validated by human A/B studies. RESOURCE2SKILL shows significant improvements (+11.9 pp) over no-skill baselines and outperforms strong harness baselines (ClaudeCode-H, Codex-H) in 26/28 cells. Ablations confirm the value of multimodal content, hierarchical organization, and online acquisition. The scaling study shows performance saturates around 200 skills. The results are robust and statistically significant.
The paper provides a code link (https://aka.ms/Resource2Skill) and detailed appendices describing the benchmark construction, judge reliability, and experimental setup. The acceptance predicates are deterministic, and the library construction is frozen before benchmarking, ensuring no data leakage. The matched-brief design and fixed seeds enhance reproducibility. The detailed description of the MCP interface and domain backends allows for replication.
The framework relies on the availability of high-quality multimodal resources (videos, repos) for each domain, which may not exist for all software domains. The automated judge, while correlated with humans, may have biases (e.g., rewarding visual polish over functional correctness in some cases). The online acquisition adds latency. The "structural executability" check ensures code runs but does not guarantee it solves the specific task correctly, leading to potential partial grounding failures as noted in case studies.
This work significantly advances the field of software agents by providing a scalable method for building and maintaining skill libraries from diverse human-generated content. It enables agents to leverage existing human knowledge (tutorials, code) more effectively, potentially lowering the barrier to entry for complex software tasks. The hierarchical wiki structure offers a promising paradigm for organizing agent memory. RESOURCE2SKILL presents a rigorous and impactful framework for distilling multimodal human resources into executable agent skills, demonstrating substantial performance gains across diverse software domains through a structured, hierarchical skill wiki and controlled online acquisition.
Video models are evolving into vision foundation models, yet they still lack human-like multi-step reasoning. Streaming autoregressive diffusion models are efficient but limited in reasoning, while bidirectional diffusion enables global revision with high inference costs due to dense frame-level denoising. Both paradigms struggle to achieve logical consistency and low-latency streaming for complex reasoning tasks. We propose HDR (Hierarchical Denoising for Visual Reasoning), a unified framework that integrates hierarchical latents into causal video generation for multi-step reasoning. HDR organizes video latents into a tree-structured hierarchy, enabling coarse-to-fine reasoning before streaming output. Coarse denoising layers preserve uncertain hypotheses for global planning, while finer layers progressively refine them into concrete visual states. A sparse hierarchical attention pattern (SHAP) further reduces temporal attention costs. We introduce a level-stratified multi-step video reasoning benchmark with out-of-distribution cases, covering six tasks: maze navigation, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, and water pouring. Compared with streaming autoregressive diffusion baselines, HDR improves success from 34.22 to 60.29 (76.2% relative gain) and increases average progress from 76.00 to 89.56, demonstrating more consistent reasoning trajectories. HDR maintains low-latency streaming at 0.70 seconds per latent, achieving 54.2 times faster inference than bidirectional diffusion. It also retains 82.9% of full-data performance with only 2% training data, compared with 52.0% for bidirectional diffusion. Real-world robot experiments further demonstrate HDR's potential for physical interaction and world modeling. Project demo: https://hierarchical-diffusion-reasoning.github.io/.
Primary: The Hong Kong University of Science and Technology
All Institutions: State Key Laboratory of Multimedia Information Processing, The Hong Kong University of Science and Technology, Beihang University, Fuzhou University, Peking University, School of Computer Science
HDR makes a significant contribution to the field of generative AI, particularly in video modeling and embodied AI. 1. **Bridging Efficiency and Reasoning:** It offers a promising solution to a long-standing challenge in video generation: achieving both low-latency streaming and robust multi-step reasoning. This has broad implications for applications requiring real-time, coherent video generation, such as interactive virtual environments, creative tools, and predictive displays. 2. **Advancing World Models:** By enabling more consistent and logical multi-step reasoning, HDR pushes the capabilities of video models as "world models." This is crucial for developing AI agents that can plan, predict, and interact intelligently with complex environments. 3. **Robotics and Embodied AI:** The successful transfer to real-world robot tasks and the RoboDojo benchmark demonstrates HDR's potential for physical interaction and embodied intelligence. Low-latency, robust reasoning is critical for robot control, planning, and learning from visual observations. 4. **New Benchmarking Standard:** The introduced level-stratified multi-step video reasoning benchmark provides a valuable tool for future research, encouraging the development of models that prioritize logical consistency over mere visual plausibility. 5. **Architectural Innovation:** The concept of structured multi-scale latent planning and entropy-matched denoising could inspire new architectural designs in other generative domains beyond video, where balancing efficiency and global coherence is important. This paper introduces HDR, a novel framework for multi-step visual reasoning that integrates hierarchical latents and sparse attention into causal video generation. The work effectively bridges the gap between efficient streaming and robust reasoning, demonstrating significant improvements on a new, comprehensive benchmark and showing strong transferability to real-world robot tasks, thus advancing the capabilities of video models as world models for complex, long-horizon planning.
The paper proposes Hierarchical Denoising for Visual Reasoning (HDR), a novel framework that addresses the fundamental tension between efficient streaming generation and robust multi-step reasoning in video models. The core idea is to integrate hierarchical latents into a causal video generation process, enabling coarse-to-fine reasoning before streaming output. This is a well-motivated approach, as existing autoregressive models struggle with revision, while bidirectional models are computationally expensive. The methodology introduces several key components: 1. **Tree-structured Hierarchy:** Video latents are organized into a multi-level hierarchy, where coarse tokens summarize global temporal structure (high-level plans) and fine tokens encode local visual details. This provides an explicit intermediate space for planning. 2. **Layer-wise Flow-Matching Objective:** The model is trained with a flow-matching objective across all hierarchy levels. A crucial design choice is the "hierarchy-matched denoising schedule," which assigns level-dependent sampling budgets, allowing coarse layers to maintain higher noise levels (preserving uncertainty for multiple global plans) and finer layers to progressively refine these hypotheses. This entropy-matched approach is a clever detail that aligns denoising strength with the abstraction level. 3. **Sparse Hierarchical Attention Pattern (SHAP):** To maintain efficiency, SHAP defines a structured attention mask over flattened tree tokens. Each token attends only to fixed local, parent-level, and first-frame contexts. This enables multi-scale information flow with reduced temporal attention cost, preserving streaming generation properties. 4. **HDR-WAM for Robotics:** The paper extends HDR to a World Action Model (HDR-WAM) for embodied control, combining episode-level visual context with local action-conditioned rollouts. This demonstrates the generality of the hierarchical approach. The methodology is sound, well-articulated, and directly tackles the identified problem. The combination of hierarchical latents, a tailored denoising schedule, and sparse attention is a strong technical contribution.
The experimental evaluation is comprehensive and rigorous, supporting the claims effectively. 1. **Novel Benchmark:** A significant contribution is the construction of a level-stratified multi-step video reasoning benchmark covering six diverse tasks (maze navigation, Tower of Hanoi, one-line drawing, sliding puzzle, Sokoban, water pouring). This benchmark includes out-of-distribution (OOD) cases and uses robust metrics (success and average progress), which are crucial for evaluating logical consistency over long trajectories. This benchmark itself is a valuable asset for the community. 2. **Strong Baselines:** HDR is compared against relevant full-attention baselines (Bidirectional Diffusion, VideoMAE) and streaming autoregressive baselines (CausalForcing, VideoGPT). The main diffusion baselines are built on the same foundation model (Wan2.2-5B-TI2V) for a controlled comparison. 3. **Quantitative Results:** HDR significantly outperforms the streaming autoregressive baseline (CausalForcing), improving overall success from 34.22 to 60.29 (76.2% relative gain) and average progress from 76.00 to 89.56. It achieves this while maintaining low-latency streaming (0.70s per latent, comparable to CausalForcing's 0.72s) and being substantially faster than bidirectional diffusion (37.92s). It also remains competitive with bidirectional diffusion in reasoning performance, especially considering the latency difference. 4. **Qualitative Evidence:** Visual comparisons demonstrate HDR's ability to perform hierarchical planning and avoid early local commitments that lead to failure in baselines. 5. **Ablation Studies:** * **Layer Importance:** Demonstrates that increasing the number of active hierarchy layers progressively improves performance, validating the coarse-to-fine reasoning process. * **Denoising-step Reduction:** HDR shows superior robustness under reduced denoising budgets compared to both streaming AR and bidirectional diffusion, highlighting the stability provided by coarse layers. * **Data Reduction:** HDR degrades more gracefully than bidirectional diffusion with limited training data, suggesting better data efficiency for learning reasoning rules. * **Entropy-Matched vs. Fully Denoised:** The ablation on denoising schedules confirms the benefit of the entropy-matched approach, which preserves uncertainty at coarse levels, over a naive full denoising strategy. 6. **Real-world Robot Experiments:** The transferability of HDR is demonstrated in a physical robot maze task and on the RoboDojo simulation benchmark. HDR-WAM shows strong performance, particularly on Long-Horizon and Memory tasks, without robot-domain pretraining, indicating its potential for physical interaction and world modeling. This is a very strong point, showcasing practical relevance. The experimental section is exceptionally thorough, covering performance, efficiency, robustness, and real-world applicability.
The paper provides a good level of detail for reproducibility. * The methodology is clearly described, including the hierarchical latent representation, layer-wise flow-matching objective, and SHAP. * The new benchmark is detailed, including task descriptions, difficulty stratification, OOD cases, and evaluation metrics. Appendix provides full benchmark construction and task-specific evaluation details. * Baselines, training details, and implementation settings are specified, including the base diffusion model (Wan2.2-5B-TI2V) and the number of hierarchy levels and denoising schedule. * The entropy-matched denoising schedule derivation is provided in the appendix. * Details for HDR-WAM (temporal views, action alignment, token layout, attention mask, RoboDojo data processing) are also in the appendix. The project demo page (https://hierarchical-diffusion-reasoning.github.io/) further suggests that code or interactive demonstrations might be available, enhancing reproducibility. Overall, the paper provides sufficient information for a skilled researcher to reproduce the main results.
1. **Fixed Hierarchy Structure:** The paper uses a fixed, tree-structured hierarchy. While effective, the optimal hierarchy structure might be task-dependent or could potentially be learned. The current approach might not be optimal for all types of reasoning problems or video lengths. 2. **Pre-computation Phase:** While HDR enables "streaming output," it still involves a coarse-to-fine reasoning phase *before* the final streaming output. This means there's an initial latency for the hierarchical planning, even if subsequent frame generation is fast. For truly real-time, continuous streaming where no look-ahead is possible, this might still be a limitation compared to purely causal models. 3. **Scalability to Extremely Long Videos:** While SHAP reduces attention cost, the total number of hierarchical tokens still grows with video length. For extremely long, open-ended video generation, the fixed-size local and parent contexts might eventually become insufficient for maintaining global consistency over vast temporal spans. 4. **Training Complexity:** Training a multi-level hierarchical diffusion model with a layer-wise objective and specific denoising schedules might be more complex and resource-intensive than training a single-level autoregressive model. 5. **Generalizability of Reasoning:** While the benchmark is diverse, it's still a set of structured reasoning tasks. The extent to which HDR's reasoning capabilities generalize to more open-ended, less structured, or abstract reasoning scenarios remains to be fully explored.
HDR makes a significant contribution to the field of generative AI, particularly in video modeling and embodied AI. 1. **Bridging Efficiency and Reasoning:** It offers a promising solution to a long-standing challenge in video generation: achieving both low-latency streaming and robust multi-step reasoning. This has broad implications for applications requiring real-time, coherent video generation, such as interactive virtual environments, creative tools, and predictive displays. 2. **Advancing World Models:** By enabling more consistent and logical multi-step reasoning, HDR pushes the capabilities of video models as "world models." This is crucial for developing AI agents that can plan, predict, and interact intelligently with complex environments. 3. **Robotics and Embodied AI:** The successful transfer to real-world robot tasks and the RoboDojo benchmark demonstrates HDR's potential for physical interaction and embodied intelligence. Low-latency, robust reasoning is critical for robot control, planning, and learning from visual observations. 4. **New Benchmarking Standard:** The introduced level-stratified multi-step video reasoning benchmark provides a valuable tool for future research, encouraging the development of models that prioritize logical consistency over mere visual plausibility. 5. **Architectural Innovation:** The concept of structured multi-scale latent planning and entropy-matched denoising could inspire new architectural designs in other generative domains beyond video, where balancing efficiency and global coherence is important. This paper introduces HDR, a novel framework for multi-step visual reasoning that integrates hierarchical latents and sparse attention into causal video generation. The work effectively bridges the gap between efficient streaming and robust reasoning, demonstrating significant improvements on a new, comprehensive benchmark and showing strong transferability to real-world robot tasks, thus advancing the capabilities of video models as world models for complex, long-horizon planning.
Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.
Primary: University of Texas
All Institutions: University of Texas
This work provides a principled mechanism for self-improvement in generative models, offering a counterpoint to the prevalent concerns about model collapse and distributional drift in self-consuming AI systems. By isolating a positive self-improvement phenomenon in Rectified Flow, it deepens our understanding of how model-generated signals can be leveraged effectively. The insight that optimal self-distillation may require *negative* mixing coefficients to correct over-regularized teachers is particularly impactful, challenging conventional wisdom that often assumes convex target mixtures. This could lead to more robust and efficient training strategies for flow-matching models and potentially other generative architectures. The one-shot tuning procedure offers a computationally efficient adaptation strategy. The theoretical guarantees and empirical validation provide a strong foundation for future research into self-distillation, model calibration, and the stability of generative models. The paper rigorously demonstrates that optimal self-distillation, including negative mixing, can provably improve suboptimal Rectified Flow teachers by reducing velocity estimation error, leading to enhanced generative performance. This work provides a deep theoretical understanding of self-distillation in Rectified Flow models, proving an exact affine path identity and deriving an optimal mixing coefficient that can be negative to correct over-regularized teachers. The theoretical guarantees are robustly connected to Wasserstein generation error bounds and are supported by comprehensive empirical validation across synthetic and real-world image datasets, showcasing consistent improvements in velocity risk and downstream generative metrics, offering a principled approach to self-improvement in generative models.
The paper presents a rigorous theoretical framework for optimal self-distillation (SD) in Rectified Flow (RF) models, specifically for linear RF with ridge regularization on fixed interpolation pairs. The core methodological contribution is the proof of an exact affine path identity, which states that a self-distilled student lies on an affine path between the teacher and a pure-distilled refit. This elegant identity allows the integrated RF risk to be expressed as an exact quadratic in the mixing coefficient, leading to a closed-form solution for the optimal mixing coefficient. A key insight is the "sign rule," which dictates that positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized ones, a departure from conventional convex distillation. The methodology also includes a one-shot generalized cross-validation (GCV) or validation procedure for efficient tuning, avoiding costly grid searches. Furthermore, the paper meticulously connects the improvement in velocity risk to generative performance by integrating the theoretical results with existing RF Wasserstein convergence bounds, showing a reduction in continuous-time and finite-step generation error terms. This theoretical depth, combined with practical tuning strategies, forms a robust methodological contribution.
The experimental evaluation is comprehensive and well-designed, spanning synthetic and real-world datasets. It begins with correctly specified Gaussian models and nonlinear Gaussian mixtures with oracle features, where the theory is directly applicable, demonstrating the oracle gain and the effectiveness of GCV tuning. Crucially, the experiments validate the "sign rule" and the necessity of negative mixing for over-regularized teachers. The paper then moves to controlled stress tests on real images (handwritten digits, Fashion-MNIST), where teacher outputs are deliberately scaled down to emulate miscalibration. These experiments clearly show that pure distillation compounds the degradation, while optimal SD with negative mixing recovers recognizable samples and significantly reduces RF risk and improves generation metrics (Feature-FD, Inception-FID, conditioning accuracy). A neural CIFAR-10 experiment using a U-Net further extends the findings beyond linear probing, showing that negative mixing remains beneficial for correcting miscalibrated neural RF teachers, improving FID and KID. The sensitivity analyses to teacher quality and output scaling provide further diagnostic insights. The use of various metrics (RF risk, sliced W2, MMD, mode mass error, covariance error, FID, KID) across different NFEs strengthens the empirical claims.
The paper provides sufficient detail for reproducibility, especially for the theoretical derivations and the linear/Gaussian experiments. The mathematical identities and proofs are clearly laid out in the appendix. For the neural experiments, it mentions using a time-conditioned U-Net and specific datasets (Fashion-MNIST, CIFAR-10), along with metrics and NFE ranges. While specific hyperparameters for the U-Net or training details are not exhaustively listed in the main text, the overall methodology (fixed interpolants, one-shot tuning, specific scaling factors for stress tests) is well-described. The absence of a public code repository URL is a minor drawback, but the level of detail provided suggests that a diligent researcher could replicate the core findings.
The paper explicitly acknowledges several limitations. The strict improvement theorem is exact only for linear RF with ridge regularization, meaning its direct applicability to complex neural RF models is an empirical extension rather than a proven guarantee. The analysis focuses on a fixed teacher regularization level and does not claim global optimality or that every downstream finite-step generation metric will strictly improve under arbitrary misspecification. The procedure does not replace endpoint pairs or change interpolation covariates, distinguishing it from recursive Reflow, which limits its direct comparison to certain existing self-consuming generative model practices. Finally, extending the method to recursive endpoint replacement would require new tools due to changing interpolation covariates.
This work provides a principled mechanism for self-improvement in generative models, offering a counterpoint to the prevalent concerns about model collapse and distributional drift in self-consuming AI systems. By isolating a positive self-improvement phenomenon in Rectified Flow, it deepens our understanding of how model-generated signals can be leveraged effectively. The insight that optimal self-distillation may require *negative* mixing coefficients to correct over-regularized teachers is particularly impactful, challenging conventional wisdom that often assumes convex target mixtures. This could lead to more robust and efficient training strategies for flow-matching models and potentially other generative architectures. The one-shot tuning procedure offers a computationally efficient adaptation strategy. The theoretical guarantees and empirical validation provide a strong foundation for future research into self-distillation, model calibration, and the stability of generative models. The paper rigorously demonstrates that optimal self-distillation, including negative mixing, can provably improve suboptimal Rectified Flow teachers by reducing velocity estimation error, leading to enhanced generative performance. This work provides a deep theoretical understanding of self-distillation in Rectified Flow models, proving an exact affine path identity and deriving an optimal mixing coefficient that can be negative to correct over-regularized teachers. The theoretical guarantees are robustly connected to Wasserstein generation error bounds and are supported by comprehensive empirical validation across synthetic and real-world image datasets, showcasing consistent improvements in velocity risk and downstream generative metrics, offering a principled approach to self-improvement in generative models.
We introduce \emph{gate-zero growth}, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces \emph{rank separation} in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation. As gates open during continual learning, function drift is $O(\|\boldsymbolα\|^2)$ and Jacobian leakage $O(\|\boldsymbolα\|_\infty)$, giving a controlled departure from the FP locus. On a $300\mathrm{M}\to857\mathrm{M}$ Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting ($Δ_A < 0.1$) under both exact-preservation (Isolation) and joint-frontier (Freeze-Nothing) operating points, while a non-FP control ($G_{\text{stack}}$) suffers an order-of-magnitude larger forgetting under the same recipe. The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in CL.
Primary: unknown
All Institutions: unknown
The paper's findings have significant positive broader impacts. By providing a principled geometric framework for function-preserving growth, it offers a pathway to adapt and expand existing large pre-trained models without catastrophic forgetting. This can substantially lower the compute and energy costs associated with obtaining more capable models, as it avoids training from scratch. This reduction in resource requirements can democratize access to advanced ML capabilities, making it more feasible for organizations without frontier-pretraining budgets to leverage and customize large models. The framework also provides a deeper understanding of widely used parameter-efficient fine-tuning (PEFT) techniques like LoRA and adapters, potentially guiding their more effective application in continual learning settings. The paper explicitly states that downstream-use mitigations (data curation, alignment, evaluation) are out of scope, which is a reasonable boundary for this technical contribution. This paper introduces gate-zero growth, a function-preserving operator, and a profound geometric framework that unifies the understanding of zero-initialization techniques (LoRA, ReZero, adapters) in continual learning. The work provides rigorous theoretical propositions on Jacobian rank separation and Fisher information structure, empirically validated through large-scale Transformer and MoE experiments demonstrating near-zero forgetting and controlled function drift, offering a principled approach to safe capacity activation and model expansion.
The paper introduces "gate-zero growth," a function-preserving (FP) operator for continual learning, and, more importantly, a comprehensive geometric framework that unifies several existing zero-initialization techniques (LoRA, ReZero, zero-init adapters) under a common theoretical lens. The core of the methodology lies in the rigorous mathematical analysis of the functional Jacobian at the growth point. Under a "transversality condition," the framework predicts "rank separation" in the Jacobian, meaning old directions are preserved, new-weight directions are exactly flat, and new gate directions are the sole first-order source of functional variation. This leads to several key theoretical propositions: 1. **Exact flat directions for new weights:** $\partial f / \partial W'_{\text{new}} = 0$ at the growth point, implying new weights can be updated without affecting the function at first order. 2. **Conditional rank additivity:** The Jacobian rank increases by the number of new gates, provided the new gate directions are linearly independent of the old Jacobian's image (transversality). 3. **Sparse Fisher block structure:** The Fisher information matrix has zero blocks for new weights and cross-terms involving new weights, simplifying second-order optimization. 4. **Isolation as a coordinate-subspace projection:** Freezing old parameters becomes an exact projection onto the new-parameter subspace, preserving the old function at the growth point. 5. **Four-way subspace partition:** A detailed orthogonal decomposition of the parameter space, quantifying plasticity and the failure mode of transversality. Beyond the growth point, the framework provides local bounds on function drift ($O(\|\alpha\|^2)$) and Jacobian leakage ($O(\|\alpha\|_\infty)$) as gates open. This theoretical depth provides a principled understanding of *why* certain CL strategies (like freezing old parameters) work effectively with zero-initialized growth methods. The methodology is highly rigorous and conceptually unifying.
The experimental evaluation is robust and conducted at a significant scale, validating the theoretical framework. 1. **Scale and Setup:** Experiments involve adapting a 300M Transformer to 857M (12 to 48 layers) and a 706M MoE to 2.5B (12 to 24 layers, 4 to 8 experts). Datasets are WikiText-103 ($D_A$) and BookCorpus ($D_B$). This large scale adds significant credibility to the findings. 2. **Key Findings:** * **Near-zero forgetting:** Gate-FP + Isolation achieves $\Delta_A < 0.1$ (specifically, +0.04 for dense, +0.20 for MoE), demonstrating near-perfect preservation of old knowledge, as predicted by the theory. * **Non-FP control:** A non-FP baseline ($G_{stack}$) suffers an order-of-magnitude larger forgetting, validating the importance of the rank-separation guarantee. * **Unification Validation:** Zero-init residual stacking (an alternative FP operator) also achieves near-zero forgetting under Isolation, confirming that the rank separation is a structural property shared by zero-init FP operators, not specific to gate-zero growth. * **Operating Points:** The paper identifies two practical operating points: Isolation (exact preservation) and Freeze-Nothing (joint $(PPL_A, PPL_B)$ optimum, leveraging the controlled-departure regime). * **MoE Specifics:** While preservation transfers to MoE, plasticity is weaker, with diagnostics localizing the issue to clone-block redundancy, providing valuable insights for future work. 3. **Ablations:** Comprehensive ablations on growth factor, freezing combinations, replay fraction, gate initialization, and growth timing further support the claims and provide practical guidance. The experimental design is thorough, using appropriate metrics (PPL, $\Delta_A$), and the results strongly support the theoretical predictions, making a compelling case for the proposed framework.
The paper provides a good level of detail for empirical reproducibility: model sizes, datasets, training epochs, GPU type (NVIDIA L20), fp16, gradient accumulation to effective batch size 128, and total compute (2,500 GPU-hours). This is commendable. However, the theoretical proofs and detailed diagnostics (e.g., for transversality, gradient-covariance, Hessian) are deferred to an appendix, which is not provided in the submitted text. Without the appendix, full theoretical reproducibility and verification of the mathematical claims are not possible. For the empirical results, the level of detail is high enough for a skilled researcher to likely reproduce the main findings, though exact cloning noise details or specific block initialization strategies might require more explicit code or pseudocode.
1. **Missing Appendix:** The most significant limitation is the absence of the appendix containing detailed proofs and diagnostics. This prevents a full verification of the theoretical claims. 2. **Stochastic Geometry Estimators:** The paper acknowledges that Hessian top eigenvalues and gradient-covariance rank diagnostics are "coarse local diagnostics," relying on single-batch or limited mini-batch samples. This suggests potential instability or approximation in these specific measurements. 3. **MoE Plasticity Gap:** While the preservation mechanism transfers to MoE, the plasticity is significantly weaker. The paper diagnoses this as clone-block redundancy but leaves the solution (MoE-specific tuning or modified operator) as future work. This indicates the current gate-zero growth operator might not be universally optimal across architectures for plasticity. 4. **Limited CL Baseline Comparison:** The paper explicitly states that $G_{stack}$ is a negative control, not a competitive CL baseline. It defers direct head-to-head comparison with gradient-projection, masking, and other state-of-the-art CL methods (EWC, A-GEM, PackNet) to future work. While it compares to LoRA-CL and zero-init stacking, a broader comparison against diverse CL strategies would strengthen the practical utility claims. 5. **No Code/Project URL:** The absence of a project page or code repository makes it harder for others to directly implement and build upon the proposed method.
The paper's findings have significant positive broader impacts. By providing a principled geometric framework for function-preserving growth, it offers a pathway to adapt and expand existing large pre-trained models without catastrophic forgetting. This can substantially lower the compute and energy costs associated with obtaining more capable models, as it avoids training from scratch. This reduction in resource requirements can democratize access to advanced ML capabilities, making it more feasible for organizations without frontier-pretraining budgets to leverage and customize large models. The framework also provides a deeper understanding of widely used parameter-efficient fine-tuning (PEFT) techniques like LoRA and adapters, potentially guiding their more effective application in continual learning settings. The paper explicitly states that downstream-use mitigations (data curation, alignment, evaluation) are out of scope, which is a reasonable boundary for this technical contribution. This paper introduces gate-zero growth, a function-preserving operator, and a profound geometric framework that unifies the understanding of zero-initialization techniques (LoRA, ReZero, adapters) in continual learning. The work provides rigorous theoretical propositions on Jacobian rank separation and Fisher information structure, empirically validated through large-scale Transformer and MoE experiments demonstrating near-zero forgetting and controlled function drift, offering a principled approach to safe capacity activation and model expansion.
Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient κ_R. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from 1/4 to 1/2 as loop count grows at fixed physical depth. The resulting method, DeepLoop, keeps the Post-LN DeepNorm architecture and sets α=(2N)^{1/2} and β=(8N)^{-1/2} for unrolled depth N. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.
Primary: Princeton University
All Institutions: Princeton University
DeepLoop offers a significant contribution to the stability and scalability of looped and recurrent Transformer architectures. By enabling stable training of deeper effective models without increasing parameter count, it facilitates more efficient use of computational resources, potentially leading to smaller models with comparable performance or larger models that can perform more complex iterative reasoning. This is particularly relevant for tasks requiring long-context processing, iterative refinement, or adaptive computation. The theoretical framework also advances the fundamental understanding of deep learning optimization, especially in the context of weight sharing and recurrent computation. There are no obvious negative societal impacts; the work primarily focuses on improving model efficiency and capability. DeepLoop introduces a novel theoretical framework to address the residual-scaling problem in looped Transformers, proposing a simple yet effective scaling rule that significantly improves stability and performance in recurrent depth settings. The paper rigorously derives a loop-aware first-order perturbation bound, formalizing the "tied-depth effect" with a visit-alignment coefficient, and empirically validates the predicted exponent shift from p=1/4 to p=1/2 on GPT-style language models and hierarchical reasoners. This work provides a crucial, parameter-free architectural correction for a growing class of efficient deep learning models, contributing both to theoretical understanding and practical model development.
The paper presents a rigorous theoretical analysis of residual scaling in looped Transformers, extending the DeepNorm framework to account for parameter reuse. The core contribution is the formalization of the "tied-depth effect" through a first-order perturbation bound controlled by a visit-alignment coefficient Îş_R. This coefficient quantifies how visit-wise gradients and sensitivities align across repeated applications of the same physical block. The derivation correctly recovers the DeepNorm exponent (p=1/4) when visits decorrelate and predicts a higher exponent (p=1/2) in the conservative aligned regime, which is crucial for fixed physical depth and increasing loop counts. The resulting method, DeepLoop, is a simple, parameter-free modification to DeepNorm's scaling rules, changing the exponent from 1/4 to 1/2. The methodology is sound, building upon established stability analyses and adapting them to a novel architectural paradigm. The extension of the framework to hierarchical recurrent reasoners, considering gradient truncation and per-module scaling, further demonstrates the robustness and applicability of the theoretical insights. The simplicity of DeepLoop (a one-line change to constants) is a significant strength.
The experimental evaluation is well-designed and provides strong empirical support for the theoretical predictions. The authors conduct controlled ablations on GPT-style looped language models at GPT-2 small and medium scales, varying the loop count R. DeepLoop consistently improves validation loss and downstream accuracy (on lm-evaluation-harness suite) as R increases, while being neutral at R=1 (where no blocks are revisited). This directly validates the paper's central hypothesis that loop-aware scaling becomes necessary when recurrent depth is activated. Furthermore, the application to a Hierarchical Reasoning Model (HRM) on ARC-AGI, where DeepLoop (with p=1/2) significantly improves voted accuracy over the baseline, confirms the theory's applicability to more complex recurrent architectures. The empirical p-sweep at R=3, showing a stability boundary near p=1/2, provides direct evidence for the predicted exponent threshold. The use of H200 GPUs and detailed compute cost reporting adds to the transparency. While single-seed runs for LLMs are a limitation, the consistent trends across scales and tasks, and the specific validation on ARC-AGI with multi-seed controls, bolster confidence in the results.
The paper provides a GitHub link to the code, which is a strong indicator of reproducibility. The experimental setup is described in detail, including model architectures (GPT-MHA-RoPE, hidden sizes, layers), training data (FineWeb-Edu 50B tokens), optimizer steps, context length, global batch size, and GPU types. The specific scaling rules for DeepLoop are clearly defined. The ARC-AGI experiments also detail the baseline, optimizer, data, and evaluation protocol. The compute resources section is commendable for its transparency. The main caveat is the reliance on single-seed runs for the LLM experiments, which the authors acknowledge, meaning full reproducibility of the exact numbers would require multi-seed runs. However, the qualitative trends are clear enough to be convincing.
The authors themselves identify several limitations: 1. Direct measurement of the visit-alignment coefficient Îş_R or cross-round gradient alignment is left for future work. 2. Testing the boundary at larger scales or under alternative parameterizations. 3. Exploring whether training can encourage decorrelated visits to use less conservative exponents. Additionally, the LLM experiments are conducted at GPT-2 small/medium scales, which are not considered "large scale" in the current landscape of LLM research. The p-sweep is limited to one scale, one loop depth, and a relatively small number of optimizer steps. The single-seed nature of the LLM experiments means that while the trends are compelling, the precise magnitude of gains might vary across runs.
DeepLoop offers a significant contribution to the stability and scalability of looped and recurrent Transformer architectures. By enabling stable training of deeper effective models without increasing parameter count, it facilitates more efficient use of computational resources, potentially leading to smaller models with comparable performance or larger models that can perform more complex iterative reasoning. This is particularly relevant for tasks requiring long-context processing, iterative refinement, or adaptive computation. The theoretical framework also advances the fundamental understanding of deep learning optimization, especially in the context of weight sharing and recurrent computation. There are no obvious negative societal impacts; the work primarily focuses on improving model efficiency and capability. DeepLoop introduces a novel theoretical framework to address the residual-scaling problem in looped Transformers, proposing a simple yet effective scaling rule that significantly improves stability and performance in recurrent depth settings. The paper rigorously derives a loop-aware first-order perturbation bound, formalizing the "tied-depth effect" with a visit-alignment coefficient, and empirically validates the predicted exponent shift from p=1/4 to p=1/2 on GPT-style language models and hierarchical reasoners. This work provides a crucial, parameter-free architectural correction for a growing class of efficient deep learning models, contributing both to theoretical understanding and practical model development.
Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, Huazhong University of Science and Technology, University of Science and Technology of China, Shenzhen Loop Area Institute
VideoRAE effectively demonstrates that frozen Video Foundation Models can be transformed into high-quality, generation-friendly latent spaces via representation alignment and multi-codebook quantization, achieving state-of-the-art reconstruction and generative performance with faster convergence.
The paper proposes VideoRAE, a novel framework that repurposes frozen Video Foundation Models (VFMs) like V-JEPA 2 and VideoMAEv2 as the backbone for video autoencoding. Instead of training encoders from scratch, it extracts hierarchical features and compresses them using a lightweight 1D self-attention projector. The key methodological innovations are: 1) A "Representation Alignment" (REPA) objective that aligns decoder features with the frozen VFM teacher, effectively replacing the traditional KL-divergence regularization in VAEs. 2) A Multi-Codebook SimVQ strategy for discrete tokenization that prevents codebook collapse in high-dimensional semantic spaces. 3) A unified architecture supporting both continuous (for Diffusion Transformers) and discrete (for Autoregressive) latent spaces. The approach is theoretically sound, leveraging the semantic richness of foundation models while addressing the specific challenges of video compression (temporal redundancy, high dimensionality).
The experimental evaluation is comprehensive and rigorous. The authors evaluate VideoRAE on reconstruction metrics (PSNR, LPIPS, rFVD) on UCF-101 and TokenBench, showing state-of-the-art or highly competitive performance against dedicated tokenizers (LARP, SweetTok, LTX-VAE, etc.). Crucially, they demonstrate the *generative* utility of these latents by training downstream models (AR and DiT) on UCF-101, achieving SOTA gFVD scores (40 for AR, 93 for DiT) with significantly fewer tokens and faster convergence (5x) compared to baselines. They also conduct a controlled 2B-scale text-to-video replacement study, showing that swapping LTX-VAE with VideoRAE improves convergence and VBench scores. The ablation studies effectively validate the components (REPA, Multi-Codebook, Feature Aggregation). The results strongly support the claim that frozen VFM representations are superior latent bases for generation.
The paper provides detailed architectural descriptions, loss functions, and training hyperparameters. The code and model weights are promised to be released at the provided URL. The use of open-source VFMs (V-JEPA 2, VideoMAEv2) and standard datasets (UCF-101, Kinetics, VideoUFO) ensures that the baseline comparisons are reproducible. The controlled T2V experiment is well-described, allowing for replication.
The primary limitation is the reliance on large, frozen foundation models, which increases inference memory overhead during the encoding phase compared to lightweight, trained-from-scratch VAEs. While the projector is lightweight, the initial feature extraction requires loading a billion-parameter model. Additionally, the paper focuses heavily on UCF-101 for generation benchmarks; while the T2V study is promising, broader evaluation on larger, more diverse video benchmarks (e.g., DAVIS, Hollywood2) for generation quality would strengthen the claims. The performance gap between V-JEPA 2 and VideoMAEv2 suggests that the choice of VFM is critical, and the generalizability to other VFM architectures (e.g., video-specific transformers vs. JEPA) is not fully explored.
This work has significant implications for the field of video generation. By demonstrating that frozen foundation models can serve as effective, semantically rich latent spaces, it bridges the gap between video understanding and generation. This could lead to more efficient training pipelines for generative models, as the "heavy lifting" of learning semantic structure is offloaded to the foundation model. It challenges the prevailing paradigm of pixel-driven autoencoders and may accelerate the development of more capable and efficient video generation systems. The potential for unifying understanding and generation tasks is a major step forward. VideoRAE effectively demonstrates that frozen Video Foundation Models can be transformed into high-quality, generation-friendly latent spaces via representation alignment and multi-codebook quantization, achieving state-of-the-art reconstruction and generative performance with faster convergence.
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.
Primary: Shanghai AI Laboratory
All Institutions: Shanghai AI Laboratory
AgentCompass provides a vital, modular infrastructure for LLM agent evaluation, significantly improving reproducibility and standardization in the field through its decoupled Benchmark-Harness-Environment architecture and comprehensive diagnostic tools.
The paper proposes AgentCompass, a modular evaluation infrastructure designed to address the fragmentation in LLM agent evaluation. The core methodological contribution is the architectural decoupling of the evaluation process into three independent components: Benchmark (task definitions), Harness (execution logic/runtimes), and Environment (interaction interfaces). This separation of concerns is a standard software engineering best practice but is explicitly framed here as a novel solution to the "tightly coupled" nature of existing agent benchmarks. The system includes a fault-tolerant asynchronous runtime and trajectory analysis tools. While the individual components are not algorithmically novel, the synthesis into a unified, extensible framework that supports over 20 benchmarks across five dimensions represents a significant engineering contribution to the field's infrastructure.
The paper demonstrates the utility of AgentCompass by running evaluations on over 20 existing benchmarks. The primary experimental result is the demonstration of reproducibility and the ability to diagnose nuanced failure modes like reward hacking through trajectory analysis. The paper likely compares the efficiency and ease of use of AgentCompass against ad-hoc evaluation scripts or older, monolithic frameworks. The results serve to validate the infrastructure's capability to handle diverse agent tasks (e.g., web navigation, coding, reasoning) rather than introducing a new SOTA model. The evaluation is sound in its purpose: proving the framework works as intended and provides insights into agent behaviors that were previously hard to capture.
High. The paper emphasizes open-source availability (linked GitHub repository) and the modular nature of the system, which inherently promotes reproducibility by allowing researchers to swap components without reimplementing execution logic. The inclusion of comprehensive trajectory analysis tools further aids in reproducing and diagnosing specific agent failures.
As an infrastructure paper, it does not propose new algorithms or models. Its impact is contingent on community adoption. The novelty is primarily architectural/engineering rather than theoretical. The paper may face the challenge of "benchmark proliferation" if it simply aggregates existing benchmarks without introducing new, rigorous evaluation protocols that challenge current SOTA models in novel ways. Additionally, the "fault-tolerance" and "asynchronous runtime" are engineering features; their technical depth is limited compared to algorithmic contributions.
Significant positive impact. By providing a unified, standardized, and open-source evaluation infrastructure, AgentCompass reduces the barrier to entry for agent research, ensures fairer comparisons between different agent architectures, and accelerates the field by preventing redundant engineering efforts. It addresses a critical pain point in the LLM agent community: the lack of standardized, reproducible evaluation pipelines. AgentCompass provides a vital, modular infrastructure for LLM agent evaluation, significantly improving reproducibility and standardization in the field through its decoupled Benchmark-Harness-Environment architecture and comprehensive diagnostic tools.
Remote sensing offers an unparalleled vantage point for observing the Earth's long-term surface evolution, yet it demands that a model not only perceive land cover at isolated moments, but also track changes, memorize evolution histories, and reason across time and space. However, existing studies lack a systematic evaluation that dissects these distinct competencies. To fill this gap, we introduce ChronoBench, a multidimensional benchmark that decomposes this task into four progressive cognitive levels (i.e., Land Cover Perception, Temporal Recognition, Long-Term Memory, and Spatio-Temporal Reasoning). The ChronoBench comprises 12 sub-tasks and 17,689 rigorously validated QA (Question-Answer) pairs. Extensive evaluations reveal that mainstream MLLMs fall drastically behind human experts, with Long-Term Memory emerging as the most critical bottleneck. Motivated by this finding, we further propose GeoChrono, an MLLM with enhanced capabilities for tracing, memorizing, and reasoning about long-term geographic evolution. Leveraging the physical prior that geographic parcels remain spatially fixed while their semantics evolve, we design a Temporal Trajectory Encoder~(TempEnc) that constructs per-location temporal trajectories for dedicated land cover evolution modeling, and we introduce a Coarse-to-Fine Token Compressor~(C2FComp) that adaptively preserves dynamic regions while compressing the static background. To support training, we also construct ChronoInstruct, a 104K-sample instruction-tuning dataset spanning all competency levels for training. GeoChrono achieves state-of-the-art performance on ChronoBench, surpassing the leading commercial MLLMs by over 20%, while C2FComp reduces visual tokens by over 56% while retaining GeoChrono's 94.6% performance. The code and data will be available at https://github.com/IntelliSensing/GeoChrono
Primary: Beijing University of Posts and Telecommunications
All Institutions: Beijing University of Posts and Telecommunications, Tsinghua University, Hunan Normal University, City University of Hong Kong
This work has significant broader impact across several domains: 1. **Environmental Monitoring and Climate Change**: Improved long-term temporal understanding of Earth's surface evolution can greatly enhance monitoring of urban expansion, deforestation, agricultural changes, water body dynamics, and the impacts of climate change, providing crucial data for policy-making and conservation efforts. 2. **Urban Planning and Development**: The ability to track and reason about urban development over extended periods can inform sustainable urban planning, infrastructure development, and resource management. 3. **Disaster Management and Recovery**: Better understanding of pre- and post-disaster land cover changes can aid in damage assessment, recovery planning, and resilience building. 4. **Scientific Research**: ChronoBench provides a foundational benchmark and a structured evaluation framework that can accelerate research in spatio-temporal AI for Earth observation, guiding the development of more capable and intelligent models. The diagnostic insights into MLLM limitations (e.g., Long-Term Memory) are valuable for the broader ML community. 5. **Technological Advancement**: The proposed GeoChrono model and its components (TempEnc, C2FComp) offer practical solutions for processing high-resolution, long-temporal remote sensing data, pushing the boundaries of what's computationally feasible and achievable with MLLMs in this domain. The paper's contributions are overwhelmingly positive, enabling more accurate and comprehensive understanding of our planet's dynamics with no apparent negative ethical implications. This paper introduces ChronoBench, a novel, multi-dimensional benchmark for long-term temporal understanding in remote sensing, and GeoChrono, a specialized Multimodal Large Language Model that leverages unique physical priors and efficient token compression to achieve state-of-the-art performance. The work provides a crucial diagnostic framework for evaluating MLLMs' cognitive competencies in Earth observation, identifies Long-Term Memory as a critical bottleneck, and offers a highly effective and efficient architectural solution, significantly advancing the field of spatio-temporal AI for remote sensing.
The methodology proposed in "GeoChrono" is exceptionally well-conceived and executed, addressing a critical gap in long-term temporal understanding within remote sensing. The paper's strength lies in its comprehensive approach: a diagnostic benchmark, a large-scale training dataset, and a specialized model architecture. ChronoBench is a standout contribution. Its decomposition of long-term temporal understanding into four progressive cognitive levels (Land Cover Perception, Temporal Recognition, Long-Term Memory, Spatio-Temporal Reasoning) is highly insightful and provides a structured framework for evaluation that moves beyond task-specific metrics. The rule-based construction pipeline, leveraging human-annotated semantic change masks and undergoing a two-stage human quality control, ensures high data quality and reproducibility. The use of both bounding box and geographic coordinate grounding for tasks further enhances its robustness. ChronoInstruct, the 104K-sample instruction-tuning dataset, is a necessary complement to the benchmark, providing ample data for training models across the defined competency levels. The use of Gemini-3-Flash for initial free-form response generation, followed by human quality control, is a pragmatic approach to scale dataset creation while maintaining quality. GeoChrono's architecture is cleverly designed to leverage the unique physical priors of remote sensing imagery. The Temporal Trajectory Encoder (TempEnc) explicitly models per-location temporal evolution by decoupling the spatio-temporal feature volume into 1D temporal trajectories. The hybrid bidirectional-causal attention effectively captures both global change patterns and chronological progression, while semantic focusing uses text guidance to filter irrelevant information. This design directly addresses the "geostationary prior" where geographic parcels remain fixed while their semantics evolve. The Coarse-to-Fine Token Compressor (C2FComp) is another innovative and highly practical component. It tackles the computational burden of high-resolution, long-temporal sequences by exploiting the "change sparsity" inherent in remote sensing. The prompt-aware scoring mechanism to selectively preserve fine-grained tokens for dynamic, task-relevant regions while compressing static backgrounds is a sophisticated solution to a common scalability challenge. The use of Gumbel-Softmax for differentiable top-K selection is a standard but effective technique. Overall, the methodology is robust, well-motivated by domain-specific characteristics, and technically sound, demonstrating a deep understanding of both remote sensing and large language model challenges.
The experimental evaluation is comprehensive and rigorous, providing strong evidence for the paper's claims. 1. **Benchmarking**: The paper conducts extensive evaluations on ChronoBench, comparing GeoChrono against a wide array of models, including commercial MLLMs (Gemini-3-Flash, GPT-5.4, Seed-1.6-Vision), open-source general MLLMs (InternVL-3.5, Qwen3-VL series), and remote sensing domain models (TEOChat, EarthDial, DVLChat). Crucially, a human-level baseline (three domain experts) is established, revealing a substantial human-machine gap, particularly in Long-Term Memory tasks. This diagnostic insight is highly valuable for guiding future research. 2. **GeoChrono Performance**: GeoChrono achieves state-of-the-art performance on ChronoBench, with an overall accuracy of 78.34%, significantly surpassing the leading commercial MLLMs by over 20%. This is a remarkable improvement, demonstrating the effectiveness of the proposed architecture and training strategy. The detailed breakdown across the four cognitive levels and 12 sub-tasks clearly shows where GeoChrono excels and where challenges remain. 3. **Generalizability**: The evaluation extends to two external benchmarks, DVL-Bench and TEOChatlas, where GeoChrono also achieves state-of-the-art results. This demonstrates the model's strong generalizability beyond its primary training and evaluation dataset. 4. **Efficiency of C2FComp**: A dedicated experiment evaluates C2FComp's effectiveness. It shows that C2FComp can reduce visual tokens by over 56% while retaining 94.6% of GeoChrono's full performance. This is a significant practical achievement, making the processing of high-resolution, long-temporal data more feasible. 5. **Ablation Studies**: Thorough ablation studies are conducted to validate the contribution of each proposed component. The results clearly show the incremental benefits of hybrid temporal attention, semantic focusing, and C2FComp, confirming their individual importance to GeoChrono's overall performance. The experiments are well-designed, the results are clearly presented, and the analysis provides deep insights into the current state of long-term temporal understanding in remote sensing.
The paper demonstrates a strong commitment to reproducibility. 1. **Code and Data Availability**: The authors explicitly state that "The code and data will be available at https://github.com/IntelliSensing/GeoChrono". This is a crucial step for reproducibility. 2. **Detailed Data Construction**: The appendix provides extensive details on the ChronoBench and ChronoInstruct data construction pipeline, including mask encoding schemes, notation, QA metadata extraction algorithms (pseudocode for class-level and object-level tasks), and comprehensive prompt templates for all 12 sub-tasks. This level of detail is exemplary and allows for independent verification and replication of the benchmark. 3. **Implementation Settings**: Key implementation details for GeoChrono are provided, such as the base MLLM (Qwen3-VL-4B-Instruct), frozen/fine-tuned components, LoRA tuning, single-layer attention architecture for TempEnc and C2FComp, and GPU configuration (4 NVIDIA H100 80GB GPUs). While full hyperparameters are relegated to the supplementary material, the provided information is sufficient for a good understanding of the setup. Given the detailed methodology, comprehensive data construction explanation, and commitment to releasing code and data, the reproducibility of this work is expected to be high.
1. **Human-Machine Gap**: Despite GeoChrono's significant improvements, a substantial performance gap still exists between the model and human experts, particularly in Spatio-Temporal Reasoning tasks. This indicates that there's still considerable room for improvement in higher-level cognitive abilities. 2. **Geographic Scope**: ChronoBench focuses on 39 major U.S. cities. While diverse, the generalizability of the benchmark and the model's performance to other geographic regions with different land cover types, urbanization patterns, or environmental dynamics (e.g., deforestation, desertification, natural disasters) might need further validation. 3. **Base MLLM Dependency**: GeoChrono is built upon Qwen3-VL-4B-Instruct. While effective, its performance might be influenced by the capabilities and limitations of the chosen base MLLM. Exploring its integration with other powerful MLLMs could yield further insights. 4. **C2FComp Hyperparameter**: The C2FComp relies on a hyperparameter `K` (top-K blocks). While the paper shows robustness across varying selection ratios, the optimal `K` might be task-dependent or sensitive to specific scene characteristics. 5. **"Geostationary Prior" Applicability**: The TempEnc heavily relies on the "geostationary prior" (spatially fixed parcels). While highly relevant for satellite remote sensing, this prior might not hold for all types of remote sensing data, such as drone imagery with varying viewpoints or aerial photography from different flight paths.
This work has significant broader impact across several domains: 1. **Environmental Monitoring and Climate Change**: Improved long-term temporal understanding of Earth's surface evolution can greatly enhance monitoring of urban expansion, deforestation, agricultural changes, water body dynamics, and the impacts of climate change, providing crucial data for policy-making and conservation efforts. 2. **Urban Planning and Development**: The ability to track and reason about urban development over extended periods can inform sustainable urban planning, infrastructure development, and resource management. 3. **Disaster Management and Recovery**: Better understanding of pre- and post-disaster land cover changes can aid in damage assessment, recovery planning, and resilience building. 4. **Scientific Research**: ChronoBench provides a foundational benchmark and a structured evaluation framework that can accelerate research in spatio-temporal AI for Earth observation, guiding the development of more capable and intelligent models. The diagnostic insights into MLLM limitations (e.g., Long-Term Memory) are valuable for the broader ML community. 5. **Technological Advancement**: The proposed GeoChrono model and its components (TempEnc, C2FComp) offer practical solutions for processing high-resolution, long-temporal remote sensing data, pushing the boundaries of what's computationally feasible and achievable with MLLMs in this domain. The paper's contributions are overwhelmingly positive, enabling more accurate and comprehensive understanding of our planet's dynamics with no apparent negative ethical implications. This paper introduces ChronoBench, a novel, multi-dimensional benchmark for long-term temporal understanding in remote sensing, and GeoChrono, a specialized Multimodal Large Language Model that leverages unique physical priors and efficient token compression to achieve state-of-the-art performance. The work provides a crucial diagnostic framework for evaluating MLLMs' cognitive competencies in Earth observation, identifies Long-Term Memory as a critical bottleneck, and offers a highly effective and efficient architectural solution, significantly advancing the field of spatio-temporal AI for remote sensing.
Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is more important than architectural novelty.
Primary: INSAIT, Sofia University ``St. Kliment Ohridski''
All Institutions: INSAIT, Sofia University ``St. Kliment Ohridski''
[One sentence main contribution]. This paper demonstrates that a general-purpose VLM, when trained with large-scale multi-task reinforcement learning and adaptive rewards, can achieve state-of-the-art performance in remote sensing without architectural specialization, challenging the necessity of domain-specific model designs. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work provides strong empirical evidence that data scale and task diversity are the primary drivers of performance in remote sensing VLMs. By successfully applying GRTO to jointly optimize a VLM and a segmentation tool, the authors create a versatile model (MLRS) that handles a broad spectrum of tasks and input types. The rigorous evaluation across numerous ID and OOD benchmarks validates the generalization capabilities of this approach. The paper's significance lies in its paradigm-shifting conclusion: for remote sensing, a simple, scalable recipe based on data and RL may be more effective than complex, specialized architectures. This insight is valuable for both researchers and practitioners, offering a practical path forward for building general Earth observation models.
The paper proposes a "simple recipe" for Remote Sensing Vision-Language Models (RS-VLMs) that challenges the prevailing trend of architectural specialization. Instead of designing new encoders or fusion modules for specific remote sensing modalities (e.g., SAR, multi-temporal, UHR), the authors leverage a general-purpose VLM (InternVL3.5-8B) and adapt it through large-scale multi-task reinforcement learning (RL). The core methodological contribution is the use of Group Relative Tool Optimization (GRTO), which jointly optimizes the VLM policy and an external segmentation tool (SAM3) using adaptive, task-specific rewards. This allows a single model to handle diverse tasks (VQA, captioning, detection, segmentation) by deciding whether to output text or invoke a localization tool. The approach is technically sound and leverages recent advances in RLHF/RLAIF (GRPO) and tool-use, applying them effectively to the remote sensing domain. The novelty lies not in the base architecture but in the systematic demonstration that data scale and task diversity, optimized via RL, can outperform specialized architectural designs.
The experimental evaluation is comprehensive and rigorous. The authors test MLRS on a wide array of in-distribution (ID) and out-of-distribution (OOD) benchmarks, including DisasterM3, DynamicVL, SARLANG-1M, EarthReason, LaSeRS, GeoSeg-Bench, XLRS-Bench, RSHR-Bench, VLRS-Bench, UrBench, and GEOBench-VLM. This covers optical, SAR, multi-temporal, multi-view, and ultra-high-resolution inputs. The results show that MLRS achieves competitive or state-of-the-art zero-shot performance on most benchmarks, often surpassing specialized models that are fine-tuned on specific tasks. The ablation studies on data scaling, model size (2B vs 8B), and training variants (GRPO vs GRTO) provide strong empirical evidence for the paper's central hypothesis: data scale and diversity are more critical than architectural novelty for RS-VLMs. The inclusion of OOD benchmarks is particularly valuable for assessing generalization.
The paper provides sufficient detail for reproduction, including the base model (InternVL3.5-8B), the tool (SAM3), the RL framework (GRPO/GRTO), and the training mix composition. The use of open-source components (InternVL, SAM3, various benchmarks) enhances reproducibility. However, the exact curation process for the 80k training set from the 2.3M raw pool is described qualitatively ("balancing task domain and question difficulty"), which might leave some ambiguity. The hyperparameters (learning rates, batch size, KL weight) are provided. The code is not explicitly linked in the text provided, but the methodology is clear enough for implementation.
The authors acknowledge several limitations. First, the scaling experiments are limited to a single VLM family (InternVL), so it is unclear if the trends hold for other architectures (e.g., LLaVA, Qwen-VL). Second, the comparison with baselines is not always under a single controlled environment, as some baselines are fine-tuned on specific benchmarks while MLRS is evaluated zero-shot. Third, the diversity of training data is proxied by the number of source datasets, which is a coarse measure. Finally, the model struggles with multi-temporal tasks, attributed to a lack of diverse training data for that specific domain, highlighting that data availability remains a bottleneck for certain capabilities.
This paper has significant implications for the field of remote sensing and VLMs. It shifts the paradigm from "architecture engineering" to "data and task scaling," suggesting that practitioners should prioritize collecting diverse, high-quality multi-task data over designing complex, domain-specific architectures. This could lead to more generalizable and robust Earth observation models. The use of RL for tool-use in remote sensing also opens new avenues for interactive and reasoning-based geospatial analysis. The findings are likely to influence future research directions in RS-VLMs, encouraging a more data-centric approach. [One sentence main contribution]. This paper demonstrates that a general-purpose VLM, when trained with large-scale multi-task reinforcement learning and adaptive rewards, can achieve state-of-the-art performance in remote sensing without architectural specialization, challenging the necessity of domain-specific model designs. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The work provides strong empirical evidence that data scale and task diversity are the primary drivers of performance in remote sensing VLMs. By successfully applying GRTO to jointly optimize a VLM and a segmentation tool, the authors create a versatile model (MLRS) that handles a broad spectrum of tasks and input types. The rigorous evaluation across numerous ID and OOD benchmarks validates the generalization capabilities of this approach. The paper's significance lies in its paradigm-shifting conclusion: for remote sensing, a simple, scalable recipe based on data and RL may be more effective than complex, specialized architectures. This insight is valuable for both researchers and practitioners, offering a practical path forward for building general Earth observation models.
3D Gaussian Splatting (3DGS) captures scenes by coupling explicit geometry (position, covariance) with view-dependent photometry (Spherical Harmonics). However, building $\mathrm{SE}(3)$-equivariant architectures on these primitives presents a fundamental representation bottleneck. Color has been treated as a signal rather than a geometric entity, making it nontrivial to unify symmetry across geometry and appearance as the camera frame changes. While translations are handled by relative coordinates, rotations act heterogeneously across attributes: $ÎĽ\mapsto RÎĽ$, $ÎŁ\mapsto RÎŁR^\top$, and $f_\ell\mapsto D^\ell(R)f_\ell$. This mismatch complicates strict equivariance, leading existing methods to either discard or flatten SH coefficients, thereby breaking symmetry. We propose a unified solution rooted in representation theory: for SH degrees $\ell\le2$, photometry is algebraically isomorphic to a rank-2 geometric tensor. We prove that the Wigner-$D$ action on these SH coefficients can be exactly reformulated as the conjugation action on $3\times3$ matrices. Leveraging this, we introduce the Unified Matrix Embedding, a lifting that maps all Gaussian attributes into a unified carrier space, $\mathfrak{gl}(3)$. Building on the "Color-as-Geometry" formulation, we present E3DGS, a rigid-body ($\mathrm{SE}(3)$) equivariant architecture that processes 3D Gaussians without Clebsch-Gordan tensor products. Evaluations on object vision and action-conditioned Gaussian world modeling demonstrate that our unified approach yields strong robustness under camera-frame changes and improved data efficiency.
Primary: University of Michigan, Ann Arbor
All Institutions: University of Michigan, Ann Arbor
The paper presents a theoretically sound and empirically validated method for achieving strict SE(3) equivariance in 3D Gaussian Splatting by unifying geometry and photometry into a matrix carrier, offering a significant advancement in robust 3D representation learning.
The paper proposes a theoretically grounded method to unify geometric and photometric representations in 3D Gaussian Splatting (3DGS) by leveraging representation theory. The core innovation is the "Color-as-Geometry" embedding, which maps Spherical Harmonic (SH) coefficients (degrees 0-2) into a $3\times3$ matrix carrier ($\mathfrak{gl}(3)$) such that the Wigner-D rotation action on SHs is isomorphic to matrix conjugation. This allows the use of ReLU/ReLN-style equivariant operations (specifically Reductive Lie Neurons) on a unified carrier space, avoiding the computational bottleneck of Clebsch-Gordan tensor products required by traditional Tensor Field Networks. The methodology is mathematically rigorous, providing explicit intertwiners for the mapping and proving the equivariance of the resulting architecture (E3DGS). The approach is elegant in its unification of disparate transformation rules (position, covariance, SH) into a single adjoint action.
The authors evaluate E3DGS on two distinct tasks: object recognition (using ShapeSplat/ModelNet10/40) and action-conditioned world modeling for robotic manipulation (using RLBench/ManiGaussian). The experiments demonstrate that E3DGS achieves superior zero-shot robustness to camera-frame changes compared to non-equivariant baselines (which degrade significantly under rotation). In the manipulation task, E3DGS shows improved data efficiency and competitive success rates against state-of-the-art methods like PerAct and GNFactor. The ablation studies on SH degrees confirm that including higher-order photometry (up to degree 2) within the equivariant framework is crucial for performance, whereas naive concatenation breaks symmetry. The results are consistent and support the claim that equivariance provides a strong inductive bias for 3D learning.
The paper provides detailed mathematical derivations, including the explicit construction of the intertwiners and the basis conventions for real SH coefficients. The architecture components (ReLN-Attention, ReLN-LayerNorm) are described with sufficient detail for implementation. However, as an arXiv preprint, the code is not explicitly linked in the text provided (though often available in supplementary materials or separate repos, none are cited here). The reliance on specific SH conventions (real vs. complex, normalization) is addressed, which aids reproducibility. The experimental setup follows established benchmarks, facilitating comparison.
The current formulation is limited to SH degrees 0-2, which corresponds to the capacity of the $3\times3$ carrier. While the authors prove that higher degrees can be handled by larger carriers ($V_k$), the empirical validation is restricted to the $3\times3$ case. The computational cost, while avoiding CG products, still involves matrix operations on $3\times3$ matrices, which may be heavier than scalar operations for simple tasks, though the paper argues for parameter efficiency. The method assumes rigid-body transformations; dynamic non-rigid deformations would require extensions. The improvement in manipulation tasks, while statistically significant in average success rate, shows variability across specific tasks (e.g., lower performance on "close jar" compared to baselines), suggesting the equivariant prior might not always align perfectly with the specific dynamics of every task.
This work bridges a significant gap in geometric deep learning by providing a practical, efficient way to incorporate view-dependent appearance into equivariant 3D models. This has broad implications for robotics (where lighting and material properties are crucial for manipulation), computer vision (robust object recognition), and generative modeling (view-consistent synthesis). By avoiding expensive tensor products, it makes strict SE(3) equivariance more accessible for large-scale 3D Gaussian scenes. The paper presents a theoretically sound and empirically validated method for achieving strict SE(3) equivariance in 3D Gaussian Splatting by unifying geometry and photometry into a matrix carrier, offering a significant advancement in robust 3D representation learning.
A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors. When sampling fails, a common fix shows the generator the gold answer and asks it to write a chain that reaches that answer. We show that this second step degrades the training data in a way that correctness filtering cannot catch. We run a controlled experiment that fixes the generator, the problem set, and the correctness filter, and varies only whether the chain is generated under answer-conditioning, the gold answer shown with a request to reach it. Training a strong instruction-tuned reasoning model on its own answer-conditioned chains sharply lowers its verifiable-reasoning accuracy. The loss grows with difficulty, reaching as much as about 27 points on the hardest competition problems. The mechanism is legible in the chains themselves, which rationalize backward from the shown answer instead of deriving it, with the early final-answer statement as the measurable symptom. The harm is a property of the data rather than the generator, read off unlabeled generations before any fine-tuning, ordering the penalty across eight thinking models from four families, and transferring across teacher families. A prompt ablation localizes it to the rationalize-toward instruction rather than the answer's bare visibility. The practical takeaway is to generate answer-blind, because no correctness filter can see this damage in the data.
Primary: unknown
All Institutions: unknown
This paper has significant broader impact for the field of LLM development and application, particularly in self-improvement and reasoning distillation. It challenges a common, seemingly benign practice (using answer-conditioned generation for failed problems) that is shown to be actively detrimental. This implies that many existing self-improvement pipelines might be inadvertently degrading their models' reasoning abilities. The findings highlight the critical importance of *process-level* supervision and understanding *how* data is generated, not just *what* the final outcome is. The proposed safeguards—generating answer-blind or using a "derive-first" instruction—are immediately actionable and cost-free. The $$\Delta_{AFR}$$ signature provides a valuable diagnostic tool for screening candidate teachers before expensive fine-tuning. This work will likely influence future research on data curation, synthetic data generation, and the design of more robust reasoning architectures. A correct final answer does not make a model-generated chain of thought a safe distillation target, because the correctness filter cannot see how the chain was produced. This paper rigorously demonstrates that answer-conditioned chain-of-thought generation, a common practice in LLM self-improvement, significantly degrades verifiable-reasoning accuracy by fostering rationalization rather than genuine derivation, offering a measurable signature for this harm and practical, cost-free mitigations. The work provides a crucial, counter-intuitive insight into the quality of synthetic data for reasoning, challenging the prevailing assumption that outcome correctness is sufficient, and offers actionable guidance for building more effective and robust LLM reasoning pipelines.
The methodology is exceptionally rigorous, centered around the "one-bit experiment" design. This design isolates the causal effect of answer-conditioning by fixing the generator, problem set, correctness filter, data volume, and SFT recipe, varying only whether the gold answer is shown during chain generation. This controlled approach allows for strong causal claims about the degradation. The paper introduces a measurable symptom, the "answer-first rate" ($$\Delta_{AFR}$$), which quantifies how often the gold answer is stated early in the chain. This metric is used both for mechanism analysis within a corpus (the "carrier experiment") and as a pre-training predictor across models. The prompt ablation study effectively disentangles the effect of answer visibility from the instruction to rationalize. The use of difference-in-differences (DiD) in the carrier experiment robustly controls for problem-subset effects. The methodology is well-designed to provide clear, actionable insights into a subtle but critical problem in LLM reasoning distillation.
The experimental evaluation is comprehensive and robust. The core finding of a significant accuracy drop (16.2 points on MATH-500, up to 27.2 points on AIME) due to answer-conditioned generation is demonstrated across multiple math benchmarks (GSM8K, Minerva, AIME) and even transfers to code generation (MBPP+, HumanEval+), where chains passing all tests still harm the student. Crucially, the effect is shown to be absent in multiple-choice knowledge tasks, aligning with the proposed mechanism that it affects derivation-based reasoning. The experiments involve eight thinking models from four distinct families (Qwen, DeepSeek, Llama-Nemotron, GLM), demonstrating the generality of the finding. The prediction of the penalty using the $$\Delta_{AFR}$$ signature before training, with a low mean absolute error of 2.9 points, is a powerful result. The length-matched controls effectively rule out chain length as the sole explanatory factor, reinforcing that the content and generation process are the culprits. Statistical significance is consistently reported using paired bootstrap and McNemar's test, adding to the credibility of the results.
The paper demonstrates a strong commitment to reproducibility. It explicitly states that code and data are released to regenerate every reported number. This includes exact generation prompts, matched problem identifiers, generated chains, metadata, full training configurations, and the evaluation harness. Correctness-critical components like the symbolic answer checker and answer-position matcher have pinned dependencies. A manifest maps every benchmark, condition, and seed cell to its evaluation record, ensuring traceability. The held-out penalty predictions were recorded *before* training, and the released code recomputes the cross-model fit, further enhancing trust. This level of detail and transparency is exemplary.
The paper acknowledges several limitations. The "one-bit experiment" primarily uses 935 matched math problems on one architecture family at a time, which, while controlled, limits the immediate generalization to all domains or problem types. The content-versus-length attribution is focused on MATH-500. The cross-model fit is based on a relatively small sample of eight models, and while robust, the Fisher-z interval on 'r' is wide. Some experiments, such as the code domain and student-output transfer, are not length-controlled, though other length-matched controls support the core claim. The single-seed rows are noted to support consistent sign rather than exact magnitude.
This paper has significant broader impact for the field of LLM development and application, particularly in self-improvement and reasoning distillation. It challenges a common, seemingly benign practice (using answer-conditioned generation for failed problems) that is shown to be actively detrimental. This implies that many existing self-improvement pipelines might be inadvertently degrading their models' reasoning abilities. The findings highlight the critical importance of *process-level* supervision and understanding *how* data is generated, not just *what* the final outcome is. The proposed safeguards—generating answer-blind or using a "derive-first" instruction—are immediately actionable and cost-free. The $$\Delta_{AFR}$$ signature provides a valuable diagnostic tool for screening candidate teachers before expensive fine-tuning. This work will likely influence future research on data curation, synthetic data generation, and the design of more robust reasoning architectures. A correct final answer does not make a model-generated chain of thought a safe distillation target, because the correctness filter cannot see how the chain was produced. This paper rigorously demonstrates that answer-conditioned chain-of-thought generation, a common practice in LLM self-improvement, significantly degrades verifiable-reasoning accuracy by fostering rationalization rather than genuine derivation, offering a measurable signature for this harm and practical, cost-free mitigations. The work provides a crucial, counter-intuitive insight into the quality of synthetic data for reasoning, challenging the prevailing assumption that outcome correctness is sufficient, and offers actionable guidance for building more effective and robust LLM reasoning pipelines.
Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.
Primary: Unknown
All Institutions: Unknown
Kaleido presents a compelling algorithm-hardware co-design that significantly accelerates Video Diffusion Transformers by exploiting latent space correlations, offering substantial speedups and energy savings while maintaining high generative quality, thereby addressing a critical bottleneck in the deployment of advanced video generation models.
The paper proposes "Kaleido," an algorithm-hardware co-design specifically targeting Video Diffusion Transformers (vDiTs). The core algorithmic insight is the exploitation of channel-wise spatiotemporal correlations in the latent space to enable computation skipping (reusing partial results). This is a significant methodological shift from standard sparse attention techniques used in LLMs, which do not account for the unique redundancy in video generation tasks. The hardware design features a systolic array-like accelerator with reconfigurable processing elements and a lightweight data dispatcher to handle the irregular sparsity introduced by the algorithm. The approach is technically sound, addressing the dominant bottleneck of self-attention in long-timestep diffusion processes. However, the novelty is somewhat incremental in the broader context of hardware acceleration, as algorithm-hardware co-design for transformers is a well-trodden path, though the specific application to vDiTs with this correlation-based reuse is a strong contribution.
The evaluation demonstrates up to 5.9x speedup and 16.0x energy savings compared to state-of-the-art accelerators across three mainstream vDiT models. The claim of preserving higher generative quality (>17 dB improvement in PSNR/SSIM relative to prior skipping methods) is a strong empirical result. The use of standard metrics and comparison against SOTA baselines provides credibility. However, the lack of specific details on the baseline models and the exact nature of the "state of the art" accelerators in the abstract limits the immediate assessability of the magnitude of the improvement. The results are promising and suggest significant practical utility for deploying vDiTs.
The paper is published on arXiv, and while the abstract provides a high-level overview, the full text would need to be scrutinized for hyperparameters, model configurations, and hardware specifications to ensure reproducibility. The mention of "lightweight" components suggests a focus on practical implementation, which often aids reproducibility if code is released. Without code availability explicitly stated in the abstract, reproducibility is assumed to be moderate to high based on the detailed algorithmic description implied.
The primary limitation is the reliance on the assumption that channel-wise spatiotemporal correlations are consistent across different vDiT architectures and video domains. The performance gain might vary depending on the specific video content and the complexity of the diffusion process. Additionally, the hardware design's flexibility (reconfigurable PEs) might come with overheads that are not fully detailed in the abstract. The comparison is against "state of the art accelerators," which may include specialized custom hardware that is not widely available, potentially skewing the perceived advantage.
This work has significant implications for the democratization of high-quality video generation by reducing the computational and energy barriers. It enables faster iteration for researchers and more accessible deployment for developers. The focus on energy efficiency also aligns with the growing need for sustainable AI. By addressing the specific bottlenecks of vDiTs, it paves the way for more complex and longer video generation tasks. Kaleido presents a compelling algorithm-hardware co-design that significantly accelerates Video Diffusion Transformers by exploiting latent space correlations, offering substantial speedups and energy savings while maintaining high generative quality, thereby addressing a critical bottleneck in the deployment of advanced video generation models.