Last 7 Days (July 11 – July 17, 2026)
Robotic manipulation is inherently multi-frame: local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame. However, modern diffusion-based visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame. We propose Mixture of Frames Policy (MoF), a diffusion policy that performs synchronized action denoising across multiple coordinate frames. MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame. To make this possible for intermediate noisy diffusion states, we introduce a column-based 6D rotation representation within an SE(3) action parameterization that supports exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold. Across nine simulated bimanual manipulation tasks, we show that the best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines. We further evaluate MoF on two real-world bimanual mobile manipulation tasks, demonstrating that it outperforms all constituent single-frame baselines. Project homepage: https://mofpo.github.io
Primary: Stanford University
All Institutions: Stanford University, Apple, Toyota Research Institute
The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
The paper proposes a novel architecture for diffusion-based visuomotor policies called Mixture of Frames (MoF). The core insight is that robotic manipulation actions are naturally represented in different coordinate frames at different stages of a task (e.g., end-effector for grasping, base for transport). Standard diffusion policies fix a single action frame, forcing the denoiser to model complex, frame-dependent distributions. MoF maintains a single canonical diffusion state but applies frame-specialized denoisers in their respective frames, fusing the noise predictions back into the canonical frame. A critical technical contribution is the introduction of a column-based 6D rotation representation for SE(3) actions. This allows for exact, differentiable frame transformations of *noisy* intermediate states without requiring projection onto the SO(3) manifold, which is non-linear and lossy. This mathematical insight is elegant and solves a significant implementation hurdle for multi-frame reasoning in diffusion policies. The method is generalizable to any diffusion policy and can be implemented as a drop-in replacement or extension.
The evaluation is comprehensive and rigorous. The authors first establish the motivation by showing that action frame choice has a substantial, task-dependent impact on performance in bimanual mobile manipulation, with a 15% gap between the best and worst single frames. They then demonstrate that MoF outperforms oracle frame selection (which knows the best frame per task) and standard Mixture-of-Experts (MoE) baselines that do not operate in multiple action frames. The ablation studies are particularly strong, showing that the proposed column-based representation is essential (orthogonalization leads to collapse in ensemble variants) and that the router effectively switches between frames based on task phase. Real-world experiments on two complex bimanual mobile manipulation tasks (Pouring, Serving) further validate the approach, showing significant improvements over single-frame baselines. The use of simulated benchmarks (BiGym, DexMimicGen) and real-world validation provides a robust evidence base.
The paper provides detailed descriptions of the method, including the specific rotation representation and the fusion mechanism. The project homepage link suggests code availability. The experimental setup is well-described, including dataset sizes, training epochs, and evaluation metrics. The ablation studies provide clear evidence of component contributions. The reliance on standard simulators (BiGym) and real-world hardware (HoMMI) enhances reproducibility.
The method relies on a designer-specified set of candidate frames. While the router learns to weight them, the set must be predefined. The paper acknowledges this and suggests future work on learning or discovering frames. Additionally, the router is supervised via denoising loss, which may not perfectly align with policy performance, though the results suggest it works well in practice. The current experiments focus on smaller diffusion policies; scaling to large VLA models is noted as future work.
This work has significant implications for robotic manipulation, particularly in complex, multi-stage tasks where different coordinate systems are advantageous. By improving the robustness and performance of visuomotor policies, it contributes to the broader goal of autonomous robots capable of operating in unstructured environments. The technical contribution regarding SE(3) transformations is also valuable for the broader machine learning community working with geometric deep learning and diffusion models. The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.
Primary: New York University
All Institutions: Carnegie Mellon University, New York University
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
The paper introduces "requential coding," a novel and highly efficient method for compressing generative models. It builds upon the principles of prequential coding and relative entropy coding (REC) but addresses their fundamental limitations. Unlike parameter-based methods (e.g., quantization) that scale with model size regardless of information stored, or prequential coding that scales with dataset size and entropy, requential coding decouples code length from both. The core idea is that a "student" model generates candidate training samples from its own distribution, and a stronger "teacher" model selects one of these candidates. The student's code then records only the index of the selected sample, which costs approximately $KL(Q_t\|P_t)$ bits, where $Q_t$ is the teacher and $P_t$ is the student. This effectively distills the teacher's knowledge into the student by efficiently transmitting only the "disagreement" between them. The methodology is rigorously defined, including the encoder and decoder protocols, and a formal bound for the expected code length is derived in the appendix, showing it scales with the cumulative teacher-student KL divergence. Practical improvements like "teacher smoothing" (using an EMA of teacher checkpoints) and "iso-loss projection" (periodically resetting the teacher to the student's performance level) are introduced to further reduce the code length without sacrificing performance. The paper also provides a detailed analysis of the runtime for code length evaluation, encoding, and decoding, acknowledging the computational cost of actual transmission (which can be high due to REC's proposal generation) but emphasizing that evaluation is much more efficient. The use of a universal integer code (Elias delta) for the index ensures the decoder doesn't need prior knowledge of the KL divergence. This is a sophisticated and well-thought-out approach that significantly advances the state of the art in model compression theory.
The experimental evaluation is comprehensive and compelling, demonstrating the power of requential coding across various tasks and scales. 1. **Compression Benchmarking**: Requential coding is benchmarked against prequential coding and idealized 4-bit post-training quantization (PTQ) on autoregressive transformers (100M parameters) trained on OpenWebText, CIFAR-5M, and FineWeb. It achieves one to two orders of magnitude shorter codes than prequential coding, dominating the Pareto frontier of loss vs. code length. The prequential heuristic (a non-rigorous estimate) also falls well above requential coding. 2. **Scaling Insights**: * **Model Size**: Experiments show that larger models (from 10M to 1B parameters) are *more compressible* at a fixed loss, despite having more parameters. This is a crucial finding, as it suggests larger models learn simpler functions relative to their capacity, which parameter-based methods fail to capture. * **Ensembles**: Larger ensembles are also shown to be more compressible, reaching lower loss at a fixed code budget. 3. **Generalization Guarantees (PAC-Bayes Bounds)**: This is a highlight. Plugging the requential code length into a PAC-Bayes bound yields state-of-the-art generalization guarantees for billion-parameter LLMs. These bounds: * Improve with model scale, mirroring the true test loss, for models trained on a fixed amount of data. * Outperform even *idealized lossless 4-bit PTQ bounds* in the compute-optimal scaling regime ($D=20N$), a significant breakthrough as previous non-vacuous bounds for LLMs were based on realistic PTQ. * Tighten with scale, as the code length per parameter decays as a power law, predicting a vanishing generalization gap. 4. **Overfitting Prediction**: The requential code successfully predicts gradual overfitting under data repetition, showing the divergence between training and test loss as the complexity penalty rises. 5. **Learnable Information Content**: The method effectively isolates learnable structure from random information in datasets, revealing that lower-entropy text holds more learnable structure than higher-entropy image data, and distinguishing it from random or trivially repeating strings. This provides a principled way to rank datasets by their "epiplexity." The experiments are well-designed, the results are clearly presented (e.g., Figure 3, 4, 5), and the conclusions drawn are profound, offering new insights into deep learning phenomena that were previously inaccessible to rigorous measurement.
The paper explicitly states that "Code available at https://github.com/shikaiqiu/requential-coding". The methodology is described with sufficient detail, including pseudocode for REC and descriptions of teacher smoothing and iso-loss projection. The appendix provides full experiment details, including how code lengths are computed, datasets used, and per-figure configurations. This commitment to open-sourcing the code and providing detailed experimental setups significantly enhances reproducibility.
The authors acknowledge several limitations: 1. **Lossy Compression of Teacher**: Requential coding provides a lossless compression for the student model, which is a distillation of the teacher. If the goal is to compress a pre-trained model not trained via distillation, it becomes a lossy compression of that original model. 2. **Encoding Runtime**: While code length *evaluation* is efficient, *actually transmitting* a model via requential coding can be prohibitively slow due to the exponential scaling of REC encoding time with KL divergence. The paper primarily presents it as a tool for *evaluation* and *understanding* rather than practical transmission. 3. **Information Forgetting**: The current code length only grows with training steps and doesn't account for information that might be learned early but subsequently forgotten. Leveraging forgotten information could potentially further reduce the code length. 4. **Teacher Optimization**: The choice of teacher sequence is simple, and further gains are expected from optimizing it.
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
Existing robotic perception is constrained by sensors that are either robot-mounted or permanently fixed in the environment, locking perception to a limited set of viewpoints. Yet as robots perform increasingly diverse tasks, the most informative viewpoint shifts from one task to the next-often somewhere onboard sensor and static infrastructure can not readily satisfy. To address this gap, we propose SensorPerch, a novel realization of active perception that decouples sensing from both the robot embodiment and the environment by treating sensors as independent physical entities that the robot can autonomously detach and re-attach within the environment. SensorPerch presents one realization of this paradigm: a lightweight, wireless, reconfigurable sensor platform that can perch on diverse surfaces, paired with a viewpoint-selection framework that determines task-optimal sensor placements. Together, these enable robots to construct task-relevant viewpoints on demand, independent of the robot's current position and available fixed infrastructure. We demonstrate the paradigm on two task classes: (i) object-coupled perception, where SensorPerch enables persistent object-state detection beyond the robot's current position, achieving successful event detection even when the robot is not nearby; and (ii) policy-coupled perception, where SensorPerch allows robots to construct diverse, policy-specific viewpoints for various policies, achieving success rates comparable to those obtained using oracle viewpoints.
Primary: Cornell University
All Institutions: Cornell University
SensorPerch represents a significant step towards more versatile and autonomous robotic systems. By decoupling perception from fixed infrastructure and robot embodiment, it enables robots to adapt their sensing capabilities dynamically to task requirements, leading to several broader impacts: 1. **Enhanced Robot Autonomy:** Robots can operate more independently in diverse, unstructured environments without relying on pre-installed cameras or human intervention for viewpoint adjustment. 2. **Improved Task Performance:** The ability to construct task-optimal viewpoints on demand can lead to higher success rates in complex manipulation, monitoring, and interaction tasks, especially in scenarios requiring persistent observation or specific perspectives. 3. **New Active Perception Paradigm:** This work establishes a new paradigm for active perception, shifting from merely adjusting onboard sensors to physically reconfiguring the sensing infrastructure itself. This could inspire future research into modular, reconfigurable robotic components beyond just sensors. 4. **Applications in Diverse Fields:** Beyond household robotics, this technology could be transformative for applications in industrial inspection, disaster response (deploying sensors in hazardous areas), construction, and even scientific exploration, where dynamic, adaptive sensing is crucial. 5. **Reduced Infrastructure Costs:** By allowing a single robot to deploy and manage its own sensing network, it could reduce the need for expensive, permanently installed camera systems or multi-robot coordination for comprehensive coverage. SensorPerch introduces a novel paradigm for active perception by treating sensors as independent, reconfigurable physical entities that robots can autonomously deploy and manage. This comprehensive system, integrating robust hardware with an intelligent software framework for task-conditioned viewpoint selection, significantly enhances robotic perception capabilities, enabling persistent object-state detection and policy-aligned viewpoints in diverse real-world tasks, thereby pushing the boundaries of robot autonomy and adaptability.
The methodology proposed in SensorPerch is a comprehensive and well-integrated system that addresses a critical limitation in robotic perception. The core idea of decoupling sensing from the robot's embodiment and fixed infrastructure by treating sensors as autonomous, reconfigurable physical entities is genuinely novel. The system comprises both robust hardware and an intelligent software framework. The hardware design is modular, lightweight (201.4g), and self-contained, featuring an onboard Raspberry Pi 4B for computation, a 2-DoF motorized gimbal for viewpoint adjustment, a compact battery for independent operation (2.25 hours), and a vacuum-based attachment mechanism for diverse surfaces. The integration of a charging dock on the robot's mobile base for autonomous recharging is a practical and essential detail for long-term operation. The cost-effectiveness ($94) is also a significant advantage. The software framework, "Where to Perch," is equally sophisticated. It leverages a hybrid scene representation combining semantic SLAM for geometric reasoning and mountable surface identification, and a radiance-field model (NeRF) for novel view synthesis. This allows for efficient evaluation of candidate viewpoints in simulation without physical deployment. Viewpoint sampling is intelligently biased towards task-relevant regions using a VLM. The task-conditioned scoring function is a flexible abstraction, instantiated for two critical use cases: object-coupled perception (using VLM queries for object state detection) and policy-coupled perception (using DINOv2 embeddings to match a policy's training distribution). Finally, the system incorporates model-based pose estimation and grasp-and-place primitives for autonomous detachment and reattachment of the sensor platforms. The overall methodology presents a complete, closed-loop system for active, reconfigurable perception.
The experimental evaluation is thorough and compelling, covering platform capabilities, task performance, and system efficiency. 1. **Platform Evaluation:** Quantitative metrics for attachment accuracy (0.42 cm, <1 degree), battery life (2.25 hours), and thermal performance demonstrate the hardware's robustness and suitability for real-world, long-horizon tasks. 2. **Task Evaluation:** This is the strongest part of the evaluation. * **Object-coupled perception:** Tested across five diverse scenarios (pot boiling, human fall, fire hazard, cup dropped, water leakage), SensorPerch consistently achieves high event detection success rates, closely matching oracle performance and significantly outperforming baselines like "wrist camera only" or "random feasible placement." This validates its ability to provide persistent, task-relevant perception beyond the robot's immediate vicinity. * **Policy-coupled perception:** Evaluated on three manipulation tasks (peeling, cutting, pouring), SensorPerch achieves success rates (80-90%) comparable to oracle viewpoints, demonstrating its capability to construct policy-specific views that align with training data distributions. This is a crucial validation for enabling robust policy execution in varied settings. * **Baselines:** The comparison against a comprehensive set of baselines (w/o 2-DoF, random, oracle, wrist-only, fixed third-person) rigorously isolates the contributions of different components and the overall system. * **Multi-platform demonstration:** The real-world demonstration with three platforms simultaneously monitoring different aspects of a breakfast preparation task (pot, cutting, human's cup) is highly impressive and showcases the system's scalability and practical utility in complex, multi-task scenarios. 3. **System Evaluation:** The latency breakdown provides valuable insights into the computational overhead. While initial scene reconstruction is time-consuming (369s), it's a one-time cost. Viewpoint sampling, synthesis, and scoring are performed in seconds (0.91s to 7.95s), and the full detach-reattach loop takes a reasonable 12.53 seconds. Streaming latency (180ms) is suitable for real-time perception, even with multiple platforms. These metrics confirm the system's practical usability.
The paper provides detailed descriptions of the hardware components, including specific models (Raspberry Pi 4B, MG90S servos, RealSense D435i, 18650 battery) and their integration. The software stack outlines the use of semantic SLAM, radiance fields (NeRF), VLM, DINOv2, and ROS. A lightweight ROS package is mentioned, which is a good step towards reproducibility. However, the absence of a public code repository or specific details on the semantic SLAM and NeRF implementation (e.g., which specific open-source projects were used, training data for NeRF, VLM model used) makes full replication challenging without further information. The experimental setup is well-described, but the exact datasets for policy training and VLM queries would be needed.
The authors acknowledge several pertinent limitations: 1. **Surface Dependence:** The vacuum-based mounting mechanism relies on the availability of flat, non-porous, rigid surfaces. This limits deployment in environments with soft, porous, or highly irregular textures (e.g., carpets, fabric, rough stone). 2. **Vacuum Reliability:** The long-term reliability of vacuum seals under various environmental conditions (dust, temperature changes, vibrations) is a potential concern, although the paper demonstrates robust attachment in its experiments. 3. **Robot Motion and Occlusions:** The current framework does not explicitly model how the robot's own motion might create or resolve occlusions, or how it affects viewpoint quality over time. This could lead to suboptimal placements if the robot itself becomes an occluder. Additional limitations could include: 4. **Scalability to Many Sensors:** While a three-sensor demo is shown, managing and coordinating a much larger fleet of reconfigurable sensors could introduce new challenges in terms of communication, power management, and optimal placement strategies. 5. **Initial Scene Reconstruction Time:** The 369-second scene reconstruction time, while a one-time cost, might be prohibitive for highly dynamic environments or those requiring frequent re-mapping.
SensorPerch represents a significant step towards more versatile and autonomous robotic systems. By decoupling perception from fixed infrastructure and robot embodiment, it enables robots to adapt their sensing capabilities dynamically to task requirements, leading to several broader impacts: 1. **Enhanced Robot Autonomy:** Robots can operate more independently in diverse, unstructured environments without relying on pre-installed cameras or human intervention for viewpoint adjustment. 2. **Improved Task Performance:** The ability to construct task-optimal viewpoints on demand can lead to higher success rates in complex manipulation, monitoring, and interaction tasks, especially in scenarios requiring persistent observation or specific perspectives. 3. **New Active Perception Paradigm:** This work establishes a new paradigm for active perception, shifting from merely adjusting onboard sensors to physically reconfiguring the sensing infrastructure itself. This could inspire future research into modular, reconfigurable robotic components beyond just sensors. 4. **Applications in Diverse Fields:** Beyond household robotics, this technology could be transformative for applications in industrial inspection, disaster response (deploying sensors in hazardous areas), construction, and even scientific exploration, where dynamic, adaptive sensing is crucial. 5. **Reduced Infrastructure Costs:** By allowing a single robot to deploy and manage its own sensing network, it could reduce the need for expensive, permanently installed camera systems or multi-robot coordination for comprehensive coverage. SensorPerch introduces a novel paradigm for active perception by treating sensors as independent, reconfigurable physical entities that robots can autonomously deploy and manage. This comprehensive system, integrating robust hardware with an intelligent software framework for task-conditioned viewpoint selection, significantly enhances robotic perception capabilities, enabling persistent object-state detection and policy-aligned viewpoints in diverse real-world tasks, thereby pushing the boundaries of robot autonomy and adaptability.
Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.
Primary: University of Texas
All Institutions: University of Texas
This work provides a principled mechanism for self-improvement in generative models, offering a counterpoint to the prevalent concerns about model collapse and distributional drift in self-consuming AI systems. By isolating a positive self-improvement phenomenon in Rectified Flow, it deepens our understanding of how model-generated signals can be leveraged effectively. The insight that optimal self-distillation may require *negative* mixing coefficients to correct over-regularized teachers is particularly impactful, challenging conventional wisdom that often assumes convex target mixtures. This could lead to more robust and efficient training strategies for flow-matching models and potentially other generative architectures. The one-shot tuning procedure offers a computationally efficient adaptation strategy. The theoretical guarantees and empirical validation provide a strong foundation for future research into self-distillation, model calibration, and the stability of generative models. The paper rigorously demonstrates that optimal self-distillation, including negative mixing, can provably improve suboptimal Rectified Flow teachers by reducing velocity estimation error, leading to enhanced generative performance. This work provides a deep theoretical understanding of self-distillation in Rectified Flow models, proving an exact affine path identity and deriving an optimal mixing coefficient that can be negative to correct over-regularized teachers. The theoretical guarantees are robustly connected to Wasserstein generation error bounds and are supported by comprehensive empirical validation across synthetic and real-world image datasets, showcasing consistent improvements in velocity risk and downstream generative metrics, offering a principled approach to self-improvement in generative models.
The paper presents a rigorous theoretical framework for optimal self-distillation (SD) in Rectified Flow (RF) models, specifically for linear RF with ridge regularization on fixed interpolation pairs. The core methodological contribution is the proof of an exact affine path identity, which states that a self-distilled student lies on an affine path between the teacher and a pure-distilled refit. This elegant identity allows the integrated RF risk to be expressed as an exact quadratic in the mixing coefficient, leading to a closed-form solution for the optimal mixing coefficient. A key insight is the "sign rule," which dictates that positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized ones, a departure from conventional convex distillation. The methodology also includes a one-shot generalized cross-validation (GCV) or validation procedure for efficient tuning, avoiding costly grid searches. Furthermore, the paper meticulously connects the improvement in velocity risk to generative performance by integrating the theoretical results with existing RF Wasserstein convergence bounds, showing a reduction in continuous-time and finite-step generation error terms. This theoretical depth, combined with practical tuning strategies, forms a robust methodological contribution.
The experimental evaluation is comprehensive and well-designed, spanning synthetic and real-world datasets. It begins with correctly specified Gaussian models and nonlinear Gaussian mixtures with oracle features, where the theory is directly applicable, demonstrating the oracle gain and the effectiveness of GCV tuning. Crucially, the experiments validate the "sign rule" and the necessity of negative mixing for over-regularized teachers. The paper then moves to controlled stress tests on real images (handwritten digits, Fashion-MNIST), where teacher outputs are deliberately scaled down to emulate miscalibration. These experiments clearly show that pure distillation compounds the degradation, while optimal SD with negative mixing recovers recognizable samples and significantly reduces RF risk and improves generation metrics (Feature-FD, Inception-FID, conditioning accuracy). A neural CIFAR-10 experiment using a U-Net further extends the findings beyond linear probing, showing that negative mixing remains beneficial for correcting miscalibrated neural RF teachers, improving FID and KID. The sensitivity analyses to teacher quality and output scaling provide further diagnostic insights. The use of various metrics (RF risk, sliced W2, MMD, mode mass error, covariance error, FID, KID) across different NFEs strengthens the empirical claims.
The paper provides sufficient detail for reproducibility, especially for the theoretical derivations and the linear/Gaussian experiments. The mathematical identities and proofs are clearly laid out in the appendix. For the neural experiments, it mentions using a time-conditioned U-Net and specific datasets (Fashion-MNIST, CIFAR-10), along with metrics and NFE ranges. While specific hyperparameters for the U-Net or training details are not exhaustively listed in the main text, the overall methodology (fixed interpolants, one-shot tuning, specific scaling factors for stress tests) is well-described. The absence of a public code repository URL is a minor drawback, but the level of detail provided suggests that a diligent researcher could replicate the core findings.
The paper explicitly acknowledges several limitations. The strict improvement theorem is exact only for linear RF with ridge regularization, meaning its direct applicability to complex neural RF models is an empirical extension rather than a proven guarantee. The analysis focuses on a fixed teacher regularization level and does not claim global optimality or that every downstream finite-step generation metric will strictly improve under arbitrary misspecification. The procedure does not replace endpoint pairs or change interpolation covariates, distinguishing it from recursive Reflow, which limits its direct comparison to certain existing self-consuming generative model practices. Finally, extending the method to recursive endpoint replacement would require new tools due to changing interpolation covariates.
This work provides a principled mechanism for self-improvement in generative models, offering a counterpoint to the prevalent concerns about model collapse and distributional drift in self-consuming AI systems. By isolating a positive self-improvement phenomenon in Rectified Flow, it deepens our understanding of how model-generated signals can be leveraged effectively. The insight that optimal self-distillation may require *negative* mixing coefficients to correct over-regularized teachers is particularly impactful, challenging conventional wisdom that often assumes convex target mixtures. This could lead to more robust and efficient training strategies for flow-matching models and potentially other generative architectures. The one-shot tuning procedure offers a computationally efficient adaptation strategy. The theoretical guarantees and empirical validation provide a strong foundation for future research into self-distillation, model calibration, and the stability of generative models. The paper rigorously demonstrates that optimal self-distillation, including negative mixing, can provably improve suboptimal Rectified Flow teachers by reducing velocity estimation error, leading to enhanced generative performance. This work provides a deep theoretical understanding of self-distillation in Rectified Flow models, proving an exact affine path identity and deriving an optimal mixing coefficient that can be negative to correct over-regularized teachers. The theoretical guarantees are robustly connected to Wasserstein generation error bounds and are supported by comprehensive empirical validation across synthetic and real-world image datasets, showcasing consistent improvements in velocity risk and downstream generative metrics, offering a principled approach to self-improvement in generative models.
We study causal inference under outcome interference for sequential, observational settings. Specifically, we consider settings where the binary outcomes over N units are Markovian across T time steps. At each time step, the outcomes of N units have dependencies captured through an Ising model; each outcome is also impacted through an external field capturing the effects of its treatment as well as latent confounders. Similar to panel data literature, these latent confounders are modeled to have a low-rank factor structure. Our data is a single sample from this high-dimensional distribution. To estimate causal quantities of interest, we provide a computationally efficient method based on Maximum Pseudo-Likelihood Estimation (MPLE) for learning the model parameters. Under mild assumptions, we establish non-asymptotic consistency for parameter estimation and show this translates to faithful estimation of causal quantities of interest after sampling from the learned model. We demonstrate the efficacy of the method through synthetic experiments as well as a real-world case-study investigating causal effects of vaccine rates on COVID-19 death rates within US counties nationwide.
Primary: Unknown
All Institutions: Unknown
This work has significant broader impact, particularly in fields requiring causal inference in complex, dynamic, and interconnected systems. * **Public Health:** The COVID-19 case study directly demonstrates its utility in understanding the causal effects of interventions like vaccination, which can inform public health policy decisions. The ability to model interference is crucial here, as vaccine efficacy is known to have spill-over effects. * **Policy Making:** The framework can be applied to other policy interventions in social sciences, economics, or environmental studies where units interact, outcomes are sequential, and latent confounders are present. * **Methodological Advancement:** It pushes the boundaries of causal inference by providing a principled approach with theoretical guarantees for a highly challenging combination of settings (sequential, interference, latent confounding, single sample). This could inspire further research in these complex domains. * **Understanding Complex Systems:** By disentangling direct and indirect causal effects in dynamic, interacting systems, the method contributes to a deeper understanding of how interventions propagate through a system. This paper introduces a robust and theoretically grounded framework for causal inference in complex sequential settings with interference and latent confounding, providing non-asymptotic consistency guarantees for parameter estimation via Maximum Pseudo-Likelihood Estimation and demonstrating its practical utility through a compelling COVID-19 case study. The work significantly advances the state-of-the-art in causal inference by addressing a challenging combination of real-world complexities with provable guarantees from a single observational sample, offering a powerful tool for informing policy decisions in interconnected and dynamic systems.
The paper proposes a sophisticated causal inference framework for sequential, observational settings characterized by outcome interference and latent confounding. The core methodology revolves around modeling binary outcomes over N units and T time steps as a Markovian process, where spatial dependencies at each time step are captured by an Ising model. Latent confounders are elegantly incorporated as a low-rank factor structure affecting the external field of the Ising model, a common approach in panel data literature. The data setting is particularly challenging: a single sample from this high-dimensional spatio-temporal distribution. To estimate model parameters, the authors employ Maximum Pseudo-Likelihood Estimation (MPLE), which is computationally efficient due to its simplified partition function compared to full MLE. The MPLE is applied sequentially across time steps. A significant strength of the work lies in its rigorous theoretical guarantees: non-asymptotic consistency for parameter estimation under mild assumptions (Dobrushin's uniqueness condition, low-rank latent structure, bounded interaction, excitability of interventions). These guarantees are then shown to translate to faithful estimation of generalized causal quantities of interest, which are estimated via sequential Gibbs sampling from the learned model. The model's ability to handle arbitrary intervention patterns is a notable advantage over some existing methods. The theoretical proofs, outlined in the main text and detailed in the appendix, demonstrate a deep understanding of high-dimensional statistics and graphical models.
The empirical evaluation is comprehensive, covering both synthetic experiments and a highly relevant real-world case study. 1. **Synthetic Experiments:** These experiments validate the method's ability to estimate parameters and causal estimands in the presence of low-rank latent confounding. The setup involves N=500 units, T=50 time steps, and k=3 latent factors. Interventions and latent confounders are generated with shared low-rank structures, ensuring confounding. The results demonstrate that the proposed MPLE method significantly outperforms baselines that ignore interference (setting $\beta=0$) or latent confounding (setting $A=0$), particularly in estimating the global average treatment effect. This provides strong evidence for the model's correctness and the necessity of its complex components. 2. **COVID-19 Vaccination Case Study:** This is a compelling application, investigating the causal effects of vaccine rates on COVID-19 death rates across US counties. * **Data:** Uses real-world data from NYT and CDC for N=3,014 counties over T=115 weeks. Outcomes (death rates) and interventions (vaccination rates) are binarized. The interaction graph is constructed using a distance kernel function, reflecting realistic spatial interference. * **Hybrid Experiments:** To address potential violations of the "excitability of interventions" assumption in real data, a hybrid experiment is conducted where interventions are partially synthetic but based on real-world patterns. This validates the identifiability of parameters under a realistic intervention distribution. * **Test Set Recovery:** The model's expressiveness is validated by its ability to recover unseen test set outcomes, showing small absolute errors comparable to the hybrid setting. This suggests the model can effectively capture the complex dynamics of COVID-19 data. * **Dobrushin's Condition:** The authors transparently acknowledge that the real-world data violates Dobrushin's condition. They empirically validate fast mixing by comparing results with different Gibbs sampling steps (B=100 vs B=500), showing robustness in practice. * **Causal Effect Estimation:** The method estimates a larger causal effect of vaccination compared to the no-interference baseline, attributing this to spill-over effects. The counterfactual analysis provides intuitive insights into the impact of different vaccination policies. Overall, the experimental evaluation is robust, well-designed, and effectively demonstrates both the theoretical validity and practical utility of the proposed method.
The paper provides an anonymous link to the code for both synthetic and real-world experiments, which is excellent for reproducibility. Details on hyperparameter selection (cross-validation) are mentioned, with further specifics deferred to an appendix. The model setup, data generation process for synthetic experiments, and real-world data sources are clearly described. The theoretical proofs are extensively detailed in the appendix. This level of detail and code availability significantly enhances reproducibility.
The authors explicitly acknowledge two main limitations: 1. **Dobrushin's Uniqueness Condition:** The theoretical results rely on this condition, which may not hold for all real-world settings, particularly those with strong outcome interference. While the authors empirically test for fast mixing in the COVID-19 case study, a theoretical understanding of performance when this condition is violated would be valuable. 2. **Unobserved Interventions:** As with any causal inference method, it's impossible to understand the performance of the model in estimating effects of *unobserved* interventions on real-world data. The validation is based on observed interventional policies. Additional limitations could include: * **Binary Outcomes:** The model is restricted to binary outcomes, which might not capture the full granularity of some real-world phenomena (e.g., continuous death rates). * **Ising Model Specificity:** While powerful, the Ising model assumes pairwise interactions. More complex, higher-order interactions might be present in some real-world networks. * **Computational Cost:** While MPLE is efficient, sequential Gibbs sampling for counterfactuals can still be computationally intensive for very large N and T, especially if many counterfactual scenarios are desired.
This work has significant broader impact, particularly in fields requiring causal inference in complex, dynamic, and interconnected systems. * **Public Health:** The COVID-19 case study directly demonstrates its utility in understanding the causal effects of interventions like vaccination, which can inform public health policy decisions. The ability to model interference is crucial here, as vaccine efficacy is known to have spill-over effects. * **Policy Making:** The framework can be applied to other policy interventions in social sciences, economics, or environmental studies where units interact, outcomes are sequential, and latent confounders are present. * **Methodological Advancement:** It pushes the boundaries of causal inference by providing a principled approach with theoretical guarantees for a highly challenging combination of settings (sequential, interference, latent confounding, single sample). This could inspire further research in these complex domains. * **Understanding Complex Systems:** By disentangling direct and indirect causal effects in dynamic, interacting systems, the method contributes to a deeper understanding of how interventions propagate through a system. This paper introduces a robust and theoretically grounded framework for causal inference in complex sequential settings with interference and latent confounding, providing non-asymptotic consistency guarantees for parameter estimation via Maximum Pseudo-Likelihood Estimation and demonstrating its practical utility through a compelling COVID-19 case study. The work significantly advances the state-of-the-art in causal inference by addressing a challenging combination of real-world complexities with provable guarantees from a single observational sample, offering a powerful tool for informing policy decisions in interconnected and dynamic systems.
Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.
Primary: Unknown
All Institutions: Unknown
This paper presents a compelling proof-of-concept for audio-native speech recognition using frozen discrete-diffusion language models, introducing a novel CTC-based grounding mechanism that overcomes gradient vanishing in attention-based adapters. While it does not yet surpass autoregressive baselines, it establishes a new architectural paradigm for parallel speech decoding and provides valuable insights into the training dynamics of frozen multimodal diffusion models.
The paper proposes a novel architecture for Automatic Speech Recognition (ASR) by interfacing a frozen discrete-diffusion language model (DiffusionGemma) with a frozen Whisper encoder via a lightweight projector and LoRA adapters. The core methodological contribution is the identification and resolution of a "grounding deadlock" where standard diffusion and autoregressive losses fail to propagate gradients to the audio projector because the frozen attention mechanism ignores the random audio embeddings. The authors introduce a Connectionist Temporal Classification (CTC) loss applied directly through the frozen output head to break this symmetry, allowing the audio embeddings to become linearly predictive of the transcript. This enables the diffusion decoder to attend to meaningful audio features. The approach is technically sound and creatively applies the "frozen backbone + adapter" paradigm, previously common in vision-language models, to the emerging field of discrete diffusion language models for speech.
The experimental evaluation is rigorous within the constraints of the data budget. The model achieves 6.6% Word Error Rate (WER) on LibriSpeech test-clean, which is a strong result given the extremely limited training data (100 hours of clean LibriSpeech + 219 hours total including multilingual). The paper provides a detailed ablation study identifying the bottleneck as data scale rather than architecture or encoder quality. The parallel decoding speed is demonstrated to be independent of transcript length, converging in ~8 steps. However, the performance trails autoregressive Whisper significantly, and the comparison is not head-to-head on equal footing (different data, different normalizers). The multilingual evaluation is limited to three languages despite training on six, which weakens the claim of robust multilingual capability.
The paper provides sufficient detail for reproduction, including the architecture of the projector, the specific diffusion process (uniform random-token corruption), the loss functions, and the training hyperparameters (AdamW, LR, batch size). The use of open-weight models (DiffusionGemma, Whisper) and standard datasets (LibriSpeech, FLEURS) enhances reproducibility. The specific "scattering" mechanism for injecting audio tokens is described clearly.
The primary limitation is the significant performance gap compared to state-of-the-art autoregressive systems (Whisper). The authors attribute this to data scale, but the model's ability to generalize to out-of-domain or noisy speech is not thoroughly tested beyond LibriSpeech and VoxPopuli. The fixed 30-second window from Whisper limits streaming applications. Additionally, the diffusion decoding introduces repetition artifacts that require post-processing. The reliance on a frozen Whisper encoder means the acoustic front-end is not optimized for the diffusion decoder, potentially capping performance.
This work demonstrates that diffusion language models, typically associated with text generation, can be effectively adapted for speech recognition, offering a parallel decoding alternative to autoregressive models. This could inspire further research into non-autoregressive speech processing and the grounding of frozen multimodal models. The "CTC unlock" technique for grounding frozen models via direct supervision is a generalizable insight that may apply to other modalities or frozen model adaptations. This paper presents a compelling proof-of-concept for audio-native speech recognition using frozen discrete-diffusion language models, introducing a novel CTC-based grounding mechanism that overcomes gradient vanishing in attention-based adapters. While it does not yet surpass autoregressive baselines, it establishes a new architectural paradigm for parallel speech decoding and provides valuable insights into the training dynamics of frozen multimodal diffusion models.
In this paper, we challenge the prevailing view that information dependency (including rote memorization) drives training data exposure to image reconstruction attacks. We show that extensive exposure can persist without rote memorization and is instead caused by a tunable connection to adversarial robustness. We begin by presenting three surprising results: (1) recent defenses that inhibit reconstruction by Model Inversion Attacks (MIAs), which evaluate leakage under an idealized attacker, do not reduce standard measures of information dependency (HSIC); (2) models that maximally memorize their training datasets remain robust to MIA reconstruction; and (3) models trained without seeing 97% of the training pixels, where recent information-theoretic bounds give arbitrarily strong privacy guarantees under standard assumptions, can still be devastatingly reconstructed by MIA. To explain these findings, we provide causal evidence that privacy under MIA arises from what the adversarial examples literature calls ``non-robust'' features (generalizable but imperceptible and unstable features). We further show that recent MIA defenses obtain their privacy improvements by unintentionally shifting models toward such features. To establish this causal relationship, we introduce Anti Adversarial Training (AT-AT), a training regime that intentionally learns non-robust features to obtain both superior reconstruction defense and higher accuracy than state-of-the-art defenses. Our results revise the prevailing understanding of training data exposure and reveal a new privacy-robustness tradeoff.
This paper fundamentally challenges the prevailing information-theoretic understanding of machine learning privacy by demonstrating that adversarial non-robust features, rather than rote memorization, are the primary driver of training data exposure in Model Inversion Attacks. The work is highly significant for the privacy and robustness communities, offering a causal explanation for why standard defenses work and introducing a novel training regime (AT-AT) that leverages this insight to improve both privacy and accuracy. While the findings are surprising and likely to reshape theoretical discussions on the relationship between robustness and privacy, the immediate impact on general ML practitioners is somewhat niche compared to foundational architectural or scaling breakthroughs, placing it in the "strong paper, likely cited widely" category rather than a field-defining landmark.
The key-value (KV) cache has become the dominant memory cost of transformer inference. It grows with batch size, context length, and depth, and at long context it, rather than the model weights, sets the ceiling on throughput. Two families of methods reduce it. Low-rank methods factor two-dimensional slices of the cache, either per-head matrices or cross-layer feature blocks, and quantization methods lower the bit-width of every entry. Neither family exploits the fact that the cache at a layer is naturally a third-order tensor whose three axes, the heads, the tokens, and the features, carry very different amounts of redundancy. We take this tensor view directly. Our method, JoLT, applies a partial Tucker decomposition that compresses only the token and feature axes while leaving the head and layer axes intact, and then restores the energy that truncation discards with a Johnson-Lindenstrauss (JL) rotated low-bit residual. A single Lagrangian dual allocates the Tucker ranks and the residual bit-widths together, per layer group and separately for keys and values, under one byte budget. The result is a near-lossless 2-3x compression: perplexity, GSM8K accuracy, and RULER needle-in-a-haystack retrieval all stay at or within statistical noise of the uncompressed baseline on both a grouped-query-attention model (Mistral-7B-v0.3) and a multi-head-attention model (LLaMA-2-13B). At 2x, JoLT reconstructs the cache to relative Frobenius error 0.009 (K) and 0.006 (V) on both architectures, roughly an order of magnitude below cross-layer SVD and 4-bit quantization. A randomized-SVD variant, FlashJoLT, delivers a 5-13x compression-time speedup at matched quality.
This paper introduces JoLT, a novel KV cache compression method that leverages tensor decomposition and Johnson-Lindenstrauss residual allocation to achieve near-lossless compression with significantly lower reconstruction error than existing low-rank or quantization baselines. The work is technically sound and addresses a critical bottleneck in LLM inference, offering a meaningful engineering advance that balances compression ratio with accuracy preservation, though it operates within the established paradigm of cache optimization rather than introducing a new foundational architecture or training paradigm.
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.
Primary: New York University
All Institutions: Carnegie Mellon University, New York University
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
The paper introduces "requential coding," a novel and highly efficient method for compressing generative models. It builds upon the principles of prequential coding and relative entropy coding (REC) but addresses their fundamental limitations. Unlike parameter-based methods (e.g., quantization) that scale with model size regardless of information stored, or prequential coding that scales with dataset size and entropy, requential coding decouples code length from both. The core idea is that a "student" model generates candidate training samples from its own distribution, and a stronger "teacher" model selects one of these candidates. The student's code then records only the index of the selected sample, which costs approximately $KL(Q_t\|P_t)$ bits, where $Q_t$ is the teacher and $P_t$ is the student. This effectively distills the teacher's knowledge into the student by efficiently transmitting only the "disagreement" between them. The methodology is rigorously defined, including the encoder and decoder protocols, and a formal bound for the expected code length is derived in the appendix, showing it scales with the cumulative teacher-student KL divergence. Practical improvements like "teacher smoothing" (using an EMA of teacher checkpoints) and "iso-loss projection" (periodically resetting the teacher to the student's performance level) are introduced to further reduce the code length without sacrificing performance. The paper also provides a detailed analysis of the runtime for code length evaluation, encoding, and decoding, acknowledging the computational cost of actual transmission (which can be high due to REC's proposal generation) but emphasizing that evaluation is much more efficient. The use of a universal integer code (Elias delta) for the index ensures the decoder doesn't need prior knowledge of the KL divergence. This is a sophisticated and well-thought-out approach that significantly advances the state of the art in model compression theory.
The experimental evaluation is comprehensive and compelling, demonstrating the power of requential coding across various tasks and scales. 1. **Compression Benchmarking**: Requential coding is benchmarked against prequential coding and idealized 4-bit post-training quantization (PTQ) on autoregressive transformers (100M parameters) trained on OpenWebText, CIFAR-5M, and FineWeb. It achieves one to two orders of magnitude shorter codes than prequential coding, dominating the Pareto frontier of loss vs. code length. The prequential heuristic (a non-rigorous estimate) also falls well above requential coding. 2. **Scaling Insights**: * **Model Size**: Experiments show that larger models (from 10M to 1B parameters) are *more compressible* at a fixed loss, despite having more parameters. This is a crucial finding, as it suggests larger models learn simpler functions relative to their capacity, which parameter-based methods fail to capture. * **Ensembles**: Larger ensembles are also shown to be more compressible, reaching lower loss at a fixed code budget. 3. **Generalization Guarantees (PAC-Bayes Bounds)**: This is a highlight. Plugging the requential code length into a PAC-Bayes bound yields state-of-the-art generalization guarantees for billion-parameter LLMs. These bounds: * Improve with model scale, mirroring the true test loss, for models trained on a fixed amount of data. * Outperform even *idealized lossless 4-bit PTQ bounds* in the compute-optimal scaling regime ($D=20N$), a significant breakthrough as previous non-vacuous bounds for LLMs were based on realistic PTQ. * Tighten with scale, as the code length per parameter decays as a power law, predicting a vanishing generalization gap. 4. **Overfitting Prediction**: The requential code successfully predicts gradual overfitting under data repetition, showing the divergence between training and test loss as the complexity penalty rises. 5. **Learnable Information Content**: The method effectively isolates learnable structure from random information in datasets, revealing that lower-entropy text holds more learnable structure than higher-entropy image data, and distinguishing it from random or trivially repeating strings. This provides a principled way to rank datasets by their "epiplexity." The experiments are well-designed, the results are clearly presented (e.g., Figure 3, 4, 5), and the conclusions drawn are profound, offering new insights into deep learning phenomena that were previously inaccessible to rigorous measurement.
The paper explicitly states that "Code available at https://github.com/shikaiqiu/requential-coding". The methodology is described with sufficient detail, including pseudocode for REC and descriptions of teacher smoothing and iso-loss projection. The appendix provides full experiment details, including how code lengths are computed, datasets used, and per-figure configurations. This commitment to open-sourcing the code and providing detailed experimental setups significantly enhances reproducibility.
The authors acknowledge several limitations: 1. **Lossy Compression of Teacher**: Requential coding provides a lossless compression for the student model, which is a distillation of the teacher. If the goal is to compress a pre-trained model not trained via distillation, it becomes a lossy compression of that original model. 2. **Encoding Runtime**: While code length *evaluation* is efficient, *actually transmitting* a model via requential coding can be prohibitively slow due to the exponential scaling of REC encoding time with KL divergence. The paper primarily presents it as a tool for *evaluation* and *understanding* rather than practical transmission. 3. **Information Forgetting**: The current code length only grows with training steps and doesn't account for information that might be learned early but subsequently forgotten. Leveraging forgotten information could potentially further reduce the code length. 4. **Teacher Optimization**: The choice of teacher sequence is simple, and further gains are expected from optimizing it.
Requential coding has a profound broader impact on the field of machine learning: 1. **Understanding Generalization**: It provides a rigorous, operational measure of model complexity that directly correlates with generalization performance, especially for large neural networks where traditional measures (like parameter count) are misleading. This is a critical step towards a deeper theoretical understanding of deep learning. 2. **Scaling Laws**: The finding that larger models are more compressible at fixed loss offers crucial insights into the efficiency of scaling, suggesting that larger models indeed learn simpler, more generalizable functions. This can guide future research into model architectures and training paradigms. 3. **PAC-Bayes Bounds**: Achieving state-of-the-art, non-vacuous PAC-Bayes generalization bounds for billion-parameter LLMs is a major theoretical breakthrough, making these bounds relevant for modern large-scale models. 4. **Data Curation and Selection**: The ability to distinguish learnable information from random content in datasets provides a principled metric for data quality and could inform strategies for more effective data curation and selection. 5. **Theoretical Tool**: Requential coding serves as a powerful analytical tool for diagnosing phenomena like overfitting and understanding the information flow during training, opening new avenues for theoretical and empirical investigation in deep learning. Requential coding introduces a novel, highly efficient model compression scheme that significantly advances the understanding of generalization in deep learning by providing a rigorous measure of model complexity independent of parameter count and data entropy. This method yields state-of-the-art PAC-Bayes generalization bounds for billion-parameter LLMs, reveals that larger models are more compressible, and offers new insights into overfitting and the learnable information content of datasets, thus serving as a powerful analytical tool for the field.
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the `winning' circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.
Primary: unknown
All Institutions: unknown
This work has significant broader impact potential for several areas of machine learning: 1. **Mechanistic Interpretability:** By providing a low-dimensional, interpretable coordinate system for circuit formation, the IMIR framework offers a powerful new tool for understanding the internal mechanisms of Transformers. This could lead to more systematic and granular circuit discovery. 2. **Predictive Theory of Learning:** The paper takes a concrete step towards a "predictive theory of how Transformers learn," which is a grand challenge in AI. Such a theory could enable better model development, debugging, and safety analysis. 3. **Model Design and Optimization:** A deeper understanding of circuit emergence could inform the design of more efficient architectures, training algorithms, and pruning strategies. Identifying and pruning "useless circuits" or describing useful ones with fewer parameters could reduce memory footprint and inference latency. 4. **Accelerating Research:** By allowing training to be constrained to the IMIR or its subsets, the framework could significantly accelerate experimentation and reduce computational demands for studying learning dynamics, making it easier to uncover novel phenomena. 5. **Generalization of Invariant Manifolds:** The success in identifying an invariant manifold for inductive tasks suggests that similar low-dimensional structures might exist for other tasks, opening new avenues for theoretical research into deep learning dynamics. This paper presents a groundbreaking theoretical framework, the Invariant Manifold of Inductive Reasoning (IMIR), that rigorously explains the emergence of inductive abilities in Transformers by confining their learning dynamics to a low-dimensional, interpretable subspace. The work provides a powerful new lens for mechanistic interpretability, offering a predictive theory for how Transformers learn specific reasoning circuits, and empirically demonstrates its utility in understanding circuit competition, data influence, and initialization effects on synthetic tasks.
This paper introduces a highly novel and rigorous theoretical framework, the Invariant Manifold of Inductive Reasoning (IMIR), to explain the emergence of inductive reasoning abilities in Transformers. The core methodological contribution is the theoretical proof that the training dynamics of attention models, when trained on a generalized class of "block-list tasks," can be confined to a low-dimensional invariant manifold. This manifold is constructed using interpretable "selection, writing, and action bases" that represent elementary operations within the Transformer architecture. The block-list task class itself is a valuable generalization, unifying several synthetic tasks like in-context n-grams and multi-hop reasoning. The paper makes several key assumptions (orthonormal centered embeddings, merged key-query parameterization, fixed orthogonal output-value maps, and specific data symmetries like token invariance and independent positional offsets) to achieve tractability. While these assumptions simplify the model (e.g., attention-only Transformers, fixed value projections), they are clearly stated and motivated, and the paper discusses potential extensions. The decomposition of learning dynamics into interpretable coordinates on this manifold is a significant step towards a more predictive and mechanistic understanding of Transformer learning.
The experimental evaluation, while conducted on synthetic block-list tasks rather than large-scale natural language datasets, effectively demonstrates the utility and explanatory power of the IMIR framework. The paper uses the framework to study three puzzling phenomena: 1. **Circuit Competition:** It analyzes the competition between in-context learning (ICL) and in-weights learning (IWL) in a 2-layer, single-head Transformer. The IMIR allows for an additive decomposition of learned weights into interpretable directions, showing how IWL emerges first and suppresses ICL gradients, and how data properties (rare tokens, burstiness) influence this competition. Theorems are provided to quantify these effects. 2. **Role of Initialization:** The framework is applied to understand how random initializations determine which of several possible induction circuits (e.g., using different layer pairs) "wins" in a multi-layer Transformer. The results show sharp phase transitions in initialization space, suggesting complex interactions beyond simple initial weight magnitudes. 3. **Automated Circuit Detection:** The interpretable nature and low dimensionality of the IMIR are leveraged for automated detection of inductive circuits in trained models. By projecting learned parameters onto the IMIR and ablating dimensions, the paper identifies not only the minimal necessary directions but also "aiding" directions that improve performance, a level of granularity often missed by existing methods. The experiments are well-designed to validate the theoretical claims and provide concrete examples of the framework's application. The figures (e.g., fig:icl-vs-iwl, fig:icl-competition, fig:autointerp) clearly illustrate the findings.
The paper provides a strong foundation for reproducibility. The theoretical framework is rigorously defined with formal assumptions, theorems, and proof sketches (with full proofs promised in appendices). The task class is precisely described. For the empirical evaluations, the paper refers to appendices for "full training details in sec:competing-induction-heads-training" and "complete algorithm in sec:autointerp-algo," which suggests that sufficient detail is provided for others to replicate the experiments. The architecture (attention-only Transformer) and training setup (cross-entropy loss, tied unembedding) are standard. The reliance on synthetic tasks also simplifies reproducibility compared to large-scale natural language experiments.
1. **Simplifying Assumptions:** The core theoretical results rely on several strong assumptions, including attention-only Transformers, merged key-query products, and fixed orthogonal output-value maps. While the paper discusses extensions, the full theoretical treatment for standard Transformers with FFNs, LayerNorms, and learnable value projections is not fully presented. 2. **Synthetic Tasks:** The empirical validation is exclusively on synthetic block-list tasks. While appropriate for foundational theoretical work, it remains to be seen how directly the IMIR framework and its specific basis constructions translate to the complexities of real-world natural language tasks and large-scale LLMs. 3. **Attractiveness of IMIR:** The paper proves the IMIR is invariant (trajectories initialized on it remain on it) but does not formally prove it is *attractive* (trajectories initialized off it converge to it), only showing this empirically. 4. **Scope of Inductive Reasoning:** While the "block-list task class" is a good generalization, it still represents a specific type of inductive reasoning. The framework's applicability to other forms of reasoning (e.g., logical deduction, planning) is not explored.
This work has significant broader impact potential for several areas of machine learning: 1. **Mechanistic Interpretability:** By providing a low-dimensional, interpretable coordinate system for circuit formation, the IMIR framework offers a powerful new tool for understanding the internal mechanisms of Transformers. This could lead to more systematic and granular circuit discovery. 2. **Predictive Theory of Learning:** The paper takes a concrete step towards a "predictive theory of how Transformers learn," which is a grand challenge in AI. Such a theory could enable better model development, debugging, and safety analysis. 3. **Model Design and Optimization:** A deeper understanding of circuit emergence could inform the design of more efficient architectures, training algorithms, and pruning strategies. Identifying and pruning "useless circuits" or describing useful ones with fewer parameters could reduce memory footprint and inference latency. 4. **Accelerating Research:** By allowing training to be constrained to the IMIR or its subsets, the framework could significantly accelerate experimentation and reduce computational demands for studying learning dynamics, making it easier to uncover novel phenomena. 5. **Generalization of Invariant Manifolds:** The success in identifying an invariant manifold for inductive tasks suggests that similar low-dimensional structures might exist for other tasks, opening new avenues for theoretical research into deep learning dynamics. This paper presents a groundbreaking theoretical framework, the Invariant Manifold of Inductive Reasoning (IMIR), that rigorously explains the emergence of inductive abilities in Transformers by confining their learning dynamics to a low-dimensional, interpretable subspace. The work provides a powerful new lens for mechanistic interpretability, offering a predictive theory for how Transformers learn specific reasoning circuits, and empirically demonstrates its utility in understanding circuit competition, data influence, and initialization effects on synthetic tasks.
Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We in troduce Transformers with Temporal Middle-Layer Recurrence (T2MLR), a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position, enabling abstract intermediate computation to persist across decoding steps with little inference overhead. Across natural-language pretraining and multi-hop reasoning finetuning, T2MLR consistently outperforms data- and parameter-matched Transformer base lines. Moreover, applying recurrence to only a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence. Im portantly, T2MLR does not require pretraining from scratch: retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption. These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.
Primary: Princeton University
All Institutions: Princeton University
The paper introduces T^2MLR, a targeted recurrent Transformer architecture that fuses cached middle-layer representations to enhance latent reasoning, demonstrating that selective recurrence is more effective than full recurrence and can be retrofitted into existing models for significant gains in reasoning tasks.
The paper proposes T^2MLR, a novel architectural modification to standard Transformers that introduces temporal recurrence specifically within the middle layers of the network. The core innovation is the fusion of a cached representation from the previous token's middle layer into the current token's earlier layers. This design choice is theoretically grounded in the hypothesis that middle layers handle abstract, compositional reasoning, while lower layers handle syntax and upper layers handle output-specific projections. By recycling these abstract features across time steps, the model aims to mitigate the "bottleneck" effect of autoregressive decoding where information degrades over long contexts. The approach is distinct from full RNN-Transformers or Mamba-like state spaces because it preserves the parallelizable nature of the lower/upper layers while adding minimal recurrence overhead only where it is most cognitively plausible. The method includes a "fusion gate" to control the influence of the cached state, allowing the model to learn when to rely on past context versus current input.
The evaluation covers natural language pretraining and multi-hop reasoning fine-tuning. The results indicate that T^2MLR consistently outperforms parameter- and data-matched Transformer baselines. A key finding is that applying recurrence to only a localized middle-layer block (e.g., 20% of the network) often outperforms full-layer recurrence, supporting the hypothesis that targeted recurrence is more efficient and effective. The paper also demonstrates that retrofitting this architecture into an existing 1.7B parameter model and briefly fine-tuning yields substantial improvements in math reasoning, suggesting practical utility without the cost of training from scratch. However, the citation count of 0 and the arXiv venue suggest this is a very recent or niche contribution, and the "substantial" improvements need to be weighed against the computational overhead of the recurrence mechanism during inference, which the abstract claims is "little" but requires rigorous benchmarking to confirm against highly optimized KV-cache implementations.
The paper provides detailed architectural descriptions, including the fusion gate mechanism and the specific layers targeted for recurrence. The authors mention providing code/scripts in the contributions section ("wrote the core training and inference script"), but no public repository URL is extracted in the text provided. The use of standard datasets for pretraining and finetuning (implied by "natural-language pretraining" and "multi-hop reasoning") aids reproducibility. The claim of "briefly finetuning" suggests a low barrier to reproduction for other researchers.
The primary limitation is the potential increase in memory bandwidth requirements during inference due to the need to access and fuse cached states, even if the compute overhead is low. The paper does not explicitly detail the latency benchmarks compared to standard FlashAttention-based Transformers, which are the current state-of-the-art for efficiency. Additionally, the "middle layer" heuristic might not generalize to all model sizes or architectures equally; the optimal placement might vary. The reliance on a specific architectural slot (middle layers) limits its applicability to models with different internal structures (e.g., MoE models where sparsity changes the effective depth).
This work contributes to the field of efficient and robust language modeling by offering a biologically-inspired (recurrent) alternative to pure attention mechanisms for long-range dependency handling. It challenges the assumption that all layers must be recurrent to benefit from recurrence, providing a more efficient path to latent reasoning. This could influence future architectures to adopt hybrid attention-recurrence designs, potentially leading to models that are both faster and better at complex reasoning tasks. The paper introduces T^2MLR, a targeted recurrent Transformer architecture that fuses cached middle-layer representations to enhance latent reasoning, demonstrating that selective recurrence is more effective than full recurrence and can be retrofitted into existing models for significant gains in reasoning tasks.
Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce \textit{expert scattering}: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose \textsc{EcoSpec}, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, \textsc{EcoSpec} favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate \textsc{EcoSpec} on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. \textsc{EcoSpec} consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to $1.62\times$ speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.
This paper introduces EcoSpec, a cost-aware speculative decoding framework that optimizes draft token selection by accounting for expert activation memory traffic in Mixture-of-Experts (MoE) models. The work is significant because it identifies and addresses a specific bottleneck—expert scattering—that limits the efficiency of standard speculative decoding in modern MoE architectures like DeepSeek-V3 and Qwen3. By integrating lightweight expert prediction into the draft tree selection process, it achieves meaningful end-to-end speedups without modifying the target model, offering a practical optimization for the rapidly scaling MoE paradigm. While the contribution is solid and addresses a timely engineering challenge in LLM inference, it remains an optimization layer rather than a fundamental architectural or algorithmic breakthrough, placing it in the upper tier of applied ML research but below the threshold for field-defining significance.
Music creation is fundamentally a process of revision. Yet symbolic music generation remains dominated by paradigms that produce complete sequences from scratch, with limited support for selective modification. Edit-based methods have proven effective for text transformation tasks, but remain largely unexplored for symbolic music. We trace this absence to the representational level: conventional event-based music encodings lack the structural properties required by explicit music editing. In contrast, the BEAT encoding, a beat-grid-anchored representation originally designed for autoregressive generation, possesses structural properties amenable to editing. We propose BeatEdit, the first framework for symbolic music generation based on explicit edit operations, recasting generation as producing new content by editing a draft rather than synthesizing from scratch. BeatEdit comprises three complementary mechanisms along an axis of increasing edit density: per-token sequence tagging for error correction, iterative refinement for accompaniment editing, and tag-then-fill for segment completion. All these mechanisms share a single encoding and pre-trained backbone, achieving higher precision and perceptual quality than autoregressive and diffusion methods across all three tasks, while remaining efficient, with single-pass inference completing in under 100 ms. Cross-encoding evaluation further reveals that encoding design substantially influences editing effectiveness, with notable encoding-method interaction effects. Code is available at https://github.com/Haoyu-Gu/BeatEdit-code
Primary: South China University of Technology
All Institutions: South China University of Technology, Nanjing University
BeatEdit establishes explicit editing as a viable and superior paradigm for symbolic music generation, demonstrating that careful design of the underlying representation is crucial for enabling efficient and high-fidelity content modification. The framework's ability to handle varying edit densities with a single backbone and its significant performance gains over autoregressive and diffusion baselines mark a substantial step forward in the field.
The paper introduces a novel paradigm for symbolic music generation by recasting it as an explicit editing problem, a significant departure from the dominant autoregressive and diffusion approaches. The core methodological contribution is the identification and formalization of three structural requirements (atomic edit units, trivial source-target alignment, and edit locality) that a music representation must satisfy to support explicit editing. The authors propose the BeatEdit framework, which utilizes the BEAT encoding (a beat-grid-anchored representation) to satisfy these requirements. They design three complementary mechanisms along an axis of edit density: per-token sequence tagging for sparse corrections, iterative refinement for accompaniment editing, and tag-then-fill for segment completion. The approach leverages a shared BERT backbone pre-trained on music, demonstrating that the representational choice is as critical as the model architecture. The methodology is rigorous, well-motivated by NLP precedents, and technically sound, with clear adaptations for music-specific challenges like relative pitch cascading.
The experimental evaluation is comprehensive and convincing. The authors evaluate on 192K piano pieces across three tasks (error correction, accompaniment editing, segment completion) using multiple baselines (AR, Diffusion, BERT-CMLM). BeatEdit consistently outperforms baselines in both objective metrics (beat exact match, note F1) and subjective listening tests (MOS). A key strength is the ablation study and cross-encoding analysis, which reveals significant interaction effects between encoding schemes (Absolute/Relative, Separated/Bundled) and editing methods. This provides deep insights into why certain representations work better for editing than others, a finding that is valuable for the broader symbolic music community. The efficiency gains (single-pass inference <100ms) are also empirically validated.
The paper provides detailed descriptions of the encoding schemes, model architectures, and training procedures. The code is made publicly available on GitHub, which significantly enhances reproducibility. The dataset (MuseScore Collection) is publicly accessible. The synthetic perturbation strategy for training is clearly defined. The inclusion of algorithms for encoding/decoding and inference steps further aids reproducibility.
The primary limitation is the reliance on synthetically perturbed data for training and evaluation. While this allows for precise control over task difficulty, it may not fully capture the complexity and nuance of real-world human edits. The paper acknowledges this gap and calls for paired draft-to-final corpora. Additionally, the current evaluation is limited to piano music (dual-voice), although some generalization to multi-instrument data is shown. The assumption of known beat/bar boundaries is also a constraint, though the authors note this can be addressed with external tools.
This work has significant implications for symbolic music generation, offering a new paradigm that aligns more closely with human creative processes (revision and editing). It could lead to more efficient and controllable music generation tools. The insights into representational requirements for editing may also influence the design of future music tokenization schemes. The open-source release promotes further research in this area. BeatEdit establishes explicit editing as a viable and superior paradigm for symbolic music generation, demonstrating that careful design of the underlying representation is crucial for enabling efficient and high-fidelity content modification. The framework's ability to handle varying edit densities with a single backbone and its significant performance gains over autoregressive and diffusion baselines mark a substantial step forward in the field.
Predictive world models enable robots to plan by imagining the outcomes of their actions, but their value for control hinges on generating many rollouts quickly. This creates a bottleneck for diffusion-based world models: multistep sampling makes each rollout expensive, limiting large-scale action search at inference time. We introduce DriftWorld, an action-conditioned world model based on drifting generative models. Rather than denoising iteratively at inference, DriftWorld learns an action-conditioned drift during training, allowing it to generate future frames from the current observation and a candidate action sequence in a single forward pass at 30+ fps, which is 17x faster on average than diffusion based baselines. We evaluate DriftWorld on standard vision-based robotic manipulation benchmarks, including Bridge-V2, RT-1, Language Table, Push-T, and Robomimic. By producing rollouts that are both accurate and fast, DriftWorld achieves state-of-the-art decision-making performance with far less inference time than diffusion-based world model baselines. Beyond online control, DriftWorld can also serve as an offline simulator for ranking real-world robot policies, with rollout-based scores correlating with ground truth at up to 0.99. These results show that drifting models are a strong fit for robot world modeling, where fast, high-quality imagination directly supports planning and policy evaluation.
Primary: University of Washington
All Institutions: University of Washington, Microsoft Research
DriftWorld introduces a fast, action-conditioned world model based on drifting generative models, achieving state-of-the-art performance on robotic manipulation benchmarks with significantly reduced inference latency compared to traditional diffusion models.
The paper introduces DriftWorld, a novel approach to action-conditioned world modeling that leverages continuous-time diffusion processes (specifically, the probability flow ODE or "drift" field) rather than discrete denoising steps. The core innovation is learning the action-conditioned drift vector field during training, which allows for single-pass generation of future frames at inference time. This effectively transforms a multi-step iterative sampling process into a deterministic (or single-sample) forward pass, drastically reducing inference latency. The methodology is theoretically grounded in the connection between diffusion models and continuous normalizing flows, applying it specifically to the constraints of robotic planning where speed is critical.
The evaluation is comprehensive, covering standard vision-based robotic manipulation benchmarks including Bridge-V2, RT-1, Language Table, Push-T, and Robomimic. The results demonstrate that DriftWorld achieves state-of-the-art decision-making performance while being 17x faster than diffusion-based baselines. The paper also highlights an interesting secondary capability: using the world model as an offline simulator for ranking real-world policies, with rollout-based scores correlating with ground truth at up to 0.99. This strong correlation suggests the model captures high-fidelity dynamics, which is crucial for sim-to-real transfer and policy evaluation.
The authors provide a project website with videos and code, indicating a commitment to reproducibility. The use of standard benchmarks ensures that comparisons are fair and replicable by the community. The description of the drift learning process appears sufficiently detailed in the full text to allow for implementation, assuming access to the provided codebase.
As a diffusion-based model, even with single-pass inference, the model may still suffer from the typical limitations of generative models, such as mode collapse or hallucination of physically impossible states, although the paper claims high accuracy. The performance gain is relative to multi-step diffusion; it may not be as fast as purely autoregressive models (like standard RNNs or Transformers) which are inherently single-pass, though those often suffer from compounding errors or lower fidelity. The paper focuses on vision-based manipulation; generalization to other modalities (e.g., proprioception-only or multi-sensor fusion) is not explicitly detailed.
This work has significant implications for real-time robotic control, enabling more complex planning algorithms that require extensive imagination of future trajectories. By making world models fast enough for online use, it lowers the barrier for deploying sophisticated planning-based controllers in dynamic environments. The ability to rank policies accurately using simulated rollouts also aids in the safe deployment of RL policies in the real world. DriftWorld introduces a fast, action-conditioned world model based on drifting generative models, achieving state-of-the-art performance on robotic manipulation benchmarks with significantly reduced inference latency compared to traditional diffusion models.
World Action Models (WAMs) enable semantically- and physically-informed control but are brittle under distribution shift. In this work, we use mechanistic interpretability to study how robustness-relevant perturbations are represented in WAM activation space. Comparing activations across successful and unsuccessful rollouts, we find some WAM architectures exhibit low-dimensional linear separability for robustness-critical features, while others do not. This motivates the use of contrastive activation directions for training-free WAM steering. We also show that local linearity in WAM activation dynamics enables efficient feedback steering via model-based optimal control, yielding World-Action Linear Quadratic Regulator (WA-LQR), a minimally-invasive reduced-order LQR controller. Via mechanistic evaluations, we predict strong steerability in the Cosmos-Policy and DiT4DiT models but weak steerability in LingBot-VA, consistent with steering intervention results. On Cosmos-Policy and DiT4DiT, WA-LQR generalizes contrastive directions to new tasks and improves robustness to camera, gripper, and visual-noise perturbations over unsteered and prompt steering baselines.
Primary: unknown
All Institutions: unknown
This paper makes a significant contribution to improving the robustness of robotic foundation models, which is a critical step towards their reliable deployment in real-world applications. 1. **Robust Robotics:** By providing a training-free method to improve WAM robustness to common OOD shifts, it directly addresses a major bottleneck in robot learning. 2. **Foundation Model Interpretability:** It extends mechanistic interpretability to a new domain (WAMs), demonstrating its utility not just for understanding but also for *intervening* in model behavior. 3. **Model Design Guidance:** The finding that steerability is architecture-dependent and predictable via hinge loss provides valuable diagnostic tools for future WAM design. Researchers can aim to build WAMs that inherently possess more steerable, linearly separable representations of robustness-critical features. 4. **Cross-Domain Insights:** The work successfully bridges concepts from mechanistic interpretability, optimal control, and robotics, demonstrating the power of interdisciplinary approaches in ML research. 5. **Inference-Time Interventions:** The focus on training-free, inference-time interventions is highly practical, as it avoids costly data collection and retraining, making it more amenable to real-world deployment scenarios. This paper presents a novel framework for diagnosing and improving the robustness of World Action Models (WAMs) using mechanistic interpretability and optimal control. It introduces WA-LQR, a training-free, closed-loop steering method that leverages low-dimensional linear activation dynamics to enhance robustness to out-of-distribution shifts, demonstrating significant performance gains and providing a valuable diagnostic tool for future WAM design.
The paper proposes a sophisticated methodology that integrates mechanistic interpretability (MI) with optimal control to enhance the robustness of World Action Models (WAMs). The core approach involves two main stages: 1. **Mechanistic Interpretability for Feature Discovery:** The authors analyze WAM activation spaces under robustness-relevant perturbations (camera, gripper, visual noise). They collect activations from successful and unsuccessful rollouts, compute contrastive directions, and then project these onto a low-dimensional PCA subspace. Linear separability is quantified using SVM hinge loss. This diagnostic step is crucial for identifying *when* and *where* robustness-critical features are linearly represented and thus steerable. This is a sound adaptation of MI techniques from LLMs/VLAs to WAMs. 2. **Training-Free Activation Steering:** * **Open-Loop Activation Addition (ActAdd):** A baseline method adapted from LLM literature, where averaged contrastive directions are simply added to WAM activations at inference time. This is a straightforward, albeit less adaptive, approach. * **World-Action Linear Quadratic Regulator (WA-LQR):** This is the main technical contribution. WA-LQR generalizes ActAdd by introducing feedback control. It first performs dimensionality reduction on contrastive directions to create a low-dimensional latent feature space. It then leverages the empirically verified local linearity of WAM activation dynamics within this latent space to formulate an LQR problem. The LQR controller computes minimum-cost interventions to steer activations towards desired feature setpoints, adapting online to the realized WAM activation. The use of Jacobian-vector products (JVPs) and vector-Jacobian products (VJPs) for efficient computation of reduced-order Jacobians ($A_{l,t}, B_{l,t}$) is a practical and necessary detail. The action-decay schedule for the control penalty is a thoughtful design choice to prioritize steering early in the action sequence. The formulation of independent L-step LQR problems for each action chunk and denoising timestep is a clever way to manage complexity. The methodology is well-articulated, building on established concepts from MI and optimal control, and adapting them thoughtfully to the specific challenges of WAMs. The diagnostic capability of predicting steerability via hinge loss is a strong point.
The experimental evaluation is comprehensive and well-structured. 1. **Models and Tasks:** The study evaluates on three distinct WAM architectures: Cosmos-Policy 2B, DiT4DiT, and LingBot-VA, covering different architectural approaches. Experiments are conducted across 10 tasks from the LIBERO-10 dataset, ensuring a diverse set of manipulation scenarios. 2. **Perturbations:** Three critical types of out-of-distribution (OOD) shifts are considered: initial gripper position, camera orientation, and Gaussian noise corruption, which are highly relevant for real-world robot deployment. 3. **Mechanistic Validation:** The paper empirically verifies key assumptions for WA-LQR: * **Contrastive Information Preservation:** Demonstrates that robustness-relevant information is concentrated in a low-dimensional subspace (Fig. 3), justifying dimensionality reduction. * **Local Linearity:** Shows that WAM latent dynamics are locally linear (cosine similarity and magnitude ratio close to 1, Fig. 2a), validating the use of LQR. * **Subspace Overlap:** Finds significant overlap in the subspaces spanned by the Jacobians ($A_{l,t}$) across different inputs and tasks (Fig. 2b), suggesting potential for reusability. 4. **Steering Performance:** * **Baselines:** Compares WA-LQR and ActAdd against unsteered models and a prompt-steering baseline. * **Results:** WA-LQR significantly improves robustness (up to 41% success rate) on Cosmos-Policy and DiT4DiT across various perturbations. ActAdd also shows improvements but can suffer from oversteering. Prompt steering is generally ineffective. * **Diagnostic Correlation:** A key finding is the strong negative correlation between SVM hinge loss (mechanistic separability) and robustness improvements from steering (Fig. 4). This validates the MI-based diagnostic as a reliable predictor of steerability. * **Architecture Dependency:** The results confirm the mechanistic prediction that Cosmos-Policy and DiT4DiT are steerable, while LingBot-VA shows weak steerability, highlighting the architecture-dependent nature of these representations. The experiments are thorough, provide strong evidence for the proposed methods, and critically validate the underlying assumptions and diagnostic capabilities.
The paper provides a detailed methodological description, including equations for WA-LQR and the MI analysis. However, it lacks specific implementation details such as hyperparameter values for LQR (e.g., Q and R matrices, decay rates), specific layer/timestep choices for intervention, or the exact procedure for computing Jacobians. Crucially, there is no mention of publicly available code or datasets, which significantly hinders direct reproducibility. While the theoretical framework is clear, reproducing the exact experimental results would require substantial effort to re-implement the methods and tune parameters.
The authors acknowledge several key limitations: 1. **Limited Applicability Across Tasks and Models:** The paper explicitly states that there is no clear interpretable method for predicting when tasks or environments share transferable representations, requiring per-setting mechanistic analysis. This limits the generalizability of specific contrastive directions. 2. **Architecture Dependence:** Steerability is strongly dependent on the WAM architecture, meaning the method is not universally applicable to all WAMs. 3. **Computational Overhead:** While WA-LQR is "efficient" in its reduced-order form, solving LQR problems for each action chunk and denoising timestep still adds computational overhead compared to unsteered inference, which might be a concern for real-time applications with very tight latency constraints. 4. **Reliance on Local Linearity:** The effectiveness of WA-LQR hinges on the assumption of local linearity in the latent activation dynamics. While empirically verified, this assumption might break down under more extreme perturbations or in different WAMs. 5. **Lack of Code/Details:** As noted in reproducibility, the absence of code or more granular implementation details is a practical limitation for adoption.
This paper makes a significant contribution to improving the robustness of robotic foundation models, which is a critical step towards their reliable deployment in real-world applications. 1. **Robust Robotics:** By providing a training-free method to improve WAM robustness to common OOD shifts, it directly addresses a major bottleneck in robot learning. 2. **Foundation Model Interpretability:** It extends mechanistic interpretability to a new domain (WAMs), demonstrating its utility not just for understanding but also for *intervening* in model behavior. 3. **Model Design Guidance:** The finding that steerability is architecture-dependent and predictable via hinge loss provides valuable diagnostic tools for future WAM design. Researchers can aim to build WAMs that inherently possess more steerable, linearly separable representations of robustness-critical features. 4. **Cross-Domain Insights:** The work successfully bridges concepts from mechanistic interpretability, optimal control, and robotics, demonstrating the power of interdisciplinary approaches in ML research. 5. **Inference-Time Interventions:** The focus on training-free, inference-time interventions is highly practical, as it avoids costly data collection and retraining, making it more amenable to real-world deployment scenarios. This paper presents a novel framework for diagnosing and improving the robustness of World Action Models (WAMs) using mechanistic interpretability and optimal control. It introduces WA-LQR, a training-free, closed-loop steering method that leverages low-dimensional linear activation dynamics to enhance robustness to out-of-distribution shifts, demonstrating significant performance gains and providing a valuable diagnostic tool for future WAM design.
Robotic manipulation is inherently multi-frame: local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame. However, modern diffusion-based visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame. We propose Mixture of Frames Policy (MoF), a diffusion policy that performs synchronized action denoising across multiple coordinate frames. MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame. To make this possible for intermediate noisy diffusion states, we introduce a column-based 6D rotation representation within an SE(3) action parameterization that supports exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold. Across nine simulated bimanual manipulation tasks, we show that the best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines. We further evaluate MoF on two real-world bimanual mobile manipulation tasks, demonstrating that it outperforms all constituent single-frame baselines. Project homepage: https://mofpo.github.io
Primary: Stanford University
All Institutions: Stanford University, Apple, Toyota Research Institute
The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
The paper proposes a novel architecture for diffusion-based visuomotor policies called Mixture of Frames (MoF). The core insight is that robotic manipulation actions are naturally represented in different coordinate frames at different stages of a task (e.g., end-effector for grasping, base for transport). Standard diffusion policies fix a single action frame, forcing the denoiser to model complex, frame-dependent distributions. MoF maintains a single canonical diffusion state but applies frame-specialized denoisers in their respective frames, fusing the noise predictions back into the canonical frame. A critical technical contribution is the introduction of a column-based 6D rotation representation for SE(3) actions. This allows for exact, differentiable frame transformations of *noisy* intermediate states without requiring projection onto the SO(3) manifold, which is non-linear and lossy. This mathematical insight is elegant and solves a significant implementation hurdle for multi-frame reasoning in diffusion policies. The method is generalizable to any diffusion policy and can be implemented as a drop-in replacement or extension.
The evaluation is comprehensive and rigorous. The authors first establish the motivation by showing that action frame choice has a substantial, task-dependent impact on performance in bimanual mobile manipulation, with a 15% gap between the best and worst single frames. They then demonstrate that MoF outperforms oracle frame selection (which knows the best frame per task) and standard Mixture-of-Experts (MoE) baselines that do not operate in multiple action frames. The ablation studies are particularly strong, showing that the proposed column-based representation is essential (orthogonalization leads to collapse in ensemble variants) and that the router effectively switches between frames based on task phase. Real-world experiments on two complex bimanual mobile manipulation tasks (Pouring, Serving) further validate the approach, showing significant improvements over single-frame baselines. The use of simulated benchmarks (BiGym, DexMimicGen) and real-world validation provides a robust evidence base.
The paper provides detailed descriptions of the method, including the specific rotation representation and the fusion mechanism. The project homepage link suggests code availability. The experimental setup is well-described, including dataset sizes, training epochs, and evaluation metrics. The ablation studies provide clear evidence of component contributions. The reliance on standard simulators (BiGym) and real-world hardware (HoMMI) enhances reproducibility.
The method relies on a designer-specified set of candidate frames. While the router learns to weight them, the set must be predefined. The paper acknowledges this and suggests future work on learning or discovering frames. Additionally, the router is supervised via denoising loss, which may not perfectly align with policy performance, though the results suggest it works well in practice. The current experiments focus on smaller diffusion policies; scaling to large VLA models is noted as future work.
This work has significant implications for robotic manipulation, particularly in complex, multi-stage tasks where different coordinate systems are advantageous. By improving the robustness and performance of visuomotor policies, it contributes to the broader goal of autonomous robots capable of operating in unstructured environments. The technical contribution regarding SE(3) transformations is also valuable for the broader machine learning community working with geometric deep learning and diffusion models. The paper presents a highly innovative and technically sound approach to improving diffusion-based visuomotor policies by leveraging multi-frame reasoning. The introduction of a column-based 6D rotation representation for exact noisy state transformation is a significant methodological advance. Comprehensive experiments in simulation and on real-world robots demonstrate substantial improvements over strong baselines, including oracle frame selection. The work is well-motivated, rigorously evaluated, and addresses a fundamental limitation in current robotic policy architectures. It is a strong candidate for top-tier publication and will likely influence future research in robotic policy learning.
Dexterous grasp generation across robot hands is challenging because hands differ in kinematic topology, actuation dimensions, and native command spaces. We introduce GraspGraphNet, a topology-aware grasp generation framework that represents each hand as a URDF-derived kinematic graph and directly generates executable palm poses and joint configurations. GraspGraphNet combines hierarchical object surface encoding, differentiable forward kinematics, and dynamic world-edge message passing to model evolving robot-object interactions. It applies conditional flow matching directly in executable palm-pose and joint-state space, avoiding post-processing optimization, inverse kinematics, and retargeting. Using a shared model trained on Barrett Hand, Allegro Hand, and Shadow Hand, GraspGraphNet achieves an average success rate of 83.48% with 40ms inference time per grasp on a 40-object benchmark. Without retraining, the same model achieves 72.70% success on controlled finger-removal variants, demonstrating robustness to hand-topology variations. These results suggest that graph-structured hand representations can effectively support dexterous grasp generation across robot hands with different kinematic structures. Project: https://lysees.github.io/graspgraphnet-page
Primary: Korea Advanced Institute of Science and Technology (KAIST)
All Institutions: Korea Advanced Institute of Science and Technology (KAIST)
GraspGraphNet presents a significant technical contribution to multi-embodiment robotic manipulation by introducing a topology-aware, graph-based flow matching framework that achieves state-of-the-art performance and efficiency in generating executable dexterous grasps across diverse robot hands.
The paper proposes GraspGraphNet, a novel framework for multi-embodiment dexterous grasp generation. The core innovation lies in representing robot hands as URDF-derived kinematic graphs and object surfaces as hierarchical point clouds. By integrating differentiable forward kinematics into a graph neural network (GNN) architecture, the model respects the topological constraints of each hand. Crucially, it employs dynamic world-edge message passing to update robot-object interaction features at every step of a conditional flow matching process. This allows the model to generate executable palm poses and joint configurations directly, bypassing the need for inverse kinematics or post-processing optimization typically required by contact-based methods. The use of flow matching for continuous state refinement is a sophisticated choice that addresses the multimodality of grasp distributions better than direct regression or standard diffusion models.
The experimental evaluation is comprehensive and rigorous. The authors evaluate on three distinct dexterous hands (Barrett, Allegro, Shadow) across 40 objects, providing a robust benchmark for multi-embodiment generalization. The results show state-of-the-art success rates (83.48% average) while significantly outperforming baselines in inference speed (40ms vs. 265ms+ for competitors). The finger-removal experiments are particularly compelling, demonstrating that the graph-based representation allows for zero-shot generalization to topologically modified hands (72.70% success), a feat previous methods struggle with. Real-world deployment on a Leap Hand further validates the practical utility of the approach. The ablation studies effectively isolate the contributions of dynamic world edges, cross-attention, and the flow-matching objective.
The paper provides sufficient implementation details, including network architecture dimensions, training hyperparameters (AdamW, learning rate, epochs), and dataset sources (CMapDataset). The use of standard simulation environments (Isaac Gym) and publicly available datasets enhances reproducibility. The inclusion of a project page suggests code availability, which is critical for replication in this domain.
The primary limitation is the reliance on URDF definitions, which requires accurate kinematic models for each hand. While this is standard in simulation, real-world deployment may require precise calibration or adaptation to handle kinematic drift or non-rigid deformations not captured in the URDF. Additionally, the model is evaluated on a specific set of hands; generalization to radically different morphologies (e.g., soft robotic hands or grippers with vastly different joint limits) is not demonstrated, though the authors acknowledge this as future work. The reliance on flow matching also introduces a small number of integration steps (K=3), which, while efficient, may occasionally produce less optimal grasps compared to methods with more extensive search or optimization phases, although the speed trade-off is favorable.
This work significantly advances the field of robotic manipulation by providing a unified, efficient, and robust method for dexterous grasp generation across diverse hardware. By eliminating the need for hand-specific retargeting and optimization, it lowers the barrier to deploying dexterous manipulation policies on new robotic platforms. The graph-based approach offers a principled way to handle structural variations, which could inspire similar topology-aware representations in other robotic control tasks. The efficiency gains (40ms inference) make real-time dexterous manipulation more feasible for practical applications. GraspGraphNet presents a significant technical contribution to multi-embodiment robotic manipulation by introducing a topology-aware, graph-based flow matching framework that achieves state-of-the-art performance and efficiency in generating executable dexterous grasps across diverse robot hands.
Bipedal robots are challenging to control because they operate close to instability, where small variations in foot-terrain contact can rapidly destabilize locomotion. On rigid terrain, bipedal robots mitigate this fragility by using well-established contact mechanics and control strategies. On flowable surfaces such as granular slopes, foot contact can induce large surface deformations and solid-fluid-like transitions, coupling terrain effects with robot dynamics, leading to underperformance or failure. This is partly due to the lack of reliable methods to represent the dynamics of flowable terrain, making it difficult to account for terrain effects in locomotion design. Here, we investigate how controlling terrain response can improve bipedal locomotion on granular slopes by studying the terradynamics of cleated feet, thin plates emanating from the foot soles. Systematic studies of a small-scale (1.4 kg) robophysical biped reveal that cleats with sparse and dense spacing lead to excessive terrain yielding and resistance, respectively, degrading performance and leading to failure. An intermediate cleat spacing distributes interaction forces to maintain substrate stresses near (or below) the yield threshold, enabling walking on granular slopes up to 30 degrees. Guided by these principles, we design a foot that actively adjusts cleat depth and accommodates both rigid and granular terrain. We also demonstrate that the principles of effective foot-terrain interaction translate to a larger (15 kg) autonomous biped. Our study presents an alternative to conventional body-centric robot control approaches, which regulate terrain-induced effects through body motion, by instead regulating terrain interactions through limb-centric approach.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, Northeastern University
This paper presents a seminal contribution to robotic locomotion on deformable terrain, establishing a mechanistic framework for cleated-foot interaction that enables robust bipedal walking on steep granular slopes. By systematically characterizing terradynamics and validating findings across multiple scales, the authors provide a foundational understanding that bridges the gap between granular physics and legged robot design, offering a pathway to more capable autonomous systems in challenging natural environments.
The paper employs a rigorous robophysical approach, combining systematic experimental variation of foot morphology (cleat spacing and depth) with granular terradynamics analysis. The methodology is grounded in physical principles rather than purely data-driven learning, utilizing Particle Image Velocimetry (PIV) and dual-plate intrusion tests to elucidate the underlying mechanics of granular flow. The introduction of a limb-centric control paradigm, where terrain interaction is regulated through foot design rather than just body dynamics, represents a significant conceptual shift in legged locomotion research. The integration of these insights into an adaptive foot mechanism (retractable cleats) and validation on a large-scale autonomous biped (HECTOR) demonstrates a robust engineering methodology.
The experimental evaluation is comprehensive and multi-scaled. It begins with a small-scale constrained robot (BLUEY) to isolate variables, proceeds to granular intrusion tests for mechanistic understanding, and culminates in challenging field-relevant scenarios with a large-scale autonomous robot (HECTOR). The use of PIV to visualize particle flow provides unique, high-quality data linking foot geometry to substrate response. The results clearly demonstrate that intermediate cleat spacing optimizes performance by balancing traction and substrate disturbance, a finding that is both counter-intuitive and practically significant. The successful translation of these principles to a 15 kg robot is a strong validation of the findings.
The paper provides detailed descriptions of the robot kinematics, gait generation (ZMP-based), and experimental setups. The inclusion of pseudocode for the adaptive cleat control and specific parameters for the granular substrate (poppy seeds, volume fraction) enhances reproducibility. However, the reliance on specific robophysical platforms and the complexity of the granular terrain preparation (air-fluidization) may pose challenges for exact replication without access to similar infrastructure. The open-source nature of the HECTOR platform aids in this regard.
The study is limited to rectangular cleats and specific granular media (poppy seeds). The scaling laws derived may not directly apply to human-scale robots or different substrate types (e.g., cohesive soils) without further validation. The gait strategies tested are relatively simple (quasi-static for BLUEY, limited MPC exploration for HECTOR), leaving room for optimization. The authors acknowledge these limitations and suggest future work involving DEM simulations and more complex terrain.
This work has significant implications for the field of legged robotics, particularly for applications in unstructured environments such as search and rescue, planetary exploration, and agricultural robotics. By providing a mechanistic framework for locomotion on flowable terrain, it enables the design of more robust and versatile robotic systems. The shift from body-centric to limb-centric control strategies offers a new paradigm for handling terrain-robot coupling, potentially influencing future research in adaptive morphology and control. This paper presents a seminal contribution to robotic locomotion on deformable terrain, establishing a mechanistic framework for cleated-foot interaction that enables robust bipedal walking on steep granular slopes. By systematically characterizing terradynamics and validating findings across multiple scales, the authors provide a foundational understanding that bridges the gap between granular physics and legged robot design, offering a pathway to more capable autonomous systems in challenging natural environments.
Existing robotic perception is constrained by sensors that are either robot-mounted or permanently fixed in the environment, locking perception to a limited set of viewpoints. Yet as robots perform increasingly diverse tasks, the most informative viewpoint shifts from one task to the next-often somewhere onboard sensor and static infrastructure can not readily satisfy. To address this gap, we propose SensorPerch, a novel realization of active perception that decouples sensing from both the robot embodiment and the environment by treating sensors as independent physical entities that the robot can autonomously detach and re-attach within the environment. SensorPerch presents one realization of this paradigm: a lightweight, wireless, reconfigurable sensor platform that can perch on diverse surfaces, paired with a viewpoint-selection framework that determines task-optimal sensor placements. Together, these enable robots to construct task-relevant viewpoints on demand, independent of the robot's current position and available fixed infrastructure. We demonstrate the paradigm on two task classes: (i) object-coupled perception, where SensorPerch enables persistent object-state detection beyond the robot's current position, achieving successful event detection even when the robot is not nearby; and (ii) policy-coupled perception, where SensorPerch allows robots to construct diverse, policy-specific viewpoints for various policies, achieving success rates comparable to those obtained using oracle viewpoints.
Primary: Cornell University
All Institutions: Cornell University
SensorPerch represents a significant step towards more versatile and autonomous robotic systems. By decoupling perception from fixed infrastructure and robot embodiment, it enables robots to adapt their sensing capabilities dynamically to task requirements, leading to several broader impacts: 1. **Enhanced Robot Autonomy:** Robots can operate more independently in diverse, unstructured environments without relying on pre-installed cameras or human intervention for viewpoint adjustment. 2. **Improved Task Performance:** The ability to construct task-optimal viewpoints on demand can lead to higher success rates in complex manipulation, monitoring, and interaction tasks, especially in scenarios requiring persistent observation or specific perspectives. 3. **New Active Perception Paradigm:** This work establishes a new paradigm for active perception, shifting from merely adjusting onboard sensors to physically reconfiguring the sensing infrastructure itself. This could inspire future research into modular, reconfigurable robotic components beyond just sensors. 4. **Applications in Diverse Fields:** Beyond household robotics, this technology could be transformative for applications in industrial inspection, disaster response (deploying sensors in hazardous areas), construction, and even scientific exploration, where dynamic, adaptive sensing is crucial. 5. **Reduced Infrastructure Costs:** By allowing a single robot to deploy and manage its own sensing network, it could reduce the need for expensive, permanently installed camera systems or multi-robot coordination for comprehensive coverage. SensorPerch introduces a novel paradigm for active perception by treating sensors as independent, reconfigurable physical entities that robots can autonomously deploy and manage. This comprehensive system, integrating robust hardware with an intelligent software framework for task-conditioned viewpoint selection, significantly enhances robotic perception capabilities, enabling persistent object-state detection and policy-aligned viewpoints in diverse real-world tasks, thereby pushing the boundaries of robot autonomy and adaptability.
The methodology proposed in SensorPerch is a comprehensive and well-integrated system that addresses a critical limitation in robotic perception. The core idea of decoupling sensing from the robot's embodiment and fixed infrastructure by treating sensors as autonomous, reconfigurable physical entities is genuinely novel. The system comprises both robust hardware and an intelligent software framework. The hardware design is modular, lightweight (201.4g), and self-contained, featuring an onboard Raspberry Pi 4B for computation, a 2-DoF motorized gimbal for viewpoint adjustment, a compact battery for independent operation (2.25 hours), and a vacuum-based attachment mechanism for diverse surfaces. The integration of a charging dock on the robot's mobile base for autonomous recharging is a practical and essential detail for long-term operation. The cost-effectiveness ($94) is also a significant advantage. The software framework, "Where to Perch," is equally sophisticated. It leverages a hybrid scene representation combining semantic SLAM for geometric reasoning and mountable surface identification, and a radiance-field model (NeRF) for novel view synthesis. This allows for efficient evaluation of candidate viewpoints in simulation without physical deployment. Viewpoint sampling is intelligently biased towards task-relevant regions using a VLM. The task-conditioned scoring function is a flexible abstraction, instantiated for two critical use cases: object-coupled perception (using VLM queries for object state detection) and policy-coupled perception (using DINOv2 embeddings to match a policy's training distribution). Finally, the system incorporates model-based pose estimation and grasp-and-place primitives for autonomous detachment and reattachment of the sensor platforms. The overall methodology presents a complete, closed-loop system for active, reconfigurable perception.
The experimental evaluation is thorough and compelling, covering platform capabilities, task performance, and system efficiency. 1. **Platform Evaluation:** Quantitative metrics for attachment accuracy (0.42 cm, <1 degree), battery life (2.25 hours), and thermal performance demonstrate the hardware's robustness and suitability for real-world, long-horizon tasks. 2. **Task Evaluation:** This is the strongest part of the evaluation. * **Object-coupled perception:** Tested across five diverse scenarios (pot boiling, human fall, fire hazard, cup dropped, water leakage), SensorPerch consistently achieves high event detection success rates, closely matching oracle performance and significantly outperforming baselines like "wrist camera only" or "random feasible placement." This validates its ability to provide persistent, task-relevant perception beyond the robot's immediate vicinity. * **Policy-coupled perception:** Evaluated on three manipulation tasks (peeling, cutting, pouring), SensorPerch achieves success rates (80-90%) comparable to oracle viewpoints, demonstrating its capability to construct policy-specific views that align with training data distributions. This is a crucial validation for enabling robust policy execution in varied settings. * **Baselines:** The comparison against a comprehensive set of baselines (w/o 2-DoF, random, oracle, wrist-only, fixed third-person) rigorously isolates the contributions of different components and the overall system. * **Multi-platform demonstration:** The real-world demonstration with three platforms simultaneously monitoring different aspects of a breakfast preparation task (pot, cutting, human's cup) is highly impressive and showcases the system's scalability and practical utility in complex, multi-task scenarios. 3. **System Evaluation:** The latency breakdown provides valuable insights into the computational overhead. While initial scene reconstruction is time-consuming (369s), it's a one-time cost. Viewpoint sampling, synthesis, and scoring are performed in seconds (0.91s to 7.95s), and the full detach-reattach loop takes a reasonable 12.53 seconds. Streaming latency (180ms) is suitable for real-time perception, even with multiple platforms. These metrics confirm the system's practical usability.
The paper provides detailed descriptions of the hardware components, including specific models (Raspberry Pi 4B, MG90S servos, RealSense D435i, 18650 battery) and their integration. The software stack outlines the use of semantic SLAM, radiance fields (NeRF), VLM, DINOv2, and ROS. A lightweight ROS package is mentioned, which is a good step towards reproducibility. However, the absence of a public code repository or specific details on the semantic SLAM and NeRF implementation (e.g., which specific open-source projects were used, training data for NeRF, VLM model used) makes full replication challenging without further information. The experimental setup is well-described, but the exact datasets for policy training and VLM queries would be needed.
The authors acknowledge several pertinent limitations: 1. **Surface Dependence:** The vacuum-based mounting mechanism relies on the availability of flat, non-porous, rigid surfaces. This limits deployment in environments with soft, porous, or highly irregular textures (e.g., carpets, fabric, rough stone). 2. **Vacuum Reliability:** The long-term reliability of vacuum seals under various environmental conditions (dust, temperature changes, vibrations) is a potential concern, although the paper demonstrates robust attachment in its experiments. 3. **Robot Motion and Occlusions:** The current framework does not explicitly model how the robot's own motion might create or resolve occlusions, or how it affects viewpoint quality over time. This could lead to suboptimal placements if the robot itself becomes an occluder. Additional limitations could include: 4. **Scalability to Many Sensors:** While a three-sensor demo is shown, managing and coordinating a much larger fleet of reconfigurable sensors could introduce new challenges in terms of communication, power management, and optimal placement strategies. 5. **Initial Scene Reconstruction Time:** The 369-second scene reconstruction time, while a one-time cost, might be prohibitive for highly dynamic environments or those requiring frequent re-mapping.
SensorPerch represents a significant step towards more versatile and autonomous robotic systems. By decoupling perception from fixed infrastructure and robot embodiment, it enables robots to adapt their sensing capabilities dynamically to task requirements, leading to several broader impacts: 1. **Enhanced Robot Autonomy:** Robots can operate more independently in diverse, unstructured environments without relying on pre-installed cameras or human intervention for viewpoint adjustment. 2. **Improved Task Performance:** The ability to construct task-optimal viewpoints on demand can lead to higher success rates in complex manipulation, monitoring, and interaction tasks, especially in scenarios requiring persistent observation or specific perspectives. 3. **New Active Perception Paradigm:** This work establishes a new paradigm for active perception, shifting from merely adjusting onboard sensors to physically reconfiguring the sensing infrastructure itself. This could inspire future research into modular, reconfigurable robotic components beyond just sensors. 4. **Applications in Diverse Fields:** Beyond household robotics, this technology could be transformative for applications in industrial inspection, disaster response (deploying sensors in hazardous areas), construction, and even scientific exploration, where dynamic, adaptive sensing is crucial. 5. **Reduced Infrastructure Costs:** By allowing a single robot to deploy and manage its own sensing network, it could reduce the need for expensive, permanently installed camera systems or multi-robot coordination for comprehensive coverage. SensorPerch introduces a novel paradigm for active perception by treating sensors as independent, reconfigurable physical entities that robots can autonomously deploy and manage. This comprehensive system, integrating robust hardware with an intelligent software framework for task-conditioned viewpoint selection, significantly enhances robotic perception capabilities, enabling persistent object-state detection and policy-aligned viewpoints in diverse real-world tasks, thereby pushing the boundaries of robot autonomy and adaptability.
Robust motion planning in dense traffic requires autonomous vehicles to interact in rare and safety-critical scenarios that are underrepresented in naturalistic driving data. Although adversarial training offers a feasible solution, existing methods often rely on external scenario generators, heuristic perturbations, or simulator-heavy rollouts, which makes them difficult to integrate with modern autoregressive planners. Here, we cast adversarially robust planner learning as a constrained min-max game and propose Adversarial World Modeling (AWM), a theoretically grounded multi-agent self-play fine-tuning framework. Since solving the exact game is intractable, AWM introduces a principled decoupled solver. In the inner minimization, the planner's predictive world model is converted into a role-conditioned adversary that learns sparse, scene-adaptive attack coalitions via counterfactual credit assignment. In the outer maximization, the ego planner optimizes a regret-aware robust best response against the frozen AWM, utilizing tail-risk weighting and reference-anchored trust regions to improve hard-case recovery while preserving nominal driving behavior. Experiments on the nuPlan and InterPlan benchmarks demonstrate that our method generates transferable adversarial interactions and yields a robust planner that achieves competitive closed-loop performance in both nominal and highly interactive long-tail scenarios. Theoretical analysis justifies the decoupled solver and the main optimization components.
Primary: unknown (information not provided in the paper text)
All Institutions: unknown (information not provided in the paper text)
This work has significant broader impact for autonomous driving and multi-agent reinforcement learning: * **Enhanced AV Safety**: By systematically exposing and mitigating vulnerabilities in rare, safety-critical scenarios, AWM contributes directly to improving the robustness and safety of autonomous vehicles in dense and interactive traffic. * **Robust ML for Robotics**: The framework provides a principled approach to adversarial training for complex, high-dimensional robotic systems, particularly those relying on autoregressive generative models. * **Multi-Agent Self-Play**: The novel counterfactual credit assignment and sparse coalition learning mechanisms advance the state-of-the-art in multi-agent self-play, offering solutions to long-standing credit assignment problems. * **Efficient Adversarial Testing**: AWM's ability to generate transferable adversarial interactions makes it a valuable tool for stress-testing and evaluating the robustness of various existing and future motion planners, potentially accelerating development and validation cycles. * **Integration with Modern Architectures**: By aligning with the autoregressive, tokenized paradigm, AWM offers a practical and scalable solution for robustifying modern generative planners, bridging a gap where traditional adversarial methods struggle. This paper introduces Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework that repurposes a planner's predictive world model as a role-conditioned adversary to robustify autoregressive motion planning. The framework's principled decoupled solver, novel counterfactual credit assignment, and regret-aware constrained optimization provide a theoretically grounded and empirically validated approach to generate transferable adversarial interactions, significantly improving planner robustness in long-tail scenarios while preserving nominal driving performance.
The paper proposes Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework for robust motion planning, framed as a constrained min-max game. The core innovation lies in repurposing the planner's own predictive world model as a role-conditioned adversary, which is a highly elegant and integrated solution compared to external scenario generators or heuristic perturbations. The methodology is theoretically grounded and addresses three key challenges: 1. **Multi-agent credit assignment**: AWM introduces a novel counterfactual credit assignment mechanism via role switching. By comparing factual active coalitions with leave-one-out and full background counterfactuals, it efficiently attributes scene-level utility changes to individual and synergistic adversarial behaviors. This is a significant improvement over naive reward sharing. 2. **Min-max optimization instability**: The paper proposes a principled decoupled solver. Stage A learns a frozen adversarial world model (AWM) that generates sparse, scene-adaptive attack coalitions. Stage B then optimizes the ego planner as a robust best response against this *frozen* AWM. This separation effectively mitigates the instability of simultaneous non-convex min-max updates. 3. **Nominal performance degradation**: The planner optimization in Stage B incorporates a regret-aware objective with tail-risk weighting (CVaR) to prioritize rare but critical failures, and crucially, reference-anchored trust regions with dual variables to preserve nominal driving behavior and enforce safety constraints. This ensures robustness is gained without sacrificing general driving competence. The detailed mechanisms for adversarial world modeling, including role-conditioned generation, sparse coalition learning (via proposal probing, primary selection, conditional pairing, and scene-adaptive calibration), are well-articulated and designed to generate plausible yet challenging multi-agent interactions. The use of a shared autoregressive tokenized rollout structure for both planning and adversarial generation ensures seamless integration. The theoretical analysis, mentioned in the abstract and contributions, provides justification for the decoupled solver and optimization components, adding significant rigor.
The experimental evaluation is comprehensive and rigorous, addressing three key questions: 1. **Closed-loop planning performance**: AWM-Planner is evaluated on the nuPlan and InterPlan benchmarks, including a challenging InterPlan-LongTail subset. It demonstrates competitive performance, significantly improving on interaction-heavy long-tail scenarios (e.g., +5.81 on InterPlan-LongTail) while maintaining or slightly improving nominal driving behavior on nuPlan (e.g., +2.60 on Test14-hard NR, -0.01 on Test14-random NR). This validates the method's ability to improve robustness without sacrificing general performance. 2. **Transferability of AWM as an adversarial simulator**: A crucial experiment deploys the learned AWM as a non-ego sim-agent traffic model for *seven different planners* (including rule-based, optimization-based, diffusion-based, and transformer-based). AWM consistently reduces the closed-loop score for all planners, demonstrating that the learned adversarial behaviors generalize beyond the planner it was trained against. This highlights AWM's potential as a valuable tool for stress-testing and robustifying various planning architectures. 3. **Component analysis (ablations)**: Extensive ablations are performed on both the AWM (Stage A) and planner (Stage B) components. * **AWM ablations**: Show that role-conditioned counterfactual credit assignment is superior to global reward broadcast, yielding stronger lower-tail degradation without sacrificing realism. The adaptive calibrated coalition learning is shown to provide sufficient adversarial stress while maintaining realism, outperforming forced K=2 attacks or simpler single-candidate approaches. * **Planner adaptation ablations**: Demonstrate the superiority of the decoupled self-play pipeline over planner-only refinement, simultaneous alternating, or alternating warmup, in terms of adversarial Tail-CVaR, closed-loop scores, and stability (lower GPU-hours, faster plateau, more stable seeds). Ablations on the Stage B objective confirm the importance of regret-aware CVaR for targeting rare severe risks and normal retention for preserving nominal performance. The use of diverse metrics (R-Score, progress, collision, TTC, drivable-area, speed-limit, driving-direction, Tail-CVaR) provides a holistic view of performance. Case studies visually illustrate the planner's improved recovery in adversarial scenarios and the nature of AWM-generated attacks.
The paper provides a high level of detail in its methodology section, including mathematical formulations for the game, decoupled solver, credit assignment, and planner objectives. It explicitly mentions appendices for "implementation details," "theoretical analysis," "architectural details," "probe signals and coverage metrics," "details for $f_pri$ and $f_pair$," "full AWM training objective," "analytical derivations and loss formulations," and "how the learned AWM is instantiated as a closed-loop sim-agent evaluator." The algorithm is summarized in Algorithm 1 in the appendix. This comprehensive documentation suggests a strong commitment to reproducibility.
The authors acknowledge several limitations: 1. **Fidelity of the tokenized world model**: AWM's effectiveness is inherently tied to the fidelity of the underlying tokenized world model. If the world model cannot accurately represent complex multi-agent dynamics, the generated adversarial scenarios may be unrealistic or ineffective. 2. **Exhaustive scenario coverage**: While AWM generates scene-adaptive attacks, it does not guarantee exhaustive coverage of all possible safety-critical scenarios. This is a common challenge in adversarial training for complex environments. 3. **Formal closed-loop safety guarantees**: The framework does not provide formal, provable safety guarantees, which remains a grand challenge for learning-based autonomous driving systems. 4. **Computational cost**: Although the decoupled solver improves stability and efficiency compared to simultaneous training, training large autoregressive models for both the world model and planner can still be computationally intensive.
This work has significant broader impact for autonomous driving and multi-agent reinforcement learning: * **Enhanced AV Safety**: By systematically exposing and mitigating vulnerabilities in rare, safety-critical scenarios, AWM contributes directly to improving the robustness and safety of autonomous vehicles in dense and interactive traffic. * **Robust ML for Robotics**: The framework provides a principled approach to adversarial training for complex, high-dimensional robotic systems, particularly those relying on autoregressive generative models. * **Multi-Agent Self-Play**: The novel counterfactual credit assignment and sparse coalition learning mechanisms advance the state-of-the-art in multi-agent self-play, offering solutions to long-standing credit assignment problems. * **Efficient Adversarial Testing**: AWM's ability to generate transferable adversarial interactions makes it a valuable tool for stress-testing and evaluating the robustness of various existing and future motion planners, potentially accelerating development and validation cycles. * **Integration with Modern Architectures**: By aligning with the autoregressive, tokenized paradigm, AWM offers a practical and scalable solution for robustifying modern generative planners, bridging a gap where traditional adversarial methods struggle. This paper introduces Adversarial World Modeling (AWM), a multi-agent self-play fine-tuning framework that repurposes a planner's predictive world model as a role-conditioned adversary to robustify autoregressive motion planning. The framework's principled decoupled solver, novel counterfactual credit assignment, and regret-aware constrained optimization provide a theoretically grounded and empirically validated approach to generate transferable adversarial interactions, significantly improving planner robustness in long-tail scenarios while preserving nominal driving performance.
The action space poses a major challenge in robot learning, since it is often high-dimensional, can span long time horizons, and frequently admits multi-modal optimal solutions. A good choice of action representation and loss function can help to address these concerns, but there are often trade offs. We propose Action Map Policy (AMP), which casts 3D closed-loop manipulation policy learning as a classification problem in image space. While classification has been an effective formulation in generative language models, applying it to robot action learning is difficult because naively discretizing high-dimensional continuous actions explodes the token vocabulary. Our key idea is to project 3D actions onto the camera image planes and treat each pixel location as a discrete class, thus controlling dimensionality while retaining multi-modality. This method supports millimeter-level precision for high-dimensional actions without requiring a prohibitively large vocabulary, while preserving fine-grained pixel-wise visual signals. Furthermore, it can predict the entire action chunk in a single forward pass, avoiding complex noise scheduling and iterative denoising while achieving substantially faster inference than diffusion policies. Experiments on various manipulation tasks show that AMP outperforms strong baselines, achieving higher success rates, faster inference, and enhanced spatial reasoning.
Primary: Brown University
All Institutions: Brown University
The authors acknowledge several limitations: 1. **Jittering**: Minor jittering in generated actions, which could be improved with geometric consistency losses. 2. **Workspace Limitation**: The workspace is limited by calibrated camera coverage, preventing actions outside the observed scope. This could be alleviated with larger-FOV or dual in-hand cameras. 3. **Gripper Specificity**: While the method is flexible, the current keypoint design focuses on a parallel-jaw gripper. Adaptation would be needed for dexterous or bimanual setups. 4. **Future Directions**: The paper also points to future work, such as integrating Q-functions for reinforcement learning and exploring pre-trained vision-language models, indicating that the current work is a foundational step. BROADER IMPACT: AMP offers a significant contribution to robot learning by providing an effective and efficient method for closed-loop 3D manipulation. Its ability to handle high-dimensional continuous actions, multi-modality, and achieve high precision with fast inference has broad implications for deploying robots in complex real-world tasks. The classification-based approach, which aligns action representation with observation space, could inspire new architectures and training paradigms in visuomotor control. The enhanced spatial reasoning demonstrated, particularly the responsiveness to small visual cues, is critical for robust and precise robot operation. This method could accelerate the development of more capable and adaptable robotic systems, potentially impacting fields like manufacturing, service robotics, and assistive technologies. The explicit action distribution output also opens avenues for safer and more interpretable robot behavior, as well as easier integration with reinforcement learning and vision-language models. Action Map Policy (AMP) introduces a highly novel and effective framework for 3D closed-loop manipulation by casting action learning as a pixel classification problem using projected keypoint heatmaps. This approach elegantly resolves the challenges of high-dimensional continuous actions, multi-modality, and precision, outperforming strong baselines in both simulated and real-world tasks with significantly faster inference and superior spatial reasoning. The paper's robust methodology, comprehensive experimental validation, and clear articulation of its advantages make it a substantial contribution to the field of robotics and machine learning.
The paper proposes Action Map Policy (AMP), a novel framework that reformulates 3D closed-loop manipulation policy learning as a pixel classification problem. The core idea is to represent 3D end-effector poses as a set of 3D keypoints, project these keypoints onto multiple camera image planes, and then predict their 2D pixel locations as discrete classes (heatmaps). This approach elegantly addresses the challenges of high-dimensional continuous action spaces, combinatorial explosion in naive discretization, and multi-modality. The use of a cross-entropy loss on soft Gaussian heatmaps for supervision is well-justified, providing stable training and naturally modeling multi-modal distributions. The X-Net architecture, a multi-view U-Net encoder combined with a Multi-View Transformer and U-Net decoder, is specifically designed to process multi-view images and output dense action feature maps, preserving spatial structure. The heatmap-to-action conversion process, involving argmax pixel selection, triangulation, and geometric pose recovery, is clearly defined and robust. A significant advantage is the single-pass inference, which is substantially faster than iterative denoising methods like diffusion policies. The method also leverages equivariant data augmentation, which is a natural fit for its spatially aligned action representation. The theoretical precision analysis demonstrating millimeter-level accuracy at reasonable image resolutions is a strong methodological justification.
The experimental evaluation is comprehensive and rigorous, spanning both simulated and real-world settings. 1. **Precision Analysis**: A theoretical analysis quantifies the achievable precision of the heatmap representation at different image resolutions, showing that $224 \times 224$ images yield approximately 1mm translation and 1.3-degree rotation precision, which is crucial for manipulation. This is a strong empirical justification for the chosen representation. 2. **Simulated Experiments**: AMP is evaluated on six challenging MimicGen tasks, covering rigid and articulated objects with varying precision requirements. It is compared against four strong baselines: Diffusion Policy (DiffPo), ACT, OAT, and Motion Track. AMP significantly outperforms these baselines on five out of six tasks, with an average gain of 20.7% in success rate. 3. **Ablation Studies**: Ablations demonstrate the importance of the in-hand camera view, soft labels (Gaussian width $\sigma=2$), and equivariant data augmentation, all contributing positively to performance. 4. **Real-World Experiments**: Three long-horizon, complex real-world tasks (make-coffee, toast-bread, steam-egg) are used to evaluate closed-loop visuomotor policy learning. AMP shows a substantial performance margin (e.g., 50% higher success rate than DiffPo on 'make-coffee') and faster inference speed (13.80 ms vs. 93.53 ms for DiffPo). 5. **Spatial Reasoning**: Two tasks (grasp-target-cup, stack-target-block) are designed to test spatial reasoning, where a laser pointer indicates the target among identical objects. AMP achieves 100% success, demonstrating its ability to respond to fine-grained visual cues, unlike baselines that struggle with mode collapse. The experiments are well-designed, the baselines are strong, and the results consistently highlight AMP's superior performance, efficiency, and robustness.
The paper provides good details for reproducibility. Key implementation choices are described, including image resolution, network architecture (U-Net encoder, MVT, U-Net decoder, skip connections), number of channels, and the use of the first 8 out of 12 predicted timesteps. Hyperparameters like learning rate, training epochs, and GPU used are specified. Crucially, the paper includes pseudocode for both training and inference, which is highly valuable. Details on data preprocessing, augmentation (including equivariant augmentation), and the soft-label width ($\sigma=2$) are provided. The description of the keypoint definition and pose recovery is also clear. The authors state that they focus on policy performance without using pretrained vision encoders for any method, ensuring a fair comparison.
The authors acknowledge several limitations: 1. **Jittering**: Minor jittering in generated actions, which could be improved with geometric consistency losses. 2. **Workspace Limitation**: The workspace is limited by calibrated camera coverage, preventing actions outside the observed scope. This could be alleviated with larger-FOV or dual in-hand cameras. 3. **Gripper Specificity**: While the method is flexible, the current keypoint design focuses on a parallel-jaw gripper. Adaptation would be needed for dexterous or bimanual setups. 4. **Future Directions**: The paper also points to future work, such as integrating Q-functions for reinforcement learning and exploring pre-trained vision-language models, indicating that the current work is a foundational step. BROADER IMPACT: AMP offers a significant contribution to robot learning by providing an effective and efficient method for closed-loop 3D manipulation. Its ability to handle high-dimensional continuous actions, multi-modality, and achieve high precision with fast inference has broad implications for deploying robots in complex real-world tasks. The classification-based approach, which aligns action representation with observation space, could inspire new architectures and training paradigms in visuomotor control. The enhanced spatial reasoning demonstrated, particularly the responsiveness to small visual cues, is critical for robust and precise robot operation. This method could accelerate the development of more capable and adaptable robotic systems, potentially impacting fields like manufacturing, service robotics, and assistive technologies. The explicit action distribution output also opens avenues for safer and more interpretable robot behavior, as well as easier integration with reinforcement learning and vision-language models. Action Map Policy (AMP) introduces a highly novel and effective framework for 3D closed-loop manipulation by casting action learning as a pixel classification problem using projected keypoint heatmaps. This approach elegantly resolves the challenges of high-dimensional continuous actions, multi-modality, and precision, outperforming strong baselines in both simulated and real-world tasks with significantly faster inference and superior spatial reasoning. The paper's robust methodology, comprehensive experimental validation, and clear articulation of its advantages make it a substantial contribution to the field of robotics and machine learning.
Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.
Primary: Unknown
All Institutions: Unknown
Kaleido presents a compelling algorithm-hardware co-design that significantly accelerates Video Diffusion Transformers by exploiting latent space correlations, offering substantial speedups and energy savings while maintaining high generative quality, thereby addressing a critical bottleneck in the deployment of advanced video generation models.
The paper proposes "Kaleido," an algorithm-hardware co-design specifically targeting Video Diffusion Transformers (vDiTs). The core algorithmic insight is the exploitation of channel-wise spatiotemporal correlations in the latent space to enable computation skipping (reusing partial results). This is a significant methodological shift from standard sparse attention techniques used in LLMs, which do not account for the unique redundancy in video generation tasks. The hardware design features a systolic array-like accelerator with reconfigurable processing elements and a lightweight data dispatcher to handle the irregular sparsity introduced by the algorithm. The approach is technically sound, addressing the dominant bottleneck of self-attention in long-timestep diffusion processes. However, the novelty is somewhat incremental in the broader context of hardware acceleration, as algorithm-hardware co-design for transformers is a well-trodden path, though the specific application to vDiTs with this correlation-based reuse is a strong contribution.
The evaluation demonstrates up to 5.9x speedup and 16.0x energy savings compared to state-of-the-art accelerators across three mainstream vDiT models. The claim of preserving higher generative quality (>17 dB improvement in PSNR/SSIM relative to prior skipping methods) is a strong empirical result. The use of standard metrics and comparison against SOTA baselines provides credibility. However, the lack of specific details on the baseline models and the exact nature of the "state of the art" accelerators in the abstract limits the immediate assessability of the magnitude of the improvement. The results are promising and suggest significant practical utility for deploying vDiTs.
The paper is published on arXiv, and while the abstract provides a high-level overview, the full text would need to be scrutinized for hyperparameters, model configurations, and hardware specifications to ensure reproducibility. The mention of "lightweight" components suggests a focus on practical implementation, which often aids reproducibility if code is released. Without code availability explicitly stated in the abstract, reproducibility is assumed to be moderate to high based on the detailed algorithmic description implied.
The primary limitation is the reliance on the assumption that channel-wise spatiotemporal correlations are consistent across different vDiT architectures and video domains. The performance gain might vary depending on the specific video content and the complexity of the diffusion process. Additionally, the hardware design's flexibility (reconfigurable PEs) might come with overheads that are not fully detailed in the abstract. The comparison is against "state of the art accelerators," which may include specialized custom hardware that is not widely available, potentially skewing the perceived advantage.
This work has significant implications for the democratization of high-quality video generation by reducing the computational and energy barriers. It enables faster iteration for researchers and more accessible deployment for developers. The focus on energy efficiency also aligns with the growing need for sustainable AI. By addressing the specific bottlenecks of vDiTs, it paves the way for more complex and longer video generation tasks. Kaleido presents a compelling algorithm-hardware co-design that significantly accelerates Video Diffusion Transformers by exploiting latent space correlations, offering substantial speedups and energy savings while maintaining high generative quality, thereby addressing a critical bottleneck in the deployment of advanced video generation models.
Modern edge system-on-chips (SoCs) combine CPUs, integrated GPUs (iGPUs), and neural processing units (NPUs), yet existing LLM runtimes typically make coarse device-level decisions or optimize operators in isolation. As a result, they underutilize heterogeneous resources, particularly on unified-memory platforms where performance depends on both device placement and task-graph coordination. We present HeteroMosaic, a heterogeneity-first scheduling framework for edge LLM inference. HeteroMosaic first uses a heterogeneous roofline model to identify when combining iGPU and NPU execution is beneficial. It then decomposes inference into dependency-preserving micro-batches that expose cross-accelerator overlap and applies trace-guided co-optimization of scheduling and device allocation under practical effects such as memory contention, DVFS, device variation, and NPU runtime overheads. We implement HeteroMosaic in PyTorch C++ and evaluate it on three AMD Ryzen AI platforms spanning NPU-heavy, balanced, and iGPU-heavy designs. On the balanced platform, HeteroMosaic achieves up to 1.73X speedup over an iGPU baseline, 1.78X over an NPU baseline, and 2.05X over frameworks such as llama.cpp, while reducing energy by up to 45.3%. It also improves performance over prior heterogeneous edge AI solutions by up to 2.35X.
Primary: University of Illinois Urbana-Champaign (UIUC)
All Institutions: University of Illinois Urbana-Champaign (UIUC), Advanced Micro Devices, Inc. (AMD)
HeteroMosaic introduces a novel scheduling framework for edge LLM inference that significantly outperforms existing baselines by exploiting fine-grained heterogeneous execution opportunities on unified-memory SoCs, offering a practical path toward energy-efficient, high-performance edge AI deployment.
The paper proposes HeteroMosaic, a scheduling framework that treats edge LLM inference as a heterogeneity-first problem rather than a simple placement problem. The core methodological contributions include: 1) A heterogeneous roofline model to theoretically bound the benefits of combining iGPU and NPU execution; 2) Dependency-preserving micro-batching to expose cross-accelerator overlap opportunities that coarse-grained batching misses; 3) Trace-guided co-optimization of scheduling and device allocation that accounts for real-world system effects like memory contention, DVFS, and NPU runtime overheads. This approach is technically sophisticated, moving beyond static operator mapping to dynamic, graph-level coordination. The insight that fine-grained heterogeneity can be exploited via scheduling on unified-memory platforms is a significant shift in how edge AI systems are designed.
The evaluation is conducted on three distinct AMD Ryzen AI platforms (NPU-heavy, balanced, iGPU-heavy), providing strong empirical evidence for the framework's adaptability. The results show substantial improvements: up to 1.73x speedup over iGPU-only, 1.78x over NPU-only, and 2.05x over llama.cpp on a balanced platform, with up to 45.3% energy reduction. The comparison against strong baselines (including existing heterogeneous solutions) demonstrates clear superiority. The use of real hardware rather than simulation adds significant credibility. The evaluation covers multiple models, ensuring the findings are not model-specific artifacts.
The authors implement HeteroMosaic in PyTorch C++, which is a standard and accessible framework for such systems research. The paper details the architectural properties required (programmable accelerators, shared memory, low-overhead sync) and the specific optimizations applied. While the exact proprietary NPU driver details might be AMD-specific, the scheduling abstraction and the trace-guided optimization methodology are described with sufficient detail for replication on similar unified-memory SoCs. The codebase structure suggests it could be open-sourced or made available upon request, which is common for top-tier systems venues.
The current implementation is tightly coupled with AMD Ryzen AI platforms. The "heterogeneous roofline model" and scheduling heuristics may need significant re-tuning for other architectures (e.g., Qualcomm Hexagon, Apple Neural Engine) due to differences in NPU capabilities, memory bandwidth, and runtime overheads. The paper acknowledges that extending this to datacenter systems would require explicit communication modeling for PCIe/CXL/NVLink, which is a non-trivial extension not covered here. Additionally, the focus is on inference; training on such heterogeneous edge devices remains an open challenge.
This work has profound implications for the deployment of Large Language Models on edge devices, enabling faster and more energy-efficient AI on smartphones, laptops, and IoT devices. By demonstrating that fine-grained heterogeneity can be practically exploited, it encourages hardware designers to prioritize balanced accelerator designs and efficient interconnects. It also provides a blueprint for runtime systems to better utilize the increasingly heterogeneous compute resources available in modern SoCs, potentially reducing the carbon footprint of AI inference. HeteroMosaic introduces a novel scheduling framework for edge LLM inference that significantly outperforms existing baselines by exploiting fine-grained heterogeneous execution opportunities on unified-memory SoCs, offering a practical path toward energy-efficient, high-performance edge AI deployment.
Large language model (LLM) inference is increasingly limited by the capacity of High-Bandwidth Memory (HBM) in GPUs, as model weights and KV cache grow rapidly. High-Bandwidth Flash (HBF) provides higher capacity than HBM while retaining comparable bandwidth, making it a promising substrate for capacity-constrained LLM inference. However, its inherently high access latency, low bandwidth utilization, and lack of support for heterogeneous resource management make it difficult to integrate HBF into GPUs for LLM inference. We present FlashAccel, a co-designed system that enables efficient LLM inference using HBF. FlashAccel integrates HBF into HBM-based GPUs, providing architectural support to mitigate access latency. It improves bandwidth utilization through specialized data layouts for both model weights and KV cache, and introduces an HBF-aware storage management layer together with a programming model to organize persistent data in HBF and coordinate heterogeneous memory resources at the system level. Experimental results demonstrate that integrating six HBF stacks into the GPU enables FlashAccel to deliver an average improvement of 2.54$\times$ and 1.93$\times$ in throughput per GPU and energy efficiency over the HBM-only GPU under 100ms latency constraint, respectively.
Primary: Institute of Computing Technology, Chinese Academy of Sciences
All Institutions: Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Institute of Microelectronics, Chinese Academy of Sciences
FlashAccel presents a compelling hardware-software co-design for integrating High-Bandwidth Flash into GPUs, effectively addressing latency and bandwidth utilization challenges to deliver significant throughput and energy efficiency gains for LLM inference.
The paper proposes FlashAccel, a hardware-software co-design that integrates High-Bandwidth Flash (HBF) into GPU architectures to address the memory capacity bottleneck in LLM inference. The methodology is rigorous and addresses three specific challenges: high access latency, low bandwidth utilization due to plane-level parallelism constraints, and heterogeneous resource management. Architecturally, it introduces SRAM buffers for prefetching. At the data layout level, it designs specialized layouts for static weights (based on execution order) and dynamic KV cache (using a unified view with selective offloading to balance plane loads). System-wise, it eliminates the Flash Translation Layer (FTL) by leveraging append-only patterns and introduces a programming model with unified virtual addressing and hyper-page granularity. The approach is technically sound and directly targets the inefficiencies of using non-volatile memory for high-bandwidth workloads.
The evaluation is conducted using an event-driven simulator built on LLMCompass and ScaleSim, extended with a NAND simulator. The authors evaluate against a baseline DGX-H200 node across multiple large models (LLaMA3.1-405B, Qwen3-235B, etc.) and sequence lengths. Results show a 2.54x throughput improvement and 1.93x energy efficiency gain under a 100ms latency constraint. The ablation study effectively isolates the contribution of each component (prefetch, weight layout, KV cache layout). The analysis of KV cache reuse in multi-turn interactions provides strong motivation for the capacity argument. The endurance analysis is realistic, acknowledging the trade-offs between retention time and P/E cycles.
The paper provides detailed architectural parameters (plane counts, latency figures, SRAM sizes) and simulation methodology. However, as a simulator-based study relying on specific extensions to LLMCompass/ScaleSim, full reproducibility requires access to the custom simulation code, which is not explicitly linked. The hardware assumptions (e.g., specific HBF stack designs) are based on cited prototypes, making the results dependent on the accuracy of those models.
The primary limitation is that the work is simulation-based; no physical HBF prototype or silicon validation is presented. The performance gains rely heavily on the assumption that HBF can achieve the stated bandwidth and latency characteristics, which are still emerging technologies. The analysis assumes a specific integration method (CSI/CLI) and does not explore other potential integration topologies. Additionally, the endurance analysis, while optimistic, still relies on SLC Flash, which may have cost implications compared to MLC/TLC if endurance can be sufficiently managed.
This work has significant implications for the future of LLM serving infrastructure. As model sizes and context lengths continue to grow, HBM capacity and cost will remain critical bottlenecks. FlashAccel demonstrates a viable path to leveraging emerging non-volatile memory technologies to decouple capacity from bandwidth constraints, potentially reducing the cost per token for large-scale inference. It also provides valuable insights for system architects designing next-generation accelerators with heterogeneous memory hierarchies. FlashAccel presents a compelling hardware-software co-design for integrating High-Bandwidth Flash into GPUs, effectively addressing latency and bandwidth utilization challenges to deliver significant throughput and energy efficiency gains for LLM inference.