Last 7 Days (October 08 – October 14, 2026)
Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.
Primary: Northeastern University
All Institutions: Northeastern University, Microsoft Research
AutoEnvScaling automates the generation of RL training environments by treating environment design as a terminal agent task, enabling a recursive self-improvement loop where agents co-evolve as both task designers and solvers. The paper demonstrates that this approach significantly increases the rate of valid environment generation and improves downstream agent performance on terminal benchmarks, offering a scalable path toward reducing human reliance in agent training pipelines.
The paper proposes AutoEnvScaling, a framework that reframes the generation of RL training environments as a terminal agent task. Instead of using static pipelines or simple prompt-based generation, the method employs a "proposer" agent operating within a sandboxed terminal environment. This agent reads solver rollouts, identifies failure modes, and iteratively builds, tests, and calibrates new Harbor tasks (instructions, environments, reference solutions, tests). The core innovation is the "data flywheel" where the solver's performance directly informs the proposer's next environment generation, and the proposer itself can be trained (via DPPO) to improve at environment design. This creates a recursive self-improvement (RSI) loop where a single model can co-evolve as both the task designer and the solver. The methodology is technically sound, leveraging existing RL techniques (DPPO) but applying them to a novel multi-agent, self-play context with distinct reward structures for design vs. solving.
The experiments are extensive and well-controlled. The authors compare their method against prompt-only and feedback-driven baselines across eight frontier models, showing a 25x increase in valid environment rates. They demonstrate downstream utility by training Qwen3.5 models on AutoEnvScaling-generated environments, showing significant gains (4.3-7.4 points) on Terminal-Bench 2.1 over static Tmax baselines. The RSI experiment is particularly compelling, showing a 12.2-point gain when the model acts as both proposer and solver. The inclusion of a qualitative study on "learning behavior" (asking for help) further validates the quality of the generated curricula. The use of matched baselines (regenerating Tmax pools with the same proposer) ensures fair comparison.
The paper provides detailed descriptions of the Harbor framework, the sandbox setup, and the validation/calibration criteria. However, specific hyperparameters for the DPPO training and the exact implementation details of the "harness optimization" are relegated to appendices or referenced generally. The reliance on proprietary frontier models (GPT-5.6, Claude Opus 5) for the proposer role limits immediate reproducibility for those without access, though the open-weight Qwen experiments provide a path for replication. The code and environment definitions are not explicitly linked in the provided text, which is a minor gap.
The primary limitation is the heavy reliance on strong frontier models to bootstrap the proposer role, which may not be accessible to all researchers. The RSI results, while impressive, are demonstrated on a specific model family (Qwen) and benchmark (Terminal-Bench), leaving open questions about generalization to other domains or model architectures. The paper also acknowledges that the validation checks cannot catch all types of errors (e.g., shared mistakes between reference solution and tests), which could lead to subtle reward hacking if not carefully monitored.
This work has significant implications for the scalability of RL training for agents. By automating the most labor-intensive part of the RL pipeline (environment design), it could drastically reduce the cost and time required to train capable agents. The concept of RSI through self-generated curricula is a major step toward autonomous AI improvement, with potential applications in scientific discovery, software engineering, and other complex problem-solving domains. It shifts the paradigm from human-designed curricula to agent-designed curricula, potentially enabling agents to discover novel skills and strategies that humans might not anticipate. AutoEnvScaling automates the generation of RL training environments by treating environment design as a terminal agent task, enabling a recursive self-improvement loop where agents co-evolve as both task designers and solvers. The paper demonstrates that this approach significantly increases the rate of valid environment generation and improves downstream agent performance on terminal benchmarks, offering a scalable path toward reducing human reliance in agent training pipelines.
A predictor is multicalibrated with respect to a class of subpopulations (formally, of functions on the inputs, called auditors) if, for every subpopulation in the class and every value the predictor outputs, the average outcome among the members of that subpopulation who receive that value is close to the value itself. For finite classes, about $\varepsilon^{-3}$ samples are necessary and sufficient to learn a predictor with multicalibration error $\varepsilon$, but the algorithms that attain this rate enumerate the class, which is impossible for the large classes of interest. We ask how many samples suffice when the learner accesses the class only through an empirical risk minimization (ERM) oracle. We give an algorithm that uses $\tilde{O}(d/\varepsilon^{3})$ samples and polynomially many ERM calls, where $d$ is the VC dimension of the class. Its dependence on $\varepsilon$ is optimal for all learners. When $d$ is at least about $1/\varepsilon$, every learner that can only access the class through polynomially many ERM calls needs $\tildeΩ(d/\varepsilon^{3})$ samples. Learners with unlimited computation need only $\tilde{O}(\varepsilon^{-3}+d\varepsilon^{-2})$ samples, so the extra factor $1/\varepsilon$ on the dimension term is the price of oracle efficiency. For swap multicalibration, which allows a different function from the class for each predicted value, $\tildeΘ(d/\varepsilon^{3})$ samples are necessary and sufficient. For estimating the error of a given predictor, the ordinary notion is the harder one: the swap error can be estimated with an ERM oracle, whereas the ordinary error is NP-hard to approximate even for classes with linear-time ERM. Consequently, there are classes for which it is computationally easier to learn a multicalibrated predictor than to compute the multicalibration error of a given predictor.
Primary: Harvard University
All Institutions: Harvard University
The paper resolves key open questions regarding the sample complexity of multicalibration under oracle efficiency constraints, providing tight upper and lower bounds that define the theoretical limits of fair learning algorithms. It introduces novel techniques combining Hadamard tests with hybrid online learning to achieve optimal rates, and demonstrates the computational hardness of auditing multicalibration error, offering profound insights into the trade-offs between statistical efficiency and computational feasibility in fairness-aware machine learning.
The paper presents a rigorous theoretical framework for understanding the sample complexity of multicalibration when the learner is restricted to using an Empirical Risk Minimization (ERM) oracle. The core methodological contribution is the development of an algorithm that achieves the optimal $\tilde{O}(d/\varepsilon^3)$ sample complexity for ordinary multicalibration, where $d$ is the VC dimension. This is achieved by replacing the exponential number of sign patterns required in previous enumeration-based approaches with a polynomial number of "signed tests" derived from Hadamard matrices. The authors cleverly combine this with a high-probability regret bound for Gaussian Follow-the-Perturbed-Leader (FTPL) in a hybrid online learning setting (random inputs, adversarial feedback). The paper also provides matching lower bounds, proving that this sample complexity is optimal for any learner making polynomially many ERM calls, thereby establishing a clear separation between oracle-efficient learning and enumeration-based learning. The extension to swap multicalibration and the analysis of error estimation (showing NP-hardness for ordinary error estimation) are also technically sound and well-supported.
As a pure theory paper, there are no empirical experiments. The "evaluation" consists of rigorous mathematical proofs of upper and lower bounds on sample complexity. The results are definitive within the theoretical model defined (VC dimension, ERM oracle access, distribution-free setting).
The paper is fully reproducible in the sense that the algorithms are described in sufficient detail for implementation, and the theoretical guarantees are proven. The algorithms rely on standard oracles (ERM) which are well-defined. The deterministic predictor construction requires a computable total order on the input space, which is a standard assumption in such contexts.
The results are theoretical and do not provide empirical validation on real-world datasets. The lower bounds rely on the assumption that the learner accesses the class only through polynomially many ERM calls; if a learner had unlimited computational power to enumerate the class (if finite), better rates are possible, but this is often infeasible for large classes. The NP-hardness result for error estimation applies to the worst-case scenario for specific classes, which may not reflect typical practical instances.
This paper significantly advances the understanding of the computational and statistical limits of fair machine learning. By establishing the optimal sample complexity for oracle-efficient multicalibration, it provides a benchmark for future algorithmic developments. The separation between learning and error estimation has important implications for the auditability of ML models, suggesting that verifying a model's fairness may be computationally harder than training a fair model. This insight is crucial for regulatory frameworks and practical deployment of fair AI systems. The paper resolves key open questions regarding the sample complexity of multicalibration under oracle efficiency constraints, providing tight upper and lower bounds that define the theoretical limits of fair learning algorithms. It introduces novel techniques combining Hadamard tests with hybrid online learning to achieve optimal rates, and demonstrates the computational hardness of auditing multicalibration error, offering profound insights into the trade-offs between statistical efficiency and computational feasibility in fairness-aware machine learning.
Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up. To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as introspective deception, the ground truth can only be determined through careful elicitation or thorough knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. Our probes also readily detect deception on prominent open-weight models which lie about politically sensitive topics, and about their beliefs when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.
Primary: FAR.AI
All Institutions: FAR.AI, Coefficient Giving
The paper presents a highly effective white-box probe architecture and dataset for detecting LLM deception, outperforming black-box monitors in agentic and introspective settings. It provides a rigorous, scalable solution to the critical problem of monitoring frontier AI agents for misaligned behavior, with significant implications for deployment safety.
The paper introduces a rigorous framework for white-box deception detection, distinguishing between "contextual falsehoods" (detectable from context) and "introspective falsehoods" (requiring knowledge of model beliefs). The core methodological contribution is the "FIBS" dataset (163k transcripts) and a novel probe architecture that applies factored causal attention over the layer-token grid, allowing for aggregation of information across many layers and tokens. This is a significant step up from standard linear probes or single-layer MLPs. The use of "contrast scores" (prefilling "Yes" vs "No" and measuring the difference) is a clever practical technique to mitigate context-specific biases in probe outputs.
The experimental setup is robust and comprehensive. The authors evaluate on SHADE-Arena, a standard benchmark for agentic sabotage, showing that their probes outperform a strong black-box baseline (Opus 5.5) with much lower computational cost. Crucially, they test generalization to "introspective" settings, including model organisms with hidden goals and politically sensitive topics where black-box monitors are structurally blind. The results showing high AUC (up to 99.7%) in distinguishing true hidden goals from false denials are particularly strong, addressing a key gap in prior work (e.g., Cooney et al. 2026).
The paper commits to releasing the FIBS dataset and trained probes. Detailed hyperparameters and implementation details are provided in the appendices. The use of standard open-weight models (Qwen, Nemotron) and established benchmarks (SHADE-Arena, MMLU) enhances reproducibility. The label noise estimation (1.4%) via LLM-as-a-judge is a transparent approach to data quality.
The primary limitation is the reliance on the model's internal representation of "belief." If a model's persona shifts such that it genuinely "believes" its deceptive output (due to training dynamics), the probe may fail. The paper acknowledges this with the "math-sandbagger" organism case. Additionally, the dataset is heavily skewed towards off-policy data, and while they test on-policy generalization, the gap between synthetic training data and real-world deployment nuances remains a concern.
This work has high impact on AI safety monitoring. By demonstrating that cheap, white-box probes can match or exceed expensive black-box monitors, it lowers the barrier to entry for effective safety monitoring in frontier deployments. The release of the FIBS dataset will likely become a standard resource for training deception detectors. The findings support the viability of interpretability-based safety measures over purely behavioral monitoring. The paper presents a highly effective white-box probe architecture and dataset for detecting LLM deception, outperforming black-box monitors in agentic and introspective settings. It provides a rigorous, scalable solution to the critical problem of monitoring frontier AI agents for misaligned behavior, with significant implications for deployment safety.
Generating an image with a diffusion or flow-matching model typically involves integrating a velocity field for hundreds of sequential steps. That computation suits a GPU poorly and an analog neuromorphic substrate well, where one fast, low-power physical relaxation could replace the whole sequence. Two things are needed: a generative algorithm whose computation is a relaxation, and a way to train it without backpropagation, whose global gradient transport is hard to realise in analog. Energy Matching supplies the first, sampling by gradient-descent relaxation on a time-independent potential - but obtaining the gradient that descent follows is itself a backward pass, once per integration step. We prove that a predictive coding network with a single linear injection term added to its energy relaxes to a state whose visible layer carries that gradient inherently, with an error that vanishes for small injection strengths. An Energy Matching model trained by backpropagation can therefore be loaded into this network unchanged and sampled by relaxation alone, matching the original model's FID to within 0.04 at CIFAR-10 and ImageNet-32 scale. The network can also be trained without backpropagation: Equilibrium Propagation reproduces backpropagation's parameter updates to a cosine similarity of 0.9998, and training end to end with those updates reaches an FID of 3.64 on unconditional CIFAR-10 against 3.54 for backpropagation. To our knowledge this is the first CIFAR-10 generative model trained with Equilibrium Propagation, and the first at competitive quality by any backpropagation-free rule. This quality does not require group normalisation or attention: an architecture of only convolutions and pointwise nonlinearities, both native to an analog crossbar, reaches an FID of 3.51 when trained by backpropagation.
Primary: University of Cambridge
All Institutions: University of Cambridge, Nokia Bell Labs
The paper presents a rigorous method for training CIFAR-scale generative models using Equilibrium Propagation, achieving competitive FID scores without backpropagation. It bridges the gap between energy-based generative models and neuromorphic hardware by proving that predictive coding networks can inherently compute the gradients required for sampling and training via local relaxation dynamics.
The paper proposes a rigorous theoretical and practical framework for training generative models without backpropagation, specifically targeting analog neuromorphic hardware. The core contribution is the "PotentialPCN," a predictive coding network (PCN) constructed from a standard feedforward energy matching model. The authors prove that by adding a single linear injection term to the energy of the PCN, the relaxed state of the network inherently computes the gradient of the potential (the velocity field required for sampling) with an error that vanishes as the injection strength approaches zero. This allows for sampling via physical relaxation without a backward pass. Furthermore, they demonstrate that Equilibrium Propagation (specifically a GradEP variant with a linearized nudge) can train these models end-to-end, reproducing backpropagation gradients with high fidelity (cosine similarity 0.9998). The methodology is mathematically sound, with clear proofs provided in the appendix regarding the correspondence between the PCN relaxation and the feedforward gradient.
The experiments are comprehensive for the scope of the claim. The authors validate the inference correspondence by showing that the PCN relaxation matches the feedforward gradient to machine precision (TF32 limits) across six decades of injection strengths. They demonstrate that an Energy Matching model trained with backprop can be sampled via PCN relaxation with negligible FID loss (within 0.04) on CIFAR-10 and ImageNet-32. Crucially, they show that training *with* Equilibrium Propagation yields competitive results: an FID of 3.64 on unconditional CIFAR-10, compared to 3.54 for backpropagation. They also provide an existence proof for hardware-plausible architectures (convolutions and pointwise nonlinearities only) achieving an FID of 3.51, demonstrating that attention and group normalization are not strictly necessary for competitive performance at this scale.
The paper provides a high level of reproducibility. The authors release code for training, sampling, and correspondence measurements. The training recipe, including hyperparameters for the Equilibrium Propagation phases (nudge strength, tether stiffness, sweep budgets), is detailed in the appendix. The distinction between the "implicit solve" (IFT) and "Equilibrium Propagation" (EP) arms is clearly defined, allowing readers to replicate the specific backprop-free training procedure.
The primary limitation is that all results are simulated on GPUs; no physical neuromorphic hardware was used, so the actual energy and latency benefits remain theoretical. The simulation cost of EP is significantly higher than backprop (16x), which is a major barrier for GPU-based research, though the paper argues this is mitigated on physical hardware. The scope is limited to CIFAR-10 for training; ImageNet-32 was only used for inference validation. Additionally, the "hardware-plausible" architecture was only trained with backprop, not with EP, leaving a gap in demonstrating the full pipeline on the simplified architecture.
This work is significant for the neuromorphic computing community as it provides a concrete, high-quality pathway to implement generative models on analog substrates. By removing the need for backpropagation during both training and inference (sampling), it addresses the two main hurdles for analog AI: global gradient transport and sequential computation. The finding that simple convolutional architectures suffice for competitive FID scores without attention is also valuable for the broader generative modeling field, suggesting that complexity may be overestimated for certain tasks. The paper presents a rigorous method for training CIFAR-scale generative models using Equilibrium Propagation, achieving competitive FID scores without backpropagation. It bridges the gap between energy-based generative models and neuromorphic hardware by proving that predictive coding networks can inherently compute the gradients required for sampling and training via local relaxation dynamics.
Advances in large language models (LLMs) have brought the vision of a general-purpose personal assistant closer to reality. Such assistants could provide a unified natural-language interface to a broad ecosystem of applications. However, real-world applications do not readily support training at scale because access is often restricted and their state is difficult to reset. To address this challenge, we introduce Terrarium, a framework that synthesizes executable App Worlds from real-world application information, generates training task packages with evaluator-side rubrics from these worlds, and iteratively validates and refines both through agent rollouts. For App World synthesis, Terrarium organizes public Web information and in-application observations into structured app specifications that guide the construction of self-contained runtimes with callable operations and mutable, resettable state. For task synthesis, Terrarium constructs tasks grounded in App World state together with evaluator-side outcome rubrics that assess success without prescribing reference trajectories. We construct a training suite of 275 App Worlds and 13,344 tasks for outcome-based reinforcement learning across three Qwen3 model scales (4B, 8B, and 14B parameters). Across these scales, models trained with Terrarium outperform the corresponding Qwen3 models by an average of 18.34 percentage points on the $τ^2$-bench and 10.94 percentage points on the AppWorld bench. These results suggest that app-grounded world and task synthesis offers a scalable path toward general-purpose assistants capable of operating across broad application ecosystems.
Primary: Fudan University
All Institutions: Beijing Tongming Lake Information Technology Application Innovation Center, Fudan University, Institute of Systems for Advanced Computing, Peking University, School of Computer Science, Simon Fraser University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a scalable framework for synthesizing executable app worlds and grounded training tasks to improve LLM agent performance, demonstrating significant gains on standard benchmarks through outcome-based reinforcement learning.
The paper introduces "Terrarium," a framework for synthesizing executable "App Worlds" from real-world application information (web search + GUI traces) to train LLM agents. The core methodological contribution is the creation of a structured intermediate representation (app specification) that guides the construction of self-contained, resettable runtimes with mutable state. This addresses the key bottleneck in agent training: the lack of scalable, realistic, and controllable environments. The pipeline involves synthesizing the world, generating grounded tasks with evaluator-side rubrics (rather than reference trajectories), and using a closed-loop validation process where agent rollouts identify and repair defects in the world, task, or evaluator. The use of Group Relative Policy Optimization (GRPO) with binary outcome rewards is a solid choice for this setting, allowing the model to learn from success/failure signals without requiring dense trajectory supervision. The distinction between "Intent Execution" and "Intent Completion" (handling missing information) is a thoughtful addition that adds nuance to the training signal.
The experimental evaluation is rigorous and comprehensive. The authors train Qwen3 models at three scales (4B, 8B, 14B) and evaluate them on two independent benchmarks: $\tau^2$-bench and AppWorld. The results show consistent and significant improvements over the base models (18.34 points on $\tau^2$-bench, 10.94 on AppWorld). The ablation studies are particularly strong, demonstrating that GRPO outperforms SFT, that combining Execution and Completion supervision is beneficial, and that performance scales with the number of synthesized App Worlds. The comparison with other synthesis methods (AWM, EnvScaler) further validates the effectiveness of the app-grounded approach. The inclusion of BFCL Multi-Turn evaluation provides additional evidence of generalization.
The paper provides extensive details on the training configuration, including hyperparameters, dataset statistics, and the judge prompt. The authors state they will open-source the framework and artifacts, which is a major plus for reproducibility. The detailed description of the closed-loop validation process and the failure taxonomy helps in understanding how the data quality is maintained. However, the actual code and data are not yet available, which is a minor limitation for immediate reproducibility.
The primary limitation is the reliance on a large, proprietary model (Qwen3.5-397B-A17B) as the evaluator-side judge. This introduces a potential bias and makes the method less accessible to smaller labs. Additionally, the synthesized worlds are surrogates and may not perfectly capture the complexity and edge cases of real-world applications. The method is also computationally expensive, requiring significant GPU resources for both synthesis and training.
This work has significant potential to impact the field of LLM agents by providing a scalable path to training general-purpose assistants. The ability to synthesize realistic, controllable environments from public information could accelerate research in agent learning and enable the development of more capable and reliable AI systems. The open-sourcing of the framework and data will likely foster further innovation in this area. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper presents a scalable framework for synthesizing executable app worlds and grounded training tasks to improve LLM agent performance, demonstrating significant gains on standard benchmarks through outcome-based reinforcement learning.
Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.
Primary: Northeastern University
All Institutions: Northeastern University, Microsoft Research
AutoEnvScaling automates the generation of RL training environments by treating environment design as a terminal agent task, enabling a recursive self-improvement loop where agents co-evolve as both task designers and solvers. The paper demonstrates that this approach significantly increases the rate of valid environment generation and improves downstream agent performance on terminal benchmarks, offering a scalable path toward reducing human reliance in agent training pipelines.
The paper proposes AutoEnvScaling, a framework that reframes the generation of RL training environments as a terminal agent task. Instead of using static pipelines or simple prompt-based generation, the method employs a "proposer" agent operating within a sandboxed terminal environment. This agent reads solver rollouts, identifies failure modes, and iteratively builds, tests, and calibrates new Harbor tasks (instructions, environments, reference solutions, tests). The core innovation is the "data flywheel" where the solver's performance directly informs the proposer's next environment generation, and the proposer itself can be trained (via DPPO) to improve at environment design. This creates a recursive self-improvement (RSI) loop where a single model can co-evolve as both the task designer and the solver. The methodology is technically sound, leveraging existing RL techniques (DPPO) but applying them to a novel multi-agent, self-play context with distinct reward structures for design vs. solving.
The experiments are extensive and well-controlled. The authors compare their method against prompt-only and feedback-driven baselines across eight frontier models, showing a 25x increase in valid environment rates. They demonstrate downstream utility by training Qwen3.5 models on AutoEnvScaling-generated environments, showing significant gains (4.3-7.4 points) on Terminal-Bench 2.1 over static Tmax baselines. The RSI experiment is particularly compelling, showing a 12.2-point gain when the model acts as both proposer and solver. The inclusion of a qualitative study on "learning behavior" (asking for help) further validates the quality of the generated curricula. The use of matched baselines (regenerating Tmax pools with the same proposer) ensures fair comparison.
The paper provides detailed descriptions of the Harbor framework, the sandbox setup, and the validation/calibration criteria. However, specific hyperparameters for the DPPO training and the exact implementation details of the "harness optimization" are relegated to appendices or referenced generally. The reliance on proprietary frontier models (GPT-5.6, Claude Opus 5) for the proposer role limits immediate reproducibility for those without access, though the open-weight Qwen experiments provide a path for replication. The code and environment definitions are not explicitly linked in the provided text, which is a minor gap.
The primary limitation is the heavy reliance on strong frontier models to bootstrap the proposer role, which may not be accessible to all researchers. The RSI results, while impressive, are demonstrated on a specific model family (Qwen) and benchmark (Terminal-Bench), leaving open questions about generalization to other domains or model architectures. The paper also acknowledges that the validation checks cannot catch all types of errors (e.g., shared mistakes between reference solution and tests), which could lead to subtle reward hacking if not carefully monitored.
This work has significant implications for the scalability of RL training for agents. By automating the most labor-intensive part of the RL pipeline (environment design), it could drastically reduce the cost and time required to train capable agents. The concept of RSI through self-generated curricula is a major step toward autonomous AI improvement, with potential applications in scientific discovery, software engineering, and other complex problem-solving domains. It shifts the paradigm from human-designed curricula to agent-designed curricula, potentially enabling agents to discover novel skills and strategies that humans might not anticipate. AutoEnvScaling automates the generation of RL training environments by treating environment design as a terminal agent task, enabling a recursive self-improvement loop where agents co-evolve as both task designers and solvers. The paper demonstrates that this approach significantly increases the rate of valid environment generation and improves downstream agent performance on terminal benchmarks, offering a scalable path toward reducing human reliance in agent training pipelines.
A predictor is multicalibrated with respect to a class of subpopulations (formally, of functions on the inputs, called auditors) if, for every subpopulation in the class and every value the predictor outputs, the average outcome among the members of that subpopulation who receive that value is close to the value itself. For finite classes, about $\varepsilon^{-3}$ samples are necessary and sufficient to learn a predictor with multicalibration error $\varepsilon$, but the algorithms that attain this rate enumerate the class, which is impossible for the large classes of interest. We ask how many samples suffice when the learner accesses the class only through an empirical risk minimization (ERM) oracle. We give an algorithm that uses $\tilde{O}(d/\varepsilon^{3})$ samples and polynomially many ERM calls, where $d$ is the VC dimension of the class. Its dependence on $\varepsilon$ is optimal for all learners. When $d$ is at least about $1/\varepsilon$, every learner that can only access the class through polynomially many ERM calls needs $\tildeΩ(d/\varepsilon^{3})$ samples. Learners with unlimited computation need only $\tilde{O}(\varepsilon^{-3}+d\varepsilon^{-2})$ samples, so the extra factor $1/\varepsilon$ on the dimension term is the price of oracle efficiency. For swap multicalibration, which allows a different function from the class for each predicted value, $\tildeΘ(d/\varepsilon^{3})$ samples are necessary and sufficient. For estimating the error of a given predictor, the ordinary notion is the harder one: the swap error can be estimated with an ERM oracle, whereas the ordinary error is NP-hard to approximate even for classes with linear-time ERM. Consequently, there are classes for which it is computationally easier to learn a multicalibrated predictor than to compute the multicalibration error of a given predictor.
Primary: Harvard University
All Institutions: Harvard University
The paper resolves key open questions regarding the sample complexity of multicalibration under oracle efficiency constraints, providing tight upper and lower bounds that define the theoretical limits of fair learning algorithms. It introduces novel techniques combining Hadamard tests with hybrid online learning to achieve optimal rates, and demonstrates the computational hardness of auditing multicalibration error, offering profound insights into the trade-offs between statistical efficiency and computational feasibility in fairness-aware machine learning.
The paper presents a rigorous theoretical framework for understanding the sample complexity of multicalibration when the learner is restricted to using an Empirical Risk Minimization (ERM) oracle. The core methodological contribution is the development of an algorithm that achieves the optimal $\tilde{O}(d/\varepsilon^3)$ sample complexity for ordinary multicalibration, where $d$ is the VC dimension. This is achieved by replacing the exponential number of sign patterns required in previous enumeration-based approaches with a polynomial number of "signed tests" derived from Hadamard matrices. The authors cleverly combine this with a high-probability regret bound for Gaussian Follow-the-Perturbed-Leader (FTPL) in a hybrid online learning setting (random inputs, adversarial feedback). The paper also provides matching lower bounds, proving that this sample complexity is optimal for any learner making polynomially many ERM calls, thereby establishing a clear separation between oracle-efficient learning and enumeration-based learning. The extension to swap multicalibration and the analysis of error estimation (showing NP-hardness for ordinary error estimation) are also technically sound and well-supported.
As a pure theory paper, there are no empirical experiments. The "evaluation" consists of rigorous mathematical proofs of upper and lower bounds on sample complexity. The results are definitive within the theoretical model defined (VC dimension, ERM oracle access, distribution-free setting).
The paper is fully reproducible in the sense that the algorithms are described in sufficient detail for implementation, and the theoretical guarantees are proven. The algorithms rely on standard oracles (ERM) which are well-defined. The deterministic predictor construction requires a computable total order on the input space, which is a standard assumption in such contexts.
The results are theoretical and do not provide empirical validation on real-world datasets. The lower bounds rely on the assumption that the learner accesses the class only through polynomially many ERM calls; if a learner had unlimited computational power to enumerate the class (if finite), better rates are possible, but this is often infeasible for large classes. The NP-hardness result for error estimation applies to the worst-case scenario for specific classes, which may not reflect typical practical instances.
This paper significantly advances the understanding of the computational and statistical limits of fair machine learning. By establishing the optimal sample complexity for oracle-efficient multicalibration, it provides a benchmark for future algorithmic developments. The separation between learning and error estimation has important implications for the auditability of ML models, suggesting that verifying a model's fairness may be computationally harder than training a fair model. This insight is crucial for regulatory frameworks and practical deployment of fair AI systems. The paper resolves key open questions regarding the sample complexity of multicalibration under oracle efficiency constraints, providing tight upper and lower bounds that define the theoretical limits of fair learning algorithms. It introduces novel techniques combining Hadamard tests with hybrid online learning to achieve optimal rates, and demonstrates the computational hardness of auditing multicalibration error, offering profound insights into the trade-offs between statistical efficiency and computational feasibility in fairness-aware machine learning.
Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.
Primary: University of Cambridge
All Institutions: University of Cambridge
The paper proposes Explanatory World Models (EWM) to improve credit assignment in RL by using LLM-generated causal explanations to redistribute delayed rewards. This approach significantly outperforms standard RL baselines in long-horizon tasks and industrial recommendation settings, demonstrating the potential of integrating causal reasoning with reinforcement learning to overcome sample inefficiency.
The paper introduces Explanatory World Models (EWM), a framework that leverages Large Language Models (LLMs) to generate causal explanations for delayed rewards in Reinforcement Learning (RL). The core innovation is the use of these natural language explanations to identify non-temporal dependencies between actions and outcomes, thereby improving credit assignment. The method employs a "conservative hierarchical operator" to redistribute reward based on the LLM's identified causal links while preserving the undiscounted episode return. The theoretical grounding draws from Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations, providing a robust conceptual foundation for what constitutes a "useful" explanation in this context. The integration of LLMs as a credit assignment mechanism rather than just a policy or value estimator is a significant methodological shift.
The evaluation is comprehensive, spanning controlled long-horizon benchmarks (Crafter, WordCraft), a public sequential recommendation dataset (KuaiRand), and an industry-scale generative recommender. The results show substantial outperformance of PPO and LaRe, particularly in scenarios with high reward delay, which is the primary pain point the paper addresses. The inclusion of an industry-scale offline evaluation adds significant practical relevance, demonstrating that the method scales beyond toy problems. The gains in Recall and behavioral proxies on the industrial dataset are strong indicators of real-world utility.
The paper provides detailed descriptions of the LLM variants and the reward redistribution operator. However, the specific prompts used to elicit explanations from the "frontier language model" and the exact hyperparameters for the conservative operator are critical for reproducibility. While the appendix is referenced, the main text suggests a level of detail that should allow for replication if the code is released (which is not provided in the text snippet). The use of a "frontier" model implies potential dependency on proprietary APIs, which could hinder open reproduction.
The reliance on LLMs introduces inference costs and latency, which may be prohibitive for real-time online RL in some settings. The method's effectiveness depends heavily on the quality of the LLM's causal reasoning, which can be inconsistent. Furthermore, the "conservative" nature of the reward redistribution might limit the magnitude of improvement compared to more aggressive credit assignment strategies. The paper does not extensively discuss the failure modes of the LLM explanations or how the system handles incorrect causal attributions.
This work has significant implications for the field of RL, particularly for applications involving sparse or delayed rewards such as robotics, long-horizon planning, and recommendation systems. By bridging the gap between symbolic causal reasoning (via LLMs) and statistical learning (RL), it offers a pathway to more sample-efficient agents. The findings could inspire further research into hybrid neuro-symbolic approaches for credit assignment and the use of generative models as world models for causal inference. The paper proposes Explanatory World Models (EWM) to improve credit assignment in RL by using LLM-generated causal explanations to redistribute delayed rewards. This approach significantly outperforms standard RL baselines in long-horizon tasks and industrial recommendation settings, demonstrating the potential of integrating causal reasoning with reinforcement learning to overcome sample inefficiency.
Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model's current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at https://github.com/Alibaba-NLP/qqr.
Primary: Alibaba Group
All Institutions: Alibaba Group, Alibaba Group Amap, Alibaba Group HKUST (Guangzhou)
The paper introduces Rollout-Time Intervention (RTI), a technique that leverages reward-side evidence to correct failed agent trajectories during rollout, enabling LLMs to learn beyond their intrinsic exploration limits. By combining a multi-agent correction pipeline with distribution gating and token reweighting, RTI effectively mitigates the on-policy bottleneck in reinforcement learning, demonstrating substantial improvements in complex agentic tasks and providing a robust framework for training agents on tasks where successful trajectories are initially sparse.
The paper proposes Rollout-Time Intervention (RTI), a method to address the "on-policy bottleneck" in RL training for LLM agents. The core idea is to use a multi-agent system (Validator-Corrector-Verifier) to repair failed trajectories during the rollout phase, effectively creating successful off-policy examples. The methodology is technically sound, addressing key challenges of off-policy learning: (1) preventing reward hacking/leakage via the Verifier, (2) managing distribution shift via perplexity-based gating, and (3) improving credit assignment via reasoning-shift-aware token reweighting. The use of self-intervention (using the policy model itself for correction) is a clever and resource-efficient design choice, though it relies on the model having sufficient latent capability to correct its own errors when guided by rubrics.
The evaluation is extensive, covering five benchmarks including a novel Navigation benchmark and the challenging Claw-Eval-Hard20. The results show consistent improvements over strong baselines like GRPO and teacher-guided methods (LUFFY, OPD). The ablation studies are rigorous, clearly isolating the contribution of each component (Post-Check, PPL Gating, Reweighting). The analysis of training dynamics (showing that the policy eventually succeeds without intervention) is particularly convincing evidence that the method teaches the model rather than just hacking the reward.
The authors provide a GitHub link and state that comprehensive details, including prompts and hyperparameters, are in the appendix. The use of standard benchmarks (tau2-bench) and open-source backbones (Qwen3) enhances reproducibility. However, the specific implementation of the MAS prompts and the "Claw-Eval" benchmark details might require careful reading of the appendix to fully replicate.
The method relies heavily on the availability of "reward-side evidence" or evaluation rubrics, which may not be available for all agentic tasks. The perplexity gating threshold requires tuning. Additionally, the method is primarily tested on reasoning and tool-use tasks; its applicability to other types of agent interactions (e.g., physical robotics) is not explored. The reliance on self-intervention means that if the model lacks the fundamental knowledge to correct the error, the intervention may fail, though the paper argues this is less of an issue than with external teachers.
This work has significant implications for the training of LLM agents, offering a practical way to overcome the sparsity of rewards in complex environments. By enabling models to learn from corrected failures, it could accelerate the development of more capable autonomous agents. The technique is modular and could potentially be integrated into other RL frameworks beyond GRPO. The paper introduces Rollout-Time Intervention (RTI), a technique that leverages reward-side evidence to correct failed agent trajectories during rollout, enabling LLMs to learn beyond their intrinsic exploration limits. By combining a multi-agent correction pipeline with distribution gating and token reweighting, RTI effectively mitigates the on-policy bottleneck in reinforcement learning, demonstrating substantial improvements in complex agentic tasks and providing a robust framework for training agents on tasks where successful trajectories are initially sparse.
In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training. However, their architecture consists of multiple transformer blocks, resulting in high computational and memory footprints. Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures. This contrast raises a fundamental question: can TFMs be simplified by removing architectural redundancies without sacrificing performance? To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged. Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most $0.002$ median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs. Furthermore, different model families are overparameterized in different ways: TabPFN-3 relies on its encoders, maintaining performance regardless of which single block is retained. Other single-axis models spread computation thinly, needing only their first block, while dual-axis models concentrate it in one critical block. Ultimately, removing depth saves up to 1.5B parameters, speeds up GPU inference by up to $6\times$ at a median cost of $0.03$ AUROC, while a single-block TabPFN-3 serves cached queries $2.4\times$ faster using $6\times$ less memory at negligible cost.
Primary: University of Oxford
All Institutions: University of Oxford
The paper demonstrates that Tabular Foundation Models are effectively shallow by showing that up to 96% of their transformer blocks can be replaced with a simple, label-free affine map with negligible performance loss, revealing significant architectural redundancy and enabling substantial efficiency gains.
The paper introduces a "closed-form, label-free affine translator" to replace contiguous windows of transformer blocks in Tabular Foundation Models (TFMs). The method fits a linear map between the hidden states of the start and end of the bypassed window using ridge regression on an unlabeled calibration set. The window selection is guided by minimizing the forward KL divergence between the full model's output distribution and the compressed model's output distribution. The methodology is clever in that it requires no gradient updates, no labeled data, and preserves the original classification head, allowing for a direct comparison of output probabilities. The per-feature parameterization for dual-axis models is a necessary and well-motivated extension.
The evaluation is extensive, covering 8 frontier TFMs from 6 families across 171 classification datasets (including clinical and standard benchmarks). The results show that replacing 50% of the blocks costs negligible performance (median AUROC drop <= 0.002), and even reducing to a single block retains competitive performance for many models. The ablation study comparing the affine translator to naive layer dropping (identity map) is crucial, showing that the blocks are not inactive but their computation is linearly absorbable. The comparison with GBDTs and the efficiency analysis (speedup/memory reduction) further solidify the practical relevance.
The paper provides a GitHub link to the code. The experimental setup is detailed, including dataset splits, hyperparameter tuning for GBDTs, and specific model configurations. The use of standard benchmarks and clear metrics (AUROC, R^2, latency, memory) ensures reproducibility.
The study is limited to classification and regression tasks; generative tasks are not covered. The pre-training study is limited to a small "nanoTabPFN" model, so it is unclear if the findings scale to much larger architectures. The per-feature translator for dual-axis models can have a high parameter count on wide datasets, potentially negating some efficiency gains.
This paper has significant implications for the deployment of foundation models in resource-constrained environments (e.g., edge devices, clinical servers). It challenges the assumption that deep architectures are necessary for tabular data, suggesting that current TFMs are overparameterized. This could lead to more efficient model design and reduced computational costs for inference. The paper demonstrates that Tabular Foundation Models are effectively shallow by showing that up to 96% of their transformer blocks can be replaced with a simple, label-free affine map with negligible performance loss, revealing significant architectural redundancy and enabling substantial efficiency gains.
Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up. To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as introspective deception, the ground truth can only be determined through careful elicitation or thorough knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. Our probes also readily detect deception on prominent open-weight models which lie about politically sensitive topics, and about their beliefs when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.
Primary: FAR.AI
All Institutions: FAR.AI, Coefficient Giving
The paper presents a highly effective white-box probe architecture and dataset for detecting LLM deception, outperforming black-box monitors in agentic and introspective settings. It provides a rigorous, scalable solution to the critical problem of monitoring frontier AI agents for misaligned behavior, with significant implications for deployment safety.
The paper introduces a rigorous framework for white-box deception detection, distinguishing between "contextual falsehoods" (detectable from context) and "introspective falsehoods" (requiring knowledge of model beliefs). The core methodological contribution is the "FIBS" dataset (163k transcripts) and a novel probe architecture that applies factored causal attention over the layer-token grid, allowing for aggregation of information across many layers and tokens. This is a significant step up from standard linear probes or single-layer MLPs. The use of "contrast scores" (prefilling "Yes" vs "No" and measuring the difference) is a clever practical technique to mitigate context-specific biases in probe outputs.
The experimental setup is robust and comprehensive. The authors evaluate on SHADE-Arena, a standard benchmark for agentic sabotage, showing that their probes outperform a strong black-box baseline (Opus 5.5) with much lower computational cost. Crucially, they test generalization to "introspective" settings, including model organisms with hidden goals and politically sensitive topics where black-box monitors are structurally blind. The results showing high AUC (up to 99.7%) in distinguishing true hidden goals from false denials are particularly strong, addressing a key gap in prior work (e.g., Cooney et al. 2026).
The paper commits to releasing the FIBS dataset and trained probes. Detailed hyperparameters and implementation details are provided in the appendices. The use of standard open-weight models (Qwen, Nemotron) and established benchmarks (SHADE-Arena, MMLU) enhances reproducibility. The label noise estimation (1.4%) via LLM-as-a-judge is a transparent approach to data quality.
The primary limitation is the reliance on the model's internal representation of "belief." If a model's persona shifts such that it genuinely "believes" its deceptive output (due to training dynamics), the probe may fail. The paper acknowledges this with the "math-sandbagger" organism case. Additionally, the dataset is heavily skewed towards off-policy data, and while they test on-policy generalization, the gap between synthetic training data and real-world deployment nuances remains a concern.
This work has high impact on AI safety monitoring. By demonstrating that cheap, white-box probes can match or exceed expensive black-box monitors, it lowers the barrier to entry for effective safety monitoring in frontier deployments. The release of the FIBS dataset will likely become a standard resource for training deception detectors. The findings support the viability of interpretability-based safety measures over purely behavioral monitoring. The paper presents a highly effective white-box probe architecture and dataset for detecting LLM deception, outperforming black-box monitors in agentic and introspective settings. It provides a rigorous, scalable solution to the critical problem of monitoring frontier AI agents for misaligned behavior, with significant implications for deployment safety.
Computer-use agents (CUAs) are rapidly improving, yet synthesizing diverse and reliable software environments at scale remains challenging. Native application environments require substantial construction and integration effort, which limits application coverage and increases the risk of application-specific overfitting. Browser-hosted environments provide limited interaction depth and workflow complexity. These limitations motivate SoftwareGym, an automated pipeline for synthesizing GUI software environments with diverse interfaces, executable functionality, and runtime-verified behavior. It generates structurally diverse designs, realizes them as runnable applications, and uses GUI evaluation with iterative repair to ensure functional reliability. Using SoftwareGym, we synthesize 182 applications across twelve categories and achieve a mean GUI task pass rate of 91.5%. We construct SoftwareGym-Bench, a 120-task evaluation suite spanning the same twelve categories, and use trajectories collected from these environments for CUA training. The resulting model improves Pass@3 from 41.39% to 56.97% on the strictly out-of-domain OSWorld benchmark and achieves 40.0% on SoftwareGym-Bench. These results demonstrate the potential of synthesized software environments as scalable and transferable supervision for CUA training.
Primary: Shanghai Jiaotong University
All Institutions: Shanghai Jiaotong University, Zhejiang University
The paper introduces a scalable, automated pipeline for synthesizing diverse and functionally reliable GUI software environments, demonstrating that training on these synthesized trajectories significantly improves the out-of-domain generalization of compact computer-use agents.
The paper proposes SoftwareGym, an automated multi-agent pipeline for synthesizing GUI software environments. The methodology is structured into three stages: Design (generating diverse functional specs and UI blueprints using layout-archetype distance and tree-edit distance to ensure structural diversity), Implementation (coding Avalonia applications with source-level verification), and Evaluation (GUI-based testing with iterative repair). The use of a "runtime-verified" approach where agents interact with the generated software to verify functionality before it enters the training pool is a strong technical contribution. The diversity metrics (Hamming distance for layout, Tree-Edit Distance for structure) are well-defined and effectively address the common issue of LLM-generated UIs converging to similar patterns.
The experiments are extensive. The authors synthesize 182 applications across 12 categories. They demonstrate that training a 9B model (Qwen3.5-9B) solely on these synthesized trajectories improves Pass@3 on the out-of-domain OSWorld benchmark from 41.39% to 56.97%. This is a significant improvement for a compact model, outperforming larger specialized agents like EvoCUA-32B and OpenCUA-32B. The ablation studies convincingly show the contribution of structured style design and GUI-guided repair. The creation of SoftwareGym-Bench (120 tasks) provides a dedicated evaluation suite, though its size is relatively small compared to OSWorld.
The paper provides code and model links. The pipeline relies on specific LLMs (GPT-5.5, Seed 2.1 Pro) for synthesis and evaluation, which may limit reproducibility for groups without access to these specific frontier models or APIs. However, the detailed description of the verification metrics and the release of the 182 applications significantly enhance reproducibility of the dataset and benchmark.
The synthesized applications are based on Avalonia (a cross-platform .NET UI framework), which may not fully represent the diversity of native desktop environments (e.g., Windows WPF, macOS SwiftUI, Linux GTK). The 40.0% score on SoftwareGym-Bench, while competitive, is lower than the top API-based agents (GPT-5.5 at 73.3%), suggesting that while the training data is effective for transfer, the 9B model still lags behind frontier API models on complex tasks. The cost of synthesis ($70.76 per app) is non-trivial, though lower than manual development.
This work addresses a critical bottleneck in Computer-Use Agent (CUA) research: the scarcity of diverse, executable software environments for training. By demonstrating that synthesized, verified environments can provide transferable supervision, this paper opens a path toward scaling CUA training without relying solely on manual data collection or limited browser-based environments. It is likely to be widely adopted as a data source or a methodological template for environment synthesis in the agent community. The paper introduces a scalable, automated pipeline for synthesizing diverse and functionally reliable GUI software environments, demonstrating that training on these synthesized trajectories significantly improves the out-of-domain generalization of compact computer-use agents.
Unified atomistic modeling has the potential to accelerate discovery in chemistry, materials science, and biology by bridging data-rich chemical domains and data-scarce biological contexts. However, existing generative approaches to atomistic modeling remain highly specialized to scientific disciplines (chemistry vs. biology) or do not leverage both high-volume organic (molecule) and inorganic (material) data for general-purpose pretraining. To this end, we introduce Zatom-2, an atomistic generative model pretrained on approximately five million structures from the OMol25 and OMat24 electronic structure datasets. Zatom-2 features a multiscale Transformer architecture coupled with conditional flow matching that supports force conditioning and foundational pretraining tasks such as generation, structure prediction, and prediction of molecular and material energies and forces. Empirically, Zatom-2 achieves better molecular distribution fidelity than Zatom-1 and achieves strong performance on existing molecule and material generation benchmarks. Zatom-2 demonstrates the ability to control sample generation across low- and high-force regimes, and enhances protein generation in a low-data setting through joint generative-predictive pretraining and transfer learning, increasing protein backbone designability in a length extrapolation setting from 67.8% without pretraining to 74.8% after finetuning on 2,000 protein domains.
Primary: University of California, Berkeley
All Institutions: University of California, Berkeley, Lawrence Berkeley National Laboratory, University of Cambridge, University of Oxford, University of Washington, University of Oxford (Inferred from authors: Mahoney, Erichson, Jacobson, Blau, Perez, Anand, Abrudan, Panescu, Li, Knowles, Morehead, Cretu)
Zatom-2 introduces a unified atomistic generative model that leverages multitask pretraining on large-scale molecular and material datasets to enable cross-domain transfer to protein generation. By combining a novel atom1 tokenization scheme with conditional flow matching and physics-grounded auxiliary objectives (energy/force prediction), the model demonstrates superior distribution fidelity and, crucially, the ability to transfer geometric representations from small molecules to larger biomolecules, achieving significant gains in protein designability in low-data and out-of-distribution settings.
The paper introduces Zatom-2, a unified atomistic generative model that bridges the gap between small molecule/material generation and protein structure prediction. The core methodological contribution is the "atom1" tokenization scheme for molecules/materials (analogous to "atom14" for proteins), which allows a single Transformer backbone to handle variable atom counts without separate diffusion tracks for atom identity. The model employs conditional flow matching for coordinate generation and incorporates a novel "force conditioning" mechanism that uses the mean force norm as a global descriptor to control the relaxation state of generated structures. A key innovation is the multitask pretraining curriculum that jointly optimizes for generation, structure prediction, and energy/force prediction (MLIP objectives). The authors hypothesize and demonstrate that the physics-grounded supervision from energy/force prediction shapes the latent representations in a way that improves generative fidelity and transferability, particularly in low-data regimes.
The experimental evaluation is extensive, covering pretraining on ~5 million structures from OMol25 and OMat24. The paper demonstrates that Zatom-2 outperforms its predecessor (Zatom-1) in distribution fidelity and achieves competitive results on standard benchmarks like QM9, MP20, and GEOM-Drugs. A significant portion of the evaluation focuses on the "force consistency" metric, showing the model can reliably distinguish between high-force (rattled) and low-force (relaxed) regimes. The most compelling results are in the transfer learning section, where Zatom-2 pretrained on small molecules/materials is fine-tuned on a small protein dataset (SCOPe-2k). The model shows significant improvements in designability and novelty, especially in out-of-distribution (longer sequence) settings, compared to training from scratch or using protein-specific models like RFdiffusion3. The scaling study confirms that both data coverage and model capacity yield complementary gains.
The paper provides a strong reproducibility statement, including a link to the open-source code repository (https://github.com/Zatom-AI/nucleus) which contains documentation, data loading, and training/inference materials. The appendix details the model architecture, hyperparameters, and compute resources used (NVIDIA A100 GPUs). The authors also provide an AI Use Statement, disclosing the use of LLMs for code and writing assistance, which adds to the transparency of the research process.
The model does not enforce exact rotational equivariance, relying instead on data augmentation and pair-biased attention, which may limit its performance in tasks requiring strict symmetry preservation compared to equivariant diffusion models. The MLIP (energy/force) prediction accuracy is lower than specialized potentials, though it serves as an auxiliary task. The protein generation experiments are limited to a small dataset (2k domains) and do not yet match the performance of state-of-the-art protein-specific models in in-distribution settings, though it shows promise in extrapolation. The paper acknowledges that precise force matching within regimes remains challenging.
This work has significant potential to accelerate discovery in chemistry, materials science, and biology by providing a unified framework for atomistic modeling. The ability to transfer knowledge from data-rich chemical domains to data-scarce biological contexts could reduce the computational cost of training specialized models. The force conditioning mechanism offers a new tool for controlling the physical state of generated structures, which is valuable for simulating non-equilibrium processes. The open-sourcing of the code and the use of large public datasets (OMol25, OMat24) will facilitate further research in this area. Zatom-2 introduces a unified atomistic generative model that leverages multitask pretraining on large-scale molecular and material datasets to enable cross-domain transfer to protein generation. By combining a novel atom1 tokenization scheme with conditional flow matching and physics-grounded auxiliary objectives (energy/force prediction), the model demonstrates superior distribution fidelity and, crucially, the ability to transfer geometric representations from small molecules to larger biomolecules, achieving significant gains in protein designability in low-data and out-of-distribution settings.
Autoencoders are the core component of latent video generative models, compressing high-dimensional videos into compact latent spaces for downstream generation. However, existing video autoencoders are primarily optimized for pixel-level reconstruction, providing limited supervision for learning structured latent representations. We argue that videos should not be treated merely as signals to be reconstructed, but rather as 2D projections of dynamic 3D worlds that exhibit explicit and well-structured geometric properties. Building on this insight, we propose \textbf{V}ideo \textbf{G}eometry \textbf{G}rounded \textbf{A}utoencoder (\textbf{VGGA}), a geometry-aware video autoencoder that explicitly incorporates geometric constraints into the reconstruction process. Specifically, VGGA grounds the learned representations along three complementary geometric dimensions: \emph{Structural Depth}, \emph{Motion Correspondence}, and \emph{Projective Warping}. These objectives are unified within a Video Geometry Decoder, which employs a shared spatio-temporal upsampling trunk to jointly optimize geometric learning and RGB reconstruction. Through these geometry-grounded objectives, VGGA encourages the latent representations to capture the underlying 3D structure and motion of dynamic scenes. VGGA consistently improves video generation across both geometry-centric and human-motion domains, reducing gFVD by 28.1\% on RealEstate10K and 37.7\% on Taichi-HD. Notably, these substantial generation improvements are achieved while maintaining comparable reconstruction quality, demonstrating that geometry-grounded representation learning yields a more effective latent space for video generation.
Primary: Unknown
All Institutions: Unknown
VGGA introduces a geometry-grounded autoencoder that significantly improves video generation quality by explicitly incorporating depth, motion, and projective constraints into the latent learning process. The paper presents a well-motivated, technically sound, and empirically validated approach that addresses a key limitation of current video VAEs, offering a robust framework for learning generation-friendly video latents.
The paper proposes VGGA, a video autoencoder that integrates geometric constraints (depth, optical flow, and projective warping) into the latent learning process. The core architectural contribution is the "Video Geometry Decoder," which uses a shared spatio-temporal upsampling trunk to align compressed latents with frame-level geometric signals. This design effectively addresses the resolution mismatch between compressed video latents and high-resolution geometric supervision. The method employs a hybrid supervision strategy, using off-the-shelf models (VGGT, RAFT) for pseudo-labels on general videos and explicit camera poses for static scenes (RealEstate10K). The integration of projective warping as a grounding objective is a strong technical choice, as it enforces camera-aware equivariance in the latent space, which is crucial for video generation tasks involving viewpoint changes.
The experimental evaluation is rigorous and comprehensive. The authors train the VAE from scratch to isolate the effect of geometry grounding, comparing against a reconstruction-only baseline and SSVAE. They evaluate on three distinct domains: geometry-centric (RealEstate10K), action-conditioned (UCF101), and human-motion (Taichi-HD). The results show significant improvements in FVD (e.g., 28.1% reduction on RealEstate10K) while maintaining reconstruction quality (PSNR/LPIPS). The ablation studies are particularly strong, demonstrating the complementary nature of the three geometric cues and the importance of the shared decoder structure. The analysis of latent diffusability (FVD convergence during generator training) and linear probing for geometric information provides deep insight into *why* the method works, confirming that the latent space is more structured and diffusion-friendly.
The paper provides high reproducibility. It details the training data mixture (10M curated + RealEstate10K), the specific pseudo-label generators (VGGT, RAFT), and the loss function components with their weights. The use of standard architectures (Wan2.2 VAE backbone, Latte-XL/1 generator) and publicly available datasets facilitates replication. The sensitivity analysis to different depth/flow teachers further supports the robustness of the method.
The primary limitation is the reliance on pseudo-labels from off-the-shelf models, which may introduce noise or bias, although the sensitivity analysis suggests robustness. The projective warping objective is only applicable to static scenes with known poses, limiting its direct application to highly dynamic scenes without camera tracking. The paper acknowledges that it does not model long-horizon dynamics or full 3D scene structure, focusing instead on frame-level and short-range consistency.
This work has significant implications for the field of video generation. By demonstrating that geometric priors can be effectively integrated into autoencoders without sacrificing reconstruction fidelity, it offers a new pathway for improving the structural coherence of generated videos. This is particularly relevant for applications in robotics, AR/VR, and film production where camera consistency and physical plausibility are critical. The approach could be extended to other modalities or integrated with 3D scene representations for more advanced generative tasks. VGGA introduces a geometry-grounded autoencoder that significantly improves video generation quality by explicitly incorporating depth, motion, and projective constraints into the latent learning process. The paper presents a well-motivated, technically sound, and empirically validated approach that addresses a key limitation of current video VAEs, offering a robust framework for learning generation-friendly video latents.
Domain generalised semantic segmentation (DGSS) has recently benefited from combining multiple foundation models, such as vision-language and vision foundation models, to leverage their diverse representations for generalisation. Existing multi-model approaches share a common design choice: cross-model fusion is restricted to depth-aligned layers, implicitly assuming that mutually beneficial information resides at the same depth across models. However, foundation models pretrained under different objectives may develop distinct representation hierarchies, making the optimality of depth-aligned fusion questionable. Moreover, we observe that the foundation models exhibit different depth-wise domain sensitivity. Motivated by these observations, we propose a Sparse All-Layer Connector (SALC) that enables each layer to access information from preceding layers of another model. As the number of accessible layers grows with depth, dense aggregation may hurt generalisation. SALC therefore learns to adaptively select a sparse subset of informative layers. We further introduce a candidate dropout regularisation that strengthens sparsity and encourages SALC to explore diverse layers, leading to more robust selection. Across four foundation model combinations and three DGSS evaluation settings, the proposed design demonstrates stronger generalisation than conventional depth-aligned fusion. The code is available at https://github.com/RuoyuGuo/SALC.
Primary: unknown
All Institutions: unknown
The paper proposes a Sparse All-Layer Connector (SALC) that enables adaptive, sparse cross-depth feature fusion between multiple foundation models for domain generalised semantic segmentation. By leveraging Sparsegen for token-wise sparsity control and introducing frequency-based candidate dropout, the method effectively selects informative layers from diverse pretrained models, outperforming conventional depth-aligned fusion baselines across multiple backbone combinations and evaluation settings.
The paper proposes the Sparse All-Layer Connector (SALC), a method for fusing features from multiple frozen foundation models (e.g., CLIP, DINOv2) in domain generalised semantic segmentation. The core innovation is moving away from depth-aligned fusion (layer $l$ to layer $l$) to a flexible cross-depth fusion where each query layer can attend to a sparse subset of preceding layers in the other model. The method utilizes Sparsegen to project importance scores into a sparse weight distribution, controlled by a token-wise sparsity factor. Additionally, a frequency-based candidate dropout regularisation is introduced to prevent over-reliance on specific layers and encourage exploration of diverse informative features. The methodology is sound, addressing a specific limitation of prior multi-model fusion approaches with a well-motivated mechanism.
The experiments are comprehensive, evaluating four foundation model combinations (DINOv2-CLIP, DINOv2-SigLIP, SAM-CLIP, DINOv2-DepthAnythingV2) across three standard DGSS settings (G->C,B,M; S->C,B,M; C->B,M). The results show consistent improvements over depth-aligned baselines (MFuser, DepthForge) and single-model SOTA methods. Ablation studies effectively validate the importance of adaptive sparsity and frequency-based dropout. The analysis of layer selection patterns provides valuable insights into how the model leverages complementary features from different backbones.
The paper provides a GitHub link for the code. Implementation details are included in the appendix, specifying hyperparameters, hardware (RTX 6000 Ada), and training protocols. The use of standard datasets and established baselines enhances reproducibility.
The improvements on the SAM-CLIP combination are marginal or slightly negative, which the authors attribute to the weak layer-wise structure of SAM features relative to CLIP. The gains on real-to-real benchmarks (C->B,M) are smaller compared to synthetic-to-real settings. The method relies on the availability of multiple large foundation models, which may increase computational overhead during inference compared to single-model approaches, though the encoders are frozen.
The work contributes to the broader field of domain generalization by demonstrating that flexible, sparse cross-model fusion can outperform rigid depth-aligned strategies. It offers insights into the complementary nature of different foundation model representations, which could inform future multi-modal or multi-encoder architectures. The paper proposes a Sparse All-Layer Connector (SALC) that enables adaptive, sparse cross-depth feature fusion between multiple foundation models for domain generalised semantic segmentation. By leveraging Sparsegen for token-wise sparsity control and introducing frequency-based candidate dropout, the method effectively selects informative layers from diverse pretrained models, outperforming conventional depth-aligned fusion baselines across multiple backbone combinations and evaluation settings.
Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs. Training a model for each modality pair scales poorly with the number of sensors and forgoes potential positive transfer across modality pairs. We introduce TartanMatch, a unified model for dense correspondence across all 25 ordered pairs among RGB, event, thermal, depth, and projected LiDAR. TartanMatch adapts an RGB-pretrained correspondence model to all modality pairs within one shared network using lightweight modality adaptation and a mixture of real and synthetic data. On zero-shot real-world benchmarks, TartanMatch reduces Endpoint Error (EPE) over the strongest baselines by 61.9% for cross-modal matching and 49.6% for same-modal matching. We further show that joint training across all modality pairs improves average matching accuracy over pair-specific training, especially on challenging cross-modal pairs. These results demonstrate that a single model achieves broad multi-modal coverage and state-of-the-art matching accuracy.
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, Fisk University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. TartanMatch introduces a unified dense correspondence model that significantly outperforms pair-specific baselines across 25 modality pairs by leveraging RGB pre-training and joint multi-modal training, demonstrating that a single network can effectively handle diverse sensor combinations with superior accuracy and scalability.
The paper proposes TartanMatch, a unified framework for dense correspondence across five distinct sensing modalities: RGB, event, thermal, depth, and projected LiDAR. The core methodological contribution is the adaptation of an RGB-pretrained correspondence model (specifically UFM) to handle all 25 ordered modality pairs within a single shared network. This is achieved through lightweight modality adaptation modules and a training strategy that leverages a mixture of real and synthetic data. The approach addresses the scalability issue of training separate models for each modality pair, arguing that joint training facilitates positive transfer, particularly for challenging cross-modal pairs. The use of a pre-trained backbone for RGB and adapting it to other modalities is a sound architectural choice that leverages the rich features learned from large-scale RGB datasets.
The experimental evaluation is extensive, covering zero-shot real-world benchmarks. The paper reports significant improvements in Endpoint Error (EPE), claiming a 61.9% reduction for cross-modal matching and 49.6% for same-modal matching compared to the strongest baselines. The radar chart visualization effectively demonstrates the model's superiority across various modality pairs. The claim that joint training improves average matching accuracy over pair-specific training is a key empirical finding that supports the unified model hypothesis. The inclusion of both real and synthetic data in the training mix is well-justified by the results, showing robustness to domain shifts.
The paper provides a project page (https://tartanmatch.github.io/) which likely contains code and additional resources. The use of standard benchmarks and clear metrics (EPE) enhances reproducibility. However, the specific details of the "lightweight modality adaptation" modules and the exact composition of the synthetic data mixture would need to be verified in the supplementary material or code repository for full reproducibility. The reliance on a specific pre-trained model (UFM) is a potential dependency, but UFM is a known baseline.
The primary limitation is the scope of modalities covered (5 modalities, 25 pairs). While this is a significant expansion over previous work, it does not cover all possible robotic sensing modalities (e.g., radar, acoustic). The performance on zero-shot real-world data is strong, but the gap between synthetic/real training data and real-world inference might still exist for unseen environments. The computational cost of training a single model for 25 pairs versus 25 separate models is not explicitly detailed in the abstract, though the "lightweight" adaptation suggests efficiency.
This work has high potential impact in the robotics and autonomous systems fields. A unified correspondence model simplifies the perception pipeline for multi-modal robots, reducing the need for maintaining and updating multiple specialized models. The demonstrated positive transfer across modalities could inspire similar unified approaches in other multi-sensor fusion tasks. The open-source nature of the project (implied by the project page) will likely facilitate adoption by the research community. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. TartanMatch introduces a unified dense correspondence model that significantly outperforms pair-specific baselines across 25 modality pairs by leveraging RGB pre-training and joint multi-modal training, demonstrating that a single network can effectively handle diverse sensor combinations with superior accuracy and scalability.
Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect a shared deficit in visual transition reasoning, and test whether this capability can serve as a shared training primitive, one that different models can learn from different supervision sources and reuse across different tasks, with a systematic training recipe. Existing benchmarks document these deficits separately but do not support controlled comparisons across scenes, actions, and reasoning operations. We therefore introduce WOVEN, a training source and benchmark for visual transition reasoning that organizes transition supervision by scene, action, and reasoning type, using diverse, realistic rollouts from video-pretrained generative models: 36,076 examples across 20 scene types, 5 action types, and 8 reasoning types. We first evaluate 38 frontier MLLMs (e.g., GPT-5.4 and Qwen3-VL-235B-A22B) and find a substantial and systematic deficit: even the strongest models fall far below humans, and the failures recur across model families and persist with scale. We then train MLLMs at multiple scales on WOVEN and find that they learn a shared capability that transfers broadly: training subsets of only about 2,000 items each collectively improve 22 of 26 external benchmarks by up to 27.3 percentage points, and WOVEN data can replace 30-50% of a task's own training data with comparable accuracy. Controlled comparisons further yield a training recipe for visual world modeling, validated prospectively on held-out benchmarks: select supervision by the reasoning operation it teaches rather than by the actions, scenes, or domains it shows, and prefer larger changes to the visual state for robustness. Our work establishes visual transition reasoning as a reusable foundation for systematic visual world-model training in MLLMs.
Primary: Northwestern University
All Institutions: Northwestern University, Carnegie Mellon University, UNC Chapel Hill
The paper establishes visual transition reasoning as a shared, trainable primitive for multimodal LLMs and provides a systematic training recipe validated by a new benchmark, WOVEN. By demonstrating that small, targeted subsets of transition supervision can broadly improve performance across diverse spatial, physical, and temporal benchmarks, the work offers a data-efficient and theoretically grounded approach to enhancing visual world modeling capabilities in current MLLMs.
The paper proposes a novel framework for "visual transition reasoning," formalizing it as inference over $(s, a, s')$ triplets. The core methodological contribution is the WOVEN dataset, which uses video-pretrained generative models (Wan2.2) to create controlled rollouts with known actions and endpoints. This allows for the construction of multiple-choice questions with error-typed distractors, enabling fine-grained analysis of model failures. The training methodology involves supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) on controlled subsets of the data. The systematic ablation study comparing supervision by reasoning operation versus action type is a strong methodological choice that isolates the source of transfer.
The experimental evaluation is extensive and rigorous. The authors evaluate 38 frontier MLLMs to establish a baseline deficit. They then demonstrate that training on small subsets (~2,000 items) of WOVEN significantly improves performance on 22 of 26 external benchmarks. A key strength is the causal intervention experiment, where replacing 30-50% of task-specific training data with WOVEN data yields comparable accuracy, proving the utility of the dataset as a data-efficient substitute. The correlation analysis ($r=0.86$) between WOVEN accuracy gains and external benchmark gains provides strong evidence for the "shared primitive" hypothesis.
The paper provides high reproducibility. Code and datasets are released on GitHub and HuggingFace. The generation process using Wan2.2 is specified, and the training recipes (SFT + GRPO) are detailed. The use of standard benchmarks for evaluation ensures that results can be verified against existing baselines.
The reliance on synthetic video generation (Wan2.2) may introduce domain gaps compared to real-world physics, although the authors argue for realism. The evaluation is primarily on multiple-choice tasks, which may not fully capture open-ended reasoning capabilities. The "frontier" models evaluated include some very recent releases (GPT-5.4, Qwen3-VL), which may limit immediate reproducibility for labs without access to these specific API versions or weights.
This paper has significant potential to influence how multimodal models are trained for embodied AI and physical reasoning. By identifying visual transition reasoning as a reusable primitive, it offers a data-efficient path to improving spatial and temporal understanding. The systematic recipe for selecting supervision (by reasoning operation rather than action) provides actionable guidelines for practitioners building world models. The paper establishes visual transition reasoning as a shared, trainable primitive for multimodal LLMs and provides a systematic training recipe validated by a new benchmark, WOVEN. By demonstrating that small, targeted subsets of transition supervision can broadly improve performance across diverse spatial, physical, and temporal benchmarks, the work offers a data-efficient and theoretically grounded approach to enhancing visual world modeling capabilities in current MLLMs.
Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusions. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local modality mixer couples patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, preserving scene context while distinguishing the target from its surroundings. Text-guided semantic conditioning complements these spatial cues with category names and object captions through stage-specific adapters for structure, geometry, and appearance generation. The generated asset initializes a multimodal agent, providing instance-specific geometry and pose for targeted structural and texture refinement through an observation-guided edit-render-review loop. Across synthetic objects, cluttered tabletops, and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons and scene-level simulation further demonstrate the reconstruction efficiency and simulation readiness. Project page: https://research.nvidia.com/labs/lpr/gator/
Primary: NVIDIA
All Institutions: NVIDIA
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. GATOR introduces a novel generative-agentic framework for 3D object reconstruction that couples a modality mixer for initial asset generation with an iterative agentic refinement loop, achieving strong geometric and appearance fidelity from casual images while recovering scene-relative pose, though the lack of detailed quantitative results in the provided text limits the full assessment of its empirical superiority over existing feed-forward and iterative baselines.
The paper proposes GATOR, a framework combining a "local modality mixer" for feature fusion (RGB, mask, pointmap) with a "multimodal agent" for iterative refinement. The core novelty lies in the agentic loop (edit-render-review) that uses the initial generative output to guide targeted structural and texture corrections. This hybrid approach of one-shot generation followed by agentic refinement is a significant architectural shift from purely feed-forward reconstruction models. The use of stage-specific adapters for text-guided conditioning is a standard but effective technique. The methodology is sound, leveraging recent trends in agentic AI for 3D tasks, though the specific implementation details of the "agent's" decision-making process (e.g., how it selects which parts to refine) are critical to its success and likely complex.
The paper claims strong geometric and appearance fidelity on synthetic objects, cluttered tabletops, and indoor scenes. It includes time-budget comparisons and scene-level simulation readiness tests. However, the provided text is a placeholder with section headers but no actual quantitative results, tables, or figures. Without specific metrics (e.g., Chamfer Distance, FID, PSNR) or comparisons to state-of-the-art baselines (like Zero-1-to-3, Wonder3D, or recent agentic 3D papers), the experimental evaluation cannot be fully verified. The claim of "strong fidelity" is unsubstantiated in the provided text.
The project page is provided, which is a positive sign for reproducibility. However, the lack of detailed hyperparameters, agent prompt engineering details, and training data specifics in the provided text makes independent reproduction difficult. The "agentic" component often relies on proprietary LLMs or specific prompt strategies that may not be fully open-sourced.
The primary limitation is the reliance on an agentic loop, which can be computationally expensive and slow compared to feed-forward methods, despite the "time-budget" claims. The performance on real-world, uncontrolled casual images (as opposed to synthetic or curated datasets) is not clearly demonstrated in the provided text. The "scene-aligned" pose recovery from sparse observations is a difficult problem, and the robustness of this component to severe occlusions or ambiguous viewpoints is a potential weakness.
This work has high potential impact in robotics, AR/VR, and digital twin creation, where accurate 3D asset recovery from casual photos is crucial. The agentic approach could be extended to other 3D tasks like scene editing or animation. It represents a step towards more interactive and intelligent 3D reconstruction pipelines. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. GATOR introduces a novel generative-agentic framework for 3D object reconstruction that couples a modality mixer for initial asset generation with an iterative agentic refinement loop, achieving strong geometric and appearance fidelity from casual images while recovering scene-relative pose, though the lack of detailed quantitative results in the provided text limits the full assessment of its empirical superiority over existing feed-forward and iterative baselines.
Frontier AI systems are increasingly natively multimodal, jointly pretrained on multiple modalities, such as text, vision, and audio. However, the scientific understanding of the multimodal data mixtures used to pretrain these systems --- how much of each modality to mix, and at what potential interference cost to the others --- is largely absent or gatekept. In this work, we derive data mixing scaling laws for three modalities: text, vision, and audio. Multimodal Large Language Models (MLLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, as MoEs scale efficiently and partially limit cross-modal interference. We therefore use MoEs to conduct 268 experiments ranging from 457M to 8.3B parameters, on up to 150 billion tokens. Our work leads to multiple actionable insights into multimodal data mixing for training MLLMs. First, compute optima are modality-specific: audio modality is model-heavy, favoring scaling parameters over tokens, while text and vision require balanced scaling of model size and data. Second, repeating vision or audio data beyond 2x yields negligible benefits. Finally, cross-modal interactions are highly asymmetric: vision strongly interferes with audio, whereas audio's interference with learning text and vision is negligible. Overall, we provide a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.
Primary: MILA - Quebec AI Institute
All Institutions: MILA - Quebec AI Institute, McGill University, University of Washington, EPFL
The paper derives scaling laws for multimodal data mixtures in MoE architectures, revealing modality-specific compute optima and asymmetric cross-modal interference. This comprehensive empirical study provides actionable guidelines for efficient pretraining of frontier multimodal large language models, significantly advancing the understanding of how to balance text, vision, and audio data.
The paper proposes a rigorous empirical framework for deriving scaling laws specifically for multimodal data mixtures (text, vision, audio) within Mixture-of-Experts (MoE) architectures. The methodology is sound, leveraging 268 controlled experiments across a wide parameter range (457M to 8.3B) and token counts (up to 150B). The choice to focus on MoE architectures is timely and methodologically justified, as MoEs are increasingly standard in frontier models and help isolate cross-modal interference effects. The derivation of modality-specific compute optima is a significant methodological advancement over previous unimodal scaling law studies.
The experimental scale is substantial, covering three modalities and a large parameter/token space. The findings are concrete and actionable: identifying audio as "model-heavy" (favoring parameters over tokens) and text/vision as requiring balanced scaling is a valuable empirical insight. The discovery of asymmetric cross-modal interference (vision interfering with audio, but not vice versa) is a surprising and useful result for practitioners. The validation section likely confirms these trends, though the specific metrics for "interference" need to be clearly defined in the full text to ensure robustness.
The paper provides detailed experimental setup information (model sizes, token counts, modality types). However, without access to the code or specific hyperparameter configurations for the MoE routing and data mixing ratios, full reproduction is difficult. The scale of the experiments (150B tokens) makes independent reproduction costly, limiting reproducibility to groups with similar computational resources.
The study is limited to three modalities (text, vision, audio) and MoE architectures. It may not generalize directly to dense transformers or other modalities (e.g., video, 3D). The "interference" metric is a key variable; if not rigorously defined, the conclusions about asymmetry could be sensitive to evaluation choices. The paper does not address the cost of data collection or curation for these mixtures.
This work provides a principled foundation for training next-generation multimodal foundation models. By quantifying the cost-benefit of mixing modalities, it helps labs allocate computational resources more efficiently. The insights on asymmetric interference can guide curriculum learning strategies and data balancing techniques, potentially reducing the compute required to achieve specific performance levels in MLLMs. The paper derives scaling laws for multimodal data mixtures in MoE architectures, revealing modality-specific compute optima and asymmetric cross-modal interference. This comprehensive empirical study provides actionable guidelines for efficient pretraining of frontier multimodal large language models, significantly advancing the understanding of how to balance text, vision, and audio data.
Recent advances in 3D reconstruction enable the extraction of temporally consistent 3D motion from RGB observations; however, current Vision-Language-Action (VLA) models still predominantly rely on 2D image-text priors. Therefore, they do not accurately represent the fundamental dynamics of the 3D world associated with physical actions. This over-reliance on static 2D visual cues limits the capacity of VLA models in achieving robust robot manipulation. To address these limitations, we introduce FlowVLA, a reconstruction-guided, pluggable 3D flow learning framework aimed at empowering VLA models with an inherent understanding of action-relevant dynamics. Instead of requiring explicit 3D sensor inputs or altering the inference-time architecture, our method is designed to utilize temporally aligned 3D flow only during training. This allows the model to learn the dynamic evolution of the physical scene while preserving the computational efficiency of conventional 2D VLA backbones. To bridge scene-level dynamics with robotic manipulation, we propose a modular 3D flow learning framework with dual-level flow supervision. Specifically, point-level flow guides the model to forecast the future displacements of task-critical 3D points, while TCP-level flow aggregates these motions to align with the robot's TCP trajectories. This dual-level flow supervision encourages the model to focus on semantically meaningful dynamic regions, effectively coupling scene-level motion with precise control signals. Extensive evaluations across simulation and real-world experiments demonstrate that FlowVLA consistently improves manipulation performance, zero-shot robustness, and optimizes data efficiency. FlowVLA adds only lightweight auxiliary heads during training and discards them at inference. Therefore, FlowVLA introduces no additional 3D sensor inputs, architectural modifications, or inference-time computation.
Primary: Unknown (Funding suggests Korean institutions, likely KAIST or similar, but affiliations are redacted/missing in text)
All Institutions: Unknown
The paper introduces a training-only 3D flow supervision framework that enhances VLA models' understanding of physical dynamics without incurring inference-time computational costs. By leveraging reconstruction-derived 3D point trajectories to supervise auxiliary heads during training, FlowVLA effectively bridges the gap between 2D semantic priors and 3D action-relevant motion, resulting in improved robustness and data efficiency in both simulation and real-world manipulation tasks.
The paper proposes FlowVLA, a framework that injects 3D motion cues into Vision-Language-Action (VLA) models using a "training-only" supervision strategy. The core methodological contribution is the use of temporally aligned 3D point trajectories derived from a feed-forward reconstruction model (VDPM) to create 3D flow targets. These targets are used to supervise two auxiliary heads: a point-level flow head (predicting future displacements of key points) and a TCP-level flow head (aggregating point flows to match robot end-effector trajectories). The key architectural insight is that these heads and the 3D inputs are discarded at inference time, allowing the model to retain the computational efficiency of a standard 2D VLA while benefiting from 3D dynamic understanding during training. The magnitude-based selection of points to focus on dynamic regions is a sensible heuristic to reduce noise from static backgrounds.
The evaluation is comprehensive, covering simulation (LIBERO, LIBERO-Plus) and real-world experiments (UR5e). The results show consistent improvements in success rates, particularly in long-horizon tasks and zero-shot robustness scenarios (viewpoint/height variations). Ablation studies confirm the necessity of coordinate alignment, reference-frame anchoring, magnitude-based selection, and the dual-level supervision. The data efficiency analysis shows that the method helps with limited data regimes, which is a strong practical advantage. The comparison against 2D optical flow and static 3D point supervision effectively isolates the benefit of dynamic 3D flow.
The paper provides detailed mathematical formulations for the flow construction and loss functions. However, specific implementation details for the VDPM reconstruction model and the exact hyperparameters for the auxiliary heads are likely in the appendix (referenced but not fully visible in the snippet). The reliance on a specific reconstruction model (VDPM) introduces a dependency that may limit immediate reproducibility if that model is not widely available or easy to train. The "pluggable" nature suggests it can be applied to existing VLAs like Seer and 0.5, which aids adoption.
The primary limitation is the dependency on the quality of the offline 3D reconstruction model. If VDPM fails to accurately track points, the supervision signal becomes noisy. The magnitude-based point selection is a heuristic and does not guarantee semantic relevance to the task, as acknowledged by the authors. The method requires an offline preprocessing step to generate 3D trajectories, which adds a pipeline complexity even if inference is fast.
This work addresses a significant gap in VLA models: the lack of explicit 3D dynamic understanding. By showing that 3D dynamics can be injected without changing the inference architecture, it offers a practical path for existing VLA developers to improve robustness. It bridges the gap between 2D semantic priors and 3D physical dynamics, potentially influencing future designs of robot foundation models to incorporate geometric priors more effectively. The paper introduces a training-only 3D flow supervision framework that enhances VLA models' understanding of physical dynamics without incurring inference-time computational costs. By leveraging reconstruction-derived 3D point trajectories to supervise auxiliary heads during training, FlowVLA effectively bridges the gap between 2D semantic priors and 3D action-relevant motion, resulting in improved robustness and data efficiency in both simulation and real-world manipulation tasks.
Generalist visual-navigation policies tell a robot which direction to go but not how far. A separate lifting step is needed at deployment to convert their unitless trajectory into a metric path that a controller can track; existing scalar lifters apply uniform scaling that misses per-waypoint corrections, while in-policy approaches that inject monocular depth into the policy inherit the prediction errors of the depth model near obstacles and at close range. We present PathLift, a per-waypoint depth-conditioned adapter that lifts a pre-trained visual-navigation policy by consuming an explicit, sensor-grade depth source (RGB-D, stereo, or projected LiDAR) and fitting per-waypoint magnitude and lateral residuals to a canonicalized prior. PathLift consistently improves closed-loop navigation across diverse visual-navigation policies in simulation and on a real wheeled robot under both stereo and LiDAR depth. By turning the policy's unitless intent into a depth-feasible metric path in a single feed-forward pass, the adapter lifts real-world success without test-time optimization.
Primary: Seoul National University
All Institutions: Seoul National University
PathLift introduces a depth-conditioned adapter that lifts unitless visual navigation trajectories to metric paths via structured-target regression, significantly improving closed-loop success rates and collision avoidance across diverse policies and real-world deployments. The paper presents a well-motivated and rigorously evaluated solution to a critical deployment bottleneck in generalist navigation, offering a modular, low-latency, and sensor-agnostic approach that is likely to be widely adopted in the field.
The paper proposes PathLift, a lightweight adapter network that converts unitless trajectories from visual navigation policies into metric paths using explicit depth inputs. The core methodological contribution is the "structured-target regression" approach, where the supervision target is not the raw expert path, but the closest candidate from a pre-defined family of intent-preserving, depth-feasible trajectories. This addresses the gap between the policy's geometric intent and physical constraints (collisions, visibility). The architecture uses Plücker ray tokenization for depth, which is a reasonable choice for geometric tasks, and a transformer-based decoder. The method is designed to be prior-agnostic and sensor-agnostic, which is a strong architectural decision.
The evaluation is rigorous and well-structured. It includes closed-loop simulation on HM3D with four different state-of-the-art policies (GNM, ViNT, NoMaD, FlowNav) and real-robot deployment. The comparison against MetricNet and MetricNav is fair, with a specific control (MetricNav*) to isolate the effect of the depth source. The results show consistent improvements in Success Rate (SR) and Success weighted by Path Length (SPL) across all priors. The ablation studies are thorough, validating the necessity of depth, the Plücker tokenization, lateral corrections, and the admissible-family supervision. The real-robot results, while on a small sample size (15 trials per condition), show significant improvements in collision-free distance and success rate.
The paper provides detailed information on the network architecture, loss functions, training schedule, and hyperparameters. The use of standard datasets (HM3D, CODa, OpenLORIS) and open-source policies aids reproducibility. However, the specific implementation of the "admissible family" candidate generation and the exact code for the adapter are not provided in the text, though the description is detailed enough for re-implementation. The lack of a provided code repository URL in the text is a minor negative for immediate reproducibility, but the methodological details are sufficient.
The primary limitation is the dependence on the quality of the base prior; if the prior points towards obstacles, PathLift cannot correct the intent, only the scale and lateral position within the feasible space. The real-robot evaluation is limited to indoor corridors and a small number of trials. The method assumes a single depth frame per inference step, which may not capture dynamic obstacles well. The paper also notes that the depth field of view is matched to the prior's training FOV, which limits awareness of obstacles outside this range.
This work has significant potential for the robotics community, particularly for deploying generalist navigation policies on real hardware. By decoupling the lifting step from the policy, it allows for the reuse of pre-trained policies across different robots and sensors without retraining. This modularity is a key trend in embodied AI. The method's low latency (5ms) makes it suitable for real-time control. It could serve as a standard component in the deployment pipeline for visual navigation systems. PathLift introduces a depth-conditioned adapter that lifts unitless visual navigation trajectories to metric paths via structured-target regression, significantly improving closed-loop success rates and collision avoidance across diverse policies and real-world deployments. The paper presents a well-motivated and rigorously evaluated solution to a critical deployment bottleneck in generalist navigation, offering a modular, low-latency, and sensor-agnostic approach that is likely to be widely adopted in the field.
Existing navigation systems commonly rely on dedicated depth sensors or explicit depth reconstruction, while learning to generate trajectories directly from RGB images remains challenging due to the sim-to-real domain gap. In this work, we propose Stereo-Planner, an end-to-end vision-to-trajectory framework that enables zero-shot sim-to-real planning from stereo RGB observations. The key idea is to formulate trajectory generation as state- and scale-conditioned querying of latent stereo geometry. A Scene Representor incorporates a pretrained vision grounding foundation model to extract geometric cues shared across simulated and real-world environments, supporting generalization across visual domains. A lightweight Trajectory Decoder then translates these representations into trajectories through interleaved attention, separating trajectory-state interaction from scene-feature retrieval and incorporating the stereo baseline as a metric-scale cue. With these designs, Stereo-Planner is trained entirely on simulated data by minimizing differentiable trajectory costs and deployed directly in real-world environments without fine-tuning. Benchmark results demonstrate strong cross-scene generalization, with a 95% success rate in simulation environments at a commanded speed of 8 m/s, and competitive performance against depth-based planners using only stereo RGB observations for scene perception. Furthermore, deployment on a compact quadrotor equipped with an NVIDIA Orin NX enables autonomous flight at speeds of up to 7 m/s, with a mean onboard vision-to-action inference time of 24.27 ms. The software, simulation environments, and hardware designs will be open-sourced at https://github.com/Stereo-Planner/Stereo-Planner.
Primary: Nanyang Technological University
All Institutions: Nanyang Technological University, Tianjin University
Stereo-Planner enables zero-shot sim-to-real quadrotor navigation using stereo RGB by leveraging a vision grounding foundation model for geometry-aware scene representation and a scale-conditioned trajectory decoder. The paper presents a significant technical contribution by demonstrating that pretrained visual features can effectively bridge the sim-to-real domain gap for end-to-end motion planning, achieving high-speed autonomous flight without depth sensors or fine-tuning.
The paper proposes Stereo-Planner, an end-to-end framework that maps stereo RGB images directly to quadrotor trajectories. The core architectural contribution is the use of a pretrained vision grounding foundation model (Depth Anything V3 encoder) as a scene representor, which extracts geometry-aware features robust to sim-to-real appearance gaps. A lightweight trajectory decoder uses interleaved attention to query these features, conditioned on the MAV state and, crucially, the stereo baseline to resolve scale ambiguity. The training strategy employs differentiable trajectory costs (safety, smoothness, goal-reaching) and a Virtual Baseline Agmentation (VBA) technique to teach the network to interpret metric scale from varying stereo baselines without re-rendering environments. This approach effectively decouples geometric perception from appearance, allowing zero-shot transfer from simulation to reality.
The evaluation is rigorous, comparing against strong baselines like YOPO V2 and Ego-Planner V2 in both simulation and real-world deployment. The paper demonstrates a 95% success rate in unseen simulated environments (forest, industrial) and successful real-world flight at 7 m/s. Ablation studies clearly isolate the contributions of the foundation model initialization, the interleaved attention mechanism, and the VBA strategy. The comparison with depth-based methods shows that stereo RGB can achieve competitive performance without the hardware cost and latency of depth sensors.
The authors commit to open-sourcing the software, simulation environments, and hardware designs. The paper provides detailed hyperparameters, loss function definitions, and architectural specifics. The use of standard Unity simulation and ROS interfaces enhances reproducibility. The specific version of the foundation model (DA3-Small) and its integration details are clearly described.
The method relies on the availability of a suitable vision grounding foundation model (DA3), which may not be universally available or optimal for all tasks. The real-world evaluation, while impressive, is limited to specific environments (forest, industrial) and may not generalize to all cluttered scenarios (e.g., dynamic obstacles, extreme weather). The computational cost, while reduced, still requires an NVIDIA Orin NX, which is a significant hardware requirement for "compact" drones.
This work bridges the gap between high-level visual foundation models and low-level robotic control, a critical area for autonomous systems. By demonstrating that RGB-only stereo vision can replace depth sensors for agile navigation, it reduces hardware costs and increases the robustness of aerial robots. The methodology of using foundation model features for sim-to-real transfer in control tasks is likely to inspire similar approaches in other robotic domains. Stereo-Planner enables zero-shot sim-to-real quadrotor navigation using stereo RGB by leveraging a vision grounding foundation model for geometry-aware scene representation and a scale-conditioned trajectory decoder. The paper presents a significant technical contribution by demonstrating that pretrained visual features can effectively bridge the sim-to-real domain gap for end-to-end motion planning, achieving high-speed autonomous flight without depth sensors or fine-tuning.
Quadruped robots deployed for inspection and field missions must cross heterogeneous terrain, from stairs, slopes, and gaps to high platforms, stepping stones, and balance beams, where safe support is sparse or see-through. Existing methods interpose a terrain map or a depth image between sensing and control. Maps are costly to maintain, corrupted by odometry drift and outliers, and misled by see-through gratings, while depth cameras degrade with lighting and surface material. In this paper, we present PointWalker, a mapless framework whose single reinforcement learning policy directly consumes raw point clouds from four light detection and ranging (LiDAR) sensors. Voxelized points are encoded by linear self-attention and fused with proprioception through cross-attention, so terrain geometry reaches the policy without maps or foothold estimation. One proximal policy optimization (PPO) run, with terrain-aware rewards and dual-channel simulation of see-through gratings, yields one policy for all seven terrain classes with a 90.1% mean success rate in simulation. Deployed zero-shot on a Deep Robotics X30 at 50 Hz onboard, it crosses 0.8-m gaps, climbs 0.8-m platforms and 45° steel-grating stairs, carries a 50-kg payload, and completes a 2.5-km mountain route.
Primary: Zhejiang University
All Institutions: Zhejiang University, Yunshenchu Technology Co. Ltd., Zhejiang Key Laboratory of Additive Manufacturing Technology and Equipment, Hangzhou Public Library
PointWalker introduces a mapless quadruped locomotion framework that directly consumes raw LiDAR point clouds via linear self-attention and cross-attention fusion, achieving robust zero-shot traversal of heterogeneous and see-through terrains. The paper demonstrates that removing intermediate terrain representations like elevation maps significantly improves performance on sparse and transparent surfaces, validated through rigorous simulation ablations and extensive real-world hardware testing on the Deep Robotics X30.
The paper proposes PointWalker, a reinforcement learning framework that eliminates intermediate terrain representations (like elevation maps or depth images) by directly processing raw LiDAR point clouds. The core architectural contribution is the use of linear self-attention to encode voxelized points into tokens, followed by cross-attention with proprioceptive history to fuse terrain geometry with robot state. This allows the policy to attend to specific relevant points (e.g., gap edges, grating bars) without quadratic computational cost. The training methodology employs a single PPO run across seven terrain classes using a curriculum, with a dual-channel simulation setup where the physics engine sees solid support while the perception channel sees sparse, see-through structures (like steel gratings). This effectively teaches the robot to distinguish between visual clutter and actual support surfaces.
The experimental evaluation is robust, featuring extensive simulation ablations comparing the proposed method against height-map encoders, depth-image encoders, and blind policies. The results show significant performance gains on sparse and see-through terrains where intermediate representations fail. Hardware validation on the Deep Robotics X30 is strong, demonstrating zero-shot transfer to complex real-world scenarios including 0.8m gaps, 45-degree steel-grating stairs, and a 2.5km mountain route. The inclusion of ablation studies on hardware (e.g., removing dual-channel simulation or foot-edge penalties) provides clear causal evidence for the necessity of specific design choices.
The paper provides detailed hyperparameters, network architectures, and simulation configurations (Isaac Lab). The use of standard RL algorithms (PPO) and well-documented domain randomization strategies enhances reproducibility. However, the specific implementation of the dual-channel simulation and the precise LiDAR noise models may require significant effort to replicate exactly. The code is not explicitly linked in the text provided, though a video is available.
The method relies on four LiDARs, which is a significant hardware cost compared to depth cameras. The perception range is limited to the local vicinity of the robot, meaning it cannot plan long-range paths or anticipate obstacles far ahead. The paper acknowledges that porting to other robot platforms with different sensor layouts is untested. Additionally, the method is reactive to local geometry and does not handle dynamic moving obstacles or support surfaces effectively.
This work advances the state-of-the-art in legged robot perception by demonstrating that raw point clouds can be effectively used for high-frequency control without intermediate maps. It addresses a critical gap in industrial inspection robots that must traverse heterogeneous, see-through terrain. The approach could inspire similar direct-perception methods in other robotic domains, such as manipulation or aerial robotics, where intermediate representations may lose critical geometric details. PointWalker introduces a mapless quadruped locomotion framework that directly consumes raw LiDAR point clouds via linear self-attention and cross-attention fusion, achieving robust zero-shot traversal of heterogeneous and see-through terrains. The paper demonstrates that removing intermediate terrain representations like elevation maps significantly improves performance on sparse and transparent surfaces, validated through rigorous simulation ablations and extensive real-world hardware testing on the Deep Robotics X30.
Camera-based occupancy prediction provides geometric and semantic scene understanding for robot perception and has been adopted in autonomous vehicle and UAV perception systems. Gaussian primitives offer a promising alternative by potentially representing only occupied regions, thereby concentrating computation and representational capacity where needed and reducing wasted computation and memory in empty space. However, existing Gaussian-based methods suffer from poor primitive utilization, as ineffective training objectives leave some Gaussians in empty space, while the limited expressiveness of Gaussian functions restricts their ability to model complex shapes. To address these limitations, we propose a novel 3D representation, Gaussian Neural Field (GNF), which can accurately map voxel positions to semantic occupancies. In GNF, Gaussians and a neural network jointly parameterize the field, where the Gaussians define its spatial support and provide features for neural decoding, while the neural network enables more expressive occupancy modeling than Gaussian functions alone. We further develop the Gaussian Neural Field Transformer (GNFormer), a streamlined and flexible method that predicts GNFs for camera-based occupancy prediction. Across Occ-ScanNet, SSCBench-KITTI-360, and nuScenes-SurroundOcc benchmarks, GNFormer achieves the highest IoU and mIoU and the fastest inference among the compared single-frame methods. In the experiments, GNFormer can represent indoor and outdoor scenes with only 256 and 1024 Gaussians, respectively, substantially fewer primitives than prior Gaussian-based methods. The code will be released.
Primary: Southern University of Science and Technology
All Institutions: Southern University of Science and Technology, Shenzhen Key Laboratory of Robotics and Computer Vision
[One sentence main contribution]. The paper introduces GNFormer, a Gaussian Neural Field Transformer that efficiently predicts camera-based occupancy by combining sparse Gaussian primitives with a neural decoder, achieving state-of-the-art accuracy and speed with significantly fewer primitives than prior methods.
The paper proposes Gaussian Neural Field (GNF), a hybrid representation that combines explicit Gaussian primitives with a neural network decoder. The core idea is to use Gaussians to define spatial support and provide local features, while the neural network handles the complex semantic mapping and shape modeling that pure Gaussian functions struggle with. This addresses the known limitations of pure Gaussian splatting or occupancy methods where primitives are either too rigid or inefficiently utilized in empty space. The architecture, GNFormer, is a transformer-based predictor that outputs these GNF parameters. The method is logically sound, addressing a specific inefficiency in current camera-based occupancy prediction by concentrating representational capacity on occupied regions.
The authors evaluate on three major benchmarks: Occ-ScanNet (indoor), SSCBench-KITTI-360, and nuScenes-SurroundOcc (outdoor). The results claim state-of-the-art IoU and mIoU among single-frame methods, along with the fastest inference speed. A key strength is the efficiency claim: using only 256 Gaussians for indoor and 1024 for outdoor scenes, which is significantly fewer than prior Gaussian-based methods. This suggests a high signal-to-noise ratio in the representation. However, the provided text is a skeleton (sections are placeholders), so the depth of ablation studies and qualitative comparisons cannot be fully verified, though the abstract claims are strong.
The paper states "The code will be released," which is standard for arXiv preprints but means reproducibility is currently pending. The method relies on standard transformer architectures and Gaussian primitives, which are well-understood components, aiding potential reproduction once code is available. The specific hyperparameters for the neural network decoder relative to the Gaussians would be critical for replication.
The primary limitation is the reliance on single-frame input, which may limit dynamic scene understanding compared to video-based methods. Additionally, the "neural field" component adds computational overhead during inference compared to pure geometric methods, though the paper claims it is still the fastest among compared methods. The generalization to unseen environments beyond the tested benchmarks is not explicitly detailed in the abstract.
This work contributes to the trend of efficient 3D scene representations in autonomous driving and robotics. By reducing the number of primitives needed, it lowers memory and computational costs, making real-time occupancy prediction more feasible for edge devices. The hybrid approach of explicit primitives + neural decoding could inspire similar architectures in other 3D reconstruction or simulation tasks. [One sentence main contribution]. The paper introduces GNFormer, a Gaussian Neural Field Transformer that efficiently predicts camera-based occupancy by combining sparse Gaussian primitives with a neural decoder, achieving state-of-the-art accuracy and speed with significantly fewer primitives than prior methods.
Flow matching is a common generative recipe for robot foundation models. Vision-language-action models (VLAs) generate action chunks, while world-action models (WAMs) additionally generate future visual states. Both start generation from a fixed Gaussian prior, despite temporal continuity between past and future states and actions. This requires costly training and iterative sampling, especially for WAMs with an additional visual branch. We introduce $Δ$-flow, which constructs the source from recent history with mild noise rather than from a fixed Gaussian prior. This moves the source closer to the target, reducing transport to a small residual $Δ$ that can be learned efficiently and enables single-step inference. We integrate $Δ$-flow into the VLA policy $π_{0.5}$ and the WAM policies ImageWAM and Fast-WAM. On LIBERO, ImageWAM reaches 90% success in one epoch rather than six, and Fast-WAM reaches 95% in two epochs rather than eight. With LoRA fine-tuning, $π_{0.5}$ exceeds its baseline's five-epoch performance after one epoch. Single-step inference gives $1.54\times$, $2.49\times$, and $4.45\times$ speedups on $π_{0.5}$, ImageWAM, and Fast-WAM, respectively. It preserves action smoothness, achieving a $4.5\times$ reduction in action jerk compared with one-step Gaussian sampling. $Δ$-flow reaches state-of-the-art generalization performance on LIBERO-Plus. Action-history sources can be applied to pretrained WAM policies without retraining. Replacing the Gaussian action source at inference largely preserves task success with a single sampling step and can slightly improve it. We further study how action representation and visual information in action learning affect history-to-future prediction in $Δ$-flow. These results show that $Δ$-flow is simple yet effective, improving the efficiency and generalization of robot foundation models.
Primary: Nanyang Technological University
All Institutions: Nanyang Technological University, MARS Lab
The paper introduces $\Delta$-flow, a simple yet effective modification to flow matching that leverages temporal history to accelerate training and enable single-step inference in robot foundation models, demonstrating substantial gains in efficiency and generalization.
The paper proposes $\Delta$-flow, a modification to flow matching for robot foundation models (VLAs and WAMs) that replaces the standard Gaussian noise source with a perturbed version of the recent action/visual history. The core insight is that since robot states and actions are temporally continuous, the "distance" between the past and the future is small, allowing the model to learn a residual transport rather than a full generation from noise. This reduces the Lipschitz constant of the learned vector field, leading to faster convergence and enabling single-step inference. The method is integrated into existing architectures ($\pi_{0.5}$, ImageWAM, Fast-WAM) without architectural changes, only modifying the source distribution and adding an endpoint consistency loss. The approach is theoretically sound within the context of conditional flow matching and builds logically on prior work in image-to-image translation (I2SB) and action-to-action flow matching.
The experiments are extensive, covering three distinct policy architectures (VLA and two WAM variants) on the LIBERO and LIBERO-Plus benchmarks. The results demonstrate significant improvements in training efficiency (reaching high success rates in fewer epochs) and inference speed (1.5x to 4.4x speedups with single-step sampling). The generalization results on LIBERO-Plus are particularly strong, showing that the history-based source improves robustness to distribution shifts. The ablation studies provide valuable insights into the necessity of the visual branch, the impact of gradient stopping, and the choice of action representation. The comparison against Gaussian-source baselines is fair and rigorous.
The paper provides detailed implementation specifics, including hyperparameters, source construction formulas, and training objectives. The use of standard benchmarks (LIBERO) and open-source backbones (FLUX, Wan, PaliGemma) enhances reproducibility. However, the specific code for the $\Delta$-flow modifications is not explicitly linked in the provided text, though the method is described sufficiently to be implemented.
The primary limitation is the reliance on the availability of accurate historical action and visual data. In scenarios with significant sensor noise or latency, the "history" source might be less reliable. Additionally, the method is evaluated primarily on simulation benchmarks (LIBERO); performance on real-world robots with higher variability is mentioned as future work. The hyperparameter $\beta_m$ (noise scale) is noted as requiring no tuning in practice, but its sensitivity in other domains is not fully explored.
This work has significant implications for the deployment of flow-based policies in real-time robotic control. By enabling single-step inference, it removes a major bottleneck for latency-sensitive applications. The insight that temporal continuity can be exploited to simplify generative modeling is likely to influence future designs of world models and action policies. It bridges the gap between high-fidelity generative models and the efficiency requirements of control systems. The paper introduces $\Delta$-flow, a simple yet effective modification to flow matching that leverages temporal history to accelerate training and enable single-step inference in robot foundation models, demonstrating substantial gains in efficiency and generalization.
We present a learning-based framework that enables a full-scale bipedal humanoid to sprint at over 7 m/s with motion-prior-guided gait and perform aggressive cornering at high speed with a pronounced trunk lean of up to $25^\circ$. Achieving this combination is challenging: above approximately 4 m/s, periodic foot impacts induce severe, high-frequency yaw-rate oscillations intrinsic to high-speed gaits. This high-frequency temporal structure corrupts the history of proprioceptive feedback that flat-MLP policies rely on, while rendering conventional yaw-rate tracking rewards counterproductive, as they fight against the robot's natural oscillatory dynamics. Moreover, a single style prior shared across skills cannot tell which skill is commanded and inherently compromises the low-speed skill fidelity. To overcome these limitations, we adopt a causal multi-head attention (MHA) module to explicitly model the high-frequency temporal feedback. In place of the oscillatory yaw rate, we design a heading-direction reward that supervises only the low-frequency velocity direction, allowing agile turning at high speed without suppressing natural gait oscillations. We further introduce skill-conditioned multi-adversarial-motion-prior (MAMP) discriminators that condition the style reward on the commanded skill. Ablation experiments confirm that each mechanism is essential: replacing structured temporal encoding with a flat-history MLP caps the top speed; reverting to yaw-rate tracking prevents high-speed turning; and a single shared discriminator makes in-place turning collapse into a quasi-static stance. Our framework bridges the gap between raw sprinting speed and reference-motion fidelity, demonstrating stable high-speed sprinting and aggressive cornering in both simulation and real-robot deployment.
Primary: Tsinghua University
All Institutions: Tsinghua University
The paper presents a learning-based framework enabling full-scale humanoids to sprint at >7 m/s and perform aggressive cornering by introducing causal multi-head attention for high-frequency temporal modeling, heading-direction rewards to avoid suppressing natural gait oscillations, and skill-conditioned adversarial discriminators to maintain motion fidelity. This work significantly advances the state-of-the-art in high-speed bipedal locomotion by identifying and solving specific control challenges inherent to high-velocity gaits, bridging the gap between raw speed and reference-motion fidelity in both simulation and real-world deployment.
The paper proposes a learning-based framework for high-speed bipedal locomotion, addressing specific failure modes of standard flat-MLP policies at high velocities. The core methodological contributions are threefold: (1) the use of a causal multi-head attention (MHA) module to explicitly model high-frequency temporal feedback, which is argued to be necessary because periodic foot impacts above 4 m/s induce yaw-rate oscillations that corrupt flat history buffers; (2) the design of a heading-direction reward that supervises only low-frequency velocity direction, replacing conventional yaw-rate tracking which is shown to be counterproductive at high speeds; and (3) the introduction of skill-conditioned multi-adversarial-motion-prior (MAMP) discriminators to maintain fidelity across different skills (sprinting vs. cornering). The ablation studies presented in the abstract confirm the necessity of each component, showing that flat MLPs cap top speed, yaw-rate tracking prevents turning, and shared discriminators degrade skill fidelity.
The evaluation includes both simulation (Gazebo) and real-robot deployment. The paper reports achieving sprinting speeds over 7 m/s and aggressive cornering with trunk lean up to 25 degrees at 5.5 m/s. The inclusion of real-robot validation is a significant strength, as high-speed bipedal control is notoriously difficult to transfer from simulation to reality. The ablation experiments are described as rigorous, isolating the impact of the temporal encoding, reward structure, and discriminator conditioning. The results demonstrate a clear capability gap closure between raw speed and motion fidelity.
The paper is accepted to CoRL 2026, a venue known for rigorous standards. While the provided text is an abstract and figure caption, the mention of specific hyperparameters (e.g., 7 m/s, 25 degrees) and the use of standard simulation environments (Gazebo) suggest a level of detail typically found in CoRL papers. However, without the full code release or detailed hyperparameter tables in the provided snippet, full reproducibility cannot be guaranteed, though the methodology is clearly defined.
The primary limitation is the reliance on a specific hardware platform (full-scale humanoid), which may limit immediate generalizability to other morphologies. Additionally, the high-speed control regime (>4 m/s) is inherently unstable and sensitive to environmental perturbations, which may not be fully explored in the provided abstract. The "causal MHA" approach may introduce computational overhead compared to flat MLPs, though this is likely negligible for onboard controllers.
This work pushes the physical limits of bipedal robotics, demonstrating that learning-based policies can handle the complex, high-frequency dynamics of sprinting. The insights regarding the failure of flat-MLP policies at high speeds and the necessity of explicit temporal modeling for high-frequency feedback are valuable for the broader robotics community. The skill-conditioned discriminator approach offers a generalizable technique for multi-skill learning in embodied agents. The paper presents a learning-based framework enabling full-scale humanoids to sprint at >7 m/s and perform aggressive cornering by introducing causal multi-head attention for high-frequency temporal modeling, heading-direction rewards to avoid suppressing natural gait oscillations, and skill-conditioned adversarial discriminators to maintain motion fidelity. This work significantly advances the state-of-the-art in high-speed bipedal locomotion by identifying and solving specific control challenges inherent to high-velocity gaits, bridging the gap between raw speed and reference-motion fidelity in both simulation and real-world deployment.