Last 7 Days (July 21 – July 27, 2026)
Scaling executable agent training data for LLM post-training is bottlenecked by substrate-bound methods that tie task generation to predefined tools, repositories, or skill graphs: expanding coverage requires manual substrate engineering, each new domain demands a bespoke pipeline, and the resulting task distributions often reflect substrate biases rather than real-world demand. We introduce NexForge, a requirement-driven framework that takes high-level capability requirements as input and synthesizes diverse, executable agent tasks and expert trajectories for SFT. NexForge first investigates real-world demand to construct representative scenarios and task profiles, then performs distribution-aware compilation to generate task directives. For each directive, NexForge automatically retrieves or constructs the required files, dependencies, and runtime configurations, and finally synthesizes expert rollouts and produces training trajectories. Without domain-specific infrastructure, NexForge produces 3.6K terminal and 2K office tasks, improving Qwen3.5-35B-A3B Base from 22.5\% to 52.0\% on Terminal-Bench 2.0 and from 813 to 1338 Elo on GDPval; scaling further to 43.2K terminal tasks yields 58.4\%, on par with Claude Opus 4.6 equipped with Claude Code. Scaled further, NexForge-synthesized data contributes to the training of Nex-N2, a family of publicly available agent models that lift Qwen3.5-35B-A3B to 75.3\% on Terminal-Bench 2.1 and to 1585 Elo on GDPval -- achieving state-of-the-art open-source performance and surpassing several frontier proprietary systems. Nex-N2 models are available at https://nex.sii.edu.cn/.
Primary: SII (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences)
All Institutions: SII (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences)
NexForge presents a compelling and effective pipeline for scaling agent training data through requirement-driven synthesis, demonstrating that high-quality, diverse task generation can significantly boost LLM agent performance, achieving state-of-the-art open-source results on key benchmarks.
The paper introduces NexForge, a framework designed to address the data bottleneck in training LLM-based agents. The core innovation lies in shifting from "substrate-bound" task generation (which relies on predefined tools or codebases) to a "requirement-driven" approach. The methodology involves three key stages: 1) Analyzing real-world demand to create representative scenarios and task profiles; 2) Distribution-aware compilation to generate high-level task directives; and 3) Automatic synthesis of executable environments (files, dependencies, runtime configs) and expert rollouts for Supervised Fine-Tuning (SFT). This approach aims to reduce manual engineering and mitigate substrate biases. The method is technically sound, leveraging existing LLM capabilities for code generation and environment setup, but the novelty is incremental rather than foundational. It represents a sophisticated engineering pipeline rather than a new algorithmic breakthrough.
The experimental section demonstrates significant empirical improvements. Using Qwen3.5-35B-A3B as the base, the authors show a jump from 22.5% to 52.0% on Terminal-Bench 2.0 and from 813 to 1338 Elo on GDPval with 3.6K terminal and 2K office tasks. Scaling to 43.2K terminal tasks pushes performance to 58.4%, which is comparable to Claude Opus 4.6 with Claude Code. Furthermore, the synthesized data is used to train "Nex-N2," achieving state-of-the-art open-source results (75.3% on Terminal-Bench 2.1, 1585 Elo on GDPval). The results are impressive and suggest that high-quality, diverse, requirement-driven data is a critical lever for agent performance. The evaluation is rigorous, covering multiple benchmarks and comparing against strong proprietary baselines.
The paper provides a project URL (https://nex.sii.edu.cn/) which likely contains code and model weights. The description of the pipeline (requirement analysis -> directive compilation -> environment synthesis -> rollout) is detailed enough to be reproducible by a team with sufficient resources. However, the "expert rollouts" likely rely on a strong teacher model or human-in-the-loop, which can introduce variability. The specific "distribution-aware compilation" algorithm is not fully detailed in the abstract, so full reproducibility depends on the completeness of the main text and code release.
The paper does not explicitly discuss the cost of generating 43.2K high-quality tasks, which can be significant. The reliance on a "requirement-driven" approach assumes that high-level requirements can be effectively mapped to executable tasks, which may fail in domains with ambiguous or complex implicit constraints. Additionally, the "substrate biases" argument, while valid, might be overstated if the underlying LLMs themselves have biases in their training data that NexForge cannot correct. The evaluation is primarily on coding/terminal tasks; generalization to other agent domains (e.g., web browsing, multi-modal reasoning) is not demonstrated.
This work has significant implications for the democratization of capable AI agents. By providing a scalable method for generating high-quality training data, it lowers the barrier to entry for developing specialized agents. The release of Nex-N2 models contributes to the open-source ecosystem. However, the potential for misuse (e.g., generating malicious code or automating cyberattacks) is a concern that should be addressed in the broader impact statement. The success of such frameworks may accelerate the arms race in agent capabilities, raising safety and alignment challenges. NexForge presents a compelling and effective pipeline for scaling agent training data through requirement-driven synthesis, demonstrating that high-quality, diverse task generation can significantly boost LLM agent performance, achieving state-of-the-art open-source results on key benchmarks.
Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy. We introduce Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude. Together with native-resolution packing and stack-level CUDA kernel fusion, the stack supports flexible-resolution training and improves end-to-end training throughput by about 2.5times. Built on this foundation, we develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. Diffusion-NFT improves prompt following, text rendering, aesthetic quality, and editing fidelity, while few-step distillation with adversarial perceptual guidance produces 4-step Turbo models for low-latency inference. Despite its compact scale, Mage-Flow and Mage-Flow-Edit achieves competitive performance across standard generation and editing benchmarks. More importantly, the Turbo variants make high-resolution generation and editing practical for interactive use: at 1024^2 resolution on a single NVIDIA A100 GPU, Mage-Flow-Turbo generates an image in 0.59s, and Mage-Flow-Edit-Turbo edits an image in 1.02s, while maintaining a small memory footprint. These results show that careful tokenizer--backbone--system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family.
Primary: Microsoft
All Institutions: Microsoft
Mage-Flow presents a significant engineering and methodological contribution by co-designing a lightweight VAE and native-resolution diffusion transformer, achieving state-of-the-art efficiency for high-resolution image generation and editing on consumer-grade hardware, thereby democratizing access to powerful generative AI tools.
The paper proposes a co-designed generative stack, Mage-Flow, consisting of a lightweight VAE (Mage-VAE) and a native-resolution diffusion transformer. The novelty lies in the system-level co-design: using one-step diffusion-style encoding/decoding with anchor-latent regularization to drastically reduce tokenization overhead, combined with native-resolution packing and CUDA kernel fusion to enable efficient training and inference. This approach addresses the computational bottlenecks of high-resolution image generation. The use of rectified flow matching and the development of specific variants (Base, RL-aligned, Turbo) for different use cases (generation vs. editing, speed vs. quality) demonstrates a comprehensive engineering and methodological effort.
The authors present a model family including Base, RL-aligned, and Turbo variants. They report competitive performance on standard generation and editing benchmarks. Crucially, they highlight inference efficiency: 0.59s for generation and 1.02s for editing at 1024^2 resolution on a single A100 GPU. These metrics are significant for practical deployment. The evaluation covers both quality (prompt following, text rendering, aesthetics) and efficiency (latency, memory footprint), providing a robust assessment of the trade-offs.
The paper provides code, models, and a project page, which strongly supports reproducibility. The description of the architecture (Mage-VAE, Native-Resolution DiT) and training techniques (rectified flow, adversarial perceptual guidance) is detailed enough for other researchers to attempt replication, assuming access to similar computational resources.
As a 4B parameter model, it may still lag behind larger foundation models (e.g., Flux, SD3, DALL-E 3) in terms of absolute peak quality or complex semantic understanding, although the paper claims competitiveness. The "native-resolution" approach, while efficient, may still face challenges with extremely high resolutions or complex multi-subject compositions compared to models specifically optimized for those edge cases. The reliance on specific CUDA kernel fusion optimizations might limit portability to non-NVIDIA hardware.
By making high-resolution, interactive image generation and editing accessible on single GPUs, this work lowers the barrier to entry for developers and researchers. It promotes more sustainable AI by reducing the energy and hardware costs associated with training and inference. The focus on editing also has implications for creative workflows and content creation industries. Mage-Flow presents a significant engineering and methodological contribution by co-designing a lightweight VAE and native-resolution diffusion transformer, achieving state-of-the-art efficiency for high-resolution image generation and editing on consumer-grade hardware, thereby democratizing access to powerful generative AI tools.
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million-token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost -- precisely where speculation is most valuable. The effect compounds with draft length (a deep native draft can turn net-negative, slower than no speculation) and sharpens under hybrid/linear-attention targets, where cheaper verification leaves the draft's full-attention read exposed. We apply a StreamingLLM-style sliding window plus attention sink to the draft's attention only (Windowed-MTP), leaving full-attention verification intact. It is training-free, drop-in, and lossless by construction: the full-attention target still decides every accepted token, so windowing changes only which tokens are proposed, never which are accepted. It bounds the draft's KV working set to a constant, dropping ~99% of KV entries at 1M. Across three architecture families (Qwen GDN-MoE 35B/122B and a Mamba2-hybrid NoPE 120B) at 1M context on a single GPU in SGLang, windowing cuts the per-decode-step cost over the shipping native MTP draft by +28% to +44%, an input-invariant margin that widens with context. Since per-token latency is this cost divided by acceptance length, at matched acceptance end-to-end decode latency improves by the same amount, and more where windowing also lifts acceptance, while preserving the target's verified output distribution. Finally, the unread draft KV -- 7.7-11% of total KV at 1M -- is reclaimed via a compact ring buffer at no acceptance or quality cost.
Primary: NVIDIA
All Institutions: NVIDIA
This paper presents a highly effective, training-free optimization for speculative decoding in long-context LLMs, solving a critical scalability bottleneck by windowing the draft head's attention, thereby enabling efficient million-token inference without compromising output quality.
The paper addresses a critical bottleneck in long-context speculative decoding: the linear scaling of KV cache access for Multi-Token Prediction (MTP) draft heads. The proposed method, Windowed-MTP, applies a StreamingLLM-style sliding window with attention sinks exclusively to the draft head, while maintaining full attention for the target verification head. This is a clever, training-free architectural tweak that decouples the draft's context dependency from the full sequence length. The methodology is sound, leveraging the observation that draft quality degrades gracefully with windowing, while verification remains exact. It effectively transforms the draft cost from $O(N)$ to $O(1)$ (relative to context length), solving the "net-negative" speculation problem at million-token scales.
The evaluation is robust, covering three distinct architecture families (Qwen GDN-MoE and Mamba2-hybrid) at 1M context. The results demonstrate a 28-44% reduction in per-decode-step cost, which translates directly to end-to-end latency improvements. The experiments are conducted in a realistic setting (SGLang on a single GPU), ensuring practical relevance. The claim of "lossless" acceptance is supported by the fact that the target head is unchanged; the paper correctly identifies that windowing only affects *which* tokens are proposed, not *which* are accepted, preserving the output distribution. The reclamation of unread KV cache via a ring buffer is a nice engineering touch that further optimizes memory.
The method is described with sufficient detail for reproduction. The use of standard components (StreamingLLM, SGLang) and the training-free nature of the approach enhance reproducibility. The codebase is likely open-source given the NVIDIA affiliation and the nature of the contribution, though a specific URL is not provided in the text. The experimental setup is clearly defined.
The primary limitation is the potential drop in draft acceptance rate due to the reduced context window in the draft head. While the paper claims this is mitigated by the attention sink and the nature of MTP, long-range dependencies that are crucial for accurate token prediction might still be missed, potentially reducing the speedup factor (acceptance length) even if the per-step cost is lower. The performance gain is also contingent on the target model being significantly more expensive to verify than the draft is to generate, which is true for MTP but might vary for other speculative setups.
This work has significant implications for the practical deployment of LLMs with million-token contexts. By making speculative decoding efficient at scale, it enables faster inference for applications like long-document analysis, code generation, and complex reasoning tasks that require large context windows. It sets a new standard for how draft heads should be designed for long-context scenarios, likely influencing future model architectures and inference engines. This paper presents a highly effective, training-free optimization for speculative decoding in long-context LLMs, solving a critical scalability bottleneck by windowing the draft head's attention, thereby enabling efficient million-token inference without compromising output quality.
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.
Primary: Unknown
All Institutions: Unknown
This empirical study establishes the scaling laws for native multimodal pre-training, demonstrating that while language learning is robust to data composition, multimodal learning is highly sensitive, thereby providing a compute-optimal framework for training vision-language models from scratch.
The paper proposes a systematic empirical investigation into the scaling laws of native multimodal pre-training (training from scratch on joint vision-language data) versus traditional late-fusion approaches. The core methodological contribution is the derivation of "allocation laws" that describe how optimal model size and token count scale with computational budget under fixed data composition constraints. The authors model the distinct scaling behaviors of language and multimodal objectives, identifying that language learning is robust to data mixture changes, while multimodal learning is highly sensitive. This involves rigorous hyperparameter sweeps and compute-optimal configuration searches, fitting power laws to loss curves. The approach is methodologically sound, relying on large-scale empirical analysis rather than novel architectural innovations.
The experimental section appears to cover a wide range of model sizes and token counts, fitting scaling laws to derive optimal configurations. The evaluation includes downstream tasks to verify that native pre-training induces positive cross-modal transfer, specifically enhancing pure-text spatial reasoning and multimodal in-context learning. The results support the claim that text-heavy mixtures become more efficient at larger scales. However, as an empirical scaling study, the novelty of the findings is somewhat incremental to the existing body of work on LLM scaling laws (Chinchilla, etc.), now applied to the multimodal domain. The "surprise" factor is moderate; the sensitivity of multimodal allocation to data composition is a key insight, but the general framework follows established scaling law methodologies.
The paper focuses on scaling laws, which typically require significant computational resources to reproduce. The abstract mentions deriving an "efficiency frontier," implying that the authors have performed extensive training runs. While the methodology is described, the sheer scale of "native pre-training from scratch" for large models makes full reproduction difficult for most labs. However, the paper likely provides sufficient hyperparameters and data mixture details to allow for verification of the scaling exponents. The lack of code release (implied by "none" for project URL) hinders immediate reproducibility, but the empirical nature of the work allows for independent verification of the scaling trends with smaller budgets.
The primary limitation is the computational cost, which restricts the scope of the search space. The study focuses on transformer-based models, so the findings may not generalize to other architectures (e.g., Mamba, hybrid models). The "native" pre-training paradigm is still emerging, and the optimal data compositions might shift as new multimodal datasets and preprocessing techniques emerge. Additionally, the evaluation of "spatial reasoning" and "in-context learning" might be limited to specific benchmarks, and the generalization to other multimodal capabilities (e.g., fine-grained object detection, video understanding) is not explicitly detailed in the abstract.
This work provides essential guidance for practitioners building multimodal foundation models, helping them allocate compute efficiently between model capacity and data volume. By establishing that language learning is stable across mixtures while multimodal learning is sensitive, it informs data curation strategies. The finding that native pre-training enhances text-only spatial reasoning suggests that multimodal data can improve general language capabilities, encouraging broader adoption of multimodal pre-training. This could lead to more capable and efficient foundation models, with positive implications for AI safety and capability research. This empirical study establishes the scaling laws for native multimodal pre-training, demonstrating that while language learning is robust to data composition, multimodal learning is highly sensitive, thereby providing a compute-optimal framework for training vision-language models from scratch.
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million-token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost -- precisely where speculation is most valuable. The effect compounds with draft length (a deep native draft can turn net-negative, slower than no speculation) and sharpens under hybrid/linear-attention targets, where cheaper verification leaves the draft's full-attention read exposed. We apply a StreamingLLM-style sliding window plus attention sink to the draft's attention only (Windowed-MTP), leaving full-attention verification intact. It is training-free, drop-in, and lossless by construction: the full-attention target still decides every accepted token, so windowing changes only which tokens are proposed, never which are accepted. It bounds the draft's KV working set to a constant, dropping ~99% of KV entries at 1M. Across three architecture families (Qwen GDN-MoE 35B/122B and a Mamba2-hybrid NoPE 120B) at 1M context on a single GPU in SGLang, windowing cuts the per-decode-step cost over the shipping native MTP draft by +28% to +44%, an input-invariant margin that widens with context. Since per-token latency is this cost divided by acceptance length, at matched acceptance end-to-end decode latency improves by the same amount, and more where windowing also lifts acceptance, while preserving the target's verified output distribution. Finally, the unread draft KV -- 7.7-11% of total KV at 1M -- is reclaimed via a compact ring buffer at no acceptance or quality cost.
Primary: NVIDIA
All Institutions: NVIDIA
This paper presents a highly effective, training-free optimization for speculative decoding in long-context LLMs, solving a critical scalability bottleneck by windowing the draft head's attention, thereby enabling efficient million-token inference without compromising output quality.
The paper addresses a critical bottleneck in long-context speculative decoding: the linear scaling of KV cache access for Multi-Token Prediction (MTP) draft heads. The proposed method, Windowed-MTP, applies a StreamingLLM-style sliding window with attention sinks exclusively to the draft head, while maintaining full attention for the target verification head. This is a clever, training-free architectural tweak that decouples the draft's context dependency from the full sequence length. The methodology is sound, leveraging the observation that draft quality degrades gracefully with windowing, while verification remains exact. It effectively transforms the draft cost from $O(N)$ to $O(1)$ (relative to context length), solving the "net-negative" speculation problem at million-token scales.
The evaluation is robust, covering three distinct architecture families (Qwen GDN-MoE and Mamba2-hybrid) at 1M context. The results demonstrate a 28-44% reduction in per-decode-step cost, which translates directly to end-to-end latency improvements. The experiments are conducted in a realistic setting (SGLang on a single GPU), ensuring practical relevance. The claim of "lossless" acceptance is supported by the fact that the target head is unchanged; the paper correctly identifies that windowing only affects *which* tokens are proposed, not *which* are accepted, preserving the output distribution. The reclamation of unread KV cache via a ring buffer is a nice engineering touch that further optimizes memory.
The method is described with sufficient detail for reproduction. The use of standard components (StreamingLLM, SGLang) and the training-free nature of the approach enhance reproducibility. The codebase is likely open-source given the NVIDIA affiliation and the nature of the contribution, though a specific URL is not provided in the text. The experimental setup is clearly defined.
The primary limitation is the potential drop in draft acceptance rate due to the reduced context window in the draft head. While the paper claims this is mitigated by the attention sink and the nature of MTP, long-range dependencies that are crucial for accurate token prediction might still be missed, potentially reducing the speedup factor (acceptance length) even if the per-step cost is lower. The performance gain is also contingent on the target model being significantly more expensive to verify than the draft is to generate, which is true for MTP but might vary for other speculative setups.
This work has significant implications for the practical deployment of LLMs with million-token contexts. By making speculative decoding efficient at scale, it enables faster inference for applications like long-document analysis, code generation, and complex reasoning tasks that require large context windows. It sets a new standard for how draft heads should be designed for long-context scenarios, likely influencing future model architectures and inference engines. This paper presents a highly effective, training-free optimization for speculative decoding in long-context LLMs, solving a critical scalability bottleneck by windowing the draft head's attention, thereby enabling efficient million-token inference without compromising output quality.
Unlike large language models (LLMs) that exhibit strong reasoning capabilities, vision-language models (VLMs) struggle with visual reasoning, even on geometry problems that admit equivalent text, diagram, and combined diagram+text views. We show that these views often elicit different behaviors: a model may solve a problem from text but fail on the corresponding diagram, or succeed visually while failing textually. This inconsistency suggests that different views expose complementary reasoning paths and failure modes that standard multimodal post-training does not fully exploit. To study and exploit this phenomenon, we construct ODA-Data, a high-quality paired multimodal geometry dataset with text-dominant, image-dominant, and combined image+text views of the same problems, together with splits for training and evaluating modality-dependent reasoning behaviors. We then develop Modality-Informed Reciprocal Reasoning Optimization (MIRROR), a reinforcement learning approach for improving multimodal reasoning via self supervision. For each problem, MIRROR evaluates the model under all views, selects the best-performing view as a teacher, and trains other views with a reverse-KL objective towards the teacher. Across reasoning benchmarks that evaluate on geometry problems, MIRROR improves over standard RL and yields more accurate and consistent behavior across modalities
Primary: unknown
All Institutions: unknown
MIRROR makes a significant contribution to the field of multimodal AI by directly addressing the critical issue of reasoning inconsistency across modalities. By demonstrating that different views expose complementary reasoning paths and failure modes, and providing a method to exploit this, the paper paves the way for more robust and reliable VLMs. The ODA-Data dataset is a valuable resource for future research, enabling more targeted studies of multimodal reasoning. The principles of reciprocal reasoning and self-supervision from "other views" could be extended to other multimodal tasks and model architectures, potentially leading to more generalizable and human-like reasoning capabilities in AI systems. This work could inspire new evaluation metrics and training paradigms that prioritize consistency alongside accuracy. This paper presents a compelling and rigorously evaluated approach to a fundamental problem in multimodal reasoning. The authors' initial observation of modality inconsistency is a strong empirical finding, which they then effectively address with the novel ODA-Data dataset and the MIRROR framework. The methodology, which combines RL with self-supervised knowledge distillation using a "best view" teacher, is innovative and well-justified, leading to significant improvements in both accuracy and consistency across modalities. The comprehensive experiments and exceptional reproducibility details make this a highly impactful contribution to the field.
The paper introduces MIRROR, a novel reinforcement learning approach for improving multimodal reasoning by leveraging "other views." The methodology is built upon a crucial empirical observation: vision-language models (VLMs) often exhibit inconsistent reasoning across different modalities (text, diagram, combined) for the same problem. This inconsistency is rigorously demonstrated through a pilot study using PaLM-2-V and LLaVA-1.5 on geometry problems. To address this, the authors construct ODA-Data, a high-quality paired multimodal geometry dataset with equivalent problems presented in text-dominant, image-dominant, and combined views, specifically designed to study and exploit modality-dependent reasoning behaviors. MIRROR's core idea is to use self-supervision where the model's best-performing view for a given problem acts as a teacher for its other, less successful views. This is formulated as a policy optimization problem within an RL framework. The objective combines a standard reward for correct answers with a reverse-KL regularization term. This reverse-KL term encourages the answer distribution of a non-teacher view to align with that of the teacher view, effectively transferring knowledge and promoting consistency. The teacher selection mechanism, which dynamically identifies the best view based on the model's current performance, is particularly clever. The overall framework is sound, combining established techniques (RL, knowledge distillation) in a novel configuration to tackle a specific and important challenge in multimodal AI.
The experimental evaluation is comprehensive and well-executed. The authors evaluate MIRROR on two prominent VLMs, PaLM-2-V and LLaVA-1.5, across multiple reasoning benchmarks: their newly introduced ODA-Data, GeoQA+, and MathVista. Baselines include standard supervised fine-tuning and an RL-only approach (REINFORCE without the reverse-KL regularization). The results consistently demonstrate that MIRROR significantly outperforms baselines in both accuracy and, critically, consistency across modalities. For instance, MIRROR improves consistency by up to 10.7% on ODA-Data. Ablation studies clearly show the importance of the reverse-KL term, confirming its role in knowledge transfer and consistency enforcement. The experiments also include detailed analysis of how MIRROR helps models correct errors in one modality by leveraging insights from another. The use of ODA-Data's modality-dependent splits allows for a granular evaluation of how MIRROR impacts reasoning across different views. The qualitative examples further illustrate the method's effectiveness in practice. The benchmarks chosen are appropriate for evaluating geometric and general mathematical reasoning, validating the method's applicability beyond the specific ODA-Data.
Reproducibility is a strong suit of this paper. The authors provide extensive details in the appendices, which include: 1. **Computation Details:** Specifics on hardware used (TPUv4, A100 GPUs). 2. **Hyperparameters:** Detailed tables of hyperparameters for both PaLM-2-V and LLaVA-1.5 across different datasets. 3. **Prompt Templates:** Examples of the exact prompt templates used for different views (text, diagram, combined). 4. **Pseudocode:** Clear and concise pseudocode for the MIRROR algorithm, making the implementation straightforward to understand. 5. **Dataset Availability:** The ODA-Data dataset is made publicly available via a GitHub repository (https://github.com/google-research/oda-data). These details, combined with the clear methodology description, make the work highly reproducible.
One potential limitation is the reliance of the teacher selection mechanism on the model being able to solve the problem correctly in at least one view. If a problem is extremely difficult and the model fails across all modalities, the "best view" teacher might not provide a strong signal for improvement, or the reverse-KL objective might not be as effective. The current scope primarily focuses on geometry and mathematical reasoning problems; while the principles might generalize, direct applicability to other multimodal reasoning tasks (e.g., visual commonsense, instruction following) would require further validation. The computational cost of RL fine-tuning, especially with large VLMs, can also be substantial, though this is a common challenge in the field.
MIRROR makes a significant contribution to the field of multimodal AI by directly addressing the critical issue of reasoning inconsistency across modalities. By demonstrating that different views expose complementary reasoning paths and failure modes, and providing a method to exploit this, the paper paves the way for more robust and reliable VLMs. The ODA-Data dataset is a valuable resource for future research, enabling more targeted studies of multimodal reasoning. The principles of reciprocal reasoning and self-supervision from "other views" could be extended to other multimodal tasks and model architectures, potentially leading to more generalizable and human-like reasoning capabilities in AI systems. This work could inspire new evaluation metrics and training paradigms that prioritize consistency alongside accuracy. This paper presents a compelling and rigorously evaluated approach to a fundamental problem in multimodal reasoning. The authors' initial observation of modality inconsistency is a strong empirical finding, which they then effectively address with the novel ODA-Data dataset and the MIRROR framework. The methodology, which combines RL with self-supervised knowledge distillation using a "best view" teacher, is innovative and well-justified, leading to significant improvements in both accuracy and consistency across modalities. The comprehensive experiments and exceptional reproducibility details make this a highly impactful contribution to the field.
Spatial intelligence is essential for agents to move from static semantic understanding toward interacting with the physical world. Many spatial tasks are grounded in continuous visual scenes, where locations, regions, and paths are more naturally expressed by pointing, marking, or drawing than by reporting precise coordinates or discrete textual symbols. Yet existing spatial reasoning benchmarks usually require coordinates, options, or text, creating an answer-interface mismatch for image-generation models. This makes it difficult to evaluate image-generation models under the same task semantics as text-output VLMs, despite their ability to externalize spatial judgments directly in pixel space. We propose ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics. ProVisE also includes an Agentic builder that constructs and validates task-specific protocols for new benchmarks. We further introduce SpatialGen-Bench, a curated diagnostic benchmark of 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms. We evaluate representative text-output VLMs and image-generation models in a unified setting and validate Agentic protocol construction on six external spatial benchmarks. Results show that image-generation models are competitive when spatial answers can be externalized directly in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning. These findings reveal complementary strengths of pixel-space expression and text-based reasoning and establish a metric-compatible testbed for studying spatial cognition in image-generation models.
Primary: Zhejiang University
All Institutions: Zhejiang University
This paper presents a rigorous and necessary evaluation framework for spatial reasoning in generative vision models, introducing ProVisE and SpatialGen-Bench to enable metric-compatible comparisons with text-output VLMs, revealing complementary strengths in visual externalization versus compositional reasoning.
The paper introduces ProVisE, a novel evaluation framework designed to bridge the modality gap between text-output Vision-Language Models (VLMs) and image-generation models in spatial reasoning tasks. The core methodological innovation lies in "protocolized visual evaluation," where image-generation models are constrained to produce visual answers (masks, points, trajectories) that are then deterministically parsed back into the structured formats required by existing benchmarks. This avoids the unreliability of VLM-as-judge approaches. The paper also proposes an "Agentic builder" to automate the creation of these protocols for new benchmarks, enhancing scalability. The approach is technically sound, leveraging existing deterministic image processing and geometry libraries for parsing, which ensures metric compatibility.
The authors construct SpatialGen-Bench, a curated dataset of 470 samples across 14 spatial subtasks and four capability levels. They evaluate a diverse set of 31 model-interface systems, including state-of-the-art VLMs (GPT-5.4, Qwen-VL) and image generators (FLUX, Seedream, Janus). The experiments are comprehensive, covering main results, interface complementarity, agentic cross-benchmark generalization, and parser sensitivity. The results reveal a clear dichotomy: image-generation models excel when spatial answers can be externalized directly in pixel space (e.g., depth, grounding), while text-output VLMs retain an advantage in compositional reasoning. The inclusion of failure attribution analysis (distinguishing generation errors from reasoning errors) adds significant depth to the empirical contribution.
The paper provides detailed descriptions of the protocol construction, parser rules, and evaluation metrics. The benchmark construction process is well-documented, including data sources and quality control steps. The use of deterministic parsers for most tasks enhances reproducibility compared to VLM-judge methods. However, the reliance on proprietary models (GPT-5.4, GPT Image 2) for some baselines and the Agentic builder's backend limits full open reproducibility of the *construction* process, though the *evaluation* of fixed protocols is reproducible. The code and protocol artifacts are mentioned as available in a repository, which is a strong positive for reproducibility.
The paper acknowledges several limitations. The Agentic protocol construction relies on strong backends (GPT-5.4/Image 2), which may bias the protocols toward representations these models can execute. The comparison between text and visual interfaces is confounded by architectural and training data differences between the model families. The framework is currently limited to static-image benchmarks, excluding video or embodied continuous control tasks. Additionally, the "Fallback" route using VLM parsers introduces some provider-dependent variability.
This work has significant implications for the evaluation of multimodal AI systems. By enabling fair comparison between generative and discriminative models on spatial tasks, it provides a more holistic view of model capabilities. It encourages the development of hybrid systems that leverage the strengths of both pixel-space expression and text-based reasoning. The benchmark and framework can serve as a standard for future research in spatial cognition and multimodal alignment. This paper presents a rigorous and necessary evaluation framework for spatial reasoning in generative vision models, introducing ProVisE and SpatialGen-Bench to enable metric-compatible comparisons with text-output VLMs, revealing complementary strengths in visual externalization versus compositional reasoning.
Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.
Primary: Dartmouth College
All Institutions: Dartmouth College
OpenForgeRL has a substantial broader impact on the field of AI agents: 1. **Democratizing Agent Research:** By providing an open-source framework to train agents in their *real* deployment harnesses, it significantly lowers the barrier for academic researchers and smaller labs to conduct end-to-end training. This directly addresses the "train-deploy mismatch" that has increasingly favored proprietary systems. 2. **Accelerating Agent Development:** Enabling end-to-end training in complex, stateful environments means agents can learn directly from real-world interactions, leading to more robust and capable agents. This could accelerate progress in domains like software engineering, tool use, and multimodal GUI control. 3. **Facilitating Deeper Analysis:** The framework allows researchers to study the impact of harness design and RL training on agent behavior in unprecedented detail, as demonstrated by the paper's discussion section. This can lead to a better understanding of what makes agents effective and how to design better harnesses and training regimes. 4. **New Benchmarking and Data Generation Paradigms:** The automated data synthesis pipeline is a valuable contribution that can help create more diverse and challenging benchmarks for agent evaluation, especially in data-scarce domains. 5. **Bridging the Gap to Frontier Models:** By allowing open models to be trained in the same sophisticated harnesses used by frontier models, OpenForgeRL helps close the capability gap between open and closed-source agent systems. Overall, OpenForgeRL is a timely and impactful contribution that promises to unlock new avenues for research and development in the rapidly evolving field of AI agents. OpenForgeRL introduces a scalable, open-source framework for training harness-based AI agents end-to-end in any environment, bridging the critical gap between complex inference harnesses and standard RL training stacks. This paper presents a robust engineering solution with a lightweight proxy and Kubernetes orchestrator, validated by extensive empirical results across diverse text-based tool-use and multimodal GUI environments, where it outperforms open baselines and provides valuable insights into harness design and the behavioral impact of RL.
OpenForgeRL addresses a critical bottleneck in training modern AI agents: the "train-deploy mismatch" caused by complex, stateful inference harnesses that are difficult to integrate with standard open-source SFT/RL stacks. The proposed methodology is a well-engineered solution built on two main components: 1. **Lightweight Proxy:** This component abstracts the harness's inference process, decoupling it from the training loop. It serves model calls from the harness while recording prompt-response pairs, which are then reconstructed into standard training samples compatible with any RL codebase (e.g., veRL). This is a clever way to bridge the gap between complex, proprietary-like harnesses and generic RL frameworks. 2. **Kubernetes Orchestrator:** Following the design of Orchard, this orchestrator manages the lifecycle of remote containerized rollouts on cloud providers like Microsoft Azure. This enables scalable, elastic execution of rollouts, addressing the challenge that complex harnesses require dedicated, containerized environments that cannot be co-located on training nodes. The paper also details practical considerations for robust operation at scale: * **Asynchronous Rollout and Timeouts:** Imposes wall-clock timeouts on remote rollout jobs to prevent unresponsive rollouts from stalling training, a crucial feature for stability in distributed systems. * **Error Handling:** Discards samples from trajectories that end in non-policy-related errors (e.g., network issues, harness crashes) to avoid injecting misleading training signals. While simple, it's a pragmatic first step. * **Data Synthesis Pipeline:** A significant methodological contribution is the automated pipeline for synthesizing SFT and RL tasks, particularly for data-scarce domains like GUI and computer-use. This pipeline mimics human curation, involving proposal, pruning, environment building, testing with an open LLM/VLM, and refinement. This addresses a major practical challenge in expanding agent research to new domains. Overall, the methodology is sound, practical, and directly tackles the stated problem with a robust system design. It leverages existing technologies (Kubernetes, proxies) in a novel configuration to solve a specific, high-impact ML engineering challenge.
The experimental evaluation is comprehensive, broad, and rigorous, demonstrating the effectiveness and versatility of OpenForgeRL across diverse agentic settings. 1. **Breadth of Environments and Harnesses:** The framework is validated across a wide spectrum: * **Claw Agents (Text-based Tool-use):** Evaluated on ClawEval, QwenClawBench, and MCPAtlas, using various harnesses like ZeroClaw, OpenClaw, and Codex, in addition to a simple loop. * **GUI Agents (Multimodal Browser/Computer-use):** Evaluated on OSWorld-Verified (computer-use), Online-Mind2Web, and WebVoyager (browser-use), using modified Kimi-Agent and Molmo-Web harnesses. This extensive coverage strongly supports the claim of "any harness in any environment." 2. **Strong Empirical Results:** * **Claw Agents:** OpenForgeClaw (30B-A3B MoE) significantly outperforms open baselines of similar size and the untrained backbone model across all three benchmarks (e.g., 31.7 pass^3 on ClawEval, 33.7 on QwenClawBench). The SFT+RL models consistently show substantial improvements over SFT-only, highlighting the efficacy of the end-to-end training. * **GUI Agents:** OpenForgeGUI (8B) achieves superior results on nearly all GUI benchmarks compared to similar-sized models, and impressively matches or surpasses models several times larger (e.g., 63.0 on Online-Mind2Web, 72.3 on WebVoyager, outperforming MolmoWeb trained on 200k tasks with only 2.5k tasks). The consistent gains from RL in this complex multimodal setting are particularly noteworthy. 3. **Valuable Discussion and Analysis:** Beyond benchmark scores, the paper provides insightful analysis enabled by the framework: * **Cross-Harness Comparison:** Reveals that simpler, better-aligned harnesses (e.g., OpenForgeRL's loop, ZeroClaw) are easier to learn and yield higher performance than more complex ones (OpenClaw, Codex), even with advanced features. * **Generalization to Unseen Harnesses:** Demonstrates that training on one harness generalizes to others, and multi-harness training further improves robustness and performance across the board. This is a crucial finding for practical agent development. * **Capabilities Learned by RL:** Detailed behavioral analysis shows that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans. It also teaches the model to prefer specialized tools over generic ones. This granular insight into *what* RL contributes is highly valuable. 4. **Data Synthesis Validation:** The data synthesis pipeline is shown to be effective in generating useful training data, especially for GUI tasks where data is scarce, enabling the strong performance observed. The experiments are well-designed, the results are compelling, and the analytical discussion adds significant depth, making a strong case for the framework's impact.
The paper makes a strong commitment to reproducibility: * **Code, Data, and Models Release:** The authors explicitly state, "We will release our code, data, and models to facilitate research on harness-based agents." This is the most critical factor for reproducibility. * **Detailed Appendices:** The appendices provide extensive details on training hyperparameters (tab:training-hparams), training curves (fig:claw-curve, fig:computeruse-curve), and a thorough description of the data synthesis pipeline. * **Environment and Harness Details:** Specifics on Kubernetes pod configurations (CPU, RAM), cloud providers (Microsoft Azure), GPU types (B200), and modifications to existing harnesses (e.g., Kimi-Agent, MolmoWeb) are provided. * **Evaluation Protocols:** Clear descriptions of evaluation benchmarks, metrics, and specific protocols (e.g., claim-coverage for MCPAtlas, AgentTrek for Online-Mind2Web) are given. While the code is not yet publicly available, the level of detail provided suggests a strong intent and capability for future reproducibility.
1. **Error Recovery Weakness:** The paper explicitly identifies that "error recovery, however, remains the weakest capability even after RL." This is a significant limitation for agents operating in real-world, noisy environments. The hypothesis that it may require dedicated data or training methods is a good starting point for future work. 2. **Cost of Data Synthesis:** The data synthesis pipeline, while effective, is noted to be "costly in both time and money," especially for RL tasks with robust verifiers (e.g., 16.1 minutes and 4.36 USD per Claw RL task, 21.3 minutes and 6.12 USD per GUI RL task). This could limit its accessibility for researchers without substantial compute budgets. 3. **Engineering-focused Solution:** While a strength in addressing a practical problem, OpenForgeRL is primarily an engineering and systems solution rather than a fundamental algorithmic breakthrough in RL or agent intelligence. Its impact relies on enabling better training of *existing* models and algorithms. 4. **Reliance on External Models for Data Synthesis/Evaluation:** The data synthesis pipeline relies on powerful, often proprietary, LLMs (Claude Opus 4.6, GPT-5.4) for task proposal, pruning, and judging. Similarly, evaluation often uses models like GPT-4o or Gemini 2.5 Pro as judges. This dependency means the quality and cost of the generated data and evaluation are tied to these external services. 5. **Overhead of Proxy/Orchestrator:** While "lightweight," introducing a proxy and Kubernetes orchestrator inherently adds some overhead and complexity compared to a fully integrated, local training setup, though this is a necessary trade-off for the problem being solved.
OpenForgeRL has a substantial broader impact on the field of AI agents: 1. **Democratizing Agent Research:** By providing an open-source framework to train agents in their *real* deployment harnesses, it significantly lowers the barrier for academic researchers and smaller labs to conduct end-to-end training. This directly addresses the "train-deploy mismatch" that has increasingly favored proprietary systems. 2. **Accelerating Agent Development:** Enabling end-to-end training in complex, stateful environments means agents can learn directly from real-world interactions, leading to more robust and capable agents. This could accelerate progress in domains like software engineering, tool use, and multimodal GUI control. 3. **Facilitating Deeper Analysis:** The framework allows researchers to study the impact of harness design and RL training on agent behavior in unprecedented detail, as demonstrated by the paper's discussion section. This can lead to a better understanding of what makes agents effective and how to design better harnesses and training regimes. 4. **New Benchmarking and Data Generation Paradigms:** The automated data synthesis pipeline is a valuable contribution that can help create more diverse and challenging benchmarks for agent evaluation, especially in data-scarce domains. 5. **Bridging the Gap to Frontier Models:** By allowing open models to be trained in the same sophisticated harnesses used by frontier models, OpenForgeRL helps close the capability gap between open and closed-source agent systems. Overall, OpenForgeRL is a timely and impactful contribution that promises to unlock new avenues for research and development in the rapidly evolving field of AI agents. OpenForgeRL introduces a scalable, open-source framework for training harness-based AI agents end-to-end in any environment, bridging the critical gap between complex inference harnesses and standard RL training stacks. This paper presents a robust engineering solution with a lightweight proxy and Kubernetes orchestrator, validated by extensive empirical results across diverse text-based tool-use and multimodal GUI environments, where it outperforms open baselines and provides valuable insights into harness design and the behavioral impact of RL.
Traditional agent development is split across prompt templates, tool schemas, callback code, and workflow graphs. We present NVIDIA Object-Oriented Agents (NOOA), a model-agnostic Python framework for building reliable AI agents. NOOA takes a simpler approach: an agent is a Python object. Its methods are the actions the model can take, fields are its state, docstrings are its prompts, and its type annotations are contracts. A method whose code body consists of "..." is completed at runtime by an LLM-driven agent loop, while methods with normal bodies remain standard deterministic Python. This gives developers and agents the same interface, so agent behavior can be tested, traced, refactored, and improved just like other software. This paper makes three contributions. (1) We present the agent-as-a-Python-object programming model and the design principles behind it. Where Python has existing abstractions, we adopt them directly. Agent-specific capabilities--context, events, state rendering, long-term memory, and validated LLM loops--are exposed through simple Pythonic APIs, so both developers and agents share one familiar programming model. (2) We identify six model-facing ideas that NOOA is, to our knowledge, the first to combine on a single surface: typed input/output, pass-by-reference over live objects, code as action, programmable loop engineering, explicit object state, and model-callable harness APIs for context and events. We find the community already converging on several of these ideas--often as experimental or partial features--and present the comparison to encourage further adoption. (3) We demonstrate that current models use this interface effectively, both in targeted capability tests and on agentic and reasoning benchmarks such as SWE-bench Verified and Terminal-Bench 2.0 and ARC-AGI-3.
Primary: NVIDIA
All Institutions: NVIDIA
NVIDIA Object-Oriented Agents (NOOA) presents a compelling framework that unifies agent development with standard Python software engineering practices, demonstrating that current LLMs can effectively utilize native object-oriented interfaces for agentic tasks, thereby improving reliability, testability, and performance on complex benchmarks.
The paper introduces NVIDIA Object-Oriented Agents (NOOA), a framework that reifies the concept of an "agent" as a standard Python class. The core methodological innovation is the use of Python's native object-oriented features (methods, fields, docstrings, type hints) to define agent behavior, state, and contracts. Specifically, methods with an ellipsis (`...`) body are treated as agentic loops executed by an LLM, while methods with concrete bodies are deterministic Python code. The framework implements a "pass-by-reference" mechanism where the LLM operates on live Python objects via a restricted REPL, using bounded previews to manage context window constraints. Context management is handled through explicit, model-callable APIs for static/dynamic blocks and event history. This approach aims to unify developer and agent interfaces, leveraging the LLM's pre-existing knowledge of Python syntax and semantics rather than forcing it into a new DSL or JSON-based tool-calling schema.
The evaluation is comprehensive and rigorous, covering both interface fluency and end-to-end performance. The authors conduct targeted capability tests on 10 models (4,400 records), demonstrating that current frontier models (GPT-5.5, Claude Opus 4.8, etc.) achieve >90% pass rates on understanding the interface, with smaller models benefiting significantly from reasoning modes. End-to-end benchmarks include SWE-bench Verified, Terminal-Bench 2.0, CyberGym L1, and ARC-AGI-3. The results show competitive performance, particularly noting that on ARC-AGI-3, NOOA achieves a better score-cost Pareto frontier by compressing multi-agent systems into a single agent with a one-page skill. The comparison with 14 other frameworks highlights NOOA's unique combination of six key interface capabilities (Typed I/O, Pass-by-reference, Code as action, Loop engineering, Object state, Harness APIs), arguing that while others have subsets, NOOA is the first to combine them natively.
The paper provides detailed descriptions of the loop mechanics, context rendering, and strategy implementations. The code examples are clear and illustrative. The evaluation methodology is well-specified, including the number of runs (5x) and the specific models tested. The comparison with other frameworks is based on pinned versions and source code analysis, adding transparency. However, as an arXiv preprint, the full codebase and exact experimental configurations for the benchmarks are likely available via a GitHub link (implied by "repository" mentions but not explicitly listed in the text provided), which is standard for this venue.
The authors explicitly acknowledge that executing model-written code in-process poses security risks, relying on sandboxing (e.g., OpenShell) for isolation, which trades off the simplicity of pass-by-reference. The framework is currently a library/harness, not a novel model architecture, meaning its impact depends on ecosystem adoption. The "pass-by-reference" feature, while powerful, requires models to handle bounded previews effectively, which may still fail on extremely complex or nested object structures not fully captured by the preview. The evaluation, while strong, relies on existing benchmarks; the "capability suite" is proprietary to the paper.
This work has significant potential to standardize agent development by aligning it with established software engineering practices. By treating agents as Python objects, it lowers the barrier to entry for developers and improves the reliability and testability of agents. It encourages the field to move away from brittle prompt-engineering and towards robust, typed, and verifiable agent architectures. The emphasis on "code as action" and "pass-by-reference" may influence future framework designs and model training objectives, pushing models to better understand and utilize live program state. NVIDIA Object-Oriented Agents (NOOA) presents a compelling framework that unifies agent development with standard Python software engineering practices, demonstrating that current LLMs can effectively utilize native object-oriented interfaces for agentic tasks, thereby improving reliability, testability, and performance on complex benchmarks.
The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce this bottleneck by either imposing sparsity, so that each query attends to only a small subset of keys, or by using low-rank/kernel sketches, so that global interactions are compressed into a lower-dimensional representation. We propose \emph{ELSAA}, an efficient low-rank and sparse approximation of attention. Importantly, ELSAA does \emph{not} decompose the learned projection or output matrices of the Transformer into sparse and low-rank factors. Instead, after dense projections produce $Q,K,V$, ELSAA approximates the induced attention score operator itself: a sparse branch captures selected high-similarity interactions, while a low-rank branch summarizes diffuse global interactions. Since the two branches can be normalized over supports with very different denominator mass, ELSAA introduces a denominator-aware fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch. This gives a practical framework for constructing low-rank and sparse attention outputs without materializing the full quadratic score matrix, aiming to enable longer-context training while preserving both sharp token-level interactions and broad contextual mixing.
Primary: KAIST
All Institutions: KAIST, DGIST
ELSAA makes a significant contribution to addressing the quadratic complexity bottleneck of self-attention, which is a central challenge for extending Transformers to longer contexts. 1. **Enabling Longer Contexts**: By providing a linear-time attention approximation that performs robustly at sequence lengths up to 64K and beyond, ELSAA enables the training and deployment of Transformers for applications requiring very long contexts (e.g., long-document understanding, high-resolution image processing, extended code generation, long-form dialogue). This pushes the boundaries of what is currently feasible with exact attention. 2. **Principled Hybrid Attention Design**: The denominator-aware fusion mechanism offers a novel and principled approach to combining different attention approximations. This insight could be adopted by other researchers developing hybrid attention methods, leading to more robust and effective designs. 3. **Efficiency and Accessibility**: Reducing the computational and memory cost of attention makes large Transformer models more accessible for training and inference, potentially lowering hardware requirements and energy consumption for certain tasks. 4. **Foundation for Future Research**: The paper lays a strong foundation for future work in several directions, including joint compression in parameter and attention space, advanced theoretical analysis of fusion, and the development of optimized hardware kernels for hybrid attention. 5. **Understanding Attention Dynamics**: The empirical observation about the different strengths of sparse vs. low-rank attention across modalities (peaked for vision, diffuse for text) provides valuable insights into the underlying attention mechanisms, which can inform future architectural designs. ELSAA introduces a principled and highly effective method for efficient low-rank and sparse attention approximation, addressing the quadratic complexity bottleneck of Transformers for long contexts. The paper's key innovation lies in its "denominator-aware fusion" mechanism, which intelligently combines separately normalized sparse (SortLSH) and low-rank (RACE) attention branches, demonstrating superior performance and scalability on diverse long-context tasks, including those where exact attention fails due to memory limitations. This work provides a robust, linear-time solution that significantly advances the practical capabilities of Transformers for processing extended sequences, supported by extensive empirical validation and theoretical rank analysis.
ELSAA proposes an efficient low-rank and sparse approximation of the attention operator for Transformers, specifically designed to handle long input sequences. The core methodology involves two distinct branches: a sparse branch and a low-rank branch, with a novel "denominator-aware fusion" mechanism. 1. **Sparse Branch (SortLSH Exact Attention)**: This branch aims to capture sharp, high-similarity token interactions. It uses a SortLSH mechanism where queries and keys are hashed, sorted, and then grouped into fixed-size blocks. Exact attention is computed only within these blocks, leveraging the expectation that high-similarity pairs will cluster together after sorting. This provides a precise, but localized, attention output and its corresponding denominator. The paper also details a recursive divide-and-conquer causal extension for this branch. 2. **Low-Rank Branch (RACE Attention)**: This branch is responsible for summarizing diffuse global interactions. It employs RACE attention, which uses soft hash-bucket summaries to compress global key/value information into a lower-dimensional representation. This provides a global context vector without materializing all pairwise scores, yielding a low-rank output and its denominator proxy. A causal extension for RACE, based on chunking and prefix sums, is also provided. 3. **Denominator-Aware Fusion**: This is the most novel methodological contribution. The key insight is that the sparse and low-rank branches, being separately normalized over potentially different supports, can have vastly different denominator masses and scales. Simply adding their outputs can distort the overall attention. ELSAA introduces a multiplier `m_sparse,i = d_sparse,i / (d_sparse,i + alpha_i * d_lr,i + epsilon)` that rescales the sparse contribution based on its estimated denominator mass relative to the low-rank branch. This is combined with learned token-wise gates `(g_sparse,i, g_lr,i)` to produce the final fused output. This mechanism provides a principled way to balance the contributions of the two branches. 4. **Theoretical Rank Analysis**: The paper includes a theoretical section analyzing the rank of a hybrid sparse + low-rank matrix `M = S_ + BA`. It leverages concepts from robust PCA and Hall's theorem to show that such a hybrid operator can achieve full rank with high probability under natural sparse-coverage conditions, motivating the expressivity of the combined approach. The methodology is well-articulated, with clear pseudocode provided for all components and their causal extensions in the appendix. The distinction between approximating the attention *operator* versus decomposing *learned projection matrices* is important and well-explained, positioning ELSAA orthogonally to many existing sparse/low-rank methods.
The experimental evaluation is comprehensive and rigorous, covering a wide array of tasks, modalities, and sequence lengths. 1. **Diverse Benchmarks**: The evaluation spans long-document text classification (ArXiv up to 64K tokens), sentiment classification (IMDB), fine-grained image classification (Food-101, Flowers-102, Oxford-IIIT Pet up to 16K tokens), low-resolution image classification (Fashion-MNIST), text retrieval (ArXiv @ 64K), and a synthetic Needle-in-a-Haystack (NIAH) benchmark (up to 65536 tokens). Causal variants are tested on autoregressive ArXiv and Tiny ImageNet. This broad coverage effectively demonstrates the general applicability of ELSAA. 2. **Strong Baselines and Ablations**: ELSAA is compared against ExactFlash (full attention baseline), RACE (low-rank baseline), Sort_Lsh (sparse baseline), and a crucial ablation, Sort_Lsh_RACE, which combines both branches but without the denominator-aware fusion (`m_sparse=1`). This ablation clearly isolates the contribution of ELSAA's novel fusion term. 3. **Key Findings**: * **Denominator-Aware Correction Efficacy**: ELSAA consistently outperforms Sort_Lsh_RACE, demonstrating the effectiveness of the denominator-aware fusion in balancing the branches. * **Modality-Specific Strengths**: The paper identifies a dichotomy where sparse attention (Sort_LSH) excels in "peaked" attention regimes (e.g., vision tasks), while low-rank attention (RACE) is strong in "diffuse" regimes (e.g., long text). ELSAA effectively combines these strengths, often dominating vision tasks and remaining competitive on text. * **Extreme Long-Context Performance**: ELSAA demonstrates superior performance at very long contexts. On the Text Retrieval @ 64K task, ExactFlash attention collapses to 50% accuracy (random guess), while ELSAA achieves 99.97%. On NIAH, ELSAA achieves perfect retrieval up to 16K and remains strong at 32K-64K, where ExactFlash runs out of memory and RACE degrades sharply. This is a critical result, showing ELSAA's ability to enable capabilities not possible with exact attention. * **Graceful Degradation**: For shorter sequences, ELSAA remains competitive with ExactFlash, indicating it's a robust general-purpose attention layer. * **Causal Variants**: Causal ELSAA matches or exceeds baselines across lengths and modalities in autoregressive settings. 4. **Complexity Analysis**: The paper provides a clear complexity analysis, showing ELSAA's linear scaling `O(N(s + L_s * 2^beta))` compared to ExactFlash's `O(N^2)`, which is crucial for long-context applications. The experiments are well-designed to highlight the strengths of ELSAA and validate its core hypotheses. The use of a powerful GPU (NVIDIA RTX PRO 6000 Blackwell with 48 GB) ensures that the OOM results for ExactFlash are genuinely due to algorithmic limitations rather than insufficient hardware for *any* exact attention.
The paper provides a good level of detail for reproducibility. * **Code Availability**: A GitHub repository URL is provided: `https://github.com/mahdiheidari721/ELSAAhere`. * **Algorithmic Details**: Detailed pseudocode for all five branch-level procedures (non-causal/causal RACE, non-causal/causal SortLSH, and causal ELSAA fusion) is included in the appendix. * **Experimental Setup**: Hardware specifications (NVIDIA RTX PRO 6000 Blackwell GPU with 48 GB), training budget, optimizer, and evaluation protocol are stated to be consistent across variants for each task. Task-specific hyperparameters are promised in an appendix (app:experiment_hyperparameters), which is standard practice. * **Theoretical Proofs**: Proofs for the rank analysis are provided in the appendix. Overall, the paper provides sufficient information to enable reproduction of the results, assuming the provided code is functional and complete.
1. **Specific Branch Choices**: The paper instantiates ELSAA with SortLSH for the sparse branch and RACE for the low-rank branch. While these are reasonable choices, the generalizability of the "denominator-aware fusion" to other sparse (e.g., learned top-k, sliding window) or low-rank (e.g., Performer, Nyströmformer) attention mechanisms is discussed but not empirically validated. 2. **Theoretical Analysis of Fusion**: The theoretical analysis focuses on the rank properties of the hybrid operator, not directly on the bias and variance of the fused estimator under the denominator-aware rule. A complementary output-level bias-variance analysis, similar to Scatterbrain's entrywise analysis but for normalized outputs, is mentioned as future work and would strengthen the theoretical foundation of the fusion mechanism. 3. **Learnable `alpha_i` Exploration**: The coefficient `alpha_i` in the denominator-aware multiplier can be fixed or learned. While the paper shows the benefit of the fusion term, it doesn't extensively explore the impact or optimal strategies for learning `alpha_i` (e.g., token-wise vs. shared, fixed vs. scheduled). 4. **Scaling Laws and Pretraining**: The paper acknowledges that a systematic study of how branch contributions evolve with model scale, data scale, and context length, as well as pretraining of large decoder language models, are important future work. The current evaluation is primarily on classification and retrieval tasks with fixed model sizes. 5. **Fused GPU Kernels**: The practical efficiency benefits could be further amplified by developing fused GPU kernels, similar to FlashAttention, which is also noted as future work.
ELSAA makes a significant contribution to addressing the quadratic complexity bottleneck of self-attention, which is a central challenge for extending Transformers to longer contexts. 1. **Enabling Longer Contexts**: By providing a linear-time attention approximation that performs robustly at sequence lengths up to 64K and beyond, ELSAA enables the training and deployment of Transformers for applications requiring very long contexts (e.g., long-document understanding, high-resolution image processing, extended code generation, long-form dialogue). This pushes the boundaries of what is currently feasible with exact attention. 2. **Principled Hybrid Attention Design**: The denominator-aware fusion mechanism offers a novel and principled approach to combining different attention approximations. This insight could be adopted by other researchers developing hybrid attention methods, leading to more robust and effective designs. 3. **Efficiency and Accessibility**: Reducing the computational and memory cost of attention makes large Transformer models more accessible for training and inference, potentially lowering hardware requirements and energy consumption for certain tasks. 4. **Foundation for Future Research**: The paper lays a strong foundation for future work in several directions, including joint compression in parameter and attention space, advanced theoretical analysis of fusion, and the development of optimized hardware kernels for hybrid attention. 5. **Understanding Attention Dynamics**: The empirical observation about the different strengths of sparse vs. low-rank attention across modalities (peaked for vision, diffuse for text) provides valuable insights into the underlying attention mechanisms, which can inform future architectural designs. ELSAA introduces a principled and highly effective method for efficient low-rank and sparse attention approximation, addressing the quadratic complexity bottleneck of Transformers for long contexts. The paper's key innovation lies in its "denominator-aware fusion" mechanism, which intelligently combines separately normalized sparse (SortLSH) and low-rank (RACE) attention branches, demonstrating superior performance and scalability on diverse long-context tasks, including those where exact attention fails due to memory limitations. This work provides a robust, linear-time solution that significantly advances the practical capabilities of Transformers for processing extended sequences, supported by extensive empirical validation and theoretical rank analysis.
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization. Existing adaptive rank methods address this problem mainly through carefully designed importance scores constructed from gradient-derived sensitivity and uncertainty measures, without an explicit statistical interpretation. In this paper, we formulate LoRA rank allocation as a statistical hypothesis testing problem and propose StatLoRA, a statistical inference-based rank allocation method. StatLoRA associates each LoRA component with a test statistic and uses estimated p-values to determine which components should be retained or pruned under a prescribed rank budget. The proposed testing procedure is supported by our central limit theory for stochastic optimizer trajectories. In particular, we establish asymptotic normality for a broad class of commonly used optimizers in deep learning, including AdamW, and derive the corresponding asymptotic distributions for the proposed component scores used in hypothesis testing. We evaluate StatLoRA on LoRA fine-tuning of DeBERTaV3-base, BART-Large, and Qwen2.5-7B across natural language understanding, natural language generation, and question answering tasks. Experiments show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and IGU-LoRA under matched rank budgets. Sensitivity analyses and empirical diagnostics further support the stability of the proposed hypothesis-testing-based allocation rule and provide empirical evidence for the asymptotic theory of component scores.
Primary: Department of Mathematics and Department of Electrical and Computer Engineering
All Institutions: Department of Mathematics, Department of Electrical and Computer Engineering
This paper has significant broader impact potential. Firstly, the establishment of central limit theory for adaptive optimizers (AdamW, Adam, Adafactor) is a fundamental theoretical contribution to the field of deep learning optimization. This work provides a rigorous framework for understanding the asymptotic behavior and uncertainty of iterates generated by these widely used optimizers, which can pave the way for more principled uncertainty quantification, confidence interval construction, and hypothesis testing in various deep learning contexts beyond LoRA. Secondly, StatLoRA introduces a statistically principled approach to resource allocation in parameter-efficient fine-tuning, offering a robust alternative to heuristic methods. This can lead to more stable, efficient, and interpretable fine-tuning processes, especially as LLMs continue to grow in size and complexity. The ability to quantify uncertainty in component contributions can help in making more informed decisions about model architecture and adaptation strategies, potentially improving generalization and reducing overfitting. This work could inspire further research into statistical inference for other aspects of deep learning training and model compression. This paper makes a significant theoretical contribution by establishing central limit theory for adaptive optimizers like AdamW, Adam, and Adafactor, and applies this theory to propose StatLoRA, a statistically principled method for LoRA rank allocation based on hypothesis testing. The methodology is rigorously developed, and the experimental evaluation demonstrates competitive performance against existing methods, offering a novel and robust approach to an important problem in parameter-efficient fine-tuning.
The methodology is exceptionally strong, combining deep theoretical contributions with a practical application. The core idea is to formulate LoRA rank allocation as a statistical hypothesis testing problem, moving beyond heuristic importance scores. This is a significant conceptual shift. The paper's most substantial methodological contribution is the establishment of central limit theory (CLT) for stochastic optimizer trajectories, specifically for adaptive optimizers like AdamW, Adam, and Adafactor. This extends classical stochastic approximation theory, which largely focused on SGD, to modern deep learning optimizers. The derivation involves representing these optimizers as general stochastic approximation recursions with augmented state variables and then applying martingale central limit theorems. The subsequent application of the delta method to derive asymptotic distributions for empirical LoRA component scores is a sound and rigorous way to bridge the gap between optimizer dynamics and component importance. StatLoRA, the proposed algorithm, leverages these derived p-values to make principled retain-or-prune decisions under a fixed rank budget. The use of Polyak-Ruppert averaging for score estimation and a batch-means estimator for variance is practical. The choice of a one-sided hypothesis test ($H_0: s_{\ell,j}^* \ge \tau$) is well-aligned with the pruning objective. The theoretical rigor and the novel application of statistical inference to a practical deep learning problem are highly commendable.
The experimental evaluation is comprehensive and well-designed. StatLoRA is evaluated on LoRA fine-tuning of three diverse large language models: DeBERTaV3-base (NLU), BART-Large (NLG), and Qwen2.5-7B (QA). This covers a good range of model sizes and tasks, demonstrating the method's applicability across different LLM domains. The comparison includes vanilla LoRA and two state-of-the-art adaptive rank allocation methods, AdaLoRA and IGU-LoRA, under matched rank budgets. The results indicate that StatLoRA achieves "comparable or better performance" than these baselines. While "comparable or better" is not a dramatic breakthrough in empirical performance, it is a strong result given the principled statistical foundation of StatLoRA, suggesting that the method is at least as effective as existing heuristics while offering explicit uncertainty quantification. Furthermore, the paper includes valuable sensitivity analyses for statistical hyperparameters (e.g., the threshold $\tau$) and empirical diagnostics to support the stability of the allocation rule and provide evidence for the asymptotic normality of component scores. These diagnostics are crucial for validating the theoretical claims in a practical setting and enhance the credibility of the approach.
The theoretical derivations, particularly the central limit theory for adaptive optimizers, are detailed in the main text and referenced appendices, providing a strong foundation for reproducibility of the theoretical results. The StatLoRA algorithm is clearly outlined, including the score definition, hypothesis testing procedure, and variance estimation. However, the main paper text does not include specific implementation details such as the exact hyperparameters used for each model/task, the specific value of $\tau$ chosen for experiments, or a link to a code repository. While appendices are mentioned for proofs, the absence of a public code release or detailed hyperparameter tables in the main text slightly hinders immediate practical reproducibility for practitioners. Assuming the appendices contain the full proofs and the authors would release code, the overall reproducibility would be high.
One limitation lies in the practical applicability of the theoretical assumptions required for the central limit theorems. Conditions such as almost sure convergence to a limit, local differentiability of the mean field, and the Hurwitz property of the Jacobian matrix might not always hold perfectly in the highly non-convex and complex landscapes encountered during deep learning training. While these are standard assumptions in stochastic approximation theory, their empirical validity in diverse deep learning scenarios needs careful consideration. Another potential limitation is the computational overhead associated with estimating the long-run variance of component scores using methods like batch-means, especially for models with a very large number of LoRA components. This might add to the training time, although the paper claims it preserves the standard LoRA training procedure. Finally, while StatLoRA achieves "comparable or better" performance, it doesn't demonstrate a significant leap in empirical performance over existing heuristic methods. The main advantage is its principled nature and uncertainty quantification, which might not immediately convince practitioners to switch if the performance gains are marginal and the method is more complex to implement. The choice of the threshold $\tau$ for the null hypothesis also remains a hyperparameter that requires tuning.
This paper has significant broader impact potential. Firstly, the establishment of central limit theory for adaptive optimizers (AdamW, Adam, Adafactor) is a fundamental theoretical contribution to the field of deep learning optimization. This work provides a rigorous framework for understanding the asymptotic behavior and uncertainty of iterates generated by these widely used optimizers, which can pave the way for more principled uncertainty quantification, confidence interval construction, and hypothesis testing in various deep learning contexts beyond LoRA. Secondly, StatLoRA introduces a statistically principled approach to resource allocation in parameter-efficient fine-tuning, offering a robust alternative to heuristic methods. This can lead to more stable, efficient, and interpretable fine-tuning processes, especially as LLMs continue to grow in size and complexity. The ability to quantify uncertainty in component contributions can help in making more informed decisions about model architecture and adaptation strategies, potentially improving generalization and reducing overfitting. This work could inspire further research into statistical inference for other aspects of deep learning training and model compression. This paper makes a significant theoretical contribution by establishing central limit theory for adaptive optimizers like AdamW, Adam, and Adafactor, and applies this theory to propose StatLoRA, a statistically principled method for LoRA rank allocation based on hypothesis testing. The methodology is rigorously developed, and the experimental evaluation demonstrates competitive performance against existing methods, offering a novel and robust approach to an important problem in parameter-efficient fine-tuning.
As autonomous agents rapidly evolve, their ability to reliably manipulate ubiquitous digital documents has become critical for enabling general-purpose AI assistants and automating complex workspace workflows. In this paper, we introduce DocOps, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities. Based on DocOps, we systematically evaluate representative closed- and open-source models across various agentic harnesses, revealing that even the most advanced frontier configurations still exhibit profound limitations when handling highly coupled, long-range tasks. Furthermore, a fine-grained analysis of existing agents' manipulation behaviors uncovers 3 key failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. Ultimately, our work exposes the capability boundaries of agents in maintaining global document consistency, shedding light on the future design of robust, non-destructive agents for complex digital ecosystems.
Primary: Not specified in the provided text (placeholder 'Address line')
All Institutions: Not specified in the provided text (placeholder 'Address line')
DocOps makes a significant contribution to the field of autonomous agents and general-purpose AI. By providing a rigorous, verifiable benchmark for complex document operations, it addresses a critical gap in evaluating agents' ability to interact with ubiquitous digital documents. The findings expose fundamental limitations of current frontier models in maintaining global document consistency and avoiding destructive modifications, shifting research focus from isolated tool invocation to state-aware, non-destructive agent design. The identified failure modes offer clear diagnostic targets for improving agent architectures, planning mechanisms, and verification capabilities. This work will likely guide the development of more robust AI assistants for workspace automation, impacting productivity across various industries. The benchmark itself is poised to become a standard tool for researchers and practitioners, fostering innovation in a crucial area of human-computer interaction. DocOps introduces a rigorously verifiable evaluation framework and benchmark for autonomous agents performing complex, stateful document operations, revealing significant limitations of current frontier models in maintaining global consistency and avoiding destructive edits. This paper makes a substantial technical contribution by defining a novel taxonomy for document manipulation, developing a deterministic artifact-level verification system, and conducting a comprehensive empirical evaluation that uncovers critical failure modes and provides actionable insights for the design of future robust, non-destructive agents.
The methodology for DocOps is exceptionally well-conceived and rigorously designed. The core contribution is a deterministically verifiable evaluation framework for autonomous agents performing complex document operations. A key strength is the hierarchical taxonomy, which deconstructs document operations along two orthogonal axes: atomic capabilities (content, format, structure) and workflow depth (L1-L4). This allows for fine-grained diagnosis of agent failures, moving beyond coarse task-level success metrics. The task construction pipeline is robust, involving seed collection, formalization, source-artifact synthesis, and iterative human review, ensuring practical relevance, clarity, and consistency across 210 tasks. Crucially, DocOps introduces a novel deterministic verifier that directly inspects output files using native document libraries. This verifier employs three types of predicates (structural, linguistic, preservation) to not only check task completion but also to ensure structural validity and preservation of out-of-scope elements, addressing a major limitation of prior benchmarks. The fidelity of this verifier is rigorously assessed through a human audit and mutation-based stress test, demonstrating high agreement. The evaluation protocol, utilizing the Harbor framework, is standard and well-defined, ensuring reproducibility. Overall, the methodology is a significant advancement in benchmarking agent capabilities for complex, stateful digital environments.
The experimental evaluation is comprehensive and insightful. The paper systematically evaluates a diverse set of models, including leading closed-source (GPT-5.5, GPT-5.4, Claude Sonnet 4.6) and open-source (DeepSeek-V4-Pro, Qwen, Gemma, GLM) LLMs. These models are tested across four distinct agentic harnesses (DocTools, Terminus-2, Claude Code, Codex) representing different interface regimes, and with/without explicit skill injection. This broad coverage provides a holistic view of current agent capabilities. The results reveal profound limitations: even the most advanced frontier configuration (GPT-5.5 with Codex and skills) achieves only a 0.671 pass rate, which drops sharply on workflow-level (L3/L4) tasks. This is a significant empirical finding. The detailed analysis further uncovers that workflow difficulty is highly dependent on the *coupling* of underlying document states (e.g., Excel tasks degrade much more severely than PDF tasks), rather than just the number of operations. The paper also identifies and quantifies three pervasive failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata, with semantic verification gaps being the most dominant. The analysis of harness impact shows that open-ended programming environments generally outperform constrained tool use, and that skills offer non-uniform benefits, sometimes even increasing cost without significant performance gains for frontier models. The experiments are well-designed, the results are clearly presented (tables, figures), and the findings provide actionable insights for future agent development.
Reproducibility is a strong suit of this paper. The authors explicitly state that "Both the code and dataset are publicly available: https://github.com/icip-cas/DocOps". Each task is packaged as a self-contained Harbor bundle, including source artifacts, natural language instructions, optional skills, and the deterministic verifier. This packaging, combined with the use of the Harbor framework for execution, greatly facilitates replication of the experiments. The detailed descriptions of the task construction pipeline, verifier design, and experimental protocol (including model IDs and access routes in the appendix) further enhance reproducibility. The deterministic nature of the verifier, independent of LLM-as-a-judge, is a critical factor in ensuring consistent evaluation outcomes.
The authors acknowledge several limitations. DocOps focuses on deterministic, offline document-editing tasks, thus not covering workflows requiring live external services, collaborative editing, or interactive user clarification. The benchmark currently contains 210 tasks, and scaling it is labor-intensive due to the need for structurally valid artifacts, clear editing scopes, and manual review. This limits the current breadth of document domains and workflow complexities. Finally, token-cost comparisons across harnesses should be interpreted with care due to varying fidelity in usage statistics exposed by different agent runtimes. These are reasonable limitations for a novel and complex benchmark, and the authors outline plans for future expansion.
DocOps makes a significant contribution to the field of autonomous agents and general-purpose AI. By providing a rigorous, verifiable benchmark for complex document operations, it addresses a critical gap in evaluating agents' ability to interact with ubiquitous digital documents. The findings expose fundamental limitations of current frontier models in maintaining global document consistency and avoiding destructive modifications, shifting research focus from isolated tool invocation to state-aware, non-destructive agent design. The identified failure modes offer clear diagnostic targets for improving agent architectures, planning mechanisms, and verification capabilities. This work will likely guide the development of more robust AI assistants for workspace automation, impacting productivity across various industries. The benchmark itself is poised to become a standard tool for researchers and practitioners, fostering innovation in a crucial area of human-computer interaction. DocOps introduces a rigorously verifiable evaluation framework and benchmark for autonomous agents performing complex, stateful document operations, revealing significant limitations of current frontier models in maintaining global consistency and avoiding destructive edits. This paper makes a substantial technical contribution by defining a novel taxonomy for document manipulation, developing a deterministic artifact-level verification system, and conducting a comprehensive empirical evaluation that uncovers critical failure modes and provides actionable insights for the design of future robust, non-destructive agents.
Bayesian online learning promises uncertainty-aware prediction on data streams, but its performance hinges on inferential choices, including learning rates, prior distributions and variational families, which are usually fixed before seeing the stream. We address this by treating Bayesian update rules as experts and aggregating the Bayesian experts according to sequential predictive losses. We prove that the resulting aggregate competes with the best expert in hindsight at an aggregation cost determined by how each expert's per-round performance is evaluated. We instantiate the framework in online conformal inference and Gaussian process regression. The conformal inference application yields a smoothed Bayesian counterpart of adaptive conformal inference with long-run randomized coverage, while the Gaussian process application gives an oracle inequality in cumulative predictive Kullback-Leibler risk and adaptation to unknown Hölder smoothness up to logarithmic factors. Experiments show that the aggregate tracks strong experts without oracle expert selection.
Primary: Inha University
All Institutions: Inha University
This work has significant broader impact for the field of online learning and Bayesian methods. It provides a principled and modular framework for making Bayesian online learning more adaptive and robust to critical inferential choices (learning rates, priors, variational families) that are often fixed arbitrarily. This can lead to: * **More reliable uncertainty quantification:** By adapting inferential choices, the resulting predictions and uncertainty estimates are less sensitive to misspecification or nonstationarity. * **Reduced hyperparameter tuning:** The aggregation mechanism automates the selection of optimal inferential settings, reducing the need for manual tuning. * **Enhanced adaptivity:** The framework allows Bayesian models to adapt to changing data stream characteristics (e.g., smoothness, noise levels, nonstationarity) in a theoretically grounded manner. * **New theoretical insights:** The distinction between mean-loss and annealed-loss aggregation offers fundamental insights into how to evaluate and combine Bayesian predictions, with implications for other areas of online learning. The applications to conformal inference and Gaussian processes demonstrate its utility in practical, high-impact areas where robust uncertainty quantification and adaptation are crucial. This paper introduces a novel expert-aggregation framework for adaptive Bayesian online learning, distinguishing between mean-loss and annealed-loss aggregation with corresponding $O(T)$ and $O(\log K)$ regret bounds, and demonstrates its effectiveness in online conformal inference and Gaussian process regression with adaptation to unknown Hölder smoothness. The work provides a principled and modular approach to address the sensitivity of Bayesian online learning to fixed inferential choices, offering strong theoretical guarantees and comprehensive empirical validation across diverse online learning settings, thereby advancing the robustness and adaptivity of uncertainty-aware prediction systems.
The paper proposes a novel two-level framework for adaptive Bayesian online learning, treating different Bayesian update rules (experts) as distribution-valued entities and aggregating their posterior predictive distributions. The core methodological contribution lies in identifying two distinct ways to evaluate these Bayesian experts: mean-loss aggregation and annealed-loss aggregation. The authors rigorously derive regret bounds for both, showing that mean-loss aggregation generally incurs an $O(T)$ cost, while annealed-loss aggregation achieves a much faster $O(\log K)$ cost. This distinction is crucial, as it links the choice of evaluation metric to the statistical properties of the target functional (e.g., posterior mean vs. full predictive distribution). The framework is modular, allowing various Bayesian online learning algorithms (SVB, OGA) to serve as base experts. The paper then instantiates this general framework in two significant applications: online conformal inference (yielding Bayes-ACI and Bayes-DtACI) and online Gaussian process regression with unknown smoothness. For the latter, the annealed-loss aggregation is shown to adapt to unknown Hölder smoothness at minimax rates up to logarithmic factors, effectively replacing a hierarchical prior with sequential aggregation. The theoretical development is sound, leveraging established results from prediction with expert advice and extending them to the Bayesian online learning context. The discussion on the curvature of loss functions and its impact on regret rates is particularly insightful.
The experimental evaluation is comprehensive and well-designed, covering three distinct online learning scenarios. 1. **Online Variational Benchmarks:** The paper evaluates the proposed expert aggregation methods (SVB-EA, OGA-EA, OGD-EA) on standard binary classification and regression datasets. Results demonstrate that the adaptive aggregates consistently track the performance of the best fixed expert in hindsight, which is a strong indicator of successful adaptation. The sensitivity of individual experts to learning rates is clearly shown, highlighting the value of aggregation. 2. **Online Conformal Inference:** Bayes-DtACI is tested in a nonstationary setting with abrupt changes in residual scale and heavy-tailed errors. It is compared against the original DtACI. Bayes-DtACI shows more stable cumulative coverage, especially under Student-$t$ errors, suggesting improved robustness due to the Gaussian-smoothed updates. The rolling coverage also adapts smoothly to regime changes. 3. **Online GP Regression:** The annealed-loss aggregation for GPs is evaluated across three stationary settings with varying function smoothness/length scales, and a nonstationary setting. The aggregate consistently tracks the best fixed bandwidth expert in terms of cumulative negative log-likelihood. Crucially, the aggregation weights dynamically reallocate, adapting to the effective smoothness of the underlying function, which is a key theoretical claim. In the nonstationary setting, GP-EA remains competitive with strong online regression baselines and demonstrates effective adaptation to regime changes by reallocating weights. Overall, the experiments provide strong empirical evidence for the theoretical claims, demonstrating both the adaptivity and robustness of the proposed framework across diverse applications. The choice of metrics and baselines is appropriate.
The paper provides a good level of detail for reproducibility. The algorithms (Alg. 1) are clearly described. Expert specifications, meta-learning rates, and sharing parameters are detailed for each experiment. The use of specific software (Python package River, version 0.20.1) is mentioned. Experiments are repeated 30 times. While specific code is not provided (common for arXiv preprints), the methodological and experimental descriptions are sufficiently thorough for a skilled researcher to reproduce the main results.
The authors acknowledge several limitations in their future work section. 1. **Fixed-share aggregation:** The current framework primarily uses fixed-share aggregation. More advanced, strongly adaptive, or parameter-free aggregation methods could offer sharper guarantees and faster adaptation to nonstationary streams. 2. **Finite, prespecified expert collection:** The theory is developed for a finite, pre-specified grid of experts. Extending this to continuous, data-dependent, or growing expert families would be more powerful but would require controlling statistical complexity and computational cost. 3. **Computational Cost:** For a large number of experts $K$, maintaining $K$ separate Bayesian updates and their aggregation can be computationally intensive, especially for complex models like GPs. 4. **Known Noise Level in GP Theory:** The theoretical analysis for GP regression assumes a known noise level, which is not practical. While the experiments address this by aggregating over a product grid of bandwidths and noise levels, the theoretical guarantees for this extended setting are not explicitly derived, though the authors suggest it's covered by lifting the parameter space.
This work has significant broader impact for the field of online learning and Bayesian methods. It provides a principled and modular framework for making Bayesian online learning more adaptive and robust to critical inferential choices (learning rates, priors, variational families) that are often fixed arbitrarily. This can lead to: * **More reliable uncertainty quantification:** By adapting inferential choices, the resulting predictions and uncertainty estimates are less sensitive to misspecification or nonstationarity. * **Reduced hyperparameter tuning:** The aggregation mechanism automates the selection of optimal inferential settings, reducing the need for manual tuning. * **Enhanced adaptivity:** The framework allows Bayesian models to adapt to changing data stream characteristics (e.g., smoothness, noise levels, nonstationarity) in a theoretically grounded manner. * **New theoretical insights:** The distinction between mean-loss and annealed-loss aggregation offers fundamental insights into how to evaluate and combine Bayesian predictions, with implications for other areas of online learning. The applications to conformal inference and Gaussian processes demonstrate its utility in practical, high-impact areas where robust uncertainty quantification and adaptation are crucial. This paper introduces a novel expert-aggregation framework for adaptive Bayesian online learning, distinguishing between mean-loss and annealed-loss aggregation with corresponding $O(T)$ and $O(\log K)$ regret bounds, and demonstrates its effectiveness in online conformal inference and Gaussian process regression with adaptation to unknown Hölder smoothness. The work provides a principled and modular approach to address the sensitivity of Bayesian online learning to fixed inferential choices, offering strong theoretical guarantees and comprehensive empirical validation across diverse online learning settings, thereby advancing the robustness and adaptivity of uncertainty-aware prediction systems.
Scaling executable agent training data for LLM post-training is bottlenecked by substrate-bound methods that tie task generation to predefined tools, repositories, or skill graphs: expanding coverage requires manual substrate engineering, each new domain demands a bespoke pipeline, and the resulting task distributions often reflect substrate biases rather than real-world demand. We introduce NexForge, a requirement-driven framework that takes high-level capability requirements as input and synthesizes diverse, executable agent tasks and expert trajectories for SFT. NexForge first investigates real-world demand to construct representative scenarios and task profiles, then performs distribution-aware compilation to generate task directives. For each directive, NexForge automatically retrieves or constructs the required files, dependencies, and runtime configurations, and finally synthesizes expert rollouts and produces training trajectories. Without domain-specific infrastructure, NexForge produces 3.6K terminal and 2K office tasks, improving Qwen3.5-35B-A3B Base from 22.5\% to 52.0\% on Terminal-Bench 2.0 and from 813 to 1338 Elo on GDPval; scaling further to 43.2K terminal tasks yields 58.4\%, on par with Claude Opus 4.6 equipped with Claude Code. Scaled further, NexForge-synthesized data contributes to the training of Nex-N2, a family of publicly available agent models that lift Qwen3.5-35B-A3B to 75.3\% on Terminal-Bench 2.1 and to 1585 Elo on GDPval -- achieving state-of-the-art open-source performance and surpassing several frontier proprietary systems. Nex-N2 models are available at https://nex.sii.edu.cn/.
Primary: SII (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences)
All Institutions: SII (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences)
NexForge presents a compelling and effective pipeline for scaling agent training data through requirement-driven synthesis, demonstrating that high-quality, diverse task generation can significantly boost LLM agent performance, achieving state-of-the-art open-source results on key benchmarks.
The paper introduces NexForge, a framework designed to address the data bottleneck in training LLM-based agents. The core innovation lies in shifting from "substrate-bound" task generation (which relies on predefined tools or codebases) to a "requirement-driven" approach. The methodology involves three key stages: 1) Analyzing real-world demand to create representative scenarios and task profiles; 2) Distribution-aware compilation to generate high-level task directives; and 3) Automatic synthesis of executable environments (files, dependencies, runtime configs) and expert rollouts for Supervised Fine-Tuning (SFT). This approach aims to reduce manual engineering and mitigate substrate biases. The method is technically sound, leveraging existing LLM capabilities for code generation and environment setup, but the novelty is incremental rather than foundational. It represents a sophisticated engineering pipeline rather than a new algorithmic breakthrough.
The experimental section demonstrates significant empirical improvements. Using Qwen3.5-35B-A3B as the base, the authors show a jump from 22.5% to 52.0% on Terminal-Bench 2.0 and from 813 to 1338 Elo on GDPval with 3.6K terminal and 2K office tasks. Scaling to 43.2K terminal tasks pushes performance to 58.4%, which is comparable to Claude Opus 4.6 with Claude Code. Furthermore, the synthesized data is used to train "Nex-N2," achieving state-of-the-art open-source results (75.3% on Terminal-Bench 2.1, 1585 Elo on GDPval). The results are impressive and suggest that high-quality, diverse, requirement-driven data is a critical lever for agent performance. The evaluation is rigorous, covering multiple benchmarks and comparing against strong proprietary baselines.
The paper provides a project URL (https://nex.sii.edu.cn/) which likely contains code and model weights. The description of the pipeline (requirement analysis -> directive compilation -> environment synthesis -> rollout) is detailed enough to be reproducible by a team with sufficient resources. However, the "expert rollouts" likely rely on a strong teacher model or human-in-the-loop, which can introduce variability. The specific "distribution-aware compilation" algorithm is not fully detailed in the abstract, so full reproducibility depends on the completeness of the main text and code release.
The paper does not explicitly discuss the cost of generating 43.2K high-quality tasks, which can be significant. The reliance on a "requirement-driven" approach assumes that high-level requirements can be effectively mapped to executable tasks, which may fail in domains with ambiguous or complex implicit constraints. Additionally, the "substrate biases" argument, while valid, might be overstated if the underlying LLMs themselves have biases in their training data that NexForge cannot correct. The evaluation is primarily on coding/terminal tasks; generalization to other agent domains (e.g., web browsing, multi-modal reasoning) is not demonstrated.
This work has significant implications for the democratization of capable AI agents. By providing a scalable method for generating high-quality training data, it lowers the barrier to entry for developing specialized agents. The release of Nex-N2 models contributes to the open-source ecosystem. However, the potential for misuse (e.g., generating malicious code or automating cyberattacks) is a concern that should be addressed in the broader impact statement. The success of such frameworks may accelerate the arms race in agent capabilities, raising safety and alignment challenges. NexForge presents a compelling and effective pipeline for scaling agent training data through requirement-driven synthesis, demonstrating that high-quality, diverse task generation can significantly boost LLM agent performance, achieving state-of-the-art open-source results on key benchmarks.
Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy. We introduce Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude. Together with native-resolution packing and stack-level CUDA kernel fusion, the stack supports flexible-resolution training and improves end-to-end training throughput by about 2.5times. Built on this foundation, we develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. Diffusion-NFT improves prompt following, text rendering, aesthetic quality, and editing fidelity, while few-step distillation with adversarial perceptual guidance produces 4-step Turbo models for low-latency inference. Despite its compact scale, Mage-Flow and Mage-Flow-Edit achieves competitive performance across standard generation and editing benchmarks. More importantly, the Turbo variants make high-resolution generation and editing practical for interactive use: at 1024^2 resolution on a single NVIDIA A100 GPU, Mage-Flow-Turbo generates an image in 0.59s, and Mage-Flow-Edit-Turbo edits an image in 1.02s, while maintaining a small memory footprint. These results show that careful tokenizer--backbone--system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family.
Primary: Microsoft
All Institutions: Microsoft
Mage-Flow presents a significant engineering and methodological contribution by co-designing a lightweight VAE and native-resolution diffusion transformer, achieving state-of-the-art efficiency for high-resolution image generation and editing on consumer-grade hardware, thereby democratizing access to powerful generative AI tools.
The paper proposes a co-designed generative stack, Mage-Flow, consisting of a lightweight VAE (Mage-VAE) and a native-resolution diffusion transformer. The novelty lies in the system-level co-design: using one-step diffusion-style encoding/decoding with anchor-latent regularization to drastically reduce tokenization overhead, combined with native-resolution packing and CUDA kernel fusion to enable efficient training and inference. This approach addresses the computational bottlenecks of high-resolution image generation. The use of rectified flow matching and the development of specific variants (Base, RL-aligned, Turbo) for different use cases (generation vs. editing, speed vs. quality) demonstrates a comprehensive engineering and methodological effort.
The authors present a model family including Base, RL-aligned, and Turbo variants. They report competitive performance on standard generation and editing benchmarks. Crucially, they highlight inference efficiency: 0.59s for generation and 1.02s for editing at 1024^2 resolution on a single A100 GPU. These metrics are significant for practical deployment. The evaluation covers both quality (prompt following, text rendering, aesthetics) and efficiency (latency, memory footprint), providing a robust assessment of the trade-offs.
The paper provides code, models, and a project page, which strongly supports reproducibility. The description of the architecture (Mage-VAE, Native-Resolution DiT) and training techniques (rectified flow, adversarial perceptual guidance) is detailed enough for other researchers to attempt replication, assuming access to similar computational resources.
As a 4B parameter model, it may still lag behind larger foundation models (e.g., Flux, SD3, DALL-E 3) in terms of absolute peak quality or complex semantic understanding, although the paper claims competitiveness. The "native-resolution" approach, while efficient, may still face challenges with extremely high resolutions or complex multi-subject compositions compared to models specifically optimized for those edge cases. The reliance on specific CUDA kernel fusion optimizations might limit portability to non-NVIDIA hardware.
By making high-resolution, interactive image generation and editing accessible on single GPUs, this work lowers the barrier to entry for developers and researchers. It promotes more sustainable AI by reducing the energy and hardware costs associated with training and inference. The focus on editing also has implications for creative workflows and content creation industries. Mage-Flow presents a significant engineering and methodological contribution by co-designing a lightweight VAE and native-resolution diffusion transformer, achieving state-of-the-art efficiency for high-resolution image generation and editing on consumer-grade hardware, thereby democratizing access to powerful generative AI tools.
LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it. Existing observability tools replay execution traces but provide little support for identifying the root cause or translating diagnosis into recovery. We present AgentDebugX, an open-source debugging framework that organizes debugging as a closed loop of Detect, Attribute, Recover, and Rerun. At its core, DeepDebug performs multi-turn root-cause diagnosis through global trajectory understanding, structure-guided investigation, and cross-examination. On the Who and When benchmark, DeepDebug achieves the best strict attribution accuracy among the evaluated methods on both tested open-weight backbones, reaching 28.8 percent exact agent-and-step accuracy on qwen3.5-9b versus 21.7 percent for the strongest single-pass baseline. On GAIA, DeepDebug repairs 13 of 73 failed tasks in a single rerun, compared with 4 to 6 for three decoupled self-correction baselines, improving overall accuracy from 55.8 percent to 63.6 percent. AgentDebugX exposes this workflow through a Python library, CLI, web console, and installable agentic skill, and provides an opt-in Error Hub for sharing scrubbed failure-diagnosis-repair bundles and reusing them as debugging memory.
Primary: Stanford University
All Institutions: Stanford University
AgentDebugX presents a significant step forward in LLM agent observability by introducing a structured, closed-loop debugging framework that connects root-cause attribution with actionable recovery, demonstrating measurable improvements in agent repair rates on complex benchmarks.
The paper proposes AgentDebugX, a comprehensive framework for LLM agent debugging that formalizes the process as a closed loop of Detect, Attribute, Recover, and Rerun. The core methodological contribution is "DeepDebug," a multi-turn root-cause diagnostic agent. Unlike single-pass attribution methods that often fail on long-horizon traces, DeepDebug employs a structured investigation strategy: a global read followed by a structure-guided probe (bisecting for single agents, tracing handoffs for multi-agent systems) and a cross-examination phase to arbitrate between conflicting hypotheses. This approach addresses the specific challenge of "latent" errors where the symptom surface is far from the root cause. The framework also introduces a portable trajectory representation and an "Error Hub" for sharing scrubbed failure-diagnosis-repair bundles, aiming to create a cumulative debugging memory. The methodology is sound and addresses a genuine gap in the current agent observability landscape, which largely focuses on logging rather than actionable diagnosis and recovery.
The evaluation is split into two parts: attribution accuracy on the Who&When benchmark and end-to-end recovery on GAIA. On Who&When, DeepDebug achieves 28.8% strict agent-and-step accuracy on Qwen3.5-9b, outperforming the strongest single-pass baseline (21.7%). While this is a relative improvement, the absolute accuracy remains low, highlighting the difficulty of the task. On GAIA, the framework repairs 13 of 73 failed tasks in a single rerun, compared to 4-6 for decoupled self-correction baselines, improving overall accuracy from 55.8% to 63.6%. The experiments are well-controlled, comparing against relevant baselines (Reflexion, CRITIC, AutoManual) and providing ablations on the diagnostic turns. However, the GAIA evaluation is limited to a single policy model and a single rerun, which may overestimate the generalizability of the recovery gains. The attribution gains are modest in absolute terms, suggesting that while the method is state-of-the-art among evaluated approaches, the problem of automated root-cause analysis for LLM agents remains unsolved.
The paper provides an open-source toolkit, a Python library, and detailed prompts for the diagnostic agents. The code is available on GitHub, and the paper includes specific details on the trace schema and evaluation protocols. The use of standard benchmarks (Who&When, GAIA) enhances reproducibility. The inclusion of an "Error Hub" format specification also aids in future reproducibility and comparison.
The authors acknowledge several limitations. The evaluation does not measure developer debugging time or UI usability, which are critical for practical adoption. The attribution gains are model-dependent, with the multi-turn approach showing less benefit on stronger hosted models (GPT-5.4-mini, Gemini-3.5-flash) where single-pass reading is already effective. The GAIA experiment evaluates the full recipe rather than isolating the effect of attribution alone. The Error Hub and taxonomy induction features are implemented but not yet evaluated. The scrubber for sensitive data is pattern-based and may not catch all PII.
AgentDebugX has the potential to make agent reliability more inspectable and measurable, moving beyond proprietary black-box debugging. By providing an open-source toolkit and a shared format for failure cases, it lowers the barrier for researchers and smaller organizations to study robustness. The Error Hub concept could foster a community-driven corpus of agent failures, accelerating progress in agent reliability. However, the collection and sharing of agent traces raise privacy and security concerns, which the paper addresses through opt-in sharing and redaction mechanisms. AgentDebugX presents a significant step forward in LLM agent observability by introducing a structured, closed-loop debugging framework that connects root-cause attribution with actionable recovery, demonstrating measurable improvements in agent repair rates on complex benchmarks.
Summation error depends on partial-sum order, which standard worst-case bounds omit. To capture this dependence, we derive an exact mean-square error (MSE) recurrence for a binary reduction tree T under conditionally unbiased rounding. With unit roundoff u, the constant-nu model sets the local variance at pre-rounding value x to nu u^2 x^2. Its leading tree-dependent cost for the input vector p is p^T K_T p, where the common-ancestor kernel K_T counts the internal ancestors shared by each pair of leaves. For i.i.d. inputs of mean mu and variance tau^2, this expected cost is tau^2 Lambda_1(T) + mu^2 Lambda_2(T), where Lambda_1 is total leaf depth and Lambda_2 sums squared internal-subtree sizes; Lambda_1 governs centered inputs, while Lambda_2 captures nonzero means. We use these statistics to characterize optimal tree topologies and schedules. Balanced and sequential trees attain the centered extrema. For k inputs, optimal two-stage sequential blocking yields root-mean-square (RMS) error scaling as k^{3/4}. For fixed-stage hierarchies, geometric schedules are optimal for centered inputs, whereas the optimal noncentered stage exponents halve successively. For independent centered inputs with unequal variances, Huffman coding minimizes variance-weighted depth over free leaf assignments. We extend the kernel to matrix multiplication through operand Gram matrices. We then test the approximation under round-to-nearest using exact residuals. Across binary64, binary32, and software-emulated binary16 and bfloat16, the model recovers the ordering among tree topologies; K_T tracks AR(1) partial-sum costs. For GEMM, independently calibrated predictions differ from measurements by at most 3% on the tested grid. A reduction tree extracted from an array library predicts the measured RMS scaling. However, stagnation and bias in positive low-precision sums limit the model's applicability.
Primary: Oak Ridge National Laboratory
All Institutions: Oak Ridge National Laboratory
This paper provides a rigorous second-moment theory for floating-point reduction trees, deriving exact error recurrences and optimal topologies that significantly advance the understanding of numerical stability in parallel reductions, with direct applications to improving the accuracy of HPC and ML libraries.
The paper presents a rigorous theoretical framework for analyzing floating-point reduction errors, specifically focusing on the variance of partial sums in reduction trees. The core methodological contribution is the derivation of an exact mean-square error (MSE) recurrence relation for binary reduction trees under the conditionally unbiased rounding model. The authors introduce the "common-ancestor kernel" $K_T$, which quantifies the structural impact of tree topology on error propagation. They decompose the expected cost into terms dependent on input variance ($\Lambda_1$) and mean ($\Lambda_2$), allowing for the characterization of optimal tree topologies (e.g., balanced vs. sequential, Huffman coding for unequal variances). The methodology extends naturally to matrix multiplication via Gram matrices, providing a unified view of reduction error in linear algebra operations. The approach is mathematically sound, leveraging statistical properties of floating-point arithmetic rather than worst-case bounds, which offers a more realistic model for typical workloads.
The authors validate their theoretical model through extensive experiments across multiple precision formats (binary64, binary32, software-emulated binary16, and bfloat16). They demonstrate that the model accurately predicts the ordering of RMS error among different tree topologies and tracks the costs of autoregressive (AR(1)) processes. For General Matrix Multiplication (GEMM), the model's predictions differ from measurements by at most 3% on the tested grid. They also test a reduction tree extracted from an actual array library, confirming the model's predictive power for real-world implementations. The experiments are well-designed, covering both synthetic i.i.d. inputs and structured data, and effectively bridge the gap between theoretical bounds and empirical behavior.
The paper provides detailed mathematical derivations and specifies the rounding models and input distributions used in experiments. The mention of "software-emulated" formats suggests that the authors have implemented or utilized existing tools for lower-precision simulation, which aids reproducibility. However, the paper does not explicitly provide a link to the source code or specific software versions used for the simulations in the abstract or main text provided. Given the institutional context (ORNL) and the nature of the work, code is likely available or reproducible, but explicit URLs are missing from the provided text.
The authors explicitly acknowledge limitations, noting that the model's applicability is limited by stagnation and bias in positive low-precision sums. This suggests that the conditionally unbiased rounding assumption may break down in specific edge cases, particularly with low-precision formats like bfloat16 or binary16 where dynamic range and precision constraints are tighter. The model is primarily statistical (expectation/variance) and may not capture worst-case scenarios or specific pathological inputs that trigger catastrophic cancellation or overflow in ways not captured by the second-moment analysis.
This work has significant implications for the design of high-performance computing (HPC) libraries and machine learning frameworks that rely heavily on parallel reductions (e.g., dot products, sums, matrix multiplications). By providing a theory for optimal tree topologies based on input statistics, it enables the development of adaptive algorithms that minimize numerical error without sacrificing performance. This is particularly relevant for mixed-precision training and inference, where understanding and controlling error propagation is critical. The insights could lead to more robust and accurate numerical libraries for deep learning and scientific computing. This paper provides a rigorous second-moment theory for floating-point reduction trees, deriving exact error recurrences and optimal topologies that significantly advance the understanding of numerical stability in parallel reductions, with direct applications to improving the accuracy of HPC and ML libraries.
Do independently trained language models come to represent the same thing in the same way? We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2.5-Coder-7B and DeepSeek-Coder-V1-6.7B -- and measuring a complete inventory of grammatical concepts (58 Python, 57 Rust) identically in all four cells: the smallest design that separates what depends on the task, the language, and the model. The answer splits into three parts. What earns dedicated circuitry is set by the task: the models agree on which concepts receive circuits (Spearman $ρ$ = 0.638 for Python, 0.673 for Rust, both p < $10^{-7}$). Where those circuits sit is set by the model: Qwen processes concepts in a late band (~L17-19), DeepSeek at L6-7, for both languages. How circuits grow across layers is also set by the model: Qwen gives its atomic concepts an early spike that DeepSeek does not. "Are circuits universal?" thus has no single answer: yes for What, no for Where and How -- universality is a property of representational content, not of computational organisation. None of this structure was fixed in advance. The agreement could have landed anywhere between independence and identity; it lands at $ρ\approx 0.65$. Rust constructs receive 2-3x more concept-specific circuitry than their Python equivalents, in both models. Both models share neurons between the languages (6/7 and 7/7 paired constructs), DeepSeek 1.94x more than Qwen -- a direction no prior result predicts. And Qwen binds nine keywords of Rust's type-and-trait machinery into one tight neuron cluster (Jaccard 0.535 vs null 0.112, p < 0.001), a semantic dimension invisible in surface syntax. Ablation and linear probes confirm the circuits are functional. All claims are scoped to this 2x2; whether the per-model profile predicts a third model is the designed next test.
Primary: University College London
All Institutions: University College London
This work has significant broader impact for the field of mechanistic interpretability and our fundamental understanding of large language models. By systematically disentangling the "What, Where, and How" of concept representations, it provides a more nuanced and accurate view of universality, moving beyond simplistic yes/no answers. This conceptual framework is crucial for developing more robust and transferable interpretability techniques. Operationally, it offers actionable guidance for practitioners, clarifying which interpretability findings (e.g., concept inventories) are likely to transfer across models and which (e.g., layer-specific interventions) require re-localization. The discovery of "two processing styles" as a model fingerprint offers a predictive framework for characterizing new models. The method's ability to uncover semantic dimensions beyond surface syntax (e.g., Rust's type-trait cluster) suggests its potential for deeper insights into model understanding. While currently applied to formal languages, the stated goal of extending to natural language promises even wider implications for understanding how models process human language. This research contributes to building a more systematic and scientific foundation for understanding the internal workings of complex neural networks. This paper systematically disentangles the roles of task, language, and model in code model representations, revealing that representational content ("What") is largely universal, while computational organization ("Where" and "How") is model-specific, and language design influences representation strength. The authors extend a concept-circuit extraction method to a rigorous 2x2 experimental design, providing compelling empirical evidence for a nuanced view of universality, identifying a "model fingerprint" of processing styles, and demonstrating the recovery of abstract semantic dimensions beyond surface syntax, all validated through causal ablation and linear probes. This work significantly advances mechanistic interpretability by providing a systematic, comparable framework for understanding internal model representations and offering actionable insights for transferring interpretability results across models.
The paper extends a recently introduced concept-circuit extraction method (Wilam 2026) to systematically identify neuron circuits corresponding to specific grammatical concepts. This method is a significant methodological advance for interpretability, as it provides a common, comparable yardstick across different models and languages. The core idea involves generating diverse "concept prompts" to isolate concept-specific circuitry through marginalization (intersection of active neurons), and contrasting these with "checker prompts" to differentiate concept-specific responses from mere token recognition. The application of this method within a novel 2x2 experimental design (Python/Rust crossed with Qwen2.5-Coder-7B/DeepSeek-Coder-V1-6.7B) is particularly elegant, as it is precisely structured to disentangle the roles of the task (concept identity), the language, and the model. The pipeline is well-defined, encompassing prompt generation with injected variance, extraction of MLP outputs, binarization of neuron activations (with a chosen threshold of 0.5 for structural signal), marginalization via intersection, and decomposition into concept-only, shared, and token-only masks. This systematic and quantitative approach addresses a key limitation of many prior interpretability methods, which are often model-specific and lack direct comparability. While the reliance on a binary neuron mask is acknowledged as a potential limitation for capturing sub-threshold distributed information, it is a deliberate design choice that enables the crucial cross-model comparison.
The experimental evaluation is comprehensive, rigorous, and yields significant empirical findings. The choice of two distinct, large-scale code models (Qwen2.5-Coder-7B, DeepSeek-Coder-V1-6.7B) and two popular formal languages (Python, Rust) provides a robust foundation for the comparative analysis. The inventory of 58 Python and 57 Rust testable concepts is thorough, focusing on constructs that allow for the critical concept-vs-token contrast. The results are clearly presented and strongly supported by quantitative metrics. The paper convincingly demonstrates the "What/Where/How" dissociation: "What" (which concepts earn circuitry) shows significant cross-model agreement (Spearman ρ ≈ 0.65), indicating a conserved ranking of concept salience. In contrast, "Where" (layer placement) and "How" (circuit growth dynamics) diverge sharply, with Qwen processing concepts in a late band (L17-19) and DeepSeek in an early band (L6-7) for both languages, and exhibiting different early-layer dynamics for atomic concepts. Beyond this core dissociation, the experiments reveal that Rust constructs consistently receive 2-3x more concept-specific circuitry than Python equivalents, highlighting a language-design effect. Cross-language neuron sharing is observed, with DeepSeek sharing more than Qwen. A particularly striking finding is Qwen's semantic clustering of Rust's type-and-trait machinery, recovering a conceptual dimension invisible in surface syntax. Causal ablation experiments on Qwen Python provide functional validation for several concepts, confirming the identified circuits' role in model behavior. Linear probes further corroborate the decodability of concept information. The experiments are exceptionally well-designed to address the research questions and provide strong empirical evidence for the paper's claims.
The paper sets an excellent standard for reproducibility. The authors explicitly state that all analysis code, figure scripts, and a frozen-numbers test suite are released on GitHub (https://github.com/piotrwilam/Atlas2x2). Crucially, the frozen experimental artifacts (neuron activations) are released as a dataset on Hugging Face (https://huggingface.co/datasets/piotrwilam/Atlas2x2). The paper guarantees that "Every number and figure in the paper regenerates from the released analysis layer without rerunning a model," which is a gold standard for transparency and allows for full verification and extension of the research by the community.
The authors provide a transparent and well-articulated discussion of the study's limitations: 1. **Threshold structure:** The method focuses on high-amplitude neurons, potentially missing distributed sub-threshold representations, though a continuous treatment is planned. 2. **Languages only:** The study is restricted to imperative languages (Python, Rust), and generalization to declarative languages or proof assistants remains an open question. 3. **Layer-count mismatch:** The models have different layer counts (28 vs. 32), which is addressed by using both absolute indices and fraction-of-depth, but is still a factor in direct layer comparisons. 4. **Validation scope:** Causal validation is performed only on the Qwen Python cell, as DeepSeek's smoother circuit dynamics lack a clear peak layer for single-layer ablation, meaning the causal evidence for the full 2x2 design is not yet complete. 5. **Consistency parameter:** A parameter that was meaningful in prior work is degenerate for the dense SwiGLU architectures used, limiting the sweep to the activation threshold alone. 6. **Concept space mismatch:** Comparisons are restricted to shared testable subsets due to differences in the full concept space across models/languages. These acknowledged limitations provide clear avenues for future research.
This work has significant broader impact for the field of mechanistic interpretability and our fundamental understanding of large language models. By systematically disentangling the "What, Where, and How" of concept representations, it provides a more nuanced and accurate view of universality, moving beyond simplistic yes/no answers. This conceptual framework is crucial for developing more robust and transferable interpretability techniques. Operationally, it offers actionable guidance for practitioners, clarifying which interpretability findings (e.g., concept inventories) are likely to transfer across models and which (e.g., layer-specific interventions) require re-localization. The discovery of "two processing styles" as a model fingerprint offers a predictive framework for characterizing new models. The method's ability to uncover semantic dimensions beyond surface syntax (e.g., Rust's type-trait cluster) suggests its potential for deeper insights into model understanding. While currently applied to formal languages, the stated goal of extending to natural language promises even wider implications for understanding how models process human language. This research contributes to building a more systematic and scientific foundation for understanding the internal workings of complex neural networks. This paper systematically disentangles the roles of task, language, and model in code model representations, revealing that representational content ("What") is largely universal, while computational organization ("Where" and "How") is model-specific, and language design influences representation strength. The authors extend a concept-circuit extraction method to a rigorous 2x2 experimental design, providing compelling empirical evidence for a nuanced view of universality, identifying a "model fingerprint" of processing styles, and demonstrating the recovery of abstract semantic dimensions beyond surface syntax, all validated through causal ablation and linear probes. This work significantly advances mechanistic interpretability by providing a systematic, comparable framework for understanding internal model representations and offering actionable insights for transferring interpretability results across models.