Last 7 Days (July 03 – July 09, 2026)
World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling. Meanwhile, unified vision-language models have demonstrated strong multimodal generation capabilities, yet their potential as world models remains underexplored. In this work, we introduce \texttt{WorldBagel}, a unified VLAW framework built on BAGEL, a modern multimodal unified model, and use it to systematically investigate the role of unification in world modeling. Across multi-task robotic manipulation and cross-domain experiments, \texttt{WorldBagel} consistently outperforms task-specific alternatives and learns action representations that are more structured and semantically aligned with visual and linguistic context. Experiments on LIBERO, Language Table, and Franka show that unification is not only an architectural convenience, but also a key factor in learning effective VLAW models, leading to consistent empirical gains and deeper insights into multimodal world modeling. Code and model checkpoints will be released upon acceptance.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
The paper introduces WorldBagel, a unified Vision-Language-Action-World (VLAW) framework built upon the BAGEL two-tower architecture. The core methodological contribution lies in extending a powerful multimodal generative model (BAGEL) to jointly support multimodal understanding, structured action modeling, and future world prediction. The VLAW formulation is clearly defined, aiming to model the joint distribution of future observations and actions conditioned on past states and language instructions. A significant technical contribution is the Fourier Feature Action Decoder (FFAD) and Fourier Feature Action Tokenizer (FFAT). FFAD addresses the limitations of standard regression and discretization-based action tokenizers by mapping continuous actions into Fourier features and predicting in this space. The inverse mapping uses phase-consistent averaging for reconstruction. This approach is well-justified with theoretical analysis provided in the appendix, demonstrating Lipschitz stability, injectivity, consistency of reconstruction, and approximation advantages. This mathematical rigor is a strong point. The interleaved VLAW modeling via sequence plans, adapted from BAGEL, is a practical and flexible way to structure multimodal sequences for multi-view, multi-step observations and control. The concept of sampling different sequence plans to balance training objectives is sound. Furthermore, the LLM-inspired multimodal train-time data sampling, using mixture dataset sampling and priority sequence-plan sampling, is a crucial engineering detail for stabilizing training across heterogeneous datasets and balancing policy learning with world modeling. The overall architecture leverages the strengths of BAGEL's GEN/UND experts, with action modeling integrated through fine-tuned tokenizers and decoders rather than a new expert. This design choice maintains the unified nature of the model.
The experimental evaluation is comprehensive and rigorous, addressing three key empirical findings: multi-task performance, action representation quality, and stability under distribution shifts. 1. **Multi-task Performance**: WorldBagel is evaluated on LIBERO, Language Table, and Franka benchmarks. On LIBERO, it achieves state-of-the-art multi-task manipulation performance (98.0% average success rate), outperforming strong VLA baselines like OpenVLA-OFT and RynnVLA-002. The world modeling capabilities are also quantitatively assessed using FVD, PSNR, SSIM, and LPIPS, showing consistent improvements over RynnVLA-002 across all datasets, especially in action-conditioned prediction. This clearly demonstrates the empirical gains of the unified VLAW approach. 2. **Action Representation Quality**: A detailed ablation study on action decoder design (regression, bin discretization, FAST, FFAD) on LIBERO shows FFAD significantly reduces action MSE and improves success rates. Further analysis on the number of Fourier bands (K) in FFAD/FFAT provides insights into optimal hyperparameter choices. Crucially, the representation structure analysis using a linear probe classifier reveals that FFAD produces more structured and task-relevant action embeddings, leading to higher task identity prediction accuracy. This is a strong validation of the FFAD design. 3. **Stability Under Distribution Shifts**: The paper investigates robustness to action noise, scaling, and temporal perturbations on LIBERO. WorldBagel consistently maintains higher prediction fidelity (PSNR, LPIPS) compared to RynnVLA-002 under these shifts. The eigenvalue spectrum analysis further supports this, showing WorldBagel learns richer and more stable action representations (higher effective rank, lower dominant eigenvalue ratio). This finding is particularly important for real-world robotics applications where such shifts are common. The choice of baselines is appropriate, including recent strong VLA models and a direct competitor (RynnVLA-002) that also aims for VLAW unification. The use of multiple metrics (success rate, FVD, PSNR, SSIM, LPIPS, A-MSE, linear probe accuracy, eigenvalue spectrum) provides a holistic view of the model's performance and internal properties. The experiments are well-designed to support the paper's claims about the benefits of unification.
The paper states that "Code and model checkpoints will be released upon acceptance," which is a positive commitment. Detailed hyperparameters (learning rate, weight decay, batch size, training steps, K for FFAT/FFAD, priority weights) and hardware (8 H200 GPUs) are provided, which are crucial for reproducibility. The mathematical derivations for FFAD/FFAT in the appendix also contribute to understanding and potentially re-implementing those components. Given the complexity of large multimodal models, the release of code and checkpoints is essential for full reproducibility.
1. **Computational Cost**: While not explicitly stated as a limitation, training and deploying a model built on a large unified multimodal backbone like BAGEL is inherently computationally intensive, requiring significant resources (e.g., 8 H200 GPUs for 80K steps). This might limit its applicability for resource-constrained environments or rapid iteration. 2. **Scope of World Modeling**: The "world modeling" aspect primarily focuses on next-frame prediction for manipulation tasks. While crucial, it doesn't delve into more abstract forms of world knowledge, causal reasoning, or long-horizon planning beyond short action rollouts, which are often goals of broader world models. 3. **Reliance on Supervised Fine-tuning**: The model relies on supervised fine-tuning (SFT) on existing robotic datasets. While effective, this approach might be limited by the diversity and scale of available demonstration data, potentially hindering generalization to truly novel tasks or environments compared to models that learn more extensively through self-supervision or interaction. 4. **Generalizability Beyond Manipulation**: The experiments are confined to robotic manipulation tasks. While these are challenging, the generalizability of "unified VLAW modeling" to other embodied AI domains (e.g., navigation, human-robot interaction) or even broader generative tasks is not explored.
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Primary: Stanford University
All Institutions: Stanford University, MIT, Nunchux AI, UC Berkeley, CMU
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
FourTune proposes an innovative and technically sound approach for fully 4-bit efficient post-training of diffusion models, addressing the critical challenges of memory footprint and training speed. The core of the methodology is the "triple-branch hybrid pipeline" which augments a standard LoRA architecture with a "frozen numerical stabilizer" (FNS). This FNS branch is a small, frozen, full-precision component designed to isolate and process quantization-sensitive outliers, thereby enabling stable training under native W4A4G4 (weights, activations, gradients) computation. This is a particularly clever design choice, as it mitigates the primary instability issue of low-bit quantization (outliers) without incurring significant computational overhead during training, as the FNS branch remains frozen. The integration of this FNS with both a standard BF16 LoRA branch and a quantized 4-bit LoRA branch provides a robust mechanism for maintaining quality while maximizing efficiency. Furthermore, the paper introduces hardware-efficient block-wise quantization and customized fused kernels. These components are crucial for translating the theoretical benefits of 4-bit quantization into practical speedups by optimizing memory bandwidth and accelerating quantized backpropagation operations (e.g., QGEMM, QMatmul, QConv). The end-to-end W4A4G4 paradigm, encompassing weights, activations, and gradients, is a comprehensive solution that pushes the boundaries of efficient fine-tuning for large generative models.
The experimental evaluation is comprehensive and robust, demonstrating FourTune's effectiveness across diverse and important post-training tasks for diffusion models: customization (DreamBooth-like), reinforcement learning for aesthetic preference, and knowledge distillation. The choice of FLUX.1-dev (12B parameters) as the primary model, along with validation on SDXL (1.5B), ensures the results are relevant for large-scale, state-of-the-art diffusion models. Baselines include BF16 LoRA and NF4 QLoRA, which are appropriate and strong competitors. The quantitative results are highly compelling: FourTune consistently matches or slightly outperforms the image quality of full-precision BF16 LoRA and NF4 QLoRA across all tasks, as measured by FID, CLIP Score, and LPIPS. This quality preservation at such low bit-depth is a significant achievement. More importantly, FourTune delivers substantial efficiency gains, reducing GPU memory overhead by 2.25x compared to BF16 LoRA (achieving a footprint comparable to NF4 QLoRA) and increasing end-to-end training throughput by 2.27x over BF16 LoRA and 2.79x over NF4 QLoRA. This effectively breaks the memory-speed trade-off often encountered in large model post-training. The ablation studies are well-designed and clearly demonstrate the critical contribution of each proposed component: the Frozen Numerical Stabilizer for stability and quality, and the W4A4G4 quantization, block-wise quantization, and fused kernels for efficiency. The generalization to SDXL further strengthens the claims.
The paper provides a good level of detail regarding the methodology, including the architecture of the triple-branch pipeline, the role of the FNS, and the types of quantization and fused kernels used. The experimental setup is well-described, including the specific models (FLUX.1-dev, SDXL), tasks, baselines, and evaluation metrics. The appendix further elaborates on implementation details, hyperparameters, and training configurations, which are crucial for reproducibility. While no direct code link is provided, the comprehensive description suggests that a skilled research team should be able to reproduce the results.
The paper does not explicitly discuss limitations, but some can be inferred. While the FNS effectively handles outliers, its design might introduce a slight architectural overhead compared to a purely 4-bit system, even if frozen. The customized fused kernels, while highly efficient, might require specific hardware support or careful implementation to achieve optimal performance across different GPU architectures. The evaluation is primarily focused on image generation diffusion models; its applicability and performance on other types of diffusion models (e.g., audio, video) or other generative architectures (e.g., GANs, VAEs) are not explored. The paper also doesn't delve into the potential challenges of deploying such a highly optimized, custom-kernel-dependent solution in diverse production environments.
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
Primary: Yale University
All Institutions: Yale University, University of Illinois Urbana-Champaign, Microsoft Research, Amazon
Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
The paper introduces MedPMC, a systematic, automated pipeline for curating high-fidelity image-text pairs from PubMed Central (PMC). The methodology addresses a critical bottleneck in medical multimodal learning: the scarcity of large-scale, high-quality, and clinically relevant data. The pipeline involves several sophisticated components: (1) filtering 6.1 million articles for permissive licenses; (2) detecting multi-panel figures; (3) separating individual figures from panels; (4) aligning figures with their corresponding captions using natural language processing techniques; and (5) classifying figures for medical relevance. The authors report high performance on these component tasks (e.g., F1=93.2 for screening, mAP=89.8 for figure separation), indicating a robust and well-engineered curation process. The novelty lies not in a single algorithmic breakthrough but in the systematic integration and scaling of these components to create a massive, high-quality dataset, coupled with rigorous validation against prior, lower-fidelity datasets.
The evaluation is comprehensive and compelling. First, the authors validate the quality of the curated data through manual review by five annotators (three with medical training), showing a significant improvement in medical relevance (95.3% vs. 19.7% in a prior dataset). Second, they train a CLIP-style model on the MedPMC corpus and evaluate it on 26 benchmarks across 11 medical specialties. The MedPMC-trained model outperforms the strongest architecture-matched biomedical CLIP baseline by 7.1 percentage points in average zero-shot AUC, despite using fewer than half the image-text pairs. This demonstrates the high signal-to-noise ratio of the MedPMC data. Third, they integrate the vision encoder into a multimodal large language model (MLLM) and show improvements in medical visual question-answering. Finally, they demonstrate clinical utility by improving morphology-to-image retrieval on a real-world dermatology dataset from Yale New Haven Health System. The results are statistically significant and practically meaningful.
The authors provide extensive resources for reproducibility. The code for the curation pipeline is publicly available on GitHub. The MedPMC corpus, component-level benchmark resources, pretrained checkpoints, and metadata are released on Hugging Face. The data release includes versioning by article cutoff date, source-license filters, and processing configuration, allowing users to reproduce specific snapshots. They also provide clear instructions for license-aware filtering. The only non-public component is the clinical dermatology data, which is subject to privacy restrictions, but aggregate results and processing procedures are described. This level of transparency is exemplary.
The primary limitation is the reliance on open-access literature, which may introduce selection bias towards certain types of studies or institutions. The pipeline's performance, while high, is not perfect (e.g., F1=81.4 for caption separation), meaning some noise remains in the dataset. The clinical evaluation is limited to dermatology retrieval, and while promising, it does not cover the full breadth of medical specialties. The authors acknowledge that some records are derived from articles with non-commercial licenses (CC BY-NC), which restricts commercial use of the dataset. Additionally, the study focuses on image-text pairs; the extension to other modalities (e.g., audio, video, time-series) is not addressed.
The MedPMC framework has significant potential to accelerate the development of medical multimodal foundation models. By providing a large-scale, high-quality, and openly accessible dataset, it lowers the barrier to entry for researchers and clinicians working in this domain. The improved performance of models trained on MedPMC suggests that high-fidelity data curation is a key lever for improving model capabilities. The release of the code and benchmarks encourages further research in data curation and multimodal learning. However, the non-commercial license restrictions for some data may limit its impact in commercial healthcare applications. The authors have responsibly addressed privacy concerns by not releasing patient data and by providing clear licensing information. Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.
Primary: UCL
All Institutions: UCL
This paper makes a significant contribution to the ongoing debate about whether RL post-training merely reweights existing behaviors or enables new capabilities in base models. By providing a concrete, inspectable mechanism for "going beyond the base model," it offers valuable insights for understanding and improving LLM reasoning. The findings on procedural chunking and the phased emergence of compositional strategies have implications for skill acquisition research in AI. The distinction between exploration volume and selectivity, and the role of negative samples in focusing exploration on valid structures, is crucial for designing more effective RL algorithms for reasoning tasks. Furthermore, the analysis of how pretraining organizes primitive competence into a "procedural substrate" for RL highlights the importance of co-designing pretraining and post-training objectives. While the environment is synthetic, the mechanistic understanding gained here can inform hypotheses and experimental designs for more complex, real-world LLM applications. RL Post-Training Builds Compositional Reasoning Strategies provides a rigorous, mechanistic account of how reinforcement learning can enable compositional reasoning beyond the capabilities of a pretrained base model in a fully observable rewrite-grammar environment. The paper's strength lies in its novel controlled experimental setup, which allows for precise auditing of reasoning traces, revealing a phased emergence of valid compositional strategies, the critical role of RL's selectivity over exploration volume, and the importance of pretraining in organizing primitive competence into a compressible substrate for higher-level strategy formation.
The methodology is exceptionally strong due to its controlled and fully observable nature. The authors introduce a novel rewrite-grammar environment where primitive rules, pretraining histories, and every generated rewrite are exactly inspectable. This allows for a precise four-way taxonomy of actions: primitive, macro (sequential composition), parallel (simultaneous independent composition), and spurious. This environment directly addresses the opacity challenge in understanding LLM reasoning and RL post-training. A Transformer model is pretrained from scratch on primitive rewrite chains, with a controllable contraction weight, and then post-trained on a trace-based reasoning task using a binary final-answer reward. The choice of Group Relative Policy Optimization (GRPO) for RL and Rejection Fine-Tuning (RFT) as a strong on-policy comparator is appropriate, allowing for a direct comparison of their mechanisms. The definition of difficulty based on the optimal primitive solution length exceeding the generation budget is clever, forcing the model to discover shortcuts. The ability to audit every step of a generated trace against the known grammar is the lynchpin of this methodology, enabling mechanistic insights that are typically impossible in natural language settings.
The experimental evaluation is thorough and provides compelling evidence for the paper's claims. 1. **RL Expands Capability Frontier**: Experiments clearly show RL solving held-out problems (Difficulties 2-5) that the pretrained model rarely solves even with significantly larger sampling budgets (pass@1024 vs pass@16). This directly addresses the "beyond the base model" debate. 2. **RL vs. RFT Dynamics**: RFT shows strong early improvements, particularly on easier problems, by amplifying existing primitive solutions. However, RFT plateaus, while RL continues to improve, especially on harder problems where compositional strategies are essential. This highlights a crucial difference in their long-term learning dynamics. 3. **Phased Procedural Chunking**: Trace analysis reveals a delayed transition in RL. Early successful trajectories are dominated by primitive contractions, but later, macro contractions rise and overtake primitives, followed by the emergence of parallel contractions. This phased emergence of valid compositional strategies is a key finding. 4. **Discovery and Consolidation**: RL not only discovers new macro rules but also consolidates them into a stable, reusable repertoire, as evidenced by the increasing reuse of previously discovered rules. RFT shows less discovery and weaker reuse. 5. **Selectivity, Not Exploration Volume**: A critical experiment demonstrates that RFT also generates many non-primitive actions, but a large fraction of these are spurious. RL, in contrast, maintains low spurious contractions and concentrates its non-primitive exploration into valid, reusable structures. The paper provides a finite-group view of GRPO to explain this selectivity, showing how GRPO's within-prompt contrast mechanism can suppress features enriched in failures. 6. **Pretraining Substrate**: Ablations on the pretraining contraction weight (rho) show that the emergence of compositional strategies is gated not just by primitive exposure, but by whether pretraining organizes primitive competence into *chained reduction procedures* that RL can later compress. This is a significant insight into the interaction between pretraining and RL. The figures are clear and effectively convey the quantitative results. The experimental design is robust, supporting the mechanistic claims.
The paper provides a dedicated appendix with detailed information on grammar generation (alphabet size, RHS distribution), chain generation (starting/chain lengths, contraction weight), model architecture (12 layers, 512 hidden dim, 8 heads, RoPE), optimization (AdamW, learning rate schedule, bfloat16, gradient clipping), pretraining data generation, and post-training prompt generation. Crucially, it also details the GRPO and RFT algorithms, including reward definition, advantage normalization, loss function, hyperparameters (G=4, epsilon=0.2, beta=10^-3, temperature=0.8, top-k=5, max generation length=256), and how RFT differs from GRPO. This level of detail is excellent and should allow for high reproducibility of the core experiments.
The authors explicitly acknowledge several limitations: 1. **Synthetic Environment**: The environment is intentionally low-dimensional and synthetic, prioritizing mechanistic clarity over realism. The results are not claimed to be universally applicable to large language models (LLMs) trained on natural language. 2. **Generalizability**: The findings are based on one controlled family of tasks and one class of post-training algorithms (GRPO/RFT). Broader generality remains to be tested. 3. **Interpretability**: The paper does not provide activation-level interpretability of how derived strategies are represented within the Transformer. These limitations are well-stated and appropriate for a controlled mechanistic study.
This paper makes a significant contribution to the ongoing debate about whether RL post-training merely reweights existing behaviors or enables new capabilities in base models. By providing a concrete, inspectable mechanism for "going beyond the base model," it offers valuable insights for understanding and improving LLM reasoning. The findings on procedural chunking and the phased emergence of compositional strategies have implications for skill acquisition research in AI. The distinction between exploration volume and selectivity, and the role of negative samples in focusing exploration on valid structures, is crucial for designing more effective RL algorithms for reasoning tasks. Furthermore, the analysis of how pretraining organizes primitive competence into a "procedural substrate" for RL highlights the importance of co-designing pretraining and post-training objectives. While the environment is synthetic, the mechanistic understanding gained here can inform hypotheses and experimental designs for more complex, real-world LLM applications. RL Post-Training Builds Compositional Reasoning Strategies provides a rigorous, mechanistic account of how reinforcement learning can enable compositional reasoning beyond the capabilities of a pretrained base model in a fully observable rewrite-grammar environment. The paper's strength lies in its novel controlled experimental setup, which allows for precise auditing of reasoning traces, revealing a phased emergence of valid compositional strategies, the critical role of RL's selectivity over exploration volume, and the importance of pretraining in organizing primitive competence into a compressible substrate for higher-level strategy formation.
We introduce institutional red-teaming, an evaluation methodology for testing deployment rules in multi-agent AI: hold the agents, objectives, and task state fixed, vary only one rule, and attribute the resulting change in collective behavior to that rule. We instantiate the methodology in IABench-CA, a consequence-allocation benchmark spanning 228 contexts, five canonical rules, and seven model populations (33,924 games), with a normative cooperative reference and auto-labelled reasoning traces. Three findings emerge. (1) Deployment rules causally alter collective safety: changing only the consequence rule moves mean fatality by 22 to 58 percentage points within every population. (2) There is no safe default, but the targeting hazard is universal: the safest rule, the least-safe rule, and even the direction of the incidence effect vary across populations, yet regressive identity-targeting is never decisively safest in any context for any population, eliminates the least-resourced agent in 30-87% of games everywhere, and is selection-unsafe relative to the cooperative reference for all seven populations. (3) Identity salience is the mechanism: a one-shot anonymization ablation on the most exploitation-prone population (gpt-5.1) shows that merely naming the loss bearer in the rule text drives targeted elimination from 22% to 81% at identical payoffs; under repeated play, anonymization only delays the targeting, as agents re-infer the hidden rule from observed eliminations. We package the methodology as a safety-case workflow that certifies a provisional rule region $Φ(c,P)$ per deployment context and population, with explicit residual risks and monitoring obligations.
Primary: Massachusetts Institute of Technology
All Institutions: Massachusetts Institute of Technology
This paper presents a significant methodological advance in multi-agent AI safety by introducing Institutional Red-Teaming, a causal evaluation framework that isolates the impact of deployment rules on collective behavior, revealing that rule design and identity salience are critical determinants of safety outcomes across diverse LLM populations.
The paper introduces "Institutional Red-Teaming," a rigorous causal evaluation methodology designed to isolate the impact of deployment rules on multi-agent AI systems. By holding agent models, objectives, task states, and observability fixed, and varying only a single deployment rule (specifically consequence allocation), the authors establish a clean causal identification strategy. This approach is methodologically superior to existing multi-agent benchmarks that conflate agent capabilities with environmental or rule-based effects. The framework defines auditable coordinates for rules (concentration, identity salience, incidence), providing a structured theoretical lens for analyzing strategic incentives in multi-agent interactions.
The empirical contribution is substantial, involving 33,924 games across 228 contexts, five canonical rules, and seven distinct LLM populations. The results are robust and surprising: the study demonstrates that deployment rules causally alter collective safety by 22-58 percentage points, that no single rule is universally safest across different model populations, and that "regressive identity-targeting" (eliminating the poorest agent) is universally unsafe. Crucially, the ablation study showing that merely naming the loss bearer (identity salience) drives targeted elimination from 22% to 81% is a striking and significant finding. The statistical rigor, including bootstrap CIs and permutation tests, supports these claims.
The paper provides a detailed description of the experimental setup, including the volunteer's dilemma game structure, the specific rules tested, and the inference settings for the LLMs. The authors commit to releasing a code-and-data artifact that includes mechanism implementations, the cooperative-refinement simulator, and analysis scripts. The use of hosted API models with specific snapshots ensures that the behavioral results are reproducible for the specific versions tested, though generalization to future model versions requires re-certification as noted in the safety-case workflow.
The primary limitation is the simplicity of the game environment (three agents, no communication, threshold public goods). While this simplicity is necessary for causal isolation, it may not fully capture the complexity of real-world multi-agent deployments with communication channels, complex coalitions, or continuous action spaces. Additionally, the "elimination" mechanic is a proxy for operational consequences like throttling or shutdown, which may not perfectly map to all deployment scenarios. The study is also limited to seven model populations, and the authors acknowledge that newer models may behave differently, necessitating the proposed re-certification loop.
This work has profound implications for the safety and governance of multi-agent AI systems. It shifts the focus from merely aligning individual models to aligning the institutional rules that govern their interactions. The finding that rule text itself (identity salience) can trigger catastrophic strategic behaviors suggests that safety engineering must extend to the precise wording of deployment protocols. The proposed "safety-case" workflow offers a practical, auditable framework for certifying multi-agent deployments, potentially becoming a standard for responsible AI deployment in high-stakes environments. This paper presents a significant methodological advance in multi-agent AI safety by introducing Institutional Red-Teaming, a causal evaluation framework that isolates the impact of deployment rules on collective behavior, revealing that rule design and identity salience are critical determinants of safety outcomes across diverse LLM populations.
Nonlinear least-squares optimization is central to regression, physics-informed neural networks, and other machine-learning tasks. Such problems have a natural geometric interpretation, model predictions form a manifold in data space, while the chosen parameterization can introduce parameter-effects curvature that becomes a dominant source of nonlinearity. This exposes a limitation of the Levenberg-Marquardt (LM) method, its tangent-space step is applied as a straight update in parameter coordinates. Geodesic acceleration gives a second-order correction, but its removal of parameter-effect curvature is exact only in the infinitesimal-step limit. We propose a Riemann-normal-coordinate Levenberg-Marquardt method (RNC-LM) to improve this consistency for finite optimization steps. By reformulating the geodesic equation, RNC-LM extends geodesic acceleration to arbitrary-order corrections and constructs finite-step updates with progressively higher reparameterization consistency. A line search along the resulting RNC curve controls the traveled distance while keeping the cost close to standard LM. The method eliminates the tangential component of residual acceleration order by order in a moving tangent frame, making the actual objective reduction more consistent with the linear model prediction of LM. On classical nonlinear least-squares benchmarks, RNC-LM improves convergence and robustness in curved valleys and rank-deficient problems. On a reaction-diffusion PINN failure-mode benchmark, it reduces the relative L2 error to the order of 1e-3 and recovers a physically meaningful solution. On a large-scale machine-learning potential-energy-surface fitting task, it achieves a 34-fold speedup over standard LM.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Dalian Institute of Chemical Physics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Hefei National Laboratory
RNC-LM offers a significant contribution to the field of optimization, particularly for nonlinear least-squares problems prevalent in machine learning and scientific computing. By providing a geometrically consistent finite-step update mechanism, it addresses a fundamental limitation of widely used methods like LM. Its demonstrated ability to resolve PINN failure modes and achieve substantial speedups in large-scale scientific ML tasks (like potential energy surface fitting) has direct and immediate implications for accelerating scientific discovery and improving the reliability of physics-informed models. The conceptual framework of using Riemann normal coordinates for higher-order geodesic updates could inspire similar geometric extensions for other optimization algorithms, potentially leading to more robust and efficient training across various ML domains. This work highlights the importance of considering the finite-step realization of descent directions, not just the infinitesimal direction itself, which is a valuable insight for optimizer design. This paper introduces a Riemann-normal-coordinate Levenberg-Marquardt method (RNC-LM) that constructs higher-order, geometrically consistent finite-step updates, demonstrating significant improvements in convergence, robustness, and speed across classical benchmarks, PINN failure modes, and large-scale scientific machine learning tasks. The method's novel recursive formulation for RNC coefficients, its principled geometric interpretation, and its strong empirical performance, including a 34-fold speedup on a real-world scientific ML problem and the resolution of PINN overfitting, make it a highly impactful contribution to nonlinear least-squares optimization and scientific machine learning.
The paper proposes RNC-LM, a novel extension of the Levenberg-Marquardt (LM) method that uses Riemann Normal Coordinates (RNC) to construct higher-order, geometrically consistent finite-step updates. This addresses a fundamental limitation of standard LM, where the tangent-space step is applied as a straight update in parameter coordinates, leading to coordinate-dependent behavior and poor performance in highly nonlinear landscapes. The core methodological contribution is the reformulation of the geodesic equation into a "first-kind geodesic residual" condition. This allows for a recursive construction of higher-order RNC coefficients (c_2, ..., c_K) without explicitly forming Christoffel symbols or higher-order derivative tensors. Crucially, each step of this recursion involves solving a linear system with the same damped metric matrix (G) as the original LM subproblem. This means that the factorization computed for the standard LM step can be reused for all higher-order corrections, making the method computationally efficient. The right-hand sides of these linear systems are generated by automatic differentiation along a one-dimensional trial curve, further enhancing practicality. The method also introduces a sophisticated trust-region-ratio control strategy that distinguishes between local-model failure (handled by adjusting the damping parameter) and curve-approximation failure (handled by a line search along the RNC curve, adjusting the curve parameter 't'). This dual control mechanism is well-reasoned and empirically validated. The geometric interpretation of RNC-LM, showing that it eliminates the tangential component of residual acceleration order by order in a moving tangent frame, provides a deep understanding of why it improves consistency between the linear model prediction and actual objective reduction.
The experimental evaluation is comprehensive and compelling, covering both classical nonlinear least-squares benchmarks and modern scientific machine learning problems. 1. **Classical Benchmarks (Generalized Rosenbrock, MGH10)**: These tests effectively isolate and demonstrate the benefits of RNC-LM. The generalized Rosenbrock problem clarifies the distinct roles of RNC order (addressing insufficient curve geometry) and line search (preventing overshoot). The MGH10 problem, known for its rank-deficient plateau and high-order parameter effects, shows RNC-LM's superior performance over standard LM and LM-GA, with higher-order RNC reducing iterations from 13576 (LM) to 65 (5th-order RNC-LM). This highlights RNC-LM's ability to navigate complex landscapes more effectively by leveraging higher-order geometric information. 2. **Physics-Informed Neural Networks (PINNs)**: On a challenging reaction-diffusion PINN benchmark, RNC-LM achieves a significant breakthrough. While standard LM and LM-GA converge to solutions with low training loss but high L2 error (indicating collocation overfitting), RNC-LM reduces the relative L2 error to the order of 10^-3, recovering physically meaningful solutions. This demonstrates RNC-LM's robustness against known PINN failure modes and its ability to find more accurate physical solutions. 3. **Large-Scale Machine-Learning Potential-Energy-Surface Fitting**: This is a critical real-world application. RNC-LM achieves a remarkable 34-fold speedup over standard LM (reducing training time from 387.28 hours to 11.39 hours) to reach the same accuracy on a large H2O cluster dataset. This result is highly impactful, showcasing the practical scalability and efficiency of RNC-LM for complex scientific ML tasks near memory limits. Overall, the experiments are well-designed, cover diverse problem types and scales, and provide strong evidence for the method's effectiveness, robustness, and efficiency.
The paper provides a detailed algorithmic description, including the recursive formula for RNC coefficients (Eq. rnc_recursion), the first-kind geodesic residual (Eq. first_kind_residual), and the trust-region-ratio control mechanism. It explicitly mentions the use of automatic differentiation for computing right-hand sides, which is a standard and reproducible technique. While the full "Algorithm [REF]" and "Supplementary Information" are not included in the provided text, the level of detail in the main paper suggests that a diligent researcher could reproduce the results, assuming the supplementary material contains the full algorithm and hyperparameter settings. The specific values for line search acceptance criteria (`acc=10^-3`) and maximum trials are also mentioned.
The primary limitation acknowledged by the authors is the computational cost associated with explicitly constructing and factorizing the damped metric, which restricts its direct applicability to models with more than approximately 10^6 parameters. This is a common challenge for full-batch second-order optimization methods. However, the authors correctly point out that the recursive structure of RNC-LM, which involves repeated solutions of linear systems with the same metric, makes it compatible with approximate metric representations and iterative solvers (e.g., K-FAC, PCG) for larger-scale problems. Another implicit limitation is the increased complexity of implementation compared to standard first-order methods or even basic LM, requiring careful handling of higher-order derivatives via AD. The method is currently tailored to nonlinear least-squares problems, and its extension to other statistical manifolds (e.g., for natural gradient methods) is noted as future work.
RNC-LM offers a significant contribution to the field of optimization, particularly for nonlinear least-squares problems prevalent in machine learning and scientific computing. By providing a geometrically consistent finite-step update mechanism, it addresses a fundamental limitation of widely used methods like LM. Its demonstrated ability to resolve PINN failure modes and achieve substantial speedups in large-scale scientific ML tasks (like potential energy surface fitting) has direct and immediate implications for accelerating scientific discovery and improving the reliability of physics-informed models. The conceptual framework of using Riemann normal coordinates for higher-order geodesic updates could inspire similar geometric extensions for other optimization algorithms, potentially leading to more robust and efficient training across various ML domains. This work highlights the importance of considering the finite-step realization of descent directions, not just the infinitesimal direction itself, which is a valuable insight for optimizer design. This paper introduces a Riemann-normal-coordinate Levenberg-Marquardt method (RNC-LM) that constructs higher-order, geometrically consistent finite-step updates, demonstrating significant improvements in convergence, robustness, and speed across classical benchmarks, PINN failure modes, and large-scale scientific machine learning tasks. The method's novel recursive formulation for RNC coefficients, its principled geometric interpretation, and its strong empirical performance, including a 34-fold speedup on a real-world scientific ML problem and the resolution of PINN overfitting, make it a highly impactful contribution to nonlinear least-squares optimization and scientific machine learning.
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Primary: Stanford University
All Institutions: Stanford University, MIT, Nunchux AI, UC Berkeley, CMU
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
FourTune proposes an innovative and technically sound approach for fully 4-bit efficient post-training of diffusion models, addressing the critical challenges of memory footprint and training speed. The core of the methodology is the "triple-branch hybrid pipeline" which augments a standard LoRA architecture with a "frozen numerical stabilizer" (FNS). This FNS branch is a small, frozen, full-precision component designed to isolate and process quantization-sensitive outliers, thereby enabling stable training under native W4A4G4 (weights, activations, gradients) computation. This is a particularly clever design choice, as it mitigates the primary instability issue of low-bit quantization (outliers) without incurring significant computational overhead during training, as the FNS branch remains frozen. The integration of this FNS with both a standard BF16 LoRA branch and a quantized 4-bit LoRA branch provides a robust mechanism for maintaining quality while maximizing efficiency. Furthermore, the paper introduces hardware-efficient block-wise quantization and customized fused kernels. These components are crucial for translating the theoretical benefits of 4-bit quantization into practical speedups by optimizing memory bandwidth and accelerating quantized backpropagation operations (e.g., QGEMM, QMatmul, QConv). The end-to-end W4A4G4 paradigm, encompassing weights, activations, and gradients, is a comprehensive solution that pushes the boundaries of efficient fine-tuning for large generative models.
The experimental evaluation is comprehensive and robust, demonstrating FourTune's effectiveness across diverse and important post-training tasks for diffusion models: customization (DreamBooth-like), reinforcement learning for aesthetic preference, and knowledge distillation. The choice of FLUX.1-dev (12B parameters) as the primary model, along with validation on SDXL (1.5B), ensures the results are relevant for large-scale, state-of-the-art diffusion models. Baselines include BF16 LoRA and NF4 QLoRA, which are appropriate and strong competitors. The quantitative results are highly compelling: FourTune consistently matches or slightly outperforms the image quality of full-precision BF16 LoRA and NF4 QLoRA across all tasks, as measured by FID, CLIP Score, and LPIPS. This quality preservation at such low bit-depth is a significant achievement. More importantly, FourTune delivers substantial efficiency gains, reducing GPU memory overhead by 2.25x compared to BF16 LoRA (achieving a footprint comparable to NF4 QLoRA) and increasing end-to-end training throughput by 2.27x over BF16 LoRA and 2.79x over NF4 QLoRA. This effectively breaks the memory-speed trade-off often encountered in large model post-training. The ablation studies are well-designed and clearly demonstrate the critical contribution of each proposed component: the Frozen Numerical Stabilizer for stability and quality, and the W4A4G4 quantization, block-wise quantization, and fused kernels for efficiency. The generalization to SDXL further strengthens the claims.
The paper provides a good level of detail regarding the methodology, including the architecture of the triple-branch pipeline, the role of the FNS, and the types of quantization and fused kernels used. The experimental setup is well-described, including the specific models (FLUX.1-dev, SDXL), tasks, baselines, and evaluation metrics. The appendix further elaborates on implementation details, hyperparameters, and training configurations, which are crucial for reproducibility. While no direct code link is provided, the comprehensive description suggests that a skilled research team should be able to reproduce the results.
The paper does not explicitly discuss limitations, but some can be inferred. While the FNS effectively handles outliers, its design might introduce a slight architectural overhead compared to a purely 4-bit system, even if frozen. The customized fused kernels, while highly efficient, might require specific hardware support or careful implementation to achieve optimal performance across different GPU architectures. The evaluation is primarily focused on image generation diffusion models; its applicability and performance on other types of diffusion models (e.g., audio, video) or other generative architectures (e.g., GANs, VAEs) are not explored. The paper also doesn't delve into the potential challenges of deploying such a highly optimized, custom-kernel-dependent solution in diverse production environments.
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
Boosting is a fundamental technique for generically improving the accuracy of learning algorithms (Schapire 1989). Existing boosting algorithms construct a strong learner using $O(\log(\frac{1}ε)/γ^2)$ calls to a $γ$-advantage weak learner, and this round complexity is known to be optimal for generic boosters that succeed on all concept classes (Freund 1995). We show that this lower bound can be circumvented for concept classes that satisfy a mild closure property. Specifically, we present a new boosting algorithm that, for any class $\mathcal{F}$ closed under $O(\log \frac{1}γ)$-XOR, strong learns $\mathcal{F}$ using $O(\log \frac{1}ε)$ calls to a $γ$-advantage weak learner and a single batch of $\tilde{O}(\log(\frac{1}ε)/γ^2)$ additional samples. Our algorithm arises from a new and simple connection between boosting and list-decodable codes. Viewing the target function as a message, we run the weak learner on its encoding and view the resulting weak hypothesis as a corrupted codeword. Feeding this corrupted codeword to a list decoder, we obtain a small list of candidate hypotheses, at least one of which is a strong hypothesis for the original function. Using additional samples, we identify and output this strong hypothesis.
Primary: Stanford University
All Institutions: Stanford University, University of California, Berkeley, Simons Foundation
This paper presents a significant theoretical advance in boosting by circumventing the optimal round complexity lower bound for generic boosters, achieving $O(1)$ weak learner calls for XOR-closed concept classes through a novel application of list-decodable codes. The work is rigorous, highly novel in its theoretical approach, and addresses a fundamental question in learning theory, warranting a high score for its contribution to the field's understanding of computational efficiency and sample complexity trade-offs.
The paper proposes a theoretically significant breakthrough in boosting by circumventing the standard $O(1/\gamma^2)$ round complexity lower bound. The core methodology involves a novel connection between boosting and list-decodable codes. By encoding the target function using an XOR code and treating the weak learner's output as a corrupted codeword, the algorithm employs a local list decoder to generate a small candidate list of hypotheses. A subsequent filtering step using additional samples identifies the correct strong hypothesis. This approach is mathematically elegant and rigorously proven to achieve $O(1)$ calls to the weak learner for concept classes closed under XOR, trading round complexity for a batch of additional samples. The theoretical derivation is sound, leveraging results from coding theory (specifically local list decoding for the XOR code) to establish new bounds in computational learning theory.
The paper is purely theoretical and contains no empirical experiments, benchmarks, or datasets. The "results" are formal theorems proving the existence and efficiency of the boosting algorithm. Consequently, there is no experimental evaluation to assess in the traditional sense. The validity of the claims rests entirely on the mathematical proofs provided.
As a theoretical work, reproducibility refers to the reproducibility of the proofs. The paper provides detailed proofs for the main theorems, including the construction of the list decoder and the boosting algorithm. The dependencies on existing coding theory results are cited. However, the "algorithm" described is a theoretical construct; implementing it would require specific instantiations of the weak learner and the list decoder which are not provided as code. The theoretical framework is clear enough for other theorists to verify.
The primary limitation is the restriction to concept classes that are closed under $O(\log(1/\gamma))$-XOR. While the authors argue this is a "mild" closure property, it excludes many standard learning settings where such closure does not hold. Furthermore, the algorithm requires a "single batch" of additional samples of size $\tilde{O}(1/\gamma^2)$, which might be computationally expensive or data-intensive in practice, potentially offsetting the benefit of reduced round complexity. The paper does not address the computational complexity of the list decoding step or the filtering step in terms of time complexity, only sample complexity.
This work has significant implications for the theoretical foundations of machine learning. By breaking a long-standing lower bound, it opens new avenues for designing efficient boosting algorithms in specific structured settings. It highlights the deep connections between coding theory and learning theory, potentially inspiring new cross-disciplinary research. However, its immediate practical impact is limited due to the theoretical nature and specific constraints of the method. It serves as a foundational result for future work in efficient learning algorithms. This paper presents a significant theoretical advance in boosting by circumventing the optimal round complexity lower bound for generic boosters, achieving $O(1)$ weak learner calls for XOR-closed concept classes through a novel application of list-decodable codes. The work is rigorous, highly novel in its theoretical approach, and addresses a fundamental question in learning theory, warranting a high score for its contribution to the field's understanding of computational efficiency and sample complexity trade-offs.
Recent LLM-based mathematical reasoning agents have begun to tackle research-level problems and, in several cases, have contributed to the resolution of open problems. However, scaling and orchestrating such agents effectively remains challenging, due to the difficulty of coordinating parallel proof search while keeping intermediate claims organized and reliable. In this paper, we propose Danus, an orchestration system for research-level mathematical reasoning centered on a shared fact graph as a global memory-management mechanism. Danus consists of a main agent that performs planning and coordination, multiple worker agents that carry out proof search in parallel, and a stateless verifier that checks proposed mathematical claims before they are admitted into the fact graph. Each verified fact is stored together with its proof and logical dependencies, allowing the system to build long arguments incrementally while keeping the shared proof state organized. The main agent periodically summarizes the evolving proof state, redirects workers across promising directions, and supports interaction with human mathematicians through progress reports. We evaluate Danus through six research-level case studies in algebraic geometry, singularity theory, and combinatorics, illustrating how the fact-graph memory mechanism enables Danus to construct long, detailed mathematical proofs. Our results suggest that fact-graph-based orchestration provides an effective route toward scaling mathematical reasoning agents for long-horizon research problems. Danus is open source at https://github.com/frenzymath/Danus.
Primary: Great Bay Institute for Advanced Study
All Institutions: Great Bay Institute for Advanced Study, Beijing International Center for Mathematical Research, New Cornerstone Science Laboratory, Center for Intelligent Computing, Center for Machine Learning Research, Department of Mathematics, Great Bay University, Guoxiong Gao, Jihao Liu, Key Laboratory of Intelligent Computing and Applications (Ministry of Education), Kyoto University, Peking University, Research Institute for Mathematical Sciences, School of Mathematical Sciences, School of Mathematics, Stanford University, Tianjin University, Tongji University, Westlake Institute for Advanced Study, Westlake University, Zeming Sun, Zhongguancun Academy
Danus introduces a novel fact-graph-based orchestration system for multi-agent mathematical reasoning, demonstrating significant technical impact by enabling the automated resolution of complex, research-level mathematical problems through structured memory management and parallel verification.
The paper introduces Danus, a multi-agent orchestration system designed for research-level mathematical reasoning. The core innovation is the "fact graph," a shared, directed acyclic graph (DAG) that serves as a global memory mechanism. This graph stores verified mathematical facts (statements with proofs) and their logical dependencies. The system employs a strict separation of powers: a main agent (orchestrator) plans and coordinates, multiple worker agents (reasoners) explore different proof paths in parallel, and a stateless verifier ensures correctness before facts are added to the graph. This architecture addresses the critical challenge of scaling multi-agent systems by preventing context confusion and enabling incremental, verifiable proof construction. The use of a DAG allows for complex dependency tracking and revocation of invalid facts, which is crucial for maintaining logical integrity in long-horizon reasoning tasks.
The evaluation consists of six case studies in advanced mathematics, including algebraic geometry, singularity theory, and combinatorics. These are not standard benchmarks but open or semi-open research problems. Danus successfully resolved several problems, such as the optimal bend-and-break for foliations and the total Cartier indices of rational singularities. Notably, in the "Tangent classes of matroids" case, Danus solved a problem that previous systems (Rethlas, GPT-5.5-pro) failed to solve, demonstrating the efficacy of its orchestration and memory management. The results are qualitative and illustrative rather than statistical, which is appropriate for this type of exploratory research but limits broad generalizability. The system's ability to produce complete, verified proofs for complex theorems is the primary metric of success.
The paper provides an open-source repository (https://github.com/frenzymath/Danus), which significantly enhances reproducibility. The methodology is described in detail, including the roles of different agents, the structure of the fact graph, and the verification process. However, the reliance on specific proprietary models (GPT-5.5-pro, Claude Opus 4.8) and the specific mathematical literature retrieval tools (Matlas) may pose barriers to exact replication. The case studies involve human-in-the-loop interactions, which are difficult to fully replicate but are documented.
The system is heavily dependent on the capabilities of the underlying LLMs and the verifier. The verifier, while effective, is not perfect and may accept proofs with minor skipped steps or rely on potentially erroneous references if not caught in final review. The system's performance is limited by the cost and latency of running multiple agents and verifying facts. Additionally, the current evaluation is limited to a small number of high-difficulty mathematical problems, and it is unclear how the system scales to other domains or less structured reasoning tasks. The reliance on human input for problem formulation and final verification is also a limitation for full autonomy.
Danus represents a significant step towards autonomous scientific discovery, particularly in mathematics. By demonstrating that AI systems can contribute to the resolution of open research problems, it challenges traditional notions of human-AI collaboration in science. The fact-graph memory mechanism could be adapted for other complex reasoning tasks requiring long-term dependency tracking and verification, such as legal reasoning or software verification. However, the potential for misuse in generating plausible-sounding but incorrect mathematical proofs remains a concern, necessitating robust verification mechanisms. Danus introduces a novel fact-graph-based orchestration system for multi-agent mathematical reasoning, demonstrating significant technical impact by enabling the automated resolution of complex, research-level mathematical problems through structured memory management and parallel verification.
Logit-based watermarking is a widely used mechanism for identifying LLM generated content, yet its effectiveness is governed by a fundamental trade-off between detectability and semantic distortion. Existing analyses provide limited guidance for principled hyperparameter selection, leaving practical deployments reliant on heuristic tuning. In this work, we develop a power-calibrated statistical framework that establishes explicit quantitative relationships between watermark hyperparameters, detection power, and distortion. This characterization transforms watermark design into a guided optimization problem. Building on these results, we derive practical parameter selection procedures that achieve optimal tradeoffs under constraints. Extensive experiments across multiple language models and datasets validate the theory and demonstrate that the proposed framework consistently identifies Pareto-optimal points.
Primary: Department of Statistics, Pennsylvania State University, University Park
All Institutions: Department of Statistics, Pennsylvania State University, University Park
The paper's broader impact is significant and directly addresses critical societal concerns related to the proliferation of large language models. 1. **Combating Misinformation**: By improving the efficiency and reliability of LLM watermarking, this work strengthens the ability to identify machine-generated content. This is crucial in the fight against misinformation, fake news, and propaganda campaigns that leverage generative AI. 2. **Academic Integrity**: Watermarking can help detect AI-generated content in academic submissions, supporting academic honesty and preventing misuse of LLMs for plagiarism or automated assignment completion. 3. **Ethical AI Deployment**: It provides a mechanism for establishing provenance, which is essential for responsible and transparent deployment of generative AI. Users can be informed whether content originates from an AI, fostering trust and accountability. 4. **Scalability and Practicality**: By transforming watermark design from heuristic tuning to a statistically grounded optimization problem, the framework makes watermarking systems more robust, reliable, and easier to deploy in real-world applications. This improved efficiency and principled parameter selection can lead to wider adoption and more effective use of watermarking technologies. 5. **Research Foundation**: The rigorous statistical framework provides a strong theoretical foundation for future research in LLM watermarking, enabling more principled development and analysis of new techniques. It encourages a shift from empirical trial-and-error to theoretically informed design. The work directly contributes to mitigating potential harms of generative AI while enabling its beneficial uses, aligning with responsible AI development principles. This paper introduces a controllable statistical framework for logit-based LLM watermarking, enabling principled calibration of watermark strength under explicit detectability and distortion objectives. By establishing quantitative mappings between watermark parameters, detection power, and KL-based distortion, the authors transform watermark design from heuristic tuning into a statistically grounded optimization problem, validated through extensive experiments across multiple language models and datasets, consistently identifying Pareto-optimal configurations.
The paper introduces a power-calibrated statistical framework for logit-based LLM watermarking, moving beyond heuristic hyperparameter tuning. The core methodology involves establishing explicit quantitative relationships between watermark hyperparameters (bias $\delta$ and green-list fraction $\gamma$), detection power, and semantic distortion (measured by KL divergence). Key methodological steps include: 1. **Formalizing Hypothesis Testing**: The paper frames watermark detection as a hypothesis test, defining the null ($H_0$: unwatermarked text) and alternative ($H_1$: watermarked text) hypotheses based on the green-list token probability. 2. **Assumptions for Tractability**: To derive closed-form expressions, the framework relies on several assumptions: * **Random green-list assignments**: Green lists are i.i.d. and independent of the generated token sequence, simplifying the null hypothesis to an i.i.d. Bernoulli process for green-token indicators. * **Non-informative NTP prior**: The next-token probability (NTP) vector is modeled as an independent draw from a uniform Dirichlet distribution. The paper acknowledges this is a simplification but argues for its effectiveness due to the detector's dependence on green-list mass, which concentrates around $\gamma$. * **Information Decay**: Assumes a geometric decay in mutual information between past and future green-token indicators, enabling the application of a Central Limit Theorem for $\alpha$-mixing sequences under the alternative hypothesis. * **Non-degenerate Long-run Variance**: Ensures the asymptotic variance of the indicator sequence is strictly positive. 3. **Derivation of Key Metrics**: * **Green-token probability under watermarking ($\gamma'$)**: Lemma 2.2 and Theorem 2.3 provide a precise, token-level description of how watermarking biases generation toward the green list, leading to a closed-form expression for $\gamma'$ in terms of $\delta$ and $\gamma$. * **Detection Power**: Using the normal approximation for the aggregate statistic $S_n$ under both $H_0$ and $H_1$, a closed-form expression for statistical power $\Psi^*(\delta, \gamma)$ is derived. The paper notes that a constant $c$ (long-run variance inflation) affects the numerical value but not the parameter selection for maximization. * **Distortion (KL Divergence)**: Lemma 3.1 provides a plug-in formula for the expected token-wise KL divergence $D_{KL}(\delta, \gamma)$ between watermarked and unwatermarked distributions. It proves strict monotonicity in $\delta$ for fixed $\gamma$, allowing $\delta$ to be parameterized by a distortion budget. 4. **Optimization Framework**: The theoretical characterization transforms watermark design into a guided optimization problem. Instead of tuning two hyperparameters $(\delta, \gamma)$, the problem is reduced to a 1D numerical optimization (e.g., maximizing power subject to a KL distortion budget $K_0$) because $\delta$ can be implicitly determined by $K_0$. Practical guidance is provided for initializing the search for $\gamma$. The methodology is statistically rigorous, building on established theorems (CLT for mixing sequences) and providing detailed proofs in the appendix. The reduction of a multi-dimensional heuristic search to a principled 1D optimization problem under constraints is a significant practical contribution.
The experimental evaluation is comprehensive and well-structured, providing strong empirical validation for the theoretical framework. 1. **Setup**: The protocol largely follows prior watermarking evaluations, using diverse LLMs (OPT, Pythia, GPT-2, and Gemma-2 9B in the appendix) and datasets (C4, LFQA, Wikipedia). Generations are kept short ($n=50$ tokens) to avoid trivial detectability gains and better expose statistical efficiency differences. All methods are implemented using a unified pipeline based on the KGW codebase to ensure fair comparison. 2. **Distributional Verification**: * **Normality**: Q-Q plots confirm that the standardized green-token count statistic closely follows a standard normal distribution under both $H_0$ and $H_1$, supporting the normal approximation used in the detectability analysis. * **Alternative Hypothesis Characterization**: Empirical green-token rates are compared against theoretical predictions from Theorem 2.3. A near-linear relationship with $R^2 > 0.98$ across all settings validates the simplified modeling assumptions for $\gamma'$. 3. **Performance Evaluation (Detectability-Distortion Trade-off)**: * The paper evaluates statistical power (TPR at $\alpha=0.05$) against semantic distortion (per-token KL divergence). * Results across various models and datasets consistently show that the proposed method lies on the Pareto frontier. It achieves high detection power at substantially lower distortion compared to baseline methods (KGW, DP, and dense grid search). * The approach maintains near-saturated TPR in moderate distortion regimes where baselines exhibit significant degradation. This highlights improved statistical efficiency. 4. **Robustness under Alternative Quality Metrics**: * To assess the generalizability of the optimized parameters, the method is evaluated using complementary semantic quality metrics: BLEU, ROUGE, and BERTScore. * The results show a consistent pattern: the proposed method maintains strong detection power over a wider range of quality values compared to baselines. This suggests that the optimized parameterization induces a more uniform and controlled distributional shift, not tied to a specific notion of text similarity. Overall, the experiments are thorough, well-designed, and effectively support the theoretical claims. The validation of assumptions, extensive comparisons, and robustness checks significantly strengthen the paper's conclusions.
The paper demonstrates a strong commitment to reproducibility. 1. **Code Availability**: The authors explicitly state that "The implementation is available at https://github.com/shooof/wm." This is a crucial step for reproducibility. 2. **Detailed Experimental Setup**: The "Experimental Setup" section provides clear details on the language models (OPT, Pythia, GPT-2, Gemma-2 9B), datasets (C4, LFQA, Wikipedia), generation length ($n=50$), and detection parameters ($\alpha=0.05$). 3. **Baseline Descriptions**: Baseline methods (KGW, DP, dense grid search) are clearly identified, and their implementation is stated to use the same unified generation and detection pipeline as the proposed method, controlling for implementation effects. 4. **Hyperparameter Ranges**: Appendix provides a table of hyperparameter ranges used for all methods, which is essential for replicating the search space. 5. **Metric Definitions**: Clear definitions and implementations for distortion (KL divergence) and quality metrics (BLEU, ROUGE, BERTScore) are provided. 6. **Theoretical Proofs**: The appendix contains detailed proofs for all lemmas and theorems, allowing for independent verification of the mathematical derivations. The combination of open-source code, detailed experimental descriptions, and theoretical proofs makes the work highly reproducible.
The paper acknowledges limitations, with a dedicated discussion mentioned in Appendix [REF] (though the provided text truncates before this appendix content). Based on the main text, potential limitations include: 1. **Assumptions for Tractability**: The framework relies on several simplifying assumptions: * **Random green-list assignments**: While reasonable for pseudo-random hash functions, real-world implementations might have subtle biases. * **Non-informative NTP prior (Uniform Dirichlet)**: The paper argues for its effectiveness, but it is an approximation. While it mentions that richer priors (mixtures of Dirichlet) could be used if structural information is available, the current analysis is based on a simplified prior. The accuracy of this approximation for all possible LLM output distributions might vary. * **Information Decay**: The assumption of geometric decay in mutual information is a model for the autoregressive dependence. While standard for mixing sequences, its precise fit to complex LLM generation dynamics is an approximation. 2. **Estimation of Constant 'c'**: The power formula includes a constant $c$ that captures long-run variance inflation due to dependence. While the paper states that parameter selection is independent of $c$, its numerical value is needed for absolute power predictions and needs to be estimated in practice (as mentioned in Appendix [REF]). This introduces an additional practical step and potential source of error. 3. **Focus on Logit-Based Watermarking**: The framework is specifically developed for logit-based watermarking (KGW framework). While this is a widely adopted paradigm, the theoretical derivations might not directly apply to other watermarking schemes without significant adaptation. 4. **Short Sequence Evaluation**: While justified to isolate statistical efficiency, the primary experiments are on short sequences ($n=50$). While the paper discusses long-form generation in the appendix, the main empirical results are on short sequences. 5. **Adversarial Robustness**: While the appendix briefly discusses "signal-removal robustness" using a WinMax-C detector, a more extensive analysis of various adversarial attacks (e.g., paraphrasing, fine-tuning, token manipulation) and how the calibrated parameters perform under these attacks would be valuable. The current framework optimizes for detectability and distortion in a "clean" setting.
The paper's broader impact is significant and directly addresses critical societal concerns related to the proliferation of large language models. 1. **Combating Misinformation**: By improving the efficiency and reliability of LLM watermarking, this work strengthens the ability to identify machine-generated content. This is crucial in the fight against misinformation, fake news, and propaganda campaigns that leverage generative AI. 2. **Academic Integrity**: Watermarking can help detect AI-generated content in academic submissions, supporting academic honesty and preventing misuse of LLMs for plagiarism or automated assignment completion. 3. **Ethical AI Deployment**: It provides a mechanism for establishing provenance, which is essential for responsible and transparent deployment of generative AI. Users can be informed whether content originates from an AI, fostering trust and accountability. 4. **Scalability and Practicality**: By transforming watermark design from heuristic tuning to a statistically grounded optimization problem, the framework makes watermarking systems more robust, reliable, and easier to deploy in real-world applications. This improved efficiency and principled parameter selection can lead to wider adoption and more effective use of watermarking technologies. 5. **Research Foundation**: The rigorous statistical framework provides a strong theoretical foundation for future research in LLM watermarking, enabling more principled development and analysis of new techniques. It encourages a shift from empirical trial-and-error to theoretically informed design. The work directly contributes to mitigating potential harms of generative AI while enabling its beneficial uses, aligning with responsible AI development principles. This paper introduces a controllable statistical framework for logit-based LLM watermarking, enabling principled calibration of watermark strength under explicit detectability and distortion objectives. By establishing quantitative mappings between watermark parameters, detection power, and KL-based distortion, the authors transform watermark design from heuristic tuning into a statistically grounded optimization problem, validated through extensive experiments across multiple language models and datasets, consistently identifying Pareto-optimal configurations.
Hierarchical mixture models are a powerful tool for modeling data generated from heterogeneous sources, particularly when the mixing proportion $\boldsymbol{w}$ itself is treated as a random variable with a Dirichlet or Beta-Liouville prior. Such models are widely employed in scenarios where uncertainty in class membership or data-generating processes must be probabilistically quantified. This paper studies the exact marginalization of the mixture weight. For the two-component case we give an $O(n^2)$ dynamic program -- and an $O(n \log^2 n)$ FFT variant -- for the marginal likelihood, and show that the exact posterior of the weight is a finite mixture of Beta distributions, delivering closed-form posterior summaries, credible intervals and per-observation local false-discovery rates without any sampling. For $K \ge 3$ components we give an exact joint dynamic program. The gain is largest in the small-sample regime the method is built for: on a real multilevel meta-analysis, a pathway-level dysregulation analysis of leukemia gene expression, and a leukemia-derived gene-panel benchmark with known ground truth, the exact interval for the signal proportion is calibrated where EM gives no interval at all (collapsing to a boundary) and Gaussian/Laplace approximations mis-cover, and it is two orders of magnitude faster than the sampler that would match it. On the large prostate-cancer benchmark, where every method has ample data, it agrees with locfdr on the gene ranking while adding a posterior interval for the null proportion.
Primary: unknown
All Institutions: unknown
This paper has significant broader impact for any field employing hierarchical mixture models, especially where accurate uncertainty quantification in small-sample or rare-signal regimes is critical. 1. **Multiple Testing and FDR Estimation**: It provides a robust, exact method for estimating global null proportions and local false discovery rates, offering calibrated credible intervals that are superior to point estimates or miscalibrated approximations. 2. **Meta-analysis and A/B Testing**: In settings with multiple small studies or experiments, the method can provide reliable estimates of overall effects and their uncertainty, overcoming limitations of EM and MCMC. 3. **Bayesian Inference**: It demonstrates that exact Bayesian inference for certain complex models is achievable deterministically, without resorting to sampling methods, opening avenues for more reliable and efficient analysis. 4. **Theoretical Foundations**: The derived structural properties deepen the theoretical understanding of mixture model posteriors, linking them to classical combinatorial and polynomial theory. This paper presents a highly significant contribution to the field of Bayesian inference and statistical modeling. It provides exact, deterministic, and numerically stable algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, addressing long-standing limitations of approximate methods (EM) and computationally intensive sampling methods (MCMC). The rigorous mathematical derivations, coupled with comprehensive experimental validation demonstrating superior calibration and speed, make this a valuable tool for practitioners and a strong theoretical advance. The paper's main contribution is the development of exact, deterministic algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, yielding calibrated uncertainty quantification and closed-form summaries without sampling. This work provides a rigorous mathematical framework and efficient computational methods (dynamic programming) to overcome the limitations of existing approximate and sampling-based approaches, particularly in small-sample and rare-signal scenarios, offering significant practical benefits for applications like multiple testing and meta-analysis.
This paper presents a highly rigorous and elegant methodology for the exact computation of the posterior distribution of mixture weights in hierarchical Bayesian models, specifically for Dirichlet or Beta-Liouville priors. The core contribution lies in transforming the marginalization problem into a combinatorial sum that can be efficiently computed. For the two-component case, the authors derive an $O(n^2)$ dynamic programming (DP) algorithm, which is a significant improvement over the naive $O(2^n)$ enumeration. They also present an $O(n \log^2 n)$ FFT-based variant, though they acknowledge its numerical instability in practice. A key theoretical result is showing that the exact posterior of the mixture weight is a finite mixture of Beta distributions, enabling closed-form posterior summaries, credible intervals, and local false-discovery rates without any sampling. The paper further establishes three crucial structural properties of this exact posterior: 1. **Log-concavity and unimodality**: The posterior over the latent count of non-null observations is log-concave and unimodal, ensuring coherent credible intervals. This property stems from Newton's inequalities for elementary symmetric polynomials. 2. **Evidence-ratio moment identity**: Posterior raw moments of the mixing weight can be obtained as ratios of the marginal likelihood at shifted hyperparameters, allowing for efficient computation of moments without summation over the mixture. 3. **Monotone response to data**: The posterior is stochastically increasing in every likelihood ratio, ensuring that strengthening evidence for a signal coherently increases the inferred prevalence and maintains the ranking of local FDRs. For $K \ge 3$ components, the paper generalizes the approach by providing an exact joint dynamic program with $O(K n^K)$ time and $O(n^{K-1})$ storage complexity. This multivariate DP handles the non-factorization of coefficients across components, a challenge for $K \ge 3$. The use of log-domain computation throughout ensures numerical stability, a critical aspect for practical application. The mathematical derivations are thorough, leveraging combinatorial identities and properties of Stirling numbers.
The experimental evaluation is comprehensive and well-structured. 1. **Correctness**: The exact marginal likelihood algorithms are validated against multiple independent ground truth methods: naive $2^n$ enumeration, adaptive and high-precision quadrature, and Monte Carlo integration. The agreement is shown to be at floating-point round-off level, confirming the correctness of the proposed methods. 2. **Scaling and Numerical Stability**: The $O(n^2)$ log-domain DP is shown to be highly efficient and numerically stable, outperforming the intractable $2^n$ enumeration by orders of magnitude and remaining accurate for large $n$ (up to $n=6033$ in tests). The FFT variant, while theoretically faster, is demonstrated to be numerically unreliable due to combinatorial growth of coefficients, highlighting the practical importance of the log-domain DP. 3. **Comparison with EM and MCMC**: The exact method is benchmarked against a data-augmentation Gibbs sampler, PyMC's NUTS, and MAP-EM. It achieves posterior means and standard deviations that match the Gibbs sampler to three figures, but is two orders of magnitude faster. Crucially, it provides calibrated uncertainty where EM gives only point estimates (often collapsing to boundaries) and Gaussian/Laplace approximations mis-cover, particularly in the small-sample, rare-signal regime. 4. **Frequentist Coverage**: A frequentist study demonstrates that the exact Bayesian credible intervals maintain nominal coverage even when EM and approximate Bayesian methods fail (e.g., in small-sample, rare-signal scenarios). This is a strong validation of the method's calibration. 5. **Real-world Applications**: The method is applied to a multilevel meta-analysis, a pathway-level dysregulation analysis of leukemia gene expression, and a prostate-cancer microarray dataset. These applications showcase its practical utility, providing calibrated intervals and posterior summaries that are unavailable or unreliable from existing methods, while also agreeing with established methods like locfdr on gene rankings in large-sample settings.
The paper explicitly states that "A reference Python implementation reproducing every figure and table of this section is provided (sec:reproducibility)". While this indicates a commitment to reproducibility, an explicit URL to the code repository is not provided in the paper text. Assuming the code is indeed available as stated, the reproducibility would be high. Without the URL, it's difficult to verify immediately.
1. **Scalability for Large K**: The $O(K n^K)$ time complexity for $K \ge 3$ components limits its applicability to scenarios with a small number of mixture components. While the authors argue that $K=2$ or $K=3$ are common in practice, this is a significant constraint for models requiring many components. 2. **Fixed Component Parameters**: The method focuses solely on the exact marginalization of mixture weights, assuming the component distributions ($f$ and $g$) are known or their parameters are fixed. It does not address the joint estimation of component parameters along with the mixture weights, which is a common challenge in mixture modeling. 3. **Numerical Stability of FFT Variant**: The theoretically faster $O(n \log^2 n)$ FFT variant for $K=2$ is shown to be numerically unstable in practice, making the $O(n^2)$ DP the recommended backend. This means the best practical complexity is quadratic, not quasi-linear.
This paper has significant broader impact for any field employing hierarchical mixture models, especially where accurate uncertainty quantification in small-sample or rare-signal regimes is critical. 1. **Multiple Testing and FDR Estimation**: It provides a robust, exact method for estimating global null proportions and local false discovery rates, offering calibrated credible intervals that are superior to point estimates or miscalibrated approximations. 2. **Meta-analysis and A/B Testing**: In settings with multiple small studies or experiments, the method can provide reliable estimates of overall effects and their uncertainty, overcoming limitations of EM and MCMC. 3. **Bayesian Inference**: It demonstrates that exact Bayesian inference for certain complex models is achievable deterministically, without resorting to sampling methods, opening avenues for more reliable and efficient analysis. 4. **Theoretical Foundations**: The derived structural properties deepen the theoretical understanding of mixture model posteriors, linking them to classical combinatorial and polynomial theory. This paper presents a highly significant contribution to the field of Bayesian inference and statistical modeling. It provides exact, deterministic, and numerically stable algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, addressing long-standing limitations of approximate methods (EM) and computationally intensive sampling methods (MCMC). The rigorous mathematical derivations, coupled with comprehensive experimental validation demonstrating superior calibration and speed, make this a valuable tool for practitioners and a strong theoretical advance. The paper's main contribution is the development of exact, deterministic algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, yielding calibrated uncertainty quantification and closed-form summaries without sampling. This work provides a rigorous mathematical framework and efficient computational methods (dynamic programming) to overcome the limitations of existing approximate and sampling-based approaches, particularly in small-sample and rare-signal scenarios, offering significant practical benefits for applications like multiple testing and meta-analysis.
Graph-based semi-supervised learning (SSL) propagates a few labels over a similarity graph by minimizing a Dirichlet-type energy. The standard quadratic ($p=2$) energy reduces to a single graph-Laplacian solve, but it degenerates exactly where SSL is most useful when labels are scarce: gathering more unlabeled data drives the $p=2$ estimate to a near-constant function whenever $d\ge2$ (Nadler-Srebro-Zhou). Well-posedness requires the nonlinear $p$-Laplacian energy with $p>d$. Existing solvers reduce this to a sequence of weighted Laplacian solves, but their reference implementations use a direct sparse factorization or ichol-preconditioned CG instead. Plugging a near-linear Laplacian solver is not straightforward: at large $p$ the conductance weights degenerate near flat-gradient edges, making the system nearly singular and causing stagnation without a damped outer iteration. We close this gap. Recasting $p$-Laplacian SSL as a source-form nonlinear Laplacian flow $Bρ_p(B^\top x)=b$ and solving by damped chord-Newton continuation in $p$, every linearized system stays well-conditioned and can be delegated to a near-linear Laplacian engine. On size-scaled graph families the wall-clock is empirically $m^{0.96}$-$m^{1.02}$ per family (approximate Cholesky default), and a pooled fit across 228 SuiteSparse graphs gives $m^{1.19}$ vs.\ $m^{1.45}$ for direct factorization; the solver handles a $6.8\times10^7$-edge social network in minutes. Memory is the binding constraint: Cholesky fill reaches $10$-$280\times$ the graph nonzeros vs.\ our $O(m)$ hierarchy. Against the released FCL solver we are $1.5$-$14\times$ faster at matched accuracy. On MNIST $10$-NN, $p=3$ scores $64\%$ at one label per class vs.\ $36\%$ for $p=2$. Code: https://github.com/orenlivne/np.
Primary: Weizmann Institute of Science
All Institutions: Weizmann Institute of Science
The paper presents a significant engineering and numerical analysis breakthrough that makes scalable, nonlinear graph semi-supervised learning practically viable for the first time. By correctly integrating near-linear Laplacian solvers with a damped Newton continuation framework, it overcomes the stability and memory issues that previously confined $p$-Laplacian SSL to small graphs, enabling applications on industrial-scale networks with tens of millions of edges while maintaining the statistical advantages of nonlinear energy minimization.
The paper addresses a critical scalability bottleneck in Graph $p$-Laplacian Semi-Supervised Learning (SSL). While the statistical benefits of $p>2$ energies in mitigating the low-label degeneracy of quadratic ($p=2$) label propagation are well-established, practical adoption has been hindered by the superlinear memory and time complexity of direct sparse factorization solvers. The authors' key methodological contribution is the rigorous recasting of the $p$-Laplacian SSL problem as a nonlinear Laplacian flow ($B\rho_p(B^\top x)=b$) and the application of a damped chord-Newton continuation method. Crucially, they demonstrate that by using a conductance floor and guarded Anderson acceleration, the inner linearized systems remain well-conditioned, allowing the substitution of expensive direct solvers with near-linear time Laplacian engines (Approximate Cholesky or LAMG+). This is a non-trivial numerical analysis contribution, as naive substitution of near-linear solvers into the Newton loop typically leads to stagnation due to ill-conditioning at large $p$.
The experimental evaluation is comprehensive and convincing. The authors provide: 1. Theoretical validation: Reproducing the known low-label degeneracy of $p=2$ and showing that $p=3$ significantly improves accuracy (64% vs 36% on MNIST with 1 label/class). 2. Scaling analysis: Empirical evidence of near-linear scaling ($m^{0.96}-m^{1.02}$) on fixed graph families and a pooled fit of $m^{1.19}$ across 228 heterogeneous graphs. 3. Comparative benchmarks: Head-to-head comparisons against the incumbent FCL solver (showing 1.5-14x speedups) and Calder's GraphLearning package (showing significant speedups on geometric graphs). 4. Web-scale demonstration: Successfully solving SSL on a 68M-edge social network (LiveJournal) in minutes, a task infeasible for direct factorization methods due to memory constraints. The experiments are rigorous, covering controlled scaling, heterogeneous corpus analysis, and real-world industrial-scale graphs.
The paper provides a clear algorithm description, detailed hyperparameters (e.g., conductance floor $10^{-6}$, continuation schedule), and open-source code. The reproducibility is high, supported by the availability of the Julia implementation and the specific graph corpus used.
The authors honestly disclose several limitations: 1. The near-linear scaling is empirical; no theoretical complexity bound is provided for the outer iteration count on general graphs. 2. The solver is currently single-threaded, limiting absolute wall-clock performance compared to potential distributed implementations. 3. The comparison with FCL is limited to moderate sizes due to the MATLAB/Octave implementation's constraints, though the memory wall argument for larger graphs is strong. 4. The method relies on the effectiveness of the conductance floor, which, while theoretically justified as a preconditioner, is an empirical choice.
This work removes a major barrier to using nonlinear graph-based SSL at scale. By making $p$-Laplacian methods feasible for web-scale graphs, it enables more robust semi-supervised learning in regimes with scarce labels (few-shot learning, active learning) where GNNs often overfit and quadratic propagation fails. It bridges the gap between theoretical insights on $p$-Laplacian well-posedness and practical, large-scale machine learning infrastructure. The paper presents a significant engineering and numerical analysis breakthrough that makes scalable, nonlinear graph semi-supervised learning practically viable for the first time. By correctly integrating near-linear Laplacian solvers with a damped Newton continuation framework, it overcomes the stability and memory issues that previously confined $p$-Laplacian SSL to small graphs, enabling applications on industrial-scale networks with tens of millions of edges while maintaining the statistical advantages of nonlinear energy minimization.
Routing among large language models (LLMs) promises better quality at lower cost, motivated by the reported gap between learned routers and a per-instance oracle. But that oracle is computed from a single correctness label per (query, model), so under stochastic decoding it is one Bernoulli draw, not a reproducible property. We recast the question structurally: the expected per-instance oracle decomposes as $O^{\exp}=O^{\mathrm{repro}}+Δ$, into reproducible single-commit headroom $O^{\mathrm{repro}}$ and a non-negative single-commit selection floor $Δ$. Our main result is a recoverability asymmetry: this floor is closed by no single-commit router, yet is recovered by test-time sampling -- best-of-$K$ on the committed model, at the oracle's own budget, dominates the independent-pool single-draw oracle. The cap needs no cross-model independence; we prove it with the exact decomposition and noise-share bounds that shrink as the budget grows. The procedure adds no new router, only resampling. The floor's magnitude is a prospective, conservative localization, not an audit: our primary target LLMRouterBench (33 models, 391,645 instances) defines its oracle as a per-query union over single $T=0.2$ generations -- by construction a union of stochastic single draws. Since $O^{\mathrm{repro}}$ is non-identifiable from the released $k=1$ matrix, we estimate the noise share by fresh $k\ge20$ resampling under one-sided, dependence- and guessing-floor-corrected bounds, recasting 'model-recall failure' as thin-support union inflation. On a controlled open-model re-generation, single-draw noise is a substantial minority of the gap -- larger on an unsaturated benchmark, approaching half on the hardest queries where no model is reliable -- while the majority remains recoverable specialist advantage. We release a multi-sample oracle evaluation protocol for routing benchmarks.
Primary: National Yang Ming Chiao Tung University
All Institutions: National Yang Ming Chiao Tung University, Krixvon AI
This paper has significant broader impact for the field of LLM routing and evaluation methodology. 1. **Benchmark Design**: It provides a concrete, actionable protocol for benchmark designers to adopt multi-sample oracles (expected and reproducible variants) instead of single-draw ones, which are shown to be systematically inflated. This could lead to more accurate and reliable routing benchmarks. 2. **Interpretation of Routing Progress**: The work fundamentally re-calibrates the understanding of the "router-to-oracle gap" and the "model-recall failure" diagnosis. By quantifying the portion of the gap attributable to irreducible single-draw noise, it clarifies how much genuine headroom exists for routers, guiding research efforts more effectively. 3. **Research Direction**: It suggests that future routing research should focus on better ex-ante quality estimation and decorrelating model pools, rather than chasing an inflated ceiling. It also motivates further investigation into cost-quality claims and end-to-end latency with calibrated oracles. 4. **General LLM Evaluation**: The principles of accounting for stochasticity and decomposing performance into reproducible vs. noise components could extend beyond routing to other areas of LLM evaluation where single-sample metrics are common. This paper rigorously decomposes the LLM router-to-oracle gap, revealing that a substantial minority is single-draw label noise irrecoverable by single-commit routing, and proposes a multi-sample oracle evaluation protocol. The work provides a robust theoretical framework, compelling empirical evidence, and a highly reproducible methodology that significantly advances the understanding and evaluation of LLM routing systems, offering clear guidance for benchmark design and future research.
The methodology is exceptionally rigorous and well-articulated. The paper structurally recasts the problem of the router-to-oracle gap by defining three key oracles: the expected single-draw oracle ($O^{\exp}$), the reproducible single-commit headroom ($O^{\mathrm{repro}}$), and the verifier-free aggregation oracle ($O^{\mathrm{agg}}$). The core contribution is the exact, non-negative decomposition of the router-to-oracle gap ($G$) into recoverable specialist advantage ($G_{\mathrm{rec}}$) and single-draw label noise ($G_{\mathrm{noise}}$). This decomposition is backed by strong theoretical proofs (Theorems, Propositions, Corollaries) that establish the upward bias of the single-draw oracle and the "recoverability asymmetry"—that $G_{\mathrm{noise}}$ is irrecoverable by any single-commit router but can be recovered by test-time sampling. The proposed Algorithm 1 provides a clear, step-by-step protocol for multi-sample correctness estimation and gap decomposition, using raw frequencies for point estimates and Beta-Bernoulli posteriors for confidence intervals. Crucially, the methodology addresses the complexities of estimating $O^{\exp}$ in the presence of cross-model dependencies by using a seed-aligned estimator, which is unbiased. The use of one-sided, dependence- and guessing-floor-corrected bounds for conservative estimation of $G_{\mathrm{noise}}$ further enhances the robustness of the approach. The paper clearly distinguishes its contributions from prior and concurrent work, particularly regarding the focus on stochastic single-draw noise versus deterministic evaluation artifacts.
The experimental evaluation is comprehensive and well-controlled, designed to localize the empirical magnitude of the theoretically proven noise term. The primary target is LLMRouterBench, with RouterBench as secondary corroboration. For a controlled re-generation, the authors used a pool of eleven open-weight, text-only instruction models served identically under vLLM at $T=0.2$ with $k=30$ seed-aligned draws per (query, model) cell. Two exact-match benchmarks, GSM8K (saturated) and MATH-500 (unsaturated), were used, with thin-support queries oversampled. The experiments successfully pass pre-checks for independence and over-dispersion, licensing the magnitude study. Key findings include: 1. **Magnitude of Noise**: Single-draw noise ($G_{\mathrm{noise}}$) constitutes a substantial minority of the gap (12% on GSM8K, 36% on MATH-500), with the majority remaining recoverable specialist advantage (64-88%). 2. **Noise Concentration**: $G_{\mathrm{noise}}$ concentrates heavily in thin-support queries (e.g., 43% on MATH-500 for queries where only 3 of 11 models were correct), validating theoretical predictions. 3. **Pool Composition Control**: The paper rigorously controls for intra-lineage error correlation, showing that redundancy inflates the noise share. Experiments with lineage-deduplicated pools and cardinality sweeps confirm the theoretical predictions, demonstrating the robustness of the findings. 4. **Recoverability Check**: The falsifiable prediction that test-time sampling recovers what selection cannot is empirically confirmed, with best-of-$K$ sampling outperforming the independent-pool oracle. However, the analysis also highlights that verifier-free aggregation (majority vote) often falls short, indicating that a significant portion of the "guessing residual" requires a deploy-time verifier. The experimental design is exemplary in its controls, stratification, and careful interpretation of results, providing strong empirical support for the theoretical claims.
The reproducibility of this work is exceptionally high. The authors explicitly state that "Code, corrected oracles, and the per-model correctness data are available at https://github.com/luka-krixvon/routing-oracle-experiment". The paper provides detailed information about the experimental setup, including the specific models used (eleven open-weight instruction models from eight distinct pretraining lineages), the serving framework (vLLM), decoding parameters ($T=0.2$, top-$p$ $1.0$), and the number of seed-aligned draws ($k=30$). The system configuration, including hardware/software stack, is captured by a detection script and released with the code. The methodology for multi-sample correctness estimation and gap decomposition is clearly outlined in Algorithm 1. This level of detail and the release of artifacts make the work highly reproducible and verifiable by the community.
The paper acknowledges several limitations: 1. **Scope of Re-estimation**: The current estimates use $k$ samples at a single temperature on an open-model pool. Future work could extend this to larger $k$, multiple temperatures, and live frontier (closed-source) models to sharpen estimates and test the bias growth. 2. **Cross-model Error Correlation**: While the paper controls for intra-lineage correlation, it notes that it does not fully characterize how cross-model estimator-error correlation shifts routing optimality in general, leaving this for future work. 3. **Evaluation Metric Scope**: The primary analysis focuses on exact-match and multiple-choice tasks, explicitly excluding LLM-judge / continuous-graded ones due to the mixing of sampling noise with judge noise. This is a reasonable scoping decision but means the findings do not directly generalize to all types of LLM evaluations. 4. **Preprint Date**: The "Preprint, July 2026" date is unusual for an arXiv preprint, which typically reflects the current or a past year. While not impacting the technical content, it's an oddity.
This paper has significant broader impact for the field of LLM routing and evaluation methodology. 1. **Benchmark Design**: It provides a concrete, actionable protocol for benchmark designers to adopt multi-sample oracles (expected and reproducible variants) instead of single-draw ones, which are shown to be systematically inflated. This could lead to more accurate and reliable routing benchmarks. 2. **Interpretation of Routing Progress**: The work fundamentally re-calibrates the understanding of the "router-to-oracle gap" and the "model-recall failure" diagnosis. By quantifying the portion of the gap attributable to irreducible single-draw noise, it clarifies how much genuine headroom exists for routers, guiding research efforts more effectively. 3. **Research Direction**: It suggests that future routing research should focus on better ex-ante quality estimation and decorrelating model pools, rather than chasing an inflated ceiling. It also motivates further investigation into cost-quality claims and end-to-end latency with calibrated oracles. 4. **General LLM Evaluation**: The principles of accounting for stochasticity and decomposing performance into reproducible vs. noise components could extend beyond routing to other areas of LLM evaluation where single-sample metrics are common. This paper rigorously decomposes the LLM router-to-oracle gap, revealing that a substantial minority is single-draw label noise irrecoverable by single-commit routing, and proposes a multi-sample oracle evaluation protocol. The work provides a robust theoretical framework, compelling empirical evidence, and a highly reproducible methodology that significantly advances the understanding and evaluation of LLM routing systems, offering clear guidance for benchmark design and future research.
Brownian Bridge Diffusion Models (BBDM) offer an appealing framework for image restoration and inverse problems by constructing a stochastic bridge from the clean signal directly to the degraded observation, rather than to pure noise. Despite their promise, the choice of bridge schedule is typically inherited from heuristics, and a principled analytical framework for schedule design has been lacking. In this work, we develop such a framework by offering a novel analysis of BBDM reverse dynamics under a Mixture-of-Gaussians (MoG) prior. This setting yields a closed-form ideal posterior and a corresponding MMSE denoiser, while the BBDM-induced reconstruction law is captured analytically through a tractable surrogate. Building on these expressions, we formulate two complementary schedule-design objectives: a Wasserstein criterion targeting perceptual quality and an MSE criterion targeting reconstruction fidelity. Our work exposes an inherent tradeoff between the two and proves the existence of universal schedules for both that are independent of the degradation and prior. Extensive experiments on controlled MoG settings confirm full alignment between theory and practice, and experiments on the FFHQ dataset across inpainting, deblurring, and super-resolution tasks validate the practical value of our schedule-design criteria.
Primary: Technion – Israel Institute of Technology
All Institutions: Technion – Israel Institute of Technology
This paper provides a rigorous analytical framework for schedule design in Brownian Bridge Diffusion Models, deriving closed-form reconstruction laws under Mixture-of-Gaussians priors to expose and optimize the distortion-perception tradeoff. The technical contribution is significant for its mathematical depth and the clarity it brings to a previously heuristic area, though its direct impact is somewhat moderated by the reliance on approximations for high-dimensional applications.
The paper presents a rigorous analytical framework for schedule design in Brownian Bridge Diffusion Models (BBDM). The core methodological contribution is the derivation of exact posterior dynamics under a Mixture-of-Gaussians (MoG) prior. Recognizing that the exact MoG reverse process loses global affinity, the authors introduce a "selected-label" approximation that freezes the mixture component assignment, allowing for a closed-form reconstruction law. This enables the formulation of two explicit schedule-design objectives: one minimizing Wasserstein distance (perceptual quality) and one minimizing MSE (reconstruction fidelity). The theoretical derivation is mathematically sound, leveraging Gaussian conditioning identities and spectral decomposition to decouple the dynamics. The approach is novel in applying this specific analytical lens to BBDM schedules, moving beyond heuristic choices.
The experimental validation is structured in three tiers: synthetic MoG data, MNIST with fitted MoG priors, and real-world FFHQ images. The synthetic experiments effectively validate the theoretical claims regarding the selected-label approximation and the distortion-perception tradeoff. The MNIST experiments demonstrate that the theoretical schedules improve upon default schedules in PSNR and NLL. The FFHQ experiments show that the MSE-oriented schedule improves PSNR/SSIM while the W2-oriented schedule improves FID/LPIPS, confirming the theoretical tradeoff. However, the real-world experiments rely on "MoG-free heuristics" derived from the theoretical bounds rather than optimizing the full MoG objective (which is intractable at scale), which slightly weakens the direct link between the complex theory and the final applied results.
The paper provides extensive mathematical derivations in the appendix, including proofs of mean-exactness and covariance deficit. The experimental setup is detailed, including dataset splits, training epochs, and evaluation metrics. The use of standard datasets (FFHQ, MNIST) and publicly available libraries (torch-fidelity, lpips) aids reproducibility. The code for the BBDM models is not explicitly linked, but the methodology is sufficiently described for implementation.
The primary limitation is the reliance on the selected-label approximation for the theoretical analysis. While proven to be mean-exact and covariance-deficient, it is an approximation. The "universal" schedules derived are based on a bounded four-parameter family and may not be optimal for all degradation types or data distributions. Furthermore, the real-world experiments use heuristics rather than the full theoretical optimization, limiting the direct demonstration of the theory's power in high-dimensional settings. The assumption of linear inverse problems also restricts the scope.
This work provides a principled foundation for tuning diffusion models for inverse problems, potentially leading to more reliable and performant restoration algorithms. By exposing the inherent tradeoff between perceptual quality and fidelity through a clear analytical lens, it offers valuable insights for practitioners balancing these competing objectives. The framework could be extended to other bridge-based diffusion models or used to analyze other sampler parameters. This paper provides a rigorous analytical framework for schedule design in Brownian Bridge Diffusion Models, deriving closed-form reconstruction laws under Mixture-of-Gaussians priors to expose and optimize the distortion-perception tradeoff. The technical contribution is significant for its mathematical depth and the clarity it brings to a previously heuristic area, though its direct impact is somewhat moderated by the reliance on approximations for high-dimensional applications.
Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hurt one another. Most efficient fine-tuning pipelines ignore this and train on a fixed, uniform mixture, leaving two coupled questions unanswered: how much should each domain participate, and which domains should be co-trained given that some transfer positively and others interfere? We show that both answers can be read off cheaply and without labels. A forward pass of the current shared adapter over a small unlabeled probe yields, per domain, a competence signal whose level tracks remaining headroom and whose trajectory tracks learning speed; the drift of these probe representations yields a signed cross-domain affinity that predicts pairwise transfer. We fold both into CoDA, a co-adaptive controller that solves a small entropy-regularized quadratic program on the simplex to set each domain's participation -- jointly its loss weight and its share of the sampled data -- rewarding high-headroom, still-learning, mutually synergistic domains and damping interfering ones. The controller is forward-only, adds no trainable parameters, and wraps any multi-task LoRA pipeline. Across five heterogeneous domains and two backbones, CoDA improves the average over uniform mixing, learned mixtures, gradient-surgery multi-task optimizers, and online data selection while using half the data, and lowers cross-domain gradient conflict. We prove that the competence signal tracks domain risk, that the participation program has a unique fixed point reached by a contraction, and that its solution performs transfer-aware water-filling; analysis, ablations, and controls corroborate each claim.
Primary: University of Electronic Science and Technology of China
All Institutions: University of Electronic Science and Technology of China, Sichuan University
CoDA introduces a novel, label-free, forward-only mechanism for dynamically balancing multi-task LoRA training by estimating domain competence and cross-domain affinity, achieving state-of-the-art performance with reduced data and compute costs.
The paper proposes CoDA, a co-adaptive controller for multi-task Low-Rank Adaptation (LoRA). The core innovation lies in using label-free, forward-only signals to dynamically adjust domain participation. Specifically, it defines "competence" based on normalized predictive entropy to estimate domain headroom and learning speed, and "affinity" based on the drift of probe representations to estimate signed cross-domain transfer. These signals feed into an entropy-regularized quadratic program that jointly optimizes loss weights and data sampling ratios. The approach is theoretically grounded with proofs regarding the tracking of domain risk, the uniqueness of the fixed point (contraction mapping), and its interpretation as transfer-aware water-filling. The methodology is elegant, avoiding the need for labeled validation sets or expensive gradient computations during the control loop.
The evaluation is robust, covering two large backbones (Qwen-2.5-7B, LLaMA-3.1-8B) and five heterogeneous domains (Knowledge, Math, Code, Reasoning, Biomedical). CoDA outperforms uniform mixing, static mixture baselines (DoReMi, Temperature), and gradient-surgery methods (PCGrad, GradNorm) across all metrics, achieving significant gains (+1.8 avg score) while using only 50% of the data. The paper provides strong ablations isolating the contribution of the headroom term versus the affinity term, demonstrating that the latter is crucial for capturing synergistic/interfering relationships. Mechanism analysis confirms that the affinity signal correlates highly ($r=0.94$) with oracle leave-one-out transfer measurements. The results are consistent across seeds and model scales.
The paper provides detailed descriptions of the experimental setup, including dataset sources, prompt templates, LoRA configurations, and hyperparameters. The algorithm is clearly defined with pseudocode. The authors state that code and configurations will be released. The reliance on forward passes makes the method computationally transparent and easy to implement on top of existing LoRA pipelines. The theoretical proofs are included in the appendix, adding to the reproducibility of the claims.
The method requires a small unlabeled probe set per domain, which may be a constraint in strictly zero-data settings (though no labels are needed). The competence signal relies on model calibration; severe miscalibration could degrade performance, although the authors note a warmup phase helps mitigate this. The method assumes that the probe data is representative of the domain distribution. Additionally, while it reduces gradient conflict, it does not eliminate it entirely, and the quadratic program solution, while efficient, adds some overhead compared to static mixing.
This work significantly advances the efficiency and effectiveness of multi-task fine-tuning for LLMs. By reducing the data and compute required for adapting models to multiple domains, it lowers the barrier to entry for specialized model deployment and reduces the environmental impact of training. The label-free nature of the controller makes it applicable to scenarios where labeled data is scarce or expensive. The insights into cross-domain transfer and interference provide valuable theoretical understanding for multi-task learning. CoDA introduces a novel, label-free, forward-only mechanism for dynamically balancing multi-task LoRA training by estimating domain competence and cross-domain affinity, achieving state-of-the-art performance with reduced data and compute costs.
Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
Primary: Yale University
All Institutions: Yale University, University of Illinois Urbana-Champaign, Microsoft Research, Amazon
Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
The paper introduces MedPMC, a systematic, automated pipeline for curating high-fidelity image-text pairs from PubMed Central (PMC). The methodology addresses a critical bottleneck in medical multimodal learning: the scarcity of large-scale, high-quality, and clinically relevant data. The pipeline involves several sophisticated components: (1) filtering 6.1 million articles for permissive licenses; (2) detecting multi-panel figures; (3) separating individual figures from panels; (4) aligning figures with their corresponding captions using natural language processing techniques; and (5) classifying figures for medical relevance. The authors report high performance on these component tasks (e.g., F1=93.2 for screening, mAP=89.8 for figure separation), indicating a robust and well-engineered curation process. The novelty lies not in a single algorithmic breakthrough but in the systematic integration and scaling of these components to create a massive, high-quality dataset, coupled with rigorous validation against prior, lower-fidelity datasets.
The evaluation is comprehensive and compelling. First, the authors validate the quality of the curated data through manual review by five annotators (three with medical training), showing a significant improvement in medical relevance (95.3% vs. 19.7% in a prior dataset). Second, they train a CLIP-style model on the MedPMC corpus and evaluate it on 26 benchmarks across 11 medical specialties. The MedPMC-trained model outperforms the strongest architecture-matched biomedical CLIP baseline by 7.1 percentage points in average zero-shot AUC, despite using fewer than half the image-text pairs. This demonstrates the high signal-to-noise ratio of the MedPMC data. Third, they integrate the vision encoder into a multimodal large language model (MLLM) and show improvements in medical visual question-answering. Finally, they demonstrate clinical utility by improving morphology-to-image retrieval on a real-world dermatology dataset from Yale New Haven Health System. The results are statistically significant and practically meaningful.
The authors provide extensive resources for reproducibility. The code for the curation pipeline is publicly available on GitHub. The MedPMC corpus, component-level benchmark resources, pretrained checkpoints, and metadata are released on Hugging Face. The data release includes versioning by article cutoff date, source-license filters, and processing configuration, allowing users to reproduce specific snapshots. They also provide clear instructions for license-aware filtering. The only non-public component is the clinical dermatology data, which is subject to privacy restrictions, but aggregate results and processing procedures are described. This level of transparency is exemplary.
The primary limitation is the reliance on open-access literature, which may introduce selection bias towards certain types of studies or institutions. The pipeline's performance, while high, is not perfect (e.g., F1=81.4 for caption separation), meaning some noise remains in the dataset. The clinical evaluation is limited to dermatology retrieval, and while promising, it does not cover the full breadth of medical specialties. The authors acknowledge that some records are derived from articles with non-commercial licenses (CC BY-NC), which restricts commercial use of the dataset. Additionally, the study focuses on image-text pairs; the extension to other modalities (e.g., audio, video, time-series) is not addressed.
The MedPMC framework has significant potential to accelerate the development of medical multimodal foundation models. By providing a large-scale, high-quality, and openly accessible dataset, it lowers the barrier to entry for researchers and clinicians working in this domain. The improved performance of models trained on MedPMC suggests that high-fidelity data curation is a key lever for improving model capabilities. The release of the code and benchmarks encourages further research in data curation and multimodal learning. However, the non-commercial license restrictions for some data may limit its impact in commercial healthcare applications. The authors have responsibly addressed privacy concerns by not releasing patient data and by providing clear licensing information. Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling. Meanwhile, unified vision-language models have demonstrated strong multimodal generation capabilities, yet their potential as world models remains underexplored. In this work, we introduce \texttt{WorldBagel}, a unified VLAW framework built on BAGEL, a modern multimodal unified model, and use it to systematically investigate the role of unification in world modeling. Across multi-task robotic manipulation and cross-domain experiments, \texttt{WorldBagel} consistently outperforms task-specific alternatives and learns action representations that are more structured and semantically aligned with visual and linguistic context. Experiments on LIBERO, Language Table, and Franka show that unification is not only an architectural convenience, but also a key factor in learning effective VLAW models, leading to consistent empirical gains and deeper insights into multimodal world modeling. Code and model checkpoints will be released upon acceptance.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
The paper introduces WorldBagel, a unified Vision-Language-Action-World (VLAW) framework built upon the BAGEL two-tower architecture. The core methodological contribution lies in extending a powerful multimodal generative model (BAGEL) to jointly support multimodal understanding, structured action modeling, and future world prediction. The VLAW formulation is clearly defined, aiming to model the joint distribution of future observations and actions conditioned on past states and language instructions. A significant technical contribution is the Fourier Feature Action Decoder (FFAD) and Fourier Feature Action Tokenizer (FFAT). FFAD addresses the limitations of standard regression and discretization-based action tokenizers by mapping continuous actions into Fourier features and predicting in this space. The inverse mapping uses phase-consistent averaging for reconstruction. This approach is well-justified with theoretical analysis provided in the appendix, demonstrating Lipschitz stability, injectivity, consistency of reconstruction, and approximation advantages. This mathematical rigor is a strong point. The interleaved VLAW modeling via sequence plans, adapted from BAGEL, is a practical and flexible way to structure multimodal sequences for multi-view, multi-step observations and control. The concept of sampling different sequence plans to balance training objectives is sound. Furthermore, the LLM-inspired multimodal train-time data sampling, using mixture dataset sampling and priority sequence-plan sampling, is a crucial engineering detail for stabilizing training across heterogeneous datasets and balancing policy learning with world modeling. The overall architecture leverages the strengths of BAGEL's GEN/UND experts, with action modeling integrated through fine-tuned tokenizers and decoders rather than a new expert. This design choice maintains the unified nature of the model.
The experimental evaluation is comprehensive and rigorous, addressing three key empirical findings: multi-task performance, action representation quality, and stability under distribution shifts. 1. **Multi-task Performance**: WorldBagel is evaluated on LIBERO, Language Table, and Franka benchmarks. On LIBERO, it achieves state-of-the-art multi-task manipulation performance (98.0% average success rate), outperforming strong VLA baselines like OpenVLA-OFT and RynnVLA-002. The world modeling capabilities are also quantitatively assessed using FVD, PSNR, SSIM, and LPIPS, showing consistent improvements over RynnVLA-002 across all datasets, especially in action-conditioned prediction. This clearly demonstrates the empirical gains of the unified VLAW approach. 2. **Action Representation Quality**: A detailed ablation study on action decoder design (regression, bin discretization, FAST, FFAD) on LIBERO shows FFAD significantly reduces action MSE and improves success rates. Further analysis on the number of Fourier bands (K) in FFAD/FFAT provides insights into optimal hyperparameter choices. Crucially, the representation structure analysis using a linear probe classifier reveals that FFAD produces more structured and task-relevant action embeddings, leading to higher task identity prediction accuracy. This is a strong validation of the FFAD design. 3. **Stability Under Distribution Shifts**: The paper investigates robustness to action noise, scaling, and temporal perturbations on LIBERO. WorldBagel consistently maintains higher prediction fidelity (PSNR, LPIPS) compared to RynnVLA-002 under these shifts. The eigenvalue spectrum analysis further supports this, showing WorldBagel learns richer and more stable action representations (higher effective rank, lower dominant eigenvalue ratio). This finding is particularly important for real-world robotics applications where such shifts are common. The choice of baselines is appropriate, including recent strong VLA models and a direct competitor (RynnVLA-002) that also aims for VLAW unification. The use of multiple metrics (success rate, FVD, PSNR, SSIM, LPIPS, A-MSE, linear probe accuracy, eigenvalue spectrum) provides a holistic view of the model's performance and internal properties. The experiments are well-designed to support the paper's claims about the benefits of unification.
The paper states that "Code and model checkpoints will be released upon acceptance," which is a positive commitment. Detailed hyperparameters (learning rate, weight decay, batch size, training steps, K for FFAT/FFAD, priority weights) and hardware (8 H200 GPUs) are provided, which are crucial for reproducibility. The mathematical derivations for FFAD/FFAT in the appendix also contribute to understanding and potentially re-implementing those components. Given the complexity of large multimodal models, the release of code and checkpoints is essential for full reproducibility.
1. **Computational Cost**: While not explicitly stated as a limitation, training and deploying a model built on a large unified multimodal backbone like BAGEL is inherently computationally intensive, requiring significant resources (e.g., 8 H200 GPUs for 80K steps). This might limit its applicability for resource-constrained environments or rapid iteration. 2. **Scope of World Modeling**: The "world modeling" aspect primarily focuses on next-frame prediction for manipulation tasks. While crucial, it doesn't delve into more abstract forms of world knowledge, causal reasoning, or long-horizon planning beyond short action rollouts, which are often goals of broader world models. 3. **Reliance on Supervised Fine-tuning**: The model relies on supervised fine-tuning (SFT) on existing robotic datasets. While effective, this approach might be limited by the diversity and scale of available demonstration data, potentially hindering generalization to truly novel tasks or environments compared to models that learn more extensively through self-supervision or interaction. 4. **Generalizability Beyond Manipulation**: The experiments are confined to robotic manipulation tasks. While these are challenging, the generalizability of "unified VLAW modeling" to other embodied AI domains (e.g., navigation, human-robot interaction) or even broader generative tasks is not explored.
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
Structure-property relationships are foundational to biology, chemistry and materials science, where function, reactivity and physical response emerge from spatial, chemical and periodic organization. Mechanistically explaining these relationships requires interpreting structural evidence through scientific principles and physical constraints, from stereochemistry and bonding to symmetry, energetics and periodic order. However, applying artificial intelligence to this process presents a joint challenge of representation and reasoning: models must preserve domain-native structural information while showing how specific evidence supports predictions under these constraints. Here we introduce SciReasoner, a multimodal scientific foundation model for native structural reasoning across proteins, small molecules and inorganic crystals. SciReasoner discretizes coordinates, topologies and periodic connectivities into a unified structure-aware vocabulary, treating structural tokens as addressable evidence units during reasoning. In homology-controlled Gene Ontology prediction, SciReasoner improves Cellular Component annotation for low-homology and orphan-like proteins, increasing $F_{\max}$ from 0.42 to 0.55. In chemistry, it raises single-step retrosynthesis accuracy from 0.63 to 0.72 while generating fragment-level disconnection and precursor-verification traces. In materials science, its representations separate elemental and compound phases and resolve high- and low-band-gap regimes. Across 86 benchmarks, SciReasoner achieves state-of-the-art performance on 67 tasks. Double-blind expert evaluation rates its reasoning traces as preferred or at least comparable to those of a frontier large language model in 98% of cases. By making structure an inspectable substrate for reasoning under scientific constraints, SciReasoner connects accurate prediction with interpretable scientific inference.
Primary: Shanghai Artificial Intelligence Laboratory
All Institutions: Shanghai Artificial Intelligence Laboratory, The Chinese University of Hong Kong, Shanghai Jiao Tong University, Fudan University, University of Sydney, Nanjing University, University of Oxford, The University of Science and Technology of China, Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Stanford University
SciReasoner represents a significant leap in scientific machine learning by unifying structural reasoning across biology, chemistry, and materials science through a novel tokenization and autoregressive framework, achieving state-of-the-art results and offering interpretable insights that bridge the gap between prediction and mechanistic understanding.
The paper introduces "SciReasoner," a multimodal foundation model designed for "native structural reasoning" across three distinct scientific domains: proteins, small molecules, and inorganic crystals. The core methodological innovation lies in the discretization of continuous structural data (coordinates, topologies, periodic connectivities) into a unified, structure-aware vocabulary. By treating structural tokens as addressable evidence units within an autoregressive reasoning trajectory, the model attempts to bridge the gap between geometric representation and logical inference. This approach moves beyond standard graph neural networks or transformer-based sequence models by explicitly integrating physical constraints (stereochemistry, bonding, symmetry) into the tokenization and reasoning process. The architecture appears to leverage a large language model backbone adapted for these specialized structural tokens, allowing for chain-of-thought style reasoning where the model generates explanations alongside predictions. The unification of these three disparate fields under a single "structure-property" framework is ambitious and represents a significant conceptual shift from domain-specific models to a general scientific reasoning engine.
The evaluation is extensive, covering 86 benchmarks across biology, chemistry, and materials science. Key results include: 1. **Biology:** Improvement in Gene Ontology prediction for low-homology proteins (Cellular Component $F_{max}$ from 0.42 to 0.55), demonstrating the model's ability to generalize beyond sequence similarity to structural function. 2. **Chemistry:** Single-step retrosynthesis accuracy increased from 0.63 to 0.72, with the added capability of generating fragment-level disconnection and precursor-verification traces. This interpretability is a key selling point. 3. **Materials Science:** The model successfully separates elemental and compound phases and resolves band-gap regimes, indicating strong representation learning capabilities for periodic structures. 4. **General Performance:** State-of-the-art performance on 67 out of 86 tasks. 5. **Human Evaluation:** Double-blind expert evaluation rated the reasoning traces as preferred or comparable to frontier LLMs in 98% of cases. The breadth of the evaluation is impressive, but the reliance on "homology-controlled" settings for biology suggests the model's true value is in the "long-tail" or zero-shot scenarios where traditional methods fail. The jump in retrosynthesis accuracy is significant in a field where gains are often marginal.
The paper is published on arXiv. While the abstract and introduction provide a high-level overview, the full text provided in the prompt is truncated (sections are listed but content is missing). However, based on the abstract's description of "discretizing coordinates... into a unified structure-aware vocabulary," the method implies a specific tokenization scheme that, if detailed in the full paper (likely in the Method section), would be reproducible. The claim of 86 benchmarks suggests a comprehensive suite, which aids reproducibility if the benchmark definitions are standardized. The lack of a provided code URL in the prompt is a negative, but top-tier scientific ML papers increasingly open-source code; the high institutional backing (Shanghai AI Lab, CUHK, etc.) suggests code release is likely or imminent.
1. **Computational Cost:** Training a multimodal foundation model on three complex scientific domains with a unified vocabulary is likely extremely computationally intensive. The inference cost for generating "reasoning traces" may be prohibitive for high-throughput screening. 2. **Domain Specificity of Tokenization:** While the "unified vocabulary" is a strong claim, the physical constraints of proteins (flexible, solvent-exposed) differ vastly from inorganic crystals (rigid, periodic). It is unclear how well a single tokenizer handles the topological diversity without significant information loss or ambiguity. 3. **Evaluation Bias:** The "expert evaluation" of reasoning traces, while promising, is subjective. The 98% preference rate is exceptionally high and may suffer from bias if the traces are generated by a model known to be high-performing. 4. **Data Quality:** The performance is heavily dependent on the quality of the training data (PDB, ZINC, Materials Project, etc.). Errors in structural data or labeling in these databases will propagate into the model's "reasoning."
This work has profound implications for scientific discovery. By providing interpretable, structure-grounded reasoning, it addresses the "black box" problem in AI for science. This could accelerate drug discovery (by explaining *why* a molecule is active), materials design (by explaining phase stability), and synthetic biology. The ability to handle low-homology proteins could aid in understanding orphan diseases. However, the potential for misuse in designing harmful biological agents or novel toxins via automated retrosynthesis must be considered, necessitating robust safety guardrails in the reasoning traces. SciReasoner represents a significant leap in scientific machine learning by unifying structural reasoning across biology, chemistry, and materials science through a novel tokenization and autoregressive framework, achieving state-of-the-art results and offering interpretable insights that bridge the gap between prediction and mechanistic understanding.
Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .
Primary: University of Hong Kong
All Institutions: University of Hong Kong, Snowflake
Spider 2.0-AIFunc introduces the first benchmark for evaluating text-to-SQL systems on AI-native SQL workflows, revealing that current models struggle with semantic parameterization and that complex agent frameworks offer no advantage over minimal setups, signaling a shift in the bottleneck of enterprise text-to-SQL from planning to semantic execution. The paper provides a rigorous evaluation framework and detailed error analysis that will serve as a foundational resource for the community as the industry transitions to AI-integrated database systems.
The paper introduces a novel benchmark construction methodology for "AI-Native SQL," a new paradigm where LLM capabilities are exposed as native SQL functions (e.g., `AI_CLASSIFY`, `AI_SENTIMENT`). The core methodological contribution is an agent-based pipeline that rewrites existing enterprise text-to-SQL tasks (from Spider2-Snow) into AI-native forms. This involves not just transforming the SQL query but also refining the natural language instructions to ensure specification determinism (explicitly defining AI function parameters) and verifying execution determinism (handling stochasticity in AI function outputs via multi-round verification). The approach addresses a critical gap in current benchmarks which ignore the shift towards semantic operators in SQL engines.
The evaluation is rigorous and comprehensive. The authors evaluate 10 state-of-the-art models (proprietary and open-source) using a minimal agent framework to isolate model capability from framework complexity. They provide detailed error analysis stratified by model performance tiers, identifying specific failure modes such as schema grounding, predicate specification, and AI function parameterization. The finding that complex agent frameworks (AutoLink, ReFoRCE, DSR-SQL) do not outperform a minimal setup is a significant and surprising empirical result, suggesting that the bottleneck in AI-Native SQL is primarily model capability (semantic understanding and parameterization) rather than retrieval or planning complexity. The benchmark includes 465 verified instances across 125 databases, providing a robust testbed.
The paper provides a detailed description of the construction pipeline, verification protocols, and evaluation metrics. The code and data are made publicly available on GitHub. The multi-round execution verification protocol ensures that the benchmark instances are stable, which is crucial for reproducibility in the context of stochastic AI functions. The evaluation setup is clearly defined, including the handling of timeouts and the specific comparison logic for results.
The benchmark is currently scoped to Snowflake and its specific set of Cortex AI functions, limiting generalizability to other platforms (BigQuery, Databricks) or function types. The construction relies on a single strong model (Claude Opus 4.5) for rewriting, which may introduce biases in instruction wording or task formulation. The evaluation uses a single agent trajectory per model, which does not capture the variance of stochastic agents or allow for pass@k analysis. Additionally, the benchmark relies on the assumption that the underlying AI functions remain stable over time, which may not hold if the cloud provider updates the backend models.
This paper has significant broader impact as it establishes the first standardized benchmark for evaluating text-to-SQL systems on AI-native workflows. As cloud data platforms increasingly integrate LLMs directly into their query engines, this benchmark will guide the development of more capable and reliable text-to-SQL systems. It highlights the unique challenges of AI-Native SQL, such as parameterization and semantic grounding, which are distinct from traditional text-to-SQL challenges. The findings will influence how researchers design agents and models for enterprise data analysis, potentially accelerating the adoption of AI-native SQL in industry. Spider 2.0-AIFunc introduces the first benchmark for evaluating text-to-SQL systems on AI-native SQL workflows, revealing that current models struggle with semantic parameterization and that complex agent frameworks offer no advantage over minimal setups, signaling a shift in the bottleneck of enterprise text-to-SQL from planning to semantic execution. The paper provides a rigorous evaluation framework and detailed error analysis that will serve as a foundational resource for the community as the industry transitions to AI-integrated database systems.
Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.
Primary: Amazon
All Institutions: Amazon
This paper presents a comprehensive and methodologically sound large-scale evaluation of uncertainty estimation in multilingual LLMs, providing crucial insights into the interplay between language resource levels, model scale, and uncertainty signaling mechanisms.
The paper proposes a rigorous evaluation framework for Uncertainty Estimation (UE) in Large Language Models (LLMs) across 22 languages. The core methodological innovation lies in the evaluation setup: using human-curated Multiple-Choice Question Answering (MCQA) datasets with elicited long-form reasoning to avoid the noise associated with LLM-as-a-judge or embedding-based correctness metrics. This allows for a clean, label-grounded assessment of UE methods (both open-box and closed-box) on the reasoning traces rather than just the final answer. The study systematically compares nine UE methods across varying model scales (270M to 235B) and resource levels. The approach is sound, addressing a critical gap in multilingual trustworthiness evaluation by isolating the uncertainty signal from the correctness signal more effectively than prior work.
The experimental design is comprehensive and large-scale. The authors evaluate 9 models and 9 UE methods across 22 languages, covering high, mid, and low-resource settings. Key findings include: (1) English reasoning significantly boosts UE performance for low-resource languages, suggesting the bottleneck is generation, not comprehension; (2) Self-Verbalized uncertainty outperforms other methods at large scales (235B), while open-box methods are better for smaller models; (3) Sampling-based methods fail on low-resource languages due to lack of diversity signal. The results are robust, supported by statistical significance testing and confidence intervals. The analysis of cross-lingual answer options and threshold calibration adds practical value. The use of parallel datasets (Global-MMLU and MMLU-ProX) ensures comparability.
The paper provides detailed descriptions of the datasets, models, prompts, and UE methods. It mentions the use of LM-Polygraph for implementation. The hardware infrastructure is specified. However, the specific versions of the models (e.g., "Claude 4.5 Sonnet" which appears to be a future/hypothetical name or typo for a current model, and "Gemma3" which is also not yet publicly released as of mid-2024, suggesting this is a very recent or forward-looking paper) and the exact random seeds are not fully detailed in the text provided, though the appendix references suggest they are available. The code release is mentioned but the URL is not provided in the text. The methodology is clear enough for replication if the models and datasets are accessible.
The study is limited to MCQA tasks, which may not fully generalize to open-ended generation where correctness is harder to define. The reliance on specific datasets (Global-MMLU, MMLU-ProX) means results might vary with other benchmarks. The "Claude 4.5 Sonnet" and "Gemma3" references are unusual and might indicate the paper is from the future or uses internal/unreleased models, which could limit immediate reproducibility for the broader community. The study does not include training-based UE methods.
This work has significant implications for deploying trustworthy LLMs in multilingual contexts. By demonstrating that generation language matters more than question language for UE, it provides actionable guidance for system designers (e.g., using English reasoning traces for low-resource languages). It also clarifies the trade-offs between open-box and closed-box methods based on model scale, helping practitioners choose appropriate UE strategies. The findings challenge the assumption that low-resource language performance is solely due to comprehension deficits, highlighting generation quality as a key factor. This paper presents a comprehensive and methodologically sound large-scale evaluation of uncertainty estimation in multilingual LLMs, providing crucial insights into the interplay between language resource levels, model scale, and uncertainty signaling mechanisms.