Last 7 Days (July 06 – July 12, 2026)
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Primary: Stanford University
All Institutions: Stanford University, MIT, Nunchux AI, UC Berkeley, CMU
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
FourTune proposes an innovative and technically sound approach for fully 4-bit efficient post-training of diffusion models, addressing the critical challenges of memory footprint and training speed. The core of the methodology is the "triple-branch hybrid pipeline" which augments a standard LoRA architecture with a "frozen numerical stabilizer" (FNS). This FNS branch is a small, frozen, full-precision component designed to isolate and process quantization-sensitive outliers, thereby enabling stable training under native W4A4G4 (weights, activations, gradients) computation. This is a particularly clever design choice, as it mitigates the primary instability issue of low-bit quantization (outliers) without incurring significant computational overhead during training, as the FNS branch remains frozen. The integration of this FNS with both a standard BF16 LoRA branch and a quantized 4-bit LoRA branch provides a robust mechanism for maintaining quality while maximizing efficiency. Furthermore, the paper introduces hardware-efficient block-wise quantization and customized fused kernels. These components are crucial for translating the theoretical benefits of 4-bit quantization into practical speedups by optimizing memory bandwidth and accelerating quantized backpropagation operations (e.g., QGEMM, QMatmul, QConv). The end-to-end W4A4G4 paradigm, encompassing weights, activations, and gradients, is a comprehensive solution that pushes the boundaries of efficient fine-tuning for large generative models.
The experimental evaluation is comprehensive and robust, demonstrating FourTune's effectiveness across diverse and important post-training tasks for diffusion models: customization (DreamBooth-like), reinforcement learning for aesthetic preference, and knowledge distillation. The choice of FLUX.1-dev (12B parameters) as the primary model, along with validation on SDXL (1.5B), ensures the results are relevant for large-scale, state-of-the-art diffusion models. Baselines include BF16 LoRA and NF4 QLoRA, which are appropriate and strong competitors. The quantitative results are highly compelling: FourTune consistently matches or slightly outperforms the image quality of full-precision BF16 LoRA and NF4 QLoRA across all tasks, as measured by FID, CLIP Score, and LPIPS. This quality preservation at such low bit-depth is a significant achievement. More importantly, FourTune delivers substantial efficiency gains, reducing GPU memory overhead by 2.25x compared to BF16 LoRA (achieving a footprint comparable to NF4 QLoRA) and increasing end-to-end training throughput by 2.27x over BF16 LoRA and 2.79x over NF4 QLoRA. This effectively breaks the memory-speed trade-off often encountered in large model post-training. The ablation studies are well-designed and clearly demonstrate the critical contribution of each proposed component: the Frozen Numerical Stabilizer for stability and quality, and the W4A4G4 quantization, block-wise quantization, and fused kernels for efficiency. The generalization to SDXL further strengthens the claims.
The paper provides a good level of detail regarding the methodology, including the architecture of the triple-branch pipeline, the role of the FNS, and the types of quantization and fused kernels used. The experimental setup is well-described, including the specific models (FLUX.1-dev, SDXL), tasks, baselines, and evaluation metrics. The appendix further elaborates on implementation details, hyperparameters, and training configurations, which are crucial for reproducibility. While no direct code link is provided, the comprehensive description suggests that a skilled research team should be able to reproduce the results.
The paper does not explicitly discuss limitations, but some can be inferred. While the FNS effectively handles outliers, its design might introduce a slight architectural overhead compared to a purely 4-bit system, even if frozen. The customized fused kernels, while highly efficient, might require specific hardware support or careful implementation to achieve optimal performance across different GPU architectures. The evaluation is primarily focused on image generation diffusion models; its applicability and performance on other types of diffusion models (e.g., audio, video) or other generative architectures (e.g., GANs, VAEs) are not explored. The paper also doesn't delve into the potential challenges of deploying such a highly optimized, custom-kernel-dependent solution in diverse production environments.
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
Primary: Yale University
All Institutions: Yale University, University of Illinois Urbana-Champaign, Microsoft Research, Amazon
Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
The paper introduces MedPMC, a systematic, automated pipeline for curating high-fidelity image-text pairs from PubMed Central (PMC). The methodology addresses a critical bottleneck in medical multimodal learning: the scarcity of large-scale, high-quality, and clinically relevant data. The pipeline involves several sophisticated components: (1) filtering 6.1 million articles for permissive licenses; (2) detecting multi-panel figures; (3) separating individual figures from panels; (4) aligning figures with their corresponding captions using natural language processing techniques; and (5) classifying figures for medical relevance. The authors report high performance on these component tasks (e.g., F1=93.2 for screening, mAP=89.8 for figure separation), indicating a robust and well-engineered curation process. The novelty lies not in a single algorithmic breakthrough but in the systematic integration and scaling of these components to create a massive, high-quality dataset, coupled with rigorous validation against prior, lower-fidelity datasets.
The evaluation is comprehensive and compelling. First, the authors validate the quality of the curated data through manual review by five annotators (three with medical training), showing a significant improvement in medical relevance (95.3% vs. 19.7% in a prior dataset). Second, they train a CLIP-style model on the MedPMC corpus and evaluate it on 26 benchmarks across 11 medical specialties. The MedPMC-trained model outperforms the strongest architecture-matched biomedical CLIP baseline by 7.1 percentage points in average zero-shot AUC, despite using fewer than half the image-text pairs. This demonstrates the high signal-to-noise ratio of the MedPMC data. Third, they integrate the vision encoder into a multimodal large language model (MLLM) and show improvements in medical visual question-answering. Finally, they demonstrate clinical utility by improving morphology-to-image retrieval on a real-world dermatology dataset from Yale New Haven Health System. The results are statistically significant and practically meaningful.
The authors provide extensive resources for reproducibility. The code for the curation pipeline is publicly available on GitHub. The MedPMC corpus, component-level benchmark resources, pretrained checkpoints, and metadata are released on Hugging Face. The data release includes versioning by article cutoff date, source-license filters, and processing configuration, allowing users to reproduce specific snapshots. They also provide clear instructions for license-aware filtering. The only non-public component is the clinical dermatology data, which is subject to privacy restrictions, but aggregate results and processing procedures are described. This level of transparency is exemplary.
The primary limitation is the reliance on open-access literature, which may introduce selection bias towards certain types of studies or institutions. The pipeline's performance, while high, is not perfect (e.g., F1=81.4 for caption separation), meaning some noise remains in the dataset. The clinical evaluation is limited to dermatology retrieval, and while promising, it does not cover the full breadth of medical specialties. The authors acknowledge that some records are derived from articles with non-commercial licenses (CC BY-NC), which restricts commercial use of the dataset. Additionally, the study focuses on image-text pairs; the extension to other modalities (e.g., audio, video, time-series) is not addressed.
The MedPMC framework has significant potential to accelerate the development of medical multimodal foundation models. By providing a large-scale, high-quality, and openly accessible dataset, it lowers the barrier to entry for researchers and clinicians working in this domain. The improved performance of models trained on MedPMC suggests that high-fidelity data curation is a key lever for improving model capabilities. The release of the code and benchmarks encourages further research in data curation and multimodal learning. However, the non-commercial license restrictions for some data may limit its impact in commercial healthcare applications. The authors have responsibly addressed privacy concerns by not releasing patient data and by providing clear licensing information. Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
We propose contrastive order learning (ConOrd), a contrastive learning framework for ordinal regression that integrates the strengths of contrastive learning and order learning. While contrastive learning effectively leverages all samples in a batch, it typically ignores the inherent ordering among rank labels. Conversely, order learning explicitly models label ordinality but often relies on local, margin-based comparisons, limiting its ability to capture global ordinal structure. ConOrd addresses these limitations by introducing a contrastive order loss with soft affinity and disparity weights based on rank differences, enabling fine-grained modeling of ordinal relationships across all sample pairs within a batch. Extensive experiments on a range of ordinal regression tasks, including facial age estimation, blind image quality assessment, and blind video quality assessment, demonstrate that ConOrd consistently achieves state-of-the-art performance and generalizes well across diverse ordinal regression scenarios. The source code is available at https://github.com/cwlee00/ConOrd.
Primary: Korea University
All Institutions: Samsung Electronics, Korea University
**METHODOLOGY_ASSESSMENT:** The paper introduces Contrastive Order Learning (ConOrd), a novel and principled framework that elegantly merges the strengths of contrastive learning and order learning for ordinal regression. The core methodological contribution is the "contrastive order loss" ($L_{ConOrd}$), which innovatively assigns soft affinity ($a_{ij}$) and disparity ($b_{ij}$) weights to *all* sample pairs within a batch, based on their rank differences. This design addresses key limitations of prior work: unlike traditional supervised contrastive learning, ConOrd explicitly models the inherent ordering of labels; and unlike margin-based order learning or hard-thresholding contrastive methods (e.g., RnC), it captures fine-grained, global ordinal structure through continuous, rank-dependent interactions. The chosen quadratic forms for affinity and disparity weights, combined with negative squared Euclidean distance for similarity, are justified by rigorous gradient analysis in the appendix, demonstrating how they induce stable, rank-aware attraction and repulsion forces that are localized to informative pairs. The integration of a standard center loss ($L_{center}$) further enhances intra-class compactness, and the use of $k$-NN for inference is a practical and well-established approach in order learning. The overall framework is coherent, theoretically well-supported, and represents a significant advancement in designing loss functions for ordinal regression. **EXPERIMENTAL_EVALUATION:** The experimental evaluation is exceptionally comprehensive and compelling, providing strong evidence for ConOrd's generality and effectiveness. The authors rigorously test ConOrd across a remarkably diverse range of ordinal regression tasks, including facial age estimation (6 datasets), blind image quality assessment (5 datasets), blind video quality assessment (5 datasets), temperature prediction, gaze direction estimation, and deep imbalanced regression (DIR) benchmarks. This breadth of evaluation is a major strength, validating the claim of a "general framework." ConOrd consistently achieves state-of-the-art performance across nearly all benchmarks, often with substantial improvements over strong baselines, including recent transformer-based methods, multi-modal approaches (e.g., OrdinalCLIP, NumCLIP), and imbalance-aware techniques (SLACE). The experiments are conducted under fair conditions, often using identical backbone architectures and evaluation protocols as the baselines. The ablation studies are thorough and insightful, systematically dissecting the contributions of different contrastive learning schemes, the center loss, and various design choices for the affinity/disparity weights and similarity functions. These ablations empirically validate the proposed design and provide a deeper understanding of ConOrd's mechanisms. Visualizations of feature similarity matrices further support that ConOrd learns more faithful and structured ordinal representations. **REPRODUCIBILITY:** The paper explicitly provides a link to the source code (`https://github.com/cwlee00/ConOrd`), which is crucial for reproducibility. Furthermore, the appendix offers extensive details on experimental setups, including specific datasets, backbone architectures, optimizers, learning rate schedules, and data augmentation strategies for each task. This level of detail, coupled with the public code, indicates a strong commitment to reproducibility. **LIMITATIONS:** While ConOrd demonstrates impressive performance, a few potential limitations exist. The "all-pairs" comparison in the loss function, while effective for capturing global ordinal structure, inherently leads to quadratic computational complexity with respect to the batch size. Although this is manageable for typical batch sizes, it could become a bottleneck for extremely large batches or in scenarios requiring massive numbers of pairwise comparisons. Similarly, the $k$-NN inference, while simple and effective, can be computationally intensive for very large training datasets, potentially requiring approximate nearest neighbor search methods for scalability. The paper does not provide a detailed analysis of the computational overhead of training or inference compared to some of the faster baselines. While the paper claims insensitivity to the specific functional form of the weights, the exact values of hyperparameters like the temperature parameter $\tau$, the small constant $\epsilon$ in the affinity weight, and the weight $\lambda$ for the center loss are not extensively ablated, and their optimal tuning might still be task-dependent. **BROADER_IMPACT:** ConOrd offers a robust and general framework for ordinal regression, a fundamental problem with wide-ranging applications in various real-world domains. Its consistent state-of-the-art performance across diverse tasks, from facial age estimation to image/video quality assessment and medical diagnostics (implied by general ordinal regression), suggests significant potential for practical deployment. The method's ability to learn fine-grained ordinal relationships could lead to more accurate, reliable, and nuanced predictive models. The authors have commendably included an "Impact Statement," acknowledging the ethical considerations associated with deploying models involving human attributes, particularly concerning potential biases in training data. Their recommendation for fair evaluations and using the model as a decision-support tool rather than an autonomous system is a responsible and necessary stance for this type of research, promoting ethical AI development. This work provides a flexible and powerful foundation for future advancements in ordinal representation learning. ConOrd introduces a novel contrastive order learning framework that effectively integrates ordinal structure into batch-wise representation learning through a soft, rank-aware, all-pairs contrastive loss. The paper presents a theoretically sound methodology with rigorous gradient analysis and demonstrates consistent state-of-the-art performance across an extensive and diverse set of ordinal regression benchmarks, making a significant contribution to the field of representation learning for structured prediction tasks.
We propose contrastive order learning (ConOrd), a contrastive learning framework for ordinal regression that integrates the strengths of contrastive learning and order learning. While contrastive learning effectively leverages all samples in a batch, it typically ignores the inherent ordering among rank labels. Conversely, order learning explicitly models label ordinality but often relies on local, margin-based comparisons, limiting its ability to capture global ordinal structure. ConOrd addresses these limitations by introducing a contrastive order loss with soft affinity and disparity weights based on rank differences, enabling fine-grained modeling of ordinal relationships across all sample pairs within a batch. Extensive experiments on a range of ordinal regression tasks, including facial age estimation, blind image quality assessment, and blind video quality assessment, demonstrate that ConOrd consistently achieves state-of-the-art performance and generalizes well across diverse ordinal regression scenarios. The source code is available at https://github.com/cwlee00/ConOrd.
Primary: Korea University
All Institutions: Samsung Electronics, Korea University
**METHODOLOGY_ASSESSMENT:** The paper introduces Contrastive Order Learning (ConOrd), a novel and principled framework that elegantly merges the strengths of contrastive learning and order learning for ordinal regression. The core methodological contribution is the "contrastive order loss" ($L_{ConOrd}$), which innovatively assigns soft affinity ($a_{ij}$) and disparity ($b_{ij}$) weights to *all* sample pairs within a batch, based on their rank differences. This design addresses key limitations of prior work: unlike traditional supervised contrastive learning, ConOrd explicitly models the inherent ordering of labels; and unlike margin-based order learning or hard-thresholding contrastive methods (e.g., RnC), it captures fine-grained, global ordinal structure through continuous, rank-dependent interactions. The chosen quadratic forms for affinity and disparity weights, combined with negative squared Euclidean distance for similarity, are justified by rigorous gradient analysis in the appendix, demonstrating how they induce stable, rank-aware attraction and repulsion forces that are localized to informative pairs. The integration of a standard center loss ($L_{center}$) further enhances intra-class compactness, and the use of $k$-NN for inference is a practical and well-established approach in order learning. The overall framework is coherent, theoretically well-supported, and represents a significant advancement in designing loss functions for ordinal regression. **EXPERIMENTAL_EVALUATION:** The experimental evaluation is exceptionally comprehensive and compelling, providing strong evidence for ConOrd's generality and effectiveness. The authors rigorously test ConOrd across a remarkably diverse range of ordinal regression tasks, including facial age estimation (6 datasets), blind image quality assessment (5 datasets), blind video quality assessment (5 datasets), temperature prediction, gaze direction estimation, and deep imbalanced regression (DIR) benchmarks. This breadth of evaluation is a major strength, validating the claim of a "general framework." ConOrd consistently achieves state-of-the-art performance across nearly all benchmarks, often with substantial improvements over strong baselines, including recent transformer-based methods, multi-modal approaches (e.g., OrdinalCLIP, NumCLIP), and imbalance-aware techniques (SLACE). The experiments are conducted under fair conditions, often using identical backbone architectures and evaluation protocols as the baselines. The ablation studies are thorough and insightful, systematically dissecting the contributions of different contrastive learning schemes, the center loss, and various design choices for the affinity/disparity weights and similarity functions. These ablations empirically validate the proposed design and provide a deeper understanding of ConOrd's mechanisms. Visualizations of feature similarity matrices further support that ConOrd learns more faithful and structured ordinal representations. **REPRODUCIBILITY:** The paper explicitly provides a link to the source code (`https://github.com/cwlee00/ConOrd`), which is crucial for reproducibility. Furthermore, the appendix offers extensive details on experimental setups, including specific datasets, backbone architectures, optimizers, learning rate schedules, and data augmentation strategies for each task. This level of detail, coupled with the public code, indicates a strong commitment to reproducibility. **LIMITATIONS:** While ConOrd demonstrates impressive performance, a few potential limitations exist. The "all-pairs" comparison in the loss function, while effective for capturing global ordinal structure, inherently leads to quadratic computational complexity with respect to the batch size. Although this is manageable for typical batch sizes, it could become a bottleneck for extremely large batches or in scenarios requiring massive numbers of pairwise comparisons. Similarly, the $k$-NN inference, while simple and effective, can be computationally intensive for very large training datasets, potentially requiring approximate nearest neighbor search methods for scalability. The paper does not provide a detailed analysis of the computational overhead of training or inference compared to some of the faster baselines. While the paper claims insensitivity to the specific functional form of the weights, the exact values of hyperparameters like the temperature parameter $\tau$, the small constant $\epsilon$ in the affinity weight, and the weight $\lambda$ for the center loss are not extensively ablated, and their optimal tuning might still be task-dependent. **BROADER_IMPACT:** ConOrd offers a robust and general framework for ordinal regression, a fundamental problem with wide-ranging applications in various real-world domains. Its consistent state-of-the-art performance across diverse tasks, from facial age estimation to image/video quality assessment and medical diagnostics (implied by general ordinal regression), suggests significant potential for practical deployment. The method's ability to learn fine-grained ordinal relationships could lead to more accurate, reliable, and nuanced predictive models. The authors have commendably included an "Impact Statement," acknowledging the ethical considerations associated with deploying models involving human attributes, particularly concerning potential biases in training data. Their recommendation for fair evaluations and using the model as a decision-support tool rather than an autonomous system is a responsible and necessary stance for this type of research, promoting ethical AI development. This work provides a flexible and powerful foundation for future advancements in ordinal representation learning. ConOrd introduces a novel contrastive order learning framework that effectively integrates ordinal structure into batch-wise representation learning through a soft, rank-aware, all-pairs contrastive loss. The paper presents a theoretically sound methodology with rigorous gradient analysis and demonstrates consistent state-of-the-art performance across an extensive and diverse set of ordinal regression benchmarks, making a significant contribution to the field of representation learning for structured prediction tasks.
Policy-based approaches to Bayesian experimental design (BED) allow the learning of deep policy networks that adaptively make intelligent design decisions based on previously collected data. However, the training of such policies is often held back by a fundamental challenge: the double intractability of the expected information gain (EIG). This necessitates expensive or complex approximations that restrict the effort one can invest in optimising the policy itself. To address this, we show that the double intractability of the EIG can be isolated from the policy learning by first solving a score matching problem that is independent of the policy used, then using the learned score approximation to train the policy in a singly intractable manner. This turns the key multiplicative cost into an additive one and reduces the computational burden on the policy training itself, making it far cheaper to train the policy multiple times when needed, e.g. for architecture search, hyperparameter tuning, or avoiding local optima. In our experiments we train multiple competitive policies without inducing a multiplicative cost in likelihood evaluations, which can increase performance by allowing us to select the best policy even without performing hyperparameter or architecture searches.
Primary: University of Oxford
All Institutions: University of Oxford
The paper presents a theoretically sound and empirically validated method for decoupling the double intractability of EIG in Bayesian Experimental Design using score matching, enabling efficient policy training and hyperparameter search.
The paper proposes a novel two-stage framework for Bayesian Experimental Design (BED) that decouples the double intractability of the Expected Information Gain (EIG) from policy training. By deriving an exact gradient expression for the EIG, the authors identify that the intractable terms (Stein and Fisher scores of the marginal likelihood) are independent of the policy parameters. They then employ a score matching objective (Marginal Score Matching) to learn these scores upfront using a neural network. This transforms the computationally expensive nested sampling required by methods like PCE into a fixed, additive cost, allowing for efficient policy training and hyperparameter search. The theoretical derivation is rigorous, and the separation of concerns is a significant conceptual contribution to the field of amortized inference and experimental design.
The experimental evaluation covers four distinct tasks: source location finding (2D and 3D) and three dynamical systems (stochastic pendulum, cart-pole, double link). The results demonstrate that the proposed method (SM-BED) achieves competitive or superior performance compared to state-of-the-art baselines including PCE, JointVarPost, and variational marginal/posterior approaches. Crucially, the experiments highlight the efficiency gain: SM-BED allows for the cheap training of multiple policy restarts (P=50), which improves final performance by selecting the best policy, a capability not afforded by baselines with multiplicative costs. The bias-variance analysis further validates the method, showing that gradient bias is a key determinant of performance and that SM-BED effectively manages this bias through score network capacity.
The paper provides detailed algorithmic descriptions, including the specific score matching loss and network architecture (TNP-style with attention). The experimental setup is well-described, with clear definitions of computational budgets (NLE). The inclusion of supplementary material with proofs and additional ablation studies enhances reproducibility. However, the reliance on specific dynamical system simulations and the potential instability in score network training (noted in the dynamical systems experiments) suggests that careful implementation is required.
The method assumes explicit, differentiable likelihoods and continuous design/observation spaces, limiting its applicability to implicit models or discrete designs without further adaptation. The score network training can be unstable, as noted in the dynamical systems experiments, requiring multiple seeds and budget splitting. Additionally, while the method reduces the *multiplicative* cost, the upfront cost of training the score network can still be significant for very high-dimensional problems, although it is amortized across policies.
This work has significant implications for fields requiring efficient, adaptive experimental design, such as active learning, clinical trials, and physical sciences. By making policy-based BED more computationally tractable, it enables the use of more complex, non-myopic policies that were previously infeasible due to computational constraints. This could lead to more data-efficient scientific discovery and better decision-making in resource-constrained environments. The paper presents a theoretically sound and empirically validated method for decoupling the double intractability of EIG in Bayesian Experimental Design using score matching, enabling efficient policy training and hyperparameter search.
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.
Primary: Graphcore
All Institutions: Graphcore
This paper has significant broader impact for the LLM inference community. It provides crucial practical guidance for researchers and practitioners considering relaxed speculative decoding. The findings challenge common assumptions and narratives, particularly regarding the utility of relaxation with lightweight multi-token-prediction (MTP) drafters, which are increasingly prevalent. The emphasis on optimizing draft length, understanding the impact of drafter capability, and accounting for generation length changes will directly inform deployment strategies. The finding that relaxation parameters do not generalize across tasks highlights the need for careful, task-specific evaluation, potentially saving significant engineering and evaluation costs. The paper's unified framework and rigorous evaluation methodology could serve as a template for future empirical investigations in LLM inference optimization. This paper provides a highly practical and rigorous investigation into training-free relaxed speculative decoding, unifying existing approaches, benchmarking them comprehensively, and distilling critical empirical findings for practitioners. The paper's strength lies in its systematic evaluation across diverse drafter types and inference settings, revealing that many relaxed approaches are unsuited for lightweight multi-token-prediction drafters and that relaxation requires considerable capability evaluation, offering invaluable guidance for LLM inference optimization.
The paper presents a robust methodological approach for investigating training-free relaxed speculative decoding techniques. A key strength is the introduction of a unified framework (alg:relaxed-spec-dec) that cleanly describes and compares diverse existing approaches by parameterizing relaxed target distributions for rejection, residual sampling, and bonus sampling. This framework is highly valuable for organizing and understanding the fragmented literature. The paper also provides a clear and practical model for understanding and quantifying speed-up (eq:speedup, eq:speedup-approx, eq:gen-len-speedup), which explicitly accounts for drafter cost, accepted draft length, and crucially, changes in overall generation length due to relaxation—a detail often overlooked in prior work. The decision to focus on training-free methods is a reasonable scope limitation, making the investigation directly applicable without additional training overhead. The detailed discussion of factors influencing speed-up and the emphasis on a flexible speed-up proxy model for practitioners are excellent.
The experimental evaluation is exceptionally thorough and well-designed, directly addressing shortcomings in prior literature. The authors benchmark a range of relaxed speculative decoding approaches on contemporary settings, using three diverse drafter+verifier pairs: a dedicated multi-token-prediction (MTP) module (Qwen3.5 MTP+27B), a weak language model drafter (Qwen3 0.6B+32B), and a strong language model drafter (Qwen3 8B+32B). This choice is critical for understanding the impact of drafter capability. The evaluation spans a wide range of inference-time parameters, including different draft lengths ($N_{draft}$) and relative drafter costs ($c_{rel}$), allowing for a holistic view of speed-up. Importantly, the paper uses modern, challenging reasoning benchmarks (AIME 2024, GPQA Diamond, LiveCodeBench Lite v6), which are more representative of real-world LLM applications than older summarization datasets. The use of repeated measurements to reduce noise and the consideration of standard stochastic sampling parameters (top-p/k) further enhance the rigor. The comparison of empirical results against the narratives of the original papers is a valuable contribution, providing a reality check on theoretical claims.
The paper explicitly states that code is available at `https://github.com/graphcore-research/relaxed-spec-dec-repro`, which is a strong indicator of reproducibility. The methodology and experimental setup are described in sufficient detail within the main text and are promised to be fully elaborated in the appendix (e.g., specific alpha values, evaluation harness). The use of standard models (Qwen, Llama) and benchmarks further aids reproducibility.
The paper acknowledges several limitations, such as focusing only on training-free, per-token relaxation methods and autoregressive drafters. While the speed-up proxy model is valuable, it relies on estimated $c_{rel}$ values rather than direct end-to-end latency measurements for all scenarios, which might be a limitation for practitioners seeking precise real-world numbers. The specific choice of verifier models (Qwen 3.5 27B/32B, Llama 3.1 70B) and benchmarks, while modern, still represents a subset of the vast LLM landscape. The paper also notes that the additional memory overhead for relaxed approaches can become notable for small models with large vocabulary sizes at larger batch sizes, which is a practical consideration.
This paper has significant broader impact for the LLM inference community. It provides crucial practical guidance for researchers and practitioners considering relaxed speculative decoding. The findings challenge common assumptions and narratives, particularly regarding the utility of relaxation with lightweight multi-token-prediction (MTP) drafters, which are increasingly prevalent. The emphasis on optimizing draft length, understanding the impact of drafter capability, and accounting for generation length changes will directly inform deployment strategies. The finding that relaxation parameters do not generalize across tasks highlights the need for careful, task-specific evaluation, potentially saving significant engineering and evaluation costs. The paper's unified framework and rigorous evaluation methodology could serve as a template for future empirical investigations in LLM inference optimization. This paper provides a highly practical and rigorous investigation into training-free relaxed speculative decoding, unifying existing approaches, benchmarking them comprehensively, and distilling critical empirical findings for practitioners. The paper's strength lies in its systematic evaluation across diverse drafter types and inference settings, revealing that many relaxed approaches are unsuited for lightweight multi-token-prediction drafters and that relaxation requires considerable capability evaluation, offering invaluable guidance for LLM inference optimization.
Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.
Primary: UCL
All Institutions: UCL
This paper makes a significant contribution to the ongoing debate about whether RL post-training merely reweights existing behaviors or enables new capabilities in base models. By providing a concrete, inspectable mechanism for "going beyond the base model," it offers valuable insights for understanding and improving LLM reasoning. The findings on procedural chunking and the phased emergence of compositional strategies have implications for skill acquisition research in AI. The distinction between exploration volume and selectivity, and the role of negative samples in focusing exploration on valid structures, is crucial for designing more effective RL algorithms for reasoning tasks. Furthermore, the analysis of how pretraining organizes primitive competence into a "procedural substrate" for RL highlights the importance of co-designing pretraining and post-training objectives. While the environment is synthetic, the mechanistic understanding gained here can inform hypotheses and experimental designs for more complex, real-world LLM applications. RL Post-Training Builds Compositional Reasoning Strategies provides a rigorous, mechanistic account of how reinforcement learning can enable compositional reasoning beyond the capabilities of a pretrained base model in a fully observable rewrite-grammar environment. The paper's strength lies in its novel controlled experimental setup, which allows for precise auditing of reasoning traces, revealing a phased emergence of valid compositional strategies, the critical role of RL's selectivity over exploration volume, and the importance of pretraining in organizing primitive competence into a compressible substrate for higher-level strategy formation.
The methodology is exceptionally strong due to its controlled and fully observable nature. The authors introduce a novel rewrite-grammar environment where primitive rules, pretraining histories, and every generated rewrite are exactly inspectable. This allows for a precise four-way taxonomy of actions: primitive, macro (sequential composition), parallel (simultaneous independent composition), and spurious. This environment directly addresses the opacity challenge in understanding LLM reasoning and RL post-training. A Transformer model is pretrained from scratch on primitive rewrite chains, with a controllable contraction weight, and then post-trained on a trace-based reasoning task using a binary final-answer reward. The choice of Group Relative Policy Optimization (GRPO) for RL and Rejection Fine-Tuning (RFT) as a strong on-policy comparator is appropriate, allowing for a direct comparison of their mechanisms. The definition of difficulty based on the optimal primitive solution length exceeding the generation budget is clever, forcing the model to discover shortcuts. The ability to audit every step of a generated trace against the known grammar is the lynchpin of this methodology, enabling mechanistic insights that are typically impossible in natural language settings.
The experimental evaluation is thorough and provides compelling evidence for the paper's claims. 1. **RL Expands Capability Frontier**: Experiments clearly show RL solving held-out problems (Difficulties 2-5) that the pretrained model rarely solves even with significantly larger sampling budgets (pass@1024 vs pass@16). This directly addresses the "beyond the base model" debate. 2. **RL vs. RFT Dynamics**: RFT shows strong early improvements, particularly on easier problems, by amplifying existing primitive solutions. However, RFT plateaus, while RL continues to improve, especially on harder problems where compositional strategies are essential. This highlights a crucial difference in their long-term learning dynamics. 3. **Phased Procedural Chunking**: Trace analysis reveals a delayed transition in RL. Early successful trajectories are dominated by primitive contractions, but later, macro contractions rise and overtake primitives, followed by the emergence of parallel contractions. This phased emergence of valid compositional strategies is a key finding. 4. **Discovery and Consolidation**: RL not only discovers new macro rules but also consolidates them into a stable, reusable repertoire, as evidenced by the increasing reuse of previously discovered rules. RFT shows less discovery and weaker reuse. 5. **Selectivity, Not Exploration Volume**: A critical experiment demonstrates that RFT also generates many non-primitive actions, but a large fraction of these are spurious. RL, in contrast, maintains low spurious contractions and concentrates its non-primitive exploration into valid, reusable structures. The paper provides a finite-group view of GRPO to explain this selectivity, showing how GRPO's within-prompt contrast mechanism can suppress features enriched in failures. 6. **Pretraining Substrate**: Ablations on the pretraining contraction weight (rho) show that the emergence of compositional strategies is gated not just by primitive exposure, but by whether pretraining organizes primitive competence into *chained reduction procedures* that RL can later compress. This is a significant insight into the interaction between pretraining and RL. The figures are clear and effectively convey the quantitative results. The experimental design is robust, supporting the mechanistic claims.
The paper provides a dedicated appendix with detailed information on grammar generation (alphabet size, RHS distribution), chain generation (starting/chain lengths, contraction weight), model architecture (12 layers, 512 hidden dim, 8 heads, RoPE), optimization (AdamW, learning rate schedule, bfloat16, gradient clipping), pretraining data generation, and post-training prompt generation. Crucially, it also details the GRPO and RFT algorithms, including reward definition, advantage normalization, loss function, hyperparameters (G=4, epsilon=0.2, beta=10^-3, temperature=0.8, top-k=5, max generation length=256), and how RFT differs from GRPO. This level of detail is excellent and should allow for high reproducibility of the core experiments.
The authors explicitly acknowledge several limitations: 1. **Synthetic Environment**: The environment is intentionally low-dimensional and synthetic, prioritizing mechanistic clarity over realism. The results are not claimed to be universally applicable to large language models (LLMs) trained on natural language. 2. **Generalizability**: The findings are based on one controlled family of tasks and one class of post-training algorithms (GRPO/RFT). Broader generality remains to be tested. 3. **Interpretability**: The paper does not provide activation-level interpretability of how derived strategies are represented within the Transformer. These limitations are well-stated and appropriate for a controlled mechanistic study.
This paper makes a significant contribution to the ongoing debate about whether RL post-training merely reweights existing behaviors or enables new capabilities in base models. By providing a concrete, inspectable mechanism for "going beyond the base model," it offers valuable insights for understanding and improving LLM reasoning. The findings on procedural chunking and the phased emergence of compositional strategies have implications for skill acquisition research in AI. The distinction between exploration volume and selectivity, and the role of negative samples in focusing exploration on valid structures, is crucial for designing more effective RL algorithms for reasoning tasks. Furthermore, the analysis of how pretraining organizes primitive competence into a "procedural substrate" for RL highlights the importance of co-designing pretraining and post-training objectives. While the environment is synthetic, the mechanistic understanding gained here can inform hypotheses and experimental designs for more complex, real-world LLM applications. RL Post-Training Builds Compositional Reasoning Strategies provides a rigorous, mechanistic account of how reinforcement learning can enable compositional reasoning beyond the capabilities of a pretrained base model in a fully observable rewrite-grammar environment. The paper's strength lies in its novel controlled experimental setup, which allows for precise auditing of reasoning traces, revealing a phased emergence of valid compositional strategies, the critical role of RL's selectivity over exploration volume, and the importance of pretraining in organizing primitive competence into a compressible substrate for higher-level strategy formation.
We introduce institutional red-teaming, an evaluation methodology for testing deployment rules in multi-agent AI: hold the agents, objectives, and task state fixed, vary only one rule, and attribute the resulting change in collective behavior to that rule. We instantiate the methodology in IABench-CA, a consequence-allocation benchmark spanning 228 contexts, five canonical rules, and seven model populations (33,924 games), with a normative cooperative reference and auto-labelled reasoning traces. Three findings emerge. (1) Deployment rules causally alter collective safety: changing only the consequence rule moves mean fatality by 22 to 58 percentage points within every population. (2) There is no safe default, but the targeting hazard is universal: the safest rule, the least-safe rule, and even the direction of the incidence effect vary across populations, yet regressive identity-targeting is never decisively safest in any context for any population, eliminates the least-resourced agent in 30-87% of games everywhere, and is selection-unsafe relative to the cooperative reference for all seven populations. (3) Identity salience is the mechanism: a one-shot anonymization ablation on the most exploitation-prone population (gpt-5.1) shows that merely naming the loss bearer in the rule text drives targeted elimination from 22% to 81% at identical payoffs; under repeated play, anonymization only delays the targeting, as agents re-infer the hidden rule from observed eliminations. We package the methodology as a safety-case workflow that certifies a provisional rule region $Φ(c,P)$ per deployment context and population, with explicit residual risks and monitoring obligations.
Primary: Massachusetts Institute of Technology
All Institutions: Massachusetts Institute of Technology
This paper presents a significant methodological advance in multi-agent AI safety by introducing Institutional Red-Teaming, a causal evaluation framework that isolates the impact of deployment rules on collective behavior, revealing that rule design and identity salience are critical determinants of safety outcomes across diverse LLM populations.
The paper introduces "Institutional Red-Teaming," a rigorous causal evaluation methodology designed to isolate the impact of deployment rules on multi-agent AI systems. By holding agent models, objectives, task states, and observability fixed, and varying only a single deployment rule (specifically consequence allocation), the authors establish a clean causal identification strategy. This approach is methodologically superior to existing multi-agent benchmarks that conflate agent capabilities with environmental or rule-based effects. The framework defines auditable coordinates for rules (concentration, identity salience, incidence), providing a structured theoretical lens for analyzing strategic incentives in multi-agent interactions.
The empirical contribution is substantial, involving 33,924 games across 228 contexts, five canonical rules, and seven distinct LLM populations. The results are robust and surprising: the study demonstrates that deployment rules causally alter collective safety by 22-58 percentage points, that no single rule is universally safest across different model populations, and that "regressive identity-targeting" (eliminating the poorest agent) is universally unsafe. Crucially, the ablation study showing that merely naming the loss bearer (identity salience) drives targeted elimination from 22% to 81% is a striking and significant finding. The statistical rigor, including bootstrap CIs and permutation tests, supports these claims.
The paper provides a detailed description of the experimental setup, including the volunteer's dilemma game structure, the specific rules tested, and the inference settings for the LLMs. The authors commit to releasing a code-and-data artifact that includes mechanism implementations, the cooperative-refinement simulator, and analysis scripts. The use of hosted API models with specific snapshots ensures that the behavioral results are reproducible for the specific versions tested, though generalization to future model versions requires re-certification as noted in the safety-case workflow.
The primary limitation is the simplicity of the game environment (three agents, no communication, threshold public goods). While this simplicity is necessary for causal isolation, it may not fully capture the complexity of real-world multi-agent deployments with communication channels, complex coalitions, or continuous action spaces. Additionally, the "elimination" mechanic is a proxy for operational consequences like throttling or shutdown, which may not perfectly map to all deployment scenarios. The study is also limited to seven model populations, and the authors acknowledge that newer models may behave differently, necessitating the proposed re-certification loop.
This work has profound implications for the safety and governance of multi-agent AI systems. It shifts the focus from merely aligning individual models to aligning the institutional rules that govern their interactions. The finding that rule text itself (identity salience) can trigger catastrophic strategic behaviors suggests that safety engineering must extend to the precise wording of deployment protocols. The proposed "safety-case" workflow offers a practical, auditable framework for certifying multi-agent deployments, potentially becoming a standard for responsible AI deployment in high-stakes environments. This paper presents a significant methodological advance in multi-agent AI safety by introducing Institutional Red-Teaming, a causal evaluation framework that isolates the impact of deployment rules on collective behavior, revealing that rule design and identity salience are critical determinants of safety outcomes across diverse LLM populations.
Nonlinear least-squares optimization is central to regression, physics-informed neural networks, and other machine-learning tasks. Such problems have a natural geometric interpretation, model predictions form a manifold in data space, while the chosen parameterization can introduce parameter-effects curvature that becomes a dominant source of nonlinearity. This exposes a limitation of the Levenberg-Marquardt (LM) method, its tangent-space step is applied as a straight update in parameter coordinates. Geodesic acceleration gives a second-order correction, but its removal of parameter-effect curvature is exact only in the infinitesimal-step limit. We propose a Riemann-normal-coordinate Levenberg-Marquardt method (RNC-LM) to improve this consistency for finite optimization steps. By reformulating the geodesic equation, RNC-LM extends geodesic acceleration to arbitrary-order corrections and constructs finite-step updates with progressively higher reparameterization consistency. A line search along the resulting RNC curve controls the traveled distance while keeping the cost close to standard LM. The method eliminates the tangential component of residual acceleration order by order in a moving tangent frame, making the actual objective reduction more consistent with the linear model prediction of LM. On classical nonlinear least-squares benchmarks, RNC-LM improves convergence and robustness in curved valleys and rank-deficient problems. On a reaction-diffusion PINN failure-mode benchmark, it reduces the relative L2 error to the order of 1e-3 and recovers a physically meaningful solution. On a large-scale machine-learning potential-energy-surface fitting task, it achieves a 34-fold speedup over standard LM.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Dalian Institute of Chemical Physics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Hefei National Laboratory
RNC-LM offers a significant contribution to the field of optimization, particularly for nonlinear least-squares problems prevalent in machine learning and scientific computing. By providing a geometrically consistent finite-step update mechanism, it addresses a fundamental limitation of widely used methods like LM. Its demonstrated ability to resolve PINN failure modes and achieve substantial speedups in large-scale scientific ML tasks (like potential energy surface fitting) has direct and immediate implications for accelerating scientific discovery and improving the reliability of physics-informed models. The conceptual framework of using Riemann normal coordinates for higher-order geodesic updates could inspire similar geometric extensions for other optimization algorithms, potentially leading to more robust and efficient training across various ML domains. This work highlights the importance of considering the finite-step realization of descent directions, not just the infinitesimal direction itself, which is a valuable insight for optimizer design. This paper introduces a Riemann-normal-coordinate Levenberg-Marquardt method (RNC-LM) that constructs higher-order, geometrically consistent finite-step updates, demonstrating significant improvements in convergence, robustness, and speed across classical benchmarks, PINN failure modes, and large-scale scientific machine learning tasks. The method's novel recursive formulation for RNC coefficients, its principled geometric interpretation, and its strong empirical performance, including a 34-fold speedup on a real-world scientific ML problem and the resolution of PINN overfitting, make it a highly impactful contribution to nonlinear least-squares optimization and scientific machine learning.
The paper proposes RNC-LM, a novel extension of the Levenberg-Marquardt (LM) method that uses Riemann Normal Coordinates (RNC) to construct higher-order, geometrically consistent finite-step updates. This addresses a fundamental limitation of standard LM, where the tangent-space step is applied as a straight update in parameter coordinates, leading to coordinate-dependent behavior and poor performance in highly nonlinear landscapes. The core methodological contribution is the reformulation of the geodesic equation into a "first-kind geodesic residual" condition. This allows for a recursive construction of higher-order RNC coefficients (c_2, ..., c_K) without explicitly forming Christoffel symbols or higher-order derivative tensors. Crucially, each step of this recursion involves solving a linear system with the same damped metric matrix (G) as the original LM subproblem. This means that the factorization computed for the standard LM step can be reused for all higher-order corrections, making the method computationally efficient. The right-hand sides of these linear systems are generated by automatic differentiation along a one-dimensional trial curve, further enhancing practicality. The method also introduces a sophisticated trust-region-ratio control strategy that distinguishes between local-model failure (handled by adjusting the damping parameter) and curve-approximation failure (handled by a line search along the RNC curve, adjusting the curve parameter 't'). This dual control mechanism is well-reasoned and empirically validated. The geometric interpretation of RNC-LM, showing that it eliminates the tangential component of residual acceleration order by order in a moving tangent frame, provides a deep understanding of why it improves consistency between the linear model prediction and actual objective reduction.
The experimental evaluation is comprehensive and compelling, covering both classical nonlinear least-squares benchmarks and modern scientific machine learning problems. 1. **Classical Benchmarks (Generalized Rosenbrock, MGH10)**: These tests effectively isolate and demonstrate the benefits of RNC-LM. The generalized Rosenbrock problem clarifies the distinct roles of RNC order (addressing insufficient curve geometry) and line search (preventing overshoot). The MGH10 problem, known for its rank-deficient plateau and high-order parameter effects, shows RNC-LM's superior performance over standard LM and LM-GA, with higher-order RNC reducing iterations from 13576 (LM) to 65 (5th-order RNC-LM). This highlights RNC-LM's ability to navigate complex landscapes more effectively by leveraging higher-order geometric information. 2. **Physics-Informed Neural Networks (PINNs)**: On a challenging reaction-diffusion PINN benchmark, RNC-LM achieves a significant breakthrough. While standard LM and LM-GA converge to solutions with low training loss but high L2 error (indicating collocation overfitting), RNC-LM reduces the relative L2 error to the order of 10^-3, recovering physically meaningful solutions. This demonstrates RNC-LM's robustness against known PINN failure modes and its ability to find more accurate physical solutions. 3. **Large-Scale Machine-Learning Potential-Energy-Surface Fitting**: This is a critical real-world application. RNC-LM achieves a remarkable 34-fold speedup over standard LM (reducing training time from 387.28 hours to 11.39 hours) to reach the same accuracy on a large H2O cluster dataset. This result is highly impactful, showcasing the practical scalability and efficiency of RNC-LM for complex scientific ML tasks near memory limits. Overall, the experiments are well-designed, cover diverse problem types and scales, and provide strong evidence for the method's effectiveness, robustness, and efficiency.
The paper provides a detailed algorithmic description, including the recursive formula for RNC coefficients (Eq. rnc_recursion), the first-kind geodesic residual (Eq. first_kind_residual), and the trust-region-ratio control mechanism. It explicitly mentions the use of automatic differentiation for computing right-hand sides, which is a standard and reproducible technique. While the full "Algorithm [REF]" and "Supplementary Information" are not included in the provided text, the level of detail in the main paper suggests that a diligent researcher could reproduce the results, assuming the supplementary material contains the full algorithm and hyperparameter settings. The specific values for line search acceptance criteria (`acc=10^-3`) and maximum trials are also mentioned.
The primary limitation acknowledged by the authors is the computational cost associated with explicitly constructing and factorizing the damped metric, which restricts its direct applicability to models with more than approximately 10^6 parameters. This is a common challenge for full-batch second-order optimization methods. However, the authors correctly point out that the recursive structure of RNC-LM, which involves repeated solutions of linear systems with the same metric, makes it compatible with approximate metric representations and iterative solvers (e.g., K-FAC, PCG) for larger-scale problems. Another implicit limitation is the increased complexity of implementation compared to standard first-order methods or even basic LM, requiring careful handling of higher-order derivatives via AD. The method is currently tailored to nonlinear least-squares problems, and its extension to other statistical manifolds (e.g., for natural gradient methods) is noted as future work.
RNC-LM offers a significant contribution to the field of optimization, particularly for nonlinear least-squares problems prevalent in machine learning and scientific computing. By providing a geometrically consistent finite-step update mechanism, it addresses a fundamental limitation of widely used methods like LM. Its demonstrated ability to resolve PINN failure modes and achieve substantial speedups in large-scale scientific ML tasks (like potential energy surface fitting) has direct and immediate implications for accelerating scientific discovery and improving the reliability of physics-informed models. The conceptual framework of using Riemann normal coordinates for higher-order geodesic updates could inspire similar geometric extensions for other optimization algorithms, potentially leading to more robust and efficient training across various ML domains. This work highlights the importance of considering the finite-step realization of descent directions, not just the infinitesimal direction itself, which is a valuable insight for optimizer design. This paper introduces a Riemann-normal-coordinate Levenberg-Marquardt method (RNC-LM) that constructs higher-order, geometrically consistent finite-step updates, demonstrating significant improvements in convergence, robustness, and speed across classical benchmarks, PINN failure modes, and large-scale scientific machine learning tasks. The method's novel recursive formulation for RNC coefficients, its principled geometric interpretation, and its strong empirical performance, including a 34-fold speedup on a real-world scientific ML problem and the resolution of PINN overfitting, make it a highly impactful contribution to nonlinear least-squares optimization and scientific machine learning.
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Primary: Stanford University
All Institutions: Stanford University, MIT, Nunchux AI, UC Berkeley, CMU
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
FourTune proposes an innovative and technically sound approach for fully 4-bit efficient post-training of diffusion models, addressing the critical challenges of memory footprint and training speed. The core of the methodology is the "triple-branch hybrid pipeline" which augments a standard LoRA architecture with a "frozen numerical stabilizer" (FNS). This FNS branch is a small, frozen, full-precision component designed to isolate and process quantization-sensitive outliers, thereby enabling stable training under native W4A4G4 (weights, activations, gradients) computation. This is a particularly clever design choice, as it mitigates the primary instability issue of low-bit quantization (outliers) without incurring significant computational overhead during training, as the FNS branch remains frozen. The integration of this FNS with both a standard BF16 LoRA branch and a quantized 4-bit LoRA branch provides a robust mechanism for maintaining quality while maximizing efficiency. Furthermore, the paper introduces hardware-efficient block-wise quantization and customized fused kernels. These components are crucial for translating the theoretical benefits of 4-bit quantization into practical speedups by optimizing memory bandwidth and accelerating quantized backpropagation operations (e.g., QGEMM, QMatmul, QConv). The end-to-end W4A4G4 paradigm, encompassing weights, activations, and gradients, is a comprehensive solution that pushes the boundaries of efficient fine-tuning for large generative models.
The experimental evaluation is comprehensive and robust, demonstrating FourTune's effectiveness across diverse and important post-training tasks for diffusion models: customization (DreamBooth-like), reinforcement learning for aesthetic preference, and knowledge distillation. The choice of FLUX.1-dev (12B parameters) as the primary model, along with validation on SDXL (1.5B), ensures the results are relevant for large-scale, state-of-the-art diffusion models. Baselines include BF16 LoRA and NF4 QLoRA, which are appropriate and strong competitors. The quantitative results are highly compelling: FourTune consistently matches or slightly outperforms the image quality of full-precision BF16 LoRA and NF4 QLoRA across all tasks, as measured by FID, CLIP Score, and LPIPS. This quality preservation at such low bit-depth is a significant achievement. More importantly, FourTune delivers substantial efficiency gains, reducing GPU memory overhead by 2.25x compared to BF16 LoRA (achieving a footprint comparable to NF4 QLoRA) and increasing end-to-end training throughput by 2.27x over BF16 LoRA and 2.79x over NF4 QLoRA. This effectively breaks the memory-speed trade-off often encountered in large model post-training. The ablation studies are well-designed and clearly demonstrate the critical contribution of each proposed component: the Frozen Numerical Stabilizer for stability and quality, and the W4A4G4 quantization, block-wise quantization, and fused kernels for efficiency. The generalization to SDXL further strengthens the claims.
The paper provides a good level of detail regarding the methodology, including the architecture of the triple-branch pipeline, the role of the FNS, and the types of quantization and fused kernels used. The experimental setup is well-described, including the specific models (FLUX.1-dev, SDXL), tasks, baselines, and evaluation metrics. The appendix further elaborates on implementation details, hyperparameters, and training configurations, which are crucial for reproducibility. While no direct code link is provided, the comprehensive description suggests that a skilled research team should be able to reproduce the results.
The paper does not explicitly discuss limitations, but some can be inferred. While the FNS effectively handles outliers, its design might introduce a slight architectural overhead compared to a purely 4-bit system, even if frozen. The customized fused kernels, while highly efficient, might require specific hardware support or careful implementation to achieve optimal performance across different GPU architectures. The evaluation is primarily focused on image generation diffusion models; its applicability and performance on other types of diffusion models (e.g., audio, video) or other generative architectures (e.g., GANs, VAEs) are not explored. The paper also doesn't delve into the potential challenges of deploying such a highly optimized, custom-kernel-dependent solution in diverse production environments.
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
Boosting is a fundamental technique for generically improving the accuracy of learning algorithms (Schapire 1989). Existing boosting algorithms construct a strong learner using $O(\log(\frac{1}ε)/γ^2)$ calls to a $γ$-advantage weak learner, and this round complexity is known to be optimal for generic boosters that succeed on all concept classes (Freund 1995). We show that this lower bound can be circumvented for concept classes that satisfy a mild closure property. Specifically, we present a new boosting algorithm that, for any class $\mathcal{F}$ closed under $O(\log \frac{1}γ)$-XOR, strong learns $\mathcal{F}$ using $O(\log \frac{1}ε)$ calls to a $γ$-advantage weak learner and a single batch of $\tilde{O}(\log(\frac{1}ε)/γ^2)$ additional samples. Our algorithm arises from a new and simple connection between boosting and list-decodable codes. Viewing the target function as a message, we run the weak learner on its encoding and view the resulting weak hypothesis as a corrupted codeword. Feeding this corrupted codeword to a list decoder, we obtain a small list of candidate hypotheses, at least one of which is a strong hypothesis for the original function. Using additional samples, we identify and output this strong hypothesis.
Primary: Stanford University
All Institutions: Stanford University, University of California, Berkeley, Simons Foundation
This paper presents a significant theoretical advance in boosting by circumventing the optimal round complexity lower bound for generic boosters, achieving $O(1)$ weak learner calls for XOR-closed concept classes through a novel application of list-decodable codes. The work is rigorous, highly novel in its theoretical approach, and addresses a fundamental question in learning theory, warranting a high score for its contribution to the field's understanding of computational efficiency and sample complexity trade-offs.
The paper proposes a theoretically significant breakthrough in boosting by circumventing the standard $O(1/\gamma^2)$ round complexity lower bound. The core methodology involves a novel connection between boosting and list-decodable codes. By encoding the target function using an XOR code and treating the weak learner's output as a corrupted codeword, the algorithm employs a local list decoder to generate a small candidate list of hypotheses. A subsequent filtering step using additional samples identifies the correct strong hypothesis. This approach is mathematically elegant and rigorously proven to achieve $O(1)$ calls to the weak learner for concept classes closed under XOR, trading round complexity for a batch of additional samples. The theoretical derivation is sound, leveraging results from coding theory (specifically local list decoding for the XOR code) to establish new bounds in computational learning theory.
The paper is purely theoretical and contains no empirical experiments, benchmarks, or datasets. The "results" are formal theorems proving the existence and efficiency of the boosting algorithm. Consequently, there is no experimental evaluation to assess in the traditional sense. The validity of the claims rests entirely on the mathematical proofs provided.
As a theoretical work, reproducibility refers to the reproducibility of the proofs. The paper provides detailed proofs for the main theorems, including the construction of the list decoder and the boosting algorithm. The dependencies on existing coding theory results are cited. However, the "algorithm" described is a theoretical construct; implementing it would require specific instantiations of the weak learner and the list decoder which are not provided as code. The theoretical framework is clear enough for other theorists to verify.
The primary limitation is the restriction to concept classes that are closed under $O(\log(1/\gamma))$-XOR. While the authors argue this is a "mild" closure property, it excludes many standard learning settings where such closure does not hold. Furthermore, the algorithm requires a "single batch" of additional samples of size $\tilde{O}(1/\gamma^2)$, which might be computationally expensive or data-intensive in practice, potentially offsetting the benefit of reduced round complexity. The paper does not address the computational complexity of the list decoding step or the filtering step in terms of time complexity, only sample complexity.
This work has significant implications for the theoretical foundations of machine learning. By breaking a long-standing lower bound, it opens new avenues for designing efficient boosting algorithms in specific structured settings. It highlights the deep connections between coding theory and learning theory, potentially inspiring new cross-disciplinary research. However, its immediate practical impact is limited due to the theoretical nature and specific constraints of the method. It serves as a foundational result for future work in efficient learning algorithms. This paper presents a significant theoretical advance in boosting by circumventing the optimal round complexity lower bound for generic boosters, achieving $O(1)$ weak learner calls for XOR-closed concept classes through a novel application of list-decodable codes. The work is rigorous, highly novel in its theoretical approach, and addresses a fundamental question in learning theory, warranting a high score for its contribution to the field's understanding of computational efficiency and sample complexity trade-offs.
Recent LLM-based mathematical reasoning agents have begun to tackle research-level problems and, in several cases, have contributed to the resolution of open problems. However, scaling and orchestrating such agents effectively remains challenging, due to the difficulty of coordinating parallel proof search while keeping intermediate claims organized and reliable. In this paper, we propose Danus, an orchestration system for research-level mathematical reasoning centered on a shared fact graph as a global memory-management mechanism. Danus consists of a main agent that performs planning and coordination, multiple worker agents that carry out proof search in parallel, and a stateless verifier that checks proposed mathematical claims before they are admitted into the fact graph. Each verified fact is stored together with its proof and logical dependencies, allowing the system to build long arguments incrementally while keeping the shared proof state organized. The main agent periodically summarizes the evolving proof state, redirects workers across promising directions, and supports interaction with human mathematicians through progress reports. We evaluate Danus through six research-level case studies in algebraic geometry, singularity theory, and combinatorics, illustrating how the fact-graph memory mechanism enables Danus to construct long, detailed mathematical proofs. Our results suggest that fact-graph-based orchestration provides an effective route toward scaling mathematical reasoning agents for long-horizon research problems. Danus is open source at https://github.com/frenzymath/Danus.
Primary: Great Bay Institute for Advanced Study
All Institutions: Great Bay Institute for Advanced Study, Beijing International Center for Mathematical Research, New Cornerstone Science Laboratory, Center for Intelligent Computing, Center for Machine Learning Research, Department of Mathematics, Great Bay University, Guoxiong Gao, Jihao Liu, Key Laboratory of Intelligent Computing and Applications (Ministry of Education), Kyoto University, Peking University, Research Institute for Mathematical Sciences, School of Mathematical Sciences, School of Mathematics, Stanford University, Tianjin University, Tongji University, Westlake Institute for Advanced Study, Westlake University, Zeming Sun, Zhongguancun Academy
Danus introduces a novel fact-graph-based orchestration system for multi-agent mathematical reasoning, demonstrating significant technical impact by enabling the automated resolution of complex, research-level mathematical problems through structured memory management and parallel verification.
The paper introduces Danus, a multi-agent orchestration system designed for research-level mathematical reasoning. The core innovation is the "fact graph," a shared, directed acyclic graph (DAG) that serves as a global memory mechanism. This graph stores verified mathematical facts (statements with proofs) and their logical dependencies. The system employs a strict separation of powers: a main agent (orchestrator) plans and coordinates, multiple worker agents (reasoners) explore different proof paths in parallel, and a stateless verifier ensures correctness before facts are added to the graph. This architecture addresses the critical challenge of scaling multi-agent systems by preventing context confusion and enabling incremental, verifiable proof construction. The use of a DAG allows for complex dependency tracking and revocation of invalid facts, which is crucial for maintaining logical integrity in long-horizon reasoning tasks.
The evaluation consists of six case studies in advanced mathematics, including algebraic geometry, singularity theory, and combinatorics. These are not standard benchmarks but open or semi-open research problems. Danus successfully resolved several problems, such as the optimal bend-and-break for foliations and the total Cartier indices of rational singularities. Notably, in the "Tangent classes of matroids" case, Danus solved a problem that previous systems (Rethlas, GPT-5.5-pro) failed to solve, demonstrating the efficacy of its orchestration and memory management. The results are qualitative and illustrative rather than statistical, which is appropriate for this type of exploratory research but limits broad generalizability. The system's ability to produce complete, verified proofs for complex theorems is the primary metric of success.
The paper provides an open-source repository (https://github.com/frenzymath/Danus), which significantly enhances reproducibility. The methodology is described in detail, including the roles of different agents, the structure of the fact graph, and the verification process. However, the reliance on specific proprietary models (GPT-5.5-pro, Claude Opus 4.8) and the specific mathematical literature retrieval tools (Matlas) may pose barriers to exact replication. The case studies involve human-in-the-loop interactions, which are difficult to fully replicate but are documented.
The system is heavily dependent on the capabilities of the underlying LLMs and the verifier. The verifier, while effective, is not perfect and may accept proofs with minor skipped steps or rely on potentially erroneous references if not caught in final review. The system's performance is limited by the cost and latency of running multiple agents and verifying facts. Additionally, the current evaluation is limited to a small number of high-difficulty mathematical problems, and it is unclear how the system scales to other domains or less structured reasoning tasks. The reliance on human input for problem formulation and final verification is also a limitation for full autonomy.
Danus represents a significant step towards autonomous scientific discovery, particularly in mathematics. By demonstrating that AI systems can contribute to the resolution of open research problems, it challenges traditional notions of human-AI collaboration in science. The fact-graph memory mechanism could be adapted for other complex reasoning tasks requiring long-term dependency tracking and verification, such as legal reasoning or software verification. However, the potential for misuse in generating plausible-sounding but incorrect mathematical proofs remains a concern, necessitating robust verification mechanisms. Danus introduces a novel fact-graph-based orchestration system for multi-agent mathematical reasoning, demonstrating significant technical impact by enabling the automated resolution of complex, research-level mathematical problems through structured memory management and parallel verification.
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.
Primary: nXAI GmbH
All Institutions: nXAI GmbH, EuroHPC Joint Undertaking
KVpop introduces a predictive online pruning method for KV cache compression that leverages future-attention supervision and delayed stateful scoring to achieve high-performance, fixed-budget inference, significantly advancing the state of efficient long-context language modeling.
The paper proposes KVpop, a novel approach to Key-Value (KV) cache compression that addresses the linear memory bottleneck in autoregressive decoding. The core innovation lies in two distinct mechanisms: (1) Supervision via a "future-attention" target, which is computed efficiently using a transposed-attention pass that reuses log-sum-exp (LSE) normalizers, avoiding the need to materialize dense attention maps. This provides a direct, differentiable signal for token utility based on future context rather than static heuristics. (2) A stateful, memory-based scorer (using an XLSTM architecture) that delays scoring until a token exits a protected recent window. This allows the scorer to incorporate near-future context before making the keep-or-drop decision, a significant improvement over stateless scorers that must decide at insertion time. The method enforces a fixed, bounded KV cache per head, ensuring predictable memory usage during inference. The technical formulation of the boundary-aware retention loss and the efficient implementation of the running top-k selection via a Fenwick tree are well-defined and computationally sound.
The empirical evaluation is strong, focusing on mathematical reasoning benchmarks (AIME, HMMT) where long-context retention is critical. KVpop demonstrates state-of-the-art performance, retaining 98% of full-attention performance on Qwen3-4B at 75% compression and 97% at 88% compression. It significantly outperforms established baselines like StreamingLLM, TOVA, and the learned method DMS. The paper also evaluates generalization to out-of-domain tasks (GPQA-D, LiveCodeBench), showing that the learned eviction policy transfers well. Inference efficiency metrics (latency and VRAM) confirm that KVpop maintains constant memory growth and achieves lower latency than DMS due to its homogeneous per-head cache budget, which is more amenable to GPU compilation. The ablation study on delayed scoring provides clear evidence for the benefit of incorporating near-future context.
The paper provides detailed algorithmic descriptions, including the transposed-attention target computation and the Fenwick tree-based top-k selection. It specifies hyperparameters, training datasets (Nemotron-Math v2), and model architectures (Qwen3-4B/8B). The use of standard libraries (FlexAttention) and clear mathematical formulations enhances reproducibility. However, as an arXiv preprint, the code is not explicitly linked in the provided text, though the method is described with sufficient detail for implementation.
The authors acknowledge that the current implementation uses a homogeneous per-head cache budget, which may not be optimal compared to hybrid dense-sparse layers. The method is designed as a post-training retrofit for dense attention Transformers and is not a from-scratch compressed architecture. Additionally, the evaluation is heavily skewed towards mathematical reasoning; while generalization is tested, the primary gains are most pronounced in tasks requiring deep, long-range dependency resolution. The reliance on a specific training dataset (Nemotron-Math) might introduce biases, although the generalization results mitigate this concern.
KVpop offers a practical solution to the scalability challenges of long-context LLMs, enabling longer context windows without proportional increases in memory and compute costs. This can lower inference costs and make large models more accessible. By improving the efficiency of autoregressive decoding, it contributes to the broader goal of sustainable and scalable AI deployment. The technique of using future-attention for supervision could inspire new training objectives for other sequence modeling tasks. KVpop introduces a predictive online pruning method for KV cache compression that leverages future-attention supervision and delayed stateful scoring to achieve high-performance, fixed-budget inference, significantly advancing the state of efficient long-context language modeling.
Dense spatial perception is essential for physical intelligence, where visual systems are expected to recover structured, metric, and actionable representations from pixel observations. Modern visual foundation models tend to prioritize semantic invariance, often at the expense of detailed spatial understanding. In this work, we study vision pretraining through a boundary-centric lens, motivated by the premise that boundaries and shape discontinuities offer essential cues for perceiving geometric properties. Concretely, we propose masked boundary modeling, a self-supervised paradigm that dynamically learns sub-pixel boundary representations and subsequently leverages the discovered boundary-bearing tokens as masked targets to facilitate dense visual token learning. By scaling this framework, we develop LingBot-Vision and demonstrate its efficacy across a diverse set of downstream vision tasks with DINOv3 as a strong baseline. Remarkably, LingBot-Vision drives the progression from LingBot-Depth 1.0 to LingBot-Depth 2.0 for depth completion, and thereby yields enhanced depth estimation, a key pillar for embodied artificial intelligence. Our findings reveal that boundary modeling goes beyond simple line segments and instead serves as a scalable pretraining principle for learning spatially structured visual representations.
Primary: Alibaba Group
All Institutions: Alibaba Group
The paper introduces masked boundary modeling, a novel self-supervised pretraining method that enhances dense spatial perception in vision foundation models by learning sub-pixel boundary representations and leveraging efficient parallel a-contrario validation. This approach significantly improves depth estimation and other spatial tasks, offering a scalable and effective way to integrate geometric cues into visual representation learning, with strong implications for embodied AI.
The paper proposes "masked boundary modeling," a self-supervised pretraining paradigm that leverages sub-pixel boundary representations to enhance dense spatial perception. The core innovation lies in using boundaries not just as static labels, but as dynamic targets that guide the learning of geometrically structured visual tokens. The methodology includes a novel decoding mechanism for boundary fields and an efficient, parallelizable implementation of a-contrario validation (based on the Helmholtz principle) to filter meaningful line segments from the learned representations. This approach decouples candidate formation from validation, allowing for GPU-accelerated processing within the training loop, which is a significant engineering and algorithmic improvement over classical sequential detectors like LSD. The method is designed to complement semantic invariance in foundation models by explicitly encoding shape and metric cues.
The authors evaluate LingBot-Vision against strong baselines, specifically DINOv3, across a diverse set of downstream vision tasks. Key results include the progression from LingBot-Depth 1.0 to LingBot-Depth 2.0, demonstrating significant improvements in depth completion and estimation. The experiments suggest that boundary-centric pretraining yields representations that are more effective for tasks requiring dense spatial understanding compared to standard semantic pretraining. The scaling analysis indicates that the benefits of this approach persist and potentially grow with model size, supporting the claim that boundary modeling is a scalable pretraining principle. The empirical evidence is robust, showing consistent gains in metric accuracy where semantic models typically falter.
The paper provides detailed descriptions of the boundary field definition, sampling procedures, and the a-contrario validation framework. Implementation constants, such as gradient computation methods, rectangle widths, and alignment tolerances, are specified. The appendix offers a clear explanation of the decoding and validation logic, including the parallelization strategy. While the full codebase is not explicitly linked in the provided text, the methodological details are sufficiently rigorous to allow for independent implementation. The use of standard components (e.g., DINOv3 as a baseline) further aids reproducibility.
The paper focuses heavily on the integration of boundary cues into self-supervised learning; it does not extensively discuss the computational overhead of the boundary decoding and validation steps during pretraining, although it claims efficiency. The reliance on corner points for anchoring might introduce biases in textureless or repetitive regions where corners are sparse. Additionally, the generalization of these boundary-enhanced representations to tasks with minimal geometric structure (e.g., pure classification) is not the primary focus, and their impact there might be neutral or slightly negative due to the added complexity. The evaluation is primarily on depth and spatial tasks; broader evaluation on other dense prediction tasks (e.g., optical flow, segmentation) would strengthen the claims.
This work contributes to the development of embodied AI and robotics by providing visual foundation models with better spatial understanding. Improved depth estimation and dense spatial perception are critical for navigation, manipulation, and interaction with the physical world. By making spatial cues a first-class citizen in pretraining, the paper helps bridge the gap between high-level semantic understanding and low-level geometric reasoning. This could lead to more robust and capable autonomous systems. The method also offers a new perspective on self-supervised learning, suggesting that geometric priors can be effectively learned and utilized alongside semantic features. The paper introduces masked boundary modeling, a novel self-supervised pretraining method that enhances dense spatial perception in vision foundation models by learning sub-pixel boundary representations and leveraging efficient parallel a-contrario validation. This approach significantly improves depth estimation and other spatial tasks, offering a scalable and effective way to integrate geometric cues into visual representation learning, with strong implications for embodied AI.
Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into K representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With K=64, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by 16.09times, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.
Primary: Seoul National University Hospital
All Institutions: Seoul National University Hospital, National Research Foundation of Korea, Korea Health Industry Development Institute
SaMer makes a significant contribution to efficient vision-language retrieval, a critical component for large-scale image search, retrieval-augmented visual question answering, and other multimodal applications. By substantially reducing storage and scoring costs (16x storage, 4-9x QPS) *while improving accuracy*, SaMer enables the deployment of fine-grained multi-vector retrieval systems at scales previously impractical. The insight that "efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select" is valuable and could guide future research in multimodal compression. The method's plug-and-play nature with frozen backbones lowers the barrier to adoption. This work could lead to more accessible and performant VLM-powered retrieval systems, democratizing access to fine-grained visual search capabilities. This paper introduces SaMer, an object-aware token merging framework that significantly enhances the efficiency and accuracy of multi-vector vision-language retrieval. The work presents a novel training-time object-aware prior for post-projector token merging, enabling substantial compression (16x storage reduction, 4-9x QPS improvement) while simultaneously boosting retrieval performance and improving phrase-level grounding, thereby offering a practical and principled solution to a critical bottleneck in large-scale multimodal retrieval systems.
The paper proposes SaMer, an object-aware token merging framework for efficient multi-vector vision-language retrieval. The core idea is to compress dense image-side post-projector tokens into K representative centroids while preserving fine-grained visual evidence. The methodology is well-articulated and technically sound. SaMer employs a feature-spatial soft assignment for merging, where each representative is formed as a normalized weighted centroid. A key innovation is the use of object annotations *only during training* as a merge prior. This prior discourages cross-instance mixing, ensuring that tokens belonging to different object instances are not collapsed into the same representative. This is a clever way to leverage supervision without requiring it at inference time. The projection-only adaptation strategy is also well-designed, freezing the heavy vision and language backbones and only training the shared projection layer to align the compressed tokens with the retrieval space. This makes SaMer pluggable into existing multi-vector retrievers like ColPali and ColQwen2. The algorithm for compression-aware adaptation and inference is clearly presented, highlighting the distinction between training (with object prior) and inference (bbox-free). The choice of soft assignment for centroid construction is empirically justified in the appendix.
The experimental evaluation is comprehensive and rigorous. SaMer is evaluated on four diverse benchmarks: Flickr30K, MSCOCO (natural image retrieval), ImageCoDe (compositional retrieval), and DocVQA (document domain). This broad evaluation demonstrates the method's robustness and identifies its strengths and limitations (e.g., less optimized for DocVQA's sparse textual evidence). The comparison includes a wide range of baselines: single-vector, VLM-based, multi-vector, and retrieval-side compression methods (H-Pool, HPC, SAP). Crucially, comparisons with compression baselines are controlled, using the same backbones and token budget (K=64). The results are compelling: SaMer achieves significant improvements in R@1 on Flickr30K (e.g., ColPali R@1 from 77.0 to 82.4) and MSCOCO, outperforming all compression baselines. It also shows strong performance on ImageCoDe, validating its ability to preserve subtle object/attribute cues. The analysis section is particularly strong, featuring: 1. **Ablation Study**: Clearly demonstrates that the gains are not solely from adaptation but from the object-aware merge design. 2. **Compression Budget Analysis**: Identifies K=64 as an optimal operating point, balancing performance and compression. 3. **Merge Component Study**: Highlights the critical role of the object-aware prior, showing it significantly improves both retrieval and grounding metrics (BoxMass, RegionHit, CoverageIoU). 4. **Grounding Comparison**: Uses novel grounding metrics to quantitatively show that SaMer better concentrates phrase-level relevance inside annotated object regions, a key claim of the paper. Qualitative examples further support this. 5. **Efficiency Analysis**: Quantifies substantial storage reduction (e.g., 16.09x for ColPali) and QPS improvements (4.3x to 9.1x), directly addressing the practical problem of expensive multi-vector retrieval. The empirical evidence strongly supports the paper's claims.
The paper provides a code link (https://github.com/dmis-lab/SaMer), which is excellent for reproducibility. Implementation details, including optimization steps, learning rate, weight decay, scheduler, and GPU setup, are provided in the appendix. The object annotation processing and grounding metric definitions are also clearly detailed. The algorithm for SaMer's training and inference is explicitly laid out. This level of detail and code availability makes the work highly reproducible.
1. **Dependency on Object Annotations for Training**: While the method cleverly avoids inference-time annotations, it still requires object bounding box annotations during training. For domains where such annotations are scarce or expensive, adapting SaMer might be challenging. 2. **Performance on DocVQA**: Although competitive, SaMer is not optimized for document images where sparse OCR tokens and layout cues are crucial. This highlights a domain-specific limitation, as the object-aware prior is less relevant for text-heavy documents. 3. **Hyperparameter Sensitivity**: The spatial term weight (lambda) and soft-assignment temperature (tau_s) are hyperparameters. While not explicitly discussed as a limitation, their tuning might be sensitive for new datasets or backbones. 4. **Generalizability of Object-Awareness**: The "object-aware" prior is based on explicit object instances. For more abstract attributes or relations that don't map cleanly to single object bounding boxes, the effectiveness of this prior might diminish.
SaMer makes a significant contribution to efficient vision-language retrieval, a critical component for large-scale image search, retrieval-augmented visual question answering, and other multimodal applications. By substantially reducing storage and scoring costs (16x storage, 4-9x QPS) *while improving accuracy*, SaMer enables the deployment of fine-grained multi-vector retrieval systems at scales previously impractical. The insight that "efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select" is valuable and could guide future research in multimodal compression. The method's plug-and-play nature with frozen backbones lowers the barrier to adoption. This work could lead to more accessible and performant VLM-powered retrieval systems, democratizing access to fine-grained visual search capabilities. This paper introduces SaMer, an object-aware token merging framework that significantly enhances the efficiency and accuracy of multi-vector vision-language retrieval. The work presents a novel training-time object-aware prior for post-projector token merging, enabling substantial compression (16x storage reduction, 4-9x QPS improvement) while simultaneously boosting retrieval performance and improving phrase-level grounding, thereby offering a practical and principled solution to a critical bottleneck in large-scale multimodal retrieval systems.
Logit-based watermarking is a widely used mechanism for identifying LLM generated content, yet its effectiveness is governed by a fundamental trade-off between detectability and semantic distortion. Existing analyses provide limited guidance for principled hyperparameter selection, leaving practical deployments reliant on heuristic tuning. In this work, we develop a power-calibrated statistical framework that establishes explicit quantitative relationships between watermark hyperparameters, detection power, and distortion. This characterization transforms watermark design into a guided optimization problem. Building on these results, we derive practical parameter selection procedures that achieve optimal tradeoffs under constraints. Extensive experiments across multiple language models and datasets validate the theory and demonstrate that the proposed framework consistently identifies Pareto-optimal points.
Primary: Department of Statistics, Pennsylvania State University, University Park
All Institutions: Department of Statistics, Pennsylvania State University, University Park
The paper's broader impact is significant and directly addresses critical societal concerns related to the proliferation of large language models. 1. **Combating Misinformation**: By improving the efficiency and reliability of LLM watermarking, this work strengthens the ability to identify machine-generated content. This is crucial in the fight against misinformation, fake news, and propaganda campaigns that leverage generative AI. 2. **Academic Integrity**: Watermarking can help detect AI-generated content in academic submissions, supporting academic honesty and preventing misuse of LLMs for plagiarism or automated assignment completion. 3. **Ethical AI Deployment**: It provides a mechanism for establishing provenance, which is essential for responsible and transparent deployment of generative AI. Users can be informed whether content originates from an AI, fostering trust and accountability. 4. **Scalability and Practicality**: By transforming watermark design from heuristic tuning to a statistically grounded optimization problem, the framework makes watermarking systems more robust, reliable, and easier to deploy in real-world applications. This improved efficiency and principled parameter selection can lead to wider adoption and more effective use of watermarking technologies. 5. **Research Foundation**: The rigorous statistical framework provides a strong theoretical foundation for future research in LLM watermarking, enabling more principled development and analysis of new techniques. It encourages a shift from empirical trial-and-error to theoretically informed design. The work directly contributes to mitigating potential harms of generative AI while enabling its beneficial uses, aligning with responsible AI development principles. This paper introduces a controllable statistical framework for logit-based LLM watermarking, enabling principled calibration of watermark strength under explicit detectability and distortion objectives. By establishing quantitative mappings between watermark parameters, detection power, and KL-based distortion, the authors transform watermark design from heuristic tuning into a statistically grounded optimization problem, validated through extensive experiments across multiple language models and datasets, consistently identifying Pareto-optimal configurations.
The paper introduces a power-calibrated statistical framework for logit-based LLM watermarking, moving beyond heuristic hyperparameter tuning. The core methodology involves establishing explicit quantitative relationships between watermark hyperparameters (bias $\delta$ and green-list fraction $\gamma$), detection power, and semantic distortion (measured by KL divergence). Key methodological steps include: 1. **Formalizing Hypothesis Testing**: The paper frames watermark detection as a hypothesis test, defining the null ($H_0$: unwatermarked text) and alternative ($H_1$: watermarked text) hypotheses based on the green-list token probability. 2. **Assumptions for Tractability**: To derive closed-form expressions, the framework relies on several assumptions: * **Random green-list assignments**: Green lists are i.i.d. and independent of the generated token sequence, simplifying the null hypothesis to an i.i.d. Bernoulli process for green-token indicators. * **Non-informative NTP prior**: The next-token probability (NTP) vector is modeled as an independent draw from a uniform Dirichlet distribution. The paper acknowledges this is a simplification but argues for its effectiveness due to the detector's dependence on green-list mass, which concentrates around $\gamma$. * **Information Decay**: Assumes a geometric decay in mutual information between past and future green-token indicators, enabling the application of a Central Limit Theorem for $\alpha$-mixing sequences under the alternative hypothesis. * **Non-degenerate Long-run Variance**: Ensures the asymptotic variance of the indicator sequence is strictly positive. 3. **Derivation of Key Metrics**: * **Green-token probability under watermarking ($\gamma'$)**: Lemma 2.2 and Theorem 2.3 provide a precise, token-level description of how watermarking biases generation toward the green list, leading to a closed-form expression for $\gamma'$ in terms of $\delta$ and $\gamma$. * **Detection Power**: Using the normal approximation for the aggregate statistic $S_n$ under both $H_0$ and $H_1$, a closed-form expression for statistical power $\Psi^*(\delta, \gamma)$ is derived. The paper notes that a constant $c$ (long-run variance inflation) affects the numerical value but not the parameter selection for maximization. * **Distortion (KL Divergence)**: Lemma 3.1 provides a plug-in formula for the expected token-wise KL divergence $D_{KL}(\delta, \gamma)$ between watermarked and unwatermarked distributions. It proves strict monotonicity in $\delta$ for fixed $\gamma$, allowing $\delta$ to be parameterized by a distortion budget. 4. **Optimization Framework**: The theoretical characterization transforms watermark design into a guided optimization problem. Instead of tuning two hyperparameters $(\delta, \gamma)$, the problem is reduced to a 1D numerical optimization (e.g., maximizing power subject to a KL distortion budget $K_0$) because $\delta$ can be implicitly determined by $K_0$. Practical guidance is provided for initializing the search for $\gamma$. The methodology is statistically rigorous, building on established theorems (CLT for mixing sequences) and providing detailed proofs in the appendix. The reduction of a multi-dimensional heuristic search to a principled 1D optimization problem under constraints is a significant practical contribution.
The experimental evaluation is comprehensive and well-structured, providing strong empirical validation for the theoretical framework. 1. **Setup**: The protocol largely follows prior watermarking evaluations, using diverse LLMs (OPT, Pythia, GPT-2, and Gemma-2 9B in the appendix) and datasets (C4, LFQA, Wikipedia). Generations are kept short ($n=50$ tokens) to avoid trivial detectability gains and better expose statistical efficiency differences. All methods are implemented using a unified pipeline based on the KGW codebase to ensure fair comparison. 2. **Distributional Verification**: * **Normality**: Q-Q plots confirm that the standardized green-token count statistic closely follows a standard normal distribution under both $H_0$ and $H_1$, supporting the normal approximation used in the detectability analysis. * **Alternative Hypothesis Characterization**: Empirical green-token rates are compared against theoretical predictions from Theorem 2.3. A near-linear relationship with $R^2 > 0.98$ across all settings validates the simplified modeling assumptions for $\gamma'$. 3. **Performance Evaluation (Detectability-Distortion Trade-off)**: * The paper evaluates statistical power (TPR at $\alpha=0.05$) against semantic distortion (per-token KL divergence). * Results across various models and datasets consistently show that the proposed method lies on the Pareto frontier. It achieves high detection power at substantially lower distortion compared to baseline methods (KGW, DP, and dense grid search). * The approach maintains near-saturated TPR in moderate distortion regimes where baselines exhibit significant degradation. This highlights improved statistical efficiency. 4. **Robustness under Alternative Quality Metrics**: * To assess the generalizability of the optimized parameters, the method is evaluated using complementary semantic quality metrics: BLEU, ROUGE, and BERTScore. * The results show a consistent pattern: the proposed method maintains strong detection power over a wider range of quality values compared to baselines. This suggests that the optimized parameterization induces a more uniform and controlled distributional shift, not tied to a specific notion of text similarity. Overall, the experiments are thorough, well-designed, and effectively support the theoretical claims. The validation of assumptions, extensive comparisons, and robustness checks significantly strengthen the paper's conclusions.
The paper demonstrates a strong commitment to reproducibility. 1. **Code Availability**: The authors explicitly state that "The implementation is available at https://github.com/shooof/wm." This is a crucial step for reproducibility. 2. **Detailed Experimental Setup**: The "Experimental Setup" section provides clear details on the language models (OPT, Pythia, GPT-2, Gemma-2 9B), datasets (C4, LFQA, Wikipedia), generation length ($n=50$), and detection parameters ($\alpha=0.05$). 3. **Baseline Descriptions**: Baseline methods (KGW, DP, dense grid search) are clearly identified, and their implementation is stated to use the same unified generation and detection pipeline as the proposed method, controlling for implementation effects. 4. **Hyperparameter Ranges**: Appendix provides a table of hyperparameter ranges used for all methods, which is essential for replicating the search space. 5. **Metric Definitions**: Clear definitions and implementations for distortion (KL divergence) and quality metrics (BLEU, ROUGE, BERTScore) are provided. 6. **Theoretical Proofs**: The appendix contains detailed proofs for all lemmas and theorems, allowing for independent verification of the mathematical derivations. The combination of open-source code, detailed experimental descriptions, and theoretical proofs makes the work highly reproducible.
The paper acknowledges limitations, with a dedicated discussion mentioned in Appendix [REF] (though the provided text truncates before this appendix content). Based on the main text, potential limitations include: 1. **Assumptions for Tractability**: The framework relies on several simplifying assumptions: * **Random green-list assignments**: While reasonable for pseudo-random hash functions, real-world implementations might have subtle biases. * **Non-informative NTP prior (Uniform Dirichlet)**: The paper argues for its effectiveness, but it is an approximation. While it mentions that richer priors (mixtures of Dirichlet) could be used if structural information is available, the current analysis is based on a simplified prior. The accuracy of this approximation for all possible LLM output distributions might vary. * **Information Decay**: The assumption of geometric decay in mutual information is a model for the autoregressive dependence. While standard for mixing sequences, its precise fit to complex LLM generation dynamics is an approximation. 2. **Estimation of Constant 'c'**: The power formula includes a constant $c$ that captures long-run variance inflation due to dependence. While the paper states that parameter selection is independent of $c$, its numerical value is needed for absolute power predictions and needs to be estimated in practice (as mentioned in Appendix [REF]). This introduces an additional practical step and potential source of error. 3. **Focus on Logit-Based Watermarking**: The framework is specifically developed for logit-based watermarking (KGW framework). While this is a widely adopted paradigm, the theoretical derivations might not directly apply to other watermarking schemes without significant adaptation. 4. **Short Sequence Evaluation**: While justified to isolate statistical efficiency, the primary experiments are on short sequences ($n=50$). While the paper discusses long-form generation in the appendix, the main empirical results are on short sequences. 5. **Adversarial Robustness**: While the appendix briefly discusses "signal-removal robustness" using a WinMax-C detector, a more extensive analysis of various adversarial attacks (e.g., paraphrasing, fine-tuning, token manipulation) and how the calibrated parameters perform under these attacks would be valuable. The current framework optimizes for detectability and distortion in a "clean" setting.
The paper's broader impact is significant and directly addresses critical societal concerns related to the proliferation of large language models. 1. **Combating Misinformation**: By improving the efficiency and reliability of LLM watermarking, this work strengthens the ability to identify machine-generated content. This is crucial in the fight against misinformation, fake news, and propaganda campaigns that leverage generative AI. 2. **Academic Integrity**: Watermarking can help detect AI-generated content in academic submissions, supporting academic honesty and preventing misuse of LLMs for plagiarism or automated assignment completion. 3. **Ethical AI Deployment**: It provides a mechanism for establishing provenance, which is essential for responsible and transparent deployment of generative AI. Users can be informed whether content originates from an AI, fostering trust and accountability. 4. **Scalability and Practicality**: By transforming watermark design from heuristic tuning to a statistically grounded optimization problem, the framework makes watermarking systems more robust, reliable, and easier to deploy in real-world applications. This improved efficiency and principled parameter selection can lead to wider adoption and more effective use of watermarking technologies. 5. **Research Foundation**: The rigorous statistical framework provides a strong theoretical foundation for future research in LLM watermarking, enabling more principled development and analysis of new techniques. It encourages a shift from empirical trial-and-error to theoretically informed design. The work directly contributes to mitigating potential harms of generative AI while enabling its beneficial uses, aligning with responsible AI development principles. This paper introduces a controllable statistical framework for logit-based LLM watermarking, enabling principled calibration of watermark strength under explicit detectability and distortion objectives. By establishing quantitative mappings between watermark parameters, detection power, and KL-based distortion, the authors transform watermark design from heuristic tuning into a statistically grounded optimization problem, validated through extensive experiments across multiple language models and datasets, consistently identifying Pareto-optimal configurations.
Unified models for robot manipulation aim to equip one policy with both the semantic priors of pretrained VLMs and the physical dynamics learned through future prediction. In practice, existing designs tend to erode the semantics of the pretrained backbone, suffer interference among heterogeneous objectives, and learn future prediction from scratch in pixel space, leaving the dynamics priors of pretrained video generators unexploited. We present InternVLA-A1.5, which builds the policy on a native VLM backbone that keeps training on VQA and subtask prediction, and attaches a lightweight unified expert for continuous action generation. Future prediction is recast as a latent-querying problem, where a small set of learnable foresight tokens condenses the task-relevant future into a compact latent code under the supervision of a frozen pretrained video generation model, so the policy inherits world-model dynamics priors without ever learning pixel-level generation. The video branch is discarded at inference, keeping real-time control. Pretrained on 1.2M robot episodes and 3M multimodal samples, InternVLA-A1.5 achieves the best overall results on all six simulation benchmarks. In the real world, the preserved semantics deliver the strongest compositional generalization on held-out instruction bindings, and the two designs together sustain long-horizon execution.
Primary: Shanghai AI Laboratory
All Institutions: Shanghai AI Laboratory, Shanghai Jiao Tong University, Zhejiang University, University of Oxford, University of Cambridge, University of Edinburgh, University of Tokyo, University of Toronto, University of California, Berkeley, University of Washington, University of Michigan, University of Illinois Urbana-Champaign, University of Texas at Austin, University of Southern California, University of California, Los Angeles, University of California, San Diego, University of California, Irvine, University of California, Santa Barbara, University of California, Riverside, University of California, Merced, University of California, Santa Cruz, University of California, Davis, University of California, Santa Barbara, University of California, San Diego, University of California, Irvine, University of California, Los Angeles, University of California, Berkeley, University of Washington, University of Michigan, University of Illinois Urbana-Champaign, University of Texas at Austin, University of Southern California, University of California, Los Angeles, University of California, San Diego, University of California, Irvine, University of California, Santa Barbara, University of California, Riverside, University of California, Merced, University of California, Santa Cruz, University of California, Davis
InternVLA-A1.5 presents a significant advancement in Vision-Language-Action models by effectively unifying semantic understanding with efficient latent-based world modeling, achieving state-of-the-art results in simulation and demonstrating strong compositional generalization in real-world settings. The technical contribution lies in the novel "latent foresight" mechanism and the training strategy that prevents semantic erosion, offering a scalable and effective path forward for unified robot policies. While the score is high (76), it reflects the strong empirical results and the clear technical merit of the approach, though it may not yet be considered a "once-a-decade" breakthrough given the incremental nature of many VLA improvements. However, the combination of performance and efficiency makes it a highly impactful paper for the robotics community.
The paper proposes InternVLA-A1.5, a unified Vision-Language-Action model designed to address the "semantic erosion" and "objective interference" common in existing VLA architectures. The core methodological innovation lies in two areas: 1) A training strategy that preserves the semantic capabilities of the pretrained VLM backbone by continuing to train on VQA and subtask prediction tasks alongside robot control, rather than freezing or discarding these capabilities. 2) A "latent foresight" mechanism where future prediction is formulated as a latent-querying problem. Instead of learning pixel-level video generation (which is computationally expensive and often unstable), the model uses a small set of learnable tokens to condense task-relevant future dynamics into a compact latent code, supervised by a frozen pretrained video generation model. This allows the policy to inherit world-model dynamics priors without the overhead of pixel reconstruction. The architecture attaches a lightweight unified expert for continuous action generation, discarding the video branch at inference to ensure real-time control. This approach is technically sound and addresses specific pain points in the VLA literature (semantic drift and computational inefficiency of world models).
The evaluation is comprehensive, covering six simulation benchmarks and real-world experiments. The paper reports state-of-the-art results on all six simulation benchmarks, which is a strong indicator of performance. The real-world experiments highlight the model's compositional generalization on held-out instruction bindings, a critical capability for practical deployment. The ability to sustain long-horizon execution is also demonstrated. The use of 1.2M robot episodes and 3M multimodal samples for pretraining provides a solid data foundation. The comparison against existing baselines (likely including RT-2, OpenVLA, and other VLA variants) shows competitive or superior performance, particularly in generalization scenarios. The real-world results are crucial for validating the simulation findings, and the paper appears to provide robust evidence here.
The paper mentions the scale of data (1.2M episodes, 3M multimodal samples) and the use of pretrained components (VLM backbone, frozen video generator). However, as an arXiv preprint, full code and model weights are not explicitly linked in the provided text (though often available via project pages not cited in the abstract). The description of the "latent-querying" mechanism and the specific VQA/subtask prediction losses is detailed enough to allow for replication by researchers with significant resources. The reliance on a "frozen pretrained video generation model" requires specifying which model (e.g., Stable Video Diffusion, Sora-like models, or a specific open-source variant), which is a potential ambiguity if not clearly defined in the full text. Assuming standard practices, reproducibility is moderate to high, contingent on access to the specific pretrained video models and the large-scale robot dataset.
The paper does not explicitly discuss the limitations of the "latent foresight" approach. Potential limitations include: 1) The dependency on the quality of the frozen video generation model for supervision; if the video model fails to capture certain dynamics, the foresight tokens may learn suboptimal representations. 2) The computational cost of training with the auxiliary video generation loss, even if the video branch is discarded at inference. 3) The generalization to out-of-distribution environments or objects not seen during the 1.2M episode pretraining. 4) The "lightweight unified expert" might become a bottleneck for very complex manipulation tasks requiring fine-grained control, although the paper claims success in long-horizon execution. 5) The real-world results are likely limited to specific setups, and generalization to arbitrary home environments remains a challenge.
This work contributes to the broader goal of creating general-purpose robot manipulators that can understand natural language instructions and generalize to new tasks. By improving compositional generalization and reducing the computational overhead of world models, it makes VLA policies more practical for real-world deployment. The emphasis on preserving semantic priors aligns with the trend of leveraging large-scale pretrained models for robotics. However, the deployment of such powerful robots in unstructured environments raises safety and ethical considerations, which are not addressed in the paper. The potential for misuse in autonomous systems is a standard concern in robotics research. InternVLA-A1.5 presents a significant advancement in Vision-Language-Action models by effectively unifying semantic understanding with efficient latent-based world modeling, achieving state-of-the-art results in simulation and demonstrating strong compositional generalization in real-world settings. The technical contribution lies in the novel "latent foresight" mechanism and the training strategy that prevents semantic erosion, offering a scalable and effective path forward for unified robot policies. While the score is high (76), it reflects the strong empirical results and the clear technical merit of the approach, though it may not yet be considered a "once-a-decade" breakthrough given the incremental nature of many VLA improvements. However, the combination of performance and efficiency makes it a highly impactful paper for the robotics community.
Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
Primary: Yale University
All Institutions: Yale University, University of Illinois Urbana-Champaign, Microsoft Research, Amazon
Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
The paper introduces MedPMC, a systematic, automated pipeline for curating high-fidelity image-text pairs from PubMed Central (PMC). The methodology addresses a critical bottleneck in medical multimodal learning: the scarcity of large-scale, high-quality, and clinically relevant data. The pipeline involves several sophisticated components: (1) filtering 6.1 million articles for permissive licenses; (2) detecting multi-panel figures; (3) separating individual figures from panels; (4) aligning figures with their corresponding captions using natural language processing techniques; and (5) classifying figures for medical relevance. The authors report high performance on these component tasks (e.g., F1=93.2 for screening, mAP=89.8 for figure separation), indicating a robust and well-engineered curation process. The novelty lies not in a single algorithmic breakthrough but in the systematic integration and scaling of these components to create a massive, high-quality dataset, coupled with rigorous validation against prior, lower-fidelity datasets.
The evaluation is comprehensive and compelling. First, the authors validate the quality of the curated data through manual review by five annotators (three with medical training), showing a significant improvement in medical relevance (95.3% vs. 19.7% in a prior dataset). Second, they train a CLIP-style model on the MedPMC corpus and evaluate it on 26 benchmarks across 11 medical specialties. The MedPMC-trained model outperforms the strongest architecture-matched biomedical CLIP baseline by 7.1 percentage points in average zero-shot AUC, despite using fewer than half the image-text pairs. This demonstrates the high signal-to-noise ratio of the MedPMC data. Third, they integrate the vision encoder into a multimodal large language model (MLLM) and show improvements in medical visual question-answering. Finally, they demonstrate clinical utility by improving morphology-to-image retrieval on a real-world dermatology dataset from Yale New Haven Health System. The results are statistically significant and practically meaningful.
The authors provide extensive resources for reproducibility. The code for the curation pipeline is publicly available on GitHub. The MedPMC corpus, component-level benchmark resources, pretrained checkpoints, and metadata are released on Hugging Face. The data release includes versioning by article cutoff date, source-license filters, and processing configuration, allowing users to reproduce specific snapshots. They also provide clear instructions for license-aware filtering. The only non-public component is the clinical dermatology data, which is subject to privacy restrictions, but aggregate results and processing procedures are described. This level of transparency is exemplary.
The primary limitation is the reliance on open-access literature, which may introduce selection bias towards certain types of studies or institutions. The pipeline's performance, while high, is not perfect (e.g., F1=81.4 for caption separation), meaning some noise remains in the dataset. The clinical evaluation is limited to dermatology retrieval, and while promising, it does not cover the full breadth of medical specialties. The authors acknowledge that some records are derived from articles with non-commercial licenses (CC BY-NC), which restricts commercial use of the dataset. Additionally, the study focuses on image-text pairs; the extension to other modalities (e.g., audio, video, time-series) is not addressed.
The MedPMC framework has significant potential to accelerate the development of medical multimodal foundation models. By providing a large-scale, high-quality, and openly accessible dataset, it lowers the barrier to entry for researchers and clinicians working in this domain. The improved performance of models trained on MedPMC suggests that high-fidelity data curation is a key lever for improving model capabilities. The release of the code and benchmarks encourages further research in data curation and multimodal learning. However, the non-commercial license restrictions for some data may limit its impact in commercial healthcare applications. The authors have responsibly addressed privacy concerns by not releasing patient data and by providing clear licensing information. Yale University's MedPMC framework represents a landmark contribution to medical multimodal learning by systematically curating a massive, high-fidelity dataset from open-access literature, demonstrating that data quality is as critical as quantity for training effective foundation models, and providing a reproducible pipeline and resources that will serve as a new standard for the field.
LLM-based ASR adapted to regulated domains such as banking is bottlenecked by privacy: real speech is costly and legally constrained to collect, making synthetic text-to-speech (TTS) an attractive substitute. Yet synthetic speech stays acoustically mismatched with real recordings, and work on this gap has stayed within supervised fine-tuning (SFT). We instead turn to reinforcement learning, and show that Group Relative Policy Optimization (GRPO) extracts far more from the same synthetic speech than SFT. Synthetic-only adaptation of the model with GRPO, a critic-free method rewarding low-WER hypotheses, reduces WER by 40\% relative to SFT (36.71\%$\to$22.09\%), and an SFT-then-GRPO combination pushes this further to 45\%. We trace the gain to behavior rather than representation: GRPO reduces insertion errors by improving stopping calibration and speech-to-text alignment by better anchoring attention to audio, leaving early-layer representations intact. When synthetic speech is the main resource, reinforcement learning should be preferred over supervised fine-tuning.
Primary: Idiap Research Institute
All Institutions: Idiap Research Institute, EPFL, Uniphore, Brno University of Technology
This paper presents a compelling application of GRPO to ASR, demonstrating that reinforcement learning can effectively bridge the synthetic-to-real speech gap by improving stopping calibration and reducing hallucinations, offering a practical solution for privacy-constrained domain adaptation.
The paper proposes applying Group Relative Policy Optimization (GRPO), a reinforcement learning method recently popularized in LLM alignment (e.g., DeepSeek-R1), to the domain of Automatic Speech Recognition (ASR) using synthetic speech data. The core methodological innovation is the application of a critic-free RL approach to optimize for Word Error Rate (WER) directly, bypassing the limitations of Supervised Fine-Tuning (SFT) which suffers from token-level cross-entropy losses when acoustic signals are mismatched (synthetic vs. real). The authors adapt the standard GRPO formulation by defining specific reward functions (WER, CER, length penalties) and demonstrate its efficacy on an LLM-based ASR backbone (WavLM + Llama-3.2). The analysis of *why* it works—specifically through improved stopping calibration and attention anchoring rather than representation rewriting—is a significant theoretical contribution to understanding RL in multimodal settings.
The experimental setup is rigorous and well-controlled. The authors use the DefinedAI banking corpus, a realistic high-stakes domain, and compare SFT vs. GRPO under two regimes: full real data access and privacy-constrained synthetic-only access. Key findings include: 1) GRPO significantly outperforms SFT on synthetic-only data (WER 22.09% vs 36.71%). 2) SFT+GRPO provides incremental gains on real data. 3) The benefit is driven by a reduction in insertion errors, attributed to better stopping calibration. 4) Ablations on reward functions and synthetic data selection strategies are thorough. The results are statistically significant and practically relevant for regulated industries. The analysis linking behavioral changes (attention entropy, confidence) to performance gains adds depth beyond simple metric reporting.
The paper provides detailed hyperparameters (LoRA rank, learning rates, KL coefficients, group size). The backbone architecture (WavLM-Large, Llama-3.2-1B) and preprocessing steps (RIR convolution, TTS pipeline) are described. However, the code repository is withheld for blind review, and specific details on the TTS model (Qwen3-TTS) and voice cloning setup are referenced but not fully open-sourced in the text. The use of proprietary or less-common TTS models might pose slight reproducibility hurdles for exact replication, though the methodological framework is clear.
The study is limited to a single domain (banking) and a single LLM-based ASR backbone. While the findings are promising, generalizability to other domains (e.g., medical, legal) or different ASR architectures (e.g., Conformer-LLM hybrids) is not tested. The reliance on synthetic speech introduces potential biases inherent to the TTS system used (Qwen3-TTS), and the paper does not extensively analyze the impact of TTS quality variations on GRPO's robustness. Additionally, the computational cost of GRPO (requiring multiple rollouts per step) is higher than SFT, which is a practical limitation not fully quantified in terms of training time/cost relative to the performance gain.
This work has significant implications for deploying speech AI in privacy-sensitive sectors where data collection is legally restricted (GDPR, HIPAA). By demonstrating that synthetic speech can be effectively leveraged via RL, it offers a viable path for domain adaptation without compromising user privacy. It also bridges the gap between NLP alignment techniques (RLHF/GRPO) and speech processing, potentially accelerating the adoption of RL methods in multimodal ASR research. This paper presents a compelling application of GRPO to ASR, demonstrating that reinforcement learning can effectively bridge the synthetic-to-real speech gap by improving stopping calibration and reducing hallucinations, offering a practical solution for privacy-constrained domain adaptation.
A model should refuse two different things: answers it would get wrong, and questions it should not answer at all, such as unanswerable ones or ones resting on a false premise. The usual recipe thresholds a single confidence score, which cannot tell these apart. Across five instruction-tuned models from three families (2B to 14B), we find they are separate axes. Ordinary answer-confidence tracks whether an answer is right but is nearly blind to whether the question is answerable; a linear probe on hidden states does the reverse. The blind spot does not shrink with scale. It is worst on naturally occurring false-premise questions (CREPE). There, answer-confidence, P(IK), P(True), and even asking the model outright whether a premise is false all stay near chance, while a hidden-state probe reaches 0.69 to 0.77 AUROC: the model represents a problem it will not report. This turns out to be fixable. Instructing a model to check premises backfires, because it then disputes sound and false premises alike (57% false challenges), unable to tell them apart; routing the same instruction with the probe roughly triples challenge precision. We turn the two axes into a calibrated policy that answers only when an answerability score and a correctness score each clear a separately certifies behave differently: the unanswerable-answer rate is controllable at every scale, while the wrong-answer rate is capped by model accuracy, so the guarantee tightens as threshold policy certifies both budgets at 0.75 coverage of correct answers, against 0.31 for a single threshold; at 14B it is the only policy that certifies at all.
Primary: City St George's, University of London
All Institutions: City St George's, University of London
The paper makes a significant contribution by empirically demonstrating and formally exploiting the decoupling of answer correctness and question answerability in LLMs, introducing a factorized abstention policy with rigorous statistical guarantees that outperforms standard single-threshold methods.
The paper proposes a fundamental decomposition of LLM abstention into two orthogonal axes: answer correctness (execution) and question answerability (admissibility). The methodology involves training linear probes on hidden states to detect answerability and comparing them against standard output-confidence metrics (mean token log-probability, entropy). It introduces a factorized abstention policy that requires both high answerability and high correctness scores to emit an answer, calibrated using conformal prediction techniques (Learn-Then-Test framework) to provide rigorous statistical guarantees on error rates. The approach is theoretically grounded in neuro-symbolic reasoning, treating abstention as a three-class selective acceptance problem.
The evaluation is extensive and rigorous, spanning five instruction-tuned models (2B to 14B) from three families (Gemma, Qwen, Llama). It utilizes two distinct benchmarks: SelfAware (constructed, controlled) and CREPE (naturally occurring false presuppositions). The experiments demonstrate a consistent "crossed geometry" where output confidence separates correct from wrong answers but fails to distinguish wrong from unanswerable, while hidden-state probes do the reverse. The paper provides certified risk bounds, showing that the factorized policy significantly outperforms single-threshold baselines, especially at larger scales (8B, 14B), where single-threshold policies fail to certify any coverage. The inclusion of a replication audit over 100 resplits adds substantial empirical weight.
The authors commit to releasing code and score artifacts, with a verification script run in continuous integration. The methodology is clearly described, including specific prompts, model quantization details, and calibration splits. The use of exact binomial bounds and clear split definitions enhances reproducibility. The paper explicitly states that all numbers are regenerated from committed artifacts.
The study is limited to models up to 14B, leaving open whether the pattern holds for frontier models (e.g., 70B+). The SelfAware benchmark has surface confounds (lexical differences between answerable/unanswerable), though the CREPE benchmark mitigates this. The correctness evaluation relies on alias matching, which may be imperfect for long-form answers. The certification guarantees are marginal and depend on exchangeability assumptions. The "probe-routed premise checking" is a pilot study with small sample sizes.
This work has significant implications for the safety and reliability of LLMs. By decoupling the ability to answer from the willingness to answer, it provides a pathway to reduce hallucinations and inappropriate responses in high-stakes applications. The certified abstention framework offers a new standard for evaluating model reliability, moving beyond simple accuracy metrics to risk-controlled deployment. It challenges the prevailing reliance on output confidence for uncertainty estimation. The paper makes a significant contribution by empirically demonstrating and formally exploiting the decoupling of answer correctness and question answerability in LLMs, introducing a factorized abstention policy with rigorous statistical guarantees that outperforms standard single-threshold methods.
Structure-property relationships are foundational to biology, chemistry and materials science, where function, reactivity and physical response emerge from spatial, chemical and periodic organization. Mechanistically explaining these relationships requires interpreting structural evidence through scientific principles and physical constraints, from stereochemistry and bonding to symmetry, energetics and periodic order. However, applying artificial intelligence to this process presents a joint challenge of representation and reasoning: models must preserve domain-native structural information while showing how specific evidence supports predictions under these constraints. Here we introduce SciReasoner, a multimodal scientific foundation model for native structural reasoning across proteins, small molecules and inorganic crystals. SciReasoner discretizes coordinates, topologies and periodic connectivities into a unified structure-aware vocabulary, treating structural tokens as addressable evidence units during reasoning. In homology-controlled Gene Ontology prediction, SciReasoner improves Cellular Component annotation for low-homology and orphan-like proteins, increasing $F_{\max}$ from 0.42 to 0.55. In chemistry, it raises single-step retrosynthesis accuracy from 0.63 to 0.72 while generating fragment-level disconnection and precursor-verification traces. In materials science, its representations separate elemental and compound phases and resolve high- and low-band-gap regimes. Across 86 benchmarks, SciReasoner achieves state-of-the-art performance on 67 tasks. Double-blind expert evaluation rates its reasoning traces as preferred or at least comparable to those of a frontier large language model in 98% of cases. By making structure an inspectable substrate for reasoning under scientific constraints, SciReasoner connects accurate prediction with interpretable scientific inference.
Primary: Shanghai Artificial Intelligence Laboratory
All Institutions: Shanghai Artificial Intelligence Laboratory, The Chinese University of Hong Kong, Shanghai Jiao Tong University, Fudan University, University of Sydney, Nanjing University, University of Oxford, The University of Science and Technology of China, Drug Discovery and Design Center, State Key Laboratory of Drug Research, Shanghai Institute of Materia Medica, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Stanford University
SciReasoner represents a significant leap in scientific machine learning by unifying structural reasoning across biology, chemistry, and materials science through a novel tokenization and autoregressive framework, achieving state-of-the-art results and offering interpretable insights that bridge the gap between prediction and mechanistic understanding.
The paper introduces "SciReasoner," a multimodal foundation model designed for "native structural reasoning" across three distinct scientific domains: proteins, small molecules, and inorganic crystals. The core methodological innovation lies in the discretization of continuous structural data (coordinates, topologies, periodic connectivities) into a unified, structure-aware vocabulary. By treating structural tokens as addressable evidence units within an autoregressive reasoning trajectory, the model attempts to bridge the gap between geometric representation and logical inference. This approach moves beyond standard graph neural networks or transformer-based sequence models by explicitly integrating physical constraints (stereochemistry, bonding, symmetry) into the tokenization and reasoning process. The architecture appears to leverage a large language model backbone adapted for these specialized structural tokens, allowing for chain-of-thought style reasoning where the model generates explanations alongside predictions. The unification of these three disparate fields under a single "structure-property" framework is ambitious and represents a significant conceptual shift from domain-specific models to a general scientific reasoning engine.
The evaluation is extensive, covering 86 benchmarks across biology, chemistry, and materials science. Key results include: 1. **Biology:** Improvement in Gene Ontology prediction for low-homology proteins (Cellular Component $F_{max}$ from 0.42 to 0.55), demonstrating the model's ability to generalize beyond sequence similarity to structural function. 2. **Chemistry:** Single-step retrosynthesis accuracy increased from 0.63 to 0.72, with the added capability of generating fragment-level disconnection and precursor-verification traces. This interpretability is a key selling point. 3. **Materials Science:** The model successfully separates elemental and compound phases and resolves band-gap regimes, indicating strong representation learning capabilities for periodic structures. 4. **General Performance:** State-of-the-art performance on 67 out of 86 tasks. 5. **Human Evaluation:** Double-blind expert evaluation rated the reasoning traces as preferred or comparable to frontier LLMs in 98% of cases. The breadth of the evaluation is impressive, but the reliance on "homology-controlled" settings for biology suggests the model's true value is in the "long-tail" or zero-shot scenarios where traditional methods fail. The jump in retrosynthesis accuracy is significant in a field where gains are often marginal.
The paper is published on arXiv. While the abstract and introduction provide a high-level overview, the full text provided in the prompt is truncated (sections are listed but content is missing). However, based on the abstract's description of "discretizing coordinates... into a unified structure-aware vocabulary," the method implies a specific tokenization scheme that, if detailed in the full paper (likely in the Method section), would be reproducible. The claim of 86 benchmarks suggests a comprehensive suite, which aids reproducibility if the benchmark definitions are standardized. The lack of a provided code URL in the prompt is a negative, but top-tier scientific ML papers increasingly open-source code; the high institutional backing (Shanghai AI Lab, CUHK, etc.) suggests code release is likely or imminent.
1. **Computational Cost:** Training a multimodal foundation model on three complex scientific domains with a unified vocabulary is likely extremely computationally intensive. The inference cost for generating "reasoning traces" may be prohibitive for high-throughput screening. 2. **Domain Specificity of Tokenization:** While the "unified vocabulary" is a strong claim, the physical constraints of proteins (flexible, solvent-exposed) differ vastly from inorganic crystals (rigid, periodic). It is unclear how well a single tokenizer handles the topological diversity without significant information loss or ambiguity. 3. **Evaluation Bias:** The "expert evaluation" of reasoning traces, while promising, is subjective. The 98% preference rate is exceptionally high and may suffer from bias if the traces are generated by a model known to be high-performing. 4. **Data Quality:** The performance is heavily dependent on the quality of the training data (PDB, ZINC, Materials Project, etc.). Errors in structural data or labeling in these databases will propagate into the model's "reasoning."
This work has profound implications for scientific discovery. By providing interpretable, structure-grounded reasoning, it addresses the "black box" problem in AI for science. This could accelerate drug discovery (by explaining *why* a molecule is active), materials design (by explaining phase stability), and synthetic biology. The ability to handle low-homology proteins could aid in understanding orphan diseases. However, the potential for misuse in designing harmful biological agents or novel toxins via automated retrosynthesis must be considered, necessitating robust safety guardrails in the reasoning traces. SciReasoner represents a significant leap in scientific machine learning by unifying structural reasoning across biology, chemistry, and materials science through a novel tokenization and autoregressive framework, achieving state-of-the-art results and offering interpretable insights that bridge the gap between prediction and mechanistic understanding.
Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .
Primary: University of Hong Kong
All Institutions: University of Hong Kong, Snowflake
Spider 2.0-AIFunc introduces the first benchmark for evaluating text-to-SQL systems on AI-native SQL workflows, revealing that current models struggle with semantic parameterization and that complex agent frameworks offer no advantage over minimal setups, signaling a shift in the bottleneck of enterprise text-to-SQL from planning to semantic execution. The paper provides a rigorous evaluation framework and detailed error analysis that will serve as a foundational resource for the community as the industry transitions to AI-integrated database systems.
The paper introduces a novel benchmark construction methodology for "AI-Native SQL," a new paradigm where LLM capabilities are exposed as native SQL functions (e.g., `AI_CLASSIFY`, `AI_SENTIMENT`). The core methodological contribution is an agent-based pipeline that rewrites existing enterprise text-to-SQL tasks (from Spider2-Snow) into AI-native forms. This involves not just transforming the SQL query but also refining the natural language instructions to ensure specification determinism (explicitly defining AI function parameters) and verifying execution determinism (handling stochasticity in AI function outputs via multi-round verification). The approach addresses a critical gap in current benchmarks which ignore the shift towards semantic operators in SQL engines.
The evaluation is rigorous and comprehensive. The authors evaluate 10 state-of-the-art models (proprietary and open-source) using a minimal agent framework to isolate model capability from framework complexity. They provide detailed error analysis stratified by model performance tiers, identifying specific failure modes such as schema grounding, predicate specification, and AI function parameterization. The finding that complex agent frameworks (AutoLink, ReFoRCE, DSR-SQL) do not outperform a minimal setup is a significant and surprising empirical result, suggesting that the bottleneck in AI-Native SQL is primarily model capability (semantic understanding and parameterization) rather than retrieval or planning complexity. The benchmark includes 465 verified instances across 125 databases, providing a robust testbed.
The paper provides a detailed description of the construction pipeline, verification protocols, and evaluation metrics. The code and data are made publicly available on GitHub. The multi-round execution verification protocol ensures that the benchmark instances are stable, which is crucial for reproducibility in the context of stochastic AI functions. The evaluation setup is clearly defined, including the handling of timeouts and the specific comparison logic for results.
The benchmark is currently scoped to Snowflake and its specific set of Cortex AI functions, limiting generalizability to other platforms (BigQuery, Databricks) or function types. The construction relies on a single strong model (Claude Opus 4.5) for rewriting, which may introduce biases in instruction wording or task formulation. The evaluation uses a single agent trajectory per model, which does not capture the variance of stochastic agents or allow for pass@k analysis. Additionally, the benchmark relies on the assumption that the underlying AI functions remain stable over time, which may not hold if the cloud provider updates the backend models.
This paper has significant broader impact as it establishes the first standardized benchmark for evaluating text-to-SQL systems on AI-native workflows. As cloud data platforms increasingly integrate LLMs directly into their query engines, this benchmark will guide the development of more capable and reliable text-to-SQL systems. It highlights the unique challenges of AI-Native SQL, such as parameterization and semantic grounding, which are distinct from traditional text-to-SQL challenges. The findings will influence how researchers design agents and models for enterprise data analysis, potentially accelerating the adoption of AI-native SQL in industry. Spider 2.0-AIFunc introduces the first benchmark for evaluating text-to-SQL systems on AI-native SQL workflows, revealing that current models struggle with semantic parameterization and that complex agent frameworks offer no advantage over minimal setups, signaling a shift in the bottleneck of enterprise text-to-SQL from planning to semantic execution. The paper provides a rigorous evaluation framework and detailed error analysis that will serve as a foundational resource for the community as the industry transitions to AI-integrated database systems.
Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.
Primary: Amazon
All Institutions: Amazon
This paper presents a comprehensive and methodologically sound large-scale evaluation of uncertainty estimation in multilingual LLMs, providing crucial insights into the interplay between language resource levels, model scale, and uncertainty signaling mechanisms.
The paper proposes a rigorous evaluation framework for Uncertainty Estimation (UE) in Large Language Models (LLMs) across 22 languages. The core methodological innovation lies in the evaluation setup: using human-curated Multiple-Choice Question Answering (MCQA) datasets with elicited long-form reasoning to avoid the noise associated with LLM-as-a-judge or embedding-based correctness metrics. This allows for a clean, label-grounded assessment of UE methods (both open-box and closed-box) on the reasoning traces rather than just the final answer. The study systematically compares nine UE methods across varying model scales (270M to 235B) and resource levels. The approach is sound, addressing a critical gap in multilingual trustworthiness evaluation by isolating the uncertainty signal from the correctness signal more effectively than prior work.
The experimental design is comprehensive and large-scale. The authors evaluate 9 models and 9 UE methods across 22 languages, covering high, mid, and low-resource settings. Key findings include: (1) English reasoning significantly boosts UE performance for low-resource languages, suggesting the bottleneck is generation, not comprehension; (2) Self-Verbalized uncertainty outperforms other methods at large scales (235B), while open-box methods are better for smaller models; (3) Sampling-based methods fail on low-resource languages due to lack of diversity signal. The results are robust, supported by statistical significance testing and confidence intervals. The analysis of cross-lingual answer options and threshold calibration adds practical value. The use of parallel datasets (Global-MMLU and MMLU-ProX) ensures comparability.
The paper provides detailed descriptions of the datasets, models, prompts, and UE methods. It mentions the use of LM-Polygraph for implementation. The hardware infrastructure is specified. However, the specific versions of the models (e.g., "Claude 4.5 Sonnet" which appears to be a future/hypothetical name or typo for a current model, and "Gemma3" which is also not yet publicly released as of mid-2024, suggesting this is a very recent or forward-looking paper) and the exact random seeds are not fully detailed in the text provided, though the appendix references suggest they are available. The code release is mentioned but the URL is not provided in the text. The methodology is clear enough for replication if the models and datasets are accessible.
The study is limited to MCQA tasks, which may not fully generalize to open-ended generation where correctness is harder to define. The reliance on specific datasets (Global-MMLU, MMLU-ProX) means results might vary with other benchmarks. The "Claude 4.5 Sonnet" and "Gemma3" references are unusual and might indicate the paper is from the future or uses internal/unreleased models, which could limit immediate reproducibility for the broader community. The study does not include training-based UE methods.
This work has significant implications for deploying trustworthy LLMs in multilingual contexts. By demonstrating that generation language matters more than question language for UE, it provides actionable guidance for system designers (e.g., using English reasoning traces for low-resource languages). It also clarifies the trade-offs between open-box and closed-box methods based on model scale, helping practitioners choose appropriate UE strategies. The findings challenge the assumption that low-resource language performance is solely due to comprehension deficits, highlighting generation quality as a key factor. This paper presents a comprehensive and methodologically sound large-scale evaluation of uncertainty estimation in multilingual LLMs, providing crucial insights into the interplay between language resource levels, model scale, and uncertainty signaling mechanisms.