Last 7 Days (July 02 – July 08, 2026)
Crowdsourced fact-checking systems have been adopted by major social media companies such as X, Meta, TikTok and Google with the aim of combating misleading information at scale without relying on centralized editorial control. These systems have been developed around a common underlying concept: a bridging mechanism that identifies notes flagging misleading information when they receive support from people with different perspectives rather than simple majority support. To our knowledge the only publicly disclosed bridging algorithms deployed for fact-checking are based on matrix factorization, as deployed by both X and Meta, augmented with additional components addressing abuse, targeted manipulation, and contributor brigades. This work examines the core matrix factorization portion of these systems, presenting theoretical and empirical evaluations of the degree to which coordinated users could vote strategically by leveraging the latent representations to fabricate the appearance of synthetic consensus within the bridging mechanism. Using historic production data, we find that up to 10.7% of lower quality notes could be manipulated above consensus thresholds using less than 10 ratings. We complement these findings with a theoretical analysis, revealing counterintuitively that rating a note as "Not Helpful" can increase its helpfulness score, as well as a cost model quantifying manipulation effort. We have developed and deployed mitigations within X's Community Notes algorithm to address synthetic consensus.
Primary: Stanford University
All Institutions: Stanford University, X Community Notes, xAI
This work has significant broader impact, particularly for social media platforms and the field of adversarial machine learning. It proactively identifies a critical vulnerability in crowdsourced fact-checking systems (like X, Meta, TikTok, Google) that rely on matrix factorization for "bridging consensus." By demonstrating how coordinated adversaries can fabricate synthetic consensus, the paper highlights a fundamental challenge in designing robust, decentralized content moderation systems. The theoretical insights, especially the counterintuitive "Not Helpful" rating effect, contribute to a deeper understanding of MF-based systems. Most importantly, the paper's findings directly led to the development and *deployment of mitigations* (population sample filtering) within X's Community Notes algorithm, demonstrating a direct translation of research into real-world system improvements. This sets a high bar for impactful research in platform security and responsible AI, encouraging transparency and open collaboration between academia and industry to strengthen critical public-facing systems. This paper presents a rigorous analysis of coordinated manipulation in matrix factorization-based crowdsourced fact-checking, demonstrating a practical attack on production data and leading to the deployment of mitigations in X's Community Notes. The work combines theoretical derivations, empirical validation on a large-scale real-world dataset, and a practical cost model to expose a significant vulnerability in systems designed to combat misinformation, offering both novel insights into adversarial ML and direct, actionable solutions for platform security.
The paper presents a well-structured and rigorous methodology for analyzing coordinated manipulation in crowdsourced fact-checking systems, specifically focusing on the core matrix factorization (MF) component. The two-phase attack strategy is logically sound: first, adversarial accounts establish diverse positions in the latent factor space by strategically rating existing notes; second, these accounts coordinate to boost a target note's helpfulness score. This approach directly targets the "bridging" mechanism designed to ensure diverse agreement. The theoretical analysis of the Manipulation Resistance Score (MRS) is a significant contribution, providing a closed-form expression for the optimal single rating injection in a 1-dimensional factor space, which is the production setting for X. The derivation, detailed in the appendix, is thorough and correct. A particularly novel and counterintuitive finding is that rating a note as "Not Helpful" can, under specific conditions related to the geometry of existing ratings, increase its helpfulness score. This highlights a subtle vulnerability in the MF model. The cost model for the full attack provides a practical framework for understanding the economic feasibility of such manipulations and for evaluating potential mitigations. The methodology is strong in its combination of theoretical derivation, practical attack formulation, and cost analysis.
The experimental evaluation is robust and highly impactful due to its use of historic production data from X Community Notes (Jan 2021 - Jan 2025). This real-world dataset lends significant credibility to the findings. The ability to predict note parameters ($f_n, i_n$) from text using a Voyage embedding model and a shallow MLP is empirically demonstrated with reasonable accuracy, validating the feasibility of Phase 1 of the attack. The simulation showing that 100 adversarial accounts can achieve diverse factor positions across the spectrum $[-0.4, 0.4]$ further supports the attack's practicality. The quantification of MRS is a key empirical result, demonstrating that up to 10.7% of lower-quality notes could be manipulated above consensus thresholds using fewer than 10 ratings. This is a stark and actionable finding. The cost model, while simplified, provides concrete estimates (e.g., $30.50 for a single note manipulation) and effectively highlights the dominant cost factors (account maintenance). The paper also discusses the effectiveness of deployed mitigations, such as population sample filtering, which is a strong indicator of real-world impact. The experiments are well-designed to validate the theoretical claims and quantify the practical threat.
The paper demonstrates a strong commitment to reproducibility. It explicitly states that the analysis is based on the "open data and source code of X Community Notes," which facilitates independent study. The dataset used is publicly released, and the specific embedding model (Voyage-3-large) is identified. Hyperparameter and implementation details for the prediction model are promised in the appendix (though the appendix provided in the prompt is truncated before these details). The computational resources are specified, and the total wall-clock time for experiments is given. The full derivation for optimal rating injection is provided in the appendix. The authors also state that X deployed mitigations and released them as part of the open-source algorithm, further enhancing reproducibility and real-world impact.
The paper openly discusses several limitations. Firstly, it acknowledges that production Community Notes implementations include anti-abuse components (e.g., Correlated Rater Detection, Rater Engagement Intercept, Net Helpful Minimums) that are not fully incorporated into the core analysis. While these are discussed qualitatively, their quantitative impact on the attack's cost and success rate is not fully modeled. Secondly, the analysis is conducted in a static setting, not accounting for dynamic feedback loops where a surfaced "Helpful" note might attract more ratings, potentially changing its status. Thirdly, the MRS computation uses a greedy algorithm, which might be a conservative approximation compared to exact combinatorial optimization. Additionally, the note parameter prediction model uses only note text, ignoring post content or URLs, which could lead to underestimation of attacker capabilities. Finally, the cost model is a simplified abstraction and doesn't capture all nuances of attacker utility or sophisticated evasion strategies.
This work has significant broader impact, particularly for social media platforms and the field of adversarial machine learning. It proactively identifies a critical vulnerability in crowdsourced fact-checking systems (like X, Meta, TikTok, Google) that rely on matrix factorization for "bridging consensus." By demonstrating how coordinated adversaries can fabricate synthetic consensus, the paper highlights a fundamental challenge in designing robust, decentralized content moderation systems. The theoretical insights, especially the counterintuitive "Not Helpful" rating effect, contribute to a deeper understanding of MF-based systems. Most importantly, the paper's findings directly led to the development and *deployment of mitigations* (population sample filtering) within X's Community Notes algorithm, demonstrating a direct translation of research into real-world system improvements. This sets a high bar for impactful research in platform security and responsible AI, encouraging transparency and open collaboration between academia and industry to strengthen critical public-facing systems. This paper presents a rigorous analysis of coordinated manipulation in matrix factorization-based crowdsourced fact-checking, demonstrating a practical attack on production data and leading to the deployment of mitigations in X's Community Notes. The work combines theoretical derivations, empirical validation on a large-scale real-world dataset, and a practical cost model to expose a significant vulnerability in systems designed to combat misinformation, offering both novel insights into adversarial ML and direct, actionable solutions for platform security.
World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling. Meanwhile, unified vision-language models have demonstrated strong multimodal generation capabilities, yet their potential as world models remains underexplored. In this work, we introduce \texttt{WorldBagel}, a unified VLAW framework built on BAGEL, a modern multimodal unified model, and use it to systematically investigate the role of unification in world modeling. Across multi-task robotic manipulation and cross-domain experiments, \texttt{WorldBagel} consistently outperforms task-specific alternatives and learns action representations that are more structured and semantically aligned with visual and linguistic context. Experiments on LIBERO, Language Table, and Franka show that unification is not only an architectural convenience, but also a key factor in learning effective VLAW models, leading to consistent empirical gains and deeper insights into multimodal world modeling. Code and model checkpoints will be released upon acceptance.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
The paper introduces WorldBagel, a unified Vision-Language-Action-World (VLAW) framework built upon the BAGEL two-tower architecture. The core methodological contribution lies in extending a powerful multimodal generative model (BAGEL) to jointly support multimodal understanding, structured action modeling, and future world prediction. The VLAW formulation is clearly defined, aiming to model the joint distribution of future observations and actions conditioned on past states and language instructions. A significant technical contribution is the Fourier Feature Action Decoder (FFAD) and Fourier Feature Action Tokenizer (FFAT). FFAD addresses the limitations of standard regression and discretization-based action tokenizers by mapping continuous actions into Fourier features and predicting in this space. The inverse mapping uses phase-consistent averaging for reconstruction. This approach is well-justified with theoretical analysis provided in the appendix, demonstrating Lipschitz stability, injectivity, consistency of reconstruction, and approximation advantages. This mathematical rigor is a strong point. The interleaved VLAW modeling via sequence plans, adapted from BAGEL, is a practical and flexible way to structure multimodal sequences for multi-view, multi-step observations and control. The concept of sampling different sequence plans to balance training objectives is sound. Furthermore, the LLM-inspired multimodal train-time data sampling, using mixture dataset sampling and priority sequence-plan sampling, is a crucial engineering detail for stabilizing training across heterogeneous datasets and balancing policy learning with world modeling. The overall architecture leverages the strengths of BAGEL's GEN/UND experts, with action modeling integrated through fine-tuned tokenizers and decoders rather than a new expert. This design choice maintains the unified nature of the model.
The experimental evaluation is comprehensive and rigorous, addressing three key empirical findings: multi-task performance, action representation quality, and stability under distribution shifts. 1. **Multi-task Performance**: WorldBagel is evaluated on LIBERO, Language Table, and Franka benchmarks. On LIBERO, it achieves state-of-the-art multi-task manipulation performance (98.0% average success rate), outperforming strong VLA baselines like OpenVLA-OFT and RynnVLA-002. The world modeling capabilities are also quantitatively assessed using FVD, PSNR, SSIM, and LPIPS, showing consistent improvements over RynnVLA-002 across all datasets, especially in action-conditioned prediction. This clearly demonstrates the empirical gains of the unified VLAW approach. 2. **Action Representation Quality**: A detailed ablation study on action decoder design (regression, bin discretization, FAST, FFAD) on LIBERO shows FFAD significantly reduces action MSE and improves success rates. Further analysis on the number of Fourier bands (K) in FFAD/FFAT provides insights into optimal hyperparameter choices. Crucially, the representation structure analysis using a linear probe classifier reveals that FFAD produces more structured and task-relevant action embeddings, leading to higher task identity prediction accuracy. This is a strong validation of the FFAD design. 3. **Stability Under Distribution Shifts**: The paper investigates robustness to action noise, scaling, and temporal perturbations on LIBERO. WorldBagel consistently maintains higher prediction fidelity (PSNR, LPIPS) compared to RynnVLA-002 under these shifts. The eigenvalue spectrum analysis further supports this, showing WorldBagel learns richer and more stable action representations (higher effective rank, lower dominant eigenvalue ratio). This finding is particularly important for real-world robotics applications where such shifts are common. The choice of baselines is appropriate, including recent strong VLA models and a direct competitor (RynnVLA-002) that also aims for VLAW unification. The use of multiple metrics (success rate, FVD, PSNR, SSIM, LPIPS, A-MSE, linear probe accuracy, eigenvalue spectrum) provides a holistic view of the model's performance and internal properties. The experiments are well-designed to support the paper's claims about the benefits of unification.
The paper states that "Code and model checkpoints will be released upon acceptance," which is a positive commitment. Detailed hyperparameters (learning rate, weight decay, batch size, training steps, K for FFAT/FFAD, priority weights) and hardware (8 H200 GPUs) are provided, which are crucial for reproducibility. The mathematical derivations for FFAD/FFAT in the appendix also contribute to understanding and potentially re-implementing those components. Given the complexity of large multimodal models, the release of code and checkpoints is essential for full reproducibility.
1. **Computational Cost**: While not explicitly stated as a limitation, training and deploying a model built on a large unified multimodal backbone like BAGEL is inherently computationally intensive, requiring significant resources (e.g., 8 H200 GPUs for 80K steps). This might limit its applicability for resource-constrained environments or rapid iteration. 2. **Scope of World Modeling**: The "world modeling" aspect primarily focuses on next-frame prediction for manipulation tasks. While crucial, it doesn't delve into more abstract forms of world knowledge, causal reasoning, or long-horizon planning beyond short action rollouts, which are often goals of broader world models. 3. **Reliance on Supervised Fine-tuning**: The model relies on supervised fine-tuning (SFT) on existing robotic datasets. While effective, this approach might be limited by the diversity and scale of available demonstration data, potentially hindering generalization to truly novel tasks or environments compared to models that learn more extensively through self-supervision or interaction. 4. **Generalizability Beyond Manipulation**: The experiments are confined to robotic manipulation tasks. While these are challenging, the generalizability of "unified VLAW modeling" to other embodied AI domains (e.g., navigation, human-robot interaction) or even broader generative tasks is not explored.
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Primary: Stanford University
All Institutions: Stanford University, MIT, Nunchux AI, UC Berkeley, CMU
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
FourTune proposes an innovative and technically sound approach for fully 4-bit efficient post-training of diffusion models, addressing the critical challenges of memory footprint and training speed. The core of the methodology is the "triple-branch hybrid pipeline" which augments a standard LoRA architecture with a "frozen numerical stabilizer" (FNS). This FNS branch is a small, frozen, full-precision component designed to isolate and process quantization-sensitive outliers, thereby enabling stable training under native W4A4G4 (weights, activations, gradients) computation. This is a particularly clever design choice, as it mitigates the primary instability issue of low-bit quantization (outliers) without incurring significant computational overhead during training, as the FNS branch remains frozen. The integration of this FNS with both a standard BF16 LoRA branch and a quantized 4-bit LoRA branch provides a robust mechanism for maintaining quality while maximizing efficiency. Furthermore, the paper introduces hardware-efficient block-wise quantization and customized fused kernels. These components are crucial for translating the theoretical benefits of 4-bit quantization into practical speedups by optimizing memory bandwidth and accelerating quantized backpropagation operations (e.g., QGEMM, QMatmul, QConv). The end-to-end W4A4G4 paradigm, encompassing weights, activations, and gradients, is a comprehensive solution that pushes the boundaries of efficient fine-tuning for large generative models.
The experimental evaluation is comprehensive and robust, demonstrating FourTune's effectiveness across diverse and important post-training tasks for diffusion models: customization (DreamBooth-like), reinforcement learning for aesthetic preference, and knowledge distillation. The choice of FLUX.1-dev (12B parameters) as the primary model, along with validation on SDXL (1.5B), ensures the results are relevant for large-scale, state-of-the-art diffusion models. Baselines include BF16 LoRA and NF4 QLoRA, which are appropriate and strong competitors. The quantitative results are highly compelling: FourTune consistently matches or slightly outperforms the image quality of full-precision BF16 LoRA and NF4 QLoRA across all tasks, as measured by FID, CLIP Score, and LPIPS. This quality preservation at such low bit-depth is a significant achievement. More importantly, FourTune delivers substantial efficiency gains, reducing GPU memory overhead by 2.25x compared to BF16 LoRA (achieving a footprint comparable to NF4 QLoRA) and increasing end-to-end training throughput by 2.27x over BF16 LoRA and 2.79x over NF4 QLoRA. This effectively breaks the memory-speed trade-off often encountered in large model post-training. The ablation studies are well-designed and clearly demonstrate the critical contribution of each proposed component: the Frozen Numerical Stabilizer for stability and quality, and the W4A4G4 quantization, block-wise quantization, and fused kernels for efficiency. The generalization to SDXL further strengthens the claims.
The paper provides a good level of detail regarding the methodology, including the architecture of the triple-branch pipeline, the role of the FNS, and the types of quantization and fused kernels used. The experimental setup is well-described, including the specific models (FLUX.1-dev, SDXL), tasks, baselines, and evaluation metrics. The appendix further elaborates on implementation details, hyperparameters, and training configurations, which are crucial for reproducibility. While no direct code link is provided, the comprehensive description suggests that a skilled research team should be able to reproduce the results.
The paper does not explicitly discuss limitations, but some can be inferred. While the FNS effectively handles outliers, its design might introduce a slight architectural overhead compared to a purely 4-bit system, even if frozen. The customized fused kernels, while highly efficient, might require specific hardware support or careful implementation to achieve optimal performance across different GPU architectures. The evaluation is primarily focused on image generation diffusion models; its applicability and performance on other types of diffusion models (e.g., audio, video) or other generative architectures (e.g., GANs, VAEs) are not explored. The paper also doesn't delve into the potential challenges of deploying such a highly optimized, custom-kernel-dependent solution in diverse production environments.
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.
Primary: Stanford University
All Institutions: Stanford University, MIT, Nunchux AI, UC Berkeley, CMU
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
FourTune proposes an innovative and technically sound approach for fully 4-bit efficient post-training of diffusion models, addressing the critical challenges of memory footprint and training speed. The core of the methodology is the "triple-branch hybrid pipeline" which augments a standard LoRA architecture with a "frozen numerical stabilizer" (FNS). This FNS branch is a small, frozen, full-precision component designed to isolate and process quantization-sensitive outliers, thereby enabling stable training under native W4A4G4 (weights, activations, gradients) computation. This is a particularly clever design choice, as it mitigates the primary instability issue of low-bit quantization (outliers) without incurring significant computational overhead during training, as the FNS branch remains frozen. The integration of this FNS with both a standard BF16 LoRA branch and a quantized 4-bit LoRA branch provides a robust mechanism for maintaining quality while maximizing efficiency. Furthermore, the paper introduces hardware-efficient block-wise quantization and customized fused kernels. These components are crucial for translating the theoretical benefits of 4-bit quantization into practical speedups by optimizing memory bandwidth and accelerating quantized backpropagation operations (e.g., QGEMM, QMatmul, QConv). The end-to-end W4A4G4 paradigm, encompassing weights, activations, and gradients, is a comprehensive solution that pushes the boundaries of efficient fine-tuning for large generative models.
The experimental evaluation is comprehensive and robust, demonstrating FourTune's effectiveness across diverse and important post-training tasks for diffusion models: customization (DreamBooth-like), reinforcement learning for aesthetic preference, and knowledge distillation. The choice of FLUX.1-dev (12B parameters) as the primary model, along with validation on SDXL (1.5B), ensures the results are relevant for large-scale, state-of-the-art diffusion models. Baselines include BF16 LoRA and NF4 QLoRA, which are appropriate and strong competitors. The quantitative results are highly compelling: FourTune consistently matches or slightly outperforms the image quality of full-precision BF16 LoRA and NF4 QLoRA across all tasks, as measured by FID, CLIP Score, and LPIPS. This quality preservation at such low bit-depth is a significant achievement. More importantly, FourTune delivers substantial efficiency gains, reducing GPU memory overhead by 2.25x compared to BF16 LoRA (achieving a footprint comparable to NF4 QLoRA) and increasing end-to-end training throughput by 2.27x over BF16 LoRA and 2.79x over NF4 QLoRA. This effectively breaks the memory-speed trade-off often encountered in large model post-training. The ablation studies are well-designed and clearly demonstrate the critical contribution of each proposed component: the Frozen Numerical Stabilizer for stability and quality, and the W4A4G4 quantization, block-wise quantization, and fused kernels for efficiency. The generalization to SDXL further strengthens the claims.
The paper provides a good level of detail regarding the methodology, including the architecture of the triple-branch pipeline, the role of the FNS, and the types of quantization and fused kernels used. The experimental setup is well-described, including the specific models (FLUX.1-dev, SDXL), tasks, baselines, and evaluation metrics. The appendix further elaborates on implementation details, hyperparameters, and training configurations, which are crucial for reproducibility. While no direct code link is provided, the comprehensive description suggests that a skilled research team should be able to reproduce the results.
The paper does not explicitly discuss limitations, but some can be inferred. While the FNS effectively handles outliers, its design might introduce a slight architectural overhead compared to a purely 4-bit system, even if frozen. The customized fused kernels, while highly efficient, might require specific hardware support or careful implementation to achieve optimal performance across different GPU architectures. The evaluation is primarily focused on image generation diffusion models; its applicability and performance on other types of diffusion models (e.g., audio, video) or other generative architectures (e.g., GANs, VAEs) are not explored. The paper also doesn't delve into the potential challenges of deploying such a highly optimized, custom-kernel-dependent solution in diverse production environments.
FourTune has a significant broader impact on the field of machine learning, particularly for generative AI. By enabling efficient 4-bit post-training of large diffusion models without compromising quality, it democratizes access to these powerful models. Researchers and practitioners with limited computational resources can now fine-tune large models faster and with less memory, accelerating research, development, and deployment of customized generative AI applications. This could lead to a proliferation of new use cases for diffusion models in various domains, from content creation to scientific discovery. The techniques developed, especially the triple-branch hybrid pipeline with the frozen numerical stabilizer, could inspire similar efficient training strategies for other large foundation models beyond diffusion models, such as large language models. The paper also includes a standard ethical impact statement, acknowledging both positive and negative societal implications of generative AI. FourTune presents a highly impactful method for fully 4-bit efficient post-training of diffusion models, achieving state-of-the-art efficiency while maintaining full-precision quality through a novel triple-branch hybrid pipeline and hardware-optimized quantization. This work significantly advances the accessibility and practical applicability of large diffusion models, enabling faster iteration and broader adoption in diverse downstream tasks.
Boosting is a fundamental technique for generically improving the accuracy of learning algorithms (Schapire 1989). Existing boosting algorithms construct a strong learner using $O(\log(\frac{1}ε)/γ^2)$ calls to a $γ$-advantage weak learner, and this round complexity is known to be optimal for generic boosters that succeed on all concept classes (Freund 1995). We show that this lower bound can be circumvented for concept classes that satisfy a mild closure property. Specifically, we present a new boosting algorithm that, for any class $\mathcal{F}$ closed under $O(\log \frac{1}γ)$-XOR, strong learns $\mathcal{F}$ using $O(\log \frac{1}ε)$ calls to a $γ$-advantage weak learner and a single batch of $\tilde{O}(\log(\frac{1}ε)/γ^2)$ additional samples. Our algorithm arises from a new and simple connection between boosting and list-decodable codes. Viewing the target function as a message, we run the weak learner on its encoding and view the resulting weak hypothesis as a corrupted codeword. Feeding this corrupted codeword to a list decoder, we obtain a small list of candidate hypotheses, at least one of which is a strong hypothesis for the original function. Using additional samples, we identify and output this strong hypothesis.
Primary: Stanford University
All Institutions: Stanford University, University of California, Berkeley, Simons Foundation
This paper presents a significant theoretical advance in boosting by circumventing the optimal round complexity lower bound for generic boosters, achieving $O(1)$ weak learner calls for XOR-closed concept classes through a novel application of list-decodable codes. The work is rigorous, highly novel in its theoretical approach, and addresses a fundamental question in learning theory, warranting a high score for its contribution to the field's understanding of computational efficiency and sample complexity trade-offs.
The paper proposes a theoretically significant breakthrough in boosting by circumventing the standard $O(1/\gamma^2)$ round complexity lower bound. The core methodology involves a novel connection between boosting and list-decodable codes. By encoding the target function using an XOR code and treating the weak learner's output as a corrupted codeword, the algorithm employs a local list decoder to generate a small candidate list of hypotheses. A subsequent filtering step using additional samples identifies the correct strong hypothesis. This approach is mathematically elegant and rigorously proven to achieve $O(1)$ calls to the weak learner for concept classes closed under XOR, trading round complexity for a batch of additional samples. The theoretical derivation is sound, leveraging results from coding theory (specifically local list decoding for the XOR code) to establish new bounds in computational learning theory.
The paper is purely theoretical and contains no empirical experiments, benchmarks, or datasets. The "results" are formal theorems proving the existence and efficiency of the boosting algorithm. Consequently, there is no experimental evaluation to assess in the traditional sense. The validity of the claims rests entirely on the mathematical proofs provided.
As a theoretical work, reproducibility refers to the reproducibility of the proofs. The paper provides detailed proofs for the main theorems, including the construction of the list decoder and the boosting algorithm. The dependencies on existing coding theory results are cited. However, the "algorithm" described is a theoretical construct; implementing it would require specific instantiations of the weak learner and the list decoder which are not provided as code. The theoretical framework is clear enough for other theorists to verify.
The primary limitation is the restriction to concept classes that are closed under $O(\log(1/\gamma))$-XOR. While the authors argue this is a "mild" closure property, it excludes many standard learning settings where such closure does not hold. Furthermore, the algorithm requires a "single batch" of additional samples of size $\tilde{O}(1/\gamma^2)$, which might be computationally expensive or data-intensive in practice, potentially offsetting the benefit of reduced round complexity. The paper does not address the computational complexity of the list decoding step or the filtering step in terms of time complexity, only sample complexity.
This work has significant implications for the theoretical foundations of machine learning. By breaking a long-standing lower bound, it opens new avenues for designing efficient boosting algorithms in specific structured settings. It highlights the deep connections between coding theory and learning theory, potentially inspiring new cross-disciplinary research. However, its immediate practical impact is limited due to the theoretical nature and specific constraints of the method. It serves as a foundational result for future work in efficient learning algorithms. This paper presents a significant theoretical advance in boosting by circumventing the optimal round complexity lower bound for generic boosters, achieving $O(1)$ weak learner calls for XOR-closed concept classes through a novel application of list-decodable codes. The work is rigorous, highly novel in its theoretical approach, and addresses a fundamental question in learning theory, warranting a high score for its contribution to the field's understanding of computational efficiency and sample complexity trade-offs.
Recent LLM-based mathematical reasoning agents have begun to tackle research-level problems and, in several cases, have contributed to the resolution of open problems. However, scaling and orchestrating such agents effectively remains challenging, due to the difficulty of coordinating parallel proof search while keeping intermediate claims organized and reliable. In this paper, we propose Danus, an orchestration system for research-level mathematical reasoning centered on a shared fact graph as a global memory-management mechanism. Danus consists of a main agent that performs planning and coordination, multiple worker agents that carry out proof search in parallel, and a stateless verifier that checks proposed mathematical claims before they are admitted into the fact graph. Each verified fact is stored together with its proof and logical dependencies, allowing the system to build long arguments incrementally while keeping the shared proof state organized. The main agent periodically summarizes the evolving proof state, redirects workers across promising directions, and supports interaction with human mathematicians through progress reports. We evaluate Danus through six research-level case studies in algebraic geometry, singularity theory, and combinatorics, illustrating how the fact-graph memory mechanism enables Danus to construct long, detailed mathematical proofs. Our results suggest that fact-graph-based orchestration provides an effective route toward scaling mathematical reasoning agents for long-horizon research problems. Danus is open source at https://github.com/frenzymath/Danus.
Primary: Great Bay Institute for Advanced Study
All Institutions: Great Bay Institute for Advanced Study, Beijing International Center for Mathematical Research, New Cornerstone Science Laboratory, Center for Intelligent Computing, Center for Machine Learning Research, Department of Mathematics, Great Bay University, Guoxiong Gao, Jihao Liu, Key Laboratory of Intelligent Computing and Applications (Ministry of Education), Kyoto University, Peking University, Research Institute for Mathematical Sciences, School of Mathematical Sciences, School of Mathematics, Stanford University, Tianjin University, Tongji University, Westlake Institute for Advanced Study, Westlake University, Zeming Sun, Zhongguancun Academy
Danus introduces a novel fact-graph-based orchestration system for multi-agent mathematical reasoning, demonstrating significant technical impact by enabling the automated resolution of complex, research-level mathematical problems through structured memory management and parallel verification.
The paper introduces Danus, a multi-agent orchestration system designed for research-level mathematical reasoning. The core innovation is the "fact graph," a shared, directed acyclic graph (DAG) that serves as a global memory mechanism. This graph stores verified mathematical facts (statements with proofs) and their logical dependencies. The system employs a strict separation of powers: a main agent (orchestrator) plans and coordinates, multiple worker agents (reasoners) explore different proof paths in parallel, and a stateless verifier ensures correctness before facts are added to the graph. This architecture addresses the critical challenge of scaling multi-agent systems by preventing context confusion and enabling incremental, verifiable proof construction. The use of a DAG allows for complex dependency tracking and revocation of invalid facts, which is crucial for maintaining logical integrity in long-horizon reasoning tasks.
The evaluation consists of six case studies in advanced mathematics, including algebraic geometry, singularity theory, and combinatorics. These are not standard benchmarks but open or semi-open research problems. Danus successfully resolved several problems, such as the optimal bend-and-break for foliations and the total Cartier indices of rational singularities. Notably, in the "Tangent classes of matroids" case, Danus solved a problem that previous systems (Rethlas, GPT-5.5-pro) failed to solve, demonstrating the efficacy of its orchestration and memory management. The results are qualitative and illustrative rather than statistical, which is appropriate for this type of exploratory research but limits broad generalizability. The system's ability to produce complete, verified proofs for complex theorems is the primary metric of success.
The paper provides an open-source repository (https://github.com/frenzymath/Danus), which significantly enhances reproducibility. The methodology is described in detail, including the roles of different agents, the structure of the fact graph, and the verification process. However, the reliance on specific proprietary models (GPT-5.5-pro, Claude Opus 4.8) and the specific mathematical literature retrieval tools (Matlas) may pose barriers to exact replication. The case studies involve human-in-the-loop interactions, which are difficult to fully replicate but are documented.
The system is heavily dependent on the capabilities of the underlying LLMs and the verifier. The verifier, while effective, is not perfect and may accept proofs with minor skipped steps or rely on potentially erroneous references if not caught in final review. The system's performance is limited by the cost and latency of running multiple agents and verifying facts. Additionally, the current evaluation is limited to a small number of high-difficulty mathematical problems, and it is unclear how the system scales to other domains or less structured reasoning tasks. The reliance on human input for problem formulation and final verification is also a limitation for full autonomy.
Danus represents a significant step towards autonomous scientific discovery, particularly in mathematics. By demonstrating that AI systems can contribute to the resolution of open research problems, it challenges traditional notions of human-AI collaboration in science. The fact-graph memory mechanism could be adapted for other complex reasoning tasks requiring long-term dependency tracking and verification, such as legal reasoning or software verification. However, the potential for misuse in generating plausible-sounding but incorrect mathematical proofs remains a concern, necessitating robust verification mechanisms. Danus introduces a novel fact-graph-based orchestration system for multi-agent mathematical reasoning, demonstrating significant technical impact by enabling the automated resolution of complex, research-level mathematical problems through structured memory management and parallel verification.
Logit-based watermarking is a widely used mechanism for identifying LLM generated content, yet its effectiveness is governed by a fundamental trade-off between detectability and semantic distortion. Existing analyses provide limited guidance for principled hyperparameter selection, leaving practical deployments reliant on heuristic tuning. In this work, we develop a power-calibrated statistical framework that establishes explicit quantitative relationships between watermark hyperparameters, detection power, and distortion. This characterization transforms watermark design into a guided optimization problem. Building on these results, we derive practical parameter selection procedures that achieve optimal tradeoffs under constraints. Extensive experiments across multiple language models and datasets validate the theory and demonstrate that the proposed framework consistently identifies Pareto-optimal points.
Primary: Department of Statistics, Pennsylvania State University, University Park
All Institutions: Department of Statistics, Pennsylvania State University, University Park
The paper's broader impact is significant and directly addresses critical societal concerns related to the proliferation of large language models. 1. **Combating Misinformation**: By improving the efficiency and reliability of LLM watermarking, this work strengthens the ability to identify machine-generated content. This is crucial in the fight against misinformation, fake news, and propaganda campaigns that leverage generative AI. 2. **Academic Integrity**: Watermarking can help detect AI-generated content in academic submissions, supporting academic honesty and preventing misuse of LLMs for plagiarism or automated assignment completion. 3. **Ethical AI Deployment**: It provides a mechanism for establishing provenance, which is essential for responsible and transparent deployment of generative AI. Users can be informed whether content originates from an AI, fostering trust and accountability. 4. **Scalability and Practicality**: By transforming watermark design from heuristic tuning to a statistically grounded optimization problem, the framework makes watermarking systems more robust, reliable, and easier to deploy in real-world applications. This improved efficiency and principled parameter selection can lead to wider adoption and more effective use of watermarking technologies. 5. **Research Foundation**: The rigorous statistical framework provides a strong theoretical foundation for future research in LLM watermarking, enabling more principled development and analysis of new techniques. It encourages a shift from empirical trial-and-error to theoretically informed design. The work directly contributes to mitigating potential harms of generative AI while enabling its beneficial uses, aligning with responsible AI development principles. This paper introduces a controllable statistical framework for logit-based LLM watermarking, enabling principled calibration of watermark strength under explicit detectability and distortion objectives. By establishing quantitative mappings between watermark parameters, detection power, and KL-based distortion, the authors transform watermark design from heuristic tuning into a statistically grounded optimization problem, validated through extensive experiments across multiple language models and datasets, consistently identifying Pareto-optimal configurations.
The paper introduces a power-calibrated statistical framework for logit-based LLM watermarking, moving beyond heuristic hyperparameter tuning. The core methodology involves establishing explicit quantitative relationships between watermark hyperparameters (bias $\delta$ and green-list fraction $\gamma$), detection power, and semantic distortion (measured by KL divergence). Key methodological steps include: 1. **Formalizing Hypothesis Testing**: The paper frames watermark detection as a hypothesis test, defining the null ($H_0$: unwatermarked text) and alternative ($H_1$: watermarked text) hypotheses based on the green-list token probability. 2. **Assumptions for Tractability**: To derive closed-form expressions, the framework relies on several assumptions: * **Random green-list assignments**: Green lists are i.i.d. and independent of the generated token sequence, simplifying the null hypothesis to an i.i.d. Bernoulli process for green-token indicators. * **Non-informative NTP prior**: The next-token probability (NTP) vector is modeled as an independent draw from a uniform Dirichlet distribution. The paper acknowledges this is a simplification but argues for its effectiveness due to the detector's dependence on green-list mass, which concentrates around $\gamma$. * **Information Decay**: Assumes a geometric decay in mutual information between past and future green-token indicators, enabling the application of a Central Limit Theorem for $\alpha$-mixing sequences under the alternative hypothesis. * **Non-degenerate Long-run Variance**: Ensures the asymptotic variance of the indicator sequence is strictly positive. 3. **Derivation of Key Metrics**: * **Green-token probability under watermarking ($\gamma'$)**: Lemma 2.2 and Theorem 2.3 provide a precise, token-level description of how watermarking biases generation toward the green list, leading to a closed-form expression for $\gamma'$ in terms of $\delta$ and $\gamma$. * **Detection Power**: Using the normal approximation for the aggregate statistic $S_n$ under both $H_0$ and $H_1$, a closed-form expression for statistical power $\Psi^*(\delta, \gamma)$ is derived. The paper notes that a constant $c$ (long-run variance inflation) affects the numerical value but not the parameter selection for maximization. * **Distortion (KL Divergence)**: Lemma 3.1 provides a plug-in formula for the expected token-wise KL divergence $D_{KL}(\delta, \gamma)$ between watermarked and unwatermarked distributions. It proves strict monotonicity in $\delta$ for fixed $\gamma$, allowing $\delta$ to be parameterized by a distortion budget. 4. **Optimization Framework**: The theoretical characterization transforms watermark design into a guided optimization problem. Instead of tuning two hyperparameters $(\delta, \gamma)$, the problem is reduced to a 1D numerical optimization (e.g., maximizing power subject to a KL distortion budget $K_0$) because $\delta$ can be implicitly determined by $K_0$. Practical guidance is provided for initializing the search for $\gamma$. The methodology is statistically rigorous, building on established theorems (CLT for mixing sequences) and providing detailed proofs in the appendix. The reduction of a multi-dimensional heuristic search to a principled 1D optimization problem under constraints is a significant practical contribution.
The experimental evaluation is comprehensive and well-structured, providing strong empirical validation for the theoretical framework. 1. **Setup**: The protocol largely follows prior watermarking evaluations, using diverse LLMs (OPT, Pythia, GPT-2, and Gemma-2 9B in the appendix) and datasets (C4, LFQA, Wikipedia). Generations are kept short ($n=50$ tokens) to avoid trivial detectability gains and better expose statistical efficiency differences. All methods are implemented using a unified pipeline based on the KGW codebase to ensure fair comparison. 2. **Distributional Verification**: * **Normality**: Q-Q plots confirm that the standardized green-token count statistic closely follows a standard normal distribution under both $H_0$ and $H_1$, supporting the normal approximation used in the detectability analysis. * **Alternative Hypothesis Characterization**: Empirical green-token rates are compared against theoretical predictions from Theorem 2.3. A near-linear relationship with $R^2 > 0.98$ across all settings validates the simplified modeling assumptions for $\gamma'$. 3. **Performance Evaluation (Detectability-Distortion Trade-off)**: * The paper evaluates statistical power (TPR at $\alpha=0.05$) against semantic distortion (per-token KL divergence). * Results across various models and datasets consistently show that the proposed method lies on the Pareto frontier. It achieves high detection power at substantially lower distortion compared to baseline methods (KGW, DP, and dense grid search). * The approach maintains near-saturated TPR in moderate distortion regimes where baselines exhibit significant degradation. This highlights improved statistical efficiency. 4. **Robustness under Alternative Quality Metrics**: * To assess the generalizability of the optimized parameters, the method is evaluated using complementary semantic quality metrics: BLEU, ROUGE, and BERTScore. * The results show a consistent pattern: the proposed method maintains strong detection power over a wider range of quality values compared to baselines. This suggests that the optimized parameterization induces a more uniform and controlled distributional shift, not tied to a specific notion of text similarity. Overall, the experiments are thorough, well-designed, and effectively support the theoretical claims. The validation of assumptions, extensive comparisons, and robustness checks significantly strengthen the paper's conclusions.
The paper demonstrates a strong commitment to reproducibility. 1. **Code Availability**: The authors explicitly state that "The implementation is available at https://github.com/shooof/wm." This is a crucial step for reproducibility. 2. **Detailed Experimental Setup**: The "Experimental Setup" section provides clear details on the language models (OPT, Pythia, GPT-2, Gemma-2 9B), datasets (C4, LFQA, Wikipedia), generation length ($n=50$), and detection parameters ($\alpha=0.05$). 3. **Baseline Descriptions**: Baseline methods (KGW, DP, dense grid search) are clearly identified, and their implementation is stated to use the same unified generation and detection pipeline as the proposed method, controlling for implementation effects. 4. **Hyperparameter Ranges**: Appendix provides a table of hyperparameter ranges used for all methods, which is essential for replicating the search space. 5. **Metric Definitions**: Clear definitions and implementations for distortion (KL divergence) and quality metrics (BLEU, ROUGE, BERTScore) are provided. 6. **Theoretical Proofs**: The appendix contains detailed proofs for all lemmas and theorems, allowing for independent verification of the mathematical derivations. The combination of open-source code, detailed experimental descriptions, and theoretical proofs makes the work highly reproducible.
The paper acknowledges limitations, with a dedicated discussion mentioned in Appendix [REF] (though the provided text truncates before this appendix content). Based on the main text, potential limitations include: 1. **Assumptions for Tractability**: The framework relies on several simplifying assumptions: * **Random green-list assignments**: While reasonable for pseudo-random hash functions, real-world implementations might have subtle biases. * **Non-informative NTP prior (Uniform Dirichlet)**: The paper argues for its effectiveness, but it is an approximation. While it mentions that richer priors (mixtures of Dirichlet) could be used if structural information is available, the current analysis is based on a simplified prior. The accuracy of this approximation for all possible LLM output distributions might vary. * **Information Decay**: The assumption of geometric decay in mutual information is a model for the autoregressive dependence. While standard for mixing sequences, its precise fit to complex LLM generation dynamics is an approximation. 2. **Estimation of Constant 'c'**: The power formula includes a constant $c$ that captures long-run variance inflation due to dependence. While the paper states that parameter selection is independent of $c$, its numerical value is needed for absolute power predictions and needs to be estimated in practice (as mentioned in Appendix [REF]). This introduces an additional practical step and potential source of error. 3. **Focus on Logit-Based Watermarking**: The framework is specifically developed for logit-based watermarking (KGW framework). While this is a widely adopted paradigm, the theoretical derivations might not directly apply to other watermarking schemes without significant adaptation. 4. **Short Sequence Evaluation**: While justified to isolate statistical efficiency, the primary experiments are on short sequences ($n=50$). While the paper discusses long-form generation in the appendix, the main empirical results are on short sequences. 5. **Adversarial Robustness**: While the appendix briefly discusses "signal-removal robustness" using a WinMax-C detector, a more extensive analysis of various adversarial attacks (e.g., paraphrasing, fine-tuning, token manipulation) and how the calibrated parameters perform under these attacks would be valuable. The current framework optimizes for detectability and distortion in a "clean" setting.
The paper's broader impact is significant and directly addresses critical societal concerns related to the proliferation of large language models. 1. **Combating Misinformation**: By improving the efficiency and reliability of LLM watermarking, this work strengthens the ability to identify machine-generated content. This is crucial in the fight against misinformation, fake news, and propaganda campaigns that leverage generative AI. 2. **Academic Integrity**: Watermarking can help detect AI-generated content in academic submissions, supporting academic honesty and preventing misuse of LLMs for plagiarism or automated assignment completion. 3. **Ethical AI Deployment**: It provides a mechanism for establishing provenance, which is essential for responsible and transparent deployment of generative AI. Users can be informed whether content originates from an AI, fostering trust and accountability. 4. **Scalability and Practicality**: By transforming watermark design from heuristic tuning to a statistically grounded optimization problem, the framework makes watermarking systems more robust, reliable, and easier to deploy in real-world applications. This improved efficiency and principled parameter selection can lead to wider adoption and more effective use of watermarking technologies. 5. **Research Foundation**: The rigorous statistical framework provides a strong theoretical foundation for future research in LLM watermarking, enabling more principled development and analysis of new techniques. It encourages a shift from empirical trial-and-error to theoretically informed design. The work directly contributes to mitigating potential harms of generative AI while enabling its beneficial uses, aligning with responsible AI development principles. This paper introduces a controllable statistical framework for logit-based LLM watermarking, enabling principled calibration of watermark strength under explicit detectability and distortion objectives. By establishing quantitative mappings between watermark parameters, detection power, and KL-based distortion, the authors transform watermark design from heuristic tuning into a statistically grounded optimization problem, validated through extensive experiments across multiple language models and datasets, consistently identifying Pareto-optimal configurations.
Hierarchical mixture models are a powerful tool for modeling data generated from heterogeneous sources, particularly when the mixing proportion $\boldsymbol{w}$ itself is treated as a random variable with a Dirichlet or Beta-Liouville prior. Such models are widely employed in scenarios where uncertainty in class membership or data-generating processes must be probabilistically quantified. This paper studies the exact marginalization of the mixture weight. For the two-component case we give an $O(n^2)$ dynamic program -- and an $O(n \log^2 n)$ FFT variant -- for the marginal likelihood, and show that the exact posterior of the weight is a finite mixture of Beta distributions, delivering closed-form posterior summaries, credible intervals and per-observation local false-discovery rates without any sampling. For $K \ge 3$ components we give an exact joint dynamic program. The gain is largest in the small-sample regime the method is built for: on a real multilevel meta-analysis, a pathway-level dysregulation analysis of leukemia gene expression, and a leukemia-derived gene-panel benchmark with known ground truth, the exact interval for the signal proportion is calibrated where EM gives no interval at all (collapsing to a boundary) and Gaussian/Laplace approximations mis-cover, and it is two orders of magnitude faster than the sampler that would match it. On the large prostate-cancer benchmark, where every method has ample data, it agrees with locfdr on the gene ranking while adding a posterior interval for the null proportion.
Primary: unknown
All Institutions: unknown
This paper has significant broader impact for any field employing hierarchical mixture models, especially where accurate uncertainty quantification in small-sample or rare-signal regimes is critical. 1. **Multiple Testing and FDR Estimation**: It provides a robust, exact method for estimating global null proportions and local false discovery rates, offering calibrated credible intervals that are superior to point estimates or miscalibrated approximations. 2. **Meta-analysis and A/B Testing**: In settings with multiple small studies or experiments, the method can provide reliable estimates of overall effects and their uncertainty, overcoming limitations of EM and MCMC. 3. **Bayesian Inference**: It demonstrates that exact Bayesian inference for certain complex models is achievable deterministically, without resorting to sampling methods, opening avenues for more reliable and efficient analysis. 4. **Theoretical Foundations**: The derived structural properties deepen the theoretical understanding of mixture model posteriors, linking them to classical combinatorial and polynomial theory. This paper presents a highly significant contribution to the field of Bayesian inference and statistical modeling. It provides exact, deterministic, and numerically stable algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, addressing long-standing limitations of approximate methods (EM) and computationally intensive sampling methods (MCMC). The rigorous mathematical derivations, coupled with comprehensive experimental validation demonstrating superior calibration and speed, make this a valuable tool for practitioners and a strong theoretical advance. The paper's main contribution is the development of exact, deterministic algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, yielding calibrated uncertainty quantification and closed-form summaries without sampling. This work provides a rigorous mathematical framework and efficient computational methods (dynamic programming) to overcome the limitations of existing approximate and sampling-based approaches, particularly in small-sample and rare-signal scenarios, offering significant practical benefits for applications like multiple testing and meta-analysis.
This paper presents a highly rigorous and elegant methodology for the exact computation of the posterior distribution of mixture weights in hierarchical Bayesian models, specifically for Dirichlet or Beta-Liouville priors. The core contribution lies in transforming the marginalization problem into a combinatorial sum that can be efficiently computed. For the two-component case, the authors derive an $O(n^2)$ dynamic programming (DP) algorithm, which is a significant improvement over the naive $O(2^n)$ enumeration. They also present an $O(n \log^2 n)$ FFT-based variant, though they acknowledge its numerical instability in practice. A key theoretical result is showing that the exact posterior of the mixture weight is a finite mixture of Beta distributions, enabling closed-form posterior summaries, credible intervals, and local false-discovery rates without any sampling. The paper further establishes three crucial structural properties of this exact posterior: 1. **Log-concavity and unimodality**: The posterior over the latent count of non-null observations is log-concave and unimodal, ensuring coherent credible intervals. This property stems from Newton's inequalities for elementary symmetric polynomials. 2. **Evidence-ratio moment identity**: Posterior raw moments of the mixing weight can be obtained as ratios of the marginal likelihood at shifted hyperparameters, allowing for efficient computation of moments without summation over the mixture. 3. **Monotone response to data**: The posterior is stochastically increasing in every likelihood ratio, ensuring that strengthening evidence for a signal coherently increases the inferred prevalence and maintains the ranking of local FDRs. For $K \ge 3$ components, the paper generalizes the approach by providing an exact joint dynamic program with $O(K n^K)$ time and $O(n^{K-1})$ storage complexity. This multivariate DP handles the non-factorization of coefficients across components, a challenge for $K \ge 3$. The use of log-domain computation throughout ensures numerical stability, a critical aspect for practical application. The mathematical derivations are thorough, leveraging combinatorial identities and properties of Stirling numbers.
The experimental evaluation is comprehensive and well-structured. 1. **Correctness**: The exact marginal likelihood algorithms are validated against multiple independent ground truth methods: naive $2^n$ enumeration, adaptive and high-precision quadrature, and Monte Carlo integration. The agreement is shown to be at floating-point round-off level, confirming the correctness of the proposed methods. 2. **Scaling and Numerical Stability**: The $O(n^2)$ log-domain DP is shown to be highly efficient and numerically stable, outperforming the intractable $2^n$ enumeration by orders of magnitude and remaining accurate for large $n$ (up to $n=6033$ in tests). The FFT variant, while theoretically faster, is demonstrated to be numerically unreliable due to combinatorial growth of coefficients, highlighting the practical importance of the log-domain DP. 3. **Comparison with EM and MCMC**: The exact method is benchmarked against a data-augmentation Gibbs sampler, PyMC's NUTS, and MAP-EM. It achieves posterior means and standard deviations that match the Gibbs sampler to three figures, but is two orders of magnitude faster. Crucially, it provides calibrated uncertainty where EM gives only point estimates (often collapsing to boundaries) and Gaussian/Laplace approximations mis-cover, particularly in the small-sample, rare-signal regime. 4. **Frequentist Coverage**: A frequentist study demonstrates that the exact Bayesian credible intervals maintain nominal coverage even when EM and approximate Bayesian methods fail (e.g., in small-sample, rare-signal scenarios). This is a strong validation of the method's calibration. 5. **Real-world Applications**: The method is applied to a multilevel meta-analysis, a pathway-level dysregulation analysis of leukemia gene expression, and a prostate-cancer microarray dataset. These applications showcase its practical utility, providing calibrated intervals and posterior summaries that are unavailable or unreliable from existing methods, while also agreeing with established methods like locfdr on gene rankings in large-sample settings.
The paper explicitly states that "A reference Python implementation reproducing every figure and table of this section is provided (sec:reproducibility)". While this indicates a commitment to reproducibility, an explicit URL to the code repository is not provided in the paper text. Assuming the code is indeed available as stated, the reproducibility would be high. Without the URL, it's difficult to verify immediately.
1. **Scalability for Large K**: The $O(K n^K)$ time complexity for $K \ge 3$ components limits its applicability to scenarios with a small number of mixture components. While the authors argue that $K=2$ or $K=3$ are common in practice, this is a significant constraint for models requiring many components. 2. **Fixed Component Parameters**: The method focuses solely on the exact marginalization of mixture weights, assuming the component distributions ($f$ and $g$) are known or their parameters are fixed. It does not address the joint estimation of component parameters along with the mixture weights, which is a common challenge in mixture modeling. 3. **Numerical Stability of FFT Variant**: The theoretically faster $O(n \log^2 n)$ FFT variant for $K=2$ is shown to be numerically unstable in practice, making the $O(n^2)$ DP the recommended backend. This means the best practical complexity is quadratic, not quasi-linear.
This paper has significant broader impact for any field employing hierarchical mixture models, especially where accurate uncertainty quantification in small-sample or rare-signal regimes is critical. 1. **Multiple Testing and FDR Estimation**: It provides a robust, exact method for estimating global null proportions and local false discovery rates, offering calibrated credible intervals that are superior to point estimates or miscalibrated approximations. 2. **Meta-analysis and A/B Testing**: In settings with multiple small studies or experiments, the method can provide reliable estimates of overall effects and their uncertainty, overcoming limitations of EM and MCMC. 3. **Bayesian Inference**: It demonstrates that exact Bayesian inference for certain complex models is achievable deterministically, without resorting to sampling methods, opening avenues for more reliable and efficient analysis. 4. **Theoretical Foundations**: The derived structural properties deepen the theoretical understanding of mixture model posteriors, linking them to classical combinatorial and polynomial theory. This paper presents a highly significant contribution to the field of Bayesian inference and statistical modeling. It provides exact, deterministic, and numerically stable algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, addressing long-standing limitations of approximate methods (EM) and computationally intensive sampling methods (MCMC). The rigorous mathematical derivations, coupled with comprehensive experimental validation demonstrating superior calibration and speed, make this a valuable tool for practitioners and a strong theoretical advance. The paper's main contribution is the development of exact, deterministic algorithms for computing the posterior distribution of mixture weights in hierarchical Bayesian models, yielding calibrated uncertainty quantification and closed-form summaries without sampling. This work provides a rigorous mathematical framework and efficient computational methods (dynamic programming) to overcome the limitations of existing approximate and sampling-based approaches, particularly in small-sample and rare-signal scenarios, offering significant practical benefits for applications like multiple testing and meta-analysis.
Graph-based semi-supervised learning (SSL) propagates a few labels over a similarity graph by minimizing a Dirichlet-type energy. The standard quadratic ($p=2$) energy reduces to a single graph-Laplacian solve, but it degenerates exactly where SSL is most useful when labels are scarce: gathering more unlabeled data drives the $p=2$ estimate to a near-constant function whenever $d\ge2$ (Nadler-Srebro-Zhou). Well-posedness requires the nonlinear $p$-Laplacian energy with $p>d$. Existing solvers reduce this to a sequence of weighted Laplacian solves, but their reference implementations use a direct sparse factorization or ichol-preconditioned CG instead. Plugging a near-linear Laplacian solver is not straightforward: at large $p$ the conductance weights degenerate near flat-gradient edges, making the system nearly singular and causing stagnation without a damped outer iteration. We close this gap. Recasting $p$-Laplacian SSL as a source-form nonlinear Laplacian flow $Bρ_p(B^\top x)=b$ and solving by damped chord-Newton continuation in $p$, every linearized system stays well-conditioned and can be delegated to a near-linear Laplacian engine. On size-scaled graph families the wall-clock is empirically $m^{0.96}$-$m^{1.02}$ per family (approximate Cholesky default), and a pooled fit across 228 SuiteSparse graphs gives $m^{1.19}$ vs.\ $m^{1.45}$ for direct factorization; the solver handles a $6.8\times10^7$-edge social network in minutes. Memory is the binding constraint: Cholesky fill reaches $10$-$280\times$ the graph nonzeros vs.\ our $O(m)$ hierarchy. Against the released FCL solver we are $1.5$-$14\times$ faster at matched accuracy. On MNIST $10$-NN, $p=3$ scores $64\%$ at one label per class vs.\ $36\%$ for $p=2$. Code: https://github.com/orenlivne/np.
Primary: Weizmann Institute of Science
All Institutions: Weizmann Institute of Science
The paper presents a significant engineering and numerical analysis breakthrough that makes scalable, nonlinear graph semi-supervised learning practically viable for the first time. By correctly integrating near-linear Laplacian solvers with a damped Newton continuation framework, it overcomes the stability and memory issues that previously confined $p$-Laplacian SSL to small graphs, enabling applications on industrial-scale networks with tens of millions of edges while maintaining the statistical advantages of nonlinear energy minimization.
The paper addresses a critical scalability bottleneck in Graph $p$-Laplacian Semi-Supervised Learning (SSL). While the statistical benefits of $p>2$ energies in mitigating the low-label degeneracy of quadratic ($p=2$) label propagation are well-established, practical adoption has been hindered by the superlinear memory and time complexity of direct sparse factorization solvers. The authors' key methodological contribution is the rigorous recasting of the $p$-Laplacian SSL problem as a nonlinear Laplacian flow ($B\rho_p(B^\top x)=b$) and the application of a damped chord-Newton continuation method. Crucially, they demonstrate that by using a conductance floor and guarded Anderson acceleration, the inner linearized systems remain well-conditioned, allowing the substitution of expensive direct solvers with near-linear time Laplacian engines (Approximate Cholesky or LAMG+). This is a non-trivial numerical analysis contribution, as naive substitution of near-linear solvers into the Newton loop typically leads to stagnation due to ill-conditioning at large $p$.
The experimental evaluation is comprehensive and convincing. The authors provide: 1. Theoretical validation: Reproducing the known low-label degeneracy of $p=2$ and showing that $p=3$ significantly improves accuracy (64% vs 36% on MNIST with 1 label/class). 2. Scaling analysis: Empirical evidence of near-linear scaling ($m^{0.96}-m^{1.02}$) on fixed graph families and a pooled fit of $m^{1.19}$ across 228 heterogeneous graphs. 3. Comparative benchmarks: Head-to-head comparisons against the incumbent FCL solver (showing 1.5-14x speedups) and Calder's GraphLearning package (showing significant speedups on geometric graphs). 4. Web-scale demonstration: Successfully solving SSL on a 68M-edge social network (LiveJournal) in minutes, a task infeasible for direct factorization methods due to memory constraints. The experiments are rigorous, covering controlled scaling, heterogeneous corpus analysis, and real-world industrial-scale graphs.
The paper provides a clear algorithm description, detailed hyperparameters (e.g., conductance floor $10^{-6}$, continuation schedule), and open-source code. The reproducibility is high, supported by the availability of the Julia implementation and the specific graph corpus used.
The authors honestly disclose several limitations: 1. The near-linear scaling is empirical; no theoretical complexity bound is provided for the outer iteration count on general graphs. 2. The solver is currently single-threaded, limiting absolute wall-clock performance compared to potential distributed implementations. 3. The comparison with FCL is limited to moderate sizes due to the MATLAB/Octave implementation's constraints, though the memory wall argument for larger graphs is strong. 4. The method relies on the effectiveness of the conductance floor, which, while theoretically justified as a preconditioner, is an empirical choice.
This work removes a major barrier to using nonlinear graph-based SSL at scale. By making $p$-Laplacian methods feasible for web-scale graphs, it enables more robust semi-supervised learning in regimes with scarce labels (few-shot learning, active learning) where GNNs often overfit and quadratic propagation fails. It bridges the gap between theoretical insights on $p$-Laplacian well-posedness and practical, large-scale machine learning infrastructure. The paper presents a significant engineering and numerical analysis breakthrough that makes scalable, nonlinear graph semi-supervised learning practically viable for the first time. By correctly integrating near-linear Laplacian solvers with a damped Newton continuation framework, it overcomes the stability and memory issues that previously confined $p$-Laplacian SSL to small graphs, enabling applications on industrial-scale networks with tens of millions of edges while maintaining the statistical advantages of nonlinear energy minimization.
Routing among large language models (LLMs) promises better quality at lower cost, motivated by the reported gap between learned routers and a per-instance oracle. But that oracle is computed from a single correctness label per (query, model), so under stochastic decoding it is one Bernoulli draw, not a reproducible property. We recast the question structurally: the expected per-instance oracle decomposes as $O^{\exp}=O^{\mathrm{repro}}+Δ$, into reproducible single-commit headroom $O^{\mathrm{repro}}$ and a non-negative single-commit selection floor $Δ$. Our main result is a recoverability asymmetry: this floor is closed by no single-commit router, yet is recovered by test-time sampling -- best-of-$K$ on the committed model, at the oracle's own budget, dominates the independent-pool single-draw oracle. The cap needs no cross-model independence; we prove it with the exact decomposition and noise-share bounds that shrink as the budget grows. The procedure adds no new router, only resampling. The floor's magnitude is a prospective, conservative localization, not an audit: our primary target LLMRouterBench (33 models, 391,645 instances) defines its oracle as a per-query union over single $T=0.2$ generations -- by construction a union of stochastic single draws. Since $O^{\mathrm{repro}}$ is non-identifiable from the released $k=1$ matrix, we estimate the noise share by fresh $k\ge20$ resampling under one-sided, dependence- and guessing-floor-corrected bounds, recasting 'model-recall failure' as thin-support union inflation. On a controlled open-model re-generation, single-draw noise is a substantial minority of the gap -- larger on an unsaturated benchmark, approaching half on the hardest queries where no model is reliable -- while the majority remains recoverable specialist advantage. We release a multi-sample oracle evaluation protocol for routing benchmarks.
Primary: National Yang Ming Chiao Tung University
All Institutions: National Yang Ming Chiao Tung University, Krixvon AI
This paper has significant broader impact for the field of LLM routing and evaluation methodology. 1. **Benchmark Design**: It provides a concrete, actionable protocol for benchmark designers to adopt multi-sample oracles (expected and reproducible variants) instead of single-draw ones, which are shown to be systematically inflated. This could lead to more accurate and reliable routing benchmarks. 2. **Interpretation of Routing Progress**: The work fundamentally re-calibrates the understanding of the "router-to-oracle gap" and the "model-recall failure" diagnosis. By quantifying the portion of the gap attributable to irreducible single-draw noise, it clarifies how much genuine headroom exists for routers, guiding research efforts more effectively. 3. **Research Direction**: It suggests that future routing research should focus on better ex-ante quality estimation and decorrelating model pools, rather than chasing an inflated ceiling. It also motivates further investigation into cost-quality claims and end-to-end latency with calibrated oracles. 4. **General LLM Evaluation**: The principles of accounting for stochasticity and decomposing performance into reproducible vs. noise components could extend beyond routing to other areas of LLM evaluation where single-sample metrics are common. This paper rigorously decomposes the LLM router-to-oracle gap, revealing that a substantial minority is single-draw label noise irrecoverable by single-commit routing, and proposes a multi-sample oracle evaluation protocol. The work provides a robust theoretical framework, compelling empirical evidence, and a highly reproducible methodology that significantly advances the understanding and evaluation of LLM routing systems, offering clear guidance for benchmark design and future research.
The methodology is exceptionally rigorous and well-articulated. The paper structurally recasts the problem of the router-to-oracle gap by defining three key oracles: the expected single-draw oracle ($O^{\exp}$), the reproducible single-commit headroom ($O^{\mathrm{repro}}$), and the verifier-free aggregation oracle ($O^{\mathrm{agg}}$). The core contribution is the exact, non-negative decomposition of the router-to-oracle gap ($G$) into recoverable specialist advantage ($G_{\mathrm{rec}}$) and single-draw label noise ($G_{\mathrm{noise}}$). This decomposition is backed by strong theoretical proofs (Theorems, Propositions, Corollaries) that establish the upward bias of the single-draw oracle and the "recoverability asymmetry"—that $G_{\mathrm{noise}}$ is irrecoverable by any single-commit router but can be recovered by test-time sampling. The proposed Algorithm 1 provides a clear, step-by-step protocol for multi-sample correctness estimation and gap decomposition, using raw frequencies for point estimates and Beta-Bernoulli posteriors for confidence intervals. Crucially, the methodology addresses the complexities of estimating $O^{\exp}$ in the presence of cross-model dependencies by using a seed-aligned estimator, which is unbiased. The use of one-sided, dependence- and guessing-floor-corrected bounds for conservative estimation of $G_{\mathrm{noise}}$ further enhances the robustness of the approach. The paper clearly distinguishes its contributions from prior and concurrent work, particularly regarding the focus on stochastic single-draw noise versus deterministic evaluation artifacts.
The experimental evaluation is comprehensive and well-controlled, designed to localize the empirical magnitude of the theoretically proven noise term. The primary target is LLMRouterBench, with RouterBench as secondary corroboration. For a controlled re-generation, the authors used a pool of eleven open-weight, text-only instruction models served identically under vLLM at $T=0.2$ with $k=30$ seed-aligned draws per (query, model) cell. Two exact-match benchmarks, GSM8K (saturated) and MATH-500 (unsaturated), were used, with thin-support queries oversampled. The experiments successfully pass pre-checks for independence and over-dispersion, licensing the magnitude study. Key findings include: 1. **Magnitude of Noise**: Single-draw noise ($G_{\mathrm{noise}}$) constitutes a substantial minority of the gap (12% on GSM8K, 36% on MATH-500), with the majority remaining recoverable specialist advantage (64-88%). 2. **Noise Concentration**: $G_{\mathrm{noise}}$ concentrates heavily in thin-support queries (e.g., 43% on MATH-500 for queries where only 3 of 11 models were correct), validating theoretical predictions. 3. **Pool Composition Control**: The paper rigorously controls for intra-lineage error correlation, showing that redundancy inflates the noise share. Experiments with lineage-deduplicated pools and cardinality sweeps confirm the theoretical predictions, demonstrating the robustness of the findings. 4. **Recoverability Check**: The falsifiable prediction that test-time sampling recovers what selection cannot is empirically confirmed, with best-of-$K$ sampling outperforming the independent-pool oracle. However, the analysis also highlights that verifier-free aggregation (majority vote) often falls short, indicating that a significant portion of the "guessing residual" requires a deploy-time verifier. The experimental design is exemplary in its controls, stratification, and careful interpretation of results, providing strong empirical support for the theoretical claims.
The reproducibility of this work is exceptionally high. The authors explicitly state that "Code, corrected oracles, and the per-model correctness data are available at https://github.com/luka-krixvon/routing-oracle-experiment". The paper provides detailed information about the experimental setup, including the specific models used (eleven open-weight instruction models from eight distinct pretraining lineages), the serving framework (vLLM), decoding parameters ($T=0.2$, top-$p$ $1.0$), and the number of seed-aligned draws ($k=30$). The system configuration, including hardware/software stack, is captured by a detection script and released with the code. The methodology for multi-sample correctness estimation and gap decomposition is clearly outlined in Algorithm 1. This level of detail and the release of artifacts make the work highly reproducible and verifiable by the community.
The paper acknowledges several limitations: 1. **Scope of Re-estimation**: The current estimates use $k$ samples at a single temperature on an open-model pool. Future work could extend this to larger $k$, multiple temperatures, and live frontier (closed-source) models to sharpen estimates and test the bias growth. 2. **Cross-model Error Correlation**: While the paper controls for intra-lineage correlation, it notes that it does not fully characterize how cross-model estimator-error correlation shifts routing optimality in general, leaving this for future work. 3. **Evaluation Metric Scope**: The primary analysis focuses on exact-match and multiple-choice tasks, explicitly excluding LLM-judge / continuous-graded ones due to the mixing of sampling noise with judge noise. This is a reasonable scoping decision but means the findings do not directly generalize to all types of LLM evaluations. 4. **Preprint Date**: The "Preprint, July 2026" date is unusual for an arXiv preprint, which typically reflects the current or a past year. While not impacting the technical content, it's an oddity.
This paper has significant broader impact for the field of LLM routing and evaluation methodology. 1. **Benchmark Design**: It provides a concrete, actionable protocol for benchmark designers to adopt multi-sample oracles (expected and reproducible variants) instead of single-draw ones, which are shown to be systematically inflated. This could lead to more accurate and reliable routing benchmarks. 2. **Interpretation of Routing Progress**: The work fundamentally re-calibrates the understanding of the "router-to-oracle gap" and the "model-recall failure" diagnosis. By quantifying the portion of the gap attributable to irreducible single-draw noise, it clarifies how much genuine headroom exists for routers, guiding research efforts more effectively. 3. **Research Direction**: It suggests that future routing research should focus on better ex-ante quality estimation and decorrelating model pools, rather than chasing an inflated ceiling. It also motivates further investigation into cost-quality claims and end-to-end latency with calibrated oracles. 4. **General LLM Evaluation**: The principles of accounting for stochasticity and decomposing performance into reproducible vs. noise components could extend beyond routing to other areas of LLM evaluation where single-sample metrics are common. This paper rigorously decomposes the LLM router-to-oracle gap, revealing that a substantial minority is single-draw label noise irrecoverable by single-commit routing, and proposes a multi-sample oracle evaluation protocol. The work provides a robust theoretical framework, compelling empirical evidence, and a highly reproducible methodology that significantly advances the understanding and evaluation of LLM routing systems, offering clear guidance for benchmark design and future research.
Brownian Bridge Diffusion Models (BBDM) offer an appealing framework for image restoration and inverse problems by constructing a stochastic bridge from the clean signal directly to the degraded observation, rather than to pure noise. Despite their promise, the choice of bridge schedule is typically inherited from heuristics, and a principled analytical framework for schedule design has been lacking. In this work, we develop such a framework by offering a novel analysis of BBDM reverse dynamics under a Mixture-of-Gaussians (MoG) prior. This setting yields a closed-form ideal posterior and a corresponding MMSE denoiser, while the BBDM-induced reconstruction law is captured analytically through a tractable surrogate. Building on these expressions, we formulate two complementary schedule-design objectives: a Wasserstein criterion targeting perceptual quality and an MSE criterion targeting reconstruction fidelity. Our work exposes an inherent tradeoff between the two and proves the existence of universal schedules for both that are independent of the degradation and prior. Extensive experiments on controlled MoG settings confirm full alignment between theory and practice, and experiments on the FFHQ dataset across inpainting, deblurring, and super-resolution tasks validate the practical value of our schedule-design criteria.
Primary: Technion – Israel Institute of Technology
All Institutions: Technion – Israel Institute of Technology
This paper provides a rigorous analytical framework for schedule design in Brownian Bridge Diffusion Models, deriving closed-form reconstruction laws under Mixture-of-Gaussians priors to expose and optimize the distortion-perception tradeoff. The technical contribution is significant for its mathematical depth and the clarity it brings to a previously heuristic area, though its direct impact is somewhat moderated by the reliance on approximations for high-dimensional applications.
The paper presents a rigorous analytical framework for schedule design in Brownian Bridge Diffusion Models (BBDM). The core methodological contribution is the derivation of exact posterior dynamics under a Mixture-of-Gaussians (MoG) prior. Recognizing that the exact MoG reverse process loses global affinity, the authors introduce a "selected-label" approximation that freezes the mixture component assignment, allowing for a closed-form reconstruction law. This enables the formulation of two explicit schedule-design objectives: one minimizing Wasserstein distance (perceptual quality) and one minimizing MSE (reconstruction fidelity). The theoretical derivation is mathematically sound, leveraging Gaussian conditioning identities and spectral decomposition to decouple the dynamics. The approach is novel in applying this specific analytical lens to BBDM schedules, moving beyond heuristic choices.
The experimental validation is structured in three tiers: synthetic MoG data, MNIST with fitted MoG priors, and real-world FFHQ images. The synthetic experiments effectively validate the theoretical claims regarding the selected-label approximation and the distortion-perception tradeoff. The MNIST experiments demonstrate that the theoretical schedules improve upon default schedules in PSNR and NLL. The FFHQ experiments show that the MSE-oriented schedule improves PSNR/SSIM while the W2-oriented schedule improves FID/LPIPS, confirming the theoretical tradeoff. However, the real-world experiments rely on "MoG-free heuristics" derived from the theoretical bounds rather than optimizing the full MoG objective (which is intractable at scale), which slightly weakens the direct link between the complex theory and the final applied results.
The paper provides extensive mathematical derivations in the appendix, including proofs of mean-exactness and covariance deficit. The experimental setup is detailed, including dataset splits, training epochs, and evaluation metrics. The use of standard datasets (FFHQ, MNIST) and publicly available libraries (torch-fidelity, lpips) aids reproducibility. The code for the BBDM models is not explicitly linked, but the methodology is sufficiently described for implementation.
The primary limitation is the reliance on the selected-label approximation for the theoretical analysis. While proven to be mean-exact and covariance-deficient, it is an approximation. The "universal" schedules derived are based on a bounded four-parameter family and may not be optimal for all degradation types or data distributions. Furthermore, the real-world experiments use heuristics rather than the full theoretical optimization, limiting the direct demonstration of the theory's power in high-dimensional settings. The assumption of linear inverse problems also restricts the scope.
This work provides a principled foundation for tuning diffusion models for inverse problems, potentially leading to more reliable and performant restoration algorithms. By exposing the inherent tradeoff between perceptual quality and fidelity through a clear analytical lens, it offers valuable insights for practitioners balancing these competing objectives. The framework could be extended to other bridge-based diffusion models or used to analyze other sampler parameters. This paper provides a rigorous analytical framework for schedule design in Brownian Bridge Diffusion Models, deriving closed-form reconstruction laws under Mixture-of-Gaussians priors to expose and optimize the distortion-perception tradeoff. The technical contribution is significant for its mathematical depth and the clarity it brings to a previously heuristic area, though its direct impact is somewhat moderated by the reliance on approximations for high-dimensional applications.
Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hurt one another. Most efficient fine-tuning pipelines ignore this and train on a fixed, uniform mixture, leaving two coupled questions unanswered: how much should each domain participate, and which domains should be co-trained given that some transfer positively and others interfere? We show that both answers can be read off cheaply and without labels. A forward pass of the current shared adapter over a small unlabeled probe yields, per domain, a competence signal whose level tracks remaining headroom and whose trajectory tracks learning speed; the drift of these probe representations yields a signed cross-domain affinity that predicts pairwise transfer. We fold both into CoDA, a co-adaptive controller that solves a small entropy-regularized quadratic program on the simplex to set each domain's participation -- jointly its loss weight and its share of the sampled data -- rewarding high-headroom, still-learning, mutually synergistic domains and damping interfering ones. The controller is forward-only, adds no trainable parameters, and wraps any multi-task LoRA pipeline. Across five heterogeneous domains and two backbones, CoDA improves the average over uniform mixing, learned mixtures, gradient-surgery multi-task optimizers, and online data selection while using half the data, and lowers cross-domain gradient conflict. We prove that the competence signal tracks domain risk, that the participation program has a unique fixed point reached by a contraction, and that its solution performs transfer-aware water-filling; analysis, ablations, and controls corroborate each claim.
Primary: University of Electronic Science and Technology of China
All Institutions: University of Electronic Science and Technology of China, Sichuan University
CoDA introduces a novel, label-free, forward-only mechanism for dynamically balancing multi-task LoRA training by estimating domain competence and cross-domain affinity, achieving state-of-the-art performance with reduced data and compute costs.
The paper proposes CoDA, a co-adaptive controller for multi-task Low-Rank Adaptation (LoRA). The core innovation lies in using label-free, forward-only signals to dynamically adjust domain participation. Specifically, it defines "competence" based on normalized predictive entropy to estimate domain headroom and learning speed, and "affinity" based on the drift of probe representations to estimate signed cross-domain transfer. These signals feed into an entropy-regularized quadratic program that jointly optimizes loss weights and data sampling ratios. The approach is theoretically grounded with proofs regarding the tracking of domain risk, the uniqueness of the fixed point (contraction mapping), and its interpretation as transfer-aware water-filling. The methodology is elegant, avoiding the need for labeled validation sets or expensive gradient computations during the control loop.
The evaluation is robust, covering two large backbones (Qwen-2.5-7B, LLaMA-3.1-8B) and five heterogeneous domains (Knowledge, Math, Code, Reasoning, Biomedical). CoDA outperforms uniform mixing, static mixture baselines (DoReMi, Temperature), and gradient-surgery methods (PCGrad, GradNorm) across all metrics, achieving significant gains (+1.8 avg score) while using only 50% of the data. The paper provides strong ablations isolating the contribution of the headroom term versus the affinity term, demonstrating that the latter is crucial for capturing synergistic/interfering relationships. Mechanism analysis confirms that the affinity signal correlates highly ($r=0.94$) with oracle leave-one-out transfer measurements. The results are consistent across seeds and model scales.
The paper provides detailed descriptions of the experimental setup, including dataset sources, prompt templates, LoRA configurations, and hyperparameters. The algorithm is clearly defined with pseudocode. The authors state that code and configurations will be released. The reliance on forward passes makes the method computationally transparent and easy to implement on top of existing LoRA pipelines. The theoretical proofs are included in the appendix, adding to the reproducibility of the claims.
The method requires a small unlabeled probe set per domain, which may be a constraint in strictly zero-data settings (though no labels are needed). The competence signal relies on model calibration; severe miscalibration could degrade performance, although the authors note a warmup phase helps mitigate this. The method assumes that the probe data is representative of the domain distribution. Additionally, while it reduces gradient conflict, it does not eliminate it entirely, and the quadratic program solution, while efficient, adds some overhead compared to static mixing.
This work significantly advances the efficiency and effectiveness of multi-task fine-tuning for LLMs. By reducing the data and compute required for adapting models to multiple domains, it lowers the barrier to entry for specialized model deployment and reduces the environmental impact of training. The label-free nature of the controller makes it applicable to scenarios where labeled data is scarce or expensive. The insights into cross-domain transfer and interference provide valuable theoretical understanding for multi-task learning. CoDA introduces a novel, label-free, forward-only mechanism for dynamically balancing multi-task LoRA training by estimating domain competence and cross-domain affinity, achieving state-of-the-art performance with reduced data and compute costs.
Crowdsourced fact-checking systems have been adopted by major social media companies such as X, Meta, TikTok and Google with the aim of combating misleading information at scale without relying on centralized editorial control. These systems have been developed around a common underlying concept: a bridging mechanism that identifies notes flagging misleading information when they receive support from people with different perspectives rather than simple majority support. To our knowledge the only publicly disclosed bridging algorithms deployed for fact-checking are based on matrix factorization, as deployed by both X and Meta, augmented with additional components addressing abuse, targeted manipulation, and contributor brigades. This work examines the core matrix factorization portion of these systems, presenting theoretical and empirical evaluations of the degree to which coordinated users could vote strategically by leveraging the latent representations to fabricate the appearance of synthetic consensus within the bridging mechanism. Using historic production data, we find that up to 10.7% of lower quality notes could be manipulated above consensus thresholds using less than 10 ratings. We complement these findings with a theoretical analysis, revealing counterintuitively that rating a note as "Not Helpful" can increase its helpfulness score, as well as a cost model quantifying manipulation effort. We have developed and deployed mitigations within X's Community Notes algorithm to address synthetic consensus.
Primary: Stanford University
All Institutions: Stanford University, X Community Notes, xAI
This work has significant broader impact, particularly for social media platforms and the field of adversarial machine learning. It proactively identifies a critical vulnerability in crowdsourced fact-checking systems (like X, Meta, TikTok, Google) that rely on matrix factorization for "bridging consensus." By demonstrating how coordinated adversaries can fabricate synthetic consensus, the paper highlights a fundamental challenge in designing robust, decentralized content moderation systems. The theoretical insights, especially the counterintuitive "Not Helpful" rating effect, contribute to a deeper understanding of MF-based systems. Most importantly, the paper's findings directly led to the development and *deployment of mitigations* (population sample filtering) within X's Community Notes algorithm, demonstrating a direct translation of research into real-world system improvements. This sets a high bar for impactful research in platform security and responsible AI, encouraging transparency and open collaboration between academia and industry to strengthen critical public-facing systems. This paper presents a rigorous analysis of coordinated manipulation in matrix factorization-based crowdsourced fact-checking, demonstrating a practical attack on production data and leading to the deployment of mitigations in X's Community Notes. The work combines theoretical derivations, empirical validation on a large-scale real-world dataset, and a practical cost model to expose a significant vulnerability in systems designed to combat misinformation, offering both novel insights into adversarial ML and direct, actionable solutions for platform security.
The paper presents a well-structured and rigorous methodology for analyzing coordinated manipulation in crowdsourced fact-checking systems, specifically focusing on the core matrix factorization (MF) component. The two-phase attack strategy is logically sound: first, adversarial accounts establish diverse positions in the latent factor space by strategically rating existing notes; second, these accounts coordinate to boost a target note's helpfulness score. This approach directly targets the "bridging" mechanism designed to ensure diverse agreement. The theoretical analysis of the Manipulation Resistance Score (MRS) is a significant contribution, providing a closed-form expression for the optimal single rating injection in a 1-dimensional factor space, which is the production setting for X. The derivation, detailed in the appendix, is thorough and correct. A particularly novel and counterintuitive finding is that rating a note as "Not Helpful" can, under specific conditions related to the geometry of existing ratings, increase its helpfulness score. This highlights a subtle vulnerability in the MF model. The cost model for the full attack provides a practical framework for understanding the economic feasibility of such manipulations and for evaluating potential mitigations. The methodology is strong in its combination of theoretical derivation, practical attack formulation, and cost analysis.
The experimental evaluation is robust and highly impactful due to its use of historic production data from X Community Notes (Jan 2021 - Jan 2025). This real-world dataset lends significant credibility to the findings. The ability to predict note parameters ($f_n, i_n$) from text using a Voyage embedding model and a shallow MLP is empirically demonstrated with reasonable accuracy, validating the feasibility of Phase 1 of the attack. The simulation showing that 100 adversarial accounts can achieve diverse factor positions across the spectrum $[-0.4, 0.4]$ further supports the attack's practicality. The quantification of MRS is a key empirical result, demonstrating that up to 10.7% of lower-quality notes could be manipulated above consensus thresholds using fewer than 10 ratings. This is a stark and actionable finding. The cost model, while simplified, provides concrete estimates (e.g., $30.50 for a single note manipulation) and effectively highlights the dominant cost factors (account maintenance). The paper also discusses the effectiveness of deployed mitigations, such as population sample filtering, which is a strong indicator of real-world impact. The experiments are well-designed to validate the theoretical claims and quantify the practical threat.
The paper demonstrates a strong commitment to reproducibility. It explicitly states that the analysis is based on the "open data and source code of X Community Notes," which facilitates independent study. The dataset used is publicly released, and the specific embedding model (Voyage-3-large) is identified. Hyperparameter and implementation details for the prediction model are promised in the appendix (though the appendix provided in the prompt is truncated before these details). The computational resources are specified, and the total wall-clock time for experiments is given. The full derivation for optimal rating injection is provided in the appendix. The authors also state that X deployed mitigations and released them as part of the open-source algorithm, further enhancing reproducibility and real-world impact.
The paper openly discusses several limitations. Firstly, it acknowledges that production Community Notes implementations include anti-abuse components (e.g., Correlated Rater Detection, Rater Engagement Intercept, Net Helpful Minimums) that are not fully incorporated into the core analysis. While these are discussed qualitatively, their quantitative impact on the attack's cost and success rate is not fully modeled. Secondly, the analysis is conducted in a static setting, not accounting for dynamic feedback loops where a surfaced "Helpful" note might attract more ratings, potentially changing its status. Thirdly, the MRS computation uses a greedy algorithm, which might be a conservative approximation compared to exact combinatorial optimization. Additionally, the note parameter prediction model uses only note text, ignoring post content or URLs, which could lead to underestimation of attacker capabilities. Finally, the cost model is a simplified abstraction and doesn't capture all nuances of attacker utility or sophisticated evasion strategies.
This work has significant broader impact, particularly for social media platforms and the field of adversarial machine learning. It proactively identifies a critical vulnerability in crowdsourced fact-checking systems (like X, Meta, TikTok, Google) that rely on matrix factorization for "bridging consensus." By demonstrating how coordinated adversaries can fabricate synthetic consensus, the paper highlights a fundamental challenge in designing robust, decentralized content moderation systems. The theoretical insights, especially the counterintuitive "Not Helpful" rating effect, contribute to a deeper understanding of MF-based systems. Most importantly, the paper's findings directly led to the development and *deployment of mitigations* (population sample filtering) within X's Community Notes algorithm, demonstrating a direct translation of research into real-world system improvements. This sets a high bar for impactful research in platform security and responsible AI, encouraging transparency and open collaboration between academia and industry to strengthen critical public-facing systems. This paper presents a rigorous analysis of coordinated manipulation in matrix factorization-based crowdsourced fact-checking, demonstrating a practical attack on production data and leading to the deployment of mitigations in X's Community Notes. The work combines theoretical derivations, empirical validation on a large-scale real-world dataset, and a practical cost model to expose a significant vulnerability in systems designed to combat misinformation, offering both novel insights into adversarial ML and direct, actionable solutions for platform security.
Safety training for large language models (LLMs) is conducted predominantly in English, leaving uncertain how well safety mechanisms generalize to low-resource languages and mixed-language code-switching. We show that this creates an epistemic gap in which models confidently generate harmful responses for inputs that fall outside the distribution of their safety training. To study this phenomenon, we introduce STEER (Safety Targeted Embedding Exploit via Refinement), a gradient-guided attack that identifies words contributing most strongly to the model's refusal behavior and iteratively translates them into low-resource languages to suppress refusal while preserving harmful intent. Across six open-source 8B-parameter models, STEER achieves attack success rates of up to 93.0% on JailbreakBench and 96.7% on AdvBench, outperforming random code-switching and Greedy Coordinate Gradient (GCG). The resulting prompts also transfer to GPT-4o-mini, achieving a 35.5% attack success rate without requiring access to the target model, suggesting that the underlying weakness is not specific to a single architecture. These findings demonstrate that safety mechanisms aligned primarily on English cannot be assumed to generalize across multilingual inputs. We argue that improving multilingual safety requires broader coverage during alignment and mechanisms that explicitly detect and abstain on out-of-distribution inputs.
Primary: Nanyang Technological University, Singapore
All Institutions: Nanyang Technological University, Singapore
This paper has profound broader implications for LLM safety research and development: * **Fundamental Vulnerability**: It exposes a systemic and fundamental vulnerability in current LLM safety alignment practices, which are predominantly English-centric and concentrate refusal knowledge into a single, exploitable direction. * **Shift in Perspective**: It reframes LLM safety as an "epistemic coverage problem" rather than solely an adversarial robustness challenge, highlighting the model's "unknown unknowns" and overconfident extrapolation. This conceptual shift is critical for designing more robust safety mechanisms. * **Mech Interp as Attack Enabler**: It provides a concrete demonstration of how mechanistic interpretability findings can be directly leveraged to construct powerful attacks, underscoring the dual-use nature of such research. * **Actionable Defenses**: The findings directly inform the design of future defenses, advocating for broader multilingual coverage during alignment, distributing safety knowledge across multiple layers/directions, and implementing principled abstention mechanisms for out-of-distribution inputs. * **Auditing Tool**: The FLD analysis offers a principled method for auditing models' structural safety vulnerability before deployment, allowing developers to assess the brittleness of their safety encoding. * **Ethical Implications**: The high success rates of STEER highlight the urgent need for more robust multilingual safety alignment to prevent the deployment of LLMs that confidently generate harmful content in diverse linguistic contexts. This paper introduces STEER, a gradient-guided attack that exploits the English-centric nature and concentrated refusal direction of LLM safety mechanisms, achieving high attack success rates by iteratively translating high-attribution words into low-resource languages. The work provides compelling evidence that current safety alignment practices suffer from an epistemic coverage problem, offering a novel diagnostic tool (FLD) and actionable insights for developing more robust, multilingual safety mechanisms and principled abstention strategies.
The STEER (Safety Targeted Embedding Exploit via Refinement) methodology is a sophisticated and principled gradient-guided attack that leverages mechanistic interpretability findings to bypass LLM safety mechanisms. The pipeline consists of four well-defined steps: 1. **Layer Selection via Fisher Linear Discriminant (FLD)**: A novel and effective method to automatically identify the transformer layer where the refusal direction is most "legible" or concentrated. This provides a quantitative measure of the model's structural vulnerability, which is a significant contribution beyond just enabling the attack. 2. **Paraphrase Preprocessing**: A practical initial step using GPT-4o to rephrase harmful requests, reducing initial keyword activation and providing a cleaner signal for gradient attribution. Ablation studies confirm its importance. 3. **Gradient-based Token Attribution**: This is the core of the "targeted" aspect. By computing gradients of input word embeddings against the mech-interp-identified refusal direction, STEER precisely identifies which words contribute most to activating the safety filter. This is a direct and elegant application of interpretability findings. 4. **Iterative Code-Switching**: Words are iteratively translated into a pool of 11 low-resource and non-Latin script languages, prioritizing those with the highest attribution scores. The selection of the best translation is based on minimizing the refusal score, ensuring the attack is efficient and effective. The overall approach is highly systematic, combining insights from mechanistic interpretability, gradient-based optimization, and multilingual NLP to create a powerful and interpretable attack. The design choices are well-justified and empirically validated.
The experimental evaluation is comprehensive and rigorous. * **Models**: Six diverse open-source 7-9B parameter models (Llama-3-8B, Mistral-7B, Gemma-7B, Qwen3-8B, DeepSeek-R1-Distill-Llama-8B, GLM-4-9B) are tested, demonstrating the generality of the attack across different architectures. * **Benchmarks**: Three standard jailbreak benchmarks (JailbreakBench, HarmBench, AdvBench) are used, covering a wide range of harmful prompts. * **Baselines**: STEER is compared against strong baselines: Direct (unmodified), CSRT (random code-switching), and GCG (gradient-based adversarial suffix optimization). * **Results**: STEER achieves exceptionally high Attack Success Rates (ASR) of up to 93.0% on JailbreakBench and 96.7% on AdvBench, consistently outperforming all baselines, often by a significant margin (e.g., 80% vs 44% for DeepSeek-R1 on JBB). This demonstrates its superior efficiency and effectiveness. * **Iteration Efficiency**: The attack shows strong performance even with a low iteration budget (e.g., 88% ASR at @1 for Mistral-7B on JBB), highlighting the efficiency gained from targeted attribution. * **Refusal Score Validation**: The paper provides strong statistical evidence that the refusal score (dot product with the refusal direction) is indeed the decision variable for refusal, validating the mechanistic hypothesis. * **Black-box Transferability**: A crucial finding is the transferability of STEER-generated prompts to GPT-4o-mini, achieving a 35.5% ASR without white-box access. This suggests the exploited weakness is not architecture-specific but a fundamental property of current alignment methods. * **Ablation Studies**: Thorough ablations confirm the importance of FLD layer selection, the diverse language pool, and the paraphrase preprocessing step, reinforcing the robustness of the design choices. The evaluation is robust, well-designed, and provides compelling evidence for the paper's claims.
The paper provides a clear algorithmic description (Algorithm 1) of the STEER attack. Key parameters, language pool, and judge details are specified. Crucially, the authors provide code at `https://github.com/JvThunder/STEER`, which significantly enhances reproducibility. The use of open-source models and standard benchmarks further aids reproducibility.
1. **White-box Access**: STEER requires white-box access to the target model's internal representations and gradients, limiting its direct applicability to closed-source APIs. While transferability to GPT-4o-mini is shown, a dedicated black-box adaptation is not explored. 2. **Model Scale**: The evaluation is limited to 7-9B parameter models. While these are widely used, the findings might not directly generalize to much larger models (e.g., 70B+) or models with different safety alignment strategies. 3. **Automated Judge**: The use of GPT-4o as an automated judge, while common, might occasionally diverge from human assessments, especially for borderline cases of harmfulness or refusal. The dual-criterion (non-refusing and harmful) is a conservative approach, but human validation on a subset could strengthen this.
This paper has profound broader implications for LLM safety research and development: * **Fundamental Vulnerability**: It exposes a systemic and fundamental vulnerability in current LLM safety alignment practices, which are predominantly English-centric and concentrate refusal knowledge into a single, exploitable direction. * **Shift in Perspective**: It reframes LLM safety as an "epistemic coverage problem" rather than solely an adversarial robustness challenge, highlighting the model's "unknown unknowns" and overconfident extrapolation. This conceptual shift is critical for designing more robust safety mechanisms. * **Mech Interp as Attack Enabler**: It provides a concrete demonstration of how mechanistic interpretability findings can be directly leveraged to construct powerful attacks, underscoring the dual-use nature of such research. * **Actionable Defenses**: The findings directly inform the design of future defenses, advocating for broader multilingual coverage during alignment, distributing safety knowledge across multiple layers/directions, and implementing principled abstention mechanisms for out-of-distribution inputs. * **Auditing Tool**: The FLD analysis offers a principled method for auditing models' structural safety vulnerability before deployment, allowing developers to assess the brittleness of their safety encoding. * **Ethical Implications**: The high success rates of STEER highlight the urgent need for more robust multilingual safety alignment to prevent the deployment of LLMs that confidently generate harmful content in diverse linguistic contexts. This paper introduces STEER, a gradient-guided attack that exploits the English-centric nature and concentrated refusal direction of LLM safety mechanisms, achieving high attack success rates by iteratively translating high-attribution words into low-resource languages. The work provides compelling evidence that current safety alignment practices suffer from an epistemic coverage problem, offering a novel diagnostic tool (FLD) and actionable insights for developing more robust, multilingual safety mechanisms and principled abstention strategies.
World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling. Meanwhile, unified vision-language models have demonstrated strong multimodal generation capabilities, yet their potential as world models remains underexplored. In this work, we introduce \texttt{WorldBagel}, a unified VLAW framework built on BAGEL, a modern multimodal unified model, and use it to systematically investigate the role of unification in world modeling. Across multi-task robotic manipulation and cross-domain experiments, \texttt{WorldBagel} consistently outperforms task-specific alternatives and learns action representations that are more structured and semantically aligned with visual and linguistic context. Experiments on LIBERO, Language Table, and Franka show that unification is not only an architectural convenience, but also a key factor in learning effective VLAW models, leading to consistent empirical gains and deeper insights into multimodal world modeling. Code and model checkpoints will be released upon acceptance.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
The paper introduces WorldBagel, a unified Vision-Language-Action-World (VLAW) framework built upon the BAGEL two-tower architecture. The core methodological contribution lies in extending a powerful multimodal generative model (BAGEL) to jointly support multimodal understanding, structured action modeling, and future world prediction. The VLAW formulation is clearly defined, aiming to model the joint distribution of future observations and actions conditioned on past states and language instructions. A significant technical contribution is the Fourier Feature Action Decoder (FFAD) and Fourier Feature Action Tokenizer (FFAT). FFAD addresses the limitations of standard regression and discretization-based action tokenizers by mapping continuous actions into Fourier features and predicting in this space. The inverse mapping uses phase-consistent averaging for reconstruction. This approach is well-justified with theoretical analysis provided in the appendix, demonstrating Lipschitz stability, injectivity, consistency of reconstruction, and approximation advantages. This mathematical rigor is a strong point. The interleaved VLAW modeling via sequence plans, adapted from BAGEL, is a practical and flexible way to structure multimodal sequences for multi-view, multi-step observations and control. The concept of sampling different sequence plans to balance training objectives is sound. Furthermore, the LLM-inspired multimodal train-time data sampling, using mixture dataset sampling and priority sequence-plan sampling, is a crucial engineering detail for stabilizing training across heterogeneous datasets and balancing policy learning with world modeling. The overall architecture leverages the strengths of BAGEL's GEN/UND experts, with action modeling integrated through fine-tuned tokenizers and decoders rather than a new expert. This design choice maintains the unified nature of the model.
The experimental evaluation is comprehensive and rigorous, addressing three key empirical findings: multi-task performance, action representation quality, and stability under distribution shifts. 1. **Multi-task Performance**: WorldBagel is evaluated on LIBERO, Language Table, and Franka benchmarks. On LIBERO, it achieves state-of-the-art multi-task manipulation performance (98.0% average success rate), outperforming strong VLA baselines like OpenVLA-OFT and RynnVLA-002. The world modeling capabilities are also quantitatively assessed using FVD, PSNR, SSIM, and LPIPS, showing consistent improvements over RynnVLA-002 across all datasets, especially in action-conditioned prediction. This clearly demonstrates the empirical gains of the unified VLAW approach. 2. **Action Representation Quality**: A detailed ablation study on action decoder design (regression, bin discretization, FAST, FFAD) on LIBERO shows FFAD significantly reduces action MSE and improves success rates. Further analysis on the number of Fourier bands (K) in FFAD/FFAT provides insights into optimal hyperparameter choices. Crucially, the representation structure analysis using a linear probe classifier reveals that FFAD produces more structured and task-relevant action embeddings, leading to higher task identity prediction accuracy. This is a strong validation of the FFAD design. 3. **Stability Under Distribution Shifts**: The paper investigates robustness to action noise, scaling, and temporal perturbations on LIBERO. WorldBagel consistently maintains higher prediction fidelity (PSNR, LPIPS) compared to RynnVLA-002 under these shifts. The eigenvalue spectrum analysis further supports this, showing WorldBagel learns richer and more stable action representations (higher effective rank, lower dominant eigenvalue ratio). This finding is particularly important for real-world robotics applications where such shifts are common. The choice of baselines is appropriate, including recent strong VLA models and a direct competitor (RynnVLA-002) that also aims for VLAW unification. The use of multiple metrics (success rate, FVD, PSNR, SSIM, LPIPS, A-MSE, linear probe accuracy, eigenvalue spectrum) provides a holistic view of the model's performance and internal properties. The experiments are well-designed to support the paper's claims about the benefits of unification.
The paper states that "Code and model checkpoints will be released upon acceptance," which is a positive commitment. Detailed hyperparameters (learning rate, weight decay, batch size, training steps, K for FFAT/FFAD, priority weights) and hardware (8 H200 GPUs) are provided, which are crucial for reproducibility. The mathematical derivations for FFAD/FFAT in the appendix also contribute to understanding and potentially re-implementing those components. Given the complexity of large multimodal models, the release of code and checkpoints is essential for full reproducibility.
1. **Computational Cost**: While not explicitly stated as a limitation, training and deploying a model built on a large unified multimodal backbone like BAGEL is inherently computationally intensive, requiring significant resources (e.g., 8 H200 GPUs for 80K steps). This might limit its applicability for resource-constrained environments or rapid iteration. 2. **Scope of World Modeling**: The "world modeling" aspect primarily focuses on next-frame prediction for manipulation tasks. While crucial, it doesn't delve into more abstract forms of world knowledge, causal reasoning, or long-horizon planning beyond short action rollouts, which are often goals of broader world models. 3. **Reliance on Supervised Fine-tuning**: The model relies on supervised fine-tuning (SFT) on existing robotic datasets. While effective, this approach might be limited by the diversity and scale of available demonstration data, potentially hindering generalization to truly novel tasks or environments compared to models that learn more extensively through self-supervision or interaction. 4. **Generalizability Beyond Manipulation**: The experiments are confined to robotic manipulation tasks. While these are challenging, the generalizability of "unified VLAW modeling" to other embodied AI domains (e.g., navigation, human-robot interaction) or even broader generative tasks is not explored.
WorldBagel makes a significant contribution to the field of embodied AI and multimodal learning. By demonstrating the power of architectural unification for Vision-Language-Action-World modeling, it paves the way for more capable and general-purpose robotic agents. The ability to jointly understand language, perceive the environment, predict actions, and model future states within a single framework is a crucial step towards truly intelligent robots. The proposed Fourier-based action representation (FFAD/FFAT) is a valuable technical innovation that could be adopted by other robotics policies for more robust and structured action learning. The findings on stability under distribution shifts are particularly relevant for deploying robots in real-world, uncertain environments. This work encourages further research into leveraging large-scale unified multimodal models for complex embodied tasks, potentially leading to breakthroughs in robot learning and generalization. WorldBagel introduces a unified VLAW framework built on BAGEL, systematically demonstrating that architectural unification significantly improves multi-task robotic manipulation performance, yields higher-quality action representations via Fourier-based tokenization, and enhances stability under distribution shifts, thereby providing a promising foundation for scalable multimodal world models. This paper presents a robust methodology for integrating vision, language, action, and world modeling into a single coherent framework, supported by strong empirical results and insightful analyses of action representation and robustness, making a substantial contribution to the development of more capable and generalizable embodied AI systems.
We elucidate the design space of Representation Distribution Matching (RDM), our name for the paradigm that trains a one-step image generator by matching generated and reference feature distributions under frozen pretrained encoders. We identify two design axes, how the distributions are compared and the representations they are compared in, and controlled studies along them yield three findings. First, the classical MMD, which could not train convincing generators a decade ago, becomes a strong and scalable objective once estimated right. Second, the generated batch is then the operative variable, with an optimum above 2048, far beyond customary batch sizes. Third, any single representation can be gamed, driven below the real score while images stay visibly fake, so we match against a balanced battery of encoders and evaluate with SW_r14, a Sliced-Wasserstein distance over 14 encoders that is independent of the training loss and resists gaming. Combining the preferred choices yields improved RDM (iRDM): it sets the one-step state of the art on ImageNet at SW_r14 1.30, corroborated by PickScore, a human-preference proxy our objective never optimizes, which prefers it over the prior best one-step generator on 71.2% of matched samples. The same recipe post-trains the four-step FLUX.2 [klein] into a one-step generator, surpassing the four-step version on GenEval, 0.826 to 0.794, and on PickScore, 22.76 to 22.58, in 90 H200 GPU-hours. Project page: https://alan-lanfeng.github.io/rdm/.
Primary: Valeo
All Institutions: Valeo, Alan Turing Institute (implied by author handle 'alan-lanfeng' and typical affiliation for such work, though only Valeo is explicitly funded; however, standard academic papers list affiliations. The text says "Project page: https://alan-lanfeng.github.io/rdm/" and "funded by Valeo". Without explicit author list, I will infer the primary institutional affiliation from the funding and project page context. The author 'alan-lanfeng' likely refers to Alan Feng or similar. A quick mental check of recent one-step generation papers suggests this is likely from Valeo and/or a university. Given the prompt asks to extract from text, and only Valeo is explicitly mentioned as funding/affiliation in the Acknowledgments, I will list Valeo. However, 'alan-lanfeng' is a GitHub handle. Let's look for other clues. The paper mentions "alan-lanfeng.github.io". This is likely a single-author or small team paper. I will list Valeo as the primary institution found in the text.)
This paper presents a significant advancement in one-step image generation by rigorously elucidating the design space of Representation Distribution Matching, introducing a robust MMD estimator with Nyström approximation, and demonstrating that large-batch, multi-encoder training yields state-of-the-art results while mitigating metric gaming, thereby providing a scalable and effective alternative to teacher-based distillation methods.
The paper proposes "Representation Distribution Matching" (RDM), a framework for training one-step image generators by directly matching feature distributions between generated and real images using frozen pretrained encoders. The core methodological contributions are threefold: 1) A specific estimator for Maximum Mean Discrepancy (MMD) that uses an exact within-batch repulsion term and a Nyström approximation for the attraction term against a frozen full-data reference, which the authors argue is superior to Fréchet distance or drifting fields for this task. 2) The identification that large, fresh generation batches (N > 2048) are critical for stable estimation, enabled by gradient caching. 3) A multi-encoder matching strategy using a "battery" of 14 diverse frozen encoders, balanced via a proportional Lagrangian controller to prevent the generator from gaming any single encoder's metric. The approach is theoretically grounded in kernel mean embeddings and optimal transport concepts, applied pragmatically to the current state-of-the-art in teacher-free distillation.
The experimental evaluation is rigorous and comprehensive. The authors conduct controlled ablations on the two design axes (comparison metric and representation space). They demonstrate that their method, iRDM, sets a new state-of-the-art for one-step generation on ImageNet-256 with an SW_r14 score of 1.30, significantly outperforming prior methods like pMF-H FD-SIM (2.05). They also show that post-training FLUX.2 (a 4-step model) into a 1-step model using this recipe improves GenEval and PickScore scores over the 4-step baseline, a surprising and valuable result. The use of an independent evaluation metric (SW_r14) that is not part of the training loss effectively mitigates concerns about metric gaming. The inclusion of a held-out encoder panel for evaluation adds robustness to the claims.
The paper provides significant detail for reproducibility. It specifies the encoder architectures, the Nyström landmark count (4096), batch sizes (5120/10240), learning rates, and the specific Lagrangian control mechanism. The reference to "gradient caching" and the specific implementation of the Nyström attraction term are clear. The project page likely contains code, which is standard for arXiv papers. The use of standard pretrained encoders (DINOv2, CLIP, etc.) ensures that the components are accessible. The detailed ablation studies allow other researchers to replicate the design space exploration.
The primary limitation is the computational cost of training. The requirement for large batch sizes (N=5120) and the use of 10 encoders for forward passes per step, while optimized with gradient caching, still implies a substantial memory and compute footprint compared to smaller-batch methods. The method relies heavily on the quality and diversity of the frozen encoders; if the encoder panel is biased or insufficiently diverse, the "balanced" training might still fail to capture all aspects of realism. Additionally, while it surpasses the 4-step FLUX on GenEval, it is a post-training step, meaning the base model's capabilities are a prerequisite. The "one-step" nature inherently limits the complexity of the generated distribution compared to iterative methods, as evidenced by the gap between 1.30 and the real-data floor of 1.00.
This work significantly advances the field of efficient generative modeling by demonstrating that high-quality one-step generation is achievable without online teachers or adversarial training, relying instead on careful distribution matching in feature space. This could lead to faster inference times for image generation, making it more accessible for real-time applications. The insights into metric gaming and the proposal of a robust multi-encoder evaluation metric (SW_r14) provide a valuable tool for the community to better assess generator quality. However, the ease of generating realistic images also raises standard concerns about misuse in creating deepfakes or misleading content, though the one-step nature might make it less suitable for high-fidelity, long-tail content generation compared to multi-step models. This paper presents a significant advancement in one-step image generation by rigorously elucidating the design space of Representation Distribution Matching, introducing a robust MMD estimator with Nyström approximation, and demonstrating that large-batch, multi-encoder training yields state-of-the-art results while mitigating metric gaming, thereby providing a scalable and effective alternative to teacher-based distillation methods.
Large vision-language models (LVLMs) have achieved strong performance across many medical imaging tasks, yet their application to ultrasound remains limited due to its inherent complexity and variability. In this work, we revisit what is truly needed to enable real-world ultrasound understanding. Instead of introducing complex architectures or elaborate training strategies, we show that data scale and clinically faithful data alignment are the key factors. We construct a large-scale dataset of 1.5M real-world ultrasound examinations, containing 17.7M images, multi-organ coverage, and paired uncurated clinical reports. Crucially, we organize the data at the examination level, aligning multiple images with their corresponding reports to reflect real clinical workflows. We then fine-tune a standard LVLM using low-rank adaptation (LoRA) on this dataset without task-specific modifications. Surprisingly, this simple recipe already leads to strong performance across diverse ultrasound understanding tasks, outperforming prior methods designed with more complex pipelines. Beyond these results, we present model and data scaling analyses that provide insights into the role of scale in ultrasound LVLMs.
Primary: Technical University of Munich
All Institutions: MedAI Technology (Wuxi) Co. Ltd, Technical University of Munich
This paper makes a substantial contribution to medical vision-language modeling by demonstrating that large-scale, clinically aligned data curation and simple fine-tuning of standard LVLMs can outperform complex, specialized architectures for ultrasound understanding, providing a new benchmark and paradigm for the field.
The paper proposes a straightforward yet effective pipeline for ultrasound understanding: constructing a massive dataset (1.5M exams, 17.7M images) and fine-tuning a standard LVLM (Qwen3-VL-4B) using LoRA. The core methodological contribution is not a new architecture, but the rigorous demonstration that "data scale + clinically faithful alignment" supersedes complex architectural modifications or specialized training strategies in this domain. The approach is simple, relying on examination-level supervision where multiple images are paired with long-form reports, mimicking real clinical workflows. This challenges the prevailing trend of designing intricate multimodal adapters for medical imaging.
The experimental evaluation is comprehensive and robust. The authors benchmark LUMI against a wide array of state-of-the-art general-purpose (InternVL3.5, Qwen3.5, Kimi-VL) and medical-domain (HuatuoGPT, Lingshu, EchoVLM) models across five major ultrasound categories. The results show significant improvements, particularly in clinical fidelity metrics (F1 score) and higher-order NLP metrics (BLEU-4, ROUGE-L). The inclusion of an LLM-based evaluator for clinical correctness is a strong methodological choice that adds depth beyond standard text similarity metrics. Scaling analyses (model and data) provide valuable empirical insights, showing saturation points that guide future resource allocation.
The paper provides detailed hyperparameters, training configurations (LoRA rank, learning rate, batch size), and data preprocessing steps. The dataset size and source descriptions are clear. However, the dataset itself (1.5M exams) is likely too large and privacy-sensitive to be fully open-sourced in its raw form, which may limit direct reproducibility of the training phase for others. The code/model availability is indicated by the project URL, which is crucial for verification.
The primary limitation is the potential for hallucination when presented with incomplete image sets at inference time, as the model is trained on complete examinations. Additionally, the reliance on uncurated, real-world reports introduces noise and variability in language style, which might affect generalization to standardized reporting formats. The study focuses on report generation and lacks detailed evaluation on downstream diagnostic tasks (e.g., specific lesion detection accuracy vs. radiologist agreement).
This work has significant implications for medical AI, demonstrating that high-quality, large-scale data alignment can drive performance gains more effectively than architectural complexity. It encourages the community to prioritize data curation and clinical fidelity in medical LVLM development. The dataset and model could accelerate research in ultrasound AI, potentially improving diagnostic support in resource-limited settings where expert sonographers are scarce. This paper makes a substantial contribution to medical vision-language modeling by demonstrating that large-scale, clinically aligned data curation and simple fine-tuning of standard LVLMs can outperform complex, specialized architectures for ultrasound understanding, providing a new benchmark and paradigm for the field.
Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .
Primary: University of Hong Kong
All Institutions: University of Hong Kong, Snowflake
Spider 2.0-AIFunc introduces the first benchmark for evaluating text-to-SQL systems on AI-native SQL workflows, revealing that current models struggle with semantic parameterization and that complex agent frameworks offer no advantage over minimal setups, signaling a shift in the bottleneck of enterprise text-to-SQL from planning to semantic execution. The paper provides a rigorous evaluation framework and detailed error analysis that will serve as a foundational resource for the community as the industry transitions to AI-integrated database systems.
The paper introduces a novel benchmark construction methodology for "AI-Native SQL," a new paradigm where LLM capabilities are exposed as native SQL functions (e.g., `AI_CLASSIFY`, `AI_SENTIMENT`). The core methodological contribution is an agent-based pipeline that rewrites existing enterprise text-to-SQL tasks (from Spider2-Snow) into AI-native forms. This involves not just transforming the SQL query but also refining the natural language instructions to ensure specification determinism (explicitly defining AI function parameters) and verifying execution determinism (handling stochasticity in AI function outputs via multi-round verification). The approach addresses a critical gap in current benchmarks which ignore the shift towards semantic operators in SQL engines.
The evaluation is rigorous and comprehensive. The authors evaluate 10 state-of-the-art models (proprietary and open-source) using a minimal agent framework to isolate model capability from framework complexity. They provide detailed error analysis stratified by model performance tiers, identifying specific failure modes such as schema grounding, predicate specification, and AI function parameterization. The finding that complex agent frameworks (AutoLink, ReFoRCE, DSR-SQL) do not outperform a minimal setup is a significant and surprising empirical result, suggesting that the bottleneck in AI-Native SQL is primarily model capability (semantic understanding and parameterization) rather than retrieval or planning complexity. The benchmark includes 465 verified instances across 125 databases, providing a robust testbed.
The paper provides a detailed description of the construction pipeline, verification protocols, and evaluation metrics. The code and data are made publicly available on GitHub. The multi-round execution verification protocol ensures that the benchmark instances are stable, which is crucial for reproducibility in the context of stochastic AI functions. The evaluation setup is clearly defined, including the handling of timeouts and the specific comparison logic for results.
The benchmark is currently scoped to Snowflake and its specific set of Cortex AI functions, limiting generalizability to other platforms (BigQuery, Databricks) or function types. The construction relies on a single strong model (Claude Opus 4.5) for rewriting, which may introduce biases in instruction wording or task formulation. The evaluation uses a single agent trajectory per model, which does not capture the variance of stochastic agents or allow for pass@k analysis. Additionally, the benchmark relies on the assumption that the underlying AI functions remain stable over time, which may not hold if the cloud provider updates the backend models.
This paper has significant broader impact as it establishes the first standardized benchmark for evaluating text-to-SQL systems on AI-native workflows. As cloud data platforms increasingly integrate LLMs directly into their query engines, this benchmark will guide the development of more capable and reliable text-to-SQL systems. It highlights the unique challenges of AI-Native SQL, such as parameterization and semantic grounding, which are distinct from traditional text-to-SQL challenges. The findings will influence how researchers design agents and models for enterprise data analysis, potentially accelerating the adoption of AI-native SQL in industry. Spider 2.0-AIFunc introduces the first benchmark for evaluating text-to-SQL systems on AI-native SQL workflows, revealing that current models struggle with semantic parameterization and that complex agent frameworks offer no advantage over minimal setups, signaling a shift in the bottleneck of enterprise text-to-SQL from planning to semantic execution. The paper provides a rigorous evaluation framework and detailed error analysis that will serve as a foundational resource for the community as the industry transitions to AI-integrated database systems.
Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.
Primary: Amazon
All Institutions: Amazon
This paper presents a comprehensive and methodologically sound large-scale evaluation of uncertainty estimation in multilingual LLMs, providing crucial insights into the interplay between language resource levels, model scale, and uncertainty signaling mechanisms.
The paper proposes a rigorous evaluation framework for Uncertainty Estimation (UE) in Large Language Models (LLMs) across 22 languages. The core methodological innovation lies in the evaluation setup: using human-curated Multiple-Choice Question Answering (MCQA) datasets with elicited long-form reasoning to avoid the noise associated with LLM-as-a-judge or embedding-based correctness metrics. This allows for a clean, label-grounded assessment of UE methods (both open-box and closed-box) on the reasoning traces rather than just the final answer. The study systematically compares nine UE methods across varying model scales (270M to 235B) and resource levels. The approach is sound, addressing a critical gap in multilingual trustworthiness evaluation by isolating the uncertainty signal from the correctness signal more effectively than prior work.
The experimental design is comprehensive and large-scale. The authors evaluate 9 models and 9 UE methods across 22 languages, covering high, mid, and low-resource settings. Key findings include: (1) English reasoning significantly boosts UE performance for low-resource languages, suggesting the bottleneck is generation, not comprehension; (2) Self-Verbalized uncertainty outperforms other methods at large scales (235B), while open-box methods are better for smaller models; (3) Sampling-based methods fail on low-resource languages due to lack of diversity signal. The results are robust, supported by statistical significance testing and confidence intervals. The analysis of cross-lingual answer options and threshold calibration adds practical value. The use of parallel datasets (Global-MMLU and MMLU-ProX) ensures comparability.
The paper provides detailed descriptions of the datasets, models, prompts, and UE methods. It mentions the use of LM-Polygraph for implementation. The hardware infrastructure is specified. However, the specific versions of the models (e.g., "Claude 4.5 Sonnet" which appears to be a future/hypothetical name or typo for a current model, and "Gemma3" which is also not yet publicly released as of mid-2024, suggesting this is a very recent or forward-looking paper) and the exact random seeds are not fully detailed in the text provided, though the appendix references suggest they are available. The code release is mentioned but the URL is not provided in the text. The methodology is clear enough for replication if the models and datasets are accessible.
The study is limited to MCQA tasks, which may not fully generalize to open-ended generation where correctness is harder to define. The reliance on specific datasets (Global-MMLU, MMLU-ProX) means results might vary with other benchmarks. The "Claude 4.5 Sonnet" and "Gemma3" references are unusual and might indicate the paper is from the future or uses internal/unreleased models, which could limit immediate reproducibility for the broader community. The study does not include training-based UE methods.
This work has significant implications for deploying trustworthy LLMs in multilingual contexts. By demonstrating that generation language matters more than question language for UE, it provides actionable guidance for system designers (e.g., using English reasoning traces for low-resource languages). It also clarifies the trade-offs between open-box and closed-box methods based on model scale, helping practitioners choose appropriate UE strategies. The findings challenge the assumption that low-resource language performance is solely due to comprehension deficits, highlighting generation quality as a key factor. This paper presents a comprehensive and methodologically sound large-scale evaluation of uncertainty estimation in multilingual LLMs, providing crucial insights into the interplay between language resource levels, model scale, and uncertainty signaling mechanisms.
Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy. We challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding. Across multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to $1.43\times$ and improving accuracy over state-of-the-art by up to $5.1\%$.
Primary: University College London
All Institutions: University College London, Huawei
Lynx introduces a progressive speculative quantization framework that decouples KV cache transfer from decoding initiation, achieving significant latency reductions without sacrificing inference accuracy in long-context LLM serving.
The paper proposes "Lynx," a novel system for disaggregated LLM inference that challenges the assumption that the Key-Value (KV) cache must be fully transferred before decoding begins. The core innovation is a hierarchical split-stream quantization scheme that partitions the KV cache into a high-priority "Anchor" stream (Most Significant Bits) and a low-priority "Residual" stream (Least Significant Bits). By transmitting the Anchor stream first, the decode instance can begin speculative token generation using the coarse-grained KV data. Once the Residual stream arrives, the system verifies the speculative tokens against the full-precision (or higher-precision) KV cache. This approach effectively overlaps network communication with computation, treating the network transfer as a draft model in speculative decoding. The methodology is technically sound, leveraging the observation that MSBs dominate attention score magnitudes due to the exponential nature of Softmax, while LSBs refine precision. The integration of non-linear logarithmic quantization and outlier-aware chunking further enhances the fidelity of the Anchor stream.
The evaluation is comprehensive, covering three models (LLaMA 3.1 8B, Qwen 3 32B, Mistral 3 24B) and three datasets (MMLU-Pro, Needle-in-the-Haystack, QMSum) across varying context lengths (up to 128K) and bandwidths (10-50 Gbps). The results demonstrate that Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit quantization while maintaining accuracy equivalent to 8-bit or BF16 inference. Specifically, it improves TTFT over standard 8-bit quantization by up to 1.43x and improves accuracy over state-of-the-art compression methods (like CacheGen) by up to 5.1%. The paper includes detailed ablation studies on context length scaling and bandwidth variations, showing that the benefits of speculative overlap increase with longer contexts and lower bandwidths. The use of Ascend NPUs (Huawei hardware) is a specific constraint but does not detract from the generalizability of the system design principles.
The paper provides significant implementation details, including the quantization algorithm (Algorithm 1), the split-stream construction logic, and the speculative verification protocol. It mentions implementation in ~2k lines of Ascend-C kernels and ~2k lines of Python, integrated into vLLM-Ascend. However, the code is not publicly available (no GitHub URL provided), and the evaluation is conducted on proprietary Huawei Ascend hardware, which may limit direct reproducibility for researchers using standard NVIDIA GPU stacks. The detailed description of the SerDes protocol and the non-blocking runtime architecture offers a strong basis for future reproduction.
The primary limitation is the reliance on specific hardware (Ascend NPUs) and the lack of public code. The speculative decoding verification introduces computational overhead; while the paper argues this is negligible compared to communication savings, this overhead scales with the number of speculative tokens and could become significant in very high-bandwidth, low-latency scenarios where the communication bottleneck is less severe. Additionally, the approach assumes a disaggregated prefill-decode architecture, which is not universal for all LLM serving setups. The accuracy guarantee relies on the verification step, which implies that if the Residual stream is delayed or lost, the system must wait, potentially negating the latency benefits in unstable network conditions.
This work has significant implications for the efficiency and scalability of long-context LLM serving, particularly in cloud environments where disaggregated inference is becoming standard. By enabling high-precision inference with lower effective latency, it allows for more responsive AI agents and retrieval-augmented generation systems. The technique of using partial data for speculative execution could inspire similar approaches in other areas of distributed machine learning where data dependencies are hierarchical or can be approximated. Lynx introduces a progressive speculative quantization framework that decouples KV cache transfer from decoding initiation, achieving significant latency reductions without sacrificing inference accuracy in long-context LLM serving.