Last 7 Days (September 08 – September 14, 2026)
We study how model post-training and test-time inference design affect natural-language proof generation for hard olympiad mathematics. Starting from Nemotron 3 Ultra, we train two specialist checkpoints using supervised fine-tuning and reinforcement learning, and evaluate checkpoint choice, verification, and refinement. Based on these findings, we present an open-model test-time-compute pipeline. The system operates entirely in natural language, with no formal prover, external tools, or internet access. Three Nemotron 3 Ultra checkpoints - the general-availability model and two post-trained specialists - power an iterative search that generates, verifies, and refines candidate proofs; a separate high-compute stage then selects each final submission. The system scored 30 out of 42 points at IMO 2026, reaching the gold-medal threshold. We release the two post-trained checkpoints as well as the training data, the training and inference code, the submitted solutions, and Nemotron-IMO-Bench, a new benchmark of 200 novel olympiad-level problems.
Primary: NVIDIA
All Institutions: NVIDIA
The paper presents an open-model test-time-compute pipeline that achieves IMO Gold medal performance using natural language proofs. It contributes a rigorous methodology for iterative proof generation and verification, along with a new benchmark and open resources, marking a significant step in the capabilities of open-source LLMs for complex reasoning tasks.
The paper proposes a sophisticated test-time compute pipeline for natural language proof generation. The core methodology involves training two specialist checkpoints from Nemotron 3 Ultra using SFT and RL, then orchestrating an iterative search process that generates, verifies, and refines candidate proofs without relying on formal provers or external tools. The inclusion of a high-compute selection stage for final submissions is a key architectural decision that differentiates this from simple best-of-N sampling.
The primary result is the achievement of 30/42 points at IMO 2026, reaching the gold-medal threshold. This is a significant empirical milestone for open models. The introduction of Nemotron-IMO-Bench (200 novel problems) provides a necessary evaluation framework, though the sample size is relatively small compared to standard benchmarks. The ablation studies on checkpoint choice and verification strategies are critical to understanding the pipeline's effectiveness.
The paper explicitly states the release of post-trained checkpoints, training data, code, and submitted solutions. This high level of transparency significantly enhances reproducibility, allowing the community to verify the results and build upon the pipeline.
The reliance on a specific base model (Nemotron 3 Ultra) may limit immediate generalizability to other architectures. The benchmark size (200 problems) is small for statistical robustness. The high compute cost of the iterative search and selection stages may be prohibitive for some users.
This work demonstrates that open models can achieve human-competitive performance in high-level mathematical reasoning, potentially accelerating research in automated theorem proving and educational AI. The open release of assets fosters community development in this niche. The paper presents an open-model test-time-compute pipeline that achieves IMO Gold medal performance using natural language proofs. It contributes a rigorous methodology for iterative proof generation and verification, along with a new benchmark and open resources, marking a significant step in the capabilities of open-source LLMs for complex reasoning tasks.
Sparsity is a powerful structural resource in optimization and statistics. We develop frameworks for leveraging sparsity in sampling problems over the Hamming slice $\mathcal{X}_k^d:=\{\mathbf{x}\in\{\pm 1\}^d:|\{i:\mathbf{x}_i=1\}|=k\}$, in high-dimensional regimes where $k\ll d$ (i.e., where $\mathcal{X}_k^d$ is \emph{highly magnetized}). We use our frameworks to design improved samplers for canonical problems in the study of \emph{Ising models} and \emph{Bayesian sparse linear regression}. Our first main result considers the \emph{Sherrington--Kirkpatrick} (SK) model restricted to fixed-magnetization slices $\mathcal{X}_k^d$. We give a polynomial-time sampler for fixed-magnetization SK models at any inverse temperature $β>0$, under arbitrary external fields, provided that $k\le c_βd$ for an appropriate constant $c_β$. By combining this result with an annealing strategy for estimating normalizing constants, we obtain polynomial-time samplers for the SK model at arbitrarily low temperatures under a sufficiently strong external field of strength $h$. In the large-$β$ limit, our framework permits sampling at field strengths within constant factors of the \emph{Almeida--Thouless line} delineating the replica-symmetric and replica-symmetry-breaking regions ([dAT78]), improving polynomially over the field strength $h(β)$ required by the recent work of [BAR26]. Our second main result concerns the measurement complexity of polynomial-time Bayesian sparse linear regression. Recent work by [KSTZ25] shows how to sample from the canonical \emph{Gaussian spike-and-slab posterior} with expected sparsity $k$, at any signal-to-noise ratio, given $n\gtrsim k^3\log^3 d$ Gaussian measurements. We improve this requirement to $n\gtrsim k^{3/2}\log^2 d+k\log^3 d$, using a common sparsity-aware framework underlying both our results.
Primary: University of Washington
All Institutions: University of Washington, Amazon AI
The paper presents a novel sparsity-aware framework for sampling in high-dimensional Ising models and Bayesian regression, achieving polynomial-time samplers at low temperatures near the Almeida-Thouless line and improving measurement complexity for sparse linear regression. This is a significant theoretical contribution to probabilistic inference and statistical physics, with rigorous proofs and clear improvements over prior work, though it lacks empirical validation and is limited to specific regimes of field strength and sparsity.
The paper introduces a "sparsity-aware framework" for sampling on the Hamming slice, which is a highly magnetized subset of the hypercube. The core technical contribution is a "trickle-down" theorem that allows bounding the Dobrushin coefficient (or similar contraction metrics) for the full Ising model by analyzing the behavior on sparser slices. This is a sophisticated theoretical tool that bridges combinatorial optimization and statistical physics. The application to the Sherrington-Kirkpatrick (SK) model is particularly strong, as it provides a polynomial-time sampler at low temperatures (high beta) under strong external fields, approaching the Almeida-Thouless line. This is a significant improvement over previous bounds which required much stronger fields. The application to Bayesian sparse linear regression improves the measurement complexity from $O(k^3 \log^3 d)$ to $O(k^{3/2} \log^2 d + k \log^3 d)$, which is a substantial polynomial improvement in the dependence on sparsity $k$. The methodology is rigorous, relying on precise probabilistic bounds and annealing strategies for normalizing constants.
This is a purely theoretical paper. There are no empirical experiments, benchmarks, or datasets evaluated. The "results" are mathematical theorems and corollaries regarding the existence and complexity of samplers. Therefore, experimental evaluation is not applicable in the traditional sense, but the theoretical guarantees are the primary contribution.
The paper is a theoretical work, so reproducibility refers to the verifiability of the proofs. The paper is well-structured with clear sections for preliminaries, main theorems, and proofs. The use of LLMs (GPT 5.6 Pro) for exploring applications and suggesting perturbation strategies is disclosed in the AI Disclosure section. While the authors state they take full responsibility, the reliance on an LLM for a key lemma's strategy is a notable aspect of the research process. However, the mathematical content stands on its own.
The results for the SK model are conditional on the external field $h$ being sufficiently strong, specifically within constant factors of the Almeida-Thouless line. This does not solve the sampling problem for the full SK model at low temperatures without strong fields, which remains a major open problem. The improvement in measurement complexity for sparse linear regression, while significant, still has a $k^{3/2}$ dependence, which may be suboptimal compared to the information-theoretic lower bounds in some regimes. The paper is highly technical and may have limited immediate practical impact for practitioners who do not work on these specific sampling problems.
The techniques developed here, particularly the sparsity-aware sampling frameworks and the trickle-down theorem, could have broader applications in other areas of statistical physics and high-dimensional statistics. The ability to sample from complex posterior distributions in sparse regimes is relevant to many machine learning tasks, including feature selection and compressed sensing. The theoretical insights into the relationship between sparsity and sampling complexity could inform the design of new algorithms for Bayesian inference. The paper presents a novel sparsity-aware framework for sampling in high-dimensional Ising models and Bayesian regression, achieving polynomial-time samplers at low temperatures near the Almeida-Thouless line and improving measurement complexity for sparse linear regression. This is a significant theoretical contribution to probabilistic inference and statistical physics, with rigorous proofs and clear improvements over prior work, though it lacks empirical validation and is limited to specific regimes of field strength and sparsity.
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
AuK introduces a unified open-source foundational model for speech generation and editing. The paper demonstrates a comprehensive approach to unifying diverse speech tasks through a shared instruction-based interface, leveraging a hybrid flow-matching architecture and advanced post-training strategies to achieve state-of-the-art performance and efficient inference.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid rectified-flow Transformer combining dual-stream MMDiT blocks for semantic-acoustic interaction and single-stream DiT blocks for refinement. The methodology is technically sound, leveraging a frozen MLLM (Qwen2.5-Omni) for semantic conditioning and a jointly trained VAE for acoustic conditioning. The training pipeline is sophisticated, involving a two-stage pre-training curriculum (generation warm-up followed by joint generation-editing), followed by post-training via human-feedback preference optimization (DPO variant) for editing and Flow-GRPO for generation. The distillation strategy using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference is a significant engineering contribution. The novelty lies primarily in the unification of diverse tasks (TTS, editing, enhancement, separation) under a single instruction-based interface and the specific combination of post-training strategies for speech.
The paper claims leading performance on zero-shot and instruction-controlled speech generation and editing benchmarks. However, the provided text is truncated before the detailed experimental results section, making it difficult to fully verify the magnitude of improvements or the specific benchmarks used (e.g., LibriSpeech, Seed-TTS, etc.). The data construction is extensive (3.03 billion instances, 1.95 million hours), which is a major strength. The evaluation of the distilled model (AuK-Flash) showing a 4.5x speedup is a strong practical result. Without the full results tables, the technical impact score is slightly conservative, but the scale of data and the comprehensive task coverage suggest high impact.
The paper states that both source code and model weights are released. The training details are provided in high granularity (hyperparameters, data ratios, loss weights, optimizer settings). The use of open-source components (Qwen2.5-Omni, F5-TTS for data synthesis, etc.) aids reproducibility. The specific data construction pipelines are described in detail, though the exact proprietary datasets used for some parts (e.g., in-house dialect corpus) may limit full reproduction of the data mix.
The model is large (1.5B parameters for the backbone, plus the MLLM and VAE), which may be resource-intensive for some users, though the distilled version mitigates this. The reliance on synthetic data for editing tasks (using other TTS models to generate targets) may introduce artifacts or bias. The evaluation of "open-ended" editing tasks is inherently subjective, and while human feedback is used for training, the final evaluation metrics for these tasks may not fully capture user satisfaction. The paper does not provide a detailed ablation study on the specific contribution of each post-training stage in the visible text.
This work has high potential impact by providing a unified, open-source foundation for speech manipulation. It bridges the gap between discrete task-specific models and a general-purpose speech AI. The release of code and weights will likely accelerate research in speech editing and generation. The efficient inference method (4-step) makes it viable for real-time applications. AuK introduces a unified open-source foundational model for speech generation and editing. The paper demonstrates a comprehensive approach to unifying diverse speech tasks through a shared instruction-based interface, leveraging a hybrid flow-matching architecture and advanced post-training strategies to achieve state-of-the-art performance and efficient inference.
Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.
Primary: Alibaba Group
All Institutions: Alibaba Group, Tsinghua University
The paper presents a comprehensive and technically rigorous approach to training large MoE agents via RL, introducing novel stabilization techniques (TITO and R3) that are critical for scaling RL to sparse models. Its strong empirical results on terminal benchmarks demonstrate a significant leap in autonomous software engineering capabilities, establishing a new standard for long-horizon agentic tasks.
The paper introduces a robust framework for training large Mixture-of-Experts (MoE) models using Reinforcement Learning (RL) in long-horizon, stateful environments (Linux terminals). The core technical contribution is the "TITO" (Token-In-Token-Out) and "R3" (Rollout Routing Replay) mechanisms. TITO addresses the training-inference mismatch caused by tokenization drift in multi-turn agentic loops by ensuring the trainer consumes the exact token identifiers generated by the sampler, repairing boundary discrepancies. R3 addresses the unique challenge of MoE models where discrete expert routing decisions made during inference can differ from those recomputed during training due to numerical precision differences; R3 records the expert masks during rollout and replays them during the training forward pass. Additionally, the paper proposes a dense process reward based on the absolute number of passing verifier assertions, which provides a much richer learning signal than binary success/fail rewards for difficult tasks. The methodology is rigorous, addressing specific, non-trivial engineering challenges in scaling RL to sparse models.
The experiments are extensive, utilizing a 122B parameter MoE model (Qwen3.5-122B-A10B) trained on 15,000 synthesized terminal tasks. The evaluation is conducted on three held-out benchmarks: Terminal-Bench 2.1, Long-Horizon Terminal-Bench (LHTB), and Terminal-Bench Hard. The results show a significant improvement from the base model (43.8%) to the final T1 model (64.0%) on Terminal-Bench 2.1, surpassing several frontier models like GPT-5.4 and GLM-5.1. The ablation studies clearly demonstrate the necessity of both TITO and R3 for stabilizing training, as well as the benefit of the dense reward over binary rewards. The analysis of training dynamics, including explained variance and turn count growth, provides deep insight into the learning process.
The paper provides high levels of detail regarding the training infrastructure, hyperparameters, and specific mechanisms like the routing replay algorithm. It describes the dataset construction process (RST) and the audit criteria for task selection. However, the specific code for the TITO and R3 implementations is not explicitly linked in the provided text (though likely available via the authors' repositories), and the exact hardware configuration for the 122B model training is only partially detailed. The use of proprietary or specific internal tools (like the "slime" framework) may limit immediate reproducibility for external labs without access to those specific infrastructure components.
The primary limitation is the reliance on a large-scale, proprietary infrastructure to train a 122B MoE model, making the results difficult to replicate for smaller labs. The evaluation is limited to terminal tasks, and while the paper argues for generalization, the specific domain of Linux shell operations is narrow. Additionally, the paper acknowledges that the model still struggles with the hardest tasks (Terminal-Bench Hard) compared to some larger dense models, and the "inefficient search" behavior (high turn counts) suggests that the model has not fully mastered optimal planning, merely improving its persistence and partial success rates.
This work is significant for the development of autonomous agents capable of performing complex, long-horizon software engineering tasks. By solving the stability issues of RL in MoE models, it paves the way for more efficient and capable agentic systems. The dense reward design is a valuable contribution to the RL community, offering a practical solution to the sparse reward problem in verifiable environments. The findings on training-inference consistency in sparse models will be relevant to any group working on large-scale MoE RL. The paper presents a comprehensive and technically rigorous approach to training large MoE agents via RL, introducing novel stabilization techniques (TITO and R3) that are critical for scaling RL to sparse models. Its strong empirical results on terminal benchmarks demonstrate a significant leap in autonomous software engineering capabilities, establishing a new standard for long-horizon agentic tasks.
Commercial text-to-image systems silently revise user prompts before generating images, a step users typically cannot disable or even see. Yet, existing audits of cultural bias examine only the final images and treat generation as a single pipeline, so they cannot tell where the bias originates. We introduce WORLDVIEW, a multilingual benchmark of 8,960 prompts across 15 languages and 31 language-context pairings. Using it, we audit the revision layer in three systems (DALL-E-3, Imagen-4, GPT-Image-1.5) through a three-step analysis of how heavily it marks each cultural context, whether it flattens that context into a narrow vocabulary, and whether that vocabulary is stereotypical. Relative to a no-context English baseline, the US is the least-marked context, while non-Western and non-Anglophone contexts are marked far more heavily, flattened into narrow vocabularies applied across topically diverse prompts, and reduced to recognizable cultural stereotypes. Comparing images from original versus revised prompts on models without a revision layer, we identify the layer itself as a previously undocumented, causal source of this stereotyping. To locate cultural bias, and fix it, we must audit the system as deployed, not the model alone.
Primary: University of Zurich
All Institutions: University of Zurich, Stanford University
The paper identifies the prompt revision layer in commercial text-to-image systems as a previously undocumented, causal source of cultural bias and stereotyping. By introducing the WORLDVIEW benchmark and a three-step audit framework, it demonstrates that non-Western cultures are disproportionately flattened into narrow, stereotypical vocabularies before image generation, necessitating a shift in bias mitigation strategies to include the entire deployed pipeline.
The paper introduces a rigorous three-step analytical framework (Contextual Markedness Score, Cultural Flattening Score, and Stereotypical Content Analysis) to audit the "prompt revision" layer in commercial text-to-image systems. By isolating this intermediate text-to-text transformation, the authors address a critical blind spot in prior bias audits that treated the generation pipeline as monolithic. The methodology is sound, utilizing sentence embeddings for markedness and TF-IDF for flattening, with a well-designed causal ablation study using open-source models to verify that the revision layer, not just the image model, drives stereotypical outputs.
The experimental setup is robust, featuring a new multilingual benchmark (WORLDVIEW) with 8,960 prompts across 15 languages and 31 contexts. The evaluation covers three major commercial systems (DALL-E-3, Imagen-4, GPT-Image-1.5). The results clearly demonstrate that non-Western contexts are disproportionately marked and flattened into stereotypical vocabularies (e.g., Egypt reduced to pyramids, Finland to snow). The causal link established via ablation on SDXL and Flux models is a strong technical contribution, proving the revision layer is a distinct source of bias.
High. The authors release the benchmark, revised prompts, metrics, and evaluation toolkit on GitHub. The use of standard APIs and open-source models for the ablation study ensures that the findings can be replicated by other researchers.
The causal ablation is restricted to English-speaking contexts due to language confounds, though a case study on Switzerland partially mitigates this. The study relies on VQA models for image description, which may introduce their own biases, although the authors argue the comparative design mitigates this. The benchmark, while broad, still underrepresents certain regions like Sub-Saharan Africa.
This paper has significant implications for AI safety and fairness research. It shifts the focus of bias mitigation from just the image generation model to the entire deployed system, including preprocessing layers. This insight is crucial for developers of commercial T2I systems, as it highlights that debiasing the image model alone is insufficient if the prompt revision layer injects stereotypes. It provides a actionable framework for auditing these hidden layers. The paper identifies the prompt revision layer in commercial text-to-image systems as a previously undocumented, causal source of cultural bias and stereotyping. By introducing the WORLDVIEW benchmark and a three-step audit framework, it demonstrates that non-Western cultures are disproportionately flattened into narrow, stereotypical vocabularies before image generation, necessitating a shift in bias mitigation strategies to include the entire deployed pipeline.
Most of mathematical knowledge has been communicated through so-called informal use of mathematics and natural language. With large language models (LLMs) being highly adept in using natural language, they achieve strong performance, yet not perfect, in informal mathematical reasoning. Restraining LLMs to informal reasoning misses out on the opportunity to use the discrete verification abilities that machines offer through machine-checkable proofs. In this paper, we bridge the gap between informal and formal reasoning by integrating Lean signals into the informal reasoning process. We introduce Magenta, a training-free agentic pipeline that, given only a natural-language problem, produces an answer, expresses it as a Lean 4 statement, and constructs a machine-checked proof. A statement judge verifies whether the formalisation preserves the original problem, while an error-attribution judge routes failed attempts either to mathematical re-derivation or local Lean repair. Magenta achieves 100% accuracy across all evaluated olympiad benchmarks, including AIME 2025, AIME 2026, and HMMT February 2026. When paired with the open-weight K2-Horizon-7B reasoner, it solves all six IMO 2026 problems. Our analysis shows that statement adjudication is essential for preventing false certificates and that feedback-guided correction outperforms independent resampling on difficult problems.
Primary: Imperial College London
All Institutions: Imperial College London, University College London
Magenta introduces a training-free agentic pipeline that integrates Lean 4 formal verification into LLM mathematical reasoning, achieving 100% accuracy on AIME/HMMT benchmarks and solving all IMO 2026 problems with a 7B model by using statement adjudication and error attribution to guide self-correction.
The paper introduces "Magenta," a training-free agentic pipeline that integrates informal LLM reasoning with formal Lean 4 verification. The core methodological contribution is the closed-loop feedback mechanism: a "Statement Judge" ensures the formalization is faithful to the natural language problem (addressing the autoformalization gap), and an "Error Attribution Judge" routes verification failures to either mathematical re-derivation or local proof repair. This distinction is crucial, as it prevents the system from wasting compute on syntax errors when the math is wrong, or vice versa. The use of a deterministic verifier (Lean) as a hard constraint on LLM outputs is a strong architectural choice that differentiates it from purely learned reward models.
The experiments are extensive, covering AIME 2025/2026, HMMT 2026, and IMO 2026. The claim of 100% accuracy on AIME/HMMT and solving all IMO 2026 problems with a 7B model is striking. Ablations effectively demonstrate the necessity of the Statement Judge (showing high false certification rates without it) and the Error Judge (showing superior performance over independent resampling). The robustness check via paraphrasing helps mitigate concerns about data contamination, though the perfect scores on recent benchmarks remain a point of scrutiny.
The paper provides high reproducibility standards, including verbatim prompt templates for all components (Reasoner, Formaliser, Judges, Prover), detailed hyperparameter settings, and specific model versions (e.g., Lean v4.29.1, Mathlib). The use of open-weight models for the core pipeline (K2-Horizon, Goedel, Leanstral) further enhances reproducibility compared to closed-source baselines.
The primary limitation is the "soft certificate" nature of the output: Lean verifies the proof relative to the generated statement, but the Statement Judge is a learned proxy that can still fail, meaning the final answer is not unconditionally guaranteed correct with respect to the original natural language problem. Additionally, the computational cost is significant, requiring multiple LLM calls and Lean compilation cycles, which may limit real-time applicability. The reliance on specific, potentially proprietary or rapidly evolving model versions (like K2-Horizon) could affect long-term reproducibility.
This work bridges a critical gap in AI for Science, enabling LLMs to produce mathematically rigorous, machine-checked proofs. It sets a new standard for mathematical reasoning benchmarks and offers a template for integrating formal verification into other domains where correctness is paramount (e.g., code generation, logic synthesis). The finding that small models can solve IMO problems with verification-guided correction challenges the assumption that scale is the only driver of capability in complex reasoning tasks. Magenta introduces a training-free agentic pipeline that integrates Lean 4 formal verification into LLM mathematical reasoning, achieving 100% accuracy on AIME/HMMT benchmarks and solving all IMO 2026 problems with a 7B model by using statement adjudication and error attribution to guide self-correction.
We study how model post-training and test-time inference design affect natural-language proof generation for hard olympiad mathematics. Starting from Nemotron 3 Ultra, we train two specialist checkpoints using supervised fine-tuning and reinforcement learning, and evaluate checkpoint choice, verification, and refinement. Based on these findings, we present an open-model test-time-compute pipeline. The system operates entirely in natural language, with no formal prover, external tools, or internet access. Three Nemotron 3 Ultra checkpoints - the general-availability model and two post-trained specialists - power an iterative search that generates, verifies, and refines candidate proofs; a separate high-compute stage then selects each final submission. The system scored 30 out of 42 points at IMO 2026, reaching the gold-medal threshold. We release the two post-trained checkpoints as well as the training data, the training and inference code, the submitted solutions, and Nemotron-IMO-Bench, a new benchmark of 200 novel olympiad-level problems.
Primary: NVIDIA
All Institutions: NVIDIA
The paper presents an open-model test-time-compute pipeline that achieves IMO Gold medal performance using natural language proofs. It contributes a rigorous methodology for iterative proof generation and verification, along with a new benchmark and open resources, marking a significant step in the capabilities of open-source LLMs for complex reasoning tasks.
The paper proposes a sophisticated test-time compute pipeline for natural language proof generation. The core methodology involves training two specialist checkpoints from Nemotron 3 Ultra using SFT and RL, then orchestrating an iterative search process that generates, verifies, and refines candidate proofs without relying on formal provers or external tools. The inclusion of a high-compute selection stage for final submissions is a key architectural decision that differentiates this from simple best-of-N sampling.
The primary result is the achievement of 30/42 points at IMO 2026, reaching the gold-medal threshold. This is a significant empirical milestone for open models. The introduction of Nemotron-IMO-Bench (200 novel problems) provides a necessary evaluation framework, though the sample size is relatively small compared to standard benchmarks. The ablation studies on checkpoint choice and verification strategies are critical to understanding the pipeline's effectiveness.
The paper explicitly states the release of post-trained checkpoints, training data, code, and submitted solutions. This high level of transparency significantly enhances reproducibility, allowing the community to verify the results and build upon the pipeline.
The reliance on a specific base model (Nemotron 3 Ultra) may limit immediate generalizability to other architectures. The benchmark size (200 problems) is small for statistical robustness. The high compute cost of the iterative search and selection stages may be prohibitive for some users.
This work demonstrates that open models can achieve human-competitive performance in high-level mathematical reasoning, potentially accelerating research in automated theorem proving and educational AI. The open release of assets fosters community development in this niche. The paper presents an open-model test-time-compute pipeline that achieves IMO Gold medal performance using natural language proofs. It contributes a rigorous methodology for iterative proof generation and verification, along with a new benchmark and open resources, marking a significant step in the capabilities of open-source LLMs for complex reasoning tasks.
Open-source large language models (LLMs) are increasingly competitive with closed-source models while offering transparency and the ability to run inference without exposing user inputs to a service provider. However, running large-scale models locally requires substantial computational resources. In practice, users may still resort to a third-party provider, giving rise to privacy and correctness concerns. Existing solutions that address these problems often impose substantial server overhead or introduce additional trust assumptions. In this paper, we present Maverick, a novel approach to private and verifiable LLM inference based on a protocol for delegating matrix-vector multiplication, a dominant operation in LLMs. At its core, Maverick provides, to our knowledge, the first information-theoretically sound verification protocol for matrix-vector multiplication delegation with transparent preprocessing, efficient (batch) verification, and virtually no server overhead. We combine this verification primitive with LPN-based pseudorandom masking to provide input privacy. We implement our matrix-vector delegation primitive and use it to build an end-to-end prototype of Maverick, which we evaluate on Qwen3-4B by measuring throughput in tokens per second. We evaluate client configurations with 1-8 threads. With one client thread and a CPU server using up to 128 threads, Maverick achieves throughput gains over local inference of up to 17x when privacy masks are generated online, 45x when they are precomputed, and 44x when only verification is required. With four client threads, the corresponding gains are 13x, 18x, and 17x. When server computation is no longer the bottleneck, client-side microbenchmarks with simulated network delay show speedups of 12x-20x, 34x-135x, and 38x-157x.
Primary: Yale University
All Institutions: Yale University
Maverick introduces a practical framework for private and verifiable LLM inference by delegating matrix-vector multiplications with minimal server overhead. The paper presents a novel information-theoretically sound verification protocol combined with LPN-based masking, achieving significant throughput gains over local inference on Qwen3-4B, thereby offering a viable path for secure, resource-efficient remote LLM usage.
The paper proposes Maverick, a system for private and verifiable LLM inference that delegates the dominant matrix-vector multiplication (MVM) operations to a server while ensuring privacy and correctness. The core contribution is a verification protocol for MVM delegation that is information-theoretically sound, features transparent preprocessing, and incurs virtually no server overhead. This is achieved by combining a novel verification primitive with LPN (Learning With Noisy) based pseudorandom masking for input privacy. The methodology is robust, addressing the two main pain points of remote inference: trust in the server's correctness and the leakage of user prompts. By shifting the heavy computational load of verification to the client (which is feasible for modern CPUs) and keeping the server's overhead minimal, the approach is architecturally sound for the specific use case of resource-constrained clients delegating to powerful servers.
The evaluation is conducted on the Qwen3-4B model, measuring throughput in tokens per second. The results are impressive, showing throughput gains of up to 17x-45x over local inference depending on the configuration (online vs. precomputed masks, verification-only vs. full privacy). The paper provides detailed breakdowns for different client thread counts (1-8 threads) and server configurations (up to 128 threads). The inclusion of microbenchmarks with simulated network delay further strengthens the evaluation by isolating client-side bottlenecks. The choice of a 4B parameter model is reasonable for a prototype, as it is large enough to be relevant but small enough to be tested on standard hardware.
The paper provides sufficient detail on the protocol design and the specific model used (Qwen3-4B). However, as an arXiv preprint with no explicit mention of a public code repository in the provided text, full reproducibility is currently limited. The authors are from Yale, a reputable institution, which increases confidence in the implementation quality, but the lack of a linked codebase is a minor drawback for immediate replication.
The primary limitation is the scope of the evaluation, which is limited to a single model size (4B parameters). It is unclear how the overhead scales for much larger models (e.g., 70B+), although the linear nature of MVM suggests it should scale well. Additionally, the reliance on LPN-based masking introduces a security parameter that must be carefully chosen to balance privacy and performance, which may not be trivial for all threat models. The paper also focuses on CPU-based servers; the applicability to GPU-accelerated servers, which are the norm for LLM inference, is not fully explored in the provided text.
This work has significant potential impact on the deployment of open-source LLMs. By enabling private and verifiable inference without requiring users to own high-end hardware, it democratizes access to powerful AI models while preserving privacy. This could be particularly impactful for mobile devices, IoT, or edge computing scenarios where local inference is too slow or resource-intensive. It also addresses the growing concern of "trust" in AI services, providing a cryptographic guarantee of correctness that is lacking in most current cloud inference APIs. Maverick introduces a practical framework for private and verifiable LLM inference by delegating matrix-vector multiplications with minimal server overhead. The paper presents a novel information-theoretically sound verification protocol combined with LPN-based masking, achieving significant throughput gains over local inference on Qwen3-4B, thereby offering a viable path for secure, resource-efficient remote LLM usage.
Sparsity is a powerful structural resource in optimization and statistics. We develop frameworks for leveraging sparsity in sampling problems over the Hamming slice $\mathcal{X}_k^d:=\{\mathbf{x}\in\{\pm 1\}^d:|\{i:\mathbf{x}_i=1\}|=k\}$, in high-dimensional regimes where $k\ll d$ (i.e., where $\mathcal{X}_k^d$ is \emph{highly magnetized}). We use our frameworks to design improved samplers for canonical problems in the study of \emph{Ising models} and \emph{Bayesian sparse linear regression}. Our first main result considers the \emph{Sherrington--Kirkpatrick} (SK) model restricted to fixed-magnetization slices $\mathcal{X}_k^d$. We give a polynomial-time sampler for fixed-magnetization SK models at any inverse temperature $β>0$, under arbitrary external fields, provided that $k\le c_βd$ for an appropriate constant $c_β$. By combining this result with an annealing strategy for estimating normalizing constants, we obtain polynomial-time samplers for the SK model at arbitrarily low temperatures under a sufficiently strong external field of strength $h$. In the large-$β$ limit, our framework permits sampling at field strengths within constant factors of the \emph{Almeida--Thouless line} delineating the replica-symmetric and replica-symmetry-breaking regions ([dAT78]), improving polynomially over the field strength $h(β)$ required by the recent work of [BAR26]. Our second main result concerns the measurement complexity of polynomial-time Bayesian sparse linear regression. Recent work by [KSTZ25] shows how to sample from the canonical \emph{Gaussian spike-and-slab posterior} with expected sparsity $k$, at any signal-to-noise ratio, given $n\gtrsim k^3\log^3 d$ Gaussian measurements. We improve this requirement to $n\gtrsim k^{3/2}\log^2 d+k\log^3 d$, using a common sparsity-aware framework underlying both our results.
Primary: University of Washington
All Institutions: University of Washington, Amazon AI
The paper presents a novel sparsity-aware framework for sampling in high-dimensional Ising models and Bayesian regression, achieving polynomial-time samplers at low temperatures near the Almeida-Thouless line and improving measurement complexity for sparse linear regression. This is a significant theoretical contribution to probabilistic inference and statistical physics, with rigorous proofs and clear improvements over prior work, though it lacks empirical validation and is limited to specific regimes of field strength and sparsity.
The paper introduces a "sparsity-aware framework" for sampling on the Hamming slice, which is a highly magnetized subset of the hypercube. The core technical contribution is a "trickle-down" theorem that allows bounding the Dobrushin coefficient (or similar contraction metrics) for the full Ising model by analyzing the behavior on sparser slices. This is a sophisticated theoretical tool that bridges combinatorial optimization and statistical physics. The application to the Sherrington-Kirkpatrick (SK) model is particularly strong, as it provides a polynomial-time sampler at low temperatures (high beta) under strong external fields, approaching the Almeida-Thouless line. This is a significant improvement over previous bounds which required much stronger fields. The application to Bayesian sparse linear regression improves the measurement complexity from $O(k^3 \log^3 d)$ to $O(k^{3/2} \log^2 d + k \log^3 d)$, which is a substantial polynomial improvement in the dependence on sparsity $k$. The methodology is rigorous, relying on precise probabilistic bounds and annealing strategies for normalizing constants.
This is a purely theoretical paper. There are no empirical experiments, benchmarks, or datasets evaluated. The "results" are mathematical theorems and corollaries regarding the existence and complexity of samplers. Therefore, experimental evaluation is not applicable in the traditional sense, but the theoretical guarantees are the primary contribution.
The paper is a theoretical work, so reproducibility refers to the verifiability of the proofs. The paper is well-structured with clear sections for preliminaries, main theorems, and proofs. The use of LLMs (GPT 5.6 Pro) for exploring applications and suggesting perturbation strategies is disclosed in the AI Disclosure section. While the authors state they take full responsibility, the reliance on an LLM for a key lemma's strategy is a notable aspect of the research process. However, the mathematical content stands on its own.
The results for the SK model are conditional on the external field $h$ being sufficiently strong, specifically within constant factors of the Almeida-Thouless line. This does not solve the sampling problem for the full SK model at low temperatures without strong fields, which remains a major open problem. The improvement in measurement complexity for sparse linear regression, while significant, still has a $k^{3/2}$ dependence, which may be suboptimal compared to the information-theoretic lower bounds in some regimes. The paper is highly technical and may have limited immediate practical impact for practitioners who do not work on these specific sampling problems.
The techniques developed here, particularly the sparsity-aware sampling frameworks and the trickle-down theorem, could have broader applications in other areas of statistical physics and high-dimensional statistics. The ability to sample from complex posterior distributions in sparse regimes is relevant to many machine learning tasks, including feature selection and compressed sensing. The theoretical insights into the relationship between sparsity and sampling complexity could inform the design of new algorithms for Bayesian inference. The paper presents a novel sparsity-aware framework for sampling in high-dimensional Ising models and Bayesian regression, achieving polynomial-time samplers at low temperatures near the Almeida-Thouless line and improving measurement complexity for sparse linear regression. This is a significant theoretical contribution to probabilistic inference and statistical physics, with rigorous proofs and clear improvements over prior work, though it lacks empirical validation and is limited to specific regimes of field strength and sparsity.
Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.
Primary: University of Illinois Urbana-Champaign (Inferred from author Junchao Huang and project page context; note: text is corrupted)
All Institutions: University of Illinois Urbana-Champaign, NVIDIA (Inferred from co-authors often associated with this work, though text is corrupted)
Proposes a dual-noise masking rollout strategy to mitigate mode collapse in autoregressive video diffusion distillation, significantly improving visual quality and motion dynamics without additional data or post-training. The paper presents a well-motivated, theoretically grounded, and empirically validated method that addresses a critical limitation in current AR video generation techniques, offering a practical and effective solution for the community.
The paper proposes "Mask Forcing," a strategy to mitigate mode collapse in autoregressive (AR) video diffusion distillation. The core innovation is the "Dual-Noise Masking Rollout," which injects cleaner signals (lower noise levels) into the student's self-rollout inputs via random spatial and temporal masks. This serves two purposes: (1) it perturbs the rollout trajectory to expose the student to broader regions of the teacher distribution, counteracting the mode-seeking behavior of the reverse KL objective in Distribution Matching Distillation (DMD); and (2) it provides cleaner context tokens that guide the denoising of noisier tokens, reducing error accumulation. The method is theoretically grounded with a KL divergence decomposition showing how marginalizing over masking trajectories increases output diversity. The approach is elegant, simple to implement (no extra data or post-training), and directly addresses a known failure mode in AR video generation.
The experiments are comprehensive, evaluating the method on three different baselines (Self Forcing, LongLive, Causal Forcing) across both chunk-wise and frame-wise settings. Metrics include HPSv3, VisionReward, VBench, and human evaluation. The results show consistent improvements in visual quality, motion dynamics, and convergence speed. Ablation studies on mask ratio, timestep window, and mask scheme are thorough and provide clear insights into the trade-offs between visual quality and motion diversity. The inclusion of a long-video generation evaluation and an interactive world model application further strengthens the empirical case.
The paper provides detailed implementation details, including hyperparameters (mask ratio, timestep window), training steps, and hardware requirements. The algorithm is provided in pseudocode. The project page likely contains code and results. The method is described as a drop-in replacement for existing self-rollout pipelines, which enhances reproducibility.
The method relies on the assumption that perturbing the rollout helps explore the teacher distribution; if the teacher itself has limited diversity, the gains may be capped. The hyperparameters (mask ratio, window size) require tuning, as shown in the ablations. The evaluation is primarily on text-to-video; performance on other modalities or tasks is not explored. The theoretical analysis assumes exact score estimation, which is an approximation in practice.
This work has significant implications for the field of real-time video generation. By improving the quality and diversity of distilled AR models without requiring additional data or complex post-training, it lowers the barrier to deploying high-fidelity video generation models. The insights into mode collapse in DMD are broadly applicable to other distillation scenarios. The method could be integrated into existing pipelines for interactive video generation, world models, and streaming video applications. Proposes a dual-noise masking rollout strategy to mitigate mode collapse in autoregressive video diffusion distillation, significantly improving visual quality and motion dynamics without additional data or post-training. The paper presents a well-motivated, theoretically grounded, and empirically validated method that addresses a critical limitation in current AR video generation techniques, offering a practical and effective solution for the community.
Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs: the optimal learning rate and batch size vary with activation ratio, and these shifts cannot be explained by either total or activated parameter count alone. To characterize this dependence, we conduct 1,800 pre-training runs spanning six activated-parameter scales and models with up to 6B total non-embedding parameters, processing approximately 20 trillion tokens at a cost of 200,000 equivalent H800 GPU-hours. Our results reconcile conflicting findings in prior work by revealing two scaling regimes. At fixed sparsity, the optimal batch size follows a power-law relationship with training tokens $D$, whereas the optimal learning rate scales with training compute $C$ and remains robust to the allocation between model size and data. Across sparsity levels, the activation ratio $A$ enters both relationships as an additional multiplicative power-law factor. These observations lead to unified hyperparameter scaling laws that transfer across MoE sparsity levels. Large-scale evaluation shows that the scaling form outperforms alternative functional forms. On a held-out ultra-sparse MoE with 12B total parameters and only 1/64 of its experts activated, the predicted hyperparameters remain close to the observed optima, supporting joint extrapolation across model scale and sparsity. Further experiments demonstrate transfer across expert granularities and isolate the effect of activation ratio from that of total expert count.
Primary: Tsinghua University
All Institutions: Tsinghua University, Microsoft Research
The paper establishes unified hyperparameter scaling laws for Mixture-of-Experts models by identifying the activation ratio as a critical scaling factor, enabling reliable hyperparameter transfer across varying sparsity levels and model scales. Through an extensive empirical study involving 1,800 pre-training runs and 20 trillion tokens, the authors demonstrate that conventional scaling laws are insufficient for ultra-sparse MoEs and propose a new framework that reconciles prior conflicting findings, offering a practical tool for optimizing the training of large-scale sparse models.
The paper proposes a unified framework for hyperparameter scaling laws in Mixture-of-Experts (MoE) models, specifically addressing the gap left by conventional scaling laws which fail in ultra-sparse regimes. The core methodological contribution is the identification of the activation ratio $A$ as a critical multiplicative factor in the scaling relationships for optimal learning rate and batch size. The authors conduct an extensive empirical study involving 1,800 pre-training runs, which is a significant resource investment. The methodology involves fitting power-law relationships to this large dataset to derive predictive formulas that transfer across different sparsity levels and model scales. The approach is rigorous, moving beyond anecdotal evidence to establish a statistical basis for hyperparameter selection in MoEs.
The experimental scale is impressive, processing approximately 20 trillion tokens with up to 6B non-embedding parameters. The evaluation includes a held-out test on a 12B parameter ultra-sparse MoE (1/64 activation), demonstrating the predictive power of the derived laws. The paper reconciles conflicting findings in prior work by distinguishing between fixed sparsity and varying sparsity regimes. The results show that the proposed scaling form outperforms alternative functional forms, providing strong evidence for its utility. The isolation of the activation ratio's effect from total expert count is a key experimental strength.
The paper provides detailed descriptions of the experimental setup, including the number of runs, parameter scales, and token counts. However, without access to the specific code repository or detailed hyperparameter grids (which are not provided in the text snippet), full reproducibility is limited to the high-level methodology. The scale of the experiments (200,000 H800 GPU-hours) makes independent replication by most groups difficult, but the theoretical framework is reproducible if the data were available.
The primary limitation is the computational cost required to validate these laws, which restricts the ability for smaller labs to verify or extend the work. The paper focuses on pre-training; the applicability of these scaling laws to fine-tuning or other training phases is not fully explored. Additionally, the laws are derived from specific model architectures (likely Transformer-based MoEs), and their generalizability to other MoE variants or non-Transformer architectures is not tested.
This work has high practical impact for the ML community, as MoE models are becoming increasingly prevalent for efficient large-scale training. Providing reliable scaling laws for hyperparameters reduces the trial-and-error cost of training large MoE models, potentially saving significant computational resources. It helps standardize best practices for MoE training, contributing to the broader goal of making large model training more predictable and efficient. The paper establishes unified hyperparameter scaling laws for Mixture-of-Experts models by identifying the activation ratio as a critical scaling factor, enabling reliable hyperparameter transfer across varying sparsity levels and model scales. Through an extensive empirical study involving 1,800 pre-training runs and 20 trillion tokens, the authors demonstrate that conventional scaling laws are insufficient for ultra-sparse MoEs and propose a new framework that reconciles prior conflicting findings, offering a practical tool for optimizing the training of large-scale sparse models.
Visual Autoregressive Models (VAR) generate images through next-scale prediction, producing all tokens within each scale in parallel. We show that this parallel decoding constitutes a mean-field-style approximation that discards spatial dependencies among same-scale tokens, causing locally incoherent samples regardless of backbone capacity -- a limitation of the decoding rule. Addressing this limitation, we introduce the Logit Refiner, a lightweight autoregressive module that restores intra-scale dependencies by sequentially sampling tokens conditioned on frozen backbone features. Adding only ~10% parameters and less than 5% of the base model's training compute, it plugs into any pretrained VAR checkpoint without retraining. Controlled ablations isolate joint intra-scale sampling -- rather than additional capacity or training -- as the critical ingredient. Across backbones from 310M to 2B parameters on class-conditional ImageNet 256x256, the refiner consistently improves generation quality, enabling a 1.1B-parameter model to surpass one twice its size. The approach further generalizes to text-to-image generation, confirming that the mean-field bottleneck persists across VAR variants and is effectively alleviated by our method. Project page: https://compvis.github.io/logit-refiner/
Primary: LMU Munich
All Institutions: LMU Munich, Munich Center for Machine Learning (MCML)
The paper introduces the Logit Refiner, a lightweight module that restores intra-scale spatial dependencies in Visual Autoregressive models by sequentially refining logits, significantly improving generation quality without retraining the base backbone. This work provides a clear diagnosis of the mean-field approximation limitation in parallel VAR decoding and offers an efficient, plug-and-play solution that demonstrates strong empirical gains across model scales and tasks, representing a solid contribution to the field of autoregressive image generation.
The paper identifies a specific structural weakness in Visual Autoregressive (VAR) models: the parallel decoding of tokens within a scale acts as a mean-field approximation, ignoring spatial dependencies between same-scale tokens. The proposed solution, the "Logit Refiner," is a lightweight autoregressive module that sequentially refines logits conditioned on frozen backbone features. This approach is elegant in its simplicity, requiring only ~10% additional parameters and minimal training compute, and it plugs into existing checkpoints without retraining the base model. The theoretical framing of the problem as a mean-field approximation is insightful and provides a clear motivation for the sequential refinement step.
The experiments are rigorous and well-controlled. The authors test across multiple backbone sizes (310M to 2B) on ImageNet 256x256, demonstrating consistent improvements in FID and IS metrics. A key strength is the ablation study that isolates the effect of intra-scale dependency modeling from mere increases in capacity or training data, confirming that the sequential sampling mechanism is the critical factor. The generalization to text-to-image generation further validates the robustness of the method. The claim that a 1.1B model with the refiner surpasses a 2B model without it is a strong empirical result.
The paper provides sufficient details on the architecture of the Logit Refiner and the training procedure. The project page likely contains code, which is standard for this group. The method's plug-and-play nature enhances reproducibility, as it does not require complex retraining pipelines for the base model.
The primary limitation is the increased inference latency due to the sequential refinement step, which partially negates the speed advantage of VAR's parallel decoding. While the paper notes the compute cost is low during training, the inference cost could be significant for real-time applications. Additionally, the method is specific to VAR architectures and may not directly apply to other autoregressive or diffusion-based models without modification.
This work has significant implications for the design of autoregressive generative models. By highlighting the trade-off between parallel efficiency and spatial coherence, it guides future research in hybrid decoding strategies. The plug-and-play nature of the refiner makes it an attractive tool for practitioners using existing VAR checkpoints, potentially extending the utility of current models without the need for expensive retraining. The paper introduces the Logit Refiner, a lightweight module that restores intra-scale spatial dependencies in Visual Autoregressive models by sequentially refining logits, significantly improving generation quality without retraining the base backbone. This work provides a clear diagnosis of the mean-field approximation limitation in parallel VAR decoding and offers an efficient, plug-and-play solution that demonstrates strong empirical gains across model scales and tasks, representing a solid contribution to the field of autoregressive image generation.
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
AuK introduces a unified open-source foundational model for speech generation and editing. The paper demonstrates a comprehensive approach to unifying diverse speech tasks through a shared instruction-based interface, leveraging a hybrid flow-matching architecture and advanced post-training strategies to achieve state-of-the-art performance and efficient inference.
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid rectified-flow Transformer combining dual-stream MMDiT blocks for semantic-acoustic interaction and single-stream DiT blocks for refinement. The methodology is technically sound, leveraging a frozen MLLM (Qwen2.5-Omni) for semantic conditioning and a jointly trained VAE for acoustic conditioning. The training pipeline is sophisticated, involving a two-stage pre-training curriculum (generation warm-up followed by joint generation-editing), followed by post-training via human-feedback preference optimization (DPO variant) for editing and Flow-GRPO for generation. The distillation strategy using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference is a significant engineering contribution. The novelty lies primarily in the unification of diverse tasks (TTS, editing, enhancement, separation) under a single instruction-based interface and the specific combination of post-training strategies for speech.
The paper claims leading performance on zero-shot and instruction-controlled speech generation and editing benchmarks. However, the provided text is truncated before the detailed experimental results section, making it difficult to fully verify the magnitude of improvements or the specific benchmarks used (e.g., LibriSpeech, Seed-TTS, etc.). The data construction is extensive (3.03 billion instances, 1.95 million hours), which is a major strength. The evaluation of the distilled model (AuK-Flash) showing a 4.5x speedup is a strong practical result. Without the full results tables, the technical impact score is slightly conservative, but the scale of data and the comprehensive task coverage suggest high impact.
The paper states that both source code and model weights are released. The training details are provided in high granularity (hyperparameters, data ratios, loss weights, optimizer settings). The use of open-source components (Qwen2.5-Omni, F5-TTS for data synthesis, etc.) aids reproducibility. The specific data construction pipelines are described in detail, though the exact proprietary datasets used for some parts (e.g., in-house dialect corpus) may limit full reproduction of the data mix.
The model is large (1.5B parameters for the backbone, plus the MLLM and VAE), which may be resource-intensive for some users, though the distilled version mitigates this. The reliance on synthetic data for editing tasks (using other TTS models to generate targets) may introduce artifacts or bias. The evaluation of "open-ended" editing tasks is inherently subjective, and while human feedback is used for training, the final evaluation metrics for these tasks may not fully capture user satisfaction. The paper does not provide a detailed ablation study on the specific contribution of each post-training stage in the visible text.
This work has high potential impact by providing a unified, open-source foundation for speech manipulation. It bridges the gap between discrete task-specific models and a general-purpose speech AI. The release of code and weights will likely accelerate research in speech editing and generation. The efficient inference method (4-step) makes it viable for real-time applications. AuK introduces a unified open-source foundational model for speech generation and editing. The paper demonstrates a comprehensive approach to unifying diverse speech tasks through a shared instruction-based interface, leveraging a hybrid flow-matching architecture and advanced post-training strategies to achieve state-of-the-art performance and efficient inference.
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Primary: Alibaba Group
All Institutions: Alibaba Group
Gander introduces a Cerebellum-Brain architecture to enable low-latency, full-duplex omni-modal interaction by separating real-time streaming processing from complex agentic reasoning. The paper presents a significant engineering and architectural contribution to the field of multimodal agents, offering a practical path toward natural, interruptible, and proactive AI interactions, though the depth of empirical validation is constrained by the provided text's lack of detailed quantitative benchmarks.
The paper proposes "Gander," a unified framework for omni-perception and agentic capabilities. The core architectural contribution is the "Cerebellum-Brain" collaborative framework. The "Cerebellum" handles low-latency, real-time streaming interaction (speech/video/text) using a "streaming Thinker-Talker" architecture that flattens inputs into ordered token streams at the chunk level. The "Brain" handles complex reasoning and tool calling. This separation of concerns is a pragmatic engineering solution to the latency-reasoning trade-off in multimodal agents. The novelty lies in the specific implementation of full-duplex, interruptible interaction within a single model framework, rather than a pipeline of separate ASR, LLM, and TTS modules.
The evaluation covers conversational ability, omni understanding, interactive capability, and agentic intelligence. The paper claims competitive performance with SOTA open-source models in spoken dialogue and robustness in noisy/multi-party scenarios. However, the provided text is a skeleton (LaTeX structure) without the actual results tables, detailed ablation studies, or specific benchmark scores. The reliance on "internal human evaluations" without public benchmark comparisons limits the objective verifiability of the claims.
The authors state they release models, code, and data. The presence of a GitHub repository and project page supports reproducibility. However, the scale of training data and specific hyperparameters for the "Cerebellum-Brain" interaction loop are not detailed in the provided abstract/skeleton, which may hinder exact reproduction of the training dynamics.
The primary limitation is the lack of detailed quantitative results in the provided text. The "Cerebellum-Brain" split may introduce latency or state-sync issues if not perfectly optimized. The evaluation relies heavily on internal human studies, which are subjective and hard to replicate externally. The paper does not clearly define the computational overhead of maintaining the continuous streaming state compared to turn-based models.
This work is significant for the development of real-time multimodal agents. By unifying perception and action in a full-duplex manner, it moves the field closer to natural human-like interaction. The open-sourcing of the model and code will likely accelerate research in streaming multimodal LLMs. Gander introduces a Cerebellum-Brain architecture to enable low-latency, full-duplex omni-modal interaction by separating real-time streaming processing from complex agentic reasoning. The paper presents a significant engineering and architectural contribution to the field of multimodal agents, offering a practical path toward natural, interruptible, and proactive AI interactions, though the depth of empirical validation is constrained by the provided text's lack of detailed quantitative benchmarks.
Humanoid locomotion on granular terrain remains a significant challenge due to its complex foot-terrain interaction dynamics that are difficult to model. Existing approaches either ignore granular contact dynamics or incorporate simplified normal force models with heuristic tangential components. In this work, we present a physics-grounded granular contact model based on three-dimensional resistive force theory (3D RFT) and efficiently simulate granular terrain for reinforcement learning (RL) training. Unlike traditional rigid contact models and simplified granular contact models with ad-hoc heuristics, our contact solver produces physically accurate granular intrusion dynamics without resorting to heuristics. It captures realistic penetration and tangential drag during training, enabling the policy to learn behaviors that transfer reliably to real-world granular terrain where rigid contact models fail. To adapt to varying terrain conditions, we train a terrain-adaptive locomotion controller via teacher-student RL, using a variational autoencoder to encode terrain information into a compact latent representation. Simulation studies using material point method (MPM) with NVIDIA Newton demonstrate that our method generalizes to unseen granular terrains, achieves a significantly higher success rate than baselines, and demonstrates zero-shot terrain identification and adaptation. We further validate our approach through extensive hardware experiments across diverse real-world granular terrains including basalt, dry sand, and beach sand. To the best of our knowledge, this is the first demonstration of agile humanoid locomotion on real-world granular terrain. Project page: https://humanoid-gm-locomotion.github.io/HUMANOID-GM/
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, Northeastern University
The paper introduces a physics-grounded granular contact model using 3D RFT and a teacher-student RL framework to enable agile humanoid locomotion on real-world granular terrain. This is a significant contribution to robotics and reinforcement learning, as it addresses the critical sim-to-real gap in complex, deformable terrain interactions, demonstrating superior performance and generalization compared to heuristic-based approaches.
The paper proposes a physics-grounded granular contact model based on 3D Resistive Force Theory (RFT) to simulate granular terrain for reinforcement learning (RL). This is a significant methodological improvement over heuristic or rigid-body approximations, as it captures realistic penetration and tangential drag. The authors employ a teacher-student RL framework where a Variational Autoencoder (VAE) encodes terrain properties into a compact latent representation, allowing the student policy to adapt to varying terrain conditions without explicit terrain labels. The use of the Material Point Method (MPM) with NVIDIA Newton for high-fidelity simulation is a strong technical choice that bridges the gap between physical accuracy and computational efficiency for RL training.
The evaluation is robust, featuring both simulation studies and extensive hardware experiments. The simulation results demonstrate generalization to unseen granular terrains and zero-shot terrain identification. Crucially, the paper validates its approach on real-world hardware across diverse terrains (basalt, dry sand, beach sand), claiming to be the first demonstration of agile humanoid locomotion on real-world granular terrain. The comparison against baselines shows significantly higher success rates, providing strong empirical evidence for the efficacy of the physics-grounded contact model.
The paper references specific tools (NVIDIA Newton, MPM) and provides a project page. However, the full text provided is a draft with placeholder sections (e.g., "First_draft/Method"), which limits the ability to verify specific hyperparameters or implementation details. The reliance on proprietary simulation tools (NVIDIA Newton) may pose some barriers to reproducibility for labs without access to such resources, though the physics-based approach (RFT) is standard.
The primary limitation is the reliance on a specific, potentially proprietary simulation stack (NVIDIA Newton). Additionally, the "first demonstration" claim, while impressive, is limited to a specific set of granular terrains; the generalization to other granular materials (e.g., snow, mud) is not explicitly tested in the provided text. The computational cost of MPM simulations, even with optimizations, remains high compared to rigid-body dynamics, which may limit real-time deployment on less powerful hardware.
This work has high potential impact in the field of legged robotics and humanoid control. By providing a physically accurate method for simulating granular terrain, it enables the development of more robust locomotion policies that can operate in unstructured environments. The teacher-student approach with latent terrain encoding offers a scalable path for adapting to diverse terrains, which is a critical challenge for deploying humanoid robots in the real world. The paper introduces a physics-grounded granular contact model using 3D RFT and a teacher-student RL framework to enable agile humanoid locomotion on real-world granular terrain. This is a significant contribution to robotics and reinforcement learning, as it addresses the critical sim-to-real gap in complex, deformable terrain interactions, demonstrating superior performance and generalization compared to heuristic-based approaches.
A hallmark of human dexterity is the cooperative use of fingers, where different fingers take on distinct yet coordinated roles to accomplish fine manipu- lation, such as capping a pen with the hand that holds it. We study this finger-level coordination through in-hand assembly: mating two rigid objects within a single dexterous hand, with no second arm and no fixture. We present a reinforcement learning formulation to solve this problem in a unified framework, which is driven by a goal relative pose between the two parts. Finger coordination is shaped by a function-based auxiliary reward and regularized toward a single human reference pose, while domain randomization and a fusion of historical proprioception and object observation confer robustness to occlusion-induced estimation noise. The same recipe solves three different assembly tasks (Bottle, Syringe, and Marker). Trained purely in simulation, the policies transfer zero-shot to hardware with a single camera, demonstrating robustness to state-estimation errors caused by oc- clusion. Our experiments also reveal that in-hand assembly places demands on hand morphology and can serve as a benchmark for modern robotic hand systems. Videos and code are available at https://ltbgbird.github.io/in-hand-assembly-page/.
Primary: The University of Hong Kong
All Institutions: The University of Hong Kong, The Hong Kong University of Science and Technology (Guangzhou), The Hong Kong University of Science and Technology, ETH Zurich
The paper introduces a robust reinforcement learning framework for fixture-free in-hand assembly, demonstrating that a single dexterous hand can coordinate multiple fingers to mate two objects with zero-shot sim-to-real transfer. By combining function-based reward shaping with human pose priors and robust state estimation, the work establishes a new benchmark for dexterous manipulation and provides critical insights into the morphological requirements for advanced robotic hands.
The paper proposes a reinforcement learning framework for "in-hand assembly," a task where a single dexterous hand manipulates two objects to mate them without external fixtures. The core methodological contribution is a unified RL formulation driven by relative pose goals, augmented with a function-based auxiliary reward to shape finger coordination (distinguishing roles like pinch vs. grasp) and a regularization term toward human reference poses. The approach relies on sim-to-real transfer using domain randomization and a fusion of proprioception and visual object state estimation. While the use of RL for dexterous manipulation is established, the specific formulation for multi-object coordination within a single hand, particularly the reward shaping for finger specialization, is a meaningful technical advance. The handling of occlusion-induced noise via historical proprioception fusion is also a solid engineering solution to a critical real-world problem.
The experiments are rigorous and well-designed. The authors evaluate three distinct tasks (Bottle, Syringe, Marker) to demonstrate generality. A key strength is the "Hand Morphology Benchmark" section, which tests the pipeline on four different robotic hands (Sharpa, Wuji, Allegro, XHand), providing valuable insights into how hardware constraints affect dexterity. The ablation studies clearly isolate the contributions of vision, recurrent architecture (LSTM vs. MLP), and specific reward components. Real-world validation is strong, with zero-shot transfer from simulation to hardware using only a single camera. The comparison between closed-loop control and open-loop replay convincingly demonstrates the necessity of feedback. The robustness tests against perturbation and wrist tilt further solidify the practical utility of the learned policies.
The paper provides high reproducibility. It details the simulation setup (IsaacSim, PhysX parameters), policy architecture (LSTM/MLP dimensions), RL hyperparameters (PPO settings), and domain randomization ranges. The code and videos are available on the project website. The description of the real-world deployment, including camera calibration and pose tracking stabilization, is detailed enough for replication. The use of standard hardware (Sharpa Wave, Franka, RealSense) and open-source simulation tools lowers the barrier to entry for other researchers.
The primary limitation is the manual initialization of objects; the system requires a human to place the objects in the hand before control is activated, meaning it does not solve the full pipeline of grasping from a table. The paper also notes that current rigid-body simulators struggle with planar contacts, leading to sim-to-real gaps in pinch stability at certain wrist tilts. The scope is limited to three specific assembly tasks, and the generalization to arbitrary part geometries (e.g., threading, friction-fit) is left for future work.
This work significantly advances the field of dexterous manipulation by demonstrating that a single hand can perform complex, coordinated multi-object tasks. It provides a valuable benchmark for evaluating robotic hand morphology and control algorithms. The insights into finger specialization and the necessity of closed-loop feedback under occlusion will influence the design of future dexterous systems. The zero-shot sim-to-real transfer capability is particularly impactful for reducing the cost and time of real-world deployment. The paper introduces a robust reinforcement learning framework for fixture-free in-hand assembly, demonstrating that a single dexterous hand can coordinate multiple fingers to mate two objects with zero-shot sim-to-real transfer. By combining function-based reward shaping with human pose priors and robust state estimation, the work establishes a new benchmark for dexterous manipulation and provides critical insights into the morphological requirements for advanced robotic hands.
Robotic manipulation tasks are often decomposed into behaviors or skills. However, one often needs to predefine these behaviors for specific tasks or try to cover a wide range of tasks using generic skills. As a result, these behaviors can remain too coarse to expose the geometric, control, and scene-dependent decisions required for execution. We introduce Grounded Task Axes v2 (GTA-2), a modular multi-VLM framework that constructs executable, task-bespoke manipulation skills from reusable object-centric task-axis components. Rather than predicting actions end-to-end or composing fixed task-level primitives, GTA-2 represents each skill as semantic subtasks comprising task-relevant keypoints and axes, controller compositions, and scene-dependent parameters. Four specialized VLM agents separately decompose the task, construct an abstract task-axis skill, assign controller parameters, and ground the required visual features from RGB-D observations. This abstraction-to-grounding factorization enables zero-shot skill generation without task-specific robot demonstrations, policy training, or fine-tuning. It also keeps intermediate decisions explicit, allowing targeted human feedback to refine an incorrect stage while preserving correct components. We evaluate GTA-2 on 14 real-robot manipulation tasks against a VLA policy pi_{0.5} and two Code-as-Policies baselines using task-axis controllers or conventional robot primitives. GTA-2 achieves an average zero-shot success rate of 73.9%, exceeding the strongest baseline by 31.4 percentage points, while targeted refinement raises GTA-2's average success rate to 90.7%. Project page: https://gta2-project.github.io/
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, Bosch Research
GTA-2 introduces a modular multi-VLM framework that synthesizes executable robot manipulation skills via grounded task axes, achieving high zero-shot success rates on real-robot tasks by factorizing skill generation into explicit, refinable stages. The paper presents a compelling alternative to end-to-end VLA policies by offering a transparent, component-based approach to skill synthesis, with strong empirical results on a real-robot benchmark, though the limited task diversity and reliance on human refinement for optimal performance are notable constraints.
The paper proposes GTA-2, a modular multi-VLM framework that decomposes robotic manipulation into "task axes" (keypoints, axes, controller compositions). The core innovation is the factorization of skill generation into four specialized VLM agents: task decomposition, abstract skill construction, parameter assignment, and visual grounding. This approach avoids end-to-end action prediction, instead generating explicit, inspectable intermediate representations that map to low-level controllers. The methodology is sound in its modular design, allowing for targeted human-in-the-loop refinement of specific stages (e.g., correcting a wrong keypoint without re-running the entire pipeline). The use of "task axes" as a semantic bridge between high-level language and low-level control is a creative contribution to the field of robot skill synthesis.
The evaluation is conducted on 14 real-robot manipulation tasks, which is a significant practical benchmark. The comparison against a VLA policy (pi_0.5) and Code-as-Policies baselines is relevant. The reported zero-shot success rate of 73.9% (outperforming the strongest baseline by 31.4 points) and the post-refinement rate of 90.7% are strong empirical results. However, the sample size of 14 tasks is relatively small for a general claim of "zero-shot skill generation," and the specific nature of the tasks (likely constrained to manipulation) limits the generalizability of the results to broader robotic domains.
The paper provides a project page, which likely contains code or further details. The modular nature of the VLM agents suggests that the pipeline is reproducible if the specific VLM models and controller libraries are available. However, the "task-axis" representation is novel, and without extensive documentation on how these axes are defined and mapped to controllers, full reproduction by external groups may be challenging. The reliance on specific VLM capabilities for grounding also introduces variability depending on the underlying model versions.
The primary limitation is the scope of the evaluation (14 tasks). While the success rates are high, it is unclear how the system performs on highly dynamic or unstructured environments beyond the tested set. The "zero-shot" claim relies on the VLMs' pre-trained knowledge, which may fail on novel object categories or task types not seen during VLM pre-training. Additionally, the human-in-the-loop refinement, while effective, introduces a dependency on human expertise for error correction, which may not scale to fully autonomous deployment.
This work contributes to the trend of using VLMs for high-level planning in robotics. By providing an explicit, editable intermediate representation (task axes), it offers a path toward more transparent and debuggable robotic systems compared to black-box VLA policies. This could be impactful for industrial applications where explainability and safety are critical. The collaboration between CMU and Bosch Research suggests potential for real-world industrial adoption. GTA-2 introduces a modular multi-VLM framework that synthesizes executable robot manipulation skills via grounded task axes, achieving high zero-shot success rates on real-robot tasks by factorizing skill generation into explicit, refinable stages. The paper presents a compelling alternative to end-to-end VLA policies by offering a transparent, component-based approach to skill synthesis, with strong empirical results on a real-robot benchmark, though the limited task diversity and reliance on human refinement for optimal performance are notable constraints.
Determinism and numerical reproducibility are increasingly required of GPU kernels in machine learning systems, yet deterministic implementations of the same kernel can still differ bit for bit. Floating-point reduction order is the primary cause, alongside partial-sum precision, fused multiply-add operations, and rounding placement. These choices may be hand-coded, selected by a block-level language such as Triton, or hidden inside a closed-source library such as cuBLAS or rocBLAS. A tile shape chosen for performance therefore also determines the arithmetic, potentially breaking batch invariance. Preserving a fixed order can cost up to 20 percent, while an autotuner cannot identify which configurations are bitwise equivalent. We characterize the factors determining the bitwise behavior of reductions and general matrix multiplication (GEMM). First, we introduce a descriptor of GEMM reduction order, including the partitioning of K in split-K GEMM. Using it, we perform the first black-box reconstruction of a closed-source library's arithmetic for bit-level correctness. Our family of Triton GEMMs matches NVIDIA cuBLAS in all tested cases on Blackwell and Hopper. For realistic LLM shapes with fused epilogues, it matches or exceeds torch.compile performance. Second, we enforce balanced-tree reduction during Triton lowering and introduce a data-layout optimization that brings 19 of 27 kernels on GB300 and H100 within 10 percent of free-order performance. Third, we develop sound static checkers for bitwise equivalence between compiled GPU kernels, including the first checker spanning NVIDIA PTX and AMD GCN. Integrated into Triton's autotuner, the checker restricts search to a single bit-equivalence class.
Primary: Georgia Institute of Technology
All Institutions: Georgia Institute of Technology, Meta
The paper presents a comprehensive framework for achieving bitwise reproducibility in GPU kernels through black-box reconstruction, compiler enforcement, and static verification. It makes significant contributions to the field of ML systems by providing practical tools and theoretical insights to ensure numerical determinism, which is increasingly critical for reliable machine learning deployments.
The paper introduces a rigorous theoretical framework for characterizing bitwise behavior in GPU kernels, specifically focusing on GEMM and reductions. The core contribution is the "GEMMDesc" descriptor, which formally captures the parameters determining floating-point reduction order (e.g., split-K cuts, instruction-level accumulation semantics). The methodology for black-box reconstruction of closed-source libraries (cuBLAS) is particularly innovative, using numerical inference with specific extreme values to reverse-engineer the internal arithmetic structure without access to source code. The implementation of a sound static equivalence checker for PTX and AMDGCN assembly is a significant technical achievement, providing a formal guarantee that two compiled kernels are bitwise equivalent, which is rare in compiler research.
The evaluation is extensive and rigorous. The authors demonstrate 100% bitwise match with cuBLAS across 110,813 random shapes on multiple GPU architectures (GB300, GB200, H100). The performance analysis is nuanced, showing that while enforcing bitwise determinism typically costs performance, their layout optimizations recover most of this cost, and in some fused kernel scenarios, the bit-exact implementation actually outperforms the baseline due to reduced launch overhead. The soundness of the static checker is validated on over 51,000 configurations, showing high precision in partitioning equivalent kernels.
The paper provides detailed descriptions of the compiler modifications and the static checker algorithm. However, as it is an arXiv preprint, the code availability is not explicitly confirmed in the text provided. The reliance on specific hardware (GB300, H100) and closed-source libraries (cuBLAS) may limit immediate reproducibility for all readers, but the methodology is clearly described.
The work is heavily focused on GEMM and standard reductions; complex kernels like Flash Attention are only theoretically analyzed. The black-box reconstruction is specific to cuBLAS versions tested and may not generalize to other libraries or future hardware without re-derivation. The static checker, while sound, has a performance overhead and may not scale to arbitrarily complex control flows without further optimization.
This work has high potential impact on the ML systems community, particularly for teams requiring strict reproducibility in training and inference. It provides tools to enforce determinism without significant performance loss, addressing a critical pain point in distributed training and RLHF pipelines. The static equivalence checker could become a standard tool in compiler verification suites. The paper presents a comprehensive framework for achieving bitwise reproducibility in GPU kernels through black-box reconstruction, compiler enforcement, and static verification. It makes significant contributions to the field of ML systems by providing practical tools and theoretical insights to ensure numerical determinism, which is increasingly critical for reliable machine learning deployments.