Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models
Abstract
Unified multimodal diffusion large language models (dLLMs) offer a single architecture for both image generation and multimodal understanding, but their iterative decoding requires tens to hundreds of forward passes. Existing few-step distillation methods largely focus on either image generation or text generation, making it unclear how to compress a fully discrete multimodal dLLM into a single efficient student while preserving both generation and understanding. We introduce Omni-Diffusion-Distill, a unified two-stage distillation framework that retains strong generation and understanding capabilities while substantially reducing the inference cost of a unified multimodal dLLM. Omni-Diffusion-Distill aligns the distillation of both generation and understanding, for both images and text, in the discrete token space. In the first stage, the student is trained to skip decoding steps by replaying cached teacher trajectories, and in the second stage the student is refined on intermediate states along its own rollouts. We further remedy two sources of degradation in unified distillation with a pairwise collision penalty that reduces repetition under parallel text decoding, and entropy-matched guidance that prevents entropy collapse caused by fitting the sharpened teacher distribution in image generation. Omni-Diffusion-Distill achieves state-of-the-art trade-offs between decoding efficiency and generation and understanding performance for multimodal dLLMs, reducing image generation from 128 to 8 decoding steps and multimodal understanding from 512 to 64, giving 18.2x and 21.2x wall-clock speedups. Under these budgets, it scores 0.828 on GenEval and 83.0 on DPG-Bench for text-to-image generation, while reaching GPT judge scores of 20.0 on MM-Vet and 57.2 on COCO captioning (twice the teacher's 28.4 at the same steps) for multimodal understanding.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a unified two-stage distillation framework for discrete multimodal dLLMs that effectively compresses inference steps for both image generation and text understanding, achieving state-of-the-art efficiency-performance trade-offs through targeted remedies for entropy collapse and parallel decoding repetition.
Methodology
The paper proposes Omni-Diffusion-Distill, a two-stage distillation framework specifically designed for unified multimodal diffusion large language models (dLLMs) that operate in a fully discrete token space. The methodology is well-structured, addressing the specific challenges of compressing iterative decoding for both image generation and text understanding within a single architecture. Stage 1 employs off-policy trajectory replay to teach the student to skip decoding steps, while Stage 2 utilizes on-policy refinement on the student's own rollouts to mitigate exposure bias. The core technical contributions are two targeted remedies for modality-specific degradation: an entropy-matched guidance mechanism to prevent entropy collapse in image generation (caused by fitting sharpened CFG targets), and a pairwise collision penalty to reduce repetition in parallel text decoding. The adaptation of distribution matching distillation (DMD) to discrete tokens via an auxiliary network and direct gradient specification is a sound technical adaptation of continuous diffusion techniques to the discrete domain.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors distill the Lumina-DiMOO model and compare against strong baselines (Di[M]O, CDLM, T3D) across 17 benchmarks covering Text-to-Image (T2I), Image-to-Image (I2I), and Multimodal Understanding (MMU). The results demonstrate significant efficiency gains (18.2x and 21.2x wall-clock speedups) with minimal performance degradation, often outperforming the few-step teacher. The inclusion of ablation studies for the collision penalty and entropy-matched guidance, along with detailed analysis of degeneration metrics (repetition, entropy), provides strong evidence for the effectiveness of the proposed components. The use of GPT-5.5 for open-ended evaluation is a modern and appropriate choice, though it introduces some dependency on external models.
Reproducibility
The paper provides high reproducibility. It details the training data mixture, optimization settings, and algorithms for both stages. The specific hyperparameters for the loss terms and the bisection method for entropy matching are described. While the code is not explicitly linked in the provided text, the level of detail in the appendices (algorithms, loss derivations, data sources) suggests that reproduction is feasible for a skilled practitioner. The use of standard benchmarks and clear evaluation protocols further supports reproducibility.
Limitations
The primary limitation is the scope of validation; the method is only tested on one specific unified dLLM architecture (Lumina-DiMOO). It is unclear how well the framework generalizes to other unified models with different attention mechanisms or tokenization schemes. Additionally, the student model still trails the full-step teacher on certain fine-grained tasks like text rendering and detailed captioning. The reliance on a large teacher model and the computational cost of collecting teacher trajectories, while one-time, remains a barrier for smaller labs.
Broader impact
This work has significant potential impact on the deployment of unified multimodal models. By drastically reducing inference costs while maintaining quality, it makes real-time multimodal interaction more feasible. The techniques developed for handling entropy collapse and parallel decoding artifacts are likely to be applicable to other discrete generative models beyond dLLMs. The framework bridges the gap between high-quality iterative samplers and efficient few-step inference, a critical area for practical AI applications. The paper introduces a unified two-stage distillation framework for discrete multimodal dLLMs that effectively compresses inference steps for both image generation and text understanding, achieving state-of-the-art efficiency-performance trade-offs through targeted remedies for entropy collapse and parallel decoding repetition.
Author of this paper? Send a correction.