High-Resolution Image Synthesis with Latent Diffusion Models
Abstract
By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs. Code is available at https://github.com/CompVis/latent-diffusion .
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Latent Diffusion Models, a two-stage architecture that decouples perceptual compression from semantic generation, drastically reducing computational costs while enabling high-fidelity, multi-conditioned image synthesis, thereby democratizing generative AI and establishing the foundational framework for modern open-weight diffusion systems.
Methodology
The paper introduces a principled two-stage generative framework that decouples perceptual compression from semantic distribution learning. Stage one trains a continuous autoencoder with perceptual and adversarial losses to map high-dimensional pixel space to a compact, perceptually equivalent latent space. Stage two trains a denoising diffusion model within this latent space, leveraging the UNet's spatial inductive biases to avoid the aggressive 1D discretization required by autoregressive latent models. The architectural innovation of injecting cross-attention layers into the UNet's intermediate feature maps enables flexible, modality-agnostic conditioning (text, bounding boxes, semantic layouts) without retraining the backbone. The mathematical formulation correctly adapts the reweighted variational bound to the latent domain, preserving the theoretical guarantees of score-based generative modeling while drastically reducing the dimensionality of the denoising trajectory.
Results & evaluation
The experimental design is comprehensive and rigorously structured. The authors systematically ablate the compression factor ($f \in \{1,2,4,8,16,32\}$), demonstrating that $f=4$ and $f=8$ optimally balance computational efficiency and detail preservation. Evaluations span unconditional generation, class-conditional ImageNet synthesis, text-to-image, layout-to-image, super-resolution, and inpainting across diverse datasets (CelebA-HQ, FFHQ, LSUN, ImageNet, LAION-400M, COCO, Places). The paper reports standard metrics (FID, IS, Precision/Recall) alongside human preference studies, showing consistent SOTA or highly competitive results while reducing training compute by 1-2 orders of magnitude compared to pixel-space diffusion baselines. The efficiency-vs-quality trade-off curves are particularly compelling and directly validate the core hypothesis.
Reproducibility
Excellent. The paper provides exhaustive implementation details, including architecture specifications, hyperparameter tables, training schedules, and evaluation protocols. The two-stage training pipeline is explicitly separated, making it straightforward to replicate the autoencoder and diffusion stages independently. Pretrained weights and a fully documented codebase are publicly released, and the authors clarify dataset preprocessing, sampling strategies (DDIM), and guidance mechanisms (classifier-free guidance). The appendix further details SNR calibration for convolutional high-res sampling and degradation pipelines for robust super-resolution.
Limitations
The authors correctly identify that sequential sampling remains slower than single-pass GANs, limiting real-time applications. The perceptual compression stage introduces an irreducible reconstruction bottleneck; tasks requiring pixel-perfect fidelity (e.g., precise medical imaging or OCR-critical generation) may suffer from latent-space quantization or smoothing artifacts. Additionally, convolutional sliding-window synthesis at megapixel resolutions can exhibit boundary inconsistencies or SNR mismatches if latent variance is not carefully rescaled. The cross-attention conditioning, while flexible, scales quadratically with sequence length, potentially bottlenecking extremely long text or dense layout inputs.
Broader impact
This work fundamentally democratizes high-resolution generative modeling by collapsing the compute barrier that previously restricted diffusion research to well-funded labs. By releasing open weights and code, it catalyzed an ecosystem of accessible, community-driven generative AI, directly enabling the development of Stable Diffusion and subsequent open-weight models. The authors responsibly address dual-use risks, including deepfake proliferation, training data memorization, and bias amplification, while highlighting the positive creative and scientific applications. The methodological shift toward latent-space diffusion has become the de facto standard for scalable generative vision, influencing subsequent work in video, 3D, and audio synthesis. The paper introduces Latent Diffusion Models, a two-stage architecture that decouples perceptual compression from semantic generation, drastically reducing computational costs while enabling high-fidelity, multi-conditioned image synthesis, thereby democratizing generative AI and establishing the foundational framework for modern open-weight diffusion systems.
Curator note
Rombach et al.; enabled open-source text-to-image at scale
Discussion
Talks & videos
Related later work
- SDXL 2023
- ControlNet 2023
- DreamBooth (personalization) 2022
- InstructPix2Pix 2022
- Sora (video diffusion) 2024
- Flux image generation 2024
Author of this paper? Send a correction.