LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining
Abstract
Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process them with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture for the audio encoder. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our resulting models, CLAP2, outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. We release our dataset, captions, model checkpoints, and code.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces LAION-Audio-XL, a 365M audio-text pair dataset, and CLAP2, a family of contrastive audio-language models, providing a rigorous scaling study that demonstrates the compute-optimal trade-off between model size and data volume in audio pretraining. The work is significant for its comprehensive data curation pipeline, detailed scaling law analysis, and the release of open resources that will enable further research in audio-language modeling.
Methodology
The paper presents a rigorous and comprehensive data engineering pipeline for constructing LAION-Audio-XL, a 365M pair corpus derived from 1.48B raw samples. The methodology is technically sound, employing multi-stage deduplication (exact MD5, acoustic Olaf fingerprints, semantic SemDeDup), coverage-aware selection based on a custom 4,416-node taxonomy, and high-quality captioning using state-of-the-art models (MOSS-Audio, Qwen3-Omni). The architectural choice of a NaFlex audio encoder with a ModernBERT-based text encoder, trained from scratch, is well-motivated for isolating scaling effects. The scaling law analysis is particularly strong, providing a clear compute-optimal allocation strategy that favors data over model size, which is a significant empirical finding for the audio domain.
Results & evaluation
The experimental evaluation is extensive, covering five model sizes (27M-340M) across 11 data scales. The use of MAEB-30, HEAR, and MSEB benchmarks provides a holistic view of performance. The results demonstrate that CLAP2 models outperform prior art of comparable size, with the largest model achieving a 53.6 MAEB-30 score. The ablation studies on audio length (30s vs 120s) and caption length are valuable, showing that shorter audio clips yield better performance with significantly less compute. The scaling law fits are robust, showing consistent gains over four orders of magnitude of compute.
Reproducibility
The paper is highly reproducible, with detailed descriptions of the data pipeline, hyperparameters, and training setup. The authors commit to releasing the dataset, captions, model checkpoints, and code. The deterministic selection process and clear documentation of the curation steps enhance transparency. The use of standard libraries (FAISS, timm, FFmpeg) and open-source models for captioning further supports reproducibility.
Limitations
The primary limitation is the reliance on web-scraped data, which may contain licensing ambiguities despite the stated use of Creative Commons and public datasets. The scaling law analysis is based on a specific architecture (NaFlex) and may not generalize to other audio encoders. The paper does not extensively discuss the computational cost of the data curation pipeline itself, which is substantial. Additionally, the performance on speech tasks is strong, but the model's ability to handle complex, multi-speaker, or noisy real-world audio scenarios is not deeply explored.
Broader impact
The release of LAION-Audio-XL and CLAP2 models will significantly impact the audio-language community by providing a large, high-quality, open dataset and competitive baseline models. The scaling law insights will guide future research in efficient audio model training. The dataset's diversity and the documented curation pipeline serve as a template for other multimodal data projects. The emphasis on open-source releases aligns with the broader trend of democratizing AI research. The paper introduces LAION-Audio-XL, a 365M audio-text pair dataset, and CLAP2, a family of contrastive audio-language models, providing a rigorous scaling study that demonstrates the compute-optimal trade-off between model size and data volume in audio pretraining. The work is significant for its comprehensive data curation pipeline, detailed scaling law analysis, and the release of open resources that will enable further research in audio-language modeling.
Author of this paper? Send a correction.