AudioLM: a Language Modeling Approach to Audio Generation
Abstract
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioLM introduces a hierarchical language modeling framework that combines semantic and acoustic discrete tokens to achieve unprecedented long-term coherence and high-fidelity audio generation, fundamentally reshaping the paradigm for neural audio synthesis and spawning a generation of subsequent audio foundation models. The paper's rigorous empirical validation, clear ablation of tokenization trade-offs, and demonstration of cross-domain applicability (speech and music) establish it as a highly influential contribution, though its autoregressive bottleneck and lack of open-source release temper its immediate reproducibility and practical deployment speed.
Methodology
The paper introduces a principled solution to the long-standing trade-off between acoustic fidelity and long-term structural coherence in neural audio generation. By decoupling semantic structure (extracted via k-means quantization of intermediate w2v-BERT representations) from acoustic details (captured by SoundStream's residual vector quantization), the authors formulate audio generation as a hierarchical, multi-stage language modeling task. The three-stage autoregressive Transformer architecture (semantic → coarse acoustic → fine acoustic) is theoretically well-motivated, leveraging conditional independence assumptions to reduce sequence length and computational burden. The hybrid tokenization scheme elegantly bridges self-supervised representation learning and neural audio compression, establishing a new paradigm that cleanly separates content from style/acoustics. The methodology is conceptually clean, scalable, and highly generalizable across audio domains.
Results & evaluation
The experimental design is comprehensive and rigorously executed. The authors validate the framework on both speech (60k hours of Libri-Light) and piano music, demonstrating strong generalization to unseen speakers and performances. Quantitative evaluations span phonetic discriminability (ABX), reconstruction quality (ViSQOL), linguistic probing (sWUGGY, sBLIMP), and speaker identity preservation (>92% accuracy). The subjective evaluation is particularly strong, showing that human listeners correctly identify synthetic continuations only ~51.2% of the time, indicating near-human indistinguishability. The inclusion of a high-accuracy synthetic speech detector demonstrates responsible evaluation practices. Results clearly isolate the contributions of semantic vs. acoustic tokens and establish strong baselines against prior textless NLP and hierarchical generation approaches.
Reproducibility
The paper provides detailed architectural specifications (12-layer decoder-only Transformers, 0.3B parameters per stage, T5-style relative positional embeddings, temperature sampling schedules, and training compute on 16 TPUv4s). Token extraction pipelines, k-means clustering procedures, and SoundStream configurations are thoroughly documented. However, reproducibility is partially constrained by the lack of open-source code, reliance on proprietary internal datasets for piano training, and the substantial compute required to pre-train w2v-BERT and SoundStream from scratch. While academic labs with sufficient resources can replicate the core methodology, full reproduction of the exact results remains challenging without released weights or training scripts.
Limitations
The autoregressive nature of the framework inherently limits inference speed and scalability for real-time or long-form generation. The hierarchical tokenization pipeline introduces latency and complexity, requiring three separate forward passes and careful temperature tuning. The model's performance on highly polyphonic music, multi-speaker dialogues, or complex environmental soundscapes is not evaluated. Additionally, the system inherits biases and failure modes from its pre-trained components (e.g., struggles with proper nouns, sensitivity to background noise, and potential degradation on underrepresented dialects). The 3-second prompt requirement, while short, still restricts true zero-shot generation capabilities.
Broader impact
AudioLM establishes a foundational architectural paradigm that has directly catalyzed the modern wave of audio foundation models, including VALL-E, MusicGen, AudioGen, and subsequent voice cloning systems. Its ability to generate syntactically and acoustically coherent speech without textual supervision democratizes high-quality audio synthesis for assistive technologies, creative composition, and scalable data augmentation. However, the near-indistinguishable quality of generated continuations raises serious ethical concerns regarding deepfake audio, biometric spoofing, and misinformation campaigns. The authors' proactive development of a high-accuracy detection classifier is a commendable step, but the work underscores the urgent need for industry-wide watermarking standards, provenance tracking, and regulatory frameworks for generative audio. AudioLM introduces a hierarchical language modeling framework that combines semantic and acoustic discrete tokens to achieve unprecedented long-term coherence and high-fidelity audio generation, fundamentally reshaping the paradigm for neural audio synthesis and spawning a generation of subsequent audio foundation models. The paper's rigorous empirical validation, clear ablation of tokenization trade-offs, and demonstration of cross-domain applicability (speech and music) establish it as a highly influential contribution, though its autoregressive bottleneck and lack of open-source release temper its immediate reproducibility and practical deployment speed.
Curator note
Borsos et al.; Google; language model for audio tokens
Author of this paper? Send a correction.