GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling
Abstract
Generative spoken language models (SLMs) offer a route to learning linguistic structure purely from speech, yet they lag far behind text-based models in both capability and data scaling efficiency. We identify two bottlenecks underlying this gap: the limited linguistic quality of existing speech units and the reliance on a single stream of discrete units to represent speech. Multiple codebooks are widely used to encode acoustic features but rarely to represent speech semantics. We present GSLM2, which learns and models complementary properties of speech at multiple levels of abstraction. Its encoder, SpidR 2.0, uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A multi-stream SLM then jointly models these units using a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. Across training data sizes and model scales, up to 6M hours and 6.5B parameters, GSLM2 consistently improves linguistic competence while scaling more efficiently with data. As a result, it substantially narrows the gap between spoken and text-based LMs; across read and conversational speech domains, we shrink the gap by about two orders of magnitude on sWuggy, sBLIMP, and tSC, while also outperforming competitive speech-text systems on all three tasks. We open-source the training code and model checkpoints at https://github.com/facebookresearch/gslm2.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GSLM2 introduces a multi-stream spoken language modeling framework that learns complementary discrete units at multiple levels of abstraction via differentiable self-supervised pretraining, significantly narrowing the data-efficiency gap between speech and text language models. The paper presents a rigorous and scalable approach to textless language modeling, demonstrating that leveraging multi-level representations from self-supervised encoders allows SLMs to achieve linguistic competence comparable to text LMs with far less data, offering a promising path for speech-native AI applications.
Methodology
The paper introduces GSLM2, a framework that addresses the data-efficiency gap between spoken language models (SLMs) and text-based LMs. The core methodological contribution is twofold: (1) SpidR 2.0, an encoder that uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels of abstraction (layers) rather than relying on post-hoc k-means on a single layer; and (2) a multi-stream SLM architecture that jointly models these complementary unit streams using a shared temporal backbone, stream-specific predictors, and an autoregressive generative head. The use of entropy regularization to stabilize the code distributions across layers is a significant technical improvement over prior work like SpidR. The approach effectively decouples the representation of fine-grained phonetic details from higher-level semantic regularities, allowing the model to leverage both for efficient language modeling.
Results & evaluation
The experiments are extensive and rigorous. The authors scale models up to 6.5B parameters and 6M hours of data, which is a substantial computational effort. They demonstrate that GSLM2 shrinks the gap to text-based LMs by about two orders of magnitude on key benchmarks (sWuggy, sBLIMP, tSC). The comparison against previous SLMs (including Moshi) shows clear superiority in parameter and data efficiency. The generative evaluation, while relying on ASR and LLM judges (which introduces noise), still shows consistent improvements in coherence and perplexity. The ablation studies on layer-wise representations provide strong evidence for the multi-level hypothesis.
Reproducibility
The paper is highly reproducible. The authors explicitly state they open-source the training code and model checkpoints. The reproducibility statement is detailed, covering datasets, preprocessing, model configurations, and evaluation protocols. The use of standard datasets (Libri-Light, LibriSpeech) and public benchmarks further enhances reproducibility.
Limitations
The paper acknowledges that it does not explicitly preserve acoustic properties like prosody, which is crucial for speech resynthesis. Multi-stream modeling increases computational cost. The evaluation of generation quality relies on indirect metrics (ASR-PPL, LLMJ) which may conflate language modeling errors with TTS/ASR errors. The analysis of how individual streams contribute to the final performance is limited to empirical probing.
Broader impact
This work has significant implications for the development of speech-first AI systems. By demonstrating that linguistic structure can be learned efficiently from speech alone using multi-level representations, it challenges the prevailing reliance on text supervision for speech models. This could lead to more robust, language-agnostic speech models that do not require written language resources. The open-sourcing of the code and models will likely accelerate research in this area. GSLM2 introduces a multi-stream spoken language modeling framework that learns complementary discrete units at multiple levels of abstraction via differentiable self-supervised pretraining, significantly narrowing the data-efficiency gap between speech and text language models. The paper presents a rigorous and scalable approach to textless language modeling, demonstrating that leveraging multi-level representations from self-supervised encoders allows SLMs to achieve linguistic competence comparable to text LMs with far less data, offering a promising path for speech-native AI applications.
Author of this paper? Send a correction.