Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)
Abstract
We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that scaling weakly supervised, multilingual audio-transcript pairs with a standard Transformer architecture yields highly robust, zero-shot speech recognition models that approach human-level generalization. By rigorously exposing the in-distribution vs. out-of-distribution evaluation gap, releasing foundational models, and establishing a new paradigm for dataset-scale speech AI, it provides a highly influential empirical foundation that has reshaped both academic research and industry deployment in audio processing.
Methodology
The paper adopts a deliberately minimalist methodological stance, prioritizing dataset scale and diversity over architectural innovation. The core pipeline involves web-scale audio-transcript collection, heuristic filtering to remove machine-generated captions and misaligned segments, and training a standard encoder-decoder Transformer on 680,000 hours of multilingual, multitask data. The multitask formulation is elegantly simple: special prompt tokens condition the decoder on language, task (transcribe/translate), and timestamping, unifying speech recognition, translation, language identification, and voice activity detection into a single sequence-to-sequence objective. While the architecture itself is a conventional Transformer with a lightweight convolutional stem, the methodological rigor lies in the data curation strategy, the explicit rejection of dataset-specific fine-tuning in favor of zero-shot transfer, and the careful handling of text normalization to ensure fair evaluation. The approach effectively demonstrates that scaling weak supervision, when paired with robust filtering and multitask conditioning, can yield highly generalizable models without complex self-supervised objectives.
Results & evaluation
The experimental design is comprehensive and sets a new standard for evaluating speech foundation models. The authors evaluate zero-shot performance across 12+ English ASR benchmarks, multilingual datasets (MLS, VoxPopuli, FLEURS), translation (CoVoST2), and long-form transcription, alongside rigorous noise robustness and scaling law analyses. A particularly strong contribution is the critical examination of the in-distribution vs. out-of-distribution evaluation gap, showing that models fine-tuned on LibriSpeech achieve artificially low WERs but degrade sharply on shifted distributions, whereas Whisper's zero-shot performance closely matches human robustness. The scaling analyses (model size, dataset size, multitask transfer) are methodical and reveal clear power-law trends with diminishing returns at scale, providing valuable empirical guidance for future work. The inclusion of commercial ASR baselines and detailed long-form decoding heuristics further grounds the evaluation in real-world applicability.
Reproducibility
High. The authors release model weights, inference code, and detailed training hyperparameters. The architecture is standard and well-documented, and the dataset construction pipeline is described with sufficient transparency for replication, though the exact web-scraped corpus is not released due to licensing constraints. The release of the text normalization script is a significant reproducibility aid, as it addresses a known pain point in ASR evaluation where minor formatting differences artificially inflate WER. The only minor reproducibility gap is the exact composition of the 680k-hour dataset, which remains proprietary, but the filtering heuristics and scaling trends are thoroughly documented.
Limitations
The paper explicitly acknowledges several important limitations. First, long-form transcription suffers from seq2seq failure modes (repetition loops, hallucination, boundary truncation) that require heuristic decoding workarounds rather than architectural solutions. Second, the dataset is heavily English-biased, leading to poor zero-shot performance on low-resource and typographically distinct languages, with performance tightly correlated to training data volume. Third, the study focuses exclusively on zero-shot transfer, leaving the potential benefits of fine-tuning unexplored. Fourth, the relative contributions of the encoder vs. decoder to robustness remain unclear, and the reliance on heuristic text normalization introduces a risk of evaluation bias if the normalizer overfits to Whisper's output quirks. Finally, the environmental and compute costs of training at this scale are not discussed.
Broader impact
This work fundamentally shifts the paradigm in speech recognition from narrow, fine-tuned systems to robust, zero-shot foundation models. By releasing high-quality models and code, it dramatically lowers the barrier to entry for speech applications, enabling researchers and developers to deploy accurate multilingual ASR without domain-specific data collection or fine-tuning. This democratization has profound implications for accessibility, content indexing, and human-computer interaction globally. However, it also raises important considerations regarding data provenance, copyright of web-scraped audio, the environmental footprint of large-scale training, and the potential displacement of specialized ASR services. The paper's emphasis on zero-shot robustness over in-distribution optimization serves as a crucial corrective for the broader ML community, advocating for evaluation protocols that better reflect real-world deployment conditions. The paper demonstrates that scaling weakly supervised, multilingual audio-transcript pairs with a standard Transformer architecture yields highly robust, zero-shot speech recognition models that approach human-level generalization. By rigorously exposing the in-distribution vs. out-of-distribution evaluation gap, releasing foundational models, and establishing a new paradigm for dataset-scale speech AI, it provides a highly influential empirical foundation that has reshaped both academic research and industry deployment in audio processing.
Curator note
Radford et al.; OpenAI; standard ASR; 680k hours weak supervision
Discussion
Talks & videos
Author of this paper? Send a correction.