Measuring Massive Multitask Language Understanding (MMLU)
Abstract
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces the Massive Multitask Language Understanding (MMLU) benchmark, a comprehensive evaluation suite spanning 57 academic and professional domains that rapidly became the field's standard for measuring LLM knowledge and reasoning, fundamentally shaping model development, scaling research, and capability assessment across academia and industry.
Methodology
The paper introduces a systematically curated benchmark comprising 57 academic and professional subjects, unified under a multiple-choice question format to enable scalable, automated evaluation. The methodology leverages a standardized 5-shot in-context learning protocol to fairly compare models of varying architectures and training scales. The dataset construction process demonstrates careful attention to difficulty stratification, domain diversity, and distractor quality. While the multiple-choice paradigm sacrifices open-ended generative evaluation, it provides a highly controlled, low-variance signal for measuring factual knowledge and reasoning breadth. The inclusion of calibration analysis (confidence vs. accuracy) and uncertainty quantification adds methodological rigor beyond simple accuracy reporting.
Results & evaluation
Experiments comprehensively evaluate a spectrum of models (GPT-2, GPT-3, UnifiedQA, and others) across all 57 tasks, clearly demonstrating scaling laws and the performance gap between current models and human experts. The empirical findings are robust, well-structured, and reveal critical insights: models exhibit severe domain imbalance, poor self-calibration, and near-random performance on socially critical subjects like law and morality. The use of both zero-shot and few-shot settings provides a nuanced view of model capabilities. However, the evaluation lacks deeper error analysis, such as categorizing failure modes (e.g., reasoning vs. knowledge retrieval vs. distractor confusion) and does not include human-in-the-loop validation for ambiguous or culturally specific questions.
Reproducibility
High. The dataset is publicly released with clear documentation, and the evaluation pipeline (prompt templates, scoring scripts, and subject splits) is standardized and open-source. The multiple-choice format ensures deterministic, easily replicable scoring. The paper provides sufficient detail on data sourcing, filtering criteria, and prompt construction to enable exact replication. Subsequent community validation and widespread adoption have further confirmed the benchmark's reproducibility and stability across different evaluation frameworks.
Limitations
The exclusive reliance on multiple-choice questions limits the benchmark's ability to assess open-ended reasoning, creative synthesis, or interactive problem-solving. The dataset exhibits geographic and cultural biases, particularly in US-centric subjects like history and law. Static benchmarking inherently risks data contamination as LLMs are increasingly trained on web corpora that may include test questions or similar formulations. The fixed 5-shot protocol, while practical, may not generalize to real-world deployment scenarios where prompt engineering, tool use, or iterative refinement are employed. Finally, the benchmark measures knowledge breadth but does not explicitly evaluate safety, alignment, or robustness to adversarial prompting.
Broader impact
MMLU has fundamentally reshaped how the ML community measures, compares, and develops large language models, becoming the de facto standard for capability evaluation across academia and industry. Its identification of calibration failures and domain-specific weaknesses has directly informed research into model alignment, uncertainty quantification, and specialized fine-tuning. However, the community's heavy reliance on a single benchmark risks incentivizing benchmark overfitting, narrow optimization, and premature deployment claims. The paper responsibly highlights the societal implications of deploying models with poor performance in high-stakes domains, advocating for cautious integration and continued research into reliability, transparency, and domain-specific validation. The paper introduces the Massive Multitask Language Understanding (MMLU) benchmark, a comprehensive evaluation suite spanning 57 academic and professional domains that rapidly became the field's standard for measuring LLM knowledge and reasoning, fundamentally shaping model development, scaling research, and capability assessment across academia and industry.
Curator note
Hendrycks et al.; 57-domain knowledge benchmark; standard LLM eval
Author of this paper? Send a correction.