Finetuned Language Models Are Zero-Shot Learners (FLAN)
Abstract
This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces instruction tuning, demonstrating that finetuning large language models on a diverse mixture of tasks framed as natural language instructions dramatically improves zero-shot generalization to unseen tasks. By rigorously establishing scaling laws for instruction tuning, providing comprehensive ablations on dataset diversity and template design, and outperforming GPT-3 across a broad benchmark suite, the work fundamentally redefined how large language models are aligned and deployed, serving as the foundational blueprint for the modern instruction-following LLM paradigm.
Methodology
The paper introduces "instruction tuning," a paradigm where a pretrained decoder-only language model is finetuned on a mixture of >60 NLP datasets, each reformulated with multiple natural language instruction templates. The methodology is deceptively simple but rigorously structured. Key strengths include the careful construction of task clusters to enforce strict zero-shot evaluation (holding out entire semantic task types during finetuning), the use of template ensembling to mitigate prompt sensitivity, and systematic ablations isolating the effects of dataset diversity, model scale, and instruction phrasing. The approach elegantly bridges multi-task learning and prompt-based inference, demonstrating that supervised finetuning on diverse, instruction-formatted tasks can teach a model to generalize to unseen tasks purely from natural language prompts.
Results & evaluation
The experimental design is comprehensive and well-controlled. The authors evaluate across 25 datasets spanning NLI, reading comprehension, closed-book QA, translation, and more, comparing against strong baselines including zero-shot and few-shot GPT-3 (175B) and GLaM. Results consistently show that instruction tuning unlocks substantial zero-shot capabilities, surpassing GPT-3 zero-shot on 20/25 tasks and few-shot on 6 tasks. The scaling ablations are particularly insightful, revealing that instruction tuning only yields positive zero-shot transfer at scales ≥68B parameters, highlighting it as an emergent capability. The paper also extends the core finding to few-shot prompting and continuous prompt tuning, demonstrating broad compatibility. A thorough data contamination analysis follows GPT-3's protocol, mitigating concerns about pretraining leakage and strengthening result validity.
Reproducibility
High. The paper provides explicit training hyperparameters (Adafactor optimizer, LR, batch size, 30k steps, sequence lengths, mixing scheme), model architecture details, and clear evaluation protocols. The instruction-tuning dataset mixture and code are publicly released, and the template construction process is well-documented. The primary barrier to exact replication is the computational cost of training/evaluating a 137B parameter model, but the methodology, data pipeline, and evaluation scripts are fully transparent and reproducible for well-resourced labs.
Limitations
The authors acknowledge several constraints: (1) subjectivity in defining task clusters, which could affect the strictness of the zero-shot split; (2) reliance on manually crafted, short instruction templates, which does not explore complex, multi-step, or crowd-sourced instructions; (3) limited gains on tasks already aligned with the autoregressive LM objective (e.g., sentence completion/commonsense), indicating instruction tuning is not universally beneficial; (4) high inference cost due to model scale; and (5) potential bias propagation from the finetuning datasets into zero-shot applications. The manual template creation per dataset also presents a scalability bottleneck for future work.
Broader impact
This work fundamentally shifted the NLP paradigm from manual prompt engineering to systematic instruction tuning, directly catalyzing the development of InstructGPT, ChatGPT, and the open-source instruction-tuned ecosystem (Alpaca, Vicuna, Llama-Instruct, etc.). It demonstrates that labeled data can be leveraged not just for specialist models, but to create highly capable generalist models that follow natural language commands. This lowers the barrier to deploying powerful LMs for non-experts while simultaneously raising ethical concerns regarding bias amplification, misuse potential, and the environmental/compute costs of scaling instruction-tuned systems. This paper introduces instruction tuning, demonstrating that finetuning large language models on a diverse mixture of tasks framed as natural language instructions dramatically improves zero-shot generalization to unseen tasks. By rigorously establishing scaling laws for instruction tuning, providing comprehensive ablations on dataset diversity and template design, and outperforming GPT-3 across a broad benchmark suite, the work fundamentally redefined how large language models are aligned and deployed, serving as the foundational blueprint for the modern instruction-following LLM paradigm.
Curator note
Wei et al.; instruction tuning; zero-shot generalization
Author of this paper? Send a correction.