Evaluating Large Language Models Trained on Code (HumanEval)
Abstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces the Codex model, the HumanEval benchmark, and the pass@k evaluation framework, fundamentally shifting code generation research from lexical matching to functional correctness. By rigorously demonstrating the scaling laws, sampling dynamics, and practical limitations of large language models trained on code, it established the methodological foundation for the modern code LLM ecosystem and set a new standard for empirical evaluation and responsible AI impact analysis in machine learning.
Methodology
The paper introduces a straightforward but highly effective methodology: fine-tuning a pre-trained GPT-3 architecture on a massive corpus of publicly available Python code (~159 GB from GitHub). While the architectural approach lacks fundamental algorithmic novelty, the methodological rigor lies in the evaluation design and sampling strategy. The authors introduce the `pass@k` metric with an unbiased estimator to measure functional correctness, explicitly moving the field away from flawed lexical overlap metrics like BLEU. They systematically analyze the relationship between sampling temperature, number of samples (`k`), and model size, demonstrating that higher temperatures are optimal for larger `k` due to increased diversity. The supervised fine-tuning (Codex-S) on curated competitive programming and CI-traced problems, along with the docstring back-translation model (Codex-D), provides a clean ablation of distribution alignment. The methodology is well-motivated, empirically grounded, and clearly separates architectural scaling from evaluation and sampling dynamics.
Results & evaluation
The experimental section is comprehensive and sets a new standard for code generation research. The authors evaluate across multiple model scales (up to 12B parameters), demonstrating clear power-law scaling for code fine-tuning. They benchmark against strong contemporaries (GPT-Neo, GPT-J, Tabnine) and validate findings on an external dataset (APPS), showing consistent gains from sampling and filtering. The analysis of ranking heuristics (mean log-probability vs. random vs. back-translation) is particularly valuable for practical deployment. The sandboxed execution environment for unit testing ensures reliable, safe evaluation. While the experiments are heavily focused on Python and single-function synthesis, the breadth of ablations (temperature, `k`, ranking, supervised fine-tuning, docstring generation) provides robust empirical evidence for the proposed claims.
Reproducibility
Excellent. The authors release the HumanEval dataset (164 hand-written problems with unit tests) and a complete evaluation framework. Training hyperparameters (optimizer, learning rate schedule, warmup, weight decay), tokenizer modifications (whitespace tokenization), sampling parameters (nucleus `p=0.95`, temperature tuning), and stop sequences are explicitly documented. The sandbox architecture (gVisor, eBPF firewall) is described in sufficient detail for replication. The numerical stability considerations for the `pass@k` estimator further enhance reproducibility. Minor limitations include the lack of released model weights (typical for proprietary LLMs at the time) and the exact GitHub repository filtering pipeline, but the dataset and evaluation code fully enable independent benchmarking.
Limitations
The authors provide a candid and thorough discussion of limitations. Key weaknesses include poor sample efficiency (requiring hundreds of millions of lines of training code), exponential performance degradation on long, chained docstrings, and systematic failures in variable-operation binding. The model struggles with system-level synthesis and often generates syntactically valid but semantically misaligned or insecure code. The reliance on massive compute and public code also raises data quality and distributional shift concerns. The paper acknowledges that `pass@k` with an oracle selection strategy does not reflect real-world single-sample deployment constraints, though they mitigate this with mean log-prob ranking. These limitations are well-documented and accurately reflect the state of early code LLMs.
Broader impact
The broader impact section is exceptionally comprehensive for its era, establishing a template for responsible AI reporting. It systematically addresses over-reliance (automation bias), misalignment (superficially correct but functionally flawed code), bias in code structure/comments, economic/labor market shifts, security vulnerabilities (insecure code generation, polymorphic malware potential), environmental footprint, and legal/IP considerations around training on public repositories. The hazard analysis is nuanced, distinguishing between capability limitations and alignment failures, and proposes concrete mitigations (UI design, rate limiting, content filtering, human oversight). While some projections are necessarily speculative, the framework encourages cross-sectoral scrutiny and sets a high bar for transparency in deploying generative code systems. This paper introduces the Codex model, the HumanEval benchmark, and the pass@k evaluation framework, fundamentally shifting code generation research from lexical matching to functional correctness. By rigorously demonstrating the scaling laws, sampling dynamics, and practical limitations of large language models trained on code, it established the methodological foundation for the modern code LLM ecosystem and set a new standard for empirical evaluation and responsible AI impact analysis in machine learning.
Curator note
Chen et al.; OpenAI; code generation benchmark
Author of this paper? Send a correction.