Code Llama: Open Foundation Models for Code
Abstract
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Code Llama, a family of state-of-the-art open foundation models for code generation, which demonstrate significant performance improvements over existing models and provide new capabilities for programming tasks. The comprehensive evaluation and innovative training methodologies position this work as a significant advancement in the field of machine learning for code.
Methodology
The paper introduces a family of large language models specifically designed for code generation, leveraging the Llama 2 architecture. The models are trained on a substantial dataset of code, with a focus on infilling capabilities and long context handling. The methodology includes specialized training for Python and instruction-following tasks, which are innovative approaches in the context of code generation. The use of self-instruction and execution feedback for dataset generation is particularly noteworthy, as it addresses the challenges of acquiring high-quality labeled data in programming tasks.
Results & evaluation
The evaluation is comprehensive, covering multiple benchmarks such as HumanEval, MBPP, and MultiPL-E, which are critical for assessing code generation capabilities. The results demonstrate state-of-the-art performance among open models, with detailed comparisons across different model sizes and configurations. The ablation studies provide insights into the impact of various training strategies, enhancing the credibility of the findings.
Reproducibility
The paper provides sufficient details on the training process, datasets, and evaluation metrics, which are essential for reproducibility. The release of the models under a permissive license further facilitates access for researchers and practitioners, promoting reproducibility and experimentation in the community.
Limitations
While the models show impressive performance, the paper does not extensively discuss the limitations of the approach, such as potential biases in the training data or the models' performance on less common programming languages. Additionally, the focus on Python may limit the generalizability of the findings to other programming contexts.
Broader impact
The Code Llama models have significant implications for software development, enabling more efficient code generation and assistance in programming tasks. Their open availability encourages further research and development in the field, potentially leading to advancements in automated programming tools and educational resources for developers. The main contribution of this paper is the introduction of Code Llama, a family of state-of-the-art open foundation models for code generation, which demonstrate significant performance improvements over existing models and provide new capabilities for programming tasks. The comprehensive evaluation and innovative training methodologies position this work as a significant advancement in the field of machine learning for code.
Curator note
Rozière et al.; Meta; open-weights code LLM; extends Llama 2 for code
Author of this paper? Send a correction.