Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Abstract
We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces phi-3, a series of language models that achieve competitive performance with larger models while being deployable on mobile devices. This work significantly advances the field of NLP by demonstrating that smaller models can be trained effectively through optimized data strategies, potentially reshaping how practitioners approach model development and deployment.
Methodology
The paper presents a novel approach to training smaller language models (phi-3-mini, phi-3-small, phi-3-medium) that achieve competitive performance with larger models through optimized data curation and training methodologies. The introduction of a "data optimal regime" is particularly noteworthy, as it deviates from conventional scaling laws by focusing on the quality and relevance of training data rather than merely increasing model size. The use of Mixture-of-Experts (MoE) architecture in phi-3.5-MoE enhances efficiency and performance, showcasing a thoughtful integration of advanced techniques.
Results & evaluation
The experimental results are robust, with the models evaluated against a wide range of benchmarks, demonstrating significant improvements over previous iterations and comparable performance to much larger models. The thorough benchmarking across various tasks, including multilingual and multimodal capabilities, adds credibility to the claims of the models' effectiveness. The paper provides detailed comparisons with state-of-the-art models, which strengthens the empirical validation of the proposed models.
Reproducibility
The paper lacks explicit URLs for code or model access, which could hinder reproducibility. While it mentions using established architectures and training methodologies, the absence of a public repository or demo limits the ability of other researchers to replicate the findings. Clearer documentation or a dedicated project page would enhance reproducibility.
Limitations
The models, while powerful, still exhibit limitations in handling factual knowledge and reasoning tasks, particularly in low-resource languages. The authors acknowledge that phi-3-mini's capacity constraints may lead to lower performance in certain benchmarks, such as TriviaQA. Additionally, challenges around safety and bias remain, indicating that further work is needed to address these issues comprehensively.
Broader impact
The ability to deploy high-performance language models on mobile devices has significant implications for accessibility and real-time applications. This advancement could democratize access to advanced AI capabilities, enabling users in various contexts to leverage powerful language processing tools. However, the potential for misuse and the need for responsible AI practices must be carefully managed. The paper introduces phi-3, a series of language models that achieve competitive performance with larger models while being deployable on mobile devices. This work significantly advances the field of NLP by demonstrating that smaller models can be trained effectively through optimized data strategies, potentially reshaping how practitioners approach model development and deployment.
Curator note
Abdin et al., Microsoft; 3.8B matches much larger models; efficient edge-deployable LLM
Author of this paper? Send a correction.