DeepSeek-V3 Technical Report
Abstract
We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
DeepSeek-V3 represents a significant advancement in the development of large language models, introducing innovative methodologies and achieving state-of-the-art performance on a variety of benchmarks. The comprehensive evaluation and robust architecture suggest that it will be a valuable resource for researchers and practitioners in the field of machine learning.
Methodology
The methodology presented in DeepSeek-V3 is robust and innovative, particularly with its introduction of the Multi-head Latent Attention (MLA) and the auxiliary-loss-free load balancing strategy. The use of FP8 mixed precision training is a significant advancement that enhances training efficiency while maintaining model performance. The multi-token prediction training objective is a novel approach that could potentially improve training signal density and model performance. Overall, the methodology is well-structured and demonstrates a clear progression from previous versions of the model.
Results & evaluation
The experimental evaluation is comprehensive, involving extensive pre-training on a large and diverse dataset of 14.8 trillion tokens. The results show that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models, which is a significant achievement. The paper provides detailed evaluation metrics across various benchmarks, including educational, factuality, coding, and mathematical reasoning tasks, showcasing the model's versatility and strength.
Reproducibility
The paper includes a URL to the model checkpoints and code repository, which is essential for reproducibility. However, it lacks detailed implementation specifics that would help other researchers replicate the results precisely. The description of the training framework and methodologies is thorough, but additional details on hyperparameters and training settings would enhance reproducibility.
Limitations
One limitation is the lack of information regarding the specific institutions involved in the research, which could impact the perceived credibility and authority of the work. Additionally, while the model shows strong performance, it does not address potential biases in the training data or the implications of deploying such a large model in real-world applications.
Broader impact
The advancements made in DeepSeek-V3 have the potential to significantly impact the field of NLP by providing a powerful open-source alternative to closed-source models. The model's capabilities in reasoning, coding, and multilingual tasks can facilitate broader applications in education, software development, and beyond. However, the implications of deploying such large models must be carefully considered, particularly concerning ethical and environmental impacts. DeepSeek-V3 represents a significant advancement in the development of large language models, introducing innovative methodologies and achieving state-of-the-art performance on a variety of benchmarks. The comprehensive evaluation and robust architecture suggest that it will be a valuable resource for researchers and practitioners in the field of machine learning.
Curator note
DeepSeek; 671B MoE; $6M training cost; matched proprietary frontier
Author of this paper? Send a correction.