Kimi k1.5: Scaling Reinforcement Learning with LLMs
Abstract
Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Kimi k1.5, a multimodal LLM that effectively scales reinforcement learning techniques to achieve state-of-the-art performance across various reasoning benchmarks. The comprehensive analysis of the methodology, experimental results, and potential implications highlights the significance of this work in advancing the field of machine learning.
Methodology
The paper presents a novel approach to scaling reinforcement learning (RL) with large language models (LLMs) through the introduction of Kimi k1.5, which integrates long context scaling and improved policy optimization methods. The methodology is well-structured, highlighting the importance of long-CoT techniques and the simplification of RL frameworks by avoiding complex techniques like Monte Carlo tree search. The authors emphasize the use of partial rollouts to enhance training efficiency, which is a significant contribution to the field. The proposed long2short methods for improving short-CoT models are particularly innovative, showcasing a thoughtful approach to leveraging long-CoT benefits in a more efficient manner.
Results & evaluation
The experimental results demonstrate that Kimi k1.5 achieves state-of-the-art performance across multiple benchmarks, which is a strong indicator of the model's effectiveness. The authors provide extensive evaluation metrics, including comparisons against existing models like OpenAI's o1 and GPT-4o, which strengthens the credibility of their claims. The use of diverse datasets and benchmarks across text, reasoning, and vision tasks adds robustness to the evaluation process. However, the paper could benefit from more detailed descriptions of the datasets used and the specific experimental setups to facilitate reproducibility.
Reproducibility
The paper lacks sufficient detail in the implementation section, making it challenging for others to reproduce the results. While the methodology is described in depth, the absence of specific URLs for code repositories or demo pages limits accessibility. Clearer documentation of the training infrastructure and hyperparameters would enhance reproducibility.
Limitations
One limitation is the reliance on the quality of the RL prompt set, which may affect the model's performance if not adequately diverse or challenging. Additionally, the paper does not address potential biases in the training data or the implications of using automated test case generation for coding problems. The absence of a comprehensive discussion on the ethical implications of deploying such models in real-world applications is also a notable gap.
Broader impact
The advancements presented in Kimi k1.5 have the potential to significantly impact various applications, including automated reasoning, coding assistance, and multimodal AI systems. The ability to effectively scale RL with LLMs could lead to more capable AI systems that can tackle complex tasks across different domains. However, careful consideration of ethical implications and biases in training data is essential to ensure responsible deployment. The main contribution of this paper is the introduction of Kimi k1.5, a multimodal LLM that effectively scales reinforcement learning techniques to achieve state-of-the-art performance across various reasoning benchmarks. The comprehensive analysis of the methodology, experimental results, and potential implications highlights the significance of this work in advancing the field of machine learning.
Curator note
Moonshot AI; RL-based reasoning with long + short CoT; competitive with o1
Author of this paper? Send a correction.