Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
Abstract
On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with $50\times$ fewer distillation FLOPs than GRPO needs to reach the same accuracy. Following the same teacher setting, we consistently observe this high efficiency with a larger student, on code, and in a second model family. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD (DeepSeek-AI, 2026) consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper provides a rigorous, FLOPs-based comparison of On-Policy Distillation and GRPO, revealing that while external-teacher OPD is generally less compute-efficient than GRPO, using the student's own GRPO checkpoint as a teacher enables significant efficiency gains, particularly in multi-domain consolidation scenarios. This work corrects the field's understanding of distillation efficiency and offers a practical, compute-aware recipe for training multi-domain LLMs.
Methodology
The paper proposes a rigorous, FLOPs-based evaluation framework for comparing On-Policy Distillation (OPD) against Group Relative Policy Optimization (GRPO). The methodology is sound, explicitly accounting for the computational cost of teacher forward passes in OPD versus the pure student cost in GRPO. It systematically ablates teacher provenance (external vs. self-GRPO), model size, and domain. The core methodological contribution is not a new algorithm, but a precise characterization of *when* OPD is efficient, identifying the "self-GRPO teacher" as the critical condition for compute efficiency.
Results & evaluation
The experiments are extensive, covering math, code, knowledge, and instruction following domains. The use of a "difficulty grid" to stratify training data is a clever diagnostic tool. The finding that external teachers generally fail to beat GRPO on FLOPs is a strong negative result that corrects misconceptions in the field. The positive result regarding multi-teacher consolidation is well-supported by the 42% FLOP reduction claim. The evaluation metrics (mean@16 accuracy) are standard for this domain.
Reproducibility
High. The paper provides detailed hyperparameters, dataset construction details, and FLOPs accounting formulas. It explicitly states the use of open-source models (Qwen3, Olmo) and public datasets (DAPO, LiveCodeBench, MMLU-Pro). The reproducibility statement is thorough, listing framework versions and specific configuration choices.
Limitations
The study is limited to the vanilla reverse-KL objective. It does not explore hybrid objectives or alternative divergences that might change the FLOPs balance. The "self-GRPO" setup is somewhat circular (training a model to distill itself), which may not generalize to all practical scenarios where a pre-trained teacher is the only option. The paper acknowledges that the student sometimes outperforms the teacher in self-GRPO settings, a phenomenon left unexplained.
Broader impact
This paper has high practical impact for LLM post-training pipelines. It provides a clear rationale for the "specialize then consolidate" strategy used by frontier labs (DeepSeek, Kimi). By quantifying the FLOPs cost, it helps practitioners decide between RL and distillation based on compute budgets rather than just accuracy. It shifts the focus from "can OPD work?" to "how much compute does OPD save compared to RL?", a more useful question for industry. The paper provides a rigorous, FLOPs-based comparison of On-Policy Distillation and GRPO, revealing that while external-teacher OPD is generally less compute-efficient than GRPO, using the student's own GRPO checkpoint as a teacher enables significant efficiency gains, particularly in multi-domain consolidation scenarios. This work corrects the field's understanding of distillation efficiency and offers a practical, compute-aware recipe for training multi-domain LLMs.
Author of this paper? Send a correction.