Compute Efficiency of On-Policy Distillation vs. GRPO
arXiv — cs.LG metadata · Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz, Xuan-Phi Nguyen, Anurag Koul, Srijan Bansal, Erik Nijkamp, Shafiq Joty, Semih Yavuz · Published on Readzer
Summary
A study compares on-policy distillation (OPD) with Group Relative Policy Optimization (GRPO) in terms of training FLOPs to reach target accuracy. With an external post-trained teacher, OPD generally underperforms GRPO, but when the teacher is the student's own GRPO checkpoint, OPD recovers the checkpoint's accuracy on math with 50x fewer distillation FLOPs. Multi-teacher OPD consolidates per-domain GRPO experts into a single student, matching each expert's accuracy with distillation costing only 5.2% (2 domains) and 8.0% (4 domains) of specialist training FLOPs, and achieving joint GRPO performance on a 2-domain mix with 42.0% fewer total FLOPs.
This is a reviewed summary. Read the complete article at the original source.
Why it matters by industry
- Artificial intelligence
Relevant to AI model training efficiency, particularly for reasoning and agent models, as it provides insights into when distillation can reduce compute compared to reinforcement learning.