arXivReinforcement LearningMethodImportance 8arXiv2mo ago

Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling

Yucheng Li, Huiqiang Jiang, Yang Xu, Jianxin Yang, Yi Zhang, Yizhong Cao, Yuhao Shen, Fan Zhou et al.

0 CitationsPopularity 0.7

AI analysis

TL;DR

To address the rollout bottleneck in RL training, we propose Bebop, which combines Multi-Token Prediction (MTP) with rejection sampling. It overcomes entropy-based acceptance rate limits and achieves up to 1.8x acceleration.

Problem

RL plays a crucial role in LLM post-training, but the rollout stage is a major bottleneck in the pipeline. While MTP-based speculative decoding has been proposed as a solution, acceptance rates degrade significantly during RL training due to model entropy changes, limiting speedup.

Approach

We discover that MTP acceptance rates are fundamentally bounded by fluctuations in model entropy, with a clear negative linear relationship between entropy increase and acceptance rate. 2. We show that probabilistic rejection sampling is less sensitive to entropy changes compared to greedy draft sampling. 3. Since conventional MTP training objectives (cross-entropy, KL) are suboptimal in this setting, we propose an end-to-end total variation (TV) loss that directly optimizes multi-step rejection sampling acceptance rates. 4. We find that training MTP with TV loss and rejection sampling before RL maintains consistent acceptance rates throughout RL, eliminating the need for costly online updates.

Results & Contribution

We improve acceptance rates up to 95% and achieve up to 25% extra inference throughput gains on mathematical reasoning, code generation, and agentic tasks. In asynchronous RL training of Qwen3.5, Qwen3.6, and Qwen3.7 models, we achieve up to 1.8x end-to-end acceleration. We systematically analyze the relationship between entropy and acceptance rate, and provide practical integration recipes.