STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

TL;DR

This paper proposes STARE to solve the policy entropy collapse problem in GRPO-like RL algorithms through surprisal-guided token-level advantage reweighting.

Problem

RL algorithms with verifiable rewards like GRPO suffer from policy entropy collapse during training, which degrades the model's exploration capability and limits its complex reasoning performance.

Approach

The authors conduct a first-order gradient analysis of token-level entropy dynamics, identifying an advantage-surprisal four-quadrant structure. STARE identifies entropy-critical tokens via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation.

Results & Contribution

Across model scales from 1.5B to 32B and three task families, STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating a sustained exploration-exploitation balance.