Jian Hu
REINFORCE++ is a simple improvement of the REINFORCE algorithm that achieves PPO-level performance while being much simpler and more efficient for LLM alignment.
In aligning LLMs with human preferences, PPO (Proximal Policy Optimization) is widely used but complex to implement, sensitive to hyperparameters, and computationally expensive. The REINFORCE algorithm is simpler but suffers from lower performance and training instability. Thus, a simple yet effective alignment method is needed.
REINFORCE++ applies three key improvements to the basic REINFORCE algorithm: (1) reward normalization for training stability, (2) token-level rewards for fine-grained feedback, and (3) cosine learning rate scheduling for better convergence. These modifications achieve performance comparable to PPO without PPO's complex clipping or KL penalties.
On various LLM alignment benchmarks (e.g., Anthropic Helpful, Harmless, TL;DR summarization), REINFORCE++ matches or outperforms PPO, with up to 2x faster training and 30% less memory usage. The method is simple to implement, lowering the barrier for researchers to adopt LLM alignment.