Scalable Best-of-N Selection for Large Language Models via Self-Certainty

TL;DR

Self-Certainty is a novel metric that evaluates response quality based on the LLM's own probability distribution, enabling efficient scaling of Best-of-N selection without external reward models.

Problem

Best-of-N selection is a key technique for improving LLM reasoning performance, but existing methods either rely on computationally expensive reward models or, like self-consistency, suffer from limited scalability and generalization in open-ended generation tasks.

Approach

We aggregate the probability distributions of multiple responses generated by the LLM to compute 'self-certainty'. This reflects the model's confidence in its outputs, based on the hypothesis that higher self-certainty correlates with more accurate responses. Using only the LLM's own probability information without external reward models ensures high computational efficiency.

Results & Contribution

Experiments on various reasoning tasks demonstrate that self-certainty (1) scales effectively with increasing sample size N, similar to reward models but with much lower computational overhead; (2) complements chain-of-thought; and (3) generalizes to open-ended tasks where self-consistency fails. This provides a practical and efficient way to improve LLM reasoning performance.