Nick Jiang, Isaac Kauvar, Jack Lindsey
This study shows that language models internally encode the success likelihood (value) of their current reasoning trajectory in a linear manner, and manipulating this axis changes the model's confidence and behaviors such as self-correction and exploration.
To investigate whether language models internally evaluate how effective their current strategy is for achieving goals during reasoning, and how this evaluation influences behavior.
Using synthetic in-context reinforcement learning data, we extract a 'value axis' from the activation vectors of Qwen3-8B. By steering activations along this axis, we analyze its effect on self-correction, exploration, and explanatory verbosity. We also show that DPO can increase the internal value of rewarded behaviors.
The value axis distinguishes between high vs. low confidence, rollouts with vs. without backtracking, and correct vs. corrupted code. Steering toward high value suppresses self-correction and reduces verbosity, while steering toward low value induces backtracking and exploration. We also confirm its validity in real-world applications, such as assigning low value to politically sensitive queries.