Closing the Confidence-Faithfulness Gap in Large Language Models

TL;DR

A study that mechanistically explains the gap between LLMs' verbalized confidence and actual accuracy, and improves calibration by steering using internal signals.

Problem

LLMs verbalize confidence scores that are often unrelated to actual accuracy. The geometric causes of this 'confidence-accuracy gap' need to be understood and effectively reduced.

Approach

Using linear probes and contrastive activation addition (CAA) steering on three open-weight models and four datasets, we analyzed how confidence and calibration signals are represented. We found that both signals are linearly encoded but orthogonal to each other. Additionally, we identified the 'Reasoning Contamination Effect' where the reasoning process disrupts the confidence signal direction when the model simultaneously reasons and verbalizes confidence. Based on this, we designed a two-stage adaptive steering pipeline that reads the model's internal accuracy estimate to adjust the confidence output.

Results & Contribution

The proposed method significantly improved calibration alignment across all evaluated models, and mechanistically revealed the root cause of the confidence-accuracy gap. This contributes to enhancing the reliability of LLMs.