Large Language Models lack essential metacognition for reliable medical reasoning

TL;DR

LLMs show high accuracy on medical exams but exhibit metacognitive deficiencies, failing to recognize knowledge limitations and confidently providing incorrect answers, as revealed by the MetaMedQA benchmark evaluation.

Problem

For LLMs to be used as clinical decision support systems in medicine, they must possess metacognitive abilities beyond simple answer prediction, such as recognizing their own uncertainty and signaling when information is insufficient. However, existing evaluations focus only on accuracy, overlooking these abilities.

Approach

The researchers developed MetaMedQA, a benchmark that integrates confidence scores and metacognitive tasks (e.g., recognizing absence of correct answer, avoiding unknown questions) into multiple-choice medical questions. They evaluated 12 LLMs on dimensions including confidence-based accuracy, missing answer recall, and unknown recall.

Results & Contribution

Despite high multiple-choice accuracy, all models showed severe metacognitive deficiencies, such as confidently answering even when no correct option was present. This reveals a disconnect between perceived and actual capabilities, highlighting risks in clinical settings. The findings emphasize the need for evaluation frameworks that incorporate metacognitive abilities.