Measuring short-form factuality in large language models

TL;DR

A short-form factuality benchmark adversarially collected against GPT-4 responses, evaluating models' factuality and ability to recognize their own limitations.

Problem

Existing factuality benchmarks for LLMs suffer from saturation or difficult grading. In particular, it is challenging to measure a model's ability to abstain from answering questions it does not know.

Approach

Each question is designed to have a single, indisputable answer. Questions are adversarially collected against GPT-4 to increase difficulty. Responses are graded as correct, incorrect, or not attempted, jointly evaluating accuracy and abstention rate.

Results & Contribution

SimpleQA achieves a difficulty level that current models find challenging, providing a simple and effective benchmark that simultaneously measures factuality and a model's awareness of its knowledge limits.