Nathaniel Bottman, Yinhong Liu, Kyle Richardson
Operadic consistency (OC) detects reasoning failures in LLMs without ground-truth labels by measuring agreement between a model's direct answer to a compositional query and the answer produced by composing a stated decomposition of the same query.
Detecting LLM reasoning failures at inference time is crucial for reliability, but existing methods (self-consistency, semantic entropy, etc.) degrade on complex multi-hop questions. In particular, they struggle to catch errors where the model fails to properly compose answers from decomposed subquestions.
Borrowing from operad theory, we compute a per-question score: the consistency between the model's direct answer and the answer obtained by decomposing the question into subquestions, answering each, and recomposing. We evaluate OC across 12 instruction-tuned LLMs (4B to 671B parameters) on four multi-hop QA datasets and compare it with existing signals.
OC shows strong correlation with accuracy across all datasets (Pearson r 0.86-0.94) and consistently outperforms existing methods in selective prediction (accuracy vs. coverage). Notably, it remains effective even when the decomposition is extracted from the model's own chain of thought in frontier models.