Nathaniel Bottman, Kyle Richardson
A theoretical study that formalizes question decomposition-based LLM reasoning using the mathematical structure of operads and proposes a new reliability metric called operadic consistency.
Although question decomposition is widely used in multi-step LLM reasoning, there is a lack of mathematical foundations to evaluate the accuracy and consistency of the decomposition process. Existing self-consistency methods rely on temperature sampling and fail to capture structural consistency.
Operads are mathematical structures that model many-in, one-out operations and their composition. We map question templates to operations and substitution of sub-answers to composition. QA models are interpreted as algebras over an operad, and operadic consistency is defined to measure answer agreement across partial collapses of a question decomposition tree.
In a companion paper (Bottman, Liu, Richardson, 2026), operadic consistency showed strong correlation with accuracy across 12 LLMs and 4 multi-hop QA datasets, outperforming temperature-based self-consistency baselines. We argue that operads are the natural mathematical home for question decomposition, and invariants like operadic consistency open new directions for analyzing and improving the reliability of multi-step reasoning.