Reasoning emerges from constrained inference manifolds in large language models

TL;DR

Reasoning in LLMs emerges under geometric constraints where internal representations compress into low-dimensional manifolds with non-degenerate information volume preserved; a label-free diagnostic method is proposed.

Problem

Existing evaluation of LLM reasoning relies on labeled benchmarks, conflating task performance with the quality of internal inference. There is a need to understand the intrinsic dynamical nature of reasoning and to assess reasoning quality without labels.

Approach

Analyze the evolution of internal representations during inference in terms of compression into low-dimensional manifolds, representational expressivity, and information preservation. Models satisfying three conditions (adequate expressivity, spontaneous manifold compression, and preservation of non-degenerate information volume within the compressed subspace) exhibit stable reasoning; a unified diagnostic computed solely from internal dynamics is proposed.

Results & Contribution

Reveals that reasoning dynamics are governed by geometric and informational constraints, providing a label-free diagnostic framework that complements benchmark-centric evaluation.