Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty

[Submitted on 15 Jun 2026]

View PDF HTML (experimental)

Abstract:Large language models can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently — a failure mode especially prevalent in multi-step deductive reasoning. Existing methods assess reliability primarily through output dispersion — measuring how much sampled answers differ — but this discards a complementary signal: whether the model can consistently rank competing reasoning candidates. We propose structural uncertainty, a consistency-aware framework derived from the stability of self-preference-induced rankings over sampled reasoning solutions. Given a query, we generate multiple candidate solutions and ask the model to judge pairwise preferences among its own outputs. We aggregate self-preferences into ranking distributions via Bradley-Terry modeling with PageRank, and decompose the signal into two entropy-based components: across-trial ranking instability and within-trial candidate ambiguity. Across five LLMs and eight benchmarks, structural signals provide information complementary to answer dispersion: on logical and mathematical reasoning tasks, the combination improves identification of unreliable instances, while on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime boundary where reasoning-level consistency evaluation is uninformative. The two components relate differently to accuracy: within-trial ambiguity correlates positively with correctness — consistent with settings where multiple plausible solution paths remain competitive — while across-trial instability correlates negatively, signaling unreliable reasoning. Structural uncertainty is best understood not as a universal confidence estimator, but as a regime-sensitive evaluator of logical reasoning consistency.

Submission history

From: Baishali Chaudhury [view email]
[v1] Mon, 15 Jun 2026 21:39:47 UTC (8,497 KB)