Technical Summary: Operadic Consistency as a Label-Free Signal for Compositional Reasoning Failures

1. Problem Statement

Modern Large Language Models (LLMs), including recent "thinking" models, frequently generate fluent, confident reasoning chains that fail to compose into a correct final answer. This phenomenon, known as the "compositionality gap," represents a critical failure mode where sub-conclusions do not logically aggregate to the intended solution. Detecting these failures at inference time without access to ground-truth labels is a central challenge in hallucination detection, model calibration, and selective prediction.

Existing inference-time confidence baselines generally fall into three categories:

1. Sample-and-aggregate methods: Such as Chain-of-Thought Self-Consistency (CoT-SC), which measures variance across temperature-sampled outputs.
2. Entropy-based methods: Such as Semantic Entropy, which computes entropy over semantic clusters of samples.
3. Self-evaluation methods: Such as $P(\text{True})$, which elicits the model's own probability that its answer is correct.

The authors argue that these methods primarily probe sample diversity or self-assessment, which are often orthogonal to the structural integrity of the reasoning process. A model can be confidently wrong with low sample variance, or correct with high variance. There is a lack of methods that directly test whether a model's reasoning is internally compositional—i.e., whether the sub-conclusions, when taken seriously and recomposed, yield the same final answer as the model's direct response.

2. Methodology: The Questions Operad and Operadic Consistency

The paper introduces a formal framework based on Operad Theory, a mathematical structure designed to study systems built by iterated substitution.

2.1 The Questions Operad ($\mathcal{Q}$)

The authors define the Questions Operad ($\mathcal{Q}$), where:

• Operations: Correspond to question templates with $k$ blanks (arity $k$).
• Composition: Defined by substituting a question template (with its own blanks) into a blank of another template. This creates a tree-shaped structure of questions (Tree of Questions, or ToQ).
• Algebras: A language model is interpreted as an algebra over $\mathcal{Q}$. Ideally, the model's answer to a composite question should equal the result of sequentially answering its sub-questions and substituting the answers back into the composite.

2.2 Operadic Consistency (OC)

Operadic Consistency is defined as the predicate where a model's direct answer to a complex query agrees with the answer produced by composing a stated decomposition of that query.

• Instantiation: The paper evaluates this on depth-2 chains (a root question decomposed into two steps: $Q_1 \to Q_2$).
• Protocol:
  1. Direct Path: The model answers the original question $Q$ directly.
  2. Decomposed Path: The model answers sub-question $Q_1$, substitutes the answer $A_1$ into $Q_2$, and answers the resulting $Q_2[A_1]$.
  3. Scoring: The OC score is the semantic agreement (0 to 1) between the direct answer and the decomposed final answer, calculated using dataset-specific scorers (e.g., SQuAD-F1, value-equality for numbers, yes/no extraction).
• Cost: This requires exactly 3 model calls per question (1 direct, 2 decomposed steps).

The framework applies to both instruction-tuned (non-thinking) models (where decompositions are supplied by human annotators or dataset metadata) and thinking models (where the decomposition is extracted from the model's own chain of thought).

3. Key Contributions

1. Theoretical Framework: The definition of the Questions Operad $\mathcal{Q}$ and the interpretation of LLMs as algebras over $\mathcal{Q}$, providing a formal basis for diagnosing compositional failures.
2. Empirical Signal (OC): The proposal of Operadic Consistency as a per-question signal that correlates strongly with accuracy.
3. Complementarity: Demonstration that OC provides predictive information distinct from and complementary to standard baselines (CoT-SC, Semantic Entropy, $P(\text{True})$).
4. Deployment Utility: Evidence that combining OC with CoT-SC improves selective prediction (accuracy at fixed coverage) at equal inference costs.

4. Experimental Results

The study evaluated 17 models (12 instruction-tuned, 5 thinking models) across four multi-hop QA datasets (HotpotQA, MuSiQue, StrategyQA, DROP) and one math dataset (GSM8K for thinking models).

4.1 Cross-Model Accuracy Correlation

• Uniform Strength: OC shows a strong Pearson correlation ($r \in [0.86, 0.94]$) with model accuracy across all four datasets.
• Comparison: CoT-SC (at $K=10$) matches OC on HotpotQA ($r=0.93$) and DROP ($r=0.87$) but collapses on MuSiQue ($r=0.43$) and StrategyQA ($r=0.46$). Semantic Entropy and $P(\text{True})$ also fail to maintain high correlation across all datasets.
• Label-Free Estimation: Using a leave-one-model-out protocol, the OC rate of a held-out model predicts its accuracy with a Mean Absolute Error (MAE) of 3.0 percentage points, significantly outperforming CoT-SC ($K=10$), which has an MAE of 4.6 pp.

4.2 Per-Question Predictive Power

• Logistic Regression: When predicting correctness on a per-question basis, OC contributes significant marginal information even when controlling for CoT-SC, Semantic Entropy, and decomposition-aware baselines (e.g., decomposed self-consistency). The coefficient for OC is highly significant ($p \le 10^{-16}$) across all datasets.
• Decomposition-Aware Baselines: The paper tested "decomposed self-consistency" (sampling only the second step of the decomposition) and a Process Reward Model (Skywork-PRM). Neither achieved the uniform cross-model correlation of OC, and neither displaced OC's significance in joint regressions.

4.3 Selective Prediction

• Non-Thinking Models: At an equal cost budget ($K=3$ calls), adding OC to a CoT-SC baseline improved AUARC by +0.086 to +0.096 and AUROC by +0.092 to +0.164 across all four datasets. Bootstrap 95% confidence intervals excluded zero for every cell.
• Thinking Models: On five frontier thinking models, where decompositions were extracted from the model's own chain of thought, the OC + Self-Consistency combiner showed positive point-estimate lifts on all 16 tested (dataset, budget, metric) cells, with 12 of 16 having 95% CIs excluding zero.

5. Significance and Claims

The paper claims that Operadic Consistency offers a structural diagnostic for LLM reasoning that is fundamentally different from variance-based signals.

• Mechanism: While CoT-SC and Semantic Entropy measure how much the model's output varies, OC measures whether the model's reasoning steps cohere. It asks: "If I break this problem down and put it back together, do I get the same answer?"
• Robustness: The signal is robust to the choice of correctness scorer (validated via LLM-judge and alternative scorers) and holds across models ranging from 4B to 671B parameters, as well as both open-weights and closed-source systems.
• Practicality: OC is a label-free, inference-time signal that requires no ground-truth data for deployment, making it suitable for real-time confidence estimation and selective prediction in production environments.
• Limitations: The authors note that OC is a necessary but not sufficient condition for correctness (false positives occur when the model is consistently wrong; false negatives occur when the direct path is correct but the decomposed path fails). Therefore, they recommend deploying OC as part of a multi-signal stack rather than a standalone gate.

The work positions the Questions Operad not just as a descriptive vocabulary, but as a generative framework for deriving new diagnostic signals, suggesting future directions involving deeper trees, branching decompositions, and algebras over other quantities (e.g., confidence, cost).