Technical Summary: Coherence Under Commitment (CUC)

Problem Statement

Large Language Models (LLMs) deployed in knowledge-intensive domains often exhibit a critical failure mode: vacuous coherence. Standard evaluation metrics prioritize logical consistency (e.g., negation consistency, where a model does not simultaneously affirm a statement and its negation). However, a model can trivially satisfy these consistency constraints by systematically abstaining from making any decisive judgment. By withholding commitment to either entailment or refutation, a model achieves zero logical contradictions while providing zero reasoning utility.

In high-stakes environments (e.g., clinical diagnostics, scientific reasoning), this "safe" behavior is indistinguishable from genuine reliability under standard evaluation. A model that refuses to answer 90% of queries appears "coherent" but is operationally useless. Current evaluation paradigms fail to detect this because they measure coherence in isolation, ignoring the epistemic commitment required for actionable reasoning.

Methodology: Coherence Under Commitment (CUC)

The authors propose Coherence Under Commitment (CUC), a dual-query evaluation paradigm designed to jointly measure logical coherence and epistemic decisiveness.

1. Dual-Query Elicitation Protocol

Instead of a single query, CUC constructs a symmetric pair of queries for every premise $P$ and hypothesis $\phi$:

• $Q_\phi$: "Is $\phi$ logically entailed by $P$?"
• $Q_{\neg\phi}$: "Is $\neg\phi$ logically entailed by $P$?"

To eliminate sampling variance and decoding artifacts, the protocol uses deterministic elicitation. It extracts normalized log-probabilities for binary responses (YES/NO) directly from the model's logits:
$$\mathcal{P}(\phi) = \frac{\exp(\log P(\text{YES}|Q_\phi))}{\exp(\log P(\text{YES}|Q_\phi)) + \exp(\log P(\text{NO}|Q_\phi))}$$
This yields a probability mass for the affirmative response for both the hypothesis and its negation.

2. Core Metrics

The framework defines two primary metrics derived from the probabilities $\mathcal{P}(\phi)$ and $\mathcal{P}(\neg\phi)$:

• Commitment Score ($c(\phi)$):
  $$c(\phi) = \mathcal{P}(\phi) + \mathcal{P}(\neg\phi)$$
  This quantifies the total probability mass allocated to decisive outcomes. A low score indicates the model treats the query as effectively unknown (abstention), regardless of whether it is coherent.

• Negation-Coherence Violation ($v_{neg}(\phi)$):
  $$v_{neg}(\phi) = \max(0, c(\phi) - 1)$$
  This measures the extent to which a model assigns more than unit probability mass to mutually exclusive outcomes. A value $>0$ indicates logical incoherence (simultaneous over-commitment).

Key Insight: The paper establishes the algebraic identity $v_{neg} = \max(0, c - 1)$. This implies that any model with $c(\phi) \leq 1$ will automatically achieve zero violation, regardless of its reasoning quality. Thus, low violation rates can be an artifact of low commitment.

3. 3-Way Decision Framework

To operationalize these metrics, the authors define a decision rule with thresholds $\tau$ (confidence) and $\delta$ (margin):

• TRUE: If $\mathcal{P}(\phi) \geq \tau$ and $\mathcal{P}(\phi) \geq \mathcal{P}(\neg\phi) + \delta$.
• FALSE: If $\mathcal{P}(\neg\phi) \geq \tau$ and $\mathcal{P}(\neg\phi) \geq \mathcal{P}(\phi) + \delta$.
• UNCERTAIN: Otherwise.

This allows the evaluation to report Coverage (fraction of non-UNCERTAIN predictions) and Conditional Accuracy ($Acc_{cov}$), distinguishing between models that are precise but abstain and those that are decisive but contradictory.

Experimental Setup

• Datasets: Evaluated on FOLIO v0.0 (204 examples with formally verified negations) and LOGIQA v2 (304 examples with rule-based negation).
• Models: Four open-weight models (1.1B–3B parameters): TinyLlama-1.1B, Phi-2, Qwen2.5-1.5B, and Qwen2.5-3B.
• Protocol: Deterministic log-probability extraction over YES/NO tokens.

Key Results

1. The Coherence-Commitment Frontier

Experiments reveal a sharp, empirically robust trade-off frontier. No model simultaneously achieves high coverage and low violation.

• Systematic Abstention (Qwen2.5-3B): Achieves the lowest violation rate ($E[v_{neg}] = 0.025$), appearing "best" under coherence-only metrics. However, it has a commitment score of $0.115$ and 7.4% coverage, withholding predictions on over 92% of examples.
• Systematic Overcommitment (TinyLlama-1.1B): Achieves 79.4% coverage but violates probability axioms on 100% of examples ($E[v_{neg}] = 0.698$), assigning $>1.6$ probability mass to contradictory outcomes.
• Intermediate Models: Phi-2 and Qwen2.5-1.5B occupy intermediate positions, exhibiting varying degrees of both abstention and contradiction.

2. Failure of Standard Evaluation

• Accuracy Blindness: Standard 3-way accuracy scores are nearly identical across models (ranging only from 0.343 to 0.441), masking the diametrically opposite failure modes of abstention vs. overcommitment.
• Coherence-Only Trap: A coherence-only evaluation would rank Qwen2.5-3B as the superior model (6.6× lower violation than the next best), incorrectly rewarding a model that provides almost no actionable reasoning.
• Calibration Illusion: Even on the small subset of committed predictions, models show poor calibration (high Expected Calibration Error). The apparent calibration of Qwen2.5-3B is an artifact of its aggressive filtering (selecting only the easiest 7% of examples).

3. Scaling Effects

Scaling Qwen2.5 from 1.5B to 3B parameters resulted in an 83% reduction in commitment and a 23.5-point drop in coverage, with negligible change in accuracy. The larger model learned to hedge (abstain) rather than reason more precisely, a behavior that standard metrics would misinterpret as an improvement in coherence.

Significance and Claims

The paper argues that evaluation of logical reasoning must shift from asking "Is the model coherent?" to "Is the model usefully coherent?"

1. Vacuous Coherence is a Blind Spot: Standard metrics cannot distinguish between a model that reasons correctly and one that simply refuses to answer. CUC exposes this by requiring non-vacuous commitment.
2. The Frontier is Inevitable: There is a fundamental tension between coherence and commitment. Models optimizing for one dimension often degrade the other.
3. Modality Agnosticism: While instantiated on text, the framework is theoretically applicable to multimodal settings (e.g., VQA, robotics) where complementary queries can be posed over images and text.
4. Methodological Necessity: The authors claim that reporting both coherence and commitment is essential for high-stakes deployments where abstention carries real operational costs. CUC provides the first principled framework to quantify this trade-off and prevent the deployment of models that are "coherent" only because they are silent.

The paper concludes that without metrics like CUC, practitioners risk deploying models that appear reliable but are functionally inert in knowledge-intensive domains.