Differentiable Conformal Training for LLM Reasoning Factuality

1. Problem Statement

Large Language Models (LLMs) frequently "hallucinate" (generate confident but false information), limiting their reliability in high-stakes applications. While Conformal Prediction (CP) offers a way to provide statistical guarantees on error rates (e.g., ensuring hallucination rates stay below a user-specified $\alpha$), existing methods for LLM factuality face a critical trade-off:

• Independent Claim Filtering: Standard CP treats claims independently. This ignores logical dependencies in multi-step reasoning (e.g., a conclusion is only valid if its premises are valid).
• Coherent Factuality (CF): A recent method (Rubin-Toles et al., 2025) addresses dependencies using Approximate Deducibility Graphs (ADGs), where claims are nodes and edges represent logical dependencies. However, CF relies on hand-crafted, frequency-based scoring (self-consistency).
• The Core Limitation: At high reliability levels (low $\alpha$), CF's rigid, non-differentiable scoring forces the removal of up to 60% of true claims to maintain coverage guarantees. The method cannot be optimized end-to-end because its discrete operations (thresholding, ancestor coherence checks, and argmax selection) are not differentiable.

2. Methodology: Differentiable Coherent Factuality (DCF)

The authors introduce Differentiable Coherent Factuality (DCF), a framework that relaxes the discrete operations of CF into differentiable surrogates, enabling gradient-based optimization of the scoring function while provably recovering the original algorithm's guarantees.

Key Technical Components

DCF mirrors the CF pipeline but replaces discrete steps with smooth approximations:

1. Soft Membership (Relaxing Thresholding & Coherence):

   • Threshold Filtering: Instead of a hard indicator $\mathbb{1}\{r_v \le \tau\}$, DCF uses a sigmoid function $\sigma(\frac{\tau - r_v}{T_p})$ to output a soft retention probability $p_{v,\tau}$.
   • Ancestor Coherence: In CF, a node is retained only if it and all ancestors are valid. DCF relaxes this logical "AND" into a weighted geometric mean of probabilities:
     $$\log q_{v,\tau} = \frac{\sum_{u \in Anc(v) \cup \{v\}} w_u \log p_{u,\tau}}{\sum w_u}$$
     This ensures that if any ancestor has a low retention probability, the node's probability drops toward zero.

2. Differentiable Calibration (Relaxing the Nonconformity Score):

   • The goal is to find the largest threshold $\tau$ such that the filtered subgraph contains no false claims.
   • Violation Measurement: A violation score $V_\tau$ is computed based on the probability of retaining false claims.
   • Soft Supremum: Instead of a hard $\sup$, DCF uses a softmax over a utility function $s_\tau = \lambda \cdot \tau - V_\tau$. This selects the threshold that maximizes the value while minimizing violations.
   • Soft Quantile: The final calibration threshold $\hat{\tau}_\alpha$ is computed using a differentiable soft quantile operator (Grover et al., 2019) instead of sorting.

3. Differentiable Prediction:

   • At test time, the method selects the best threshold below $\hat{\tau}_\alpha$. DCF relaxes the constrained $\arg\max$ using a gated softmax, where a sigmoid gate enforces the constraint $\tau < \hat{\tau}_\alpha$.

4. Training Objective:

   • The system is trained end-to-end by simulating the full calibration and prediction pipeline on a split dataset.
   • The loss function maximizes the number of true claims retained ($L_{retention} = -\sum y_{i,v} \cdot q_{i,v}$), backpropagating gradients through the quantile estimation and graph coherence constraints to the scorer parameters $\theta$.

Theoretical Guarantees

The authors prove (Theorems 3.1 and 3.2) that as the temperature parameters ($T_p, \tau_s, \tau_z$) approach their limits (0 or $\infty$), the soft operations converge to the original hard CF algorithm. This ensures that while training uses gradients, the deployed model retains the statistical coverage guarantees of the original discrete CF.

3. Key Contributions

1. Differentiable Relaxation of Graph-Structured CP: The first framework to make Conformal Prediction differentiable for graph-structured reasoning (ADGs), handling coupled discrete operations like ancestor coherence and constrained argmax.
2. End-to-End Optimization: Enables learning scoring functions that combine multiple features (graph metrics, semantic coherence, frequency) to optimize claim retention, rather than relying on hand-crafted heuristics.
3. Theoretical Convergence: Proves that the differentiable surrogate faithfully recovers the original CF algorithm in the limit, ensuring valid statistical guarantees at test time.
4. Empirical Validation: Demonstrates significant improvements in retaining correct claims without sacrificing reliability.

4. Experimental Results

The method was evaluated on two benchmarks: MATH (complex reasoning, 7.3 claims/problem) and FELM (diverse domains, 4.0 claims/problem).

• Performance Gains:
  • MATH: DCF achieved up to 141% improvement in claim retention compared to the frequency-based CF baseline at $\alpha=0.03$ (97% reliability), while maintaining coverage within 0.5 percentage points of the target.
  • FELM: Achieved up to 61% improvement in retention.
• Baseline Comparisons:
  • DCF outperformed Independent Factuality (proving graph structure is necessary).
  • DCF outperformed Boosted Independent (proving that learning alone without graph structure is insufficient).
  • DCF outperformed XGBoost + Conformal (proving that standard classifiers cannot replace the differentiable conformal training pipeline).
• Feature Analysis:
  • SHAP analysis revealed that DCF learns to combine complementary signals. For example, when the standard "frequency" score was 0 (indicating low confidence), graph metrics like reachability and claim index compensated, allowing the model to retain valid claims that baselines would discard.
  • On MATH, graph structure features dominated; on FELM (simpler chains), frequency was more dominant, but DCF still adapted effectively.

5. Significance

This work bridges the gap between formal statistical guarantees and practical usability in LLM systems.

• Solving the Conservatism Problem: Previous conformal methods were too conservative for practical use, discarding too much correct information to guarantee safety. DCF makes these guarantees "usable" by significantly increasing the amount of retained, factual information.
• Learned vs. Hand-Crafted: It demonstrates that for complex reasoning tasks, learned scoring functions that respect logical dependencies are superior to static, hand-crafted heuristics.
• Deployment Ready: Because the final deployment uses the original discrete algorithm (with the learned scorer), the system maintains the rigorous theoretical guarantees required for high-stakes decision-making.

In summary, DCF provides a robust, mathematically grounded method to make LLM reasoning more reliable and informative, enabling the deployment of conformal factuality in real-world applications where both accuracy and information density are critical.