💬 NLP

Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence

The paper introduces DoublyCal, a framework that enhances the accuracy and reliability of black-box LLMs by employing a lightweight proxy model to generate knowledge graph evidence with calibrated confidence, thereby enabling the LLM to produce well-calibrated predictions that trace back to the uncertainty of the supporting evidence.

Original authors: Yuyin Lu, Ziran Liang, Yanghui Rao, Wenqi Fan, Fu Lee Wang, Qing Li

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuyin Lu, Ziran Liang, Yanghui Rao, Wenqi Fan, Fu Lee Wang, Qing Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, but sometimes overconfident, librarian (the Large Language Model or LLM) a tricky question. The librarian is great at talking, but they often make things up (hallucinate) or are too sure of themselves even when they are wrong.

To fix this, people started giving the librarian a reference book (a Knowledge Graph) to check facts. But there's a new problem: What if the reference book itself has missing pages or confusing entries? The librarian might read a half-finished sentence and confidently declare, "I know the answer!" when they actually don't.

The paper "Double-Calibration" introduces a new system called DoublyCal to solve this. Think of it as a two-step quality control process that ensures the librarian is only confident when they should be.

Here is how it works, using simple analogies:

The Problem: The "Confident but Wrong" Librarian

Currently, if you ask a librarian, "Who is Snoopy's brother?" and they find a note saying "Belle is a sibling," they might confidently say, "It's Belle!" even if the note is incomplete or ambiguous. They don't know how sure that note is. They just guess and sound very certain.

The Solution: The "Double-Check" System (DoublyCal)

The authors created a system that acts like a smart assistant who checks the reference book before the librarian gives the final answer. This assistant does two specific things:

Step 1: Calibrating the Evidence (The "Fact-Checker")

Before the librarian even looks at the question, a lightweight "proxy" model (the assistant) looks at the reference book (Knowledge Graph).

The Analogy: Imagine the assistant is a detective looking at a clue. The clue says, "Snoopy has a brother named Spike."
The Innovation: The assistant doesn't just hand over the clue; it attaches a confidence score to it, like a weather forecast.
- "This clue is 100% reliable." (High confidence)
- "This clue is shaky; maybe it's true, maybe not." (Low confidence)
How they do it: They use a mathematical trick called "Bayesian smoothing." Think of this as a safety net. If the reference book is sparse (has few facts), the math prevents the assistant from saying "100% sure" just because it found one tiny piece of evidence. It keeps the confidence score honest.

Step 2: Calibrating the Reasoning (The "Smart Librarian")

Now, the assistant hands the clue and its confidence score to the main librarian (the LLM).

The Analogy: The librarian reads the clue: "Snoopy's brother is Spike [Confidence: 1.0]" and "Snoopy's brother is Belle [Confidence: 0.5]."
The Result: Because the librarian sees the confidence scores, they can weigh the answers. They realize, "Okay, the 'Spike' clue is very strong, but the 'Belle' clue is weak."
The Outcome: The librarian gives the final answer ("It's Spike") and says, "I am very confident about this." If the clues were weak, the librarian would say, "I'm not sure," instead of guessing confidently.

Why is this "Double" Calibration?

Most other systems only do the second step: they ask the librarian, "How sure are you?" after the librarian has already guessed. But the librarian is bad at judging their own certainty.

DoublyCal does it twice:

First: It calibrates the evidence (the reference book notes) to make sure the facts are trustworthy.
Second: It calibrates the final answer based on those trustworthy facts.

The Results

The paper tested this on difficult trivia questions. They found that:

Accuracy went up: The system got more questions right because it stopped guessing on weak clues.
Confidence went down (in a good way): The system stopped saying "100% sure" when it was actually unsure. It became much better at admitting when it didn't know.
It's cheap: The "assistant" (proxy model) is small and fast, so it doesn't cost a lot of money or time to run.

In a Nutshell

Think of DoublyCal as a quality control manager for AI. Instead of letting the AI guess and then ask, "Am I right?", this system checks the source material first, assigns a "trust score" to every piece of information, and then guides the AI to give an answer that matches that trust score. It stops the AI from being confidently wrong.

Technical Summary: Double-Calibration for Reliable LLMs via Calibrating Knowledge and Reasoning Confidence

1. Problem Statement

Large Language Models (LLMs) suffer from hallucinations and unreliable reasoning, driven by both intrinsic epistemic uncertainty (knowledge gaps) and extrinsic aleatoric uncertainty (data ambiguity). While Knowledge Graph-augmented Retrieval-Augmented Generation (KG-RAG) has emerged as a leading paradigm to improve factual accuracy by retrieving structured evidence, current methods possess a critical flaw: they assume retrieved evidence is always sufficient and certain.

In practice, retrieved evidence can be noisy, incomplete, or ambiguous. Existing KG-RAG approaches fail to quantify the uncertainty of the retrieved evidence itself. Consequently, LLMs may produce confidently stated but incorrect predictions when relying on partial or flawed evidence. Furthermore, existing Uncertainty Quantification (UQ) methods for LLMs predominantly focus on calibrating the final output confidence. These methods often rely on the model's internal states or self-assessment, making them opaque and non-traceable. They cannot distinguish whether uncertainty stems from flawed evidence, reasoning deficiencies, or task difficulty, thereby failing to synergize effectively with KG-RAG to provide a stepwise-calibrated view of the reasoning chain.

2. Methodology: DoublyCal

To address these gaps, the authors propose DoublyCal, a framework built on a novel double-calibration principle. This approach explicitly estimates and makes confidence traceable from the retrieved KG evidence to the final LLM prediction. The framework operates through three core components:

2.1 Bayesian Calibration of KG Evidence

The framework first formalizes KG evidence as constrained relational paths. Unlike simple relational paths or subgraphs, these paths consist of a core relational path augmented with an optional constraint derived from the candidate answer's neighborhood, balancing informativeness and interpretability.

To estimate a statistically grounded confidence for each evidence path, the authors employ a Beta-Bernoulli model:

The correctness of a candidate answer given evidence is modeled as a Bernoulli distribution with parameter $p$ .
To account for KG incompleteness and sparse grounding, a conjugate Beta prior ( $p \sim \text{Beta}(\alpha, \beta)$ ) is imposed.
The calibrated confidence score is calculated as the posterior mean, blending empirical accuracy with prior belief (e.g., using a Jeffreys prior $\alpha=\beta=0.5$ ). This prevents overconfidence in cases where the number of retrieved candidates is small.

2.2 Proxy Model for Evidence Generation and Calibration

Since the Bayesian calculation is retrospective, a lightweight LM-based Evidence Generator (Proxy Model) is trained to perform prospective evidence retrieval and confidence estimation during inference. The training involves two stages:

Supervised Fine-Tuning (SFT): The proxy is trained to generate KG evidence paths and their corresponding Bayesian confidence scores, mimicking the statistical signal.
Reinforcement Learning (RL): The proxy is further refined using Group Relative Policy Optimization (GRPO). The reward function jointly optimizes for inferential quality (matching gold evidence) and calibration alignment (minimizing the difference between predicted confidence and the Bayesian ground truth).

2.3 LLM Reasoning with Calibrated Evidence

During inference, the trained proxy generates a set of candidate evidence-confidence pairs. These pairs are verbalized and integrated into the prompt for a black-box primary LLM. The LLM is instructed to reason over these paths and their associated confidence scores to produce a final answer and a well-calibrated prediction confidence. This establishes a traceable chain where the final confidence is anchored in the verifiable, calibrated evidence rather than the LLM's internal state alone.

3. Key Contributions

Double-Calibration Principle: The paper establishes a new paradigm requiring explicit confidence calibration for both the retrieved KG evidence and the final LLM predictions, moving beyond single-stage output calibration.
DoublyCal Framework: A practical implementation utilizing a Bayesian-calibrated proxy model to provide black-box LLMs with KG evidence accompanied by evidence confidence.
Empirical Validation: Demonstrations that DoublyCal significantly improves both accuracy and confidence calibration across diverse black-box LLMs on knowledge-intensive benchmarks while maintaining low token costs.

4. Experimental Results

The framework was evaluated on WebQSP and CWQ benchmarks, comparing DoublyCal against Single-Calibration (SingleCal) baselines (standard KG-RAG methods combined with verbalized UQ techniques like Vanilla, CoT, and Self-Probing).

Accuracy and Calibration: DoublyCal (specifically the RL-enhanced version) achieved state-of-the-art performance, leading in Hits, Recall, and F1 scores while significantly reducing Expected Calibration Error (ECE) and Adaptive Calibration Error (ACE). For instance, on WebQSP, RL-DoublyCal reduced ECE to ~3.9% compared to ~18-38% for baselines.
Cost Efficiency: DoublyCal demonstrated superior token efficiency. It outperformed SubgraphRAG with significantly fewer input tokens (approx. 39% of SubgraphRAG's cost) due to the high information density of its constrained relational paths.
Cross-Model Compatibility: The framework was tested on various black-box LLMs (GPT-3.5-turbo, GPT-4o-mini, DeepSeek-V3, Gemini-2.5-flash). DoublyCal consistently decoupled the trade-off between accuracy and reliability, reducing average ECE from 29.4% to 7.6% across models while boosting average F1 from 45.8% to 76.5%.
Ablation Studies: Removing the evidence confidence calibration (SingleCal variant) resulted in a drastic increase in calibration error (ECE rising from ~4% to >20%), confirming that externally calibrated evidence is essential for reliable self-assessment by black-box LLMs.

5. Significance and Claims

The paper claims that DoublyCal offers a principled solution for systematically managing the propagation of uncertainty in KG-augmented LLMs. By grounding confidence in externally verifiable evidence, the framework provides a traceable foundation for the entire reasoning chain, addressing the "black box" nature of current UQ methods.

The authors assert that their work represents a concrete step toward building more reliable and transparent LLM systems. They highlight that while KG-RAG improves factuality, it is the double-calibration of both evidence and reasoning that is necessary to achieve truly reliable reasoning. The framework is presented as a plug-and-play solution that enhances the trustworthiness of diverse black-box LLMs without requiring access to their internal parameters.

Limitations Acknowledged:
The authors note that the robustness of the Bayesian confidence estimator may decline when the KG is extremely sparse. Additionally, the fixed form of constrained relational paths may limit performance on complex questions involving aggregation or comparison, suggesting future work could explore richer evidence forms or agentic planning paradigms.