💬 NLP

Calibrated Confidence Estimation for Tabular Question Answering

This paper presents the first systematic study of confidence calibration for tabular question answering, revealing that large language models are severely overconfident on structured data and proposing Multi-Format Agreement—a cost-effective method leveraging serialization variations—to significantly improve uncertainty estimation and calibration performance.

Original authors: Lukas Voss

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lukas Voss

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read assistant (a Large Language Model, or LLM) who is great at reading stories and answering questions about them. You ask them, "Who is the main character?" and they say, "It's Harry," with 100% confidence. You trust them because they sound sure.

Now, imagine you give this assistant a spreadsheet full of financial data instead of a story. You ask, "What was the total profit in 2023?" They say, "$5 million," again with 100% confidence.

The Problem:
This paper discovers that when these AI assistants look at spreadsheets (tables), they are terrible at knowing when they are wrong. They are like a confident tourist who gets lost in a foreign city but insists, "I'm absolutely sure this is the way to the museum!" even when they are walking in the opposite direction.

In the world of text, these AIs are only slightly overconfident. But with tables, they are severely overconfident. They think they are right 99% of the time, but they are actually only right about 65–75% of the time. If you built a financial app on this, you'd lose money because the AI would never admit it's unsure.

The Two Ways to Check Confidence

The researchers tested five different ways to ask the AI, "Are you sure?" They found a massive split between two types of methods:

1. The "Self-Reflection" Method (The Bad Idea)

How it works: You ask the AI, "You just gave me an answer. On a scale of 1 to 100, how sure are you?" or "Is your answer correct?"
The Analogy: This is like asking a student who just failed a math test, "How confident are you that you got the right answer?" The student says, "100% sure!" even though they got it wrong.
The Result: This method failed miserably. The AI's self-assessment was basically a random guess. It couldn't tell the difference between a right answer and a wrong one.

2. The "Perturbation" Method (The Good Idea)

How it works: Instead of asking the AI to judge itself, you change the question slightly and see if the answer stays the same.
The Analogy: Imagine you are trying to find a hidden treasure.
- Method A: You ask a guide, "Are you sure this is the spot?" (Self-reflection).
- Method B: You ask the guide to look at the spot from the North, then the South, then the East, then the West. If they say "Treasure here!" from all four angles, you trust them. If they say "Treasure here!" from the North but "No, it's over there!" from the South, you know they are confused.

The Star of the Show: Multi-Format Agreement (MFA)

The paper introduces a clever new trick called Multi-Format Agreement (MFA). This is the "gold standard" they found for tables.

How it works:
Tables can be written in many different "languages" (formats) that mean the exact same thing:

Markdown (like a simple text list)
HTML (like a web page)
JSON (like a code dictionary)
CSV (like a comma-separated list)

The researchers took the same table and asked the AI to answer the question four times, once for each format.

If the AI gives the same answer for all four formats, it's probably right. (High Confidence)
If the AI gives different answers depending on the format, it's likely just guessing based on how the words look, not understanding the data. (Low Confidence)

Why is this special?

It's Cheaper: It costs 20% less to run than other "guessing" methods because it doesn't need to generate random variations; it just swaps the file format.
It's Deterministic: It's not a roll of the dice. If you run it twice, you get the exact same result.
It Works: It fixed the AI's confidence. Instead of being wildly overconfident, the AI's confidence scores actually matched its accuracy.

The "Magic" Fix: Structure-Aware Recalibration

Even with the best methods, the AI's confidence numbers can still be a bit off (e.g., it says 80% sure, but it's actually 70% right).

The researchers added a "smart filter" called Structure-Aware Recalibration.

The Analogy: Imagine a weatherman who always says "80% chance of rain." But you notice he only says that when it's a small, easy-to-predict drizzle. When it's a massive, complex storm, he still says 80%.
The Fix: The researchers taught the system to look at the complexity of the table.
- "This is a tiny table with 3 rows and a simple question." -> Okay, 80% confidence is probably accurate.
- "This is a huge table with 500 rows and a complex math question." -> Wait, 80% confidence is too high! Let's lower it to 60%.

By looking at the "shape" of the data (how big the table is, how hard the question is), they could fine-tune the confidence scores to be much more accurate.

The Big Takeaway

Don't trust the AI's voice: When an AI answers a question about a spreadsheet, do not believe it when it says, "I'm 100% sure." It is almost certainly lying (unintentionally).
Change the format to test it: The best way to know if an AI is right about a table is to ask it the same question in four different file formats. If it changes its mind, it's confused.
Context matters: A simple lookup is easier than a complex calculation. The AI needs to be told, "Hey, this is a hard question, so don't be so confident."

In short: To make AI trustworthy with data, stop asking it "Are you sure?" and start asking it, "Does the answer change if I show you the data differently?"

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed for Tabular Question Answering (Tabular QA) in high-stakes domains like finance and business intelligence. However, a critical reliability gap exists:

Severe Overconfidence: LLMs exhibit significantly higher overconfidence on structured tabular data compared to textual QA. While textual QA models typically show a Smooth Expected Calibration Error (smECE) of 0.10–0.15, tabular models show smECE values between 0.35 and 0.64.
Lack of Systematic Study: Prior research has extensively studied LLM calibration on text or text-to-SQL tasks, but no systematic comparison of general-purpose confidence estimation methods exists for direct tabular QA.
Unique Uncertainty Sources: Tabular tasks introduce specific uncertainty sources absent in text, including schema linking ambiguity, aggregation errors (SUM vs. COUNT), numerical precision issues, and format sensitivity (where the same table serialized differently yields different answers).

2. Methodology

The study evaluates five confidence estimation methods across five frontier LLMs (GPT-4o, GPT-4o-mini, Gemini 2.5 Flash, Llama-3.3-70B, DeepSeek-V3) on two benchmarks: WikiTableQuestions (WTQ) and TableBench.

The methods are categorized into two paradigms:

Self-Evaluation:
- Verbalized Confidence: The model outputs an answer and a confidence score (0–100%) in a single pass.
- P(True): A two-pass method where the model generates an answer, then is asked "Is your answer correct?" to self-assess probability.
Perturbation (Input/Output Variation):
- Self-Consistency (SC): Sampling $N=5$ responses at temperature $\tau > 0$ ; confidence is the agreement rate.
- Semantic Entropy (SE): Clustering $N=5$ sampled responses by semantic equivalence; confidence is $1 - \text{Entropy}$ .
- Multi-Format Agreement (MFA): A novel method proposed in this paper. It exploits the unique property of structured data: lossless and deterministic format variation. The same table is serialized into four formats (Markdown, HTML, JSON, CSV) and queried at temperature 0. Confidence is the agreement rate across these formats.

Recalibration: The paper also proposes Structure-Aware Recalibration, an extension of Platt scaling that incorporates structural covariates (table dimensions, column types, query complexity) to improve discrimination.

3. Key Contributions

A. Discovery of the Self-Evaluation vs. Perturbation Dichotomy

The paper establishes a consistent, statistically significant dichotomy across all models and benchmarks:

Self-Evaluation Methods (Verbalized, P(True)): Perform poorly, achieving AUROC between 0.42 and 0.76 (often near random chance, 0.5). They fail to distinguish correct from incorrect answers reliably.
Perturbation Methods (MFA, SC, SE): Perform significantly better, achieving AUROC between 0.78 and 0.86.
Statistical Significance: Paired bootstrap tests with Holm-Bonferroni correction reject the null hypothesis ( $p < 0.001$ ) for the performance gap between the two paradigms.

B. Multi-Format Agreement (MFA)

The paper introduces MFA as a domain-specific instantiation of input-perturbation uncertainty.

Mechanism: It treats format sensitivity as an uncertainty signal. If a model changes its answer based on whether the table is in JSON vs. CSV, it is likely pattern-matching tokens rather than reasoning about structure.
Efficiency: MFA requires 4 API calls per question (one per format), whereas sampling baselines (SC/SE) require 5 calls. This results in a 20% reduction in API cost while matching the performance of sampling methods.
Performance: MFA reduces ECE by 44–63% and achieves a mean AUROC of 0.80 on TableBench.

C. Structure-Aware Recalibration

Standard post-hoc recalibration (e.g., Platt scaling) fixes calibration (ECE) but does not improve discrimination (AUROC). By adding structural covariates (e.g., log rows, query complexity) to the recalibration model, the paper improves AUROC by +10 percentage points, proving that task-specific structural features provide orthogonal signals for reliability.

4. Key Results

Overconfidence: All models are severely overconfident. For example, GPT-4o reports a mean confidence of 99.5% despite an accuracy of 76.2%, resulting in an AUROC of only 0.522 (near random).
MFA vs. Verbalized: On Llama-3.3-70B, MFA improves AUROC from 0.523 (Verbalized) to 0.737.
MFA vs. Sampling: MFA achieves comparable AUROC to Self-Consistency and Semantic Entropy but at a lower cost.
- Llama-3.3-70B: MFA (0.737) vs. SC (0.752) vs. SE (0.753).
Ensemble Synergy: Combining MFA (input perturbation) with Self-Consistency (output perturbation) creates a complementary signal. An ensemble of MFA + SC + SE lifts AUROC from 0.74 (best single method) to 0.82.
Cross-Benchmark Generalization: The findings hold true on the harder TableBench benchmark (accuracy $\le$ 43%), confirming the robustness of the perturbation paradigm.
Gemini Outlier: Gemini 2.5 Flash is the only model where verbalized confidence performs well (AUROC 0.812), but even for Gemini, the P(True) method fails (AUROC drops to 0.507), reinforcing the unreliability of self-evaluation.

5. Significance and Implications

Practical Guidance: The paper definitively advises practitioners against using verbalized confidence or P(True) for tabular QA. Instead, input or output perturbation methods are required for reliable selective prediction (abstention).
Cost-Efficiency: MFA offers the most cost-effective solution for structured data, providing high-confidence estimates with fewer API calls than sampling-based methods.
Theoretical Insight: The study confirms that format sensitivity is a valid and powerful uncertainty signal for LLMs. When a model's answer depends on the serialization format, it indicates a lack of true structural reasoning.
Future Directions: The authors suggest applying conformal prediction to MFA scores for formal coverage guarantees and integrating MFA into tool-routing systems (e.g., deciding between SQL, Python, or direct reasoning).

In conclusion, this paper provides the first systematic evidence that perturbation-based methods, specifically the novel Multi-Format Agreement, are essential for calibrating LLMs on tabular data, outperforming self-evaluation by a wide margin while offering a more efficient path to reliable AI in data-driven applications.