💬 NLP

Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

This paper introduces ExtractConf, a cross-domain confidence engine that significantly improves the reliability of LLM-based document field extraction by fusing signals from two structurally distinct extraction strategies (a field-guided "Hunter" and a document-guided "Mapper") with image and layout features to effectively distinguish trustworthy extractions from hallucinations, thereby enabling safe human-in-the-loop automation.

Original authors: Nitesh Kumar

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nitesh Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy factory that processes thousands of business documents every day, like invoices and receipts. You've hired a super-smart AI robot (an LLM) to read these documents and pull out specific numbers, like "Total Amount" or "Tax ID."

The problem? Sometimes, the robot makes a mistake. It might confidently write down a number that doesn't exist, or it might read a smudge on the paper as a zero. In a high-stakes factory, a "silent mistake" (where the robot gives a wrong answer but acts like it's sure) is worse than the robot just saying, "I can't read this."

This paper introduces a new system called EXTRACTCONF. Think of it not as a better reader, but as a super-smart quality control inspector that stands next to the robot to decide: "Can we trust this answer, or should we send it to a human to check?"

Here is how it works, broken down into simple concepts:

1. The "Two-Headed" Strategy (Hunter vs. Mapper)

Most systems ask the AI one question: "What is the Total Amount?" If the AI guesses, it might be wrong.

EXTRACTCONF asks the AI to read the document twice, but in two completely different ways:

The Hunter: This is like a detective looking for a specific suspect. It is told, "Find the 'Total Amount' field." Because it is under pressure to find that specific slot, it might accidentally "hallucinate" (make up) a number if the field is blurry or missing.
The Mapper: This is like a tourist looking at a map. It is told, "Scan the whole document and tell me what important numbers you see." It only reports what it actually sees. If the "Total Amount" is missing or blurry, the Mapper stays silent.

The Magic: If the Hunter says "The total is $500" but the Mapper says "I don't see a total," the system knows something is wrong. If they both agree on "$500," the system feels much more confident. This "disagreement" is a huge clue that the paper is hard to read or the answer is tricky.

2. Looking at the Paper, Not Just the Robot

The paper argues that the robot's confidence (how sure it feels) is often a lie. If the paper is blurry, the robot might still feel very confident about the wrong answer.

So, EXTRACTCONF doesn't just listen to the robot. It also checks:

The "Camera" (OCR): How clear is the text on the actual image? If the camera sees a smudge, the system knows the answer is risky, even if the robot says it's 100% sure.
The "Map" (Spatial Layout): Where did the robot look? If the Hunter looked at the top-left corner and the Mapper looked at the bottom-right, they are looking at different things. That's a red flag.
The "Texture" (Image Quality): Is the specific area of the paper where the number should be blurry or faded?

3. The Final Verdict

All these clues (the two different readings, the camera quality, the location, and the robot's internal confidence) are fed into a traffic light system.

Green Light: The robot and the inspector agree, the paper is clear, and the location makes sense. Automate it! (Send it to the computer).
Red Light: The robot and inspector disagree, or the paper is blurry. Stop! (Send it to a human to check).

What Did They Find?

The researchers tested this on thousands of real invoices. Here is what happened:

Old Way: If they just trusted the robot's "confidence score," the system would try to automate almost everything, including the mistakes. It was like a traffic light that was stuck on green.
New Way (EXTRACTCONF): By using the "Two-Headed" strategy and checking the paper quality, they could filter out the bad answers.
- When they let the system automate the "Green Light" answers, 99.1% of them were correct.
- Without this system, the robot was only correct about 73.3% of the time.
- They reduced the risk of making a mistake by 70% compared to the old methods.

The "Zero-Shot" Surprise

The coolest part is that they trained this system on invoices, and then tested it on receipts (a totally different type of paper) without teaching it anything new. It worked just as well. This proves that the system isn't just memorizing what an invoice looks like; it's actually learning how to tell if any document is hard to read.

In a Nutshell

EXTRACTCONF is a safety net. It doesn't try to make the AI smarter at reading; instead, it builds a smarter judge that knows when the AI is guessing and when it's actually seeing the truth. It saves money by automating the easy stuff and sending the tricky stuff to humans, ensuring that no wrong numbers slip through the cracks.

Technical Summary: Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

Problem Statement

In high-stakes document processing pipelines (e.g., financial reconciliation, compliance), the primary challenge is not merely extraction accuracy but reliable confidence estimation. An extraction that is "silently wrong" poses a greater risk than one that is visibly absent. Current methods for estimating confidence in Large Language Model (LLM) extractions—specifically token-level log-probabilities, verbalized self-assessment, and multi-sample self-consistency—fail to provide reliable separation between trustworthy and untrustworthy outputs. These methods often collapse into "all-positive" classifiers at practical thresholds because they measure confidence in the generated token sequence rather than the grounding of the extraction in the source document. Consequently, a model may confidently transcribe unreadable OCR noise or hallucinate values for absent fields, yielding high log-probabilities for incorrect answers.

Methodology: The EXTRACTCONF Architecture

The paper proposes EXTRACTCONF, a cross-domain, field-agnostic confidence engine that constructs reliability signals from multiple heterogeneous sources. The core innovation is a Dual-Call Hunter–Mapper design that reads the same document twice using structurally asymmetric prompts to induce different failure modes:

Hunter Call (Field-Guided): Receives a prompt to extract a specific field $f$ under schema-slot completion pressure. It tends to hallucinate plausible values for ambiguous or absent fields to satisfy the schema.
Mapper Call (Document-Guided): Receives a prompt to holistically scan the document and surface candidate values based on what is actually present, without being anchored to a specific field. It tends to miss fields that are visually non-salient but remains reliable when it does surface a value.

The disagreement between these two calls serves as an independent reliability signal. If Hunter reports a value while Mapper is silent, or if their values clash, it indicates genuine extraction ambiguity.

Feature Fusion:
EXTRACTCONF fuses 40 features across five signal groups into a gradient-boosted binary classifier (CatBoost):

LLM Internal Uncertainty: Token-level statistics (mean, min, percentiles) of log-probabilities and Shannon entropy from both calls.
OCR Grounding & Value Confidence: Region-level confidence scores derived from matching extracted values and labels against the raw OCR token list, including "value-in-OCR" match indicators and candidate competition metrics.
Cross-Call Agreement: Exact match indicators and similarity scores (Levenshtein/FuzzyWuzzy) between Hunter and Mapper outputs.
Spatial Neighbourhood & Centroid: Features derived from the spatial bounding boxes of attended regions, including centroid distance (divergence between calls) and Normalised Laplacian sharpness (local image quality).
Field Type Encoding: One-hot encoding of inferred field types (numeric, date, etc.) to enable type-conditional calibration.

Post-Processing:
The system supports post-hoc recalibration using Isotonic Regression (M7) or Lasso-regularised logistic regression (M8) to improve calibration metrics (ECE, Brier score) without altering the base model weights.

Key Contributions

Dual-Call Asymmetry: Introduces a Hunter–Mapper architecture where two structurally different LLM calls approach the same document, generating failure modes that are complementary and informative when they disagree.
Multi-Signal Fusion: Demonstrates that fusing cross-call disagreement, LLM internal uncertainty, OCR quality, and spatial layout outperforms single-signal approaches.
Baseline Failure Analysis: Shows that standard baselines (logprob-mean, verbalized confidence, and 5x self-consistency) fail as routing signals, with self-consistency incurring 5x API costs for negligible improvement in routing quality (AURC).
Document Quality Dominance: Establishes that document-side signals (OCR confidence, image sharpness) are primary predictors of correctness, often outperforming model-internal uncertainty metrics.
Zero-Shot Generalization: Proves that reliability signals generalize across document domains (e.g., from invoices to receipts) via lightweight Lasso recalibration without retraining the core classifier.

Experimental Results

Evaluated on DocILE (55-field invoices with a 26% natural failure rate) and CORD (receipts) for zero-shot transfer:

Discrimination (AUC): EXTRACTCONF achieves 0.928 ROC AUC, significantly outperforming logprob-mean (0.705) and self-consistency (0.744).
Routing Quality (AURC): The system reduces the Area Under the Risk-Coverage Curve (AURC) by 70% compared to logprob-mean (0.043 vs. 0.145).
Operational Accuracy: At 80% coverage (automating the top 80% of predictions), EXTRACTCONF achieves 99.1% accuracy, a 25.8-point improvement over the base rate of 73.3%.
Calibration: Post-hoc recalibration (M7/M8) reduces Expected Calibration Error (ECE) by up to 83%.
Zero-Shot Transfer: On CORD receipts, without retraining, the system achieves 0.858 AUC. Lasso recalibration reduces ECE by 89% and Brier score by 43%, confirming the document-agnostic nature of the signals.

Significance and Claims

The paper argues that extraction errors in document processing are document-caused (e.g., unreadable source material, ambiguous layouts) rather than model-caused. Therefore, relying on token probabilities is structurally flawed because a model can confidently hallucinate based on noise.

EXTRACTCONF's significance lies in enabling a practical human-in-the-loop workflow: automating high-confidence predictions while routing low-confidence cases to human review. The authors claim that the Hunter–Mapper disagreement provides a reliability signal unavailable to single-call or resampling approaches, and that the multi-signal fusion creates a robust, domain-agnostic confidence engine that outperforms existing baselines in both discrimination and calibration, all while maintaining a fixed, low API cost (two calls per document).