💬 NLP

Pre-Inference Routing for Cost-Efficient Document Field Extraction

This paper proposes a pre-inference routing framework that predicts document difficulty using inexpensive features to dynamically select between cheap and strong extraction models, achieving significant cost reductions (up to 77%) without sacrificing accuracy, but only for specific genres where the cheaper model frequently fails and those failures are predictable.

Original authors: Sreerekha Rajendran

Published 2026-08-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sreerekha Rajendran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive library where you need to sort thousands of different documents every day. Some are simple, like a clean, typed receipt from a coffee shop. Others are messy, like a crumpled, faxed political ad form with blurry ink and torn edges. In the world of artificial intelligence, "models" are the smart computers that read these documents and pull out specific information, like prices or dates. Usually, libraries use one giant, super-smart computer for everything. It's like hiring a world-class detective to find a lost set of car keys in a tidy bedroom. It works perfectly, but it's expensive and overkill.

The big question researchers are asking is: Can we be smarter about who does the work? What if we could quickly glance at a document and say, "Hey, this one is easy; let's use a cheap, fast intern to read it," while saving the expensive detective for the messy, confusing ones? This is called "routing." The goal is to save money without losing accuracy. But there's a catch: if you guess wrong and send a messy document to the cheap intern, you might get the wrong answer. So, how do you know when to switch?

This paper tackles that exact puzzle. The researchers, Sreerekha Rajendran and their team, investigated whether we can predict how "hard" a document is before we even ask the computer to read it. They didn't just guess; they built a system that looks at the document's "vibe"—things like how blurry the photo is, how crowded the text looks, or how many tiny fragments the text is broken into. They call this "pre-inference routing." Think of it like a bouncer at a club who looks at your shoes and your posture to decide if you need a VIP pass (the expensive model) or if you can get in with a regular ticket (the cheap model).

The team tested this idea on five different types of documents, from grocery receipts to political forms. They found that this "bouncer" system works incredibly well, but only under two specific conditions. First, there must be a real difference in difficulty; the cheap model has to actually struggle with some of the documents. Second, the clues that make a document hard must be visible to the bouncer before they start reading.

When both conditions were met, the results were like magic. For messy, degraded political forms (called "ad-buy forms"), the system cut the cost by a massive 77% while keeping the quality almost identical to using the expensive model all the time. For receipts, they saved between 31% and 33%. However, the system hit a wall with other documents. For clean, digital invoices, the cheap model was already so good that there was no money to save. For nutrition labels, the documents were so simple that the cheap model was already at its maximum performance ceiling. In these cases, the "bouncer" couldn't find any reason to switch, and trying to force a switch would have just wasted time.

The researchers also discovered something surprising about the "bouncer" itself. They expected that a complex, engineered system looking at image quality and layout would be the best judge. Instead, they found that a very simple system that just counted words (a "bag-of-words" approach) worked just as well. This suggests that the secret isn't the complexity of the bouncer, but the nature of the documents themselves. If the document type is inherently predictable, a simple bouncer works. If the difficulty is hidden deep inside the meaning of the text (like in some invoices), no amount of looking at the picture will help.

Perhaps the most important lesson is that you can't just build one "universal" bouncer for every library. A system trained on receipts doesn't work on political forms, and a system trained on one set of receipts doesn't work on a different set. The "bouncer" needs to be retrained for every new job. But here is the good news: you don't need a huge team to do this. The paper shows that you can run a tiny, cheap test on a small pile of documents first. If that test shows that the documents are hard enough to need a switch, and that the switch is predictable, then you go ahead. If not, you save your money and stick with the one model.

In the end, this paper doesn't promise a magic wand that solves every problem. Instead, it offers a practical checklist. Before you spend a fortune trying to route your documents, take a small sample, check if the "hard" ones are actually visible, and see if the cheap model really struggles. If the answer is yes, you can slash your costs by up to three-quarters. If the answer is no, don't bother trying to be clever; just use the one model you have. It's a reminder that sometimes, the smartest move is knowing exactly when not to overcomplicate things.

Technical Summary: Pre-Inference Routing for Cost-Efficient Document Field Extraction

Problem Statement
Organizations typically employ a single extraction model for all document types (e.g., invoices, receipts, forms). This "one-size-fits-all" approach is inefficient: it incurs unnecessary costs on easy documents that a cheaper model could handle, while potentially limiting the utility of stronger models for difficult cases. The paper investigates whether it is possible to predict a document's extraction difficulty before inference using low-cost, document-based signals. The goal is to route documents to either a cheaper or a stronger extractor to reduce overall cost while maintaining accuracy within a specific tolerance.

Methodology
The author proposes a pre-inference routing framework where a router selects a single model tier (cheap vs. expensive) for each document based solely on intrinsic features of the document image and its OCR text, without running any extraction model first.

Problem Formulation:
- Let $d$ be a document and $x(d)$ be a vector of 13 pre-inference features.
- Two extractors are available: a cheap tier $s$ and an expensive tier $\ell$ .
- The "difficulty gap" is defined as $g(d) = q_\ell(d) - q_s(d)$ , where $q$ represents field-level F1 scores.
- A document is labeled as requiring the large model ( $y(d)=1$ ) if the gap exceeds a threshold $\tau$ (set to 0.02 F1 points).
- The router estimates the probability $p(d) = P(y(d)=1 | x(d))$ and routes to the large model if $p(d) \ge t$ .
- The objective is to find a threshold $t^*$ that minimizes cost $C(t)$ while ensuring the aggregate quality $Q(t)$ remains within $\delta$ (0.02) of the "always-large" baseline.
Features:
- The system uses 13 interpretable, document-intrinsic features computed from the image, OCR text, or OCR box geometry. These fall into four categories:
  - OCR Quality: Confidence scores, standard deviation, preprocessing stages.
  - Image Quality: Blur (Laplacian variance), contrast, word-height variation.
  - Layout: Line density, crowded line fraction, aspect ratio.
  - Content/Structure: Item density, token counts, token fragmentation.
- Crucially, these features are derived only from the document artifact, never from ground-truth annotations, ensuring strict pre-inference capability.
Experimental Setup:
- Datasets: Five genres were tested: Receipts (CORD, SROIE), Ad-buy forms (DeepForm), Invoices (DocILE), Nutrition labels (POIE), and Registration forms (VRDU).
- Models: The primary experiment used Claude Haiku 4.5 (cheap) and Claude Opus 4.8 (expensive), representing a 5× cost ratio. A secondary experiment used Haiku vs. Sonnet 4.6 (3× ratio).
- Baselines: Comparisons included "always-small," "always-large," a single-feature rule (OCR confidence), and a confidence cascade (run cheap first, escalate if uncertain).

Key Contributions

A Diagnostic Framework: The paper introduces a two-part test to determine if routing is viable for a specific dataset:
1. Headroom: Is there a significant fraction of documents where the cheap model fails (gap > $\tau$ )?
2. Predictability: Can pre-inference features predict these failures (high AUC)?
  The author demonstrates that a small labeled pilot can accurately predict whether routing will yield savings before full deployment.
Genre-Specific Success: Routing is shown to be effective only when both conditions are met. It succeeded on Receipts (31–33% cost reduction) and Degraded Ad-buy Forms (77% cost reduction with the 5× model pair, 65% with the 3× pair). It failed on Invoices (features could not predict semantic difficulty) and Nutrition Labels (cheap model was already near ceiling, leaving no headroom).
Router Simplicity vs. Interpretability: The author found that a simple bag-of-words text router performed comparably to their engineered feature set. They argue that the primary constraint is the genre (the document type), not the router design. They retain the engineered features not for superior performance, but for their interpretability, which helps explain why certain genres are routable (e.g., image degradation, layout complexity).
Deployment Specificity: The router does not generalize across datasets, even within the same genre (e.g., CORD to SROIE transfer failed). Therefore, the router must be retrained on a small labeled sample for each new deployment.
Cost Efficiency vs. Cascades: Under full-document re-extraction, pre-inference routing is strictly cheaper than confidence-based cascades. Cascades incur the cost of the cheap model on every document before deciding to escalate, whereas pre-inference routing avoids the cheap call entirely for documents routed directly to the expensive tier.

Results

Receipts (CORD/SROIE): The router achieved a held-out AUC of ~0.71. It reduced costs by 33% (CORD) and 31% (SROIE) while keeping quality within 0.02 F1 of the always-large baseline.
DeepForm (Ad-buy forms): This non-receipt genre showed the strongest results. With the 5× cost ratio (Haiku/Opus), the router achieved a held-out AUC of 0.916 and reduced costs by 77% while maintaining quality within tolerance. With the 3× cost ratio (Haiku/Sonnet), the AUC was 0.845 and savings were 65%.
Negative Cases:
- Invoices: High headroom (67% of docs needed the large model) but low predictability (AUC 0.52), indicating the difficulty is semantic and invisible to visual/OCR features.
- Nutrition Labels: High predictability but low headroom (16% needed the large model), as the cheap model already performed near ceiling.
Transferability: Cross-genre and cross-dataset transfer failed (AUC ~0.55), confirming the need for per-deployment retraining. However, the approach generalized across model pairs (5× and 3× cost ratios) when retrained on the new pair's gap distribution.

Significance and Claims
The paper modestly claims that the primary contribution is the diagnostic rather than a universal routing algorithm. The author asserts that pre-inference routing is not a one-size-fits-all solution; its viability is contingent on the specific characteristics of the document genre.

The significance lies in providing a practical, low-cost method to determine if routing is worth the engineering effort for a new dataset. By running a small pilot to measure "headroom" and "feature predictability," organizations can avoid deploying routing systems that will not yield savings (e.g., on clean digital invoices). The paper concludes that when the two diagnostic conditions are met, pre-inference routing offers substantial cost reductions (up to 77% with a 5× cost ratio) without sacrificing quality, outperforming traditional cascade approaches by eliminating redundant cheap-model calls.

Technical Summary: Pre-Inference Routing for Cost-Efficient Document Field Extraction

More like this