🤖 AI

Active Learning for Cascaded Object Detection: Balancing Coverage and Uncertainty in Table Extraction Pipelines

This paper introduces a novel active learning framework that adapts Uncertainty Herding for cascaded table extraction pipelines by proposing two pipeline-aware variants, RankFusion and CAPA, which effectively balance coverage and uncertainty to significantly reduce annotation costs while outperforming standard baselines across multiple datasets.

Original authors: Eliott Thomas, Mickael Coustaty, Aurelie Joseph, Gaspar Deloin, Vincent Poulain d'Andecy, Jean-Marc Ogier

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eliott Thomas, Mickael Coustaty, Aurelie Joseph, Gaspar Deloin, Vincent Poulain d'Andecy, Jean-Marc Ogier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot assistant to read and understand business documents like invoices and contracts. These documents are full of tables (rows and columns of data), and the robot needs to do two things to understand them:

Find the Table: First, it has to spot where the table is on the page (like finding a specific box in a messy room).
Read the Table: Second, it has to understand the inside of that box—figuring out which lines are headers, which are columns, and which are rows.

The problem is that teaching a robot to do this is expensive. You have to hire humans to draw boxes around tables and then painstakingly label every single cell inside them. You can't afford to label every document in the world, so you need a smart way to pick just the right ones to teach the robot. This is where Active Learning comes in. It's like a teacher who doesn't just pick random students to quiz, but specifically picks the ones who are struggling the most or who represent a unique type of problem, so the class learns faster with fewer quizzes.

The Problem: A Two-Step Relay Race

The paper points out a flaw in how we usually teach these robots. Most "smart pickers" treat the robot as a single brain. But in reality, this is a relay race.

Runner 1 (Table Detection): Finds the table.
Runner 2 (Table Structure): Reads the table.

If Runner 1 drops the baton (misses the table), Runner 2 never gets a chance to run. No matter how good Runner 2 is, if they never see the table, they can't learn. Conversely, if Runner 1 is great but Runner 2 is confused, the whole race fails.

Standard "smart pickers" don't realize this connection. They might pick a document that is perfect for teaching Runner 2, but if Runner 1 can't even find the table in that document, the lesson is wasted.

The Solution: A New Strategy for the Relay

The authors, Eliott Thomas and his team, took an existing smart-picking method called UHerding (which balances "covering new ground" with "focusing on confusion") and upgraded it for this two-step relay. They created two new versions:

1. RankFusion: The "Double-Check" Strategy

Imagine you are looking for a lost item.

Old way: You look at the whole room (the document) to see where you haven't looked yet.
RankFusion way: You look at the whole room AND you zoom in on the specific drawer (the table) to see if you've missed anything there.

This method picks documents that are interesting for both finding the table and understanding the table's inside. It's like saying, "Let's pick a document that is weird enough to teach us how to find tables, but also complex enough to teach us how to read the numbers inside."

2. CAPA: The "Team Captain" Strategy

This is the most advanced version. CAPA acts like a smart team captain who watches the race in real-time.

The Gating Mechanism: If the captain sees that Runner 1 (Table Detection) is failing hard, the captain says, "Stop! Don't waste time teaching Runner 2 yet. Let's focus all our energy on helping Runner 1 find the tables first." It ignores documents where the table is missing because teaching the second step is useless there.
Dynamic Weighting: If Runner 1 gets good at their job, the captain shifts focus to help Runner 2. It constantly adjusts the training plan based on which runner is currently the "bottleneck" (the weak link).

What They Found

The team tested these strategies on four different types of documents (academic papers, financial reports, invoices, and mixed business docs).

The Result: Both new strategies (RankFusion and CAPA) were better than the old methods. They helped the robot learn faster and more accurately with the same amount of human labeling effort.
The Trade-off:
- RankFusion was the "high risk, high reward" player. It sometimes got the best scores, but its performance varied a lot depending on the document type.
- CAPA was the "consistent champion." It wasn't always the absolute fastest, but it was the most reliable. It never did poorly, making it the safest bet if you aren't sure what kind of documents you are dealing with.

The Big Takeaway

The paper concludes that when you have a multi-step process (like a relay race), you can't just treat it as one big task. You have to understand that if the first step fails, the second step doesn't matter.

By building a system that knows which step is currently struggling and focuses its teaching efforts there, you can train powerful AI systems much more efficiently. It's not just about picking the "hardest" examples; it's about picking the examples that fix the specific broken link in your chain.

Technical Summary: Active Learning for Cascaded Object Detection in Table Extraction

Problem Definition
Table extraction from business documents typically relies on a cascaded pipeline comprising two stages: Table Detection (TD), which localizes tables within a document page, and Table Structure Recognition (TSR), which recovers the internal layout (rows, columns, headers) of the detected regions. Constructing training sets for this pipeline is costly, particularly for TSR, which requires fine-grained structural annotations compared to the single bounding box required for TD. While Active Learning (AL) offers a principled approach to reduce annotation burdens by selecting the most informative samples, existing strategies are primarily designed for single-model tasks. They fail to account for the structural dependencies in cascaded architectures, where TSR performance is strictly bounded by the quality of upstream TD predictions. Furthermore, the two stages differ in input scope (full page vs. cropped table) and task complexity, suggesting they may require different training signals and do not improve at identical rates as data is added.

Methodology
The authors adapt Uncertainty Herding (UHerding), a hybrid sampling method that balances coverage (diversity) and uncertainty, to the cascaded table extraction setting. The core adaptation involves modifying the coverage and uncertainty components to handle multiple stages with distinct embedding spaces.

Generalized UHerding Adaptation:
- Embedding: Instead of natural image features, a pretrained document image encoder (DiT) is used to generate document-level representations.
- Uncertainty: A combined uncertainty signal is derived from the confidence scores of both the TD and TSR models. If predictions fall below a threshold, uncertainty defaults to 1.
- Calibration: To handle the lack of standard Expected Calibration Error (ECE) formulations for object detection, the authors use Spearman correlation between uncertainty and prediction error (GriTS-Con F1) on a validation set to dynamically adjust the influence of uncertainty versus coverage.
Pipeline-Aware Extensions:
Recognizing that treating stages symmetrically is suboptimal, two extensions are proposed:
- RankFusion: This approach introduces dual-manifold coverage. It employs a second encoder (reusing TSR features) to generate table-level representations. Coverage is computed independently in both document-level and table-level spaces. Instead of summing scores (which requires normalization across different scales), the method converts coverage gains into ranks and fuses them using Reciprocal Rank Fusion (RRF). It retains a shared uncertainty signal and calibration parameter.
- CAPA (Calibrated and Pipeline-Aware): This method further relaxes symmetry assumptions by explicitly modeling the TD-to-TSR dependency:
  - Adaptive Weighting: Fusion weights ( $\beta$ ) are dynamically adjusted based on the validation performance of each stage. If TD is the bottleneck, the weight shifts toward the TD manifold; if TD is reliable, weight shifts to TSR.
  - Stage-Dependent Gating: TSR uncertainty is suppressed for documents where TD confidence is low, preventing the selection of samples for structural annotation when the table itself is not detected.
  - Per-Task Calibration: Separate calibration parameters ( $\tau_{td}, \tau_{tsr}$ ) are computed for each stage based on their specific uncertainty-error correlations.

Experimental Setup
The methods were evaluated on four datasets: two public (PubTables-1M, FinTabNet) and two private (invoice-int, business-int). The scope was restricted to single-table documents without spanning cells to isolate the sampling strategy's effect. Experiments utilized D-FINE-X detectors for both stages, retrained from scratch at each AL iteration. The evaluation covered annotation budgets ranging from 71 to 500 documents, with results averaged over 10 random seeds. Performance was measured using the normalized Area Under the Curve (AUC) of median learning curves based on the GriTS-Con F1 metric.

Key Results

Generalization: The adapted UHerding strategy outperformed single-signal baselines (pure coverage via MaxHerding and pure uncertainty via Confidence) across all datasets, confirming that hybrid sampling generalizes well to cascaded detection.
Pipeline-Aware Performance:
- RankFusion achieved the highest mean AUC (65.74%), particularly excelling on the business-int dataset where dual-manifold coverage amplified diversity signals. However, it exhibited higher variance and performed less consistently on PubTables-1M, where high model performance made uncertainty the dominant signal.
- CAPA emerged as the most consistent strategy, outperforming the adapted UHerding baseline on three of the four datasets. It achieved the best results on FinTabNet (70.52% AUC) and maintained a high floor (never ranking below 4th), making it a robust choice when the specific bottleneck of a target dataset is unknown.
Bottleneck Dynamics: Analysis revealed that the performance bottleneck (TD vs. TSR) is dataset-dependent. CAPA's ability to reallocate coverage toward the bottleneck stage (e.g., shifting focus to TSR when detection is reliable) was visually confirmed through embedding space coverage analysis.

Significance and Contributions
The paper claims three primary contributions:

First Adaptation of UHerding: It presents the first application of Uncertainty Herding to a cascaded object detection pipeline, moving beyond single-model image classification.
Pipeline-Aware Extensions: It introduces RankFusion and CAPA, which explicitly exploit the inter-stage dependency between detection and structure recognition. CAPA, in particular, demonstrates that dynamically gating uncertainty and weighting coverage based on stage reliability yields consistent improvements.
Empirical Validation: Through extensive evaluation across diverse domains and annotation budgets, the work establishes a trade-off between expected gain (RankFusion) and consistency (CAPA).

The authors conclude that the bottleneck between pipeline stages shapes strategy preferences more than the acquisition function itself. They suggest that this framework is generalizable to any cascaded detection task with sequential dependencies, where the "where" of the failure matters as much as the "how" of sample selection. The work does not claim to solve complex multi-table or spanning-cell scenarios, noting these as limitations for future investigation.