🤖 AI

ConRTF: Edge-Constrained Boundary Distribution Refinement for Realtime TransFormer Table Structure Recognition

ConRTF is a real-time table structure recognition method that introduces an Edge-constrained Fine-grained Localization (EFL) loss to encode structural asymmetry between rows and columns during training, thereby refining boundary distributions and improving accuracy on both public and private datasets without altering the inference pipeline.

Original authors: Eliott Thomas, Tri-Cong Pham, Mickael Coustaty, Aurelie Joseph, Gaspar Deloin, Vincent Poulain d'Andecy, Jean-Marc Ogier, Antoine Doucet

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eliott Thomas, Tri-Cong Pham, Mickael Coustaty, Aurelie Joseph, Gaspar Deloin, Vincent Poulain d'Andecy, Jean-Marc Ogier, Antoine Doucet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to read a messy spreadsheet or a financial report. The computer needs to figure out where the rows end, where the columns begin, and how the data is grouped. This is called Table Structure Recognition (TSR).

Think of a table like a grid of Lego bricks. If you build a wall of bricks, the most important lines are the horizontal ones that separate one row of bricks from the next. If those lines are crooked, the whole wall looks wrong. Similarly, in a table, the horizontal lines define the rows, and the vertical lines define the columns.

The Problem: "One Size Fits All" Doesn't Work

Current AI tools that read tables often treat every line on the page the same way. They look at a horizontal line and a vertical line and say, "Okay, both of these are just edges; let's try to find them equally well."

The authors of this paper argue that this is like trying to paint a long, thin fence by using the same amount of paint on the top and bottom as you do on the sides. It's inefficient.

For a Row: The top and bottom edges are the "bosses." They define the row. The left and right edges are just the "fence posts" marking the end.
For a Column: The left and right edges are the "bosses." The top and bottom are just the fence posts.

If the AI gets the "boss" edges slightly wrong, the whole table structure falls apart. But if it gets the "fence post" edges slightly wrong, the table still looks mostly okay.

The Solution: ConRTF (The "Smart Painter")

The paper introduces a new method called ConRTF. Think of it as a smart painter who knows exactly which parts of the fence need the most attention.

The Special Paint (EFL Loss): The core innovation is a new rule for training the AI, called Edge-constrained Fine-grained Localization (EFL).
- When the AI is learning to draw a row, the EFL rule says: "Hey, pay extra attention to the top and bottom lines! Make them perfect. The side lines can be a little looser."
- When the AI is learning to draw a column, the rule flips: "Focus hard on the left and right lines! The top and bottom can be a bit more flexible."
Training Only: This special rule only applies while the AI is studying (training). Once the AI is finished learning and is ready to work (inference), it doesn't need to follow these extra rules. It just works as fast as before. It's like a student using a special highlighter to study for a test; once the test starts, they just write normally, but they remember the important parts better.
Data Efficiency: The authors found that this method is so good at teaching the AI what matters that it doesn't need a massive library of examples to learn. It can learn to read tables very well even with a relatively small number of examples (around 2,000 to 3,000), whereas other methods might need much more data to get the same result.

The Results: Faster and More Accurate

The team tested their "Smart Painter" (ConRTF) against other top-tier AI models on huge datasets of real-world tables (like scientific papers and business documents).

Better Accuracy: ConRTF made fewer mistakes in how it drew the table lines. It improved the "GriTS" score (a measure of how well the table structure is understood) by up to 1.6 points compared to the best existing real-time models.
Speed: Because the special rules only happen during training, the AI runs just as fast as the standard models when it's actually reading a document. It didn't slow anything down.
Handling the Hard Stuff: The biggest improvements happened on the "messy" tables where other AI models struggled. ConRTF was better at untangling complex layouts where rows and columns overlap or are hard to distinguish.

In a Nutshell

The paper presents a way to teach AI to read tables by teaching it that not all lines are created equal. By telling the AI to focus its energy on the specific lines that define the shape of a row or a column, it learns faster, needs less data, and produces much cleaner, more accurate table structures without slowing down the process.

Technical Summary: ConRTF – Edge-Constrained Boundary Distribution Refinement for Realtime TransFormer Table Structure Recognition

1. Problem Statement

Table Structure Recognition (TSR) aims to recover the row and column layout of tables from document images, a critical component of document understanding pipelines. The paper identifies that accurate TSR is highly sensitive to localization errors; unlike generic object detection, small inaccuracies in row or column boundaries propagate into incorrect cell assignments and structural inconsistencies. Metrics such as TEDS and GriTS penalize even minor boundary misalignments.

Existing approaches face two primary limitations:

Structural Asymmetry Ignored: Most detection-based methods treat table elements (rows, columns, headers) as generic objects, ignoring the fundamental geometric asymmetry where rows are defined by horizontal boundaries and columns by vertical boundaries.
Real-time vs. Accuracy Trade-off: While Transformer-based multi-stage pipelines achieve high accuracy, their depth and post-processing hinder real-time deployment. Conversely, real-time detectors often lack the fine-grained boundary precision required for complex table layouts.

2. Methodology

The authors propose ConRTF, a real-time architecture that reformulates TSR as a boundary-centric task using distribution-based boundary representations. The core innovation is the Edge-constrained Fine-grained Localization (EFL) loss.

A. Boundary-Centric Modeling
Instead of detecting cells or generating token sequences, ConRTF models table elements as "row-like" (rows, headers, section separators) and "column-like" (columns) objects. It adopts the D-FINE architecture, which uses a Transformer decoder to iteratively refine boundary distributions. Rather than predicting coordinates directly, the model predicts probability distributions over candidate offsets for the top, bottom, left, and right edges.

B. Edge-constrained Fine-grained Localization (EFL) Loss
The standard Fine-Grained Localization (FGL) loss used in D-FINE supervises all four edges of an object with equal weight. ConRTF introduces EFL to encode table-specific geometric priors:

Structural Asymmetry: For row-like elements, horizontal boundaries (top/bottom) are structurally defining, while vertical boundaries merely mark extent. The reverse is true for column-like elements.
Weighted Supervision: EFL assigns higher weights ( $\lambda_h$ $λ_{h}$ ) to structurally defining edges and lower weights ( $\lambda_l$ $λ_{l}$ ) to extent edges.
- Row-like: $w = [\lambda_h, \lambda_h, \lambda_l, \lambda_l]$ (Top, Bottom, Left, Right).
- Column-like: $w = [\lambda_l, \lambda_l, \lambda_h, \lambda_h]$ .
Derivation of Weights: The optimal ratio between $\lambda_h$ and $\lambda_l$ is derived from an IoU-sensitivity analysis. The authors demonstrate that an error on a defining edge causes significantly more IoU degradation than an error on an extent edge, proportional to the aspect ratio ($AR$) of the element. The weights are constrained to sum to the uniform baseline total to ensure fair comparison.
Training vs. Inference: EFL operates only during training. It guides the iterative refinement of boundary distributions toward structurally meaningful adjustments. At inference, the pipeline remains identical to the baseline D-FINE, incurring zero additional computational cost.

C. Data Efficiency
The system incorporates Global Optimal Localization Self-Distillation (GO-LSD) from D-FINE, which transfers refined boundary distributions from deeper layers to shallower ones. This stabilizes early-layer predictions, allowing ConRTF to maintain robust accuracy even with limited annotated data (2k–3k tables).

3. Key Contributions

Boundary-Centric Reformulation: The authors reformulate TSR as a boundary-centric task using distribution-based representations, enabling fine-grained localization within a compact, real-time Transformer architecture.
EFL Loss: Introduction of a loss function that encodes geometric priors via edge weights derived from IoU-sensitivity analysis. This loss explicitly differentiates between structurally defining and extent boundaries without adding inference overhead.
Data Efficiency: Demonstration that the approach remains robust with small annotated datasets (2k–3k tables), making it suitable for deployment where annotation budgets are limited.
Comprehensive Evaluation: Extensive experiments on the large-scale PubTables-1M benchmark and two private datasets (Business and Finance), evaluating both structural accuracy (TEDS, GriTS) and real-time efficiency.

4. Experimental Results

The authors evaluated ConRTF against optimized baselines (D-FINE-X) and other real-time detectors (RT-DETRv2, YOLOv10/11/12).

PubTables-1M: ConRTF achieved consistent improvements over the D-FINE-X baseline across all metrics. Notably, it reached 99.47% TEDS-Struct and 99.37% GriTS-Top, outperforming the baseline by +0.07 and +0.10 points respectively. On a "hard" subset of tables where the baseline scored below 99% GriTS-Con, ConRTF showed a statistically significant gain of +2.6 GriTS-Con points.
Private Datasets (Limited Data): On the Business and Finance datasets (with training sets of ~3k and ~2k tables respectively), the gap between DETR-based and YOLO-based methods widened. ConRTF maintained high accuracy where YOLO models struggled significantly (e.g., YOLOv10-X dropped to 48.8% TEDS-Struct on Finance). ConRTF outperformed D-FINE-X on both datasets, with gains of +1.29 GriTS-Con on the Finance dataset.
Score Distribution: Analysis of per-document scores revealed that ConRTF effectively shifts predictions from the "low" (<70% score) and "moderate" bands to the "high" (≥90%) band, particularly in low-data regimes.
Inference Speed: As EFL is training-only, ConRTF shares identical parameters (62M), FLOPS (202G), and inference speed (15.42 FPS on GPU) with D-FINE-X.

5. Significance and Claims

The paper claims that ConRTF addresses the specific geometric asymmetry of table layouts, which generic object detection losses overlook. By formalizing this asymmetry into the training objective, the method achieves:

Higher Structural Accuracy: Improved boundary localization leads to better cell assignment and structural recovery, particularly for complex or ambiguous layouts.
Real-Time Viability: The approach matches the speed of state-of-the-art real-time detectors while surpassing their accuracy, bridging the gap between high-precision multi-stage pipelines and efficient single-stage detectors.
Robustness in Low-Data Scenarios: The integration of geometric priors allows the model to generalize effectively even when trained on significantly smaller datasets than the massive PubTables-1M, making it practical for domain-specific applications.

The authors conclude that while the current method targets axis-aligned layouts, future work could explore per-class or instance-adaptive edge weights and zero-shot settings.