💻 computer science

LEGR: Learnable-Edge Graph Refinement for Table Structure Recognition

This paper introduces LEGR, an end-to-end framework for Table Structure Recognition that replaces static geometric heuristics with learnable relational edge features and a cascaded Graph Attention Network refinement module to achieve robust, domain-invariant performance by iteratively correcting predictions through learned global layout reasoning.

Original authors: Saeed Ahmed, Muhammad Mudassir Mehmood, Muhammad Imran Malik, Muhammad Naseer Bajwa, Waheed Ahmed

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saeed Ahmed, Muhammad Mudassir Mehmood, Muhammad Imran Malik, Muhammad Naseer Bajwa, Waheed Ahmed

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of digital documents, tables are the quiet workhorses of information. They organize financial reports, scientific data, and administrative records into neat grids of rows and columns. For a human reader, the structure is obvious; the lines and spacing guide the eye, making it easy to see how a value in one cell relates to another. For a computer, however, a table is just a chaotic collection of pixels. To turn a picture of a table into useful data, software must perform a difficult task called table structure recognition. It has to figure out which text belongs to the same row, which belongs to the same column, and which cells stretch across multiple spaces. This is not merely about reading the words; it is about understanding the invisible logic that holds the grid together. Without this understanding, a computer cannot extract the data to answer questions, analyze trends, or build databases.

For years, researchers have tried to teach computers to see these structures by treating the table as a map of connections. In this view, every piece of text is a point on a map, and the lines connecting them represent relationships like "next to" or "above." The challenge has always been deciding how to draw those lines. Traditional methods relied on rigid, pre-written rules based on geometry. If two text boxes were close to each other, the computer assumed they were connected. If they were far apart, the connection was ignored. This approach worked well for simple, clean tables but often failed when the layout was complex, with merged cells or missing borders. The computer was following a static map that didn't account for the unique context of each document.

A team of researchers from the National University of Sciences and Technology in Pakistan has proposed a different way forward. They developed a system called LEGR, which stands for Learnable-Edge Graph Refinement. Instead of using fixed rules to decide which text segments are related, LEGR learns to make those decisions itself. Imagine a student learning to read a map: at first, they might follow a simple set of directions, but with practice, they learn to recognize patterns and relationships that aren't explicitly drawn on the page. LEGR does something similar. It starts with a rough guess about the table's structure and then iteratively improves that guess by learning which connections matter most.

The core of this system is a shift from static rules to dynamic learning. In previous systems, the computer would look at two text boxes and measure the distance between them. If the distance was small, it would draw a line. LEGR, however, looks at the content and context of the text boxes to decide if they belong together. It uses a special training task where it tries to predict the logical difference between two connected pieces of text. By doing this, the system learns to recognize that a header at the top of a page and a data point at the bottom are related, even if they are far apart in physical space. This allows the system to understand the "relational substrate"—the underlying logic of the table—rather than just the physical layout.

Once the system has learned these relationships, it refines its understanding through a two-step process. First, it makes an initial, broad guess about where every piece of text belongs in the grid. Then, it passes this information through a network that allows the different parts of the table to "talk" to each other. This network corrects small errors and resolves ambiguities. If the system initially thinks two cells are in the same row but realizes they don't align with the rest of the table, it can change its mind. This iterative correction happens in two stages, allowing the system to first fix large structural mistakes and then fine-tune the details.

The researchers tested their system on several large collections of tables from scientific papers and financial reports. On standard tests, LEGR performed as well as the best existing methods, achieving high accuracy in reconstructing the table structure. However, the most significant finding came when they tested the system on a completely new type of document it had never seen before. Without any extra training, the system achieved a success rate of 98.46 percent on this unseen data. This suggests that the system learned general principles of how tables work, rather than just memorizing the specific layouts of the documents it was trained on.

To prove that their approach was truly superior, the researchers ran a series of experiments where they swapped out their learning system with traditional, rule-based methods. When they replaced the learned connections with a comprehensive set of hand-crafted geometric rules, the system's performance dropped significantly, falling by about four percentage points. This confirmed that the ability to learn the connections was the key to success. Furthermore, they tested what happened if they randomly connected the text segments instead of using a logical map. Surprisingly, the system's performance barely changed. This indicates that the system is so good at learning the relationships from the text itself that the initial way the connections are drawn matters very little. The system can figure out the correct structure regardless of how the map is initially sketched.

The implications of this work extend beyond just reading tables. It demonstrates that for tasks involving complex, structured relationships, learning the connections is more important than engineering them. By letting the computer discover the logic of the data rather than forcing it to follow a pre-set map, the researchers have created a more robust and adaptable tool. This approach could eventually help computers understand other complex layouts, such as forms or hierarchical documents, where the relationships between elements are not always obvious. The study shows that when a system is allowed to learn the rules of the game, it can play better than one that is simply told how to move.

Technical Summary: LEGR - Learnable-Edge Graph Refinement for Table Structure Recognition

1. Problem Definition

Table Structure Recognition (TSR) aims to convert visual table images into machine-readable logical structures (row/column indices and spans). While physical structure (bounding boxes) can be obtained via off-the-shelf OCR, inferring the logical structure is a complex relational problem.

Existing graph-based TSR approaches treat cells as nodes and spatial relations as edges. However, these methods suffer from two primary limitations:

Static Graph Construction: They rely on hand-crafted geometric heuristics (e.g., $k$ -nearest neighbors based on bounding box centroids) to define graph connectivity and edge features. This fixes the relational substrate before learning begins, preventing the model from capturing non-local dependencies (e.g., header-to-cell alignment across long rows) or handling border-less and spanning-cell layouts effectively.
Single-Pass Prediction: Most pipelines predict the final structure in a single forward pass, lacking a mechanism to revise initial estimates once a global view of the table is assembled.

2. Methodology: LEGR Framework

LEGR (Learnable-Edge Graph Refinement) is an end-to-end framework that learns the relational graph structure and iteratively refines predictions. The architecture consists of four functional blocks:

2.1 Visual and Node Encoding

Visual Backbone: A Swin Transformer generates a four-level feature pyramid. These are fused via a Large-Kernel Pyramid Attention Network (LKPAN) to produce a unified stride-4 feature map ( $FFPN$ ) for fine-grained boundary preservation and a stride-32 global context map ( $C_4$ ).
Node Embeddings: For each OCR-detected bounding box, an initial embedding ( $h_i^{in}$ $h_{i}^{in}$ ) is constructed by fusing:
- Visual features (via RoIAlign on $FFPN$ ).
- Learnable geometric embeddings of the bounding box coordinates.
- Absolute 2D sinusoidal positional encodings.
Context Encoder: A transformer-based encoder processes these embeddings using self-attention (for inter-segment dependencies) and cross-attention (injecting global context from $C_4$ ). This produces contextualized node features ( $H_0$ ).

2.2 Relational Edge Learning

Instead of static heuristics, LEGR employs a Relational Edge Learner:

Graph Construction: During training, a neighborhood for each node is sampled comprising both geometrically close neighbors and random long-range neighbors to prevent overfitting to local geometry.
Edge Features: Edge features are generated by a shared Context Encoder and an Edge Head (MLP). The feature vector for an edge $(v_i, v_j)$ concatenates node embeddings, their difference, and their element-wise product.
Auxiliary Supervision: The Edge Head is supervised by an auxiliary task: predicting the logical-coordinate difference (signed offsets for row and column start/end) between connected nodes. This forces the network to learn context-aware relational features rather than copying raw coordinate offsets.

2.3 Cascaded Iterative Refinement

LEGR replaces single-pass prediction with a two-stage cascaded refinement module using Graph Attention Networks (GATs):

Stage 1: A Node Head produces initial coarse logits ( $L_0$ ). These are fused with node features and passed through a GAT ( $GAT_1$ ) to predict a corrective delta ( $\delta L_1$ ). The refined logits are $L_1 = L_0 + \delta L_1$ .
Stage 2: The process repeats with a second GAT ( $GAT_2$ ) operating on the Stage 1 features and the running estimate $L_1$ , producing a final correction $\delta L_2$ and logits $L_2 = L_1 + \delta L_2$ .
This coarse-to-fine approach allows the model to first resolve large-scale structural ambiguities and then perform fine-grained local adjustments.

2.4 Training Objective

The model is trained with a composite loss function:
$\mathcal{L}_{total} = (1 - \lambda) \mathcal{L}_{coord} + \lambda (\alpha \mathcal{L}_{aux} + \beta \mathcal{L}_{edge})$
Where $\mathcal{L}_{coord}$ aggregates cross-entropy losses for the initial and two refinement stages, $\mathcal{L}_{aux}$ is a multi-positive contrastive loss to cluster segments belonging to the same cell, and $\mathcal{L}_{edge}$ is the cross-entropy loss for the coordinate-difference prediction.

3. Key Contributions

Learnable Edge Features: Replaces static geometric heuristics with a relational edge learner supervised by an auxiliary logical-coordinate-difference task. This enables the capture of non-local dependencies and context-aware relations.
Cascaded Refinement: Introduces a two-stage GAT-based refinement module that iteratively corrects initial predictions, enabling the model to reason globally and self-correct local errors.
Domain-Invariant Reasoning: Demonstrates that learning the relational substrate yields representations that generalize well across domains without fine-tuning.

4. Experimental Results

LEGR was evaluated on three benchmarks: PubTabNet (scientific articles), FinTabNet (financial reports), and SciTSR (scientific papers).

In-Domain Performance (PubTabNet & FinTabNet):
- On PubTabNet, LEGR achieved 97.91 TEDS-S, placing it second only to TFLOP (98.38) among state-of-the-art methods, despite using a non-autoregressive architecture.
- On FinTabNet, LEGR achieved 99.10 TEDS-S, again second to TFLOP (99.56).
- Notably, LEGR achieves these results without an autoregressive HTML decoder, offering constant inference cost relative to table size.
Cross-Domain Generalization (SciTSR):
- In a zero-shot setting (trained on PubTabNet, tested on SciTSR without fine-tuning), LEGR achieved an F1 score of 98.46. This is within 0.84 points of fully supervised state-of-the-art methods (e.g., LORE at 99.30), demonstrating strong domain invariance.
- With fine-tuning, LEGR reached 99.39 F1, surpassing previous graph-based methods.
Comparison with Graph-Based Methods:
- LEGR significantly outperforms prior graph-based methods like NCGM and GraphTSR. On PubTabNet, it improves TEDS-S by 2.51 points over NCGM. On SciTSR, it improves F1 by 5.99 points over GraphTSR.

5. Ablation Studies and Analysis

Learned vs. Hand-Crafted Edges: Replacing the learned edge features with a comprehensive hand-crafted baseline (PCEF) caused a drop of approximately 4.0 points in TEDS on FinTabNet (from 99.10 to 95.13). This confirms that relational features for complex layouts are not expressible as fixed functions of bounding boxes.
Graph Adjacency Robustness: Randomizing the input graph adjacency (replacing $k$ -NN with random connections) resulted in negligible performance loss (<0.15 points). This indicates that the heavy lifting occurs within the learned Edge Head and GATs, not in the specific choice of neighbors.
Refinement Stages: The second refinement stage consistently improved performance over the first, confirming the value of iterative correction.
Difference Prediction vs. Affinity: Supervising edges via coordinate differences outperformed an alternative affinity-based formulation (predicting same-row/column probabilities), attributed to avoiding optimization conflicts between competing constraints.

6. Significance and Claims

The paper claims that learning the relational substrate is more critical than engineering it. The central thesis is that static geometric heuristics are insufficient for capturing the complex, non-local dependencies inherent in table structures.

Robustness: The ability of LEGR to transfer to the unseen SciTSR dataset in a zero-shot setting proves that the learned relational reasoning captures domain-invariant cues (what makes two segments share a row/column) rather than memorizing dataset-specific layout statistics.
Efficiency: By avoiding autoregressive decoders, LEGR offers a more efficient inference path for large tables while maintaining competitive accuracy.
Paradigm Shift: The work suggests that iteratively refined graphs with learned edges offer a viable alternative to autoregressive structured prediction for hierarchical and tabular data, potentially extending to layout analysis and form understanding.

The authors acknowledge limitations regarding noisy, scanned, or camera-captured tables and assume reliable input OCR segments, identifying these as areas for future work.