TDATR: Improving End-to-End Table Recognition via Table Detail-Aware Learning and Cell-Level Visual Alignment

1. Problem Statement

Table Recognition (TR) is a critical task in document analysis, converting table images into machine-readable formats (e.g., HTML, LaTeX). Existing approaches suffer from two main limitations:

• Modular Pipelines: Traditional methods separate Table Structure Recognition (TSR) and Table Content Recognition (TCR). This decoupling leads to suboptimal integration, error accumulation during post-processing, and complex workflows.
• Data-Dependent End-to-End Models: Recent unified Vision-Language (VLM) approaches simplify the pipeline but rely heavily on large-scale, annotated TR datasets. In data-constrained real-world scenarios, these models struggle with generalization. Furthermore, they often lack explicit spatial correspondence (cell locations), limiting interpretability and downstream utility.

The core challenge is achieving robust, end-to-end table recognition that generalizes well with limited labeled TR data while maintaining high interpretability through precise cell localization.

2. Methodology

The authors propose TDATR (Table Detail-Aware Table Recognition), a framework built on a "Perceive-then-Fuse" strategy. The architecture consists of a Visual Encoder (Swin Transformer), a Multi-modal Language Decoder, and a novel Structure-Guided Cell Localization (SGCL) module.

A. "Perceive-then-Fuse" Training Strategy

The training process is divided into two distinct stages to decouple capability learning from specific task modeling:

1. Table Detail-Aware Learning (Perception Stage):

   • Goal: Equip the model with fine-grained perception of table structures and content using diverse, large-scale document data (not limited to tables).
   • Approach: Under a unified language modeling paradigm, the model is pre-trained on multiple tasks:
     • Content Tasks: Spatially ordered text spotting, text spotting with box queries, and Markdown parsing. These leverage rich-text corpora to enhance text recognition, localization, and layout awareness.
     • Structure Tasks: Table element detection (cells, span cells, rows, columns) and table structure parsing (HTML/Markdown). These tasks teach the model to understand hierarchical table logic and spatial dependencies.
   • Benefit: This stage allows the model to learn robust structural and textual priors from massive, diverse document datasets, reducing reliance on scarce TR data.

2. Table Detail Fusion (Fusion Stage):

   • Goal: Integrate the learned implicit details into a unified end-to-end TR model.
   • Approach: The model is fine-tuned on a limited set of TR data to perform Table HTML Parsing. It generates structured HTML sequences that jointly encode structure and content.
   • Integration: During this stage, the Structure-Guided Cell Localization (SGCL) module is trained to predict precise cell coordinates, ensuring spatial consistency with the generated text.

B. Structure-Guided Cell Localization (SGCL) Module

To address the lack of explicit cell locations in standard VLMs, TDATR introduces the SGCL module:

• Representation Aggregation: It extracts cell-level representations from the hidden states of the language decoder (combining shallow visual cues and deep structural/linguistic information).
• Structure-Guided Attention: It projects cell representations into row and column feature spaces to compute adjacency matrices. These masks guide bidirectional contextual attention, allowing cells to focus on their neighbors within the same row or column.
• Refinement: Initial cell boxes are regressed from these enhanced representations and further refined using multi-resolution visual features via DAB-DETR decoder layers. This eliminates the need for unstable bipartite matching and post-processing.

3. Key Contributions

1. "Perceive-then-Fuse" Paradigm: A novel strategy that reduces reliance on large-scale labeled TR data by first establishing robust structure/content perception via diverse document data, then fusing these capabilities for end-to-end TR.
2. Table Detail-Aware Learning: A unified pre-training framework that combines structure understanding (cell/row/column detection) and content recognition (text spotting, Markdown parsing) to enhance model robustness across domains.
3. Structure-Guided Cell Localization: A module that refines cell boundaries using structural priors and multi-level visual features, significantly improving visual-language alignment and interpretability without post-processing.
4. Unified Model Performance: A single model evaluated across seven diverse benchmarks without dataset-specific fine-tuning, demonstrating state-of-the-art (SOTA) or highly competitive results.

4. Experimental Results

The authors evaluated TDATR on seven public benchmarks covering real-world (Chinese/English, low quality, borderless) and digitally-born tables.

• Real-World Tables (iFLYTAB-full, TabRecSet):
  • TDATR achieved SOTA performance on both datasets.
  • On iFLYTAB-full, it outperformed the best modular TR method (SEMv3+PPOCR) by 15.82% in overall TR accuracy (TEDS).
  • It surpassed the end-to-end baseline EDD by 6.59% on TabRecSet.
• Digitally-Born Tables (PubTabNet, PubTables-1M):
  • Achieved SOTA TR performance (TEDS) on PubTables-1M (97.97) and PubTabNet (95.12).
  • Demonstrated superior alignment between structure and content (lowest TEDS-Delta), mitigating error accumulation seen in modular approaches.
• Generalization & Efficiency:
  • The model performed competitively against much larger OCR-VLMs (e.g., MinerU2.5, PaddleOCR-VL) and general MLLMs (e.g., Qwen2.5-VL) on benchmarks like OmniDocBench and OCRBench, despite being trained with significantly less data.
  • Cell Localization: Achieved SOTA AP50 scores (94.37 on iFLYTAB-full, 91.80 on PubTabNet), outperforming methods that rely solely on visual cues or sequential coordinate generation.
  • Inference: The parallel nature of SGCL makes it 1.6x faster than autoregressive coordinate generation baselines.

5. Significance

TDATR addresses the critical bottleneck in table recognition: the scarcity of high-quality, annotated table data. By decoupling the learning of "table details" (structure and content) from the specific task of "table recognition," the framework leverages the vast availability of general document data to build a robust foundation.

The introduction of cell-level visual alignment directly within the end-to-end framework bridges the gap between semantic output and spatial layout, enhancing both the accuracy and interpretability of the results. This approach offers a scalable solution for document digitization and retrieval-augmented generation (RAG) systems, proving that high-performance TR does not necessarily require massive, task-specific datasets.