💬 NLP

PulseBench-Tab: A Multilingual Benchmark for Table Extraction with Graph-Based Evaluation

This paper introduces PulseBench-Tab, a multilingual benchmark comprising 1,820 human-annotated tables across nine languages and four scripts, alongside a novel graph-based evaluation metric called T-LAG, to assess the performance of various table extraction systems on complex, real-world document images.

Original authors: Ritvik Pandey, Sid Manchkanti, Mohammed Wazir Adain, Mohammed Hadi, Dushyanth Sekhar

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ritvik Pandey, Sid Manchkanti, Mohammed Wazir Adain, Mohammed Hadi, Dushyanth Sekhar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of documents—financial reports, government forms, and corporate disclosures—printed in nine different languages. Inside these documents are thousands of tables, like spreadsheets, filled with numbers and text. The goal of this paper is to see how well computers can "read" these tables, copy them perfectly, and understand where every single piece of information belongs.

Here is the breakdown of PulseBench-Tab and T-LAG, explained simply.

1. The Problem: Why Tables are Tricky

Reading a table isn't just about recognizing letters (like reading a book). It's about understanding geometry.

The Analogy: Imagine a table is a jigsaw puzzle. If you take a piece with the number "100" and put it in the wrong spot, the picture is ruined. In a computer system, if a number ends up in the wrong cell, it's not just a typo; it's a disaster that breaks any calculations done later.
The Gap: Previous tests mostly looked at English tables or treated tables like long lists of words (flattening them), which loses the 2D structure (rows and columns). They also didn't test enough languages that write right-to-left (like Arabic) or use complex characters (like Chinese or Japanese).

2. The Solution: A New "Table Gym" (The Dataset)

The authors built a massive training ground called PulseBench-Tab.

The Size: It contains 1,820 real-world tables.
The Variety: These tables come from 380 different documents in 9 languages (English, Chinese, Arabic, Russian, etc.) and use 4 different writing systems.
The Difficulty: Some tables are tiny (just 2 cells), while others are monsters with over 1,000 cells. About half of them have "merged" cells (where one big cell covers the space of two or three smaller ones), which is like a puzzle piece that covers multiple spots on the board.
The Gold Standard: Every single table was checked by human experts who spoke the specific language to ensure the "answer key" was 100% correct.

3. The New Ruler: T-LAG (The Graph Metric)

To grade the computers, the authors invented a new scoring system called T-LAG.

The Old Way: Previous methods often checked if the computer got the right words, but they didn't care enough if those words were in the right neighbors. It was like grading a student on a spelling test but ignoring if they put the words in the right sentences.
The New Way (T-LAG): Imagine the table as a map of cities connected by roads.
- Nodes (Cities): The individual cells.
- Edges (Roads): The connections between cells (e.g., "Cell A is to the right of Cell B" or "Cell C is below Cell D").
How it Scores: T-LAG looks at the "roads." It asks: "Did the computer build the same map of roads as the real one?"
- If the computer gets the text right but puts it in the wrong column, the "road" is broken, and the score drops.
- It uses a special math trick (the Hungarian Algorithm) to find the best possible match between the real table and the computer's table, ensuring it doesn't just guess randomly.
- The "Strictness" Dial: The authors set the scoring to be very strict (a "k=7" setting). In the real world, if a financial report says "$1 million" instead of "$10 million," it's a failure, not a "close enough" attempt. This metric treats small errors as big failures to reflect real-world needs.

4. The Race: Who Won?

The authors tested 9 different computer systems (including big tech APIs, open-source tools, and their own system) on this dataset.

The Winner: Pulse Ultra 2 (the authors' own system) came in first with a score of 93.5%. It was the only one to get a "perfect" score on more than half of the tables.
The Runner-Up: Gemini 3.1 came in second with 81.6%.
The Gap: There was a huge drop-off after the top two. The bottom third of the systems scored below 72%, meaning they struggled significantly.
The Hardest Challenge: Non-Latin scripts (like Arabic, Korean, and Chinese) were the hardest for everyone.
- Analogy: Think of it like a runner who is great on a flat, paved road (English) but trips over every rock on a mountain trail (Arabic/Korean). Even the best systems saw their scores drop significantly when the language changed.
The "Silent Failures": Some systems didn't just give bad answers; they gave no answer at all for about 20% of the tables. In a real business, this is like a robot that just stops working when it sees a difficult document.

5. The Takeaway

The paper concludes that while computers are getting better at reading tables, there is still a massive gap between the best systems and the rest of the field, especially when dealing with complex layouts or languages that aren't English.

They have released the dataset, the scoring code, and the results to the public. This is like opening the gym doors to everyone, so researchers can see exactly where the machines are failing and try to fix them, rather than guessing.

In short: They built a tough, multilingual obstacle course for table-reading AI, invented a new ruler to measure how well the AI navigated it, and found that while one system is currently leading the pack, most others are still struggling to keep their footing on complex, non-English terrain.

Technical Summary: PulseBench-Tab and T-LAG

1. Problem Statement

Table extraction from document images is a foundational task in document intelligence, critical for financial analysis, regulatory compliance, and enterprise data processing. However, existing benchmarks and evaluation metrics suffer from significant limitations:

Structural vs. Formatting Conflation: Metrics like TEDS operate on HTML DOM trees, conflating formatting conventions (e.g., <thead> wrappers) with structural errors.
Loss of 2D Topology: Sequence-alignment approaches (e.g., Needleman-Wunsch) flatten tables into 1D sequences, losing the distinction between horizontal and vertical adjacency.
Linguistic Bias: Most benchmarks focus primarily on English, leaving performance on non-Latin scripts (Arabic, Cyrillic), right-to-left (RTL) languages, and CJK (Chinese, Japanese, Korean) text largely unmeasured.
Semantic Sensitivity: In downstream applications, a number in the wrong cell is a semantic failure, not merely an OCR error. Current metrics often fail to penalize positional errors sufficiently.

2. Methodology

2.1 PulseBench-Tab Dataset

The authors introduce PulseBench-Tab, an open, multilingual benchmark designed to address these gaps.

Scale and Scope: The dataset contains 1,820 human-annotated table images extracted from 380 unique source documents (financial filings, government reports, regulatory disclosures).
Linguistic Diversity: It spans 9 languages across 4 scripts (Latin, CJK, Arabic, Cyrillic). English comprises 32.6% of samples, with the remainder distributed among Chinese, Spanish, Russian, French, Japanese, Arabic, German, and Korean.
Complexity: Tables range from 2 to 1,183 cells. Notably, 48.1% of tables contain merged or spanning cells (rowspan/colspan), and the dataset includes dense spreadsheets and multi-level headers.
Ground Truth: Annotations were produced via a multi-stage pipeline involving native-speaking domain experts, cross-lingual quality control, and adversarial cell-by-cell audits against source images. Ground truth is provided as fully structured HTML.

2.2 T-LAG Evaluation Metric

To evaluate extraction quality, the authors propose T-LAG (Table Logical Adjacency Graph), a novel metric that models tables as directed graphs to capture both structural fidelity and content accuracy in a single score.

Core Philosophy:
Unlike node-based metrics that verify cell content independently, T-LAG operates on edges (adjacency relationships). This ensures that a cell is evaluated not just by its text, but by its position relative to its neighbors. A transposed column, for instance, results in different adjacency edges and is thus penalized.

Pipeline:

Grid Parsing: Both ground truth and predicted HTML are parsed into a cell-position grid matrix, preserving rowspan and colspan attributes.
Edge Extraction: Directed edges are emitted for RIGHT and BELOW adjacencies between cells with different IDs. Edges within spanning cells are suppressed.
Weighting ( $\Psi$ Function): For matching edges, a similarity weight is computed using a text similarity kernel.
- Texts are normalized (whitespace, Unicode dashes, null markers).
- Similarity is calculated as $1 - \frac{d_{Lev}}{\max(|g|, |p|)}$ , raised to an exponent $k=7$ .
- Rationale for $k=7$ : This aggressive decay ensures that near-miss text (e.g., 90% character accuracy) is heavily penalized ( $\Psi \approx 0.48$ ), reflecting enterprise requirements where a single digit error in a financial figure renders data unusable.
Optimal Matching: The Hungarian algorithm is used to find the globally optimal one-to-one assignment between ground truth and predicted edges, maximizing total matched weight. This avoids the greedy alignment issues of previous metrics.
Scoring: Precision, Recall, and F1 are computed based on the weighted matched edges.
- Precision: Penalizes hallucinated rows/columns.
- Recall: Penalizes dropped rows or missing merged cells.
- Fallback: For single-cell tables, the score defaults to direct text similarity.

3. Key Contributions

Multilingual Benchmark: A large-scale, high-quality dataset covering 9 languages and 4 scripts, specifically addressing the underrepresentation of RTL and CJK languages in table extraction research.
T-LAG Metric: A graph-based evaluation method that preserves 2D structural topology and uses optimal bipartite matching, distinguishing it from tree-edit distance (TEDS) and 1D sequence alignment methods.
Comprehensive Evaluation: A rigorous benchmarking of 9 commercial and open-source systems (including Pulse Ultra 2, Gemini 3.1, LlamaParse, AWS Textract, and Azure DI) with public release of all outputs and scoring code.

4. Results

The evaluation of 9 providers across 1,820 samples reveals significant performance disparities:

Top Performer: Pulse Ultra 2 leads with a T-LAG score of 0.935, achieving a median score of 1.0 (perfect extraction) on 57.9% of samples. It outperforms the second-ranked system by 11.9 percentage points.
General Performance: Only one provider exceeds a 90% mean T-LAG score. The bottom third of providers score below 0.72, indicating fundamental limitations in current multilingual table extraction capabilities.
Structural Hallucinations: Even high-performing systems (e.g., Gemini 3.1) achieve perfect extraction on only ~28% of tables, frequently introducing fabricated rows, invented content, or incorrect span attributes.
Linguistic Variance:
- Easiest: French and Spanish show the highest performance across all providers, likely due to shared Latin scripts and standard formatting.
- Hardest: Arabic and Korean exhibit the widest cross-provider variance (spreads >75 percentage points).
- CJK Challenges: Providers with strong English performance (e.g., AWS Textract at 0.724) drop significantly on CJK languages (0.324 on Chinese, 0.254 on Korean), highlighting OCR weaknesses.
Coverage: Some "Agentic" systems failed to produce output for ~20% of samples, a critical factor for production reliability.

5. Significance and Claims

The paper positions PulseBench-Tab and T-LAG as essential tools for advancing the field of document intelligence:

Realistic Evaluation: By enforcing strict penalties for positional and structural errors (via the $k=7$ exponent and edge-based matching), the metric reflects the operational reality of enterprise data processing, where approximate extraction is functionally equivalent to failure.
Reproducibility: The authors emphasize that the full dataset, scoring code, and all provider outputs are publicly available to support reproducible evaluation.
Future Direction: The results highlight that non-Latin scripts remain a major unsolved challenge. The authors note that while the benchmark is large, further scaling is needed for smaller languages (e.g., Korean, German) to narrow confidence intervals.
Limitations: The authors modestly acknowledge current limitations, including the assumption of single-table-per-image and single-page tables, noting that multi-table layouts and multi-page tables are not yet evaluated.

In conclusion, the paper argues that current table extraction systems are not yet robust enough for high-stakes enterprise applications without additional verification, particularly in multilingual and structurally complex scenarios. PulseBench-Tab provides the necessary infrastructure to measure progress toward this goal.