💻 computer science

ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction

This paper introduces ShredBench, a novel benchmark with an automated generation pipeline for evaluating Multimodal Large Language Models on the challenging task of reconstructing shredded documents, revealing that current models struggle significantly with the fine-grained cross-modal reasoning required to bridge visual discontinuities compared to their performance on intact documents.

Original authors: Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a newspaper, a piece of code, or a financial spreadsheet. Now, imagine someone takes a pair of scissors and shreds that document into 8, 12, or even 16 random, jagged pieces. They then throw those pieces into a pile, mix them up, and maybe even crumple them a bit.

Your task? Look at that messy pile of paper scraps and tell a computer exactly what the original document said, in the correct order.

This is the core challenge of a new study called ShredBench. The researchers wanted to see if the newest, smartest AI models (called Multimodal Large Language Models, or MLLMs) can act like a human detective when faced with a destroyed document.

Here is a simple breakdown of what they did and what they found:

1. The "Shredded Puzzle" vs. The "Jigsaw Puzzle"

Usually, when we test AI on puzzles, we use standard jigsaw puzzles where you match the picture on the edge of one piece to the edge of another. That's a visual game.

But ShredBench is different. It's a semantic game.

The Analogy: Imagine you have a sentence: "The algorithm optimiz-" on one piece and "-es the loss function" on another. The edges don't match perfectly because the cut is jagged. A human doesn't need to see the exact pixel match; they use their brain to know that "optimizes" is the word that fits there.
The Goal: The researchers wanted to see if AI could use its "brain" (language knowledge) to piece together the meaning, rather than just its "eyes" (matching pixels).

2. How They Built the Test (The "Shredder")

To make a fair test, the researchers didn't just take real photos of torn paper. They built a digital "shredder" pipeline:

The Source: They grabbed real news articles (in English and Chinese), computer code (Python, Java, C++), and complex tables.
The Cut: They used a mathematical method (Voronoi tessellation) to slice the digital documents into irregular, jagged shapes, just like a real shredder would.
The 3D Effect: They simulated physics in a 3D program to add shadows, crinkles, and depth, making the digital pieces look like real, messy paper scraps.
The Mix: They scrambled the pieces so the AI had to figure out the order from scratch.

3. The Results: "Smart" but "Fragile"

The researchers tested 14 of the world's most advanced AI models, including big names like GPT-5, Gemini 3, and Qwen.

The Good News: When the documents were whole and clean, almost all the models performed like human experts. They could read and understand the text perfectly.
The Bad News: As soon as the documents were shredded, the performance of most models collapsed.
- Think of it like a student who can ace a math test if the questions are printed clearly, but fails miserably if the test paper is torn into confetti.
- As the number of pieces increased (from 8 to 16), the AI's ability to reconstruct the text dropped sharply.
- Many models started "hallucinating"—making up words that weren't there or putting sentences in the wrong order.

4. The Winners and Losers

The Champion: Gemini 3 Pro was the clear winner. It was the most resilient, able to handle the shredded pieces better than anyone else. It could still "read" the text even when the visual clues were messy.
The Strugglers:
- Chinese Text: Models struggled more with Chinese than English. The researchers explain that in English, a tear might cut through a word, but you can often guess the rest. In Chinese, a single character is a whole unit of meaning; if you cut a character in half, the meaning is often lost completely, making it much harder for the AI to guess.
- Code: Computer code was very difficult. Code relies on strict rules (like indentation and brackets). When the paper is shredded, those visual rules get broken, and the AI gets confused about which line of code comes next.
- Tables: Tables are like grids. When you shred a grid, the rows and columns get mixed up. Even the best models had a hard time putting the cells back in the right 2D arrangement.

5. What the AI Actually Did (Successes and Failures)

Success: In some cases, the AI was amazing. If a word like "school" was cut in half between "sch" and "ool," the AI didn't just read the fragments; it used its knowledge of language to "bridge the gap" and realize the word was "school."
Failure: The AI often failed at ordering. In a story, context tells you what comes next. In code, logic tells you. The AI often got the lines of code mixed up, putting the "end" of a program before the "beginning," because it couldn't see the logical flow through the visual chaos.

The Bottom Line

The paper concludes that while AI is great at reading clean documents, it currently lacks the fine-grained reasoning needed to reconstruct physically broken information. It treats the paper scraps as separate, isolated islands rather than parts of a single, cohesive story.

The researchers built this benchmark (ShredBench) not to sell a product, but to show the scientific community: "Hey, our AI is smart, but it still struggles when the world gets messy and broken." It highlights a specific gap in how these models connect what they see with what they know.

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) have achieved state-of-the-art performance in Visually Rich Document Understanding (VRDU) tasks, such as Optical Character Recognition (OCR) and information extraction. However, these evaluations typically rely on pristine, well-structured document images.

The paper addresses a critical gap: the ability of MLLMs to reconstruct documents that are physically fragmented (shredded). This task is distinct from traditional jigsaw puzzles because:

Visual Cues are Ambiguous: Unlike natural images with clear edge textures, shredded documents often have uniform backgrounds and sparse text, making edge-matching insufficient.
Semantic Reasoning is Required: Successful reconstruction demands integrating visual pattern recognition with deep semantic reasoning (e.g., syntactic expectations in code, logical flow in news, or structural alignment in tables) to bridge content discontinuities.
Real-World Relevance: Real-world document processing often involves damaged, torn, or occluded inputs, a scenario currently underexplored in benchmarking.

2. Methodology: The ShredBench Framework

The authors propose SHREDBENCH, a comprehensive benchmark and an automated generation pipeline designed to stress-test MLLMs.

A. Data Generation Pipeline

The pipeline ensures evaluation validity by generating fragmented documents directly from Markdown sources, preventing training data contamination. It consists of three stages:

Data Collection: A diverse corpus of 756 documents across four domains:
- English & Chinese News: Sourced from China Daily and People's Daily (800–2,500 characters).
- Source Code: Python, C++, and Java snippets (1KB–4KB) from GitHub.
- Scientific Tables: Complex 2D structures from the SWHL dataset.
Shredding Simulation (Physics-Based):
- Rendering: Raw text is rendered into high-resolution images (1600px) with simulated paper textures (noise, fonts).
- Voronoi Cutting: Instead of simple grid cuts, the system uses Voronoi tessellation with $N$ seed points ( $N \in \{8, 12, 16\}$ ) to create irregular, jagged boundaries mimicking manual shredding.
- 3D Synthesis: Fragments are imported into Blender for physical simulation. A Solidify modifier, displacement maps (Marble/Musgrave textures), and global illumination are applied to create realistic shadows, crumpling, and spatial depth.
Task Formulation: The task is defined as a set-to-sequence problem. The model receives an unordered set of scattered image fragments (potentially rotated) and must output the reconstructed text string, implicitly solving the jigsaw puzzle and performing OCR.

B. Evaluation Metrics

The benchmark employs four primary metrics to assess reconstruction quality:

NED (Normalized Edit Distance): Measures general text similarity (lower is better).
TEDS (Tree-Edit-Distance-based Similarity): Specifically for tables, assessing structural accuracy (higher is better).
BLEU & ROUGE-L: Standard metrics for n-gram precision and longest common subsequence to evaluate fluency and sentence structure.

C. Experimental Setup

The authors evaluated 14 representative MLLMs, including:

Proprietary: GPT-5 (Mini/5.1), Gemini 3 (Flash/Pro), Qwen-VL (Plus/Flash).
Open-Source: InternVL series, Mistral3-Reasoning, GLM-4.6v, DeepSeek-OCR, Hunyuan-OCR.

3. Key Contributions

First Specialized Benchmark: Introduced SHREDBENCH, the first benchmark specifically designed to evaluate semantic reasoning capabilities of MLLMs via shredded content restoration.
Automated, Contamination-Free Pipeline: Developed a pipeline that renders fragmented documents from Markdown, allowing for the synthesis of diverse, unseen samples (English, Chinese, Code, Tables) with adjustable granularity (8, 12, 16 pieces).
Comprehensive Evaluation & Baselines: Established the first quantitative baselines across 14 models, revealing significant performance gaps and identifying specific failure modes in handling visual structural noise.

4. Key Results & Findings

A. Performance Collapse on Fragmented Data

While models perform well on intact documents, their capabilities collapse significantly when documents are shredded.

NED Degradation: As fragmentation increases (8 $\to$ 16 pieces), NED scores rise sharply (indicating worse performance). In the hardest setting (16 fragments), the average NED reached 0.73, even for top models.
Reading Order Errors: Models frequently fail to identify the correct reading order or hallucinate non-existent bridging text.

B. Model Comparisons

Top Performer: Gemini 3 Pro demonstrated the strongest resilience, achieving the lowest NED (0.33) and highest ROUGE (0.83) at the 8-fragment level. It maintained a "flatter" decay curve as fragmentation increased compared to others.
Language Disparity: Models performed significantly worse on Chinese News than English News. The authors attribute this to the high information density of Chinese characters; a physical tear through a single character often obliterates its semantic identity, whereas Latin scripts have more redundancy across multi-letter words.
Code vs. Prose: Structured languages with explicit delimiters (Java, C++) were easier to reconstruct than Python, which relies on whitespace for indentation. Shredding disrupts spatial layout, making Python's logical scope difficult to infer.
Table Reconstruction: This was the most challenging domain. Interestingly, Gemini 3 Flash outperformed Gemini 3 Pro on tables, suggesting Flash's architecture may be better optimized for rigid 2D spatial structures, while Pro prioritizes semantic flow.

C. Qualitative Analysis

Success Cases: Top models (e.g., Gemini 3 Pro) showed visual semantic bridging, successfully recovering words split across fragments (e.g., "sch" + "ool" $\to$ "school") by synthesizing disjointed visual cues.
Failure Modes:
- Logical Misalignment: In code, models often swapped the order of independent statements because visual cues were ambiguous.
- Hallucinated Deletion: Models frequently omitted narrow strips of text (e.g., single lines of code), treating them as visual noise rather than integrating them into the context.

5. Significance and Future Directions

Critical Gap Identified: The study highlights that current MLLMs lack the fine-grained cross-modal reasoning required to bridge visual discontinuities. They tend to treat fragments as independent entities rather than parts of a cohesive whole.
Research Impact: SHREDBENCH provides a rigorous probe for evaluating the robustness of MLLMs in real-world, non-ideal conditions, moving beyond standard OCR tasks to complex structural reasoning.
Limitations: The current study assumes 2D flat fragments and regular cuts. Future work should address 3D physical complexities (folding, overlapping, occlusion) and irregular tearing mechanics found in real-world destruction.

In conclusion, SHREDBENCH reveals that while MLLMs are powerful, they are not yet robust enough to handle physically fragmented documents without significant performance degradation, particularly in structured data and non-Latin scripts. This benchmark sets a new standard for evaluating the next generation of robust VRDU models.