MoDora: Tree-Based Semi-Structured Document Analysis System

MoDora is an LLM-powered system that addresses the challenges of analyzing semi-structured documents by employing a local-alignment aggregation strategy, a Component-Correlation Tree for hierarchical organization, and a question-type-aware retrieval mechanism to significantly improve accuracy in natural language question answering.

The Gist

Imagine you have a massive, messy library. But this isn't a normal library with books neatly stacked on shelves. This library is filled with semi-structured documents.

Think of these documents like a chaotic collage: they have paragraphs of text, but also tables, charts, sidebars, footnotes, and images all mixed together on the same page. Sometimes a table is on page 1, but the explanation for it is on page 5. Sometimes a chart is floating in the middle of a paragraph.

The Problem:
If you ask a human librarian, "What was the feather score for the winter experiment?" they can easily flip through the pages, find the paragraph about "winter," look at the table on the next page, and give you the answer.

But if you ask a computer (specifically, current AI models) this question, it often gets lost.

• Old Method 1 (The Scanner): It reads the text but ignores the layout. It sees the word "winter" and the word "table" but doesn't know they are connected. It's like reading a recipe where the ingredients list is on page 1 and the cooking instructions are on page 10, but the book is torn up and shuffled.
• Old Method 2 (The Image Viewer): It looks at the whole page as a picture. It might see the table, but it misses the tiny text saying "this experiment happened in winter." It's like looking at a map but ignoring the legend.
• Old Method 3 (The Search Engine): It grabs random chunks of text that sound similar to your question but misses the big picture. It's like finding a sentence about "feathers" in a completely different chapter.

The Solution: MoDora
The authors of this paper built a new system called MoDora. Think of MoDora not as a scanner or a search engine, but as a super-organized architect who rebuilds the messy library into a perfect, logical treehouse.

Here is how MoDora works, step-by-step:

1. The "Local-Alignment" Strategy (Grouping the Clutter)

Imagine you walk into a room where someone has thrown all the furniture, books, and pictures on the floor.

• What MoDora does: It doesn't just look at individual items. It says, "Okay, this title belongs with these three paragraphs. This chart belongs with that specific table."
• The Analogy: It's like a detective gathering clues. It groups the "Title" with the "Story" and the "Chart" with its "Data" into neat, self-contained bundles called Components. It ignores the messy page numbers and footers for a moment to focus on the meaningful groups.

2. Building the "CCTree" (The Organized Treehouse)

Once the items are grouped, MoDora builds a Component-Correlation Tree (CCTree).

• The Analogy: Imagine a family tree, but instead of people, it's made of document sections.
  • The Root is the main title of the document.
  • The Branches are the chapters (Introduction, Methods, Results).
  • The Leaves are the specific details (a paragraph, a table, a chart).
• Why this helps: In a normal document, a table might be far away from the text explaining it. In MoDora's tree, the table is a "child" of the text that explains it. The system knows, "Ah, this chart is a child of the 'Experiment Design' branch." It preserves the hierarchy (who is the boss of what) and the layout (where things are on the page).

3. The "Bottom-Up Summarization" (The Smart Summary)

Now, imagine you are the boss of this treehouse. You don't want to read every single leaf to know what's happening.

• What MoDora does: It starts at the bottom (the leaves) and asks the AI, "What is the main point of this paragraph?" Then it moves up one level and asks, "What is the main point of this chapter, including the summary of the paragraph below?"
• The Analogy: It's like a news network. The reporter at the scene (the leaf) sends a quick summary to the editor (the branch), who sends a summary to the anchor (the root). By the time the question reaches the top, the system has a "map" of the whole document without needing to read every single word again.

4. The "Question-Type-Aware" Detective (Finding the Answer)

When you ask a question, MoDora doesn't just search for keywords. It acts like a detective with a specific plan based on the type of question:

• If you ask "Where is the logo?" (Location Question): MoDora looks at the "map" of the tree. It knows the logo is in the "Header" branch, specifically in the top-right corner of Page 1. It goes straight there.
• If you ask "What did the experiment find?" (Semantic Question): MoDora uses its "Smart Summary" (the CCTree) to skip irrelevant branches. It says, "The 'Introduction' branch isn't relevant, but the 'Results' branch is." It then zooms in on the specific table or paragraph needed.
• The Safety Net: If the AI isn't 100% sure, it uses a "Verifier" (another AI) to double-check the evidence before giving an answer, ensuring it doesn't make things up (hallucinate).

The Result

In tests, MoDora was much better at answering questions about these messy documents than previous methods.

• Old methods were like trying to find a needle in a haystack by guessing.
• MoDora is like having a magnet that knows exactly where the needle is, because it understands the structure of the haystack.

In short: MoDora takes a chaotic, multi-page document, organizes it into a logical family tree, summarizes the relationships, and then uses that structure to find the exact answer you need, whether it's a number in a table or a location on a page. It turns a messy collage into a clear story.

Technical Summary

1. Problem Statement

Semi-structured documents (e.g., scientific reports, financial statements, technical manuals) contain diverse, interleaved data elements such as tables, charts, hierarchical paragraphs, and images arranged in complex, often irregular layouts. While these documents are ubiquitous, existing methods struggle to support Natural Language Question Answering (NLQA) over them due to three primary technical challenges:

1. Fragmented Context: Elements extracted via OCR are often isolated and stripped of their original semantic context (e.g., a table detached from its title or a chart separated from its description), making independent analysis inadequate.
2. Lack of Structural Representation: Current approaches fail to effectively capture hierarchical structures (e.g., associating a table with a nested chapter title) or preserve layout-specific distinctions (e.g., differentiating sidebars from main content).
3. Scattered Evidence Retrieval: Answering complex questions often requires retrieving and aligning information scattered across multiple regions or pages (e.g., linking a paragraph describing an experiment on page 1 to a specific data cell in a table on page 2).

Existing methods fall short: Content extraction methods discard layout; structure extraction methods lack fine-grained modeling; End-to-End models (MLLMs) miss fine-grained details and layout relations; and standard RAG approaches fail to align structural and semantic cues.

2. Methodology: MoDora

MoDora is an LLM-powered system designed to analyze semi-structured documents through a three-stage pipeline: Document Preprocessing, Tree Construction, and Tree-Based Analysis.

A. Document Preprocessing

The system transforms raw OCR outputs into self-contained, semantically rich Components (COs) using a Local-Alignment Aggregation Strategy:

• Aggregation: It groups fragmented OCR elements into coherent components based on reading order and layout rules.
  • Text Aggregation: Merges titles with their corresponding paragraphs.
  • Non-Text Aggregation: Associates tables, charts, and images with their titles (or assigns default titles if missing).
  • Supplement Aggregation: Isolates headers, footers, and sidebars.
• Type-Specific Information Extraction:
  • Hierarchical Titles: Uses a format-aware LLM prompt (combining text and visual crops) to detect title levels (e.g., H1, H2).
  • Non-Text Elements: Employs a template-based approach to extract structured semantic triples (title, metadata, data) for tables and charts, enriching them with schema and representative values.

B. Tree Construction: Component-Correlation Tree (CCTree)

To model global structure and inter-component relations, MoDora constructs a CCTree:

• Nodes: Each node represents a document component.
• Edges: Three types of relationships are modeled:
  1. Text-to-Text: Based on title hierarchy (parent-child).
  2. Text-to-Other: Non-text elements (tables/charts) are linked as children to their surrounding text components.
  3. Independent Supplementary: Sidebars and footers are attached to a separate branch to prevent semantic interference.
• Bottom-Up Cascade Summarization: To enable efficient retrieval, the system recursively propagates summaries upward. It uses an information attenuation formula to determine the number of keywords generated for each node, ensuring higher-level nodes retain a concise yet comprehensive summary of their subtrees.

C. Tree-Based Document Analysis

MoDora employs a Question-Type-Aware Retrieval Strategy to locate evidence:

• Location-Based Retrieval: For spatial questions (e.g., "bottom of page 1"), it maps the query to a 3x3 grid system and retrieves nodes overlapping with those coordinates.
• Semantics-Based Retrieval: A two-step iterative process:
  1. Forward Search (Pruning): An LLM selects candidate nodes based on metadata and indices. Unselected subtrees are flattened and searched via embedding as a fallback to ensure high recall.
  2. Backward Verification: An MLLM verifier checks the detailed content (text + cropped image regions) of candidate nodes to filter out false positives, ensuring high precision.
• Answer Generation: The final answer is generated by an MLLM using the aggregated textual evidence, locational evidence (cropped regions), and the hierarchical context of the CCTree.

3. Key Contributions

1. MoDora System: A novel framework for semi-structured document analysis that supports diverse layouts and handles both semantic and location-based queries.
2. Local-Alignment Aggregation: A strategy to convert OCR elements into self-contained components with enriched hierarchical and non-text semantics.
3. Component-Correlation Tree (CCTree): A hierarchical representation that explicitly models inter-component relations and layout distinctions, supported by bottom-up summarization for efficient reasoning.
4. Question-Type-Aware Retrieval: A hybrid retrieval mechanism combining layout-based grid partitioning, LLM-guided pruning, embedding fallback, and cross-modal verification.
5. MMDA Benchmark: The introduction of a new benchmark (Multi-Modal Document Analysis) comprising 537 documents and 1,065 QA pairs, featuring higher element and layout diversity than existing datasets.

4. Experimental Results

The authors evaluated MoDora against eight baselines (including GPT-5, DocOwl2, ZenDB, TextRAG, and M3DocRAG) on standard datasets (DUDE, M3DocVQA, MP-DocVQA) and their new MMDA benchmark.

• Performance: MoDora outperformed all baselines across all datasets and metrics.
  • Accuracy Gains: It achieved improvements ranging from 5.97% to 61.07% over the best-performing baselines.
  • Specific Metrics: On the MMDA benchmark, MoDora achieved an AIC-Acc of 73.33%, significantly higher than the next best (GPT-5 at 66.20% or DocAgent at 65.50%).
• Question Types: MoDora excelled in all four categories:
  • Hierarchy: 76.73% (vs. 55.97% for DocAgent).
  • Text: 79.95%.
  • Hybrid: 68.00% (crucial for table/chart reasoning).
  • Location: 68.21% (leveraging grid mapping).
• Ablation Studies: Removing the tree structure caused a ~15% drop in accuracy, while removing component construction caused a ~37% drop, proving the necessity of structural modeling.
• Cost Efficiency: Despite using multiple LLM calls, MoDora's cost per query ($0.025) is competitive with simpler baselines and significantly cheaper than multi-round agent frameworks like DocAgent ($0.09), while offering much higher accuracy.

5. Significance

MoDora addresses a critical gap in document AI by moving beyond simple text extraction or flat image processing. Its significance lies in:

• Structural Reasoning: It demonstrates that explicitly modeling document hierarchy and layout relationships is essential for accurate reasoning in complex documents.
• Robustness: By combining LLM reasoning with structured tree traversal and embedding fallbacks, it mitigates the hallucination and retrieval failures common in pure RAG or End-to-End models.
• Practical Applicability: The system is designed to work with OCR outputs, making it applicable to both digital-native and scanned/rasterized documents without requiring native PDF structure.
• Benchmarking: The introduction of MMDA provides a more rigorous testbed for future research, highlighting the limitations of current datasets in capturing real-world document complexity.

In conclusion, MoDora establishes a new state-of-the-art for semi-structured document analysis by unifying layout awareness, hierarchical modeling, and advanced retrieval strategies.