SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding

1. Problem Statement

The paper addresses the significant challenges in understanding multi-page, multi-modal visual documents (e.g., slide decks, brochures, manuals, financial reports). While Multimodal Large Language Models (MLLMs) have advanced document understanding, they face three critical limitations when applied to complex slide decks:

1. Lack of Scalable Fine-Grained Reasoning: Current MLLMs often process pages holistically or are limited in the number of images they can ingest. They struggle to isolate specific visual elements (e.g., a single chart segment within a cluttered slide) required for precise reasoning, leading to errors in counting or data extraction.
2. Deficits in Domain-Specific Visual Semantics: Pre-trained on natural images, MLLMs often fail to interpret domain-specific visual cues found in documents, such as financial color coding (red/green for loss/gain), abstract icons, spatial hierarchies, and the distinction between decorative elements (logos) and content.
3. Metadata-Free Integration Issues: Many existing systems rely on clean metadata (text layers, hierarchy tags) which is often unavailable in real-world scenarios (e.g., flattened PDFs, screenshots, scanned documents). Metadata-free methods often lack the structural understanding needed for complex reasoning.

2. Methodology: SlideAgent Framework

SlideAgent is a hierarchical, agentic framework designed to decompose reasoning into three specialized levels: Global, Page, and Element. It operates in two distinct stages: Knowledge Construction and Inference.

A. Knowledge Construction Stage

In this offline stage, SlideAgent builds a structured, query-agnostic knowledge base ($K = \{K_g, K_p, K_e\}$) using specialized agents:

• Global Agent ($M_g$): Analyzes the document (sampling the first few pages initially) to generate a high-level summary, including the document's purpose, narrative flow, and overarching themes. This knowledge is later refined using evidence from all pages to ensure comprehensive coverage.
• Page Agent ($M_p$): Processes each page sequentially. It generates page-level knowledge ($K_p$) that captures specific content while maintaining cross-page context (linking to the previous page's summary and the global context). This creates a coherent narrative across the deck.
• Element Agent ($M_e$): This is the core innovation for fine-grained reasoning. It uses a layout parsing pipeline (integrating OCR, layout detection, and classification) to decompose each page into discrete elements (text blocks, charts, images, icons). For each element, it generates metadata including:
  • Bounding box coordinates.
  • Verbatim text or visual description.
  • Semantic role and functional purpose.
  • Relation to the slide's overall message.
  • Note: It includes a graph-based merging algorithm to reconstruct fragmented text spans into coherent blocks.

B. Inference Stage

When a user query ($q$) is received, the system performs retrieval and reasoning:

1. Query Classification: An orchestrator classifies the query into one of four types (Global Understanding, Fact-based, Multi-hop Reasoning, Layout/Visual Relationships) to determine which agents to activate.
2. Subquery Generation & Retrieval: The system generates subqueries ($\hat{Q}$) targeting key entities. It retrieves relevant pages ($\hat{P}$) and elements ($\hat{E}$) from the constructed knowledge base using text-based or multimodal retrievers.
3. Hierarchical Reasoning:
   • The Global Agent provides broad context.
   • The Page Agent narrows down to specific slides and cross-slide relationships.
   • The Element Agent grounds the answer in specific visual/textual components (e.g., reading a specific bar in a chart).
4. Answer Synthesis: An answer synthesizer ($\phi$) combines the reasoning traces and outputs from the activated agents. If agents agree, the answer is finalized; otherwise, the synthesizer resolves conflicts to produce the final answer ($a$).

3. Key Contributions

• Hierarchical Agentic Architecture: Proposes a novel three-level decomposition (Global, Page, Element) that mimics human information processing, allowing for both broad context and fine-grained detail.
• Metadata-Free Element Parsing: Introduces a robust method for parsing visual documents without relying on clean metadata, making it effective for real-world, flattened, or scanned documents.
• Structured Knowledge Construction: Moves beyond simple retrieval by generating structured, query-agnostic knowledge representations ($K_g, K_p, K_e$) that serve as a "cache" for multiple queries.
• Enhanced Spatial Reasoning: Explicitly addresses the weakness of MLLMs in spatial reasoning by isolating elements and their bounding boxes, enabling precise counting and location-based queries.

4. Experimental Results

The authors evaluated SlideAgent on multiple datasets (SlideVQA, TechSlides, FinSlides, InfoVQA) against 15 MLLMs (including GPT-4o, Gemini, Claude, Llama, InternVL) and existing RAG/Agentic baselines.

• Performance Gains:
  • Proprietary Models: SlideAgent improved accuracy by +7.9% over the base GPT-4o model.
  • Open-Source Models: It achieved a +9.8% improvement over the base InternVL3-8B model.
  • State-of-the-Art: SlideAgent outperformed all baseline RAG and multi-agent systems (e.g., VisRAG, VDocRAG, ViDoRAG) across all metrics.
• Query Type Analysis: The framework showed the most significant improvements in Multi-hop Reasoning (+9.8%) and Visual/Layout Reasoning (+7.7%), demonstrating its ability to handle complex, cross-element queries that standard MLLMs fail at.
• Efficiency: Despite the multi-agent overhead, SlideAgent reduced input token usage by >90% (via retrieval of relevant snippets rather than full documents) while maintaining competitive latency.
• Ablation Studies: Removing the Page Agent caused the steepest performance drop, highlighting the critical importance of cross-slide coherence. Element-level reasoning was also crucial for numeric precision.

5. Significance and Future Work

Significance:
SlideAgent represents a paradigm shift from "end-to-end" MLLM processing to structured, hierarchical reasoning for visual documents. It effectively bridges the gap between the high-level comprehension of LLMs and the precise, localized reasoning required for technical and financial documents. By operating without metadata, it is highly applicable to real-world enterprise scenarios where document structures are often degraded.

Future Work:
The authors suggest future improvements in:

• Robustness to low-contrast layouts.
• Modeling inter-element relations via graph structures.
• Replacing the "first-N-pages" heuristic with content-aware page selection.
• Extending the framework to multi-turn conversational scenarios.

In conclusion, SlideAgent provides a scalable, high-accuracy solution for multi-page visual document understanding, proving that decomposing the reasoning process into specialized, hierarchical agents significantly outperforms monolithic MLLM approaches.