💻 computer science

SciPostLayoutTree: A Dataset for Structural Analysis of Scientific Posters

This paper introduces SciPostLayoutTree, a dataset of approximately 8,000 annotated scientific posters designed to address the underexplored challenge of structural analysis in poster layouts, alongside a novel Layout Tree Decoder model that leverages visual and bounding box features to accurately predict complex spatial relations like reading order and parent-child hierarchies.

Original authors: Shohei Tanaka, Atsushi Hashimoto, Yoshitaka Ushiku

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shohei Tanaka, Atsushi Hashimoto, Yoshitaka Ushiku

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a library and see a massive, colorful poster on the wall. It's packed with charts, bullet points, photos, and paragraphs of text. To a human, it's easy to figure out where to start reading and how the ideas connect. You might start at the big title at the top, move down to the "Methods" section, glance at a graph on the right, and then jump back to the conclusion at the bottom.

But for a computer, this poster is just a chaotic mess of colored rectangles. It doesn't know that the graph on the right belongs to the paragraph on the left, or that you should read the author's names before the abstract. It sees a jigsaw puzzle where all the pieces are mixed up.

This paper introduces a solution to teach computers how to "read" these scientific posters the way humans do. Here is the breakdown in simple terms:

1. The Problem: The Computer is Lost

Scientists have spent years teaching computers to read standard documents like PDFs or Word files. These are usually straight lines of text, top-to-bottom. But scientific posters are different. They are visual. They have:

Upward jumps: You read a section, then look up at a chart above it.
Side-to-side jumps: You read a column on the left, then jump to the right column.
Long leaps: You read the title, then skip way down to the conclusion.

Existing computer models get confused by these "spatial jumps." They try to read everything in a straight line and end up making a mess of the story.

2. The Solution: A New Map (The Dataset)

The authors created a massive new training map called SciPostLayoutTree.

The Scale: They collected about 8,000 real scientific posters.
The Annotation: Humans carefully drew lines connecting every box on the poster. They decided: "This box is the parent of that box," and "You read this box before that one."
The Tree: They organized this information into a Tree Structure. Imagine a family tree. The "Root" is the whole poster. The "Children" are the sections (like Introduction, Methods). The "Grandchildren" are the specific paragraphs or images inside those sections.
The Twist: Unlike standard documents, this tree has branches that go up, sideways, and far away. It's a much more complex map than what computers are used to.

3. The New Brain: Layout Tree Decoder

To read this complex map, the authors built a new AI model called the Layout Tree Decoder. Think of this model as a detective with two special tools:

Tool A: The "Eyes" (Visual Features): The model looks at the picture. It sees that a box contains a pie chart or a bold title.
Tool B: The "Ruler" (BBox Features): This is the secret sauce. The model also looks at the coordinates (where the box is) and the category (what kind of box it is).
- Analogy: Imagine trying to find your way in a city. "Eyes" help you recognize the Eiffel Tower. "The Ruler" tells you, "The Eiffel Tower is 5 blocks north of the hotel." By combining what it sees with where things are, the model understands that a graph on the right is related to text on the left, even if they aren't touching.
Tool C: The "Thinker" (Beam Search):
- The Old Way (Greedy Decoding): Imagine a hiker who always picks the path that looks best right now. If the path looks slightly downhill, they take it, even if it leads to a dead end. This is how old models worked. They made a mistake early on and couldn't recover.
- The New Way (Beam Search): This is like a hiker who keeps multiple paths open in their mind at the same time. "Maybe I should go left, but maybe right is better? Let's explore both possibilities for a few steps before deciding." This allows the model to realize, "Oh, going straight down was a trap; the correct path actually went up!"

4. The Results: A Smarter Reader

When they tested this new system:

It got much better at understanding those tricky "upward" and "sideways" jumps that confused older models.
It created a much more accurate "story flow" of the poster.
It proved that to understand posters, you can't just look at the words; you have to understand the layout and the geometry of the page.

Why Does This Matter?

If a computer can finally "read" a poster correctly, it can:

Summarize research for you instantly.
Answer questions like, "What was the result of the experiment shown in Figure 3?"
Read it aloud to visually impaired researchers in the correct order, not just randomly.

In short, this paper gave computers a new pair of glasses and a better map, allowing them to finally understand the complex, visual language of scientific posters.

1. Problem Statement

Scientific posters are a primary medium for academic communication, yet they remain significantly underexplored in the field of structural analysis compared to traditional textual documents (e.g., research papers). Existing Document Structure Analysis (DSA) research primarily focuses on linear, text-heavy documents where reading order is predominantly top-to-bottom and hierarchical structures are relatively shallow.

Scientific posters present unique challenges:

Complex Spatial Relations: Posters often utilize non-linear layouts, including upward reading flows, horizontal transitions, and long-distance dependencies between elements.
Visual Density: Posters contain a high density of figures, tables, and lists, requiring heavy reliance on visual features rather than just text.
Lack of Data: There was no large-scale dataset annotated with both reading order and parent-child hierarchical relations specifically for scientific posters.

The core problem addressed is the inability of current ML models to accurately infer the Depth-First Search (DFS) ordered tree structure (comprising reading order and parent-child relations) of scientific posters due to these spatial complexities and the lack of appropriate training data.

2. Methodology

A. Dataset Construction: SciPostLayoutTree

The authors constructed SciPostLayoutTree, a novel dataset containing approximately 8,000 scientific posters.

Annotation: Each poster is annotated with bounding boxes (BBoxes) categorized into 8 classes (Title, Author Info, Section, Text, List, Figure, Table, Caption).
Structure: The annotations form a DFS-ordered tree where:
- Reading Order: A sequence $\Phi$ representing the logical flow of content.
- Parent-Child Relations: A tree structure where a "Root" node represents the poster, and other nodes are linked based on containment or logical grouping (e.g., a Figure is a child of a Section).
Statistical Properties: Compared to existing document datasets (like DocHieNet), SciPostLayoutTree features:
- More BBoxes per page (avg. ~25 vs. ~12).
- Shallower but wider trees.
- A significantly higher frequency of spatially challenging relations: upward, horizontal, and long-distance transitions.

B. Model Architecture: Layout Tree Decoder

To address the challenges posed by the dataset, the authors developed Layout Tree Decoder, an extension of the existing Document Relation Graph Generator (DRGG) model. Key architectural innovations include:

Multimodal Feature Integration:
- Visual Features: Extracted via a Visual Feature Extractor (VFE) using backbones like ResNet, ViT, or Swin, followed by ROI Align.
- BBox Embedding: A critical addition where normalized coordinates (position) and category information are embedded and concatenated with visual features. This explicitly encodes spatial and categorical context, which is vital for resolving complex layouts.
Relation Prediction:
- The model predicts two score matrices:
  - Subsequent Score ( $S$ ): Probability that BBox $j$ follows BBox $i$ in reading order.
  - Parent Score ( $P$ ): Probability that BBox $j$ is the parent of BBox $i$ .
Decoding Algorithm with Beam Search:
- Unlike previous works that used greedy decoding (selecting the highest local score at each step), the authors implemented Beam Search for both reading order and parent-child prediction.
- Rationale: Greedy decoding fails on spatially challenging relations because they often have lower local scores than standard downward transitions. Beam search maintains multiple candidate sequences, allowing the model to capture sequence-level plausibility and recover from local errors.

3. Key Contributions

SciPostLayoutTree Dataset: The first large-scale dataset (8k posters) annotated with reading order and parent-child relations, specifically highlighting the prevalence of spatially challenging structural patterns.
Identification of Novel Challenges: The paper demonstrates that poster structures introduce technical difficulties (upward/horizontal/long-distance relations) not typically found in standard document analysis tasks.
Layout Tree Decoder: A robust baseline model that integrates BBox embeddings and beam search.
- BBox Embeddings significantly improve parent-child prediction accuracy by leveraging spatial context.
- Beam Search significantly improves reading order prediction, particularly for non-standard directional flows.

4. Experimental Results

The model was evaluated using Tree Edit Distance (TED), Semantic Tree Edit Distance Score (STEDS), and Reading Edit Distance Score (REDS) across five different visual backbones (ResNet-50, ViT, Swin, DiT, InternImage).

Overall Performance: The full model (DRGG-BEBS, combining BBox Embedding and Beam Search) achieved the highest scores across all metrics.
- STEDS: Improved from 68-80 (baseline) to **88-90**.
- REDS: Improved from 75-85 to **90-91**.
Impact of Components:
- BBox Embedding: Provided the most significant gain in STEDS (parent-child accuracy), proving that spatial/categorical features are crucial for hierarchical structure.
- Beam Search: Provided the most significant gain in REDS (reading order), confirming its ability to handle sequential dependencies and non-local transitions.
Spatial Analysis:
- The model showed substantial accuracy improvements on upward and horizontal relations compared to the baseline.
- Long-distance relations (distance > 8) saw significant accuracy boosts with beam search.
Error Analysis:
- Remaining errors often involve misidentifying the "Root" as a parent for nodes that should belong to a "Section," or failing to capture semantic grouping in complex Z-shaped layouts.
- Text embeddings (OCR) were tested but provided no significant improvement, suggesting that for posters, visual layout and spatial cues are more dominant than textual content for structural inference.

5. Significance and Future Directions

Foundation for Poster Intelligence: This work establishes a foundational framework for enabling machines to "read" and understand scientific posters, facilitating applications like automated text-to-speech for posters, content retrieval, and layout assessment.
Shift in Paradigm: It highlights that structural analysis for visual-rich documents requires moving beyond simple top-down assumptions and incorporating explicit spatial reasoning and sequence-level planning (beam search).
Baseline for Future Research: The dataset and the Layout Tree Decoder provide a solid, reproducible baseline for the community to tackle the specific challenges of scientific poster layouts.

In conclusion, the paper successfully bridges the gap between document structure analysis and scientific poster understanding by providing a specialized dataset and a model architecture tailored to the unique spatial and hierarchical complexities of the poster medium.