Re:Verse -- Can Your VLM Read a Manga?

1. Problem Statement

Current Vision-Language Models (VLMs) exhibit a critical gap between surface-level recognition (identifying objects or text in a single image) and deep narrative reasoning (understanding sequential storytelling). While models excel at interpreting individual manga panels, they systematically fail at:

• Temporal Causality: Understanding cause-and-effect relationships across multiple panels.
• Cross-Panel Cohesion: Maintaining character consistency and tracking plot progression over long sequences.
• Non-Linear Narratives: Handling complex storytelling structures (e.g., time loops, flashbacks) common in manga.

Existing benchmarks (e.g., Manga109, MangaUB) are limited to single panels or short sequences, focusing on structural analysis or multiple-choice questions, failing to evaluate long-form narrative comprehension across entire chapters.

2. Methodology: The Re:Verse Benchmark

The authors introduce Re:Verse, a comprehensive benchmark designed to evaluate VLMs on sequential visual storytelling using the first arc of the popular manga Re:Zero - Starting Life in Another World.

Dataset Construction

• Scale: 308 manually annotated panels spanning 11 chapters.
• Alignment: Each manga panel is rigorously aligned with corresponding text from the original Re:Zero light novel.
• Annotation Schema:
  • Spatial: Fine-grained bounding boxes for text bubbles, thought bubbles, and scene elements.
  • Semantic Tags: Distinct tags for Dialogue (<D>) and Internal Thoughts (<T>).
  • Ground Truth: Manual curation ensures precise narrative correspondence between visual and textual modalities.

Evaluation Framework

The study evaluates 8 state-of-the-art open-source VLMs (e.g., InternVL, Ovis, Qwen2.5-VL) across three core axes:

1. Story Synthesis (Generative Tasks):
   • Story Generation: Converting manga chapters into coherent prose.
   • Summary Generation: Producing concise plot summaries.
   • Metrics: NER density (character consistency), STTR (style/tone), BERTScore, ROUGE, and a novel penalty-based narrative quality metric (0–100 scale) evaluating Character, Plot, Setting, and Coherence.
2. Character Grounding (Attribution Tasks):
   • Extracting dialogue/thoughts and attributing them to specific characters based on visual cues (e.g., speech bubble proximity, character gaze).
   • Metrics: Precision, Recall, F1-Score, and Character Attribution Accuracy.
3. Temporal Reasoning:
   • Next Page Prediction (NPP) & Intermediate Page Prediction (IPP): Predicting missing pages in a sequence to test causal inference.
   • Visual Question Answering (VQA): 55 complex questions requiring cross-page information retrieval.
   • Retrieval-Augmented Generation (RAG): Testing if external knowledge retrieval mitigates reasoning failures.

3. Key Results

The evaluation reveals that current VLMs lack "genuine story-level intelligence," struggling significantly with discrete temporal structures.

• Character Consistency Failure:
  • Models show a 3x to 10x gap in Named Entity Recognition (NER) density compared to human gold standards (Model NER: ~0.01–0.02 vs. Human: 0.087).
  • Character Attribution Accuracy is near-zero (0.00% – 1.11%). Even models that successfully extract dialogue text fail to identify who is speaking.
• Visual-Textual Integration Bottleneck:
  • While spatial localization (IoU) is reasonable (~0.60), text extraction F1 scores are critically low (<0.25).
  • Automated metrics (BERTScore) show minimal difference between text-only and image-text inputs, suggesting current metrics are insufficient for evaluating visual narrative understanding.
• Temporal Reasoning Degradation:
  • Overall VQA accuracy is 28.5% (1.43/5.0).
  • Performance degrades as narrative complexity increases; models perform worst on the final chapters (Ch 11) where plot threads converge.
  • RAG Limitations: Adding external knowledge retrieval reduces total failures but does not improve high-quality reasoning, indicating the issue is architectural (processing discrete sequences) rather than a lack of knowledge.
• Semantic Discontinuity:
  • Vision encoders (CLIP, SIGLIP, etc.) struggle to maintain semantic continuity between consecutive pages. Similarity scores often drop to near-zero or negative values, confirming that models treat pages as isolated images rather than a continuous narrative.
• Architecture vs. Scale:
  • Performance is more dependent on architecture design than parameter count. For instance, InternVL3-8B outperformed larger models from other architectures in temporal tasks, suggesting specialized mechanisms for sequential processing are crucial.

4. Key Contributions

1. Re:Verse Benchmark: The first dataset and evaluation framework specifically designed for chapter-length manga narrative understanding, featuring aligned visual-textual data and fine-grained semantic tags.
2. Systematic Limitation Analysis: The first study to quantify the "inferent gap" in VLMs, demonstrating that current models fail to maintain character consistency, dialogue attribution, and temporal causality in long-form sequences.
3. Novel Evaluation Methodology: Introduction of a penalty-based, state-incremental evaluation system (SCORE framework extension) that provides fine-grained scoring (0–100) for narrative quality, moving beyond binary pass/fail metrics.
4. Quantification of the Inferent Gap: Empirical evidence showing that models struggle to bridge narrative discontinuities, with counter-intuitive findings (e.g., some models performing better on 3-missing-page scenarios than 2-missing-page ones).

5. Significance

• Research Impact: Re:Verse establishes a new standard for evaluating discrete visual narratives, shifting the focus from static image recognition to dynamic, sequential reasoning.
• Architectural Insights: The findings suggest that increasing parameter count alone is insufficient; future VLMs require architectural innovations specifically targeting temporal coherence and cross-modal binding over extended contexts.
• Cultural & Practical Application: The benchmark addresses a gap in understanding culturally significant media (manga/comics), paving the way for AI assistants capable of editing, summarizing, or expanding visual stories for creators and researchers.
• Open Science: The dataset and code are released to foster reproducible research in sequential visual storytelling.

In conclusion, the paper argues that while VLMs have mastered individual image understanding, they currently lack the narrative intelligence required to comprehend the complex, non-linear, and temporally dependent nature of manga storytelling.