🤖 machine learning

FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&A

FLOWREADER addresses the challenge of answering questions from fragmented multimodal long documents by reframing evidence assembly as a min-cost flow optimization problem on a node graph, which unifies scoring, routing, and adaptive compute to outperform top- $k$ retrieval baselines on benchmarks dominated by scattered evidence.

Original authors: Ambuj Mehrish, Sebatiano Vascon

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ambuj Mehrish, Sebatiano Vascon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery, but the clues are scattered across a massive library. Some clues are written on sticky notes (text), some are drawn on whiteboards (images), and others are hidden inside complex spreadsheets (tables). The problem is that these clues are often broken up: a single fact might be split between a chart on page 5 and a paragraph on page 12, or a table might be so wide it spans three different slides.

Most current AI systems act like a frantic librarian who grabs the top 5 most relevant-looking pages and hands them to a detective (the AI). If the answer requires connecting a chart on page 5 to a sentence on page 12, the librarian often misses the connection because they are looking at pages in isolation.

FLOWREADER is a new system that changes how the librarian works. Instead of just grabbing pages, it treats the entire library as a giant, interconnected map and uses a mathematical concept called "Minimum-Cost Flow" to find the best path to the answer.

Here is how it works, step-by-step:

1. The Map (The Multimodal Graph)

First, FLOWREADER builds a map of the document.

Nodes: Every piece of information (a paragraph, a cell in a table, a chart) is a "node" on the map.
Edges: Lines connect these nodes if they are related. A line might connect a chart to the text describing it, or a table header to the data rows below it.
The Goal: The system wants to find a path from the "Question" (where you start) to the "Answer" (where you end up) by traveling through the most useful clues.

2. The Traffic Flow (Min-Cost Flow)

Instead of just picking the "best" 5 pages, FLOWREADER treats the search like managing traffic on a highway.

The Budget: Imagine you have a fixed amount of "fuel" (a budget) to send a convoy of trucks from the start to the finish.
The Cost: Every road (connection between clues) has a "cost."
- If a road connects two very relevant, high-quality clues, the cost is low (easy to travel).
- If a road connects weak or irrelevant clues, the cost is high (hard to travel).
The Optimization: The system calculates the most efficient way to send your "fuel" to reach the answer. It naturally avoids dead ends and finds the smoothest, most logical chain of evidence, even if that chain jumps between text, tables, and images.

3. The Filter (Replicator Dynamics)

Once the system finds all the possible routes, it might have too many of them, and some might be duplicates.

Think of this like a reality TV show elimination round.
The system runs a game where different routes compete. The "winners" are routes that are both high quality (good clues) and diverse (not just repeating the same fact).
This ensures the final list of clues is short, non-redundant, and covers all the necessary angles.

4. The Double-Check (System 2 Gate)

Sometimes, the first attempt isn't enough. Maybe the clues are too fragmented, or the answers from different routes contradict each other.

The Gatekeeper: A smart gatekeeper checks the situation. If the "traffic" is stuck (low saturation) or the drivers (AI workers) are giving conflicting reports, the gate opens.
The Refinement: This triggers a "System 2" pass—a slower, more deliberate second look. The system might add a new bridge between two disconnected parts of the map or re-evaluate the clues.
Efficiency: Crucially, this second look only happens when absolutely necessary, saving time and computing power.

Why It Matters (The Results)

The paper tested this system on VisDoMBench, a benchmark full of difficult questions about scientific papers, slides, and tables where information is often broken apart.

The Problem with Old Methods: Traditional methods (Top-K retrieval) often fail here because they can't "see" the connection between a chart and a distant paragraph.
FLOWREADER's Success: By using the "traffic flow" method, FLOWREADER excelled at these fragmented tasks.
- It beat the previous best system on PaperTab (long tables) by a significant margin.
- It also improved performance on SlideVQA (slides with mixed text and images).
- Overall, it was highly competitive across all tests, proving that treating evidence assembly as a flow problem works better than just grabbing the top few chunks.

In Summary

FLOWREADER stops treating a document like a pile of separate pages. Instead, it sees the document as a living network. It uses math to route its "attention" like water flowing through pipes, finding the most efficient, connected path to the answer, and only doing extra work when the path looks shaky. This makes it much better at solving puzzles where the clues are scattered across different types of media.

Technical Summary: FLOWREADER

Problem Statement

Long, multimodal documents present a significant challenge for Retrieval-Augmented Generation (RAG) systems. Evidence required to answer complex queries is often fragmented across text, tables, charts, and slides, distributed across multiple cells, pages, or modalities. Traditional top- $k$ chunk retrieval approaches fail in this context because they treat evidence fragments independently. This independence prevents the system from representing structural connections, multi-hop dependencies, or the way evidence must be assembled from disjoint sources (e.g., a figure and its discussion, or a table header and a distant row). Furthermore, fixed retrieval policies allocate the same computational budget to queries of varying complexity, leading to inefficiencies.

Methodology

FLOWREADER reframes the evidence assembly problem as a minimum-cost flow (min-cost flow) optimization task on a multimodal node graph. The system operates through three structured phases:

1. Multi-Modal Graph Construction (Offline)

The document corpus is pre-processed into a directed, weighted, heterogeneous graph $G = (V, E)$ .

Nodes ( $V$ ): Partitioned into textual ( $V_{txt}$ ), visual ( $V_{vis}$ ), and table ( $V_{tbl}$ ) units. Each node contains an LLM-generated summary, indexing keywords, a modality tag, and an embedding.
Edges ( $E$ ): Directed edges carry weights encoding semantic affinity. The graph is constructed using a dual-evolving process (inspired by G2-Reader) that iteratively refines node attributes and adjacency based on semantic neighbors and layout.

2. Multi-Signal Node Scoring and Flow Routing (Online)

At query time, a single scalar score $h_i$ is computed for every node, aggregating three signals:

Query Relevance ( $r_i$ ): A fusion of dense cosine similarity and BM25 scores.
Structural Propagation ( $\phi_i$ ): Relevance propagated along the graph to capture nodes that are part of tight clusters of high-relevance nodes, even if their individual relevance is moderate.
Locality Diffusion ( $\psi_i$ ): A personalized PageRank score capturing longer-range connections within the query's topical neighborhood.

These scores drive a min-cost flow problem on an augmented network $G^*$ :

Sources and Sinks: Sources are selected from query-aligned nodes using Maximal Marginal Relevance (MMR) to ensure diversity. Sinks are selected based on an "answerability" score combining relevance and information density.
Costs and Capacities: Edge costs penalize semantically incoherent transitions and detours through low-scoring nodes. Capacities are bounded by the quality of the connected nodes.
Optimization: A linear program routes a fixed evidence budget $F$ from sources to sinks, minimizing total cost. The optimal flow is decomposed into a set of candidate evidence paths (chains).

3. Path Selection and Adaptive Synthesis

The decomposed flow yields many candidate paths, often with overlapping nodes. FLOWREADER selects a compact, non-redundant subset using entropy-regularized replicator dynamics (RD):

Quality-Diversity Game: Paths are treated as strategies in a game where the payoff matrix balances individual path quality against pairwise diversity (measured by Jaccard non-overlap).
Selection: RD iteratively updates the population of paths, converging to a weighted maximal clique that maximizes the quality-diversity trade-off.
Answer Synthesis: Selected paths are processed in parallel by Visual Language Model (VLM) workers.
Dual-Process Gating: A "System 1" pass generates initial answers. A gate monitors answer consistency across workers and flow saturation. If consistency is low or the flow is strained (indicating insufficient evidence), a single "System 2" refinement pass is triggered. This pass modifies the graph (e.g., adding bridge edges or re-weighting edges) and re-runs the pipeline once.

Key Contributions

Min-Cost Flow Formulation: The paper introduces a unified framework where a single parameterization (the node score $h$ ) controls source selection, sink selection, edge costs, and capacities. This collapses the traditionally separate stages of scoring, routing, and selection into a single optimization problem.
Handling Fragmented Evidence: By modeling evidence assembly as flow transport, the system explicitly captures multi-hop dependencies and structural connections that top- $k$ retrieval misses.
Adaptive Compute: The integration of a dual-process gate allows the system to allocate additional computational resources (a System 2 refinement pass) only when the initial retrieval and synthesis are deemed unreliable, balancing performance and cost.
Entropy-Regularized Selection: The use of replicator dynamics provides a principled, game-theoretic approach to selecting a diverse set of evidence paths, avoiding the collapse onto a single high-scoring but potentially redundant path.

Experimental Results

The system was evaluated on VisDoMBench, covering five multimodal subsets: SPIQA, FetaTab, PaperTab, SciGraphQA, and SlideVQA.

Performance on Fragmented Evidence: FLOWREADER achieved the best results on the two subsets dominated by fragmented evidence:
- PaperTab: 58.40 (a +1.30 improvement over the strong baseline G2-Reader).
- SlideVQA: 72.93 (a +0.62 improvement over G2-Reader).
Overall Performance: The macro-averaged score across all five subsets was 65.47, within 0.74 points of the strongest baseline (G2-Reader, 66.21).
Competitive Landscape: While FLOWREADER excelled on fragmented tasks, it was competitive but not superior on other subsets (e.g., trailing VisDoMRAG on SPIQA). The authors attribute the performance gap on SciGraphQA to the use of a smaller open-source encoder (nomic-embed-text-v1.5) compared to proprietary encoders used by baselines, and strict entity-anchored prompts that penalize paraphrased answers common in that dataset.
Efficiency: The system operates with a bounded VLM call budget. System 2 is triggered at most once per query, resulting in an average of 30.8–47.3 calls, compared to the unbounded replanning loops of some baselines.

Significance and Claims

The paper claims that min-cost flow is an effective mechanism for assembling fragmented multimodal evidence where traditional top- $k$ retrieval fails. By unifying scoring, routing, and selection under a single optimization objective, FLOWREADER provides a principled way to handle the structural complexity of long documents.

The authors emphasize that their approach offers a unified control mechanism for the entire retrieval pipeline, allowing for adaptive compute allocation without the need for complex, multi-stage tuning. The results suggest that in regimes where evidence is highly fragmented (tables, slides), structured routing is more critical than simply increasing the scale of the generator model. The work positions itself as a step toward moving RAG beyond fixed heuristics toward learned, structured processes for evidence assembly within controlled compute budgets.