💬 NLP

HELIOS: Harmonizing Early Fusion, Late Fusion, and LLM Reasoning for Multi-Granular Table-Text Retrieval

HELIOS is a novel framework for multi-granular table-text retrieval that harmonizes edge-based bipartite subgraph retrieval, query-relevant node expansion, and star-based LLM refinement to overcome the limitations of existing early and late fusion methods, thereby significantly improving performance on open-domain question answering benchmarks.

Original authors: Sungho Park, Joohyung Yun, Jongwuk Lee, Wook-Shin Han

Published 2026-03-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sungho Park, Joohyung Yun, Jongwuk Lee, Wook-Shin Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a complex mystery. You have two types of evidence: Tables (like a spreadsheet of facts) and Text Passages (like newspaper articles or encyclopedia entries). Your goal is to find the specific pieces of evidence needed to answer a tricky question.

The paper introduces a new detective system called HELIOS. To understand why HELIOS is special, let's look at how the "old detectives" (previous AI models) tried to solve these cases and where they failed.

The Problem with the Old Detectives

1. The "Bundle" Detective (Early Fusion)

How they worked: Imagine this detective grabs a whole folder of files the moment they see a name. If the question is about "Notre Dame," they grab the Notre Dame table and every single article ever written about Notre Dame, even the ones about its history or football team that aren't relevant to the specific question.
The Flaw: They get overwhelmed by too much noise. It's like trying to find a specific needle in a haystack, but the detective just dumped the entire barn on you. They also miss subtle clues because they didn't look at the question before grabbing the files.

2. The "Scattergun" Detective (Late Fusion)

How they worked: This detective looks at the question first, then grabs individual sentences or single rows from tables one by one.
The Flaw: They often miss the big picture. If the answer requires connecting a table row to a specific paragraph, this detective might grab the row but forget the paragraph, or grab the wrong paragraph because it looked similar on the surface. They struggle with "multi-hop" reasoning (connecting A to B, then B to C).

3. The "Literal" Detective (No Reasoning)

The Flaw: Both old detectives were bad at logic. If the question asked, "Who is the most recent player of the month?", they might just look for the word "recent" and miss the fact that they need to compare dates across the whole table to find the winner.

Enter HELIOS: The Master Detective

HELIOS is a new system that combines the best of both worlds and adds a "Chief Inspector" (a Large Language Model) to handle the hard logic. It works in three clever steps:

Step 1: The "Edge" Search (Finding the Right Connections)

Instead of grabbing whole folders (tables) or random sentences, HELIOS looks for edges.

Analogy: Imagine a map where tables and articles are islands. Old detectives grabbed whole islands. HELIOS looks for the specific bridges connecting a specific row in a table to a specific sentence in an article.
Why it helps: It avoids the "noise" of the Bundle Detective. It only grabs the exact bridge that might lead to the answer, ignoring irrelevant parts of the island.

Step 2: The "Seed" Expansion (Growing the Map)

Once HELIOS finds a few promising bridges, it doesn't stop. It asks, "Who else is connected to these bridges?"

Analogy: It's like following a trail of breadcrumbs. If you find a clue about "Brendan King," HELIOS doesn't just stop there; it expands the map to include the university he attended and the articles about that university's colors.
Why it helps: This fixes the "Scattergun" problem. It dynamically builds a small, perfect network of evidence that grows only as much as the question needs, ensuring no important context is left behind.

Step 3: The "Chief Inspector" (LLM Reasoning)

This is the magic sauce. HELIOS takes this small, perfect network of evidence and hands it to a super-smart AI (the LLM) to act as a Chief Inspector.

Analogy: The previous detectives just collected evidence. The Chief Inspector thinks about it.
- Aggregation: If the question is "Who is the most recent player?", the Chief Inspector looks at the whole table, compares the dates, and picks the winner.
- Verification: If the evidence is confusing, the Chief Inspector double-checks: "Does this article actually answer the question, or is it just a red herring?"
Why it helps: It solves the logic puzzles that the old systems couldn't handle.

The Result

When the researchers tested HELIOS on a giant database of tables and text (the OTT-QA benchmark), it was a game-changer.

It found the right answers 42% better than the best previous methods at the very top of the list.
It was much better at ranking the answers correctly, meaning the right answer was almost always near the top.

Summary

Think of HELIOS as a detective who:

Doesn't grab the whole library (avoids noise).
Doesn't just grab random pages (avoids missing context).
Builds a custom map of connections based on the specific question.
Hires a genius logician to solve the puzzle using that map.

By harmonizing these three approaches, HELOS solves the "Table-Text Retrieval" problem faster and more accurately than ever before.

1. Problem Definition

The paper addresses Table-Text Retrieval for Open-Domain Question Answering (ODQA). The goal is to retrieve relevant structured data (tables) and unstructured data (text passages) to answer complex queries that often require multi-hop reasoning and logical inference across modalities.

Existing methods generally fall into two categories, both of which have significant limitations:

Early Fusion: Pre-aligns table rows with associated passages (forming "fused blocks" or "stars") via entity linking before the query is presented.
- Limitation: Includes query-irrelevant contexts (noise) and misses query-dependent relationships that are not predefined.
Late Fusion: Dynamically retrieves individual table segments and passages based on query similarity.
- Limitation: Risks missing relevant contexts if the initial retrieval of a single node is incomplete; struggles with advanced reasoning tasks like column-wise aggregation or multi-hop inference that require structural understanding.

Core Challenge: Current systems fail to balance retrieval granularity, miss dynamic query-specific relationships, and lack the capability to perform advanced logical reasoning (e.g., finding the "most recent" entry in a table or aggregating data across rows).

2. Methodology: The HELIOS Framework

HELIOS proposes a novel graph-based retrieval framework that harmonizes early fusion, late fusion, and Large Language Model (LLM) reasoning. It models the retrieval corpus as a bipartite graph where nodes represent table segments and passages, and edges represent their relationships.

The framework operates in three distinct stages:

Stage 1: Edge-Based Bipartite Subgraph Retrieval (Early Fusion + Fine-Grained Retrieval)

Offline Construction: A bipartite data graph ( $G_d$ ) is constructed by linking table segments and passages via entity linking.
Edge-Level Retrieval: Unlike previous methods that retrieve "nodes" (entire rows) or "stars" (rows + all linked passages), HELIOS retrieves edges (specific table segment–passage pairs).
Mechanism:
1. Edges are linearized and embedded using a multi-vector encoder (ColBERTv2).
2. The system retrieves top- $k$ edges based on query-edge similarity.
3. A reranker refines these edges to form a candidate bipartite subgraph ( $G_c$ ).
Benefit: This granularity avoids the noise of retrieving entire "stars" while preventing the partial information loss of retrieving isolated nodes.

Stage 2: Query-Relevant Node Expansion (Late Fusion)

Goal: To address the limitation of early fusion missing query-dependent relationships not present in the initial graph.
Mechanism:
1. Seed Selection: The system identifies the most query-relevant nodes within the candidate subgraph ( $G_c$ ) using a node reranker.
2. Beam Search Expansion: Using an expanded query retrieval technique, the system dynamically searches the full corpus for new edges connected to these seed nodes.
3. It calculates the probability $p(u, v|q)$ to select the most promising new edges, expanding the graph into a larger structure ( $G_l$ ).
Benefit: Dynamically incorporates relationships that emerge only in the context of the specific query, mitigating the risk of missing critical evidence.

Stage 3: Star-Based LLM Refinement (Advanced Reasoning)

Goal: To perform logical inference (aggregation, multi-hop reasoning) that semantic similarity alone cannot achieve.
Mechanism:
1. Decomposition: The expanded graph is decomposed into star graphs (a central node and its immediate neighbors) rather than feeding the entire graph to the LLM. This reduces hallucination and improves focus.
2. Column-wise Aggregation: The LLM determines if the query requires aggregation (e.g., "most recent," "highest"). If so, it reconstructs the original table from segments and identifies the correct rows.
3. Passage Verification: The LLM performs binary verification on edges to filter out hard negatives (irrelevant passages) that passed previous retrieval stages.
Benefit: Leverages LLMs for complex reasoning while maintaining efficiency by processing smaller, structured units (stars).

3. Key Contributions

Multi-Granular Retrieval Strategy: HELIOS is the first to systematically combine edge-level retrieval (for noise reduction), node-level expansion (for dynamic relationship discovery), and star-level LLM refinement (for logical inference).
Graph Refinement via LLM: It introduces a specific refinement stage where LLMs are used not just for answering, but for retrieval refinement (aggregating table rows and verifying passage relevance), effectively bridging the gap between retrieval and reasoning.
Formalization: It formalizes table-text retrieval as a bipartite graph problem, providing a unified theoretical framework to analyze and improve existing early/late fusion methods.
Handling Advanced Reasoning: The system explicitly handles tasks like column-wise aggregation and multi-hop reasoning, which are major failure points for standard semantic retrieval models.

4. Experimental Results

The authors evaluated HELIOS on the OTT-QA (Open Table-Text Question Answering) and MultimodalQA (MMQA) benchmarks.

Performance on OTT-QA:
- Recall (AR@2): HELIOS achieved 63.3%, outperforming the state-of-the-art (SOTA) late fusion model (COS) by 42.6%.
- Ranking Quality (nDCG@50): HELIOS achieved 47.0, a 39.9% improvement over COS.
- End-to-End QA: HELIOS improved Exact Match (EM) by 4.2% and F1 by 4.1% over COS on the development set.
Comparison with Other Models:
- Outperformed early fusion models (Fusion-Retriever, OTTeR, DoTTeR) and late fusion models (CORE, COS).
- Significantly outperformed graph-based multi-hop QA methods like HOLMES (88.4% higher EM), attributed to HELIOS's seed node selection and query-conditioned reasoning.
Ablation Studies:
- Removing Node Expansion (QNE) caused a ~2.1% drop in recall, proving the necessity of dynamic graph growth.
- Removing LLM Refinement (SLR) caused a significant drop in AR@2 and AR@5, highlighting the importance of logical inference for top-k precision.
- Granularity Analysis: Edge-based retrieval outperformed node-based and star-based retrieval by 6.9% and 12.4% respectively, confirming the optimal balance of information density.

5. Significance

Paradigm Shift: HELOS moves beyond the binary choice of "early vs. late" fusion, demonstrating that a hybrid, multi-stage approach with varying granularities yields superior results.
Scalability and Efficiency: Despite the complexity, HELIOS maintains a competitive execution time (approx. 5.14s), which is only slightly higher than SOTA late fusion models but delivers massive accuracy gains.
Robustness: The method shows strong generalization on the MultimodalQA dataset, indicating its applicability beyond just text-table pairs.
Future Direction: The paper lays the groundwork for extending graph-based retrieval to other modalities (e.g., images) and improving LLM reliability through self-evaluation to further reduce hallucinations.

In conclusion, HELIOS represents a significant advancement in open-domain QA by effectively integrating structured and unstructured data retrieval with advanced reasoning capabilities, setting a new state-of-the-art for complex multi-hop questions.