💬 NLP

IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answering

IterCOMP is a training-free, iterative prompt compression framework that enhances multi-hop question answering by dynamically decomposing documents and generating targeted follow-up questions to construct compact, reasoning-oriented prompts, thereby improving accuracy and efficiency while reducing token usage.

Original authors: JungMin Yun, YoungBin Kim

Published 2026-08-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: JungMin Yun, YoungBin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, multi-layered mystery. You have a giant stack of old newspapers, magazines, and handwritten notes scattered across your floor. Your goal is to find the answer to a tricky question that doesn't have a single obvious clue, but instead requires you to connect dots between three or four different articles. This is the world of "Multi-hop Question Answering," a special challenge for Artificial Intelligence (AI). In this corner of computer science, AI acts like a detective that reads through mountains of text to find answers. But here's the problem: if you give the AI the entire stack of newspapers at once, it gets overwhelmed. It's like trying to find a specific needle in a haystack while someone is shouting a thousand other random facts in your ear. The AI gets confused, slows down, and might even miss the answer because the "noise" drowns out the important clues. This is why researchers are always looking for ways to help the AI focus on just the right pieces of paper, throwing away the rest without losing the story.

Enter IterCOMP, a new method proposed by researchers Jungmin Yun and Youngbin Kim that acts like a super-smart, patient editor for these AI detectives. Instead of just blindly cutting out words to make the text shorter, IterCOMP uses a clever, step-by-step strategy to figure out exactly what the AI needs to solve the puzzle. Think of it as a game of "20 Questions" played with the AI itself. The system first looks at the clues it has and asks, "Do we have enough to solve this yet?" If the answer is "No," the AI doesn't just guess; it asks itself, "What specific piece of information is missing?" It then generates a new, targeted question to hunt down that missing clue, finds the relevant sentence, and adds it to its notes. It repeats this loop—checking, questioning, and gathering—until it has built a perfect, compact story that contains everything needed to solve the mystery.

The paper suggests that this approach is a significant improvement over existing methods. While older techniques often tried to compress text by simply removing "unimportant" words or sentences based on a single glance at the question, IterCOMP argues that this fails for complex, multi-step puzzles. The researchers found that by letting the AI reason through the gaps in its knowledge, they could create a much shorter prompt (the set of instructions and clues given to the AI) that actually performed better. In their tests on three different puzzle datasets (MusiQue, 2WikiMultiHopQA, and HotpotQA), IterCOMP didn't just save space; it made the AI smarter. For example, on the HotpotQA dataset, the method boosted the AI's accuracy score (F1) from 43.63 to 51.78, while shrinking the amount of text the AI had to read by about 81% (a compression ratio of 0.19). This means the AI could solve harder problems faster and cheaper, without needing to be retrained or taught new tricks.

The beauty of IterCOMP is that it doesn't require the AI to learn anything new. It works with the AI's existing brainpower, using a "training-free" loop to guide the search. The researchers showed that as the puzzles got harder (requiring more "hops" or steps of reasoning), the system naturally adjusted, asking more follow-up questions and gathering more evidence until it felt confident. However, they also noted that this isn't a magic wand; if the AI makes a mistake in judging what it knows, it might stop too early or gather the wrong clues. But overall, the study suggests that by turning prompt compression into an active reasoning process, we can help AI navigate the noise of the internet and find the truth, even when the answer is hidden deep inside a maze of documents.

Technical Summary: IterCOMP

Problem Statement

Multi-hop Question Answering (QA) requires reasoning across multiple evidence segments to derive a correct answer. While Retrieval-Augmented Generation (RAG) systems mitigate knowledge limitations of Large Language Models (LLMs) by incorporating external information, they face significant challenges in efficiency and effectiveness. The retrieval of lengthy documents introduces high inference latency, increased computational costs, and "lost-in-the-middle" phenomena where models struggle to attend to relevant information amidst noise.

Existing prompt compression methods attempt to address these issues by reducing input length. However, most are designed for single-turn queries and rely on surface-level relevance matching. They fail to capture the interdependent, sequential reasoning steps required in multi-hop QA, often discarding critical intermediate clues or retaining irrelevant content that distracts the model. Consequently, current approaches struggle to synthesize information from multiple sources effectively, leading to incomplete contextual understanding.

Methodology: IterCOMP

The authors propose IterCOMP, a unified, training-free prompt compression framework designed specifically for multi-hop QA. Unlike static compression methods, IterCOMP integrates multi-hop reasoning directly into the compression loop through an iterative refinement process. The framework operates without fine-tuning the underlying LLMs, making it applicable to both open-source and commercial black-box models.

The pipeline consists of four key stages:

Document Decomposition: Retrieved documents are segmented into smaller evidence units (specifically sentences in this work) to preserve local semantic context while enabling granular filtering.
Relevant Evidence Filtering: A dual-aspect relevance scoring mechanism filters candidate segments. This combines:
- Semantic Similarity: Computed via dense vector embeddings to capture contextual alignment.
- Lexical Similarity: Computed via weighted token overlap to capture precise keyword matches, which are crucial for multi-hop reasoning.
- These scores are combined into a single metric, and segments exceeding a percentile-based threshold are retained.
Answerability Judgment & Missing Information Identification: An LLM acts as a reasoning controller to evaluate the accumulated evidence set ( $E_{cand}$ $E_{c an d}$ ):
- Answerable: If the evidence is sufficient to answer the original question, the process terminates, and the current set becomes the final compressed prompt.
- Unanswerable: If evidence is insufficient, the LLM identifies the specific information gap and generates a targeted follow-up question.
Iterative Accumulation: The generated follow-up question serves as a new query to retrieve and filter additional evidence. This cycle repeats until the question is deemed answerable or a maximum iteration limit is reached, progressively constructing a compact, reasoning-oriented prompt.

Key Contributions

Iterative Reasoning Loop: The paper introduces a framework that explicitly incorporates multi-hop reasoning into the compression process, moving beyond single-query relevance matching to handle interdependent reasoning steps.
Training-Free Design: IterCOMP operates without training or fine-tuning the compression model, relying instead on the inherent reasoning capabilities of pre-trained LLMs for judgment and question generation.
Dual-Aspect Filtering: The method combines semantic and lexical signals to ensure both broad contextual relevance and precise keyword coverage, addressing the limitations of purely semantic or purely lexical compression.
Dynamic Evidence Accumulation: By using answerability judgments to trigger targeted follow-up questions, the framework dynamically accumulates only the critical evidence needed, avoiding the information loss common in static compression.

Experimental Results

The authors evaluated IterCOMP on three multi-hop QA benchmarks: MuSiQue, 2WikiMultiHopQA, and HotpotQA. The framework was compared against state-of-the-art hard prompt compression baselines (e.g., LLMLingua, LongLLMLingua, R2C) and a "Raw Documents" baseline.

Performance Gains: IterCOMP achieved the highest Exact Match (EM) and F1 scores across all datasets. For instance, on MuSiQue, it improved the F1 score from 19.92 (Raw Documents) to 27.36, significantly outperforming the second-best baseline (R2C) by 4.92 points.
Efficiency: The framework achieved substantial token reduction (e.g., a 7× reduction on MuSiQue) while maintaining or improving accuracy compared to uncompressed baselines.
Robustness: While performance naturally declines as reasoning complexity increases (e.g., F1 scores on MuSiQue drop from 30.19 at 2-hop to 20.72 at 4-hop), IterCOMP demonstrates robustness by dynamically adjusting its behavior. Specifically, it increases the average number of iterations (from 1.98 to 3.08) and expands the retained evidence base (from 228 to 403 tokens) to better support longer reasoning chains, thereby mitigating the severity of performance degradation compared to static methods.
Cost and Speed: In experiments with API-based commercial LLMs (GPT-3.5, GPT-4o, Gemini, Claude), the framework reduced operational costs by 75.5%–79.4% and accelerated inference by 1.13× to 2.01×.
Ablation Studies: Removing the iterative refinement component caused the largest performance drop, confirming the necessity of evidence accumulation. Similarly, the combination of semantic and lexical scoring proved superior to using either signal alone.

Significance and Claims

The paper claims that IterCOMP offers a robust and scalable solution for complex multi-document LLM applications by addressing both performance degradation in long contexts and the economic burden of large-scale inference.

Key claims regarding its significance include:

Bridging the Gap: IterCOMP effectively bridges the performance gap between raw document inputs and ideal "Oracle" settings (where only gold documents are provided), closing over 50% of the gap on HotpotQA.
General Applicability: Its model-agnostic and training-free nature allows for immediate integration with commercial black-box LLMs where fine-tuning is infeasible.
Reasoning-Aware Compression: The work demonstrates that effective compression for multi-hop QA requires more than just reducing length; it necessitates a reasoning-aware process that iteratively identifies and fills information gaps.

The authors acknowledge limitations, noting that the framework's effectiveness depends on the underlying LLM's reasoning capabilities (risking error propagation) and that the iterative design incurs additional overhead compared to single-pass methods. They suggest future work could focus on adaptive hyperparameter tuning and more sophisticated scoring schemes.