Verification Without Sufficiency: Per-Chunk Filtering Fails on Multi-Hop RAG, and Decomposition Repairs It

The Detective and the Missing Clue

Imagine you are a detective trying to solve a mystery, but instead of a notebook, you have a super-smart robot assistant. This robot is great at writing stories and answering questions, but it sometimes makes things up. To stop it from lying, you give it a stack of newspaper clippings (retrieved text) and ask it to find the specific facts hidden inside them before it writes its answer. This setup is called Retrieval-Augmented Generation, or RAG. The idea is simple: the robot reads the clues, checks if they make sense, and then tells you the truth.

But what happens when the mystery is a "multi-hop" puzzle? In a normal question, the answer is right there in one paragraph. In a multi-hop puzzle, the answer is hidden in a chain of clues. You need to read one paragraph to find a name, then use that name to find a second paragraph that holds the real answer. The robot's job is to verify: "Does this paragraph help solve the puzzle?" The big question researchers are asking is: Can we just check each paragraph one by one to see if it's useful, or does the puzzle break if we look at the pieces in isolation?

---

The Trap of Looking at One Piece at a Time

In this paper, a researcher named Randhir Kumar investigates a common strategy used to fix these robot assistants. The strategy is called "per-chunk filtering." Imagine you have a pile of 10 newspaper clippings. The standard advice is to look at each one individually, give it a score, and throw away the ones that don't seem to contain the answer. It sounds logical, like a bouncer checking IDs at a club door. If the ID doesn't match the guest list, you don't let them in.

The paper shows that for multi-hop puzzles, this "bouncer" strategy is actually a disaster. It's not just that the bouncer is bad at his job; it's that the job description is impossible.

Here is the twist: In a multi-hop puzzle, the paragraph that actually holds the final answer is usually the one the question doesn't mention. For example, if the question is "Who was the wife of the actor who played the villain in Movie X?", the question names the movie and the actor. The paragraph about the movie is easy to find. But the paragraph about the actor's wife? The question never mentions her name. If you ask the robot, "Does this paragraph about the wife help answer the question?", the robot looks at the question, sees no mention of the wife, and says, "Nope, this is irrelevant." It throws the most important clue in the trash.

The researchers tested this on three different puzzle datasets (HotpotQA, 2WikiMultihopQA, and MuSiQue) and found that this "one-by-one" checking method failed miserably. When they tried to score paragraphs individually, the system couldn't tell the difference between a helpful clue and a fake one. The success rate (measured as AUC) hovered around 0.54 to 0.64, which is barely better than flipping a coin. In fact, on the hardest puzzles, the system was so confused that it often kept the wrong paragraphs and threw away the right ones.

Why the "Bouncer" Fails

The paper rules out several excuses for why this might be happening. It's not because the robot is too dumb (they tested bigger and smaller robots, and the problem got worse with smarter ones). It's not because the paragraphs were too short or too long. It's not because the "bouncer" was too strict or too lenient with its rules.

The real culprit is sufficiency. The "bouncer" assumes that a single paragraph should be enough to prove the answer. But in a multi-hop puzzle, no single paragraph is enough. You need the combination of the first clue and the second clue to make sense.

To prove this, the researchers did a clever experiment. They took the two correct paragraphs and glued them together into one long piece of text. When they asked the robot to check this combined text, the success rate jumped from a coin-flip 0.66 up to a solid 0.88. This showed that the information was there; the robot just couldn't see it when the clues were separated.

They also found that the problem gets worse the more steps the puzzle has. If a puzzle needs two steps to solve, the robot is okay-ish. If it needs four steps, the robot is completely lost. It's like trying to find a needle in a haystack by looking at one blade of grass at a time; you'll never find the needle until you look at the whole pile.

The Fix: Breaking the Problem Down

So, if checking one paragraph at a time fails, what works? The paper suggests a repair that changes how we ask the question.

Instead of asking the robot, "Does this paragraph answer the original question?", the researchers suggest asking, "Does this paragraph answer the next step of the puzzle?"

Imagine the puzzle is a treasure hunt.

• The Old Way: You show the robot a map of the island and ask, "Does this map show the treasure?" The robot says, "No, the treasure isn't on this map," and throws the map away. But the map actually shows the location of the key needed to open the treasure chest.
• The New Way: You first figure out the first step: "Where is the key?" You find the map with the key. Then, you ask the robot, "Does this next map show the treasure given that we have the key?" Suddenly, the robot understands. It sees the connection.

The researchers tested this by using a "decomposer"—a tool that breaks the big question into smaller sub-questions. When they used these smaller questions to check the paragraphs, the success rate skyrocketed. On the hardest puzzles, the score jumped from 0.546 (random guessing) to 0.840. This is a massive improvement, proving that if you give the robot the right context for the specific step it's looking at, it can find the answer.

The Cost of Getting It Wrong

The paper also looked at what happens when you use these methods to actually generate answers. They found that using the "one-by-one" bouncer method was the worst possible choice. It was so bad that it actually made the robot's answers worse than if you had just let the robot read everything without filtering at all.

In fact, the smarter the robot was, the more it suffered from this bad filtering. A slightly smarter robot lost 4.6 points in accuracy, but a very smart robot lost 19.4 points. It's like giving a brilliant chef a recipe where you've thrown away the main ingredient because it didn't match the title of the dish. The chef is so good at cooking that they can tell you exactly what's missing, but they can't make the dish without it.

The Takeaway

The main lesson here is that you can't judge a multi-step puzzle by looking at the steps in isolation. The "bouncer" approach of checking each paragraph against the original question fails because the answer is hidden in the connection between paragraphs, not in the paragraphs themselves.

The paper doesn't claim to have solved the whole problem perfectly. Even with the new "decomposed" method, there is still room for improvement, and the researchers admit their tool isn't perfect yet. But they have proven that the old way of filtering is broken and that the path forward lies in breaking the big question into smaller, manageable pieces before checking the clues. It's a reminder that sometimes, to find the answer, you have to stop looking at the whole picture and start looking at the next step.