Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression

Imagine you are trying to solve a giant jigsaw puzzle, but instead of a picture on the box, you have a massive, thousand-page encyclopedia. You need to find the answer to a tricky question hidden somewhere inside those pages. The problem is, your brain (or in this case, a super-smart computer program called a Large Language Model) gets overwhelmed if it tries to read every single word. It's like trying to drink from a firehose; you might drown in the information before you find the one drop you need. To fix this, scientists use "prompt compression." Think of this as a super-fast editor that scans the encyclopedia and deletes the boring parts, keeping only the most exciting sentences so the computer can read the story quickly and cheaply. The goal is to keep the story short but still make sure the computer can solve the puzzle.

But here is the catch: sometimes, this editor gets a little too enthusiastic. It might keep the sentence that says "The answer is McDonald County" but delete the sentence right before it that says "Tim DuBois was born in Southwest City." Without that missing link, the computer sees the answer but has no idea why it's the answer. It's like finding a treasure map that says "X marks the spot" but having the page with the landmark description torn out. The computer is left staring at a dangling clue, confused and unable to connect the dots. This paper investigates exactly how often this "referential dangling" happens, proves that it's a major flaw in how these editors currently work, and shows that we can fix the broken chains of logic to help the computer solve the puzzle again.

The Great "Dangling" Disaster

The researchers discovered that the current way these computers "compress" text is like a game of musical chairs where the rules are broken. The standard method scores every sentence or chunk of text individually, asking, "Is this sentence important?" If the score is high, it stays; if it's low, it gets tossed. The problem is that these editors don't look at how sentences depend on each other. They treat every sentence like an island.

The authors call this failure mode "referential dangling." Imagine you are telling a story: "I went to the store. The store was closed." If you delete the first sentence, the second one makes no sense. The word "The store" is now dangling in the air with nothing to hold it up. In the world of AI, this happens when the computer keeps the answer (e.g., "McDonald County") but deletes the bridge that explains the connection (e.g., "Tim DuBois was born in Southwest City, which is in McDonald County"). The answer is still there, visible and intact, but the logic chain is snapped. The AI sees the answer but can't figure out how to get there, leading to confusion or wrong answers.

How Bad Is It? (The Numbers Don't Lie)

The team tested this on a popular compression tool called Beaver and found that the problem is everywhere. At a compression ratio of 0.30 (meaning they kept only 30% of the original text), they found that 34% to 54% of the tricky, multi-step questions ended up with these broken chains. That's more than half the time!

They didn't just blame Beaver. They tested six different compression tools using various methods (some looking at grammar, some at word importance, some at how surprising a word is). The results were shocking: every single one of them suffered from this dangling problem. On a shared set of difficult questions, the failure rates ranged from 32% to nearly 60%. Even worse, when they looked at long documents (like legal or academic papers), every single document they tested contained at least one dangling reference. It seems that no matter how smart the compression tool is, if it just picks the "best" sentences one by one, it will inevitably break the story.

The "Oh No, We Fixed It" Experiment

The big question was: Is this just a quirk of the current tools, or is the whole idea of picking sentences one-by-one doomed? To find out, the researchers played a game of "what if." They took the broken, compressed stories and manually put the missing "bridge" sentences back in. But to keep the story short (staying within the same 30% budget), they had to cut out some other, less important sentences to make room.

The result was a massive success. When they fixed the broken chains by reinserting the missing logic, the computer's accuracy jumped by 29 to 34 points on difficult questions. This wasn't a tiny improvement; it was a huge leap. It proved that the problem wasn't that the computer was too dumb to understand the text, but that the text it was given was logically incomplete. The "dangling" was the real culprit, not the AI's intelligence.

Even more interesting, they tested if stronger, smarter AI models could just "figure it out" on their own. They tried using a super-powerful model called GPT-5.5, thinking maybe it was smart enough to guess the missing links. But no luck. Even the smartest model was 8.8 points less accurate when the logic chain was broken compared to when it was whole. This suggests that no matter how smart the AI gets, it still needs the full story to work properly.

A Smart "Auto-Rescue" System

Finally, the team asked: Can we fix this automatically without a human needing to check every sentence? They built a tiny, fast "rescue robot" (a small classifier) that looks at the sentences the editor threw away. Its job is to ask, "Hey, does this deleted sentence explain the one we kept?" If the answer is yes, the robot puts it back in.

They tested this on the HotpotQA dataset. By using this automatic rescue system, they improved the accuracy by 4.7 points. The best part? They only had to increase the text size by a tiny bit, from a compression ratio of 0.30 to 0.31. It was a small price to pay to save the logic chain.

The Takeaway

This paper doesn't just point out a bug; it reveals a fundamental flaw in how we currently try to shrink text for AI. The lesson is clear: Relevance isn't enough. Just because a sentence is important doesn't mean it's useful if you delete the sentence that gives it meaning. To make AI truly efficient, we need compressors that don't just pick the "best" words, but that also make sure the story stays connected. If we want our AI to solve puzzles, we have to make sure we don't throw away the pieces of the puzzle that hold the picture together.