Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles

Imagine you are trying to teach a super-smart robot how to understand the world by showing it pictures. To do this, the robot doesn't just "see" the image like a human; it breaks the picture down into thousands of tiny digital puzzle pieces called "tokens." Think of these tokens like individual notes in a massive, chaotic orchestra. The robot has to listen to every single note to understand the song. But here's the problem: for high-resolution images, that orchestra is so huge it slows the robot down, making it sluggish and hungry for computer memory.

Scientists have been trying to fix this by teaching the robot to ignore the "boring" notes—those that seem to repeat themselves or don't carry much meaning. This is called "token pruning." It's like a conductor telling the orchestra to stop playing the notes that sound the same, hoping the music still sounds perfect but plays much faster. Recently, a new tool called "EmbedLens" helped researchers realize that these tokens aren't all the same. Some are "alive" (carrying the real story of the picture), some are "sinks" (repeating patterns that just sit there), and some are "dead" (completely redundant noise). The big question was: If we know which tokens are "dead," can we just throw them all away to speed things up?

This paper, titled "Not All Redundant Tokens Are Alike," dives into that very question. The researchers took a close look at how different pruning methods decide which tokens to cut. They discovered something surprising: the methods that try to cut the "dead" tokens don't always work better than the ones that don't. In fact, their experiments suggest that even the "dead" tokens might be doing something important behind the scenes, like holding the orchestra together. By protecting these "dead" tokens from being deleted, the robot sometimes actually got better at answering questions, not worse. It turns out that just because a token looks useless on its own doesn't mean it's safe to throw away; sometimes, the whole group needs to stay together for the music to sound right.

The Story of the Digital Orchestra

To understand why this matters, let's look at how these Vision-Language Models (VLMs) work. Imagine you have a robot that can read and see. When you show it a photo of a cat, the robot's "eye" (a vision encoder) breaks that photo into a long list of tiny data chunks. These chunks are then handed over to the robot's "brain" (a large language model) to figure out what the cat is doing.

The problem is that a single photo can generate hundreds of these chunks. If you have a video or a very detailed image, the list gets so long that the robot gets overwhelmed. It takes too long to process the first part of the image (called "prefill latency"), uses up too much memory, and costs a lot of energy. To fix this, researchers developed "token pruning." This is like a smart editor who looks at the list of chunks and says, "Okay, we don't need these 500 copies of the same blue sky; let's just keep 50."

For a while, everyone assumed the rule was simple: "Keep the interesting stuff, throw away the boring stuff." Recently, a tool called EmbedLens changed the game. It looked inside the robot's brain and found that the tokens fall into three distinct groups:

1. Alive Tokens: These are the stars. They carry the specific details of the image, like "fluffy fur" or "green grass."
2. Sink Tokens: These are like the background hum. They are repetitive and don't really talk about the specific image, but they are stable and always there.
3. Dead Tokens: These are the ultimate noise. They seem to have no connection to the image or the words at all.

The logical guess was: "If we have Dead Tokens, let's just delete them! They are useless!"

The Great Pruning Experiment

The authors of this paper decided to test that guess. They took three popular pruning methods (FastV, DART, and DivPrune) and asked: "When you cut down the number of tokens, which ones are you actually cutting?"

They ran these methods on ten different test sets, ranging from simple questions like "What color is the car?" to complex science questions. They tracked exactly how many "Alive," "Sink," and "Dead" tokens were removed at different levels of cutting (from 12.5% to 87.5% of the tokens removed).

What they found was a bit messy.
The different pruning methods didn't agree on what was "useless."

• DivPrune was very aggressive about cutting "Dead" tokens. It seemed to follow the rule: "Delete the noise."
• FastV and DART, however, were weirdly protective of the "Sink" tokens. They kept the repetitive background hum even though it seemed redundant.

Here is the twist: Just because a method cut more "Dead" tokens didn't mean it performed better. In fact, DivPrune (which cut the most dead tokens) was sometimes the best, but sometimes FastV (which kept the dead tokens) was better. There was no perfect match between "cutting the dead stuff" and "getting a good score."

The "Protect the Dead" Surprise

To figure out why this was happening, the researchers did a clever experiment. Instead of just letting the pruning methods decide, they forced the robot to keep specific types of tokens. They said, "Okay, FastV, you can cut whatever you want, but you must keep all the Dead Tokens."

They expected the performance to drop. After all, if Dead Tokens are useless, keeping them should just waste space and confuse the robot.

The result? It didn't drop. In many cases, it went up.

• When they protected the Alive tokens, performance improved (which makes sense).
• But when they protected the Dead tokens, the robot's performance actually improved by about 0.93% on average for the DivPrune method.

This was a shock. It suggested that even though a single "Dead" token might look like garbage, deleting all of them might break something else.

The Hidden Orchestra Conductor

Why would keeping "useless" tokens help? The authors looked at how the tokens talked to each other using something called attention. Imagine the tokens are musicians passing notes to each other.

They found that while a single "Dead" token doesn't get much attention (it's not a soloist), there are so many of them that, as a group, they take up a huge chunk of the "attention budget." They are like a massive choir of backup singers. If you cut them all out, the remaining musicians (the Alive tokens) lose their context. The "Dead" tokens might not be singing the melody, but they are holding the rhythm and the structure of the song.

When the researchers protected the Dead tokens, the attention patterns among the remaining tokens stayed more balanced. The "Alive" tokens could still interact properly because the "Dead" tokens were still there to support the structure.

The Takeaway

The paper suggests that we can't just look at a token and say, "You are dead, go away." It's not that simple. The "Dead" tokens might be weak on their own, but they are strong as a group.

The authors conclude that effective pruning shouldn't just be about finding the most important tokens. It needs to be about composition. We need to make sure we don't accidentally strip away a whole category of tokens that, while individually boring, are essential for the group to function. It's a reminder that in a complex system like an AI, even the "redundant" parts might be doing a job we don't fully understand yet.

So, the next time you think about cleaning up a messy room, remember: sometimes the stuff you think is just clutter is actually holding the shelf up.