GEB-Bench: Abstract Structures Told in Many Voices

The Great Shape-Shifter Test

Imagine you are looking at a river delta spreading out into the ocean. Now, imagine you are looking at a jagged lightning bolt striking the sky. To a human, these look completely different: one is water and sand, the other is fire and air. But if you squint and look at the shape of the branches, you might realize they share the same secret blueprint. They both follow a pattern of splitting and spreading that mathematicians call a "fractal" or a specific type of branching structure. This ability to see the invisible skeleton underneath the messy surface is what we call abstract reasoning. It's the superpower that lets us understand that a story about a king and a story about a computer program can both be about "power," even if the words are totally different.

For a long time, scientists have wondered if artificial intelligence (AI) has this superpower. We've built models that can name a cat in a photo or write a poem about the moon, but can they actually see the deep, hidden structures that connect a river, a story, a math equation, and a computer code? This is the big question. If an AI can't do this, it's just a very fancy parrot that mimics patterns without understanding the "why" or the "how" behind them. It's like having a robot that can recite the recipe for a cake but doesn't understand what "baking" actually means.

The "Many Voices" Challenge

Enter GEB-BENCH, a new, tricky test designed to see if AI models can spot these hidden shapes. The creators named it after a famous book called Gödel, Escher, Bach, which is all about how the same weird, looping ideas show up in math, art, and music. The test is built on a simple but sneaky idea: take one single abstract shape (like a loop, a spiral, or a mirror image) and tell the same story about it in four completely different "voices."

Think of it like a game of "Telephone," but instead of whispering a message, you are translating a shape.

1. The Scene Voice: A picture of a natural thing, like a nautilus shell or a river delta, where the shape is hidden in the composition.
2. The Story Voice: A short folk tale where the shape is hidden in the way the story is told. For example, the story might be a "crab canon," where the second half of the story is the first half read backward, word for word.
3. The Math Voice: A clean, precise math theorem that describes the shape using symbols.
4. The Skeleton Voice: A bare-bones line drawing, like a wireframe, that shows the shape without any fancy details.

The catch? The test makers scrubbed away all the "fluff." They made sure the river delta and the lightning bolt didn't share any colors or textures. They made sure the stories didn't use the same words. The only thing that matters is the invisible structure. The AI has to look at a picture of a river and say, "Ah, this is the same shape as that math theorem!" or "This story is telling the same structural joke as that line drawing."

The Findings: The "Tax" of Translation

The researchers tested 37 different AI models, from the tiny, open-source ones to the massive, super-smart "frontier" models from big tech companies. They found something fascinating and a little bit disappointing: AI is good at recognizing a shape in one voice, but terrible at carrying it over to another.

Imagine you are really good at recognizing a friend when they are wearing their favorite red jacket (the "Math Voice"). But if that same friend shows up wearing a clown costume (the "Story Voice") or a camouflage suit (the "Scene Voice"), you might not recognize them at all. The AI models could often identify the shape when it was presented as a clean math equation or a simple line drawing. But the moment they had to translate that shape into a natural scene or a folk tale, their performance dropped like a stone.

The paper calls this the "mapping tax." Every model, no matter how smart, has to pay this tax. Even the most advanced models, which are usually considered the "frontier" of AI, struggle to connect the dots between a picture and a story. The study found that while these models could recognize the shape in the math voice about 86% of the time, their ability to match that same shape to a story voice dropped to around 44%. It's a huge gap.

The "Formal Geometry" Trap

Here is the really wild part: the AI didn't just guess randomly. When they got it wrong, they made specific mistakes that followed a logical pattern. The paper calls this "formal geometry."

Imagine you are trying to tell the difference between a "loop" (a circle) and a "strange loop" (a circle that twists back on itself in a confusing way). The AI models often confused these two. They didn't get confused because the pictures looked similar; they got confused because the mathematical rules defining the two shapes are neighbors. It's like if you were learning to drive and you kept confusing a "stop sign" with a "yield sign" because they are both red octagons, even though the rules for using them are different.

The study showed that the AI's errors were much more predictable based on these mathematical rules than on how the pictures actually looked to a human. If you showed the AI a picture that looked like a "strange loop," it would often call it a "loop" because, in the world of math, those two concepts are right next to each other. This suggests the AI isn't just "seeing" the image; it's trying to match it to a library of mathematical concepts, but it's getting stuck on the fine print.

The "Surface" Problem

The researchers also found that the "noise" of the real world makes things harder for AI. They tested the models with images that went from a simple line drawing (very clean) to a cluttered, realistic photograph (very messy). As the images got more realistic and "noisy," the AI's performance got worse.

It's like trying to hear a song in a quiet room versus a loud, crowded concert. The AI can hear the melody (the structure) in the quiet room, but the crowd noise (the surface details of the photo) drowns it out. The study found that having a bigger, more powerful model didn't make the AI immune to this noise; it just gave it a little more "headroom" to handle the mess. It didn't solve the problem; it just delayed the crash.

The Bottom Line

The most important takeaway from this paper is that recognizing a structure and translating it across different worlds are two different skills. The AI models we have today are getting very good at recognizing structures when they are presented clearly (like in math or simple diagrams). But they are still struggling to take that understanding and apply it to the messy, complex, and varied world of natural scenes and stories.

The paper doesn't say AI is "broken" or that it will never learn. It just says that right now, there is a specific, measurable gap. The models can see the shape in the math book, but they can't yet see the same shape in the river delta or the folk tale. And until they can bridge that gap, they are missing the true "aha!" moment of abstract reasoning—the ability to see that the river, the story, and the math are all singing the same song, just in different voices.