Evaluation Resolution Confounds Learning-Rule Comparisons in Model-Brain RSA of Early Visual Cortex

Imagine you are trying to teach a robot to see the world the way a human does. Scientists in a field called NeuroAI are obsessed with this question: "What is the best set of rules to teach a computer how to recognize a cat or a car?" They have a few different rulebooks to choose from. One is the standard, super-efficient method used in almost all modern AI (called backpropagation). The others are "biologically plausible" rules, which try to mimic how real brains actually learn, like how neurons fire in response to timing or how the brain predicts what it will see next.

To see which rulebook works best, researchers use a clever trick called Representational Similarity Analysis (RSA). Think of it like a game of "Guess the Connection." They take the robot's internal "thoughts" (its digital representations) about an image and compare them to the actual electrical activity in a human's brain when looking at that same image. If the robot's thoughts match the brain's activity closely, we say the robot has a "brain-like" understanding. The big hope is that by finding the rule that makes the robot most like the brain, we might finally crack the code of human vision. But there's a catch: the real brain is incredibly complex, and the robots we can currently train with these fancy biological rules are still quite small and simple.

This paper pulls back the curtain on a hidden trap in these comparisons. The researcher discovered that the answer to "which rule is best?" changes completely depending on one thing: the size of the picture you show the robot. It turns out that when scientists compare these small, biologically-inspired robots to the human brain, they often use high-resolution, detailed photos (like 224 pixels wide) to test them, even though the robots were only trained on tiny, blurry 32-pixel images. The study finds that this mismatch creates a magical illusion. When tested on big, detailed pictures, the untrained, random robot (which hasn't learned anything yet) suddenly looks just as good as, or even better than, the smart, trained robot at understanding the early parts of the visual system. The author shows that this isn't because the random robot is secretly brilliant; it's because the way we are measuring it is biased by the size of the image.

The Resolution Trap

The core discovery here is that the "winner" of the brain-matching contest depends entirely on the resolution of the test images. Imagine you are judging a cooking contest. You have a chef who practiced making a dish on a tiny, 3-inch pan, and you have a random person who just threw ingredients into a bowl without cooking. If you ask them to serve the dish on a massive, 2-foot platter, the random person might accidentally look better because the huge plate hides the fact that the food is raw. But if you serve it on the same tiny 3-inch pan they practiced on, the trained chef shines, and the random person looks terrible.

In this study, the "tiny pan" is the 32-pixel resolution the robots were trained on, and the "massive platter" is the 224-pixel resolution often used to test them against brain data. When the researcher tested the robots on the same 32-pixel size they learned on, the trained robots (using backpropagation) were slightly better than the untrained ones, just as you'd expect. But when they switched to the larger 224-pixel images, the results flipped. The untrained, random network suddenly looked much more like the human brain's early visual cortex (V1) than the trained one did. The gap between them grew steadily as the image got bigger, going from a tiny difference of -0.001 at 32 pixels to a large difference of +0.044 at 224 pixels.

Ruling Out the Suspects

The author didn't just stop at finding the problem; they played detective to figure out why this was happening. They had four main suspects, and they cleared all of them one by one.

First, they suspected it was just a "mismatch" issue: maybe the trained robots just perform worse when shown images they weren't trained on. To test this, they tried robots that were actually trained on the big 224-pixel images (like a ResNet-50 and a Swin-Tiny transformer). If the mismatch theory were true, these big-trained robots should have done best at 224 pixels. But they didn't. They also performed best at low resolutions and got worse as the images got bigger. So, it wasn't just about being trained on the wrong size.

Second, they looked at the "untrained" robot's settings. Untrained robots often have their "normalization" settings (which help keep data balanced) set to default values, while trained robots have learned settings. The author thought maybe this default setting was the secret sauce. They manually adjusted the untrained robot's settings to match the training data perfectly. Even with these perfect settings, the untrained robot still beat the trained one at high resolutions. So, it wasn't a settings glitch.

Third, they checked if the robots were just mimicking simple visual patterns, like the way light and dark patches (Gabor filters) or raw pixels look. They found that the robots that looked most like these simple patterns didn't necessarily look most like the brain. In fact, one of the trained robots looked very "Gabor-like" but still didn't match the brain well. So, simple patterns weren't the whole story.

Fourth, they considered that as the image gets bigger, the robot is averaging over more and more pixels, which might make its final answer just a simple measure of the image's overall brightness. They found that a single number representing the average brightness of an image actually matched the brain data almost as well as the untrained robot did. This was a huge clue: the untrained robot was doing a lot of its "matching" just by guessing the brightness. However, even after accounting for brightness, the resolution effect didn't disappear completely.

The Real Culprit: Image Detail

To find the true cause, the author ran a final, clever experiment. They took the big images, shrunk them down to the tiny 32-pixel size (capping the detail), and then blew them back up to the big size. This meant the robot was still looking at a big image and averaging over many pixels, but the actual information in the image was stuck at the low resolution.

The result was dramatic. When the detail was capped at the training resolution, the trained robots stopped getting worse as the image got bigger. The huge gap between the trained and untrained robots almost vanished. This proved that the effect wasn't caused by the robot looking at more pixels (the pooling), but by the robot seeing more detail that it hadn't learned to handle. The extra detail in the high-resolution images was hurting the trained robots and helping the random ones, but the author admits they don't fully understand why the random filters benefit from this extra detail while the trained ones suffer.

What Actually Matters

The paper ends with a reassuring twist. While the early part of the visual system (V1) was getting confused by the image size, the higher-level parts of the brain (called LOC) told a different story. At these higher levels, the trained robots consistently beat the untrained ones, no matter the image size. This means learning does change the brain-like nature of the network, but the "noise" of the image resolution was drowning out the signal in the early stages.

The main takeaway is a warning for scientists: if you want to know which learning rule is best, you have to be very careful about the size of the pictures you use to test them. The "winner" isn't a fixed property of the rule; it's a result of how you choose to measure it. Until scientists agree on a standard way to match the image size to the brain area they are studying, comparisons at the early visual cortex might be misleading. The paper suggests that we should test at the resolution the model was trained on, and report that resolution clearly, so we aren't fooled by the resolution trap.