Reassessing Number-Detector Units in Convolutional Neural Networks

By applying pruning to CORnet to overcome the limitations of classical Representational Similarity Analysis, this study demonstrates that specialized number-detector units are not critical for population-level representations of numerosity, thereby challenging the hypothesis that such units are necessary for numerosity discrimination in convolutional neural networks.

The Gist

The Big Question: Do AI "Count" Like We Do?

Imagine you are looking at a bowl of fruit. You don't need to count each apple one by one to know there are "about five" of them. Your brain just sees the number. Scientists call this numerosity (or "number sense").

For a long time, researchers have been trying to build Artificial Intelligence (AI) that thinks like a human brain. They use models called Convolutional Neural Networks (CNNs). These are like digital brains made of layers of tiny processing units (neurons).

The Big Debate:
Some scientists thought these digital brains must have special "number detectors"—specific little neurons dedicated solely to counting, just like some animals have in their actual brains. They believed that if you found these special neurons in the AI, the AI was successfully learning to count.

The New Study:
This paper asks: Are these "number detector" neurons actually the heroes of the story, or are they just side characters?

The Experiment: The "Choir" vs. The "Soloists"

To find out, the researchers used a special AI model called CORnet, which is designed to look and act very much like a primate's visual system. They trained these AIs in three different ways:

1. The Blank Slate: An AI with random settings (never learned anything).
2. The Object Learner: An AI trained to recognize cats, dogs, and cars (like a standard photo app).
3. The Counter: An AI specifically trained to tell the difference between groups of dots (e.g., "Is this 6 dots or 10 dots?").

They then looked at the AI's "brain" to see how it reacted to different numbers of dots.

The Old Way of Checking (The "Equal Vote" Problem)

Previously, scientists used a method called Representational Similarity Analysis (RSA). Imagine you are trying to guess a song by listening to a choir.

• The Old Method: You assume every single singer in the choir contributes exactly the same amount to the final sound. If the choir sounds like the song, you say, "Great job, everyone!"
• The Problem: In a real choir, some singers might be singing the melody (super important), while others are humming background noise (not important). If you treat everyone equally, you might miss the fact that the melody singers are doing all the work, or you might think the background noise is crucial when it's not.

The New Way of Checking (The "Pruning" Method)

The researchers used a technique called Pruning.

• The Analogy: Imagine you have a giant choir of 10,000 singers. You want to know which singers are actually needed to recreate a specific song.
• The Process: You start by silencing singers one by one.
  • If silencing a singer makes the song sound terrible, that singer is essential.
  • If silencing a singer makes no difference (or even makes it sound better by removing noise), that singer is redundant.
• The Result: You end up with a tiny, super-efficient group of "retained singers" who are the only ones actually needed to match the human experience of seeing numbers.

The Surprising Findings

When the researchers applied this "Pruning" method, they found something shocking:

1. The "Number Detectors" were mostly useless.
   The specific neurons that looked like they were "counting" (the number-detector units) were often the ones that got cut out during pruning. They were like the background singers who were humming off-key. Even though they existed, the AI didn't need them to understand the concept of "how many."

2. The "Whole Crowd" matters more than the "Specialists."
   The AI understood numbers best when it used a broad mix of many different neurons working together, rather than relying on a few "specialist" neurons. It's like realizing that a song is made by the entire choir's harmony, not just one soloist.

3. Even untrained AIs had "detectors."
   Interestingly, even the AI that had never been taught anything (the "Blank Slate") had these "number detector" neurons. This suggests they might just be a natural byproduct of how the AI is built, not a sign that the AI has truly learned to count.

The Takeaway

The "Specialist" Myth:
We used to think that to understand numbers, an AI (or a brain) needed a specific "number neuron" to do the heavy lifting. This paper suggests that's not true.

The "Teamwork" Reality:
Understanding numbers is a team sport. It emerges from the collective activity of thousands of different neurons working together. The "specialist" neurons that look like they are counting are often just noise or side effects.

Why this matters:
If we want to build AI that truly understands the world like humans do, we shouldn't just look for "specialist" neurons. We need to look at how the whole network works together. It's not about finding the one genius in the room; it's about understanding how the whole team solves the problem.

Technical Summary

1. Problem Statement

Convolutional Neural Networks (CNNs) are widely used to model biological vision and predict neural activity. A specific area of debate concerns numerosity (the ability to perceive the number of items without counting). Previous studies suggested that CNNs trained on object recognition develop specialized "number-detector units" (analogous to "number neurons" found in primate brains) that are critical for representing numerosity.

However, recent findings indicate that standard CNNs fail to fully explain the variance in human parietal cortex activity regarding numerosity. A critical limitation of the Representational Similarity Analysis (RSA) framework used in these studies is the "equal weights" assumption: it treats all neural units as equally important when comparing model representations to behavioral data. This assumption may obscure the true contribution of specific units, potentially overemphasizing irrelevant features or underestimating the collective dynamics of the population. The authors aim to determine if the identified "number-detector units" are actually critical for population-level numerosity representation or if they are artifacts of the analysis method.

2. Methodology

Models and Architecture

The study utilized CORnet-Z and CORnet-S, biologically inspired CNN architectures that mimic the primate visual hierarchy (V1, V2, V4, and IT layers).

• CORnet-Z: A feedforward network (lightweight alternative to AlexNet).
• CORnet-S: Includes recurrent connectivity and is optimized for "Brain-Score."

Training Conditions

Three versions of each model were tested to isolate the effects of training:

1. Untrained: Randomly initialized weights (to test architectural bias).
2. ImageNet-trained: Trained on 1.2 million object images (standard object recognition).
3. DeWind-trained: Specifically trained to discriminate 10 numerosity values (6, 7, 9, 10, 12, 14, 17, 20, 24, 29) using stimuli generated via the DeWind method to decorrelate low-level visual features (dot size, density) from numerosity.

Stimuli and Data Collection

• Testing: 32 conditions varying in numerosity (4 levels), average item area, and total field area.
• Activation Extraction: Activations were extracted from V1, V2, V4, and IT layers. 100 instances per condition were averaged to create robust activity vectors.

Analytical Techniques

1. Identification of Number-Detector Units (Baseline):
   • Used Three-way ANOVA (factors: numerosity, total field area, average item area).
   • A unit was classified as a "number-detector" if it showed a significant response to numerosity ($p < 0.01$) but no significant response to low-level features or their interactions.
2. Pruning (Feature Selection):
   • Adapted from Tarigopula et al. (2023) to address the "equal weights" limitation of classical RSA.
   • Process:
     1. Assess unit importance by removing one unit at a time and computing the drop in RSA score against a Number RDM (Representational Dissimilarity Matrix based on human behavioral data).
     2. Rank units by importance.
     3. Sequentially add units back to an empty set until the RSA score is maximized.
   • The resulting subset is termed "Retained Units."
3. Comparison:
   • The study compared the RSA fit (correlation with the Number RDM) of three sets: (1) The Full Set of units, (2) The Retained Units (via pruning), and (3) The Number-Detector Units (via ANOVA).

3. Key Contributions

• Methodological Innovation: Applied a pruning-based feature selection approach to CNN analysis, moving beyond the classical RSA assumption that all units contribute equally. This allows for the identification of the specific subset of units most relevant to behavioral similarity.
• Re-evaluation of "Number-Neurons": Directly challenges the prevailing hypothesis that sparse, specialized number-detector units are the primary drivers of numerosity representation in CNNs.
• Population-Level Focus: Shifts the focus from isolated unit selectivity to population-level dynamics, demonstrating that numerosity is likely an emergent property of the collective network rather than a function of specific "tuned" neurons.

4. Results

Distinct Unit Sets

• The set of units identified as "number-detectors" via ANOVA and the set of "retained units" identified via pruning were largely disjoint.
• Overlap: In most cases, the overlap between the two sets was less than 5%.
• Quantity: The number of retained units after pruning was significantly higher (often 20–50% of the layer) compared to the tiny fraction of ANOVA-identified number-detectors (often <0.5%).

Performance in Modeling Behavior

• Pruned Models Outperform: The Retained Units (selected via pruning) consistently provided a better fit to the human behavioral Number RDM than the Full Set of units. This confirms that classical RSA is suboptimal due to noise from irrelevant units.
• Number-Detectors Underperform: In 8 out of 22 cases, the ANOVA-identified number-detector units performed worse than the full set of units. In most other cases, they performed worse than the pruned set.
• Conclusion: The specific units identified as "number-detectors" are not critical for capturing the population-level representation of numerosity that aligns with human behavior.

Training Effects

• Training specifically on numerosity (DeWind) reorganized the network but did not make the ANOVA-identified units more critical for the overall population representation compared to the pruned set.
• Even in untrained networks, both types of units (detectors and retained) existed, suggesting numerosity representation is partially architectural but relies on distributed populations rather than sparse detectors.

5. Significance and Implications

• Challenge to "Number-Neuron" Hypothesis: The study suggests that the emergence of "number-detector units" in CNNs may be a byproduct of the analysis method (ANOVA on isolated units) rather than the fundamental mechanism of numerosity processing.
• Distributed Representation: Numerosity is likely represented by the collective dynamics of a large, distributed population of neurons, where many units contribute weakly but synergistically, rather than relying on a few highly specialized "expert" units.
• Improved Model-Brain Alignment: By using pruning to filter out noise and irrelevant features, researchers can achieve a more accurate alignment between CNN representations and human behavioral/neural data. This suggests that future models of cognitive functions should prioritize population-level feature selection over the search for isolated, specialized units.
• Methodological Shift: The paper advocates for moving away from "equal weight" RSA assumptions in cognitive neuroscience, promoting feature selection techniques that better reflect the biological reality of neural coding.