Feedforward computational models of vision do not explain expert neural processing of visual Braille in the human visual system

This study demonstrates that feedforward computational models fail to replicate the expert human neural processing of visual Braille, suggesting that reading relies on complex interactive mechanisms between visual and linguistic systems rather than simple feedforward visual processing alone.

The Gist

The Big Idea: Why Robots Struggle to Read "Dot" Words

Imagine you are teaching a robot how to read. You give it a picture of the word "CAT" written in standard letters, and it learns quickly. But then, you show it the word "CAT" written in Braille (a system of raised dots used by blind people). You might expect the robot to learn this just as fast, because it's still the same word, just written differently.

However, this paper shows that standard computer vision models (robots) fail miserably at this. They get stuck on the "dots" and can't figure out that it's a word. In contrast, human brains are amazing at this. Even if a person has never seen Braille before, once they learn it, their brain treats it exactly like regular text.

The researchers wanted to know: Why can humans read Braille so easily, but our best AI models can't?

---

The Experiment: The "Line" vs. The "Dot"

To solve this mystery, the researchers set up two experiments using two different types of AI "brains" (neural networks): AlexNet and CORnet Z. Think of these as digital brains that have been trained to recognize objects like cats, dogs, and cars, but they have never been taught to read.

Experiment 1: The "Illiterate" Robot

First, they showed the AI single letters in three different styles:

1. Latin: Standard letters (like "A", "B", "C").
2. Braille: The standard dot system.
3. Line Braille: A fake script where the Braille dots were connected by lines (making them look like squiggly lines instead of dots).

The Result:
The AI immediately liked the Latin and Line Braille. It saw them as "cousins" because they both use lines and corners.
However, it treated Braille (the dots) as a total alien. To the AI, a dot looks nothing like a letter. It's like trying to teach a fish to fly; the AI's "vision" is built for lines, not dots.

Experiment 2: The "Literacy" Training

Next, they tried to teach the AI to read words.

• They taught it to read Dutch words in Latin first (like a child learning to read).
• Then, they tried to teach it to read the same words in Braille or Line Braille.

The Result:

• Line Braille: The AI learned this quickly. It was easy because it just had to recognize lines again.
• Real Braille: The AI struggled immensely. Even after lots of training, it was much slower and less accurate than with the line-based scripts.

The Human Comparison:
When real humans learn Braille, they do show a tiny initial struggle, but they catch up very fast (within a few days). The AI, however, never caught up. The gap between the AI and the human was huge.

---

The "Aha!" Moment: What's Missing?

The researchers dug deeper to see how the AI was thinking about these words. They looked at whether the AI could tell the difference between:

• Real Words (e.g., "Dog")
• Fake Words (e.g., "Dag" - looks like a word but isn't)
• Nonsense Strings (e.g., "Xqz")

In Humans:
When a human expert reads Braille, their brain organizes these words perfectly. It groups "Real Words" together and separates them from "Nonsense," regardless of whether the word is in Latin or Braille. The brain understands the meaning and the sound of the word, not just the shape.

In the AI:
The AI failed to do this.

• It didn't group words by their meaning or sound.
• It only grouped them by how they looked.
• Even after being "trained" to read Braille, the AI's internal map of Braille words looked nothing like a human's map. It was still just looking at dots, not reading words.

The Conclusion: The "Language" Gap

The paper concludes that vision alone is not enough to explain how humans read.

• The AI Model: Is like a very smart camera. It sees lines and dots. If the dots don't look like lines, it gets confused. It is a "bottom-up" processor (it only sees what is in front of it).
• The Human Brain: Is like a super-connected library. When we see Braille, our eyes send a signal, but our language centers (the parts of the brain that handle sound and meaning) jump in and help. They say, "Hey, those dots actually spell 'CAT'!" This top-down help allows us to read Braille effortlessly, even though it looks nothing like a letter.

The Takeaway

Current AI models are great at recognizing pictures, but they are missing the conversation between the eyes and the language center that happens in the human brain. To build a robot that can truly read Braille (or any strange script), we can't just give it better eyes; we have to give it a "language brain" that can talk to its eyes.

In short: Humans read with their minds; current robots only read with their eyes.

Technical Summary

1. Problem Statement

The study addresses a fundamental gap in understanding how the human brain processes non-canonical visual scripts, specifically Braille. While the Visual Word Form Area (VWFA) in the human brain adapts to process diverse scripts (including Braille, lip-reading, and sign language) by integrating linguistic properties (orthography, phonology, semantics), current feedforward Deep Neural Networks (DNNs)—which serve as computational models of the visual hierarchy—fail to replicate this adaptability.

Existing models (e.g., AlexNet, CORnet Z) suggest that reading relies heavily on a predisposition for line junctions (features common in Latin scripts). However, it remains unclear whether these purely visual, bottom-up models can explain the human ability to expertly read Braille (a dot-based script lacking line junctions) and how linguistic content influences visual representations in the absence of top-down language feedback. The authors hypothesize that simple feedforward visual processing is insufficient to explain human expert performance on Braille and that interactive mechanisms between visual and linguistic systems are required.

2. Methodology

The researchers conducted two experiments using two benchmark DNN architectures: AlexNet and CORnet Z.

Stimuli

Three script types were compared:

1. Latin Alphabet: Standard Arial font (line-based).
2. Braille: Standard dot-based representation.
3. Line Braille: A custom script where Braille dots were connected by lines to form line junctions, controlling for structure while introducing the "line" feature.

Experiment 1: Illiterate Network Analysis

• Goal: To determine if a visual network pre-trained only on natural objects (ImageNet) has an inherent bias toward line junctions over dot-based structures.
• Setup: An "illiterate" AlexNet (pre-trained on ImageNet, not letters) was presented with single letters from the three scripts.
• Analysis: Representational Dissimilarity Matrices (RDMs) were computed using Euclidean distances between layer activations to measure how the network distinguishes between scripts.

Experiment 2: Expertise Acquisition and Linguistic Representation

• Goal: To simulate literacy acquisition and expert processing, and to test if networks develop human-like linguistic representations.
• Setup:
  • Literacy Phase: Networks were trained to classify Dutch words in the Latin alphabet.
  • Expertise Phase: Networks were fine-tuned on one of three conditions: Latin only (Naïve), Latin + Line Braille, or Latin + Braille (Expert).
  • Testing: Both "Naïve" and "Expert" networks were tested on a set of stimuli varying in linguistic properties:
    • Real Words (RW): High frequency, meaningful.
    • Pseudo Words (PW): pronounceable, no meaning.
    • Non Words (NW): low-frequency consonant strings.
    • Fake Script (FS): rearranged line junctions.
• Analysis:
  • Clustering: Measured the separation between linguistic categories vs. within categories.
  • Correlation: Compared network RDMs against a theoretical linguistic model (based on the cumulative number of linguistic properties: orthography, phonology, semantics) and human fMRI data.

3. Key Results

A. Line Junction Bias in Illiterate Networks (Exp 1)

• Even without explicit training on letters, the visual hierarchy of AlexNet showed a strong predisposition for line junctions.
• Line Braille and Latin scripts clustered together in mid-to-high layers, showing high similarity.
• Braille (dot-based) was treated as a distinct outlier with significantly higher dissimilarity compared to line-based scripts.
• Conclusion: The visual system's initial bias favors line junctions, suggesting an inherent advantage for line-based scripts in purely visual processing.

B. Learning Curves and Expertise (Exp 2)

• Performance Gap: While human participants show only a small, temporary delay in learning Braille compared to Line Braille, the DNNs showed a massive and persistent performance gap.
• Convergence Failure: Even after extensive training (10–15 epochs), the networks trained on Braille failed to reach the classification accuracy of those trained on Line Braille or Latin. The "expert" networks did not converge to the same performance levels as the "line-based" experts.
• Architecture Differences: CORnet Z showed a slight improvement in Braille processing in later layers (IT), but both architectures struggled significantly more with Braille than humans do.

C. Linguistic Representation and Clustering

• Lack of Linguistic Tuning: Unlike human VWFA, which organizes representations based on linguistic properties (e.g., separating Real Words from Fake Script regardless of script type), the DNNs failed to cluster stimuli by linguistic category.
• Script-Specific Clustering: Clustering was driven primarily by the visual features of the script (Latin vs. Braille) rather than the linguistic content.
• No Correlation with Human Data: The representational patterns in the expert networks did not correlate with the theoretical linguistic model or with human fMRI data.
  • In humans, expert Braille and Latin representations converge.
  • In DNNs, expert Braille representations remained distinct and did not align with the linguistic model.

4. Key Contributions

1. Demonstration of Model Limitations: The study provides empirical evidence that current state-of-the-art feedforward vision models (AlexNet, CORnet Z) cannot fully explain human expert reading of non-canonical scripts like Braille.
2. Quantification of the "Line Junction" Bias: It confirms that purely visual networks possess a strong, persistent bias for line junctions that hinders the processing of dot-based scripts, a bias that is much stronger and longer-lasting in AI than in human learners.
3. Decoupling Visual and Linguistic Processing: The results show that without explicit language feedback, visual networks do not naturally develop representations that align with linguistic categories (semantics, phonology).
4. Benchmarking Human vs. AI Learning: It highlights a critical discrepancy: humans compensate for the lack of line junctions in Braille through rapid linguistic integration, whereas AI models require mechanisms beyond simple bottom-up visual processing to achieve similar flexibility.

5. Significance and Implications

• Theoretical Impact: The findings challenge the "visual recycling" hypothesis if interpreted strictly as a bottom-up visual process. Instead, they support an interactive account where the VWFA's selectivity for scripts relies on bidirectional connections with language areas (phonology, semantics).
• Future Directions: The authors argue that future computational models of reading must move beyond pure feedforward vision. They propose integrating Vision-Language Models (VLMs) (e.g., CLIP) or combining visual networks with specific language-processing layers to simulate the interactive mechanisms that allow humans to read Braille fluently.
• Neuroscience Relevance: The study suggests that the human brain's ability to read Braille is not merely a visual adaptation but a result of the visual system being "tuned" by linguistic constraints, a process currently missing in standard DNNs.