Neuro-Symbolic Decoding of Neural Activity

The paper introduces NEURONA, a neuro-symbolic framework that integrates symbolic reasoning with fMRI grounding to decode interacting visual concepts, demonstrating that incorporating structural priors significantly enhances both decoding accuracy and generalization to unseen queries.

The Gist

Imagine your brain is a massive, bustling library. Inside, billions of neurons are constantly whispering to each other, creating a complex, noisy symphony of thoughts, memories, and perceptions whenever you look at something.

For decades, scientists have tried to "read" this library. They use a machine called an fMRI scanner (think of it as a high-tech camera that takes pictures of the brain's activity) to see which parts of the library light up when you see a picture of a cat or a car.

The Problem: The "Blurry Photo" Approach
Most previous attempts to decode these brain signals were like trying to guess the plot of a movie by looking at a single, blurry, black-and-white photo of the theater.

• Simple models were like trying to guess the movie just by counting how many people were in the room. They missed the details.
• Complex AI models were like trying to guess the movie by looking at the whole theater at once. They could tell you "It's a movie," but they struggled to explain exactly what was happening in the scene (e.g., "Is the person holding the bat, or is the bat holding the person?"). They treated the brain's activity as one big, messy blob rather than a structured story.

The Solution: NEURONA (The Detective with a Script)
This paper introduces a new framework called NEURONA. Think of NEURONA not as a blurry camera, but as a super-smart detective who has a specific script (a "symbolic" plan) for solving the mystery.

Here is how NEURONA works, using a simple analogy:

1. The Script (Symbolic Reasoning)

Imagine you are asked a question about a picture: "Is there a person holding a baseball bat?"
Old AI models might just guess "Yes" or "No" based on a gut feeling. NEURONA, however, breaks the question down into a logical script, like a recipe:

• Step 1: Find the "Person."
• Step 2: Find the "Baseball Bat."
• Step 3: Check if the "Person" and "Bat" are connected by the action "Holding."

2. The Map (Neural Grounding)

Now, the detective looks at the brain's library (the fMRI data).

• Instead of looking at the whole library at once, NEURONA asks: "Okay, which specific shelves in the library are talking about 'People'? Which shelves are talking about 'Bats'?"
• It turns out that different parts of the brain handle different concepts. Some parts handle "people," others handle "tools," and others handle "actions."

3. The Magic Trick (Compositional Execution)

This is where NEURONA shines. It doesn't just look for "holding" in isolation. It uses the script to guide its search.

• It says: "I found the 'Person' on Shelf A. I found the 'Bat' on Shelf B. Now, let's look specifically at the connection between Shelf A and Shelf B to see if the 'Holding' action is happening there."

It's like a detective who knows that if you are looking for a "kiss," you shouldn't just look at the lips; you should look at the space between two people. By understanding the relationship between the parts, NEURONA can solve the puzzle much better than models that just look at the parts separately.

Why This Matters

The paper tested this on two huge datasets (BOLD5000 and CNeuroMod), which are like massive libraries of brain scans while people looked at thousands of images and videos.

• The Result: NEURONA was significantly better at answering questions like "What is the person doing?" or "Where is the cat sitting?" compared to all previous methods.
• The Superpower: The best part? NEURONA could answer questions it had never seen before. If it learned how "holding" works with a "person" and a "bat," it could instantly figure out how "holding" works with a "woman" and a "coffee cup." It understood the logic of the relationship, not just the specific words.

The Big Picture

In simple terms, this research shows that the brain organizes its thoughts like a structured story, not just a random pile of words. By building an AI that respects this structure—by treating the brain like a library with a logical filing system rather than a messy pile of papers—we can finally start reading the human mind with much greater clarity and accuracy.

NEURONA is the first step toward a future where we can decode not just what you are seeing, but the complex, relational story of how you are thinking about it.

Technical Summary

Here is a detailed technical summary of the paper "Neuro-Symbolic Decoding of Neural Activity" (NEURONA), published as a conference paper at ICLR 2026.

1. Problem Statement

The paper addresses a critical gap in functional magnetic resonance imaging (fMRI) decoding: the ability to decode high-level relational meaning and compositional semantics from neural activity.

• Limitations of Current Methods: Existing large-scale fMRI decoding studies typically focus on either isolated concepts (e.g., identifying a single object) or holistic stimulus reconstruction (generating pixel-level images/videos). They often fail to capture the interactions between multiple visual concepts (e.g., "a person holding a baseball bat") and the specific relations binding them.
• The Challenge: Standard linear models lack the capacity to model interactions between components, while purely end-to-end neural decoders (often using language model backbones) tend to encode stimuli holistically. This leads to coarse alignment between brain activity and language, failing to generalize to unseen compositional queries (e.g., predicting the relation between a "surfboard" and a "person" if the model only saw "person" and "kite" during training).
• Goal: To determine if explicitly modeling the compositional structure of cognition (predicates and arguments) improves the accuracy, precision, and generalization of neural decoding.

2. Methodology: NEURONA Framework

The authors propose NEURONA (NEURO-symbolic framework for decoding in Neural Activity), a neuro-symbolic approach that integrates symbolic reasoning with neural network expressivity.

A. Core Architecture

NEURONA treats fMRI decoding as a Question Answering (QA) task where the input is a visual stimulus and an fMRI recording, and the output is an answer to a structured query (e.g., "Is there a person holding a baseball bat?").

1. Symbolic Parsing: Each query is parsed into a symbolic expression (e.g., exists(holding(person, baseball_bat))).
2. Candidate Entity Generation: The fMRI signal is parcellated into functional brain regions (using atlases like Yeo-17, DiFuMo, or Schaefer). These regions serve as candidate neural entities.
3. Concept Grounding Modules:
   • Unary Concepts: For entities (e.g., "person"), a linear layer maps region embeddings to concept-specific scores.
   • Relational Concepts: For predicates (e.g., "holding"), the model computes scores over pairs of regions to capture interactions.
4. Differentiable Executor: A differentiable executor composes the grounded scores according to the symbolic expression to produce a final answer (Boolean or classification).

B. Structural Priors and Hypotheses

A key innovation is the use of structural priors to guide how concepts are grounded. The model tests five hypotheses regarding how predicates relate to their arguments (subject/object):

• H1 (Single-region): Concepts are localized to a single brain region.
• H2 (Multi-region): Concepts rely on co-activation across region pairs.
• H3-H5 (Guided Grounding): The model conditions the predicate grounding on the evidence provided by the subject and/or object arguments.
  • H5 (Full Argument-Guided): The predicate grounding is explicitly guided by the activation patterns of both the subject and object regions. This is the core mechanism of NEURONA, allowing the model to route brain activity through specific concept modules based on the query structure.

C. Datasets: BOLD5000-QA and CNeuroMod-QA

To train and evaluate this framework, the authors constructed two new datasets by pairing existing fMRI data with structured queries:

• BOLD5000-QA: Derived from static images (Scene, COCO, ImageNet). Contains ~135k training examples and 2k test examples.
• CNeuroMod-QA: Derived from naturalistic videos (Friends TV show). Contains ~157k training examples and 30k test examples.
• Generation Process: Scene graphs are extracted from stimuli using vision-language models, then converted into structured QA pairs (e.g., "What is the relation between X and Y?").

3. Key Contributions

1. New Datasets: Introduction of BOLD5000-QA and CNeuroMod-QA, the first large-scale fMRI datasets specifically designed for compositional question answering, emphasizing fine-grained relational semantics.
2. Neuro-Symbolic Framework (NEURONA): A novel architecture that decouples concept grounding from final reasoning, allowing the model to learn how specific brain regions support specific concepts and their interactions without direct supervision on intermediate groundings.
3. Demonstration of Structural Priors: Empirical evidence showing that incorporating predicate-argument dependencies as inductive biases significantly outperforms both linear baselines and state-of-the-art end-to-end neural decoders.
4. Generalization: Proof that neuro-symbolic models generalize robustly to unseen compositions of concepts (e.g., new subject-predicate-object combinations) where purely neural baselines fail.

4. Results

The authors evaluated NEURONA against strong baselines (Linear, UMBRAE, SDRecon, BrainCap) on both datasets.

• Quantitative Performance:

  • NEURONA achieved an overall accuracy of ~70.4% on BOLD5000-QA and 70.5% on CNeuroMod-QA.
  • This represents a ~47% relative improvement over the strongest prior neural baseline (UMBRAE).
  • Gains were most pronounced in Action and Position tasks, which require precise relational reasoning.

• Generalization to Unseen Queries:

  • In zero-shot composition tests (where training and test sets share no overlapping entity-relation pairs), NEURONA maintained high accuracy (~68-69%), while baselines dropped to near-chance levels. This confirms that NEURONA learns systematic recombination rules rather than memorizing specific patterns.

• Ablation Studies:

  • Guided Grounding is Crucial: The "Full Argument-Guided Multi-region" hypothesis (H5) significantly outperformed single-region and unguided multi-region baselines.
  • Predicate-Argument Binding: Analysis showed that guided predicates have significantly higher correlation with their arguments than unguided ones, validating the model's ability to bind relations to specific neural substrates.

• Consistency:

  • Concept grounding was found to be highly consistent across different stimuli and subjects, significantly outperforming a random null model.
  • Qualitative analysis revealed that predicates (e.g., "hold") are decoded from different regions depending on the object (e.g., "kite" vs. "surfboard"), aligning with known neuroscientific findings about motor and prefrontal engagement.

5. Significance and Implications

• Cognitive Science: The results support the Language of Thought (LoT) hypothesis, suggesting that human cognition operates over structured, compositional representations that are reflected in the brain's functional organization.
• Neuroimaging: The work demonstrates that neuro-symbolic modeling is a superior paradigm for decoding complex neural activity compared to purely data-driven deep learning. It suggests that the brain's representation of meaning is not just a holistic embedding but a structured composition of parts.
• Future Directions: The paper highlights the need for datasets with systematic combinatorial manipulations to further validate representational compositionality. It also suggests that scaling NEURONA to voxel-level data is a promising next step.

In summary, NEURONA bridges the gap between symbolic AI and neuroscience, proving that explicitly modeling the structural relationships between concepts leads to more accurate, precise, and generalizable decoding of human brain activity.