Bottom-up and generative computations uniquely explain neural responses across the social brain

This preregistered fMRI study challenges the traditional view of a strict spatial division between social perception and mentalizing brain regions by demonstrating that both posterior STS and TPJ integrate bottom-up relational processing and top-down generative inverse-planning computations, likely operating on distinct temporal scales.

The Gist

Imagine your brain is a bustling newsroom trying to figure out a chaotic scene: two animated shapes are moving around a screen. One is chasing the other. Is it a playful game of tag? Is it a bully chasing a victim? Or are they just ignoring each other?

For decades, scientists believed the brain solved this mystery using a strict assembly line. They thought:

1. The "Perception Team" (located in the back of the brain, called pSTS) would quickly look at the shapes and say, "Hey, they are touching! One is moving faster!" (Bottom-up processing).
2. The "Mentalizing Team" (located further forward, called TPJ) would then take that report and think, "Okay, since they are touching and moving fast, the first one intends to tag the second one." (Top-down, goal-based reasoning).

The Big Question: Is the brain really a factory with separate stations for "seeing" and "thinking"? Or do these teams work together in the same room?

The Experiment: A New Kind of Detective Work

The researchers (Manasi Malik and her team) decided to test this by building two "AI detectives" and seeing which one matched the brain's activity best.

1. Detective "SocialGNN" (The Bottom-Up Observer): This AI is like a security camera with a super-fast eye. It doesn't know about "goals" or "feelings." It only looks at the raw data: Where are the shapes? How fast are they moving? Did they bump into each other? It builds a picture based purely on what it sees.
2. Detective "SIMPLE" (The Inverse-Planner): This AI is like a Sherlock Holmes. It doesn't just watch; it imagines. It asks, "If the red shape wanted to catch the blue shape, what path would it take?" It simulates the world in its head, guessing the agents' hidden goals and beliefs to explain the movement.

They put 25 people in an MRI machine, showed them videos of these animated shapes, and watched which parts of their brains lit up. Then, they compared the brain's "light patterns" to the "thought patterns" of their two AI detectives.

The Surprise: The Brain is a Hybrid

The researchers expected to find that the "Perception Team" (pSTS) only liked the security camera AI, and the "Mentalizing Team" (TPJ) only liked the Sherlock Holmes AI.

They were wrong.

Instead, they found that both brain regions loved both detectives.

• The pSTS (the "perception" area) wasn't just watching the shapes; it was also figuring out the hidden goals, just like the Sherlock AI.
• The TPJ (the "thinking" area) wasn't just guessing goals; it was also paying close attention to the raw movement and bumps, just like the security camera AI.

The Analogy: It's not a factory assembly line where the product moves from Station A to Station B. It's more like a kitchen.

• Imagine a chef (the brain) making a complex dish.
• You might think the "chopping station" only chops vegetables and the "sauce station" only mixes spices.
• But this study shows that the chef at the chopping station is also tasting the sauce, and the chef at the sauce station is also chopping vegetables. They are both doing everything, just at slightly different speeds.

The Twist: It's About Timing, Not Location

If both brain areas are doing both types of work, why do we have two different areas? The answer might be time, not space.

The researchers looked at when the brain reacted during the 10-second video:

1. Early in the video: The "Security Camera" (SocialGNN) style of thinking fired up first. The brain quickly grabbed the visual facts: "They are moving! They bumped!"
2. Later in the video: The "Sherlock Holmes" (SIMPLE) style of thinking took over. The brain started to slow down and reason: "Oh, I see, the red one is trying to trap the blue one."

So, the brain isn't split into "seeing" and "thinking" rooms. Instead, it's a single, powerful engine that starts with fast visual observation and gradually shifts into deep, goal-based reasoning.

Why This Matters

This changes how we understand human social intelligence. We don't just "see" a social interaction and then "think" about it later. Our brains are constantly running two simulations at once: one that tracks the physical world and one that predicts the mental world.

It's like driving a car. You don't just look at the road (bottom-up) and then later decide to turn the wheel (top-down). You are constantly doing both: your eyes track the lane lines while your brain predicts where the other car is going to be in three seconds. The brain is a master of doing both at the same time, using different tools to solve the complex puzzle of human connection.

Technical Summary

1. Problem Statement

The human ability to infer social interactions (e.g., cooperation vs. competition) from visual input relies on a network of brain regions, primarily the posterior superior temporal sulcus (pSTS) and the temporoparietal junction (TPJ).

• The Hypothesis: A prevailing theory suggests a strict hierarchical and spatial division of labor:
  • pSTS (Social Perception): Implements fast, bottom-up relational computations based on visual features.
  • TPJ (Mentalizing): Performs slower, generative inverse-planning computations to infer latent goals, beliefs, and intentions.
• The Gap: This computational-neural mapping has never been formally tested because successful computational models capable of distinguishing between these two specific processing strategies (relational bottom-up vs. generative inverse-planning) were previously unavailable.

2. Methodology

Experimental Design

• Participants: 25 adults (after exclusions) viewed 10-second videos of animated agents interacting in a 2D environment (from the PHASE dataset).
• Stimuli: Videos depicted "friendly," "neutral," or "adversarial" interactions generated by a physics simulator and hierarchical planner.
• Data Collection: Whole-brain fMRI was recorded. Participants also completed functional localizers to define Regions of Interest (ROIs) for social perception (pSTS) and theory-of-mind (TPJ). Post-scan behavioral ratings were collected to create a ground-truth behavioral Representational Dissimilarity Matrix (RDM).

Computational Models

The study compared neural data against two distinct computational models:

1. SocialGNN (Bottom-Up/Relational):
   • A Graph Neural Network (GNN) that processes video frames as graphs.
   • Nodes: Agents/objects with features (position, velocity, orientation).
   • Edges: Physical contacts.
   • Mechanism: Relies purely on visual features and relational inductive biases without explicit goal inference. It uses an LSTM to integrate temporal information.
2. SIMPLE (Generative/Inverse-Planning):
   • A generative model that simulates agents' mental states.
   • Mechanism: Hypothesizes physical/social goals, simulates trajectories using a physics engine and hierarchical planner, and compares simulated trajectories to observed ones to infer the most likely relationship (inverse planning).

Analysis Pipeline

• Representational Similarity Analysis (RSA): Neural RDMs (derived from fMRI activity patterns) were compared against model RDMs and behavioral RDMs using Spearman's rank correlation.
• Variance Partitioning: Semi-partial correlations were used to determine the unique variance explained by each model while controlling for the other, low-level visual features (Motion Energy), and non-relational baselines (ControlRNN).
• Time-Resolved Analysis: The 10-second videos were split into five 2-second windows to examine temporal dynamics.

3. Key Results

A. Human Ratings and Neural Activity

• Human behavioral judgments (friendly/neutral/adversarial) significantly correlated with neural representations in both pSTS and TPJ, even after controlling for low-level motion energy. This confirms these regions are sensitive to high-level social categories.

B. Model-Brain Correspondence (The Core Finding)

Contrary to the strict spatial hierarchy hypothesis:

• Both Models Explain Both Regions: Both SocialGNN (bottom-up) and SIMPLE (inverse-planning) showed significant representational similarity to neural activity in both the pSTS and the TPJ.
• Unique Variance: Variance partitioning revealed that each model explained unique variance in both regions. This indicates that pSTS and TPJ do not exclusively perform one type of computation; rather, both regions support a combination of relational bottom-up processing and higher-order inferential reasoning.
• Relational Necessity: The relational structure of SocialGNN was critical. A control model (ControlRNN) without graph structure failed to explain variance in pSTS and TPJ, highlighting that relational processing is essential for social brain responses.

C. Temporal Dynamics

Exploratory time-resolved analysis revealed a temporal, rather than spatial, hierarchy:

• SocialGNN (Bottom-up): Showed a "rise-and-fall" pattern, peaking in the early-to-middle windows of the video.
• SIMPLE (Inverse-planning): Showed a gradual, steady increase, becoming more dominant in later windows.
• Interaction: A significant Model × Time interaction was found in pSTS, suggesting that while both computations occur in both regions, they unfold on different timescales (fast visual processing followed by slower inferential reasoning).

4. Key Contributions

1. First Formal Test of Computational-Neural Mapping: This is the first study to formally test the hypothesis that pSTS and TPJ implement specific computational strategies (bottom-up vs. inverse-planning) using distinct, validated models.
2. Refutation of Strict Spatial Segregation: The findings challenge the view that social perception and mentalizing are strictly segregated by brain region. Instead, both regions appear to integrate both types of computations.
3. Temporal Hierarchy Proposal: The study proposes that the "hierarchy" in social cognition may be temporal (fast bottom-up followed by slow generative inference) rather than strictly spatial (pSTS $\to$ TPJ).
4. Model Validation: It demonstrates that theory-driven cognitive models (specifically inverse-planning) can successfully predict fMRI responses to complex social scenes, bridging the gap between cognitive science and neuroscience.

5. Significance

• Theoretical Impact: The results suggest a more integrated and dynamic architecture for the "social brain," where regions like pSTS and TPJ are not isolated processors but rather hubs that combine rapid perceptual structuring with deep inferential reasoning.
• Methodological Advance: The study establishes a robust framework for using procedurally generated stimuli combined with computational modeling to dissect neural mechanisms. This approach allows for precise control over variables that is difficult to achieve with naturalistic stimuli.
• Future Directions: The findings open avenues for developing hybrid models that integrate bottom-up priors with top-down generative planning. It also suggests that future research should focus on the temporal dynamics of social inference using high-temporal-resolution methods (e.g., intracranial recordings) to distinguish between within-region processing and rapid inter-regional communication.

In summary, the paper argues that the social brain does not simply pass information from a "perception" module to a "reasoning" module. Instead, both the pSTS and TPJ simultaneously or sequentially engage in a dual-process system: rapidly extracting relational visual features and gradually inferring the underlying mental states driving those interactions.