Distinct cortical regions support the coding of order across visual and auditory working memory

This study demonstrates that while visual and auditory working memory share a common frontoparietal network for order encoding, they rely on distinct neural mechanisms in the inferior frontal gyrus, anterior intraparietal sulcus, and posterior hippocampus to process serial positions, suggesting modality-specific spatial and temporal coding strategies.

The Gist

The Big Idea: How Our Brain Organizes a List

Imagine you are trying to remember a shopping list: Bananas, Blueberries, Cherries, and Dates.

You don't just remember the items; you remember the order. But how does your brain do that? Does it just keep a mental to-do list? Or does it actually place those items on a mental map, like putting them on a shelf from left to right?

This study investigates a phenomenon called the SPoARC effect. This is a fancy way of saying that our brains often turn a list of words into a mental line.

• The Rule: We tend to put the first items on the left and the last items on the right.
• The Proof: If you are asked to sort these fruits, you will press a "Left" button faster for "Bananas" (the first item) and a "Right" button faster for "Dates" (the last item).

The Mystery: Does It Matter How You Hear or See the List?

Previous studies showed this "mental line" trick works when you see the words. But what if you hear them?

• If someone reads the list to you, do you still build a left-to-right line in your head?
• Or does your brain use a completely different strategy for sounds?

The researchers wanted to know: Is the brain's "filing cabinet" for lists the same whether the information comes through your eyes or your ears?

The Experiment: A Brain Scan Adventure

The researchers put 49 people inside an MRI machine (a giant camera that takes pictures of the brain).

1. The Visual Task: People saw pictures of fruits appear on a screen one by one.
2. The Auditory Task: People heard the names of the same fruits played through headphones.
3. The Test: After memorizing the list, they had to identify if a fruit was part of the list, pressing buttons with their left or right hands.

While they did this, the scientists looked at two things:

1. How loud the brain was shouting (Univariate analysis).
2. The specific pattern of the shout (Multivariate analysis). Think of this like the difference between hearing a crowd cheer "Woo!" (loudness) vs. hearing the crowd chant a specific song (pattern).

The Findings: Same Building, Different Rooms

1. The "Loudness" Check (Univariate Results)

When looking at how much the brain lit up, the results were surprisingly similar. Whether the list was seen or heard, the brain used the same major "office buildings" (the front and top parts of the brain) to do the work.

• Analogy: It's like ordering a pizza. Whether you order by phone or online, the same kitchen (the brain's working memory network) is cooking the pizza.

2. The "Pattern" Check (Multivariate Results)

This is where it got interesting. When the scientists looked at the specific patterns of activity (the "song" the brain was singing), they found differences based on how the information arrived.

• The Visual List (Seeing): The brain used the Parietal Lobe (a region near the top-back of the brain, like a "spatial map center").
  • Analogy: When you see the list, your brain treats it like a physical shelf. It places the first item on the left and the last on the right. It uses a 2D map (like a piece of paper).
• The Auditory List (Hearing): The brain leaned heavily on the Hippocampus (a deep, seahorse-shaped structure usually known for long-term memory and navigation).
  • Analogy: When you hear the list, there is no physical "left" or "right" on the air. So, your brain builds a more complex, 3D "cognitive map" (like a GPS system) to keep track of the order. It's a higher-dimensional space.

The "Mental Whiteboard" vs. The "GPS"

The authors propose a cool theory based on these findings:

• Visual Input = The Mental Whiteboard: When you see things, your brain naturally draws a line on a whiteboard. It's simple, flat, and left-to-right. This is why the "SPoARC effect" (the left-hand/right-hand speed difference) is very strong when you see the list.
• Auditory Input = The GPS System: When you hear things, your brain can't draw a line on a whiteboard because sound doesn't have a shape. Instead, it uses a complex, multi-layered map (like a GPS) to navigate the sequence. This map is so complex that it doesn't naturally translate to a simple "left vs. right" button press, which is why the SPoARC effect was weaker or missing in the hearing task.

The Takeaway

Our brains are incredibly adaptable.

• If you show us a list, we turn it into a picture on a shelf (using the Parietal Lobe).
• If you tell us a list, we turn it into a journey through a complex map (using the Hippocampus).

Even though we are doing the same task (remembering a list), the brain switches its internal software depending on whether the input is visual or auditory. It's not just one "memory center"; it's a flexible system that builds different kinds of mental maps for different kinds of information.

Technical Summary

1. Problem Statement and Research Gap

• Core Phenomenon: The study investigates the SPoARC effect (Spatial-Positional Association of Response Codes), where individuals implicitly map the serial order of non-spatial items (e.g., words) onto a mental spatial line (left-to-right). This supports the "Mental Whiteboard" hypothesis, suggesting order is coded via spatial markers.
• The Gap: While the SPoARC effect has been robustly demonstrated for visual-verbal stimuli, its neural mechanisms in the auditory-verbal modality remain unclear. Previous neuroimaging studies on SPoARC (e.g., Cristoforetti et al., 2022; Zhou et al., 2021) used exclusively visual stimuli.
• Key Question: Do visual and auditory working memory (WM) tasks rely on shared, modality-independent neural mechanisms for order coding, or do they engage distinct, modality-specific patterns of activation within the fronto-parietal network and hippocampus?

2. Methodology

• Participants: 49 French-speaking right-handed students (final $N=41$ after exclusions for motion and hearing issues).
• Task Design: A within-subjects fMRI experiment comparing Visual and Auditory SPoARC tasks using the same verbal material (10 fruit names).
  • Procedure: Participants memorized sequences of 4 items (encoding phase) followed by a recognition phase where they judged if probe items belonged to the sequence using left/right hand responses.
  • Behavioral Measure: Reaction times (RT) were analyzed to detect the SPoARC effect (faster left-hand responses for early sequence positions, faster right-hand for late positions).
• Imaging Protocol:
  • Scanner: 3T GE Discovery.
  • Sequence: Whole-brain EPI (TR=2000ms).
  • Analysis Strategy:
    1. Univariate Analysis (GLM): To identify general activation networks and compare mean activation amplitudes between modalities.
    2. Multivariate Pattern Analysis (MVPA) & Representational Similarity Analysis (RSA): To decode the pattern of activity rather than just amplitude.
       • Regions of Interest (ROIs): Anterior/Posterior Intraparietal Sulcus (IPS), Inferior/Middle/ Superior Frontal Gyri (IFG, MFG, SFG), Anterior/Posterior Hippocampus, and SMA.
       • Theoretical Models: Neural Representational Dissimilarity Matrices (RDMs) were compared against four theoretical models:
         • Across-modality vs. Within-modality: Do patterns look similar across vision/hearing, or distinct?
         • Early-late vs. Boundary: Do patterns distinguish early (1-2) vs. late (3-4) positions, or boundary (1-4) vs. middle (2-3) positions?

3. Key Results

A. Behavioral Results

• Visual Modality: A significant SPoARC effect was observed (participants responded faster with the left hand to early items and right hand to late items).
• Auditory Modality: The SPoARC effect was not significant.
  • Interpretation: The authors suggest this may be due to scanner noise interfering with auditory processing or distinct memorization strategies, though accuracy was comparable between modalities.

B. Univariate fMRI Results (Activation Amplitude)

• Shared Network: Both modalities activated a canonical fronto-parieto-cerebellar network (including DLPFC, IPS, SMA, and cerebellum), consistent with general working memory processing.
• Modality-Specific Sensory Areas:
  • Visual: Stronger activation in occipital gyri (visual processing).
  • Auditory: Stronger activation in Heschl's gyrus (auditory processing) and greater activation in the IFG and posterior IPS during encoding (potentially reflecting higher attentional demands).
• Conclusion: The magnitude of activation in the core WM network is largely similar across modalities, but sensory processing remains distinct.

C. Multivariate Results (Activation Patterns & RSA)

This was the study's critical finding, revealing differences invisible to univariate analysis:

• Modality-Specific Patterns: Despite similar activation levels, the neural patterns (spatial distribution of voxels) were distinct between visual and auditory conditions in the IFG, IPS, and Hippocampus.
• Position Coding (Early vs. Late):
  • Visual Modality: Distinct patterns for early vs. late positions were found in the Anterior IPS and IFG.
  • Auditory Modality: Distinct patterns for early vs. late positions were found in the Posterior Hippocampus and IFG.
  • SMA: Showed robust position coding (early vs. late) that was modality-independent (shared across both).

4. Key Contributions and Theoretical Framework

The study proposes a novel framework based on Bottini & Doeller (2020) regarding internal representational spaces:

1. Visual Order Coding (IPS):

   • Visual stimuli (presented on a 2D screen) naturally encourage low-dimensional spatial coding (e.g., a simple left-to-right mental line).
   • This relies heavily on the Intraparietal Sulcus (IPS), which supports egocentric, low-dimensional spatial attention and mapping.
   • Result: Strong SPoARC effect; distinct patterns in Anterior IPS.

2. Auditory Order Coding (Hippocampus):

   • Auditory stimuli lack an inherent spatial layout. To encode order, the brain may rely on high-dimensional, allocentric spatial representations or complex temporal maps.
   • This relies heavily on the Hippocampus, which is known for complex spatial navigation and encoding non-spatial dimensions in high-dimensional spaces.
   • Result: Weaker/absent SPoARC effect (as the mapping isn't a simple left-right line); distinct patterns in Posterior Hippocampus.

3. Shared vs. Distinct Mechanisms:

   • The study challenges the idea of a purely modality-general order code. While the network (fronto-parietal) is shared, the representational format within specific nodes (IPS vs. Hippocampus) is modality-dependent.

5. Significance and Limitations

• Significance:

  • First direct fMRI comparison of visual vs. auditory SPoARC.
  • Demonstrates that multivariate pattern analysis is necessary to detect modality differences that univariate amplitude analysis misses.
  • Provides neural evidence that the brain adapts its order-coding strategy (spatial vs. temporal/complex spatial) based on the input modality.
  • Bridges the gap between working memory theories and spatial navigation frameworks (hippocampal involvement in non-spatial WM).

• Limitations:

  • Scanner Noise: The lack of SPoARC in the auditory condition may be confounded by MRI acoustic noise, making it difficult to isolate the pure modality effect.
  • Task Design: The task did not perfectly dissociate item memory from order memory.
  • Recognition Phase: The signal-to-noise ratio was lower for the recognition phase, leading to less robust findings for that stage compared to encoding.

Conclusion

The paper concludes that order coding in working memory is not a single, uniform mechanism. Instead, it is a flexible process where the Intraparietal Sulcus supports low-dimensional, egocentric spatial coding (dominant in vision), while the Hippocampus supports high-dimensional, allocentric or temporal coding (dominant in audition). This suggests that the "Mental Whiteboard" is not a single static board but a dynamic system that recruits different cortical substrates depending on the sensory nature of the information.