🧬 biology

Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization

The paper introduces WiCAT, a multi-subject foundation model for widefield calcium imaging that utilizes atlas-aligned spatiotemporal tokenization and self-supervised pretraining to achieve superior performance over single-session baselines while enabling zero-shot behavior decoding and brain region reconstruction on unseen subjects.

Original authors: Mohammad Hosseini, Eray Erturk, Saba Hashemi, Maryam M. Shanechi

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad Hosseini, Eray Erturk, Saba Hashemi, Maryam M. Shanechi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain is a massive, bustling city. Usually, when scientists want to study how this city works, they look at just one neighborhood at a time, or maybe they study one specific person's commute. They build a map for that person, on that day, for that specific task. But what if you wanted a map that works for everyone in the city, instantly, without needing to relearn the streets every time a new person arrives? That's the big dream, and until now, it's been a bit of a puzzle.

Enter WiCAT, a new "brain translator" created by researchers at USC. Think of WiCAT as a super-smart tour guide who has studied the blueprints of the entire city (the brain's anatomy) rather than just memorizing the habits of individual tourists.

The Problem: Too Much Noise, Too Many Maps

Widefield calcium imaging is like a high-definition drone camera flying over the brain's surface. It captures the glowing activity of millions of neurons at once, creating a massive, swirling video of the brain in action. But here's the catch: these videos are huge, messy, and full of "background noise"—like the brain humming to itself or reacting to things that have nothing to do with the task at hand.

Because of this mess, scientists usually had to throw away the data from one session and start fresh for the next. They couldn't easily combine data from 38 different mice, 378 different recording sessions, and two different types of tasks to build one giant, powerful model. It was like trying to learn a language by only reading one sentence from one book, then throwing the book away and starting over with a different book.

The Solution: The "Atlas" Trick

The researchers realized that while every mouse is unique, their brain "city maps" are actually built on the same blueprint. To solve the problem, they invented a way to force every single mouse's brain data to fit onto the same standard map, called the Allen Brain Atlas.

Think of it like this: Imagine you have 38 different people drawing maps of their own houses. One uses a grid of 10x10 squares, another uses 20x20, and they all draw their kitchens in slightly different spots. WiCAT takes all these messy drawings and snaps them onto a single, perfect, pre-drawn grid. Suddenly, the "kitchen" of Mouse A lines up exactly with the "kitchen" of Mouse B.

Once everything is aligned, WiCAT chops the brain video into tiny puzzle pieces (tokens) and feeds them into a giant AI brain (a Transformer) that learns to predict what the missing pieces look like. It's like playing a game where you cover up 90% of a brain video and ask the AI to guess what's hidden underneath. To win, the AI has to learn the rules of how the brain moves and thinks, not just memorize the specific faces of the mice it's seen before.

The Magic: Zero-Shot Superpowers

The real magic happens when they test WiCAT on a mouse it has never seen before.

In the past, if you showed a new mouse to an old model, the model would be confused and fail. But WiCAT is different. Because it learned the "universal rules" of the brain city during its training, it can look at a brand-new mouse and immediately start decoding its behavior.

Zero-Shot Behavior Decoding: The model can watch a new mouse and guess what it's doing (like reaching for a lever or turning its head) with surprising accuracy, without needing any practice data from that specific mouse first. It's like meeting a stranger and instantly knowing how they walk, just because you've studied the blueprint of human legs.
Filling in the Blanks: The model can even guess what's happening in a part of the brain it can't "see" (a region that was masked out), just by looking at the activity in the rest of the brain. It's like being able to hear a conversation in a room you can't see, just by listening to the muffled sounds from the hallway.

What WiCAT is NOT (and what it rules out)

It's important to know what this model doesn't do, because the paper is very clear about that.

It's not a "one-size-fits-all" magic wand that ignores anatomy. The paper explicitly argues against models that try to learn without a map. If you try to train a model without aligning the brains to the atlas, it fails to generalize to new subjects. The "map" is essential.
It doesn't need to be retrained for every new mouse. The paper tested models that tried to learn "session-specific" details (like a unique ID for each mouse). Those models failed when faced with a new mouse. WiCAT proves you don't need to retrain the whole system for every new animal; the shared "brain language" is enough.
It's not a "perfect" solution yet. The paper suggests that while the results are strong, the model still relies on the brain being aligned correctly. If the map is twisted or misaligned, the model's performance drops. It's robust, but not invincible.

The Numbers (The Proof)

The researchers didn't just guess; they measured it.

They trained on data from 38 subjects across 378 sessions.
When tested on new, unseen mice, WiCAT achieved a behavior decoding score (R²) of 0.3274 on the Musall dataset and 0.1840 on the Kondo dataset.
Compare that to the old "single-session" models, which scored around 0.0477 and 0.0573 respectively on the same new mice. That's a huge jump!
Even better, if you give the model just a tiny bit of help (about 16 trials or roughly 100 seconds of recording), it can adapt and get even better, reaching scores of 0.4336 and 0.2869.

The Bottom Line

WiCAT is a major step toward a "foundation model" for the brain—a single, powerful AI that understands the general language of brain activity across different animals and tasks. It suggests that if we align our data correctly and teach the AI to look for universal patterns rather than specific details, we can build models that generalize to new subjects instantly.

The paper doesn't claim this solves everything or that it works for every type of brain data (like deep brain recordings) yet. But for widefield calcium imaging, it suggests a new way forward: stop treating every brain as a unique puzzle and start teaching the AI the shared blueprint that connects them all. And the best part? The code is open, so anyone can try to build on this "brain translator" themselves.

Technical Summary: Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization

Problem Statement
Widefield calcium imaging offers unprecedented access to brain-wide cortical dynamics in behaving animals, providing mesoscale neural activity with high temporal resolution and broad spatial coverage. However, the high dimensionality, complex spatiotemporal structure, and substantial task-irrelevant activity in these recordings have historically restricted modeling efforts to single-session or single-subject analyses. Existing approaches typically rely on session-specific parameters or preprocessing steps (e.g., PCA, LocaNMF) that limit scalability and cross-subject generalization. While multi-subject pretrained models have shown promise in other neural modalities (e.g., spiking, LFP), they have not been demonstrated for widefield calcium imaging. Furthermore, achieving subject-invariant zero-shot behavior decoding and neural reconstruction across unseen subjects remains an open challenge.

Methodology: WiCAT
The authors introduce WiCAT (Widefield Calcium imaging Atlas-aligned Tokenizer model), a multi-subject foundation model designed to learn shared spatiotemporal representations across diverse subjects, sessions, and datasets. The architecture consists of three core components:

Atlas-Grounded Tokenization: To enable parameter sharing across subjects with varying cortical morphologies and recording setups, all widefield recordings are spatially registered to a common anatomical reference frame (the Allen Brain Atlas Common Coordinate Framework) and resampled to a unified spatial resolution (128 × 128). The aligned frames are partitioned into non-overlapping spatiotemporal patches. Crucially, the model employs global spatial embeddings associated with each atlas-aligned patch index. These embeddings are shared across all subjects, sessions, and datasets, providing a consistent representation of cortical location without session-specific components.
Spatiotemporal Transformer Backbone: The tokenized data is processed by a Transformer encoder that applies joint spatial and temporal attention. Rotary positional embeddings (RoPE) are used to encode relative temporal information. The model operates on a unified sequence of tokens, allowing it to capture interactions across cortical regions and time within a single attention mechanism.
Self-Supervised Pretraining (MAE): The model is pretrained using a Masked Autoencoder (MAE) objective. During pretraining, 90% of spatiotemporal tokens are randomly masked across space and time. The encoder processes only the unmasked tokens, while a lightweight Transformer decoder reconstructs the masked patches using a Mean Squared Error (MSE) loss. Critically, no subject, session, or dataset identifiers are provided during pretraining; the model must rely solely on the sparse context and shared global spatial embeddings to reconstruct neural activity.

Key Contributions

Multi-Subject Modeling: WiCAT is the first model to map neural recordings from different subjects into a shared representation space for widefield calcium imaging, moving beyond the single-session limitations of prior work.
Atlas-Based Tokenization: The authors propose a tokenization scheme using global spatial embeddings that capture subject-invariant anatomical organization, enabling cross-subject modeling without session-specific parameters.
Challenging Pretraining Framework: A self-supervised MAE framework is designed to learn spatiotemporal representations that reconstruct neural activity without access to subject identifiers, forcing the model to learn global dynamics rather than session-specific noise.
Zero-Shot Capabilities: The model demonstrates robust zero-shot continuous behavior decoding and efficient few-shot adaptation on unseen subjects. It also achieves zero-shot reconstruction of left-out brain regions, indicating the learned representations capture cross-region dependencies that generalize across subjects.
Cross-Dataset Transfer: The pretrained representations transfer effectively across datasets with different recording setups, spatial resolutions, and behavioral tasks.

Results
Evaluated on two large-scale datasets (Musall and Kondo) spanning 38 subjects and 378 sessions, WiCAT outperformed baseline models, including single-session dynamical models (SBIND), linear regression on PCA/LocaNMF features, and multi-session extensions of SBIND.

Behavior Decoding: WiCAT achieved significantly higher behavior decoding performance ( $R^2$ ) on both finetune and zero-shot sets compared to all baselines. Notably, on the zero-shot set (unseen subjects), WiCAT Linear probing achieved $R^2$ scores of 0.3274 (Musall) and 0.1840 (Kondo), substantially outperforming the next best multi-session baseline.
Zero-Shot Generalization: The model maintained strong performance on unseen subjects without any gradient updates to the backbone or decoder.
Neural Reconstruction: In zero-shot left-out region reconstruction tasks, WiCAT consistently achieved lower reconstruction error (MSE) than a linear cross-region decoder baseline across major cortical areas (e.g., SS, VIS, MO), preserving region-specific temporal structure.
Ablation Studies: Removing global spatial embeddings or introducing session-specific parameters significantly degraded zero-shot performance, confirming the necessity of the proposed architecture for cross-subject transfer. Additionally, MAE pretraining outperformed JEPA-style pretraining for this modality.
Scaling: Performance improved with increased pretraining data volume and model capacity, suggesting the framework can leverage larger datasets effectively.

Significance
The paper positions WiCAT as a foundational step toward "foundation-style" modeling for widefield calcium imaging. By demonstrating that shared representations can be learned across subjects and sessions without session-specific tuning, the work addresses critical barriers to scalability and generalization in neural imaging. The ability to perform zero-shot behavior decoding and neural reconstruction on unseen subjects indicates that the model captures fundamental, subject-invariant spatiotemporal dynamics of cortical activity. This approach facilitates the reuse of learned representations across animals, sessions, and experimental conditions, potentially accelerating research into distributed population dynamics and large-scale functional organization. The authors note that while the current scale of public widefield data is smaller than other modalities, the framework is designed to scale as larger, more diverse datasets become available.