🧠 neuroscience

Learning Interpretable Switching Dynamics in Shared Neural-Behavioral Latent Space

This paper introduces Switching Shared Latent Dynamics (SSLD), a unified framework that learns an interpretable shared latent space with nonlinear switching dynamics to simultaneously model the relationship between high-dimensional neural activity and behavior while isolating modality-specific variability across diverse experimental datasets.

Original authors: Zhang, Y., Caro, J. O., Oren, R., Higley, M. J., Cardin, J., Churchland, A., Saxena, S.

Published 2026-10-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhang, Y., Caro, J. O., Oren, R., Higley, M. J., Cardin, J., Churchland, A., Saxena, S.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The brain is a vast, noisy machine. At any given moment, billions of neurons are firing in complex patterns, creating a signal that looks like static on a radio. Yet, within that chaos, specific groups of cells are working together to drive our movements, decisions, and sensations. For decades, scientists have tried to find the hidden thread that connects this neural noise to the clean, observable actions of an animal. The challenge is that the brain does not just react to behavior; it also generates its own internal rhythms and fluctuations that have nothing to do with what the animal is currently doing. To understand how the brain controls behavior, researchers must first separate the signal that matters from the noise that does not. They need a way to find the shared language between the brain and the body, and then understand how that language changes as the animal moves from one state to another, such as from resting to reaching for a target.

A team of researchers at Yale University and UCLA has developed a new method to solve this problem, called Switching Shared Latent Dynamics. Imagine trying to understand a conversation between two people by listening to a recording that contains their voices mixed with the sound of a busy street. The new method acts like a sophisticated filter that can isolate the voices of the two speakers from the street noise, while also tracking how their conversation shifts as the topic changes. In this case, the "speakers" are the brain's neural activity and the animal's behavior, and the "street noise" is the internal brain activity that has nothing to do with the current task. The researchers built a computer model that learns to identify the specific patterns in the brain that are directly linked to what the animal is doing, while ignoring the rest. Crucially, the model recognizes that these patterns do not stay the same; they switch between different modes, much like a car shifting gears, depending on whether the animal is preparing to move, executing a movement, or resting.

The researchers tested this approach on four very different sets of data to see if it would hold up in the real world. They started with a computer simulation where they knew the exact answer, and the model successfully recovered the hidden patterns and the moments when the system switched modes. They then moved to real experiments involving monkeys. In one experiment, monkeys reached for targets in twenty-seven different directions. The model analyzed the electrical activity of 180 neurons in the motor and premotor cortex while the monkeys moved their hands. It found that the information about the hand's movement was concentrated in a specific part of the brain's activity that the model had identified as "shared" between the brain and the body. The rest of the brain activity, which the model labeled as "private" to the neurons, contained complex patterns that did not predict the hand's movement. This confirmed that the model could successfully separate the behavior-driving signals from the brain's internal chatter.

In a second monkey experiment, the animals performed a task where they had to push against a force, a scenario that involves both active movement and passive resistance. The model again succeeded, identifying distinct periods of activity that matched the phases of the task, such as the moment before the monkey moved and the moment it was pushing. The researchers found that the model's ability to spot these switches was far superior to older methods that tried to find patterns without knowing anything about the behavior, or methods that treated the brain's activity as a single, unchanging stream. By explicitly teaching the model to look for these switches in the shared part of the brain's activity, it could pinpoint the exact moments when the brain's strategy changed to match the animal's actions.

The team did not stop with monkeys. They applied the same method to mice, using a different type of recording that captures activity across the entire top surface of the brain. In one set of experiments, mice made their own decisions to perform a task, and in another, the mice simply moved around freely without a specific goal. Even in these more complex and less controlled situations, the model worked. It reconstructed the neural activity and the mouse's movements with high accuracy. It identified that the shared part of the brain's activity contained the information about what the mouse was doing, while the private part held the rest of the brain's activity. When the mice were making decisions, the model detected a specific shift in the brain's dynamics that corresponded to the moment the mouse began to accumulate information before acting.

The results suggest that the brain does not operate in a single, steady state. Instead, the part of the brain that controls behavior seems to jump between different modes of operation, each suited to a specific phase of a task. The new model provides a clear window into these shifts. It shows that when researchers try to understand how the brain controls behavior, they must look for these discrete changes in the shared signals, rather than assuming the brain's activity is a smooth, continuous flow. The study also highlights that ignoring the behavior when analyzing the brain leads to a loss of clarity; by including the behavior in the learning process, the model can better identify which parts of the brain are actually doing the work of controlling the body. This approach offers a more interpretable way to study the brain, moving beyond simple descriptions of activity to a deeper understanding of how neural circuits reorganize themselves to produce the rich variety of behaviors we see in the natural world.

Technical Summary: Learning Interpretable Switching Dynamics in Shared Neural-Behavioral Latent Space

Problem Statement
Modern neuroscience recording technologies now enable the simultaneous measurement of high-dimensional neural activity and rich behavioral variables. This creates a dual challenge: identifying the shared latent structure that mediates the brain-behavior relationship and modeling the underlying dynamics through which this structure evolves across behavioral states. Existing approaches typically address only part of this problem. Shared representation learning methods (e.g., CEBRA, PSID) identify neural-behavioral subspaces but generally fail to model the temporal dynamical structure. Conversely, dynamical models (e.g., SLDS, SRNNs) capture temporal variation and discrete state transitions in neural activity but do not explicitly disentangle shared neural-behavioral structure from modality-specific variability. Consequently, current models struggle to isolate behaviorally relevant signals from neural noise while simultaneously capturing how these signals change across discrete behavioral epochs.

Methodology: Switching Shared Latent Dynamics (SSLD)
The authors introduce Switching Shared Latent Dynamics (SSLD), a unified framework that learns a latent representation shared between neural activity and behavior, endowing this shared space with nonlinear switching recurrent dynamics. The model architecture decomposes information into three latent components:

Shared Latent ( $s_t$ ): Captures information common to both neural activity and behavior. It evolves according to a Switching Recurrent Neural Network (SRNN), where a discrete switching state ( $z_t$ ) selects one of $K$ distinct nonlinear dynamical regimes. This design is motivated by the hypothesis that behaviorally relevant neural representations exhibit structured dynamical changes across behavioral states.
Private Neural Latent ( $h^n_t$ ): Captures neural variability not explained by behavior.
Private Behavioral Latent ( $h^b_t$ ): Captures behavioral variability not explained by the recorded neural activity.

Inference and Training:

Encoders: Neural and behavioral observations are encoded separately using Transformer-based encoders into initial latent representations. These are then processed to extract the shared and private latents via linear networks.
Generative Model: The shared latent evolves via the SRNN process. The observed neural and behavioral signals are reconstructed by concatenating the shared latent with their respective private latents and passing them through decoder networks.
Variational Inference: The model employs amortized variational inference with a factored posterior. Crucially, the inference network conditions on both modalities to infer the shared latent and switching states, unlike previous SRNNs that condition only on neural data.
Loss Function: The objective combines a Variational Evidence Lower Bound (ELBO) with two regularization terms:
- Behavioral Supervision: A cross-entropy loss ( $L_{switch}$ ) encourages the inferred switching states to align with experimentally defined behavioral epochs during training.
- Disentanglement: A Cauchy-Schwarz (CS) divergence term ( $L_{DT}$ ) maximizes the divergence between the shared latent distribution and the private latent distributions, ensuring that behaviorally relevant dynamics are isolated from modality-specific noise.

Key Results
SSLD was evaluated on a simulated dataset and four diverse experimental datasets spanning species (monkey, mouse), brain regions (motor, premotor, somatosensory, dorsal cortex), and recording modalities (electrophysiology, widefield calcium imaging):

Reconstruction Accuracy: SSLD accurately reconstructed both neural population activity and behavioral trajectories across all datasets, outperforming baseline models in mean squared error.
Disentanglement: Linear decoding analyses confirmed that behavioral information was strongly encoded in the shared and private behavioral latents but was largely absent from the private neural latent. This demonstrates the model's ability to separate behaviorally relevant shared structure from neural-specific variability.
State Alignment: The discrete states inferred within the shared latent space aligned significantly better with ground-truth behavioral epochs than states inferred by unsupervised baselines (k-means, HMM, SLDS) or supervised models operating on neural data alone. In the reaching task, SSLD achieved 96% state alignment accuracy, comparable to fully supervised baselines but without requiring direct supervision of the latent dynamics.
Dynamical Interpretability: Visualization of flow fields and line-attractor scores revealed that the model learned distinct, state-dependent dynamical regimes. For example, in the reaching task, the dynamics transitioned from preparatory states to stable movement-execution regimes following the "go cue."
Ablation Studies: Removing any of the three core components—shared representation learning, behavioral supervision, or switching dynamics—resulted in degraded performance. Specifically, linear methods (CCA) failed to generalize, and removing behavioral supervision reduced the model's ability to capture shared structure, while removing switching dynamics lowered state-alignment accuracy.

Significance and Claims
The paper claims that SSLD offers an interpretable approach to modeling shared neural-behavioral dynamics. By placing switching dynamics specifically within the shared latent subspace, the model focuses its discrete state inference on the component of neural activity most directly linked to behavior, rather than on arbitrary neural variability.

The authors position SSLD as a complementary architectural motif for the emerging field of neuroscience foundation models. While recent foundation models have focused on scaling across sessions and subjects, they have largely treated dynamics as stationary. SSLD demonstrates that incorporating switching dynamics in a shared latent space provides a principled bridge between neural activity and behavior, offering a path toward more interpretable and scalable models that can generalize across diverse neural datasets, brain regions, and behavioral contexts. The work suggests that jointly modeling shared structure and switching dynamics is essential for linking neural activity to behavior, though the authors note limitations regarding the reliance on well-defined behavioral priors and the lack of explicit modeling for external inputs.

Technical Summary: Learning Interpretable Switching Dynamics in Shared Neural-Behavioral Latent Space

More like this