🧬 biology

Feature leakage and the identifiability of direct-dependency entropy models of neural activity

This paper demonstrates that successful prediction by direct-input entropy models of neural activity often reflects feature leakage from omitted interactions or hidden states rather than true mechanistic simplicity, and proposes diagnostic reweighting methods to distinguish between in-distribution predictive performance and the actual recovery of underlying neural response rules.

Original authors: Houman Safaai, Bernardo L. Sabatini

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Houman Safaai, Bernardo L. Sabatini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to figure out how a complex machine works by watching its inputs and outputs. In the brain, neurons are these machines. They receive thousands of signals (inputs) from other neurons and decide whether to fire an electrical spike (output).

For a long time, scientists have used a simple rule to guess how neurons work: "The neuron is like a scale." If you put enough weight (signals) on the scale, it tips over and fires. In this simple view, every input adds a little bit of weight independently. If Input A adds 1 unit and Input B adds 2 units, the total is just 3. This is called a "direct" or "additive" model.

However, real neurons are messy. They have tiny branches where signals can mix, cancel each other out, or amplify each other in complex ways (like a recipe where mixing flour and sugar creates a new taste, rather than just tasting flour plus sugar).

This paper asks a crucial question: If a simple "scale" model predicts a neuron's behavior perfectly, does that mean the neuron is actually simple? Or is it just a trick of the data?

The authors say: It's often a trick.

Here is the breakdown of their findings using everyday analogies:

1. The "Crowded Room" Illusion (Feature Leakage)

Imagine you are trying to guess who is leaving a party based on who is arriving.

The Simple Model: You assume people leave independently. If Alice leaves, she leaves. If Bob leaves, he leaves.
The Reality: Alice and Bob are best friends. They always leave together.

Now, imagine you only watch the party during a specific hour when Alice and Bob happen to be the only ones arriving. Because they always arrive together, your data looks like a single unit. If you try to predict who leaves using a simple model, it will look perfect! You might conclude, "See? They act independently!"

But you are wrong. They are deeply connected. The "connection" (the friendship) got leaked into your simple model because the data you collected (the specific hour) was biased. The complex interaction was hidden inside the simple numbers because the inputs were correlated.

The paper calls this "feature leakage." When inputs are correlated (like friends arriving together), a complex, non-linear interaction can look exactly like a simple, linear sum.

2. The "Coskewness" Shortcut

The authors found a specific mathematical reason why this happens in the brain, especially when neurons are "sparse" (they are quiet most of the time and only fire rarely).

Think of it like a shadow.

If you shine a light on a 3D object (a complex interaction) from a specific angle (the specific way the brain is firing right now), the shadow on the wall might look like a perfect circle (a simple linear rule).
But if you move the light to a different angle (change the pattern of inputs), the shadow changes, and you realize it was actually a complex shape all along.

The paper shows that in the brain, the "light" (the natural firing patterns) often hits the "object" (the neuron's complex rules) in a way that casts a perfect "circle" shadow. This makes the neuron look simple, even if it's not.

3. The New Test: "Reweighting"

So, how do we stop being fooled? The authors propose a new diagnostic tool called "State Reweighting."

Imagine you have a photo of a crowd.

The Old Way: You count the people exactly as they appear in the photo. If the photo is taken at a concert where everyone is wearing red, you conclude, "Everyone in this crowd loves red."
The New Way (Reweighting): You take that same photo, but you digitally adjust the weights. You say, "Okay, let's pretend this crowd is a random mix of people, not just concert-goers." You force the data to look at the rules of the crowd, not just the specific snapshot.

The authors applied this to real brain recordings from the hippocampus (the memory center).

Result: When they looked at the data the "old way" (using the natural, biased patterns), about 75% of the neurons looked like they followed simple, linear rules.
Result: When they applied the "rewighting" test (checking if the rules hold up under different conditions), that number dropped to about 45%.

The Takeaway: Roughly half of the neurons that looked simple were actually hiding complex interactions. The simple model was just "cheating" by relying on the specific way the data was collected.

4. The "Raw Coactivity" Trap

Scientists also check if a simple model can predict "raw coactivities" (how often inputs and outputs happen together). The paper shows that even if a simple model predicts these perfectly, it doesn't prove the neuron is simple.

It's like predicting the weather. If you notice that "umbrellas" and "rain" always happen together, a simple model predicts rain perfectly. But if you only look at data from a rainy city, you might miss the fact that in a desert, umbrellas and rain don't go together. The prediction works only because of the specific environment you are in, not because the underlying rule is simple.

Summary

The paper doesn't say simple models are useless. They are great for prediction (guessing what will happen next).

However, the paper warns us against using them for discovery (figuring out how the brain actually works).

Prediction: "If I see inputs A and B, the neuron will fire." (Simple models are good at this).
Mechanism: "The neuron adds inputs A and B together." (Simple models might be lying here).

The Bottom Line: Just because a neuron looks like a simple calculator under current conditions doesn't mean it is a simple calculator. It might be a complex, non-linear machine that is just hiding in plain sight because of how we are looking at it. To find the truth, we have to change the "light" (the data distribution) and see if the shadow changes.

Technical Summary: Feature Leakage and the Identifiability of Direct-Dependency Entropy Models of Neural Activity

Problem Statement
Biological neurons integrate thousands of synaptic inputs through complex, nonlinear dendritic mechanisms. However, population activity is frequently modeled using "direct" or additive input-output rules (e.g., logistic units) where each input contributes independently to a scalar drive. A critical gap exists between the predictive success of these direct models and the mechanistic interpretation of that success. Previous work has shown that direct maximum-entropy (MaxEnt) models can predict a large fraction of higher-order and time-delayed coactivities in neural datasets. The central question addressed here is whether this predictive accuracy implies that the underlying neural mechanism is truly additive, or if the success is an artifact of the specific distribution of sampled input states.

The paper argues that in sparse, binarized neural recordings, omitted higher-order interactions, temporal lags, or hidden states can be "absorbed" into fitted first-order parameters if they are correlated with the included statistics under the empirical input distribution. This phenomenon, termed feature leakage, means that a model with high goodness-of-fit may be mechanistically incorrect. Specifically, for sparse binary inputs, a "coskewness" channel allows interaction terms to project onto first-order terms, creating an illusion of additivity.

Methodology
The authors develop an identifiability framework for conditional MaxEnt models, utilizing information geometry and specific diagnostics to separate in-distribution prediction from the recovery of the true response rule ( $P(y|x)$ ).

Information Projection Analysis: The authors formalize how omitted sufficient statistics ( $G$ ) shift fitted first-order parameters ( $\hat{\theta}$ ) in a restricted MaxEnt model. They derive that the shift depends on the Fisher cross-covariance between included ( $F$ ) and omitted ( $G$ ) statistics under the sampled input distribution ( $P_{emp}$ ). If this covariance is non-zero, omitted effects leak into the fitted parameters.
Coskewness Identity: For sparse binary inputs, the authors derive a closed-form identity showing that the projection of a centered pairwise product ( $\delta_i \delta_j$ ) onto a first-order feature ( $\delta_i$ ) is proportional to the input covariance and the sparsity factor $(1-2p_i)$ . In sparse regimes ( $p_i \ll 0.5$ ), this factor is near 1, creating a strong channel for higher-order features to masquerade as first-order ones.
Diagnostics: To distinguish stable first-order mechanisms from distribution-dependent artifacts, the paper introduces three controls:
- State Reweighting: Holding the estimated response table $P(y|x)$ fixed while changing the input state weights $P(x)$ from empirical ( $q_{emp}$ ) to product-marginal ( $q_{prod}$ ) or balanced ( $q_{bal}$ ). If first-order attribution drops significantly under reweighting, the initial success was distribution-sensitive.
- Conditional Log-Odds Contrasts: Testing the additivity of the local response table directly via $\Delta_{ij}$ (the interaction term in the log-odds), which is independent of $P(x)$ .
- Temporal Leakage Simulation: Quantifying how instantaneous statistics can absorb lagged mechanisms when inputs are autocorrelated.

Key Results

Ground-Truth Simulations: In simulations with purely higher-order generators (zero linear drive), the apparent fraction of recoverable information captured by a direct model ( $\phi_{dir}$ ) varied drastically based on input representation and correlation. For sparse, correlated binary inputs, $\phi_{dir}$ reached ~0.84, despite the ground truth being purely interaction-based. Conversely, under balanced sampling or symmetric representations, $\phi_{dir}$ dropped to ~0. The state-reweighting control successfully separated these cases: high-order responses lost their first-order attribution under balanced weights, while genuine additive responses remained stable.
CA1 Hippocampal Recordings: Applied to local three-input "cubes" from mouse CA1 recordings (selected for high coactivity and well-sampled states), the diagnostics revealed that approximately half of the tables appearing first-order under empirical weights became distribution-sensitive under balanced reweighting.
- Under empirical weights, the median first-order attribution was 0.754.
- Under product-marginal control, it dropped to 0.679.
- Under balanced stress-test weights, it dropped to 0.459.
- A matched additive-surrogate null confirmed that this drop was not a finite-sample artifact; genuinely additive tables remained stable, while the observed "sensitive" fraction (48%) far exceeded the null expectation.
- Crucially, even tables that remained "successful" (retained) under reweighting often showed significant conditional log-odds contrasts ( $\Delta_{ij}$ ), indicating they were not truly additive despite their predictive power.
Visual Cortex Recordings: Analysis of mouse visual cortex data showed a smaller but measurable distribution-sensitive component. Most local successes were more robust than in CA1, but the effect persisted, suggesting it is not unique to the hippocampus.
Raw Coactivity Prediction: The paper demonstrates that high $R^2$ values in predicting raw higher-order coactivities (e.g., $\langle y x_i x_j \rangle$ ) do not confirm conditional additivity. These metrics are averages over $P_{emp}(x)$ and can be high even when the conditional response is purely interactive, provided the sampling geometry projects those interactions onto first-order features.

Significance and Claims
The paper claims that direct entropy-explained fractions and raw coactivity predictions are best interpreted as statements about prediction under the observed state distribution ( $P_{emp}$ ), not as evidence that mechanisms outside the direct model are absent or small.

The authors do not claim that direct models are useless; rather, they argue that their successful prediction does not identify the absence of higher-order, temporal, or latent mechanisms. The framework provides quantitative tools to state the scope of a direct-dependency MaxEnt model's predictive contribution. The key takeaway is that without controls like state reweighting and conditional log-odds contrasts, one cannot distinguish whether a biological neuron nonlinearly integrates synaptic information or is effectively an additive logistic unit under the specific sampling conditions of the experiment. The observed "first-order" behavior in many datasets may largely reflect the geometry of the sampled inputs rather than the intrinsic biophysics of the neuron.