🧬 biology

A global predicted-fMRI drive signal from TRIBE does not predict YouTube replay heatmaps

This study demonstrates that a state-of-the-art deep multimodal brain-encoding model (TRIBE), despite accurately predicting fMRI responses to naturalistic video, fails to forecast viewer re-watch behavior as measured by YouTube replay heatmaps, showing no significant correlation beyond simple acoustic and motion baselines.

Original authors: Barada Sahu, Shivesh Pandey

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Barada Sahu, Shivesh Pandey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you have a super-smart robot that can look at a video and guess exactly how a human brain would react to it. This robot, called TRIBE, is a "brain-encoding model." It's so good at its job that it recently won a major competition to predict brain scans (fMRI) just by watching videos.

The big question the researchers asked was: "If this robot can predict what the brain is doing, can it also predict what people will actually do?"

Specifically, they wanted to know if the robot's predictions could tell them which parts of a YouTube video people would rewatch.

The Experiment: The "Rewatch" Test

To test this, the team took 48 different YouTube videos (everything from music and comedy to science and tech). They ran these videos through the TRIBE robot to generate a "brain reaction curve"—a line graph showing how active the brain is predicted to be every second.

Then, they compared this robot-generated curve against real human behavior: YouTube's "Most Replayed" heatmaps. These heatmaps are like a map showing exactly where millions of viewers hit the "rewind" button. If the robot was a true "mind reader," its curve should match the heatmap perfectly.

The Result: The Robot Got It Wrong

The answer was a resounding no.

The researchers found that the robot's predictions had almost zero connection to where people actually rewound the video.

The Score: The connection was so weak it was statistically indistinguishable from zero.
The Baseline: The robot didn't even do better than a very simple computer program that just measured how loud the video was or how much the pixels were moving. In fact, the robot's performance was basically the same as just guessing based on volume.

Why Did the Music Videos Seem Different?

In a small pilot test with just music videos, the robot did seem to predict re-watches. However, the researchers realized this was a trick. Music videos often have a loud intro or a specific beat drop that people naturally rewind to. The robot was just picking up on the fact that "intros are loud," not on the actual content of the song. Once they tested other genres like comedy, science, and talk shows, this "magic" disappeared completely.

The "Whole Brain" vs. "Specific Parts" Check

The researchers wondered if they were looking at the wrong part of the brain. Maybe the robot was right about the "reward center" or the "visual center," but they were looking at the whole brain at once?

They tried looking at specific brain networks (like the visual system or the reward system).
Result: Still nothing. No specific part of the predicted brain activity could predict re-watching behavior.

The Takeaway: A "Brain Map" Isn't a "Behavior Map"

Think of it like this:

The Robot (TRIBE) is like a highly accurate weather forecast. It can tell you exactly how much rain will fall in a city (predicting the brain scan).
The Behavior (Re-watching) is like asking, "Will people carry umbrellas?"

The study shows that just because you know exactly how much rain is falling (the brain scan), it doesn't mean you can predict if people will carry umbrellas (re-watch the video). The robot is great at simulating the biology of the brain, but it fails to capture the human choices that drive people to hit the rewind button.

In short: A model that is perfect at predicting what a brain looks like when watching a video is not necessarily good at predicting what a human will do when watching that same video. The "brain signal" and the "behavior" are two different things.

Technical Summary: Global Predicted-fMRI Drive Signals and YouTube Replay Heatmaps

Problem Statement
Deep multimodal brain-encoding models, such as TRIBE (the winner of the 2025 Algonauts challenge), have achieved high accuracy in predicting fMRI responses to naturalistic video stimuli. A critical unanswered question is whether these predicted neural signals, which are computationally inexpensive and require no scanner, inherit the "neuroforecasting" capabilities of measured neural signals. Previous literature suggests that measured neural activity can predict aggregate behavioral engagement (e.g., cultural popularity, crowdfunding success) beyond self-reports. However, it is unknown if the predicted signals from group-trained encoders preserve the specific individual variability and reward-region dynamics necessary for such behavioral prediction, or if they regress toward the group mean, discarding the very structure required to forecast re-watching behavior.

Methodology
The study evaluates whether TRIBE's predicted cortical response can forecast YouTube's "most replayed" heatmaps, a passive proxy for viewer engagement and re-watching behavior.

Model & Input: The authors utilized the unmodified TRIBE model (a 1B-parameter trimodal encoder fusing Llama-3.2, V-JEPA2, and Wav2Vec-BERT features) to generate per-TR (1-second) predicted cortical responses on the fsaverage5 surface for 48 YouTube videos across 11 categories.
Engagement Readout: The high-dimensional predicted response ( $P \in \mathbb{R}^{T \times V}$ ) was reduced to a scalar per-second engagement curve using Global Field Power (GFP), defined as the root-mean-square over all cortical vertices.
Behavioral Target: The target variable was the "most replayed" heatmap provided by YouTube, representing normalized re-watch intensity. This was linearly interpolated to match the model's temporal grid.
Statistical Controls: To isolate content-specific prediction from temporal confounds (e.g., the tendency to re-watch intros or the general decay of engagement over time), the primary metric was a position-controlled partial correlation. Both the predicted engagement curve and the replay heatmap were regressed against a quadratic time basis ( $[1, t, t^2]$ ), and the correlation was computed on the residuals.
Baselines: The model's performance was compared against low-level feature baselines: audio loudness (RMS energy) and visual motion (mean absolute frame difference).
Data Acquisition: Due to YouTube's SABR (server-side adaptive bitrate) streaming, which blocks standard download tools, the authors developed a custom acquisition pipeline using the NewPipe Android client driven via ADB to retrieve video content, while fetching replay heatmaps as metadata.
Robustness Checks: The analysis included region-specific readouts (restricted to five functional networks: visual, auditory, salience, frontal, parietal) and a non-parametric circular-shift permutation test to account for temporal autocorrelation.

Key Results
The study found no evidence that the global predicted-fMRI drive signal predicts re-watch behavior:

Primary Correlation: The pooled position-controlled partial correlation between TRIBE's predicted engagement and the replay heatmap was +0.058 (95% CI [−0.04, 0.15]). This is statistically indistinguishable from zero ( $t(47) = 1.21, p = 0.23$ ).
Baseline Comparison: TRIBE's predictive power did not significantly exceed simple low-level baselines. The difference between TRIBE and the loudness baseline was negligible (paired $p = 0.74$ ).
Raw Correlation: The raw (uncontrolled) correlation was also near zero (+0.036). Moderate raw correlations observed in music videos were identified as a genre-specific artifact related to intro/onset replay rather than genuine content prediction, as they vanished in other genres (talk, tech, comedy, etc.).
Region-Specific Analysis: Restricting the readout to specific cortical networks (including salience/reward regions) did not recover a significant signal. All network-specific partial correlations remained near zero and inconsistent.
Video-Level Ranking: TRIBE summaries (mean/peak engagement) showed no ability to rank videos by overall popularity (views/likes), with correlations near zero or slightly negative.

Contributions and Significance
The paper makes three primary contributions:

Empirical Null Result: It establishes that, for the specific target of YouTube replay heatmaps and the readout of global field power, state-of-the-art predicted neural signals do not forecast behavioral engagement. This challenges the assumption that accurate fMRI encoders automatically inherit neuroforecasting capabilities.
Methodological Rigor: The study rigorously separates genuine null results from confounds by implementing position controls, comparing against low-level baselines, and utilizing permutation tests that preserve temporal autocorrelation.
Open Infrastructure: The authors release the code, the video-ID manifest, and a novel acquisition pipeline capable of bypassing YouTube's SABR streaming restrictions, enabling reproducibility and future testing with different models or behavioral targets.

Claims and Limitations
The authors frame their claims modestly. They assert that for this specific target (replay heatmaps), this specific readout (global GFP), and this specific model (TRIBE optimized for fMRI accuracy), the predicted signal fails to forecast re-watch behavior beyond temporal position and low-level features.

The paper explicitly notes that the null result may be bounded by:

The noise and bias inherent in "most replayed" heatmaps (e.g., intro/onset effects).
The limited analysis window (60 seconds).
The fact that TRIBE was optimized for fMRI reconstruction accuracy rather than behavioral endpoints.

The authors conclude that while a richer readout or a cleaner behavioral signal might alter the result, the current evidence suggests that off-the-shelf brain-encoding models do not yet serve as reliable engagement predictors for naturalistic video content.