💬 NLP

M $^3$ Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

This paper introduces M $^3$ Eval, the first cognitively-grounded benchmark designed to systematically evaluate memory capabilities in multi-modal models through specialized video tasks, revealing critical weaknesses such as poor disentanglement in parallel streams and limited symbolic memory that highlight the need for improved memory mechanisms.

Original authors: Jie Huang, Ruixun Liu, Sirui Sun, Xinyi Yang, Yin Li, Yixin Zhu, Yiwu Zhong

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jie Huang, Ruixun Liu, Sirui Sun, Xinyi Yang, Yin Li, Yixin Zhu, Yiwu Zhong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a new student who is incredibly smart at looking at pictures and reading text. They can describe a single movie scene perfectly. But now, you want to see if they can actually remember what happened in a whole movie, especially when things get complicated, like when two movies are playing at once or when similar scenes are mixed together.

The paper "M 3Eval" is like a new, very specific report card designed just for this. The researchers realized that while we have tests for how well AI "sees" and "thinks," we don't have good tests for how well it "remembers." So, they built a gym for AI memory, based on how human psychologists test our own brains.

Here is how they tested the AI, using four simple games:

1. The "Split-Screen" Test (Divided Attention)

The Setup: Imagine watching two different cooking shows at the same time, side-by-side on your TV. One is on the left, one is on the right. Every few seconds, the TV swaps them so the left one moves to the right and vice versa.
The Question: "In the video that was originally on the left, did the chef add onions or tomatoes?"
The Result: The AI got very confused. It couldn't keep the two stories separate. It was like a person trying to listen to two different conversations at a loud party; the AI started mixing up who said what and where. When the videos swapped places, the AI lost track of which story belonged to which screen.

2. The "Interference" Test (Memory Interference)

The Setup: Imagine you watch a video of a man baking a cake (Video A). Immediately after, you watch a very similar video of a man baking a pie (Video B).
The Question: "What did the man in the first video (the cake) put on top?"
The Result: The AI struggled to keep the two memories apart. It often answered with details from the pie video instead of the cake video.
The Twist: The researchers found something surprising. If they showed the cake video twice before showing the pie, the AI remembered the cake much better. It's like if you hear a song twice, you're less likely to forget it when a similar song plays right after.

3. The "Scrambled Story" Test (Interleaved Events)

The Setup: Imagine taking two different movies, cutting them into 10 tiny pieces each, and then shuffling them together like a deck of cards (Piece 1 of Movie A, Piece 1 of Movie B, Piece 2 of Movie A, Piece 2 of Movie B...).
The Question: "Can you tell me the correct order of events for just Movie A?"
The Result: The AI was terrible at this. It couldn't untangle the two stories. It was like trying to reassemble two different jigsaw puzzles that have been mixed into one big pile. The AI also had a hard time knowing when something happened (temporal order) compared to where it happened (spatial location). It was better at remembering "The pot was on the left" than "The pot was added before the onions."

4. The "N-Back" Test (Symbolic Memory)

The Setup: This is a classic brain game. You watch a long stream of short video clips. The rule is: "If the current clip looks like the one you saw 3 clips ago, say 'Yes'."
The Result: Humans get worse at this as the gap gets bigger (it's hard to remember 10 steps back). But the AI was weird: it didn't get worse as the gap got bigger, but it got terrible as the total number of clips got longer.
The Metaphor: Imagine a human has a small notepad and can only write down the last few things they saw. If you show them too many things, they forget the old ones. The AI, however, has a giant library where it keeps everything, but it doesn't know how to ignore the junk. It gets overwhelmed by the sheer volume of information, not by the distance in time.

The Big Takeaway

The paper concludes that while AI is getting better at understanding video, its "memory" is still very different from a human's.

Humans are good at filtering out noise and remembering the order of events, even if we forget details over time.
AI struggles to keep parallel stories separate, gets confused by similar-looking videos, and has a hard time organizing a long list of events in the right order.

The researchers hope that by using these specific "games," other scientists can build better AI systems that don't just "see" the world, but actually remember it the way we do. They have made their test questions and videos available for everyone to use.

Technical Summary: M 3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

Problem Statement

As multi-modal models (MLMs) advance toward long-form video understanding, driven by expanding context windows, memory has emerged as a critical capability. However, existing benchmarks and datasets for video understanding primarily focus on visual perception and reasoning. While some tasks implicitly involve memory (e.g., long-video understanding or video reasoning), they fail to systematically isolate memory mechanisms. Consequently, current evaluations provide only partial and indirect assessments, lacking the ability to disentangle specific memory dimensions such as capacity (how much information is retained), fidelity (how accurately information is preserved), and robustness (how well representations withstand interference). There is currently no dedicated evaluation protocol to systematically probe what models retain, how faithfully information is preserved, and how robust memory remains under interference.

Methodology

The authors introduce M 3Eval, the first comprehensive evaluation framework and benchmark designed to probe different memory dimensions in multi-modal models. The methodology is grounded in cognitive psychology, adapting controlled experimental paradigms to the video domain. The framework consists of four distinct evaluation paradigms, each instantiated through carefully constructed video tasks and targeted questions:

Divided Attention (Encoding under Concurrent Load):
- Setup: Two semantically similar videos are displayed side-by-side (split-screen), either without swapping or with frequent left/right swaps.
- Goal: Evaluate the model's ability to maintain independent representations for parallel streams.
- Metrics: Source Identification (distinguishing content origin), Order Understanding (temporal/logical sequence), and Content Retention (plot details).
- Failure Modes: Source confusion, order confusion, and content forgetting.
Memory Interference (Robustness to Distraction):
- Setup: Two semantically similar videos are concatenated sequentially. Questions target one specific video while the other acts as interference.
- Goal: Isolate proactive interference (earlier content disrupting later recall) and retroactive interference (later content disrupting earlier recall).
- Metrics: Accuracy and Intrusion Rate (IR), measuring the proportion of responses selecting options from the competing video.
Interleaved Events (Temporal Organization):
- Setup: Segments from two source videos are interleaved into a single stream (e.g., A1–B1–A2–B2...).
- Goal: Assess the ability to disentangle interleaved segments and recover the internal temporal order of a specific source.
- Metrics: Source Identification, Order Understanding, Content Retention, and False Memory Discrimination (identifying queries that do not belong to either video).
N-Back (Symbol Grounding and Memory Capacity):
- Setup: A sequence of short video clips is presented. The model must determine if the final clip matches the clip $N$ positions earlier based on abstract attributes (Scene or Action).
- Goal: Probe symbolic memory and the ability to filter irrelevant information over temporal gaps.
- Metrics: Accuracy on scene and action matching across varying lag distances ( $N$ ) and sequence lengths ( $K$ ).

Dataset Construction:
The benchmark comprises 2,403 questions over 451 videos (approx. 403 hours) sourced from five public datasets (HourVideo, Video-MME, LVBench, InfiniBench, CrossVid). Video pairs are selected based on semantic similarity to maximize interference. Questions are generated using LLMs (Qwen3.5-27B) and refined through manual review to ensure logical consistency and eliminate ambiguity.

Key Contributions

M 3Eval Benchmark: The first benchmark to systematically evaluate multi-modal memory across four distinct dimensions (Divided Attention, Interference, Interleaved Events, N-Back) using video tasks.
Cognitively-Grounded Design: A novel evaluation design that isolates specific memory mechanisms by adapting established psychological paradigms (e.g., dual-task, N-Back, interference theory) to video understanding.
Systematic Evaluation: A comprehensive assessment across diverse models, including proprietary models (Gemini-3.1-Pro-Preview, GPT-5.4), open-weight models (Qwen3-VL, InternVL3.5), and agentic methods (VideoLucy, M3-Agent), compared against human performance.

Experimental Results

The evaluation reveals consistent weaknesses and distinctive behaviors in current multi-modal models compared to humans:

Parallel Processing Failure: Models struggle to maintain disentangled representations when processing parallel video streams. Performance drops significantly on Source Identification in split-screen settings, suggesting attention confusion across concurrent visual inputs.
Interference Patterns: Unlike humans, who exhibit significantly stronger retroactive interference than proactive interference, models show comparable levels of both. Furthermore, models struggle to resist interference from semantically similar content, with high intrusion rates. Interestingly, repeating target or interfering videos surprisingly improves model accuracy, likely due to strengthened representations via causal attention.
Temporal vs. Spatial Grounding: Models are less capable than humans at organizing temporally interleaved information. Analysis shows that spatial source grounding (split-screen) is consistently stronger than temporal source grounding (interleaved events), where many models fall below random baselines.
Symbolic Memory and Filtering: Models lag far behind humans in symbolic memory (N-Back tasks). While human accuracy declines monotonically with increasing temporal gap ( $N$ ) due to capacity limits, model accuracy often remains flat or improves (due to global attention). However, models degrade sharply as the total number of symbols ( $K$ ) increases, revealing a fundamental inability to filter out irrelevant information from memory.
Hallucination: In False Memory Discrimination tasks, most models perform below the random baseline, indicating severe hallucination when presented with queries unrelated to the video content.

Significance

The paper positions M 3Eval as a valuable diagnostic resource for future research. The findings highlight memory as a fundamental yet underexplored capability in multi-modal models. The authors argue that current benchmarks conflate memory with perception and reasoning, obscuring specific failure modes. By isolating these mechanisms, M 3Eval offers insights for designing more effective memory mechanisms, such as refining attention for parallel streams, leveraging repetition strategies, strengthening temporal source grounding, and improving symbolic memory to support the filtering of task-irrelevant data. The work aims to motivate the development of multi-modal systems equipped with robust, structured, and human-aligned memory capabilities.