📄 other

FalseResMem: A Neural Network to Predict False Memories in Visual Recognition Tasks

The paper introduces FalseResMem, a novel neural network that combines ResNet50 and AlexNet architectures to accurately predict image-level false alarm rates in visual recognition tasks, demonstrating robust generalization across diverse image categories.

Original authors: Anastasiia Mikhailova, Wilma A. Bainbridge

Published 2026-07-25

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Anastasiia Mikhailova, Wilma A. Bainbridge

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain is a super-advanced security guard for your memories. Its job is to decide if a new picture you see is a "stranger" or an "old friend." Usually, this works great. But sometimes, the guard gets tricked. You see a brand-new photo of a zebra, and your brain screams, "I know this guy! We've met before!" even though you've never seen it in your life. This is called a false alarm or a false memory. It's the feeling of déjà vu—that spooky sense that a new experience is actually familiar. Scientists have long known that some people are more prone to these mix-ups than others, but they also noticed something weird: certain pictures seem to trick everyone. A specific photo of a waterfall might make a whole crowd of people feel like they've seen it before, while a photo of a weirdly dressed cat might not trick anyone at all. For a long time, researchers thought these mistakes were just random noise or a result of how a specific person's brain was wired. But what if the mistake isn't in the person, but in the picture itself? What if some images are just "deceptively familiar"?

This is exactly the mystery a team of researchers at the University of Chicago set out to solve. They built a new kind of computer brain, a neural network they named FalseResMem, designed with one specific superpower: to look at a picture and predict how likely it is to trick human memory. Instead of trying to guess what you will remember, this model guesses what you will mistakenly think you remember. They trained it on a massive library of 10,000 images, teaching it to spot the hidden visual tricks that make a photo feel familiar when it's actually brand new. The results are fascinating: the computer learned that simple, nature-like scenes (like waterfalls) or high-contrast patterns (like zebra stripes) are the ultimate memory traps. The model is so good that it can predict these "false alarms" almost as well as a group of humans can agree with each other. It's like giving scientists a crystal ball that tells them, "Hey, if you show people this specific image, they are going to get confused," which could help us understand why our memories sometimes lie to us and how to stop them.

The Story of the "Deceptive" Picture

Think of your memory like a giant filing cabinet. When you see a new photo, your brain pulls out a file to see if it matches anything already in the cabinet. If it doesn't match, you say, "New!" If it does, you say, "Old!" But sometimes, the brain gets lazy or confused. It sees a picture that looks sort of like something in the cabinet and says, "Old!" even though it's a total stranger. This is a false alarm.

For years, scientists thought these mistakes were mostly about the person looking at the photo. Maybe they were tired, or maybe they just had a "yes, I know this" attitude. But recent discoveries showed that some photos are just bad actors. They are consistently tricking people, no matter who is looking at them. This is the "Mandela Effect" in action—where huge groups of people share the same wrong memory about a famous logo or event. The researchers behind this paper wanted to know: Can we build a computer that looks at a photo and tells us, "This one is a master of disguise"?

Enter FalseResMem: The Memory Detective

The team created FalseResMem, a special computer program that acts like a detective for memory errors. Here's how it works in plain English:

The Training: They fed the computer a huge dataset called MemCat, which contains 10,000 images of animals, food, landscapes, sports, and vehicles. They didn't just show the pictures; they showed them to nearly 100 real people and recorded how often each person made a mistake. If 50 out of 100 people thought a photo of a bear was familiar when it wasn't, that photo got a "False Alarm Rate" (FAR) score of 0.5.
The Brain Power: The computer uses two different "eyes" to look at the photos. One eye is a pre-trained expert (ResNet50) that knows a lot about general shapes and objects. The other eye is a custom-built detective (an AlexNet-like structure) that was specifically taught to look for the tricks that cause confusion.
The Prediction: The computer combines what it sees to give every single image a score between 0 and 1. A score of 0 means "No one will be tricked by this." A score of 1 means "Everyone will be tricked."

What Did They Find?

The results were surprisingly clear. The computer learned to predict these false memories with a high degree of accuracy. When they tested the model, it matched human agreement levels with a correlation of 0.60. This means the computer is almost as good as a group of humans agreeing on which photos are tricky.

But the most fun part is what the computer decided was tricky:

The Masters of Disguise: The images that got the highest "trick scores" were often simple, nature-based scenes. Think of waterfalls or zebras. These images have repetitive patterns and high contrast (like black and white stripes) that seem to make our brains go, "I've seen this pattern before!" even if we haven't. They are "deceptively familiar."
The Safe Bets: The images that were least likely to trick people were complex scenes with weird or surprising content, like an animal doing something totally unexpected. Because these scenes are so unique and strange, our brains say, "Okay, I definitely haven't seen this before," and we don't get fooled.
The Food Problem: Interestingly, the model struggled a bit with pictures of food. While it was great at predicting tricks for landscapes and animals, it wasn't as good with food. This suggests that our brains might use different rules for remembering a burger versus remembering a mountain.

Why Does This Matter?

This isn't just a cool trick for a video game. The researchers showed that FalseResMem can look at pictures it has never seen before—like faces, art, or symbols—and still guess how likely they are to cause a false memory.

For Eyewitnesses: If you are a police officer trying to identify a suspect, you don't want to show a witness a photo that is "deceptively familiar" by nature, because they might think they've seen the suspect before even if they haven't. This tool could help design better lineups that are less likely to cause wrongful convictions.
For Scientists: It gives researchers a way to test their theories. Instead of guessing which photos will confuse people, they can use this model to pick the perfect "tricky" photos for experiments.
For Understanding Our Brains: It proves that false memories aren't just random mistakes. They are driven by specific features in the images themselves. Some pictures are just wired to make us feel like we know them.

The paper concludes that we can now treat these memory errors not as random noise, but as a predictable signal. By understanding the "deceptive" features of images, we can start to understand why our brains sometimes play tricks on us, and maybe even learn how to stop them. The computer didn't just learn to see; it learned to see why we get confused.

Technical Summary: FalseResMem – A Neural Network to Predict False Memories in Visual Recognition Tasks

Problem Statement

Visual memory research has traditionally relied on Signal Detection Theory (SDT) to characterize recognition performance, separating memory strength (hits) from decision criteria (false alarms). In this framework, the False Alarm Rate (FAR)—the incorrect classification of novel images as familiar—is often treated as a global, participant-level error or noise arising from a liberal response criterion or low sensitivity. However, recent evidence suggests that false memories are not merely random errors but are consistent, image-specific phenomena. Certain images consistently elicit higher false alarms across different observers (e.g., the "Mandela Effect"), independent of the observer's general memory traits.

Current deep learning models, such as ResMem, successfully predict image memorability (Hit Rate) by extracting intrinsic visual features. However, no existing model explicitly targets the prediction of image-level false alarms. Treating FAR as mere noise obscures the systematic visual properties that drive false recognition. The challenge is to develop a model that can predict the likelihood of a novel image being falsely recognized as familiar based solely on its visual features, moving beyond participant-level summaries to image-specific error modeling.

Methodology

Dataset

The model was trained exclusively on the MemCat dataset (Goetschalckx & Wagemans, 2019), a large-scale collection of 10,000 natural images across five semantic categories: animals, food, landscapes, sports, and vehicles.

Ground Truth: FAR scores were derived from a continuous recognition task involving an average of 99 participants per image. A false alarm was recorded when a participant incorrectly identified a novel image as a repeat.
Rationale: MemCat was selected over the LaMem dataset due to its superior inter-observer FAR consistency ( $\rho = 0.78$ vs. $0.67$) and higher data quality (e.g., absence of duplicates). Attempts to fine-tune on LaMem did not improve performance.

Model Architecture: FalseResMem

FalseResMem is a hybrid deep learning architecture designed to predict a single FAR score (0–1) for any input image. It adapts the architecture of ResMem but is trained specifically for error prediction. The model consists of two parallel feature extraction streams:

ResNet-50 Stream: Utilizes layers 0–8 (conv1 through adaptive average pooling) of a ResNet-50 pretrained on ImageNet-1K. This branch extracts 2,048-dimensional global features. Extensive testing showed that ResNet-50 with ImageNet pretraining outperformed other variants (ResNet-18, 34, 101, 152) and training from scratch.
AlexNet-Inspired Stream: A five-layer convolutional stack trained from scratch (initialized with random weights). This stream processes images through a sequence of convolutional layers (11x11, 5x5, 3x3 kernels), ReLU activations, max-pooling, and Local Response Normalization (LRN), outputting a 9,216-dimensional feature vector. This branch is hypothesized to capture specific false-alarm-related features distinct from general memorability.

Fusion and Head:
The 2,048-dimensional ResNet features and the 9,216-dimensional AlexNet features are concatenated into an 11,264-dimensional vector. This is fed into a seven-layer fully connected head:

Layers 1–6: Dense layers with dimensions 4096, 4096, 2048, 1024, 512, and 256, each followed by ReLU activation and dropout ( $p=0.5$ ) after layers 1, 3, and 4.
Layer 7: A final dense layer (256 $\to$ 1) with a sigmoid activation to output the predicted FAR probability.

Training and Evaluation

Optimization: The model was trained using Stochastic Gradient Descent (SGD) with momentum (0.9) and Mean Squared Error (MSE) loss.
Hyperparameter Tuning: 30 trials were conducted using Optuna to optimize learning rate ( $10^{-5}$ to $10^{-2}$ ) and batch size (8–128). The best configuration used a learning rate of 0.009381 and batch size of 32.
Validation: Performance was evaluated using 10-fold cross-validation. The ceiling for model performance was established by calculating the split-half inter-observer consistency of human FAR scores ( $\rho = 0.60 \pm 0.29$ ).
Interpretability: Feature importance was analyzed using Ablation-CAM (to handle gradient vanishing issues in deeper layers) and occlusion sensitivity (sliding 32x32 occluders) to identify spatial regions critical for high FAR predictions.

Key Results

Predictive Performance

FalseResMem achieved robust performance on the MemCat test set:

Cross-Validation: Mean Spearman's rank correlation ( $\rho$ ) of 0.48 $\pm$ 0.13.
Best Case: The top-performing model achieved a test correlation of $\rho = 0.60$ (MSE = 0.002), effectively reaching the ceiling of human inter-observer consistency.
Generalization: The model successfully generalized to nine independent datasets, including scenes (FIGRIM, LaMem), faces, conventional symbols, and art, with statistically significant correlations ( $\rho$ ranging from 0.06 to 0.28).

Category-Specific Performance

Performance varied across semantic categories:

Highest Accuracy: Animals ( $\rho = 0.673$ ) and Landscapes ( $\rho = 0.605$ ) showed the strongest alignment with human data.
Lowest Accuracy: Food images showed the largest deviation from ground truth ( $\rho = 0.477$ ), a finding consistent with the model's failure to generalize to the FoodFolio dataset.
Generalization Limits: While the model generalized to faces and conventional symbols, it failed to predict FAR for "engineered" symbols (designed to be memorable/forgettable) and showed weak correlation for art images.

Interpretability Findings

High FAR Images: Associated with visually simple, high-contrast, and structurally neutral patterns (e.g., waterfalls, zebra stripes). Ablation-CAM and occlusion analyses revealed that the model relies on localized, low-intensity activations in these regions.
Low FAR Images: Associated with complex, semantically surprising, or incongruent scenes. These elicited diffuse, whole-image activations, suggesting the model rejects them as novel based on global complexity.
Feature Contribution: The AlexNet stream was critical for capturing the specific local features driving false alarms, while the ResNet stream provided global context.

Key Contributions

First Image-Level FAR Predictor: FalseResMem is the first neural network explicitly designed and trained to predict image-level false alarm rates, shifting the focus from treating FAR as noise to modeling it as a systematic, image-driven signal.
Hybrid Architecture: The introduction of a dual-stream architecture combining ImageNet-pretrained ResNet-50 with a retrained AlexNet-like stack to capture both global and specific features driving false recognition.
Practical Tool: The authors released FalseResMem as an open-source PyPI package (falseresmem), providing researchers in psychology, neuroscience, and computer vision with an accessible tool to estimate FAR without programming expertise.
Empirical Validation: The study provides empirical evidence that a substantial portion of false recognition variability is predictable from visual features alone, reaching the theoretical ceiling of human consistency.

Significance and Claims

The paper claims that FalseResMem offers a paradigm shift in visual memory modeling by reframing false alarms not as random errors or participant traits, but as predictable properties of images. By achieving performance that matches human inter-observer consistency, the model demonstrates that intrinsic visual features (such as contrast boundaries and spatial uniformity) systematically drive the "deceptive familiarity" associated with false memories.

The authors emphasize the utility of this tool for:

Experimental Design: Allowing researchers to pre-screen stimuli to intentionally elicit or eliminate false memories.
Forensic and Clinical Applications: Potentially improving eyewitness testimony procedures by identifying images prone to false recognition and probing memory biases in clinical populations.
Theoretical Advancement: Providing a quantitative framework to study the "Mandela Effect" and other collective false memories, bridging the gap between classic memory theory and modern deep learning.

The paper concludes that integrating error-based metrics like FAR into visual memory modeling is essential for developing comprehensive theories of memory that account for both accurate recall and systematic distortions.