Technical Summary: Do You Remember? Toward Memory-Centric Multimodal AI

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) operate on a unidirectional paradigm: images are processed by a frozen visual encoder, projected into an LLM's embedding space, and consumed to generate text. Once the text is produced, the visual information within the LLM's hidden states is discarded. The authors argue that this architecture lacks a mechanism for "reconstructive memory"—the ability to recall and verify what was perceived.

Drawing from cognitive science (Bartlett, 1932), the paper posits that human memory is not a faithful recording but a reconstructive process. The central hypothesis is that for MLLMs to truly "remember," they must be able to reconstruct perceptual content from their internal representations. However, the authors identify a fundamental disconnect: while MLLMs may "understand" an image (generating accurate semantic analysis), their hidden states may not retain any "decodable visual representation" necessary for reconstruction. Furthermore, attempts to train a shared memory matrix to bridge this gap have historically failed due to gradient cancellation and the irreversible nature of the LLM's information transformation.

2. Methodology: The DoYouRemember Architecture

The authors propose DoYouRemember, a three-stage architecture designed to introduce reconstructive memory into MLLMs:

1. Sensory Compression (Stage 0): A Vector Quantized Variational Autoencoder (VQ-VAE) compresses 256×256 images into 1,024 discrete visual tokens. This stage serves as a reversible, lossy compression baseline (approx. 200:1 ratio) that preserves diagnostically relevant features while discarding pixel redundancy.
2. Conscious Reasoning (Stage 1): Visual tokens and text prompts are processed by a LoRA-fine-tuned LLM (Qwen2.5-7B-Instruct). The model generates dermatological analysis text while preserving hidden states at visual token positions.
3. Reconstructive Verification (Stage 2): A Diffusion Transformer (DiT-style) decoder attempts to reconstruct the original image from the LLM's hidden states or a dedicated memory matrix.

Experimental Design

The authors conducted a series of controlled experiments to isolate information flow:

• Upper Bound Experiment: Comparing reconstruction from VQ-VAE tokens (pre-LLM) vs. LLM hidden states (post-LLM) using the same Diffusion Decoder.
• Memory Architectures: Four generations of shared memory modules (SM-v1 to SM-v4) were tested, including attention-based writing, spatial sharding, and direct gradient paths.
• Training Strategies: The authors tested backpropagation-based methods (MSE, DDPM) and contrastive learning (SimCLR, BYOL, SimSiam) before settling on Local EMA (Exponential Moving Average).

Datasets

• Labeled: 1,000 3D facial skin texture maps with expert dermatological analyses.
• Unlabeled: 99,000 unlabeled frontal facial images used for unsupervised memory accumulation.

3. Key Findings and Results

A. The "Understanding vs. Memory" Gap

The most critical finding is that LLM hidden states contain approximately zero recoverable visual information.

• Evidence: The Diffusion Decoder successfully reconstructed clear facial features from VQ-VAE tokens (pre-LLM) but produced pure high-frequency noise when fed LLM hidden states (post-LLM).
• Implication: The LLM transforms visual information into semantic meaning for text generation, rendering the visual substrate irrecoverable. This is not merely "lossy compression" (like VQ-VAE) but "irreversible information transformation." The model understands the image but does not remember it.

B. The Failure of Backpropagation for Shared Memory

The authors systematically demonstrated that training a shared memory matrix ($M$) via backpropagation fails across all tested architectures and contrastive methods.

• Gradient Cancellation: When a shared matrix serves $N$ heterogeneous inputs, per-sample gradients cancel out. The effective gradient magnitude scales as $O(1/\sqrt{N})$. With $N=99,000$, the gradient signal is attenuated by a factor of ~314, falling below initialization noise.
• The Chicken-and-Egg Deadlock: Reconstruction loss cannot train the compressor because the compressor must already know what to preserve to minimize loss, but it cannot learn what to preserve without a gradient signal.
• Global EMA Failure: Global smoothing of memory slots via EMA destroys inter-slot diversity, causing all slots to converge to near-identical vectors.

C. The Success of Local EMA

The authors propose Local EMA as a non-backpropagation solution that resolves the identified root causes.

• Mechanism: Each image updates only its top-$k$ (e.g., 8) most relevant memory slots out of a total of $K$ (e.g., 64) via exponential moving average. No gradients are computed; updates are driven by activation similarity.
• Results:
  • Diversity: Local EMA preserves and increases inter-slot diversity (variance increased from 0.040 to 0.194), avoiding the collapse seen in global EMA.
  • Reconstruction Quality: A 229K-parameter memory matrix ($K=64$) achieved reconstruction quality approaching the VQ upper bound on unseen test images (LPIPS 0.123 vs. 0.071; SSIM 0.775 vs. 0.877).
  • Scaling: Increasing memory slots to $K=1024$ (3.67M parameters) allowed the memory to surpass the VQ upper bound (LPIPS 0.056 vs. 0.071). This verifies that the continuous representation of $M$ avoids VQ's quantization error, provided the slot count is sufficient to reduce EMA cancellation below the quantization threshold.

4. Theoretical Contributions

The paper unifies these findings under an Information-Theoretic Framework:

1. Memory as Lossy Compression: Memory is defined as lossy compression, and recall as decompression.
2. Rate-Distortion Limit: Gradient cancellation is identified as the mathematical manifestation of the rate-distortion limit when a single shared representation serves heterogeneous inputs.
3. Local Learning Rules: The paper argues that shared memory requires local learning rules (like Hebbian plasticity) rather than global backpropagation. Local EMA succeeds because it bypasses gradient cancellation by restricting updates to semantically similar clusters.
4. Hallucination as Decompression Artifact: The authors reframe hallucination not as a defect but as an inherent property of lossy decompression. When information is lost during compression, the decompressor must interpolate missing data, leading to plausible but potentially inaccurate outputs. This applies to LLMs, human recall, and diffusion models alike.

5. Significance and Claims

The paper claims to establish a new paradigm for Memory-Centric Multimodal AI:

• Architectural Shift: It proposes moving away from the "bolt-on" vision encoder + LLM pipeline toward a shared memory matrix ($M$) as the computational center. In this paradigm, perception encoders act as write-heads, the LLM acts as a consciousness layer for semantic organization, and recall is an iterative decompression process.
• Validation of Local Rules: It provides the first experimental validation that local learning rules (Local EMA) can successfully train shared memory representations where backpropagation fails.
• Domain Application: The authors identify skin health imaging as an ideal validation domain due to its high density of co-registered multi-modal data (20+ image types per patient), which allows for testing the "redundant coding" hypothesis (that multi-modal encoding reduces hallucination).

The work concludes that the distinction between "understanding" (semantic extraction) and "memory" (retention of decodable perceptual representation) is fundamental to current Transformer architectures. Achieving true reconstructive memory requires architectural changes that prioritize local learning rules and persistent, shared memory substrates over unidirectional text-generation objectives.