🤖 AI

Agentic Learner with Grow-and-Refine Multimodal Semantic Memory

This paper introduces ViLoMem, a dual-stream memory framework that employs a grow-and-refine principle to separately encode visual distraction patterns and logical reasoning errors, thereby enabling multimodal large language models to overcome the limitations of trajectory-based memory and achieve improved accuracy with reduced repeated mistakes across diverse benchmarks.

Original authors: Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu, Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He, Xiaofan Li, Na Zhao, Jingdong Wang, Zechao Li

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu, Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He, Xiaofan Li, Na Zhao, Jingdong Wang, Zechao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly clumsy, robot to solve puzzles that involve both pictures and math problems.

Right now, most of these robots (called Multimodal Large Language Models) are like students who take a test, get a question wrong, and then immediately forget the mistake. When they see a similar question on the next test, they make the exact same error again. They solve every problem from scratch, as if they've never seen a puzzle before.

Some researchers tried to fix this by giving the robot a "notebook" to write down past mistakes. But these notebooks were flawed. They only wrote down the logic of the mistake (e.g., "I used the wrong formula") and ignored the visual part (e.g., "I looked at the wrong number in the picture"). It's like a teacher telling a student, "You did the math wrong," but never pointing out that the student was looking at the wrong line on the graph.

Enter ViLoMem: The Robot's "Dual-Brain" Memory System

The paper introduces ViLoMem, a new system that gives the robot a smarter, two-part memory, inspired by how human brains work. Instead of one messy notebook, ViLoMem uses two distinct, specialized "streams" of memory that work together but stay separate.

1. The Two Streams of Memory

Think of the robot's brain as having two different librarians:

The Visual Librarian (Visual Memory): This librarian only cares about what the robot sees. If the robot mistakes a shiny metal sphere for a rubber ball, or misreads a tiny number on a chart, this librarian writes down a rule like: "When you see a shiny object, check if it reflects light like metal, not rubber." It also draws a little "heat map" (like a spotlight) on future pictures to show the robot exactly where to look so it doesn't miss the important details.
The Logic Librarian (Logical Memory): This librarian only cares about the thinking. If the robot uses the wrong math formula or adds numbers incorrectly, this librarian writes down a rule like: "Remember, the area of a triangle is ½ × base × height, not just adding the sides together."

2. How It Learns: The "Grow and Refine" Cycle

The system works like a continuous loop of practice and correction:

The Attempt: The robot tries to solve a problem using its current memory.
The Check: A "Verifier" (like a strict teacher) checks the answer.
The Diagnosis: If the robot is wrong, the system asks: "Was this a seeing error or a thinking error?"
- If it was a seeing error, the Visual Librarian updates its rules and draws a new spotlight for next time.
- If it was a thinking error, the Logic Librarian updates its rules.
The Cleanup: The system is smart enough to avoid clutter. If the robot makes the same mistake twice, it doesn't write two new notes; it refines the existing note to be clearer. This prevents the memory from getting too big and confusing (a problem called "catastrophic forgetting").

3. Why This Matters (The Results)

The researchers tested this system on six different types of difficult puzzles involving math, science, and real-world images.

The Result: Robots using ViLoMem got significantly better at solving these problems. They stopped repeating the same visual mistakes (like misreading a diagram) and the same logical mistakes (like using the wrong formula).
The Analogy: Imagine a robot trying to calculate the area of a triangle.
- Without ViLoMem: It might look at the wrong side of the triangle (Visual error) and then use the wrong formula (Logic error). It fails, forgets, and fails again.
- With ViLoMem: After failing once, the Visual Librarian says, "Next time, look at the side labeled '12', not '5'." The Logic Librarian says, "Next time, remember to multiply by ½." On the next try, the robot looks at the right spot and uses the right math, getting the answer correct.

4. The "Cross-Training" Bonus

One of the coolest findings is that this memory system is portable. The researchers showed that a smaller, less powerful robot could learn from the "notebooks" of a much larger, smarter robot. It's like a junior student reading the study notes of a top student and instantly getting smarter without having to go through all the hard work of making the mistakes themselves.

In Summary:
ViLoMem is a system that teaches AI to learn from its mistakes by separating "what I saw" from "what I thought." By keeping these two types of learning in separate, organized files, the AI stops making the same silly errors over and over, becoming a more reliable and accurate problem-solver.

Technical Summary: ViLoMem

Problem Statement

Multimodal Large Language Models (MLLMs) currently exhibit a critical limitation: they solve problems de novo, treating each query independently without retaining or learning from past experiences. This leads to the repetition of identical mistakes and a failure to accumulate domain knowledge over time. While recent memory-augmented agents attempt to store past trajectories, existing approaches suffer from three primary deficiencies:

Brevity Bias: Iterative rewriting of memory strips away essential details, causing a loss of domain-specific knowledge.
Unimodal Focus: Current systems primarily record high-level logical summaries, discarding the visual grounding and perceptual cues essential for multimodal reasoning.
Misalignment with Human Cognition: Human semantic memory is multimodal and integrated, preserving visual and abstract knowledge through distinct but coordinated streams. Existing AI systems fail to replicate this separation, often conflating visual attention errors with logical reasoning errors.

Research indicates that visual perception errors (e.g., diagram misinterpretation) frequently exceed logical reasoning errors and act as a primary bottleneck, causing downstream logical hallucinations. Therefore, a memory framework that treats visual and logical errors as distinct, structured entities is necessary for progressive learning.

Methodology: ViLoMem

The authors propose ViLoMem, a dual-stream memory framework designed for agentic learners. It operates on a "grow-and-refine" principle, constructing compact, schema-based memory that separately encodes visual distraction patterns and logical reasoning errors.

Core Architecture

The system maintains two distinct memory banks:

Logic Memory ( $M_L$ ): Stores textual reasoning guidelines, theorem applications, and logical fallacies.
Visual Memory ( $M_V$ ): Stores visual guidelines paired with source images, focusing on perceptual traps, object attributes, and spatial relationships.

The Memory Cycle

The framework operates through a closed-loop cycle involving four stages:

Retrieval: Given a new multimodal input (image $I_i$ $I_{i}$ , question $q_i$ $q_{i}$ ), the system performs parallel retrieval from both memory banks.
- Visual Retrieval: Uses a two-stage pipeline. First, it performs image-embedding similarity search to find visually similar diagrams. Second, it applies text-embedding filtering using an enriched query (original question + problem analysis) to ensure semantic relevance. Crucially, it generates question-aware attention maps to highlight historically error-prone regions in the query image.
- Logical Retrieval: Uses problem analysis to extract subject domains and key concepts, constructing an enriched query for text-embedding similarity matching against logical schemas.
Utilization: The solver (MLLM) integrates the original inputs with the retrieved logical and visual guidelines to generate a candidate answer.
Verification: A verifier compares the candidate answer against the ground truth.
Generation (Update): Upon detecting an error, a parallel generation process attributes the failure to either the visual or logical stream (or both).
- Visual Generation: An MLLM analyzes the image and reasoning trace to identify visual misinterpretations (e.g., object confusion, overlooked symbols) and generates a structured visual guideline.
- Logical Generation: An LLM analyzes the reasoning chain for non-visual errors (e.g., formula misapplication, calculation errors) and generates a logical guideline.
- Merge/Refine: Before storage, the system performs a similarity check against existing memories. If a new guideline is sufficiently similar to an existing one, they are merged to prevent redundancy and detail erosion. Otherwise, a new entry is created.

Key Technical Innovations

Dual-Stream Decoupling: By separating visual and logical error patterns, the system avoids the "detail erosion" common in single-stream memory and allows for targeted correction of specific failure modes.
Question-Aware Visual Attention: Unlike standard image retrieval, ViLoMem generates attention maps conditioned on the specific question and known error patterns, guiding the model to look at task-relevant regions.
Grow-and-Refine Mechanism: The system incrementally accumulates knowledge while filtering similar error patterns, ensuring long-term scalability without catastrophic forgetting.

Key Contributions

Dual-Stream Memory Framework: The introduction of ViLoMem, which explicitly models visual distraction patterns and logical hallucination errors as separate, structured schemas, inspired by human semantic memory architecture.
Specialized Retrieval Strategies: The development of a two-stage visual retrieval pipeline (image similarity + question-specific text filtering) and a logical retrieval mechanism based on problem analysis, enabling precise memory matching.
Progressive Learning without Forgetting: A mechanism that allows MLLMs to learn from both successful and failed experiences over time, reducing repeated errors through iterative self-correction.
Cross-Model Knowledge Transfer: Demonstration that memory distilled from stronger models can effectively improve the performance of smaller, resource-constrained models without explicit fine-tuning.

Experimental Results

ViLoMem was evaluated across six multimodal benchmarks: MMMU, MathVista, MathVision, HallusionBench, MMStar, and RealWorldQA.

Performance Gains: The framework consistently improved pass@1 accuracy across all tested models (GPT-4.1, Qwen3-VL-235B, and Qwen3-VL-8B). Notable improvements were observed in mathematical reasoning tasks, where visual-logical coupling is critical (e.g., GPT-4.1 saw a +6.48 point gain on MathVision).
Ablation Studies: Removing either the visual or logical stream resulted in consistent performance degradation, confirming that both streams are necessary and complementary. The visual stream was found to be the dominant factor in error generation (59%–93% of stored cases), highlighting visual perception as the primary bottleneck.
Scalability: Experiments on the WeMath benchmark showed that accuracy improved as the memory pool grew from 15k to 150k tokens, demonstrating effective long-term scaling without catastrophic forgetting. Cross-domain memory transfer showed that task-aligned memories are essential for optimal performance, while cross-domain transfer provided only partial benefits.
Efficiency: ViLoMem achieved higher accuracy with significantly lower retrieval latency and storage costs compared to existing memory methods like Dynamic-Cheatsheet, due to its selective update strategy (only storing incorrect answers) and efficient two-stage retrieval.

Significance and Claims

The paper claims that ViLoMem represents a fundamental shift from isolated problem solving to progressive, lifelong learning for multimodal agents. By aligning AI memory mechanisms with human cognitive principles—specifically the separation of visual and logical processing streams—the system enables models to:

Systematically reduce repeated visual and logical errors.
Transfer knowledge across domains and model scales.
Maintain stable, generalizable strategies while avoiding the "brevity bias" of iterative context engineering.

The authors position ViLoMem not merely as a retrieval augmentation tool, but as a foundational mechanism for building agentic learners capable of continuous self-improvement in complex, multimodal scientific reasoning tasks. The work suggests that future advancements in MLLMs will depend on the ability to decouple and coordinate distinct modalities of memory, rather than treating them as a monolithic context.