💻 computer science

Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

This paper introduces a unified, memory-augmented Vision-Language agent that resolves object description inconsistencies across viewpoints by integrating data association, captioning, and exploration into a single autoregressive framework, achieving significant improvements in semantic consistency and captioning scores through self-supervised training on photorealistic 3D environments.

Original authors: Tommaso Galliena, Stefano Rosa, Tommaso Apicella, Pietro Morerio, Alessio Del Bue, Lorenzo Natale

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tommaso Galliena, Stefano Rosa, Tommaso Apicella, Pietro Morerio, Alessio Del Bue, Lorenzo Natale

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a giant, unfamiliar house with a friend who is terrible at remembering things. Every time you turn a corner and look at the same couch from a different angle, your friend gets confused.

From the front, they say, "That's a gray sofa."
From the side, they panic and say, "Wait, that's a bed!"
From the back, they insist, "No, it's definitely a giant armchair!"

Your friend is a standard Vision-Language Model (VLM). They are incredibly smart at describing what they see right now, but they have no memory of what they saw a moment ago. They treat every new glance as a brand-new mystery, leading to a chaotic and inconsistent story about the world.

This paper introduces EPOS-VLM, a new kind of AI agent that solves this problem by giving the robot a memory notebook and a curious personality.

Here is how it works, broken down into simple concepts:

1. The "Memory Notebook" (Episodic Memory)

Instead of just looking at the current picture, EPOS-VLM keeps a running diary.

The Old Way: The robot sees an object, describes it, and then immediately forgets it.
The EPOS Way: The robot writes down: "I saw a gray couch at 2:00 PM. At 2:05 PM, I saw it again from the side, and it looked like a bed. Let me check my notes... Ah, it's the same couch! I'll update my description to be more accurate."

The model turns this memory into a list of text tokens (like a shopping list of objects) that it can read while it's thinking. This allows it to say, "I know this object because I've seen it three times already," ensuring the description stays consistent no matter how the robot moves.

2. The "Detective's Hat" (Data Association)

Imagine you are at a party and you see a person from behind. You think, "Is that Bob?" Then you walk around and see their face. "Yes, it's Bob!"

Standard robots often get confused here. They might think the person from behind is "Bob" and the person from the front is "Dave," creating two different people out of one.
EPOS-VLM acts like a detective. It uses its memory notebook to link the "person from behind" and the "person from front" to the same ID. It knows that even though the view changed, the object is the same. It stops the robot from saying, "That's a new object!" every time it turns its head.

3. The "Curious Explorer" (Active Exploration)

This is the coolest part. Most robots just wander randomly or follow a map. EPOS-VLM is curious.

If the robot looks at a chair and its memory says, "I thought this was a chair, but now it looks like a table," it gets confused.
Instead of ignoring the confusion, EPOS-VLM says, "I need to solve this!" It actively decides to walk around the object to get a better angle.
It treats the world like a puzzle. If a description is shaky, it moves to a spot where the answer becomes clear. It doesn't just wait for the answer; it goes out and finds it.

4. The "Self-Taught Teacher" (Training)

How do you teach a robot to be consistent without a human writing down the perfect description for every single object in the world?

The researchers created a system where the robot explores a virtual house and generates its own "best guess" descriptions.
If the robot sees a couch from 10 different angles and 9 times it says "couch" but once it says "bed," the system learns that "couch" is the truth.
It uses this "consensus" to teach itself. It's like a student studying for a test by taking practice quizzes and correcting their own mistakes until they get the right answer every time.

The Result: A Stable World

When you use EPOS-VLM, the robot builds a stable, consistent map of the world.

No more identity crises: A couch is always a couch, even if you see it from the ceiling or the floor.
Better descriptions: It combines all the little details it saw from different angles into one perfect sentence (e.g., "A gray couch with a blanket and a wooden table next to it").
Efficiency: It doesn't need to store massive 3D point clouds (which take up huge amounts of computer memory). It just stores a simple text list of what it knows, making it fast and lightweight.

In short: EPOS-VLM turns a robot that sees the world as a series of disconnected, confusing snapshots into a smart explorer that builds a coherent, long-term story of the places it visits. It remembers who it met, what they looked like, and keeps its story straight no matter how many times it turns around.

1. Problem Statement

Modern Vision-Language Models (VLMs) excel at captioning static images but fail in embodied navigation scenarios. When an agent moves through a 3D environment, changes in viewpoint, occlusion, and distance cause standard VLMs to generate semantically inconsistent descriptions for the same physical object (e.g., describing a couch as a "bed" from one angle and a "sofa" from another).

Existing solutions suffer from two main limitations:

Decoupled Pipelines: They separate exploration, data association, and captioning into distinct stages, preventing the agent from actively selecting viewpoints to resolve ambiguities.
Lack of Persistent Memory: They rely on offline aggregation or post-hoc consensus, lacking an integrated mechanism to maintain object identity and semantic consistency over long sequences within a single autoregressive framework.

2. Methodology: EPOS-VLM

The authors propose EPOS-VLM (Embodied Persistent Object Semantics), a unified, memory-augmented Vision-Language-Action model. It operates within a single autoregressive framework to jointly perform data association, object-level captioning, and navigation action prediction.

Core Architecture Components:

Inputs:
- Current Observation ( $I_t$ ): RGB image with detected object bounding boxes and transient IDs.
- Explored Map ( $E_t$ ): A top-down RGB map of visited regions.
- Episodic Memory ( $M_{t-1}$ ): A structured, tokenized history of previously observed objects.
Episodic Object Memory:
- Stores persistent object identifiers ($oid$), 3D positions, and a history of captions with occurrence counts.
- Tokenization: The memory is serialized into a structured text format (e.g., [OBJ-ID], [POS], [CAPTIONS]) and injected directly into the language context of the pretrained VLM. This allows the model to reason over long-horizon object histories without architectural changes.
Autoregressive Output:
The model generates a structured sequence containing three distinct tasks:
1. Data Association ([MATCH]): Links current transient object IDs to persistent memory IDs or assigns [NEW_ID] for novel objects.
2. Object Captioning ([CAPTION]): Generates a viewpoint-invariant description based on the current view and the accumulated memory history.
3. Action Prediction ([ACTION]): Selects navigation commands (e.g., move_forward) to maximize information gain.

Training Strategy (Self-Supervised):

3D-Aware Consistent Pseudo-Captioner (3D-CPS): A teacher model generates high-quality "pseudo-captions" for training. It aggregates multi-view observations, filters out view-specific noise (occlusions, perspective artifacts), and uses geometric 3D data to select the most informative image crops.
Disagreement-Based Exploration: During data collection, an exploration policy targets objects with high cross-view caption disagreement to generate diverse training data.
Joint Optimization: The model is fine-tuned to minimize cross-entropy loss across all three output streams (association, caption, action) simultaneously, forcing the agent to learn that consistent descriptions require active exploration and memory retention.

3. Key Contributions

Unified Framework: The first model to jointly optimize data association, object captioning, and exploration policy in a single end-to-end autoregressive policy.
Structured Memory Tokenization: A novel method to embed episodic object memory (positions and caption histories) directly into the VLM's token stream, enabling long-horizon reasoning.
Embodied Dataset & Benchmark: Creation of a dataset with navigation trajectories and object-level pseudo-captions, along with a manually annotated benchmark for evaluating cross-view consistency.
Active Resolution of Ambiguity: The agent learns to revisit semantically unstable objects to resolve identity and attribute ambiguities, rather than passively aggregating data.

4. Experimental Results

The model was evaluated on the HM3D and Gibson datasets (simulated environments) and compared against state-of-the-art VLMs (Qwen-VL, BLIP-2, InternVL) and prior multi-stage pipelines (LD-CPS, IC3).

Caption Accuracy: EPOS-VLM outperformed the best baseline VLM (Intern-VL) by +9.76% in SPICE and +0.89 in CIDEr. It also surpassed the previous multi-stage pipeline (Galliena et al.) by +7.12% in cross-view consistency.
Semantic Consistency: The model achieved a Mean Cosine Similarity (CS) of 89.37 (vs. 61.18 for InternVL) and a significantly lower Interquartile Range (IQR) of 2.79, indicating highly consistent descriptions across viewpoints.
Data Association & Scalability:
- Achieved 93% association accuracy, comparable to dense point-cloud methods.
- Efficiency: Unlike point-cloud baselines that scale linearly in memory (reaching ~~1GB) and time, EPOS-VLM maintains a **constant inference time (~~0.7s)** and low memory footprint (<10kB) due to its sparse, tokenized memory representation.
Generalization: When trained on HM3D and tested on the unseen Gibson dataset, EPOS-VLM maintained superior consistency and accuracy, demonstrating robust transferability.
Ablation Studies: Removing the episodic memory module caused a drastic drop in performance (-14.15% SPICE), confirming memory is essential for consistency. Furthermore, using the EPOS-VLM action policy yielded better results than random or frontier exploration, proving that active exploration driven by semantic uncertainty is critical.

5. Significance

This work represents a paradigm shift in embodied AI by demonstrating that semantic consistency is not an inherent property of visual inputs but a result of memory and active reasoning.

Theoretical Impact: It challenges the static image-captioning paradigm, showing that embodied agents require persistent, object-level memory to construct stable world models.
Practical Impact: By unifying perception, memory, and action, EPOS-VLM enables agents to operate in complex, dynamic 3D environments with reduced computational overhead. This is a crucial step toward deploying reliable VLMs for real-world robotics tasks like navigation, mapping, and long-horizon reasoning.
Efficiency: The demonstration that sparse, tokenized memory can replace dense 3D point clouds for data association offers a scalable path forward for resource-constrained embodied agents.