Technical Summary: Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

1. Problem Statement

Recent unified multimodal models have demonstrated the capability to perform vision-language understanding and image generation/editing within a single architecture. However, these monolithic designs face critical scalability limitations in long-horizon multimodal dialogue (e.g., multi-turn conversations spanning 20+ turns).

The primary challenges identified are:

• Visual Token Explosion: Repeatedly feeding all historical visual and textual inputs into a shared context window leads to prohibitive token usage, crowding out the budget available for reasoning.
• Unreliable Cross-Turn Referencing: Reliance on implicit attention over extended contexts weakens the model's ability to retrieve specific visual episodes from earlier turns, resulting in retrieval errors and semantic drift.
• Data Scarcity: Existing multimodal dialogue datasets lack fine-grained, turn-level supervision for visual retrieval, making it difficult to train agents to selectively recall relevant historical images.

Current unified models, even large-scale ones (e.g., 32B parameters), struggle to retrieve correct visual episodes in difficult, long-range scenarios, achieving less than 3% accuracy on the hardest turns of the proposed benchmark.

2. Methodology

The authors propose the Cognitive-structured Multimodal Agent, a framework that decouples perception, memory, retrieval, and execution to address the limitations of monolithic scaling. The system externalizes visual information into structured memory units rather than treating them as persistent tokens in the context window.

Core Architecture

The agent consists of three specialized modules:

1. Perceptual Abstraction Engine (PAE): Processes incoming images to generate structured semantic abstractions, including descriptive captions, attribute tags, and compact thumbnails. These are stored in an external Episodic Visual Memory (EVM).
2. Cognitive Retrieval Engine (CoRE): Given the current dialogue state and user query, CoRE performs cross-modal reasoning to retrieve only the relevant visual episodes from the EVM.
3. Multimodal Executive Controller (MEC): Integrates the dialogue context with the retrieved memory. It autonomously infers task intent (understanding, generation, editing, composition, or chat) and orchestrates the appropriate action, dispatching requests to specific generation or editing models (e.g., Qwen-Image-Edit).

Training and Data Construction

To overcome the lack of retrieval supervision in existing datasets, the authors developed:

• Unified Scenario Engine: A programmable pipeline that generates structured multi-turn conversations with fine-grained retrieval annotations. It simulates diverse visual scenes and dialogue trajectories, automatically specifying which historical visual episode is relevant to each query.
• Staged SFT + RL Pipeline:
  • Supervised Fine-Tuning (SFT): Trains the CoRE to retrieve relevant indices based on ground-truth retrieval sets.
  • Reinforcement Learning (RL): Utilizes the DAPO algorithm to optimize both the retriever and the memory writer (PAE). The reward function is based on the difficulty-weighted Jaccard similarity between predicted and ground-truth retrieval sets, rather than traditional captioning metrics. This ensures memory representations are optimized for downstream retrieval success.

Benchmark Construction

The authors constructed the Multi-turn Context Agent Benchmark (M2CA-Bench), comprising 100 sessions of 20 turns each (2,000 annotated turns). The benchmark stratifies turns into four difficulty levels (Easy, Medium, Hard, Very Hard) based on temporal span, topic-shift frequency, multi-image interaction, and referential ambiguity. It includes hard negatives (high-similarity confounders) to prevent shortcut learning.

3. Key Contributions

1. Cognitive-structured Agent Architecture: A novel framework for long-horizon dialogue that externalizes visual history into episodic memory and enables autonomous task inference through modular decision-making.
2. Unified Scenario Engine: A tool for programmatically generating structured multi-turn multimodal conversations with turn-level retrieval supervision, addressing a critical gap in training data.
3. M2CA-Bench: A specialized benchmark designed to evaluate cross-turn visual retrieval and episodic memory in sustained multimodal interaction.
4. Optimization Techniques: The application of staged SFT and RL to optimize perceptual abstraction and retrieval policies, enabling a smaller 8B model to outperform larger unified baselines.
5. CMA-Harness: A tool-augmented deployment of the cognitive structure that integrates persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving.

4. Experimental Results

Experiments were conducted on M2CA-Bench using an 8B agent (Qwen3-VL-8B backbone) compared against unified baselines (BAGEL) and modular baselines (8B/32B All-Context agents).

• Retrieval Accuracy: The 8B Cognitive-structured Agent achieved 91.4% retrieval accuracy over 20-turn English sessions, surpassing the strongest 32B All-Context baseline by +8.2%. On the "Hard" subset (long-range callbacks and complex references), the gap widened to +18.4%.
• Efficiency: By retrieving only relevant visual episodes, the agent nearly halved the per-turn inference time (12.7s vs. 23.1s for the 32B All-Context baseline) while maintaining lower visual token usage.
• Generation Quality: The agent achieved the highest generation quality scores (8.49 EN / 8.53 CN), with performance gains directly correlating with improved retrieval accuracy.
• Ablation Studies:
  • Staged training (SFT + RL) significantly improved performance, with RL on the PAE providing an additional 4.6% gain on the Hard subset.
  • Text-only memory ablations (removing thumbnails) caused a sharp performance drop on hard turns (down to ~24-39%), confirming the necessity of visual metadata for fine-grained recall.

5. Significance and Claims

The paper claims that structured memory and modular decision-making provide a more scalable and efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling.

Key takeaways emphasized by the authors include:

• Scalability: Externalizing visual history allows the system to maintain long-range consistency without the exponential token costs associated with context window expansion.
• Efficiency: Decoupling the executive controller from the retrieval engine allows for independent scaling and optimization, allocating model capacity where it matters most for the performance-latency trade-off.
• Practicality: The CMA-Harness demonstrates that this architecture can be deployed as a functional, tool-augmented assistant capable of handling open-ended multimodal workflows, including web search and complex image composition.

The authors conclude that while unified models excel at single-turn tasks, they are insufficient for sustained interaction, and that explicit memory management is essential for the next generation of multimodal agents.