OCR-Memory: Optical Context Retrieval for Long-Horizon Agent Memory

1. Problem Statement

Autonomous Large Language Model (LLM) agents operating in long-horizon, interactive environments (e.g., web automation, mobile app control) face a critical bottleneck: finite context windows.

• The Conflict: Agents accumulate extensive interaction histories (reasoning traces, tool invocations, feedback) that are crucial for future tasks. However, storing these high-fidelity trajectories as raw text is prohibitively expensive in terms of token budgets.
• Limitations of Existing Solutions:
  • Raw Storage: Exceeds context limits.
  • Summarization/Abstraction: Saves tokens but leads to information loss, fragmentation, and the loss of critical structural or procedural details needed for debugging and complex planning.
  • Text-Based Retrieval (RAG): Often relies on semantic similarity, which can retrieve topically related but logically irrelevant snippets, or fail to capture precise causal dependencies.

2. Methodology: OCR-Memory

The authors propose OCR-Memory, a framework that shifts memory storage and retrieval from the text domain to the visual domain. It treats interaction trajectories as high-density visual data to maximize information retention within minimal token budgets.

Core Components

1. Visual Encoding (Storage):

   • Interaction logs are rendered into images.
   • Multi-Resolution Strategy (Optical Forgetting): To manage long-term storage costs, the system mimics human memory decay. Recent interactions are stored as high-resolution images, while older history is downsampled to low-resolution thumbnails.
   • Active Recall Upsampling: If a low-resolution thumbnail is identified as relevant during retrieval, the system dynamically re-renders it to high fidelity from the original text logs, ensuring full detail is available for the current task.

2. Locate-and-Transcribe Retrieval:

   • Instead of generating free-form text summaries (which causes hallucinations), the system uses a Set-of-Mark (SoM) prompting technique.
   • Visual Anchors: Text segments in the memory images are annotated with unique numerical IDs inside red bounding boxes.
   • Selection Mechanism: The optical retrieval model scans the image and outputs a binary relevance vector (selecting specific IDs) rather than generating text.
   • Deterministic Fetching: Once relevant IDs are selected, the system retrieves the verbatim text from an external database. This decouples "search" (visual) from "generation" (text), eliminating hallucinations.

3. Training Strategy:

   • Backbone: Built upon DeepSeek-OCR.
   • Fine-tuning: The model is fine-tuned on a repurposed HotpotQA dataset. The task is transformed from text generation to discriminative retrieval (predicting which paragraphs support a query).
   • Optimization: Uses a weighted Binary Cross-Entropy loss to prioritize recall (penalizing missed evidence more than false positives).
   • Resolution Curriculum: During training, images are randomly downsampled to simulate the low-resolution conditions of long-term memory, ensuring the model remains robust to visual degradation.

3. Key Contributions

• First Visual Memory Framework: Introduces the first agent memory system that stores interaction history as images, enabling the retention of arbitrarily long episodic experiences under strict context limits.
• Locate-and-Transcribe Paradigm: Mitigates hallucination by replacing free-form text generation with explicit index selection via visual anchors, followed by deterministic text fetching.
• Adaptive Resolution & Active Recall: A novel mechanism that balances storage efficiency (low-res for old data) with fidelity (high-res for active data), mimicking human memory dynamics.
• Empirical Validation: Comprehensive experiments demonstrating state-of-the-art performance on long-horizon benchmarks.

4. Experimental Results

The system was evaluated on Mind2Web (web navigation) and AppWorld (API interactions).

• Performance Gains:
  • Mind2Web: OCR-Memory achieved a Task Success Rate (SR) of 4.8%, outperforming the strong baseline AWM (4.3%) and standard Retrieval (2.7%). It also improved Element Accuracy to 53.8% (vs. 49.1% for AWM).
  • AppWorld: Achieved an average SR of 58.1%, with a significant lead on "Hard" tasks (30.8% vs. 27.2% for AWM).
• Token Efficiency:
  • Under strict token limits (e.g., 1024 tokens), OCR-Memory maintained high accuracy, whereas text-based RAG degraded significantly due to information loss.
  • Visual encoding achieved a 10x+ compression ratio compared to raw text while maintaining 94.1% retrieval accuracy even at 32k token context lengths (NIAH benchmark).
• Faithfulness:
  • OCR-Memory achieved 100% faithfulness in evidence recovery (verbatim text retrieval), whereas generative retrieval methods suffered from hallucinations (84.3% faithfulness).
• System Trade-offs:
  • Pros: Drastically reduces reasoning context tokens (from ~3,980 to ~596 per step).
  • Cons: Increases disk storage usage (from 18 KB to 1.47 MB per episode) and retrieval latency (0.3s to 1.7s). The authors argue this is an acceptable trade-off as reasoning tokens are the scarcest resource.

5. Significance

This work fundamentally rethinks how agents manage long-term memory. By leveraging the visual modality as a high-density, lossless storage medium, OCR-Memory solves the "context window vs. information completeness" dilemma.

• Scalability: It enables agents to operate over extended histories without losing critical structural details.
• Reliability: The "Locate-and-Transcribe" approach ensures that retrieved evidence is exact and hallucination-free, which is critical for complex, multi-step planning and debugging.
• Generalizability: The performance gains were consistent across different backbone models (GPT-4 and Qwen3-32B), indicating the method is a robust architectural improvement rather than a model-specific artifact.

In summary, OCR-Memory demonstrates that optical encoding is a superior strategy for long-horizon agent memory, offering a path to scalable, faithful, and token-efficient autonomous systems.