Wontopos Tablet 2: Measuring Multilingual and Multimodal Memory Retrieval Without Lexical Matching

Technical Summary: Wontopos Tablet 2 Benchmark Report

Problem Statement
The paper addresses the difficulty of evaluating long-term memory retrieval systems for language models (LLMs) in isolation. Standard benchmarks typically measure the performance of a retrieval engine coupled with a specific reader model and judge, making it impossible to distinguish whether a score reflects the quality of the retrieval or the reasoning capability of the reader. Furthermore, existing evaluations rarely test retrieval across languages or modalities without relying on lexical matching (term overlap), which fails when queries and documents share no common vocabulary or when documents contain no text (e.g., photographs).

Methodology
The author reports measurements of tablet-2, a production memory engine, across three distinct axes:

1. Text Benchmarks (LongMemEval-S and BEAM-1M):

   • LongMemEval-S: 500 questions over multi-session dialogues.
   • BEAM-1M: 700 questions over a corpus of 74,630 turns (2.21M memories).
   • Configuration Control: The author systematically varies the reader model (e.g., Claude Opus 5 vs. GPT-5.6-sol), the "re-ask" budget (allowing up to 3 retrieval passes), and context expansion policies. They explicitly measure the impact of these settings on the final score.
   • Token Counting: Unlike previous reports that estimated tokens from character counts, this study uses a tokeniser (o200k_base) to count delivered tokens, correcting a 33–72% overstatement in prior figures.

2. Multilingual Multimodal Retrieval (Lexical Control):

   • Synthetic Corpus: 30 images with captions in 10 languages. The core test condition involves storing photographs with no caption and querying them with text descriptions.
   • Baseline: A maximally configured BM25 (Okapi) system. Since BM25 relies on term overlap, it cannot score documents with no text.
   • Real-World Corpus: 300 photographs from Crossmodal-3600 with human-written captions in 14 languages (no translations). The task is to retrieve the correct image from a text query in a different language.

3. Open Baselines:

   • Two open-source models (CLIP with an English-only text tower and a multilingual text tower) are run on the same 300 caption-less photographs to determine if "dense" retrieval inherently confers language independence.

Key Contributions

1. Production Engine Measurement: Provides detailed performance metrics for a production memory engine on two major text benchmarks, including latency, delivered token costs, and ablation studies on reader models and retrieval budgets.
2. Retrieval Isolation: Demonstrates that retrieval quality is often confounded by reader behavior. By disabling "re-ask" (iterative retrieval), the author isolates the engine's first-pass retrieval score, showing it can match a stronger reader's performance with significantly fewer delivered memories.
3. Lexical Independence Proof: Establishes that lexical matching (BM25) is structurally incapable of retrieving uncaptioned images or cross-lingual text without pre-indexed translations. The paper shows BM25 achieves 0% recall on caption-less images, whereas tablet-2 achieves high recall.
4. Dense Retrieval Limitations: Shows that dense retrieval is not automatically language-agnostic. Open baselines using multilingual text towers perform well on languages they were trained on but collapse on low-resource languages (e.g., Swahili, Telugu), proving that language independence is a property of training data, not just the absence of term matching.
5. Negative Results and Defect Attribution:
   • Low-Resource Degradation: The system performs poorly on Swahili (53.0% recall@5) and Telugu (64.0%), though this is an order of magnitude better than open baselines (6.0% and 5.0%).
   • Caption Interference: Attaching English captions to images degrades cross-lingual retrieval for other languages by an average of 11.4 points, identified as a defect in the current product logic rather than a benchmark artifact.
   • Configuration Error: A missing configuration field for Korean short queries caused a 37-point drop in top-1 accuracy, which was identified by isolating the retrieval stage from the full system.

Results

• LongMemEval-S: tablet-2 scores 95.7% (95% CI: [93.4, 97.1]) with Opus 5 max and re-ask enabled. Disabling re-ask drops the score to 93.8%, while changing the reader model shifts scores by ~2.0 points.
• BEAM-1M: tablet-2 scores 67.5% (95% CI: [64.8, 70.2]). The "re-ask" budget alone accounts for an 8.9-point swing (58.6% to 67.5%). Context expansion increases the score to 70.3% but at a 6.7x token cost.
• Multilingual Photo Retrieval:
  • No Captions: tablet-2 achieves 91.4% mean recall@5 across 14 languages. BM25 achieves 0.0%.
  • Open Baselines: An English-trained text tower scores 91.0% on English but 4.7% on Russian. A multilingual tower is flat across trained languages but drops to ~5% on Swahili and Telugu.
  • Lexical Baseline: BM25 reaches 100% only when the query and document share the exact language; it fails completely on cross-lingual pairs and caption-less images.

Significance and Claims
The paper claims that the value of these measurements lies not in establishing a new ranking (which the author argues is impossible due to uncontrolled variables like reader models and retrieval budgets in other reports) but in isolating the retrieval component and bounding its capabilities.

• On Ranking: The author explicitly states that comparing their BEAM-1M score (67.5%) against other published scores (e.g., 75.0%) is not a valid ranking because the other systems use different reader models and unreported retrieval budgets. The gap between systems is often smaller than the variance introduced by changing the reader or the re-ask budget.
• On Language Independence: The paper argues that "language-agnostic" retrieval is not an inherent property of dense models. It is a function of the training data's language coverage. The system's performance on low-resource languages is limited by the underlying retrieval stage, not the surrounding system logic.
• On Methodology: The author emphasizes a procedure for debugging: when a specific language or condition performs poorly, one must measure the specific retrieval stage in isolation. If the isolated stage matches the full system's poor performance, the limit is inherent to that stage; if the full system is worse, the limit is in the surrounding handling logic. This procedure allowed them to identify and fix a configuration error that had been misattributed to the model's capabilities.

The paper concludes that while tablet-2 demonstrates robust cross-lingual and multimodal retrieval without lexical matching, it still suffers from specific defects (caption interference) and inherits the limitations of its underlying components regarding low-resource languages. The primary contribution is a transparent, reproducible measurement framework that separates retrieval quality from reader performance and exposes the structural limits of both lexical and dense retrieval strategies.