From Organizational Knowledge to AI Agent Memory: Empirical Validation of the SECI Model on the LongMemEval Benchmark

Technical Summary: From Organizational Knowledge to AI Agent Memory

Problem Statement
AI agent memory systems face a fundamental architectural design question: should conversational history be stored verbatim or transformed into structured knowledge before retrieval? While various engineering solutions exist (e.g., raw storage, LLM-based extraction, virtual memory hierarchies), the field lacks a principled framework to explain why certain architectures retrieve information more effectively than others. Furthermore, empirical comparisons often conflate architectural choices with implementation variables, such as the embedding model used or retrieval depth, making it difficult to isolate the specific mechanism driving performance gains.

Methodology
The authors propose adapting the SECI model of organizational knowledge creation (Nonaka and Takeuchi, 1995) to the context of AI agent memory. The study maps the four SECI phases to memory operations:

• Socialization: The occurrence of a conversation session (tacit knowledge creation).
• Externalization: The extraction of tacit session knowledge into explicit, structured facts (topic, follow-up, resolution).
• Combination: Merging and deduplicating explicit knowledge entries.
• Internalization: Loading retrieved explicit knowledge back into the agent's context for a new session.

To empirically validate this framework, the authors conducted controlled retrieval experiments on the LongMemEval S benchmark (Wu et al., 2025), which contains 500 questions across six task types. The experimental design strictly controlled variables to isolate the effect of representation:

• Fixed Variables: The embedding model (all-MiniLM-L6-v2, max 256 tokens), the vector database (ChromaDB), and retrieval depth.
• Variable: The representation of session content.

Four approaches were compared:

1. Raw All Turns: Storing full session text.
2. Raw User-Only: Storing only user turns (following the MemPalace method).
3. SECI Extraction: A rule-based (non-LLM) condensation of sessions into structured facts (~500 characters).
4. SECI Hybrid: Fusing SECI extraction with Raw User-Only using Reciprocal Rank Fusion (RRF) and keyword boosting.

Key Results
The study yielded three primary findings:

1. Performance of Structured Extraction: The SECI-based extraction approach achieved a 93.7% Recall@5, outperforming the Raw User-Only baseline (92.1%) by 1.6 percentage points. The SECI Hybrid approach achieved the highest score of 93.9% Recall@5.
2. The Embedding Truncation Effect: The dominant factor driving performance differences was identified as embedding window truncation. The all-MiniLM-L6-v2 model processes a maximum of 256 tokens (~1,000 characters). Since LongMemEval sessions average 10,000 characters, raw storage loses approximately 90% of the content at the embedding layer. In contrast, the SECI extraction condenses sessions to ~500 characters, ensuring the entire representation fits within the embedding window.
3. Task-Specific Nuances: While extraction outperformed raw storage on four of six task types (notably single-session-user questions), the Raw User-Only approach outperformed extraction on knowledge-update questions. The authors attribute this to the fact that knowledge updates often rely on the specific wording of temporal changes, which can be lost during condensation.

Key Contributions
The paper claims four specific contributions:

1. Theoretical Framework: A mapping of the SECI model to AI agent memory operations, providing a vocabulary to compare memory architectures based on knowledge conversion modes.
2. Controlled Empirical Comparison: A head-to-head evaluation of structured extraction vs. raw storage on 500 LongMemEval questions, holding the embedding infrastructure constant.
3. Identification of a Confound: The isolation of embedding window truncation as a previously unreported but critical factor in long-term memory retrieval. The study quantifies that raw storage loses ~90% of content under standard short-context embedders.
4. Mechanistic Design Guidance: Evidence that rule-based Externalization (without LLMs in the indexing path) recovers most of the retrieval advantage attributed to more complex LLM-based memory systems. The paper suggests that the "gain" from LLM-based memory construction is largely attributable to the general principle of condensing information to fit retrieval windows, rather than the semantic sophistication of the LLM itself.

Significance and Claims
The authors position this work as a bridge between Knowledge Management (KM) theory and AI memory engineering.

• For KM: It provides an empirical validation of the SECI model in a novel, automated domain.
• For AI Engineering: It offers a mechanistic explanation for retrieval performance, arguing that the primary constraint is not the "richness" of the representation but its compatibility with the embedding model's input window.

The paper concludes that developers should apply the SECI framework selectively: use Externalization (extraction) for stable knowledge to ensure full visibility to the embedding model, but retain raw storage for knowledge updates where temporal nuance and original wording are critical. The authors emphasize that their findings are specific to short-context embedding models and that the performance gap may narrow with longer-context models (e.g., 8,192 tokens), a direction left for future work.