SuperLocalMemory: Privacy-Preserving Multi-Agent Memory with Bayesian Trust Defense Against Memory Poisoning

1. Problem Statement

As AI agents increasingly rely on persistent memory to retain factual knowledge, context, and user preferences, existing memory architectures face critical security and privacy vulnerabilities:

• Memory Poisoning (OWASP ASI06): Malicious actors can inject false information into shared memory, which persists across sessions and influences future agent decisions. Current systems lack robust defenses against "sleeper agents" (agents that behave normally before attacking) or indirect injection via external tools.
• Cloud-Centric Risks: Most memory systems (e.g., Mem0, MemOS) are cloud-default. This creates centralized attack surfaces, multi-tenant exposure (one compromised agent affects others), opaque provenance (users cannot audit data origins), and vendor lock-in.
• Lack of Personalization without LLMs: Existing systems often rely on Large Language Model (LLM) inference for retrieval ranking or personalization, introducing latency, cost, and privacy concerns (data leaving the local machine).
• Performance vs. Complexity: There is a trade-off between sophisticated memory structures (knowledge graphs) and search latency, particularly on commodity hardware.

2. Methodology: SuperLocalMemory Architecture

The authors propose SuperLocalMemory, a local-first system designed to eliminate cloud dependencies while providing trust-aware coordination. The architecture consists of four progressive layers and a coordination framework:

A. Four-Layer Memory Stack

1. Layer 1: Storage Engine: Built on SQLite with FTS5 for full-text search. It implements Write-Ahead Logging (WAL) for concurrency and a thread-safe write queue. It stores content, tags, importance scores, and timestamps.
2. Layer 2: Hierarchical Index: Uses a materialized path scheme to maintain parent-child relationships, enabling O(1) parent lookups and contextual retrieval (e.g., retrieving a memory and all its sub-memories).
3. Layer 3: Knowledge Graph: Extracts key terms via TF-IDF and constructs edges based on cosine similarity. It uses the Leiden algorithm for community detection to create hierarchical sub-clustering. To manage computational cost, it caps graph construction at 10,000 memories (with optional HNSW indexing for scaling).
4. Layer 4: Pattern Learning: Tracks user preferences across 8 technology categories using a Beta-Binomial Bayesian model. It calculates confidence scores based on evidence counts to adapt retrieval without LLMs.

B. Event Coordination & Trust Defense

• Event Bus: An event-driven layer (v2.5) using Server-Sent Events (SSE), WebSockets, and Webhooks to broadcast memory events (creation, recall, agent connection).
• Bayesian Trust Scoring:
  • Agents start with a trust score of 1.0.
  • Trust evolves via a Beta-Binomial posterior inference model.
  • Asymmetric Signals: Negative signals (contradictory writes, flagged content) have a larger magnitude than positive signals, making trust harder to gain than to lose.
  • Enforcement: Agents with trust scores below a threshold (default < 0.3) are blocked from write/delete operations.
  • Provenance Tracking: Every memory record includes created_by, source_protocol, trust_score, and a provenance_chain (JSON history of modifications), enabling forensic isolation.

C. Adaptive Learning-to-Rank (v2.7)

To address the limitation that structural layers (Graph/Patterns) do not inherently improve search ranking, an Adaptive Learning Layer was added:

• Three-Layer Behavioral Analysis: Mines preferences via (1) cross-project tech preferences, (2) project context detection, and (3) workflow pattern mining (time-weighted sliding windows).
• Zero-LLM Re-ranking:
  • Phase 0 (<20 signals): Returns baseline results.
  • Phase 1 (20–199 signals): Rule-based deterministic boosting based on a 9-dimensional feature vector (BM25, TF-IDF, tech match, etc.).
  • Phase 2 (200+ signals): A Gradient Boosted Decision Tree (LambdaRank) re-ranker trained on real feedback.
• Privacy Isolation: All behavioral data is stored in a separate learning.db, allowing for GDPR Article 17 "right to be forgotten" via one-command deletion without affecting the memory store.

3. Key Contributions

1. Local-First Trust Architecture: A system that eliminates cloud attack surfaces entirely, combining SQLite storage with per-agent provenance tracking and Bayesian trust scoring to defend against OWASP ASI06.
2. Bayesian Trust Framework: A novel scoring mechanism that achieves a 0.90 trust gap between benign and malicious agents and detects "sleeper" attacks with 72% trust degradation.
3. Zero-LLM Adaptive Ranking: A privacy-preserving re-ranking framework that improves retrieval relevance by 104% (NDCG@5) without requiring any LLM inference calls, using only local behavioral data.
4. Comprehensive Evaluation: Honest benchmarking across seven dimensions, including search latency, concurrency, graph scaling, and human-validated pilot studies.

4. Experimental Results

Evaluated on an Apple M4 Pro (24GB RAM) with synthetic and real-world data:

• Search Performance:
  • Latency: 10.6ms median latency for 100 memories; scales roughly linearly up to 1,000 memories.
  • Storage Efficiency: 1.4KB per memory at scale (13.6MB for 10,000 memories).
• Concurrency:
  • Achieved zero "database is locked" errors under 10 simultaneous agents using WAL mode.
  • Peak throughput of ~220 writes/sec with 2 concurrent agents.
• Trust Defense:
  • Benign Agents: Converged to a trust score of 0.945.
  • Malicious Agents: Trust dropped to 0.048 (Single Poisoner) or 0.249 (Sleeper Agent), successfully triggering enforcement blocks.
• Retrieval Quality:
  • Base System (FTS5): MRR of 0.90, but low NDCG@5 (0.441), indicating arbitrary ordering within relevant topics.
  • With Adaptive Ranker: NDCG@5 improved to 0.900 (+104%) and NDCG@10 to 0.728, with only a 20ms latency overhead.
  • Human Validation: A pilot study with a developer user confirmed an MRR of 0.70 and NDCG@5 of 0.90, validating real-world utility.

5. Significance and Impact

• Security Paradigm Shift: SuperLocalMemory is the first system to combine a local-first architecture with active trust defense specifically targeting memory poisoning. By keeping data on the user's machine, it removes the centralized attack surface inherent in cloud-based memory services.
• Privacy-Preserving Personalization: It demonstrates that high-quality, personalized retrieval can be achieved through local behavioral analysis and statistical learning (Beta-Binomial, LambdaRank) without the privacy risks or costs of sending data to LLMs.
• Practical Deployment: The system requires zero external dependencies for core operation (only Python standard library), integrates with 17+ tools via the Model Context Protocol (MCP), and is open-source (MIT License).
• Future-Proofing: The architecture is designed to support future inter-agent communication (A2A Protocol) and compliance with regulations like the EU AI Act, offering a robust foundation for secure, multi-agent ecosystems.

In conclusion, SuperLocalMemory addresses the critical intersection of security, privacy, and performance in multi-agent systems, proving that local-first architectures can effectively mitigate memory poisoning while delivering personalized, high-speed retrieval.