Facts Without Owners. Consolidating a Conversation into a Language Model's Weights Stores What Was Said, Not Who Said It

Technical Summary: Facts Without Owners

Problem Statement
The paper investigates the limitations of "internal" conversational memory, where a language model (LM) consolidates a user's conversation history directly into its weights via fine-tuning, as opposed to "external" memory which relies on retrieving data from an archive. While internal memory promises resilience against retrieval failure and seamless integration with prior knowledge, it carries risks such as hallucination and directional learning failures. The specific research question is: When a real-length conversation is consolidated into a model's weights, what information is stored, and what is lost? Specifically, does the model learn who said a fact, or merely what was said?

Methodology
The study utilizes three real-length, two-person conversations from the LoCoMo benchmark (conv-42, conv-43, conv-48), containing 629–681 turns each. These conversations include a specific adversarial category (Category 5) known as "swapped-person" questions, where a question asks about Speaker A regarding a fact that only Speaker B mentioned. The correct behavior for the model is to abstain, as the fact does not belong to Speaker A.

• Model & Consolidation: The base model is Qwen3-4B. The author employs a LoRA (Low-Rank Adaptation) adapter (Rank 16) trained on self-generated questions. The base model reads conversation windows and writes short-answer questions. Crucially, 15% of questions about one speaker are "traps" where the name is swapped, and the target answer is "Not mentioned."
• Training Recipe (D1): The adapter is trained on these generated questions for three epochs. To prevent catastrophic forgetting of general knowledge, a KL-divergence term ties the adapter to the base model on anchor texts (SQuAD and chat requests).
• Evaluation: The model is tested on the original LoCoMo benchmark questions (which it never saw during training) across four arms: Base model without archive (B0), Consolidated model without archive (D), Base with archive (B0S), and Consolidated with archive (DS).
• Intervention Variants: Seven pre-registered variants (D2–D6) were tested to fix attribution errors using only weights or model signals (e.g., inverse questions, "I don't know" examples with swapped speakers, and confidence thresholds).
• Provenance Check (D7): A model-free check against the raw archive was implemented. It compares the retrieved turns for the answer against the speaker named in the question. If the answer aligns more strongly with the other speaker's turns, the model is forced to abstain.
• Rigor: All experiments were pre-registered. Labels were generated by an LLM (Claude) following fixed rules, with strict handling of borderline cases.

Key Results

1. Consolidation Works for Facts, Not Owners:

   • Without an archive, the base model (B0) abstains on nearly 100% of questions (0.00 accuracy).
   • The consolidated model (D) successfully retrieves facts, achieving 28–39% accuracy on standard benchmark questions (Categories 1–4) across the three conversations.
   • General knowledge (TriviaQA) remained stable, showing no detectable damage from consolidation.

2. The Attribution Failure:

   • The dominant failure mode is attribution. On "swapped-person" questions, the consolidated model abstains only 9–33% of the time.
   • Instead of abstaining, the model often answers confidently with the fact, even though it belongs to the wrong speaker. In roughly 30% of these errors, the model's answer matches a first-person turn of the other speaker.
   • The model learns the topic-to-answer association but treats the speaker's name as a weak cue.

3. Ineffectiveness of Weight-Only Fixes:

   • Variants attempting to teach the model to abstain (D2–D6) using inverse questions, "I don't know" examples, or confidence thresholds failed to replicate on held-out conversations.
   • These methods created a trade-off line: increasing abstention on swapped-person questions came at a high cost to overall accuracy (losing 0.09–0.17 accuracy for every 0.40 increase in abstention).
   • The "I don't know" examples taught the model the form of the trap rather than the actual knowledge boundaries. The model's confidence scores were indistinguishable between correct and incorrect attribution (AUROC ~0.54–0.56 on held-out data).

4. Efficiency of Archive Checks:

   • The model-free provenance check (D7) against the raw archive significantly outperformed weight-only methods.
   • It raised abstention on swapped-person questions from ~0.17 to 0.45–0.47 while costing only 0.005–0.042 in accuracy.
   • This cost is one-fifth to one-half of the cost incurred by weight-only variants at the same abstention level.
   • Note on Pre-registration: In one of two runs, the accuracy loss (7.0%) slightly exceeded the pre-registered limit of 5%, leading the author to report the check as "not adopted" under their own strict rules, despite the significant efficiency gain.

Significance and Claims
The paper concludes that consolidating a conversation into a model's weights stores "what was said," but not "who said it."

• Complementary Roles: The author argues that parametric memory (weights) and external archives are complementary. Weights are effective for holding knowledge, while the archive is essential for holding provenance (ownership of facts).
• Limitations of Current Remedies: Current techniques to force models to abstain (via training data manipulation or confidence thresholds) are inefficient because the model is technically "correct" about the fact, making it difficult for internal signals to detect the attribution error.
• Practical Recommendation: For personal assistants, relying solely on weight consolidation is insufficient for accurate attribution. The archive should be retained, not necessarily for every retrieval, but as a source for provenance checks to verify speaker ownership.

The study emphasizes that while internal memory is a viable mechanism for knowledge retention, it currently lacks the inherent ability to bind facts to their specific speakers without external verification.