Technical Summary: Knowledgeless Language Models (KLLMs)

Problem Statement

Large Language Models (LLMs) acquire substantial factual knowledge during pretraining, which is encoded in their parameters. While this enables broad capability, it often leads to unreliable behavior, specifically hallucination, when the internal parametric knowledge is outdated, incomplete, or misaligned with the provided context. Standard mitigation strategies, such as Retrieval-Augmented Generation (RAG), often fail because the models are still pretrained to memorize and reproduce facts, creating a mismatch where models rely on parametric recall rather than external evidence even when context is available. The central question addressed is whether the pretraining signal can be modified to systematically shift models away from parametric recall and toward evidence-grounded reasoning without sacrificing reasoning ability.

Methodology

The authors introduce Knowledgeless Language Models (KLLMs), a training paradigm that modifies the pretraining signal by systematically anonymizing named entities in the training corpus.

• Corpus Anonymization: The pretraining data is processed using a state-of-the-art Named Entity Recognition (NER) model (Flair framework with XLM-R embeddings). All identified named entities (e.g., people, organizations, locations) are replaced with unique placeholder tokens (e.g., PERSON184, ORG42). Crucially, numerical and temporal values are retained to preserve semantic coherence and readability.
• Training Procedure: KLLMs are trained from scratch on this anonymized corpus using standard autoregressive next-token prediction objectives. No architectural changes are made; the models share identical architectures, tokenizers, and optimization settings with standard Language Models (SLMs) trained on the original, non-anonymized data.
• Inference Pipeline: During inference, the same anonymization procedure is applied to the query and any accompanying context. The model generates a response based on the anonymized input, and entity names are restored as a post-processing step to map the output back to standard natural language.
• Experimental Setup: The study evaluates models across multiple scales (from 135M to 3B parameters) and data sizes (2.5B, 10B, and 20B tokens). Comparisons are made against matched SLM baselines and original pretrained checkpoints.

Key Contributions

1. Epistemic Training Paradigm: The paper proposes a fundamental shift in how LLMs acquire knowledge, demonstrating that suppressing entity-linked supervision during pretraining induces a reliance on external evidence rather than parametric memory.
2. Controlled Empirical Study: Through rigorous ablation studies and matched baselines, the authors isolate the effect of entity anonymization from generic data corruption or regularization, showing that the specific structure of entity removal is the driving mechanism.
3. Comprehensive Evaluation: The work evaluates KLLMs across four distinct dimensions:
   • Closed-book factual recall (to measure parametric knowledge suppression).
   • Context-based question answering and reasoning.
   • Retrieval-grounded inference (including sparse, dense, and tool-augmented settings).
   • Calibration and selective prediction (abstention behavior).

Results

The empirical findings reveal a consistent pattern across model scales:

• Suppression of Parametric Recall: KLLMs exhibit a near-complete collapse in closed-book factual recall. On benchmarks like LAMA and SQuAD without context, KLLM accuracy drops to near-random levels (e.g., ~3-4% vs. ~35-40% for SLMs), confirming the successful removal of entity-linked factual memory.
• Improved Contextual Reasoning: When relevant information is provided as context, KLLMs match or outperform standard LLMs. They show consistent gains in factual QA, fact verification (FEVER), and hallucination detection (HaluBench), with improvements reaching up to +10.4% accuracy on hallucination detection.
• Robustness in Retrieval-Grounded Settings: In settings with imperfect or noisy evidence, KLLMs demonstrate superior robustness. They achieve relative gains of 20–25% over SLMs in retrieval-augmented QA. Notably, KLLMs exhibit higher abstention rates when evidence is insufficient, indicating better alignment with evidence quality.
• Enhanced Calibration: KLLMs show improved calibration across multiple metrics, including lower Expected Calibration Error (ECE) and Brier scores, and higher AUROC. They provide more reliable confidence signals and better separation between correct and incorrect predictions.
• Commonsense Reasoning: Contrary to the hypothesis that removing entities might harm general reasoning, KLLMs consistently outperform baselines on commonsense benchmarks (CommonsenseQA, StrategyQA, PIQA), suggesting that anonymization encourages reliance on structural and relational patterns rather than memorized associations.
• Training Dynamics: KLLMs exhibit higher pretraining loss than SLMs, indicating that the anonymized objective is more difficult and not a result of easier optimization or undertraining.

Significance and Claims

The paper claims that pretraining objectives fundamentally shape the epistemic behavior of language models. By suppressing entity-linked supervision, KLLMs are induced to rely less on parametric knowledge and more on external evidence. This suggests that pretraining-time control over knowledge acquisition can complement retrieval-augmented and tool-based systems by providing a base model that is inherently more sensitive to evidence quality and less prone to hallucination.

The authors position KLLMs not as a replacement for standard models in all contexts, but as a specialized solution for scenarios where reliability, groundedness, and the ability to abstain from unsupported answers are critical. The work demonstrates that it is possible to decouple reasoning capabilities from factual memorization, offering a pathway to more controllable and reliable language models. The observed effects persist across scaling laws and ablation analyses, supporting the conclusion that the anonymization mechanism is the key driver of these epistemic shifts.