Technical Summary: Activation Dispersion Separates Entity Familiarity from Factual Reliability

Problem Statement

Large Language Models (LLMs) frequently hallucinate when queried about entities they have not encountered during pretraining. While mechanistic studies suggest that knowledge retrieval involves localized activation patterns and hallucination involves delocalized ones, it remains untested whether unsupervised, single-forward-pass dispersion statistics of internal activations can detect entity familiarity before any answer token is generated. Furthermore, it is unclear if this internal "familiarity signal" correlates with the model's behavioral factual reliability (i.e., whether the model actually answers correctly about known entities).

This study addresses these gaps using the Bielik family of Polish instruction-tuned models (1.5B to 11B parameters). It investigates whether activation dispersion can distinguish between known, real-but-obscure, and fabricated entities, and whether this signal tracks the model's ability to answer correctly.

Methodology

Dataset and Experimental Design

The authors constructed a dataset across four entity domains: athletes, cities, writers, and musicians. Each domain contains 126 entities divided into three conditions (42 per condition):

1. Known: Famous entities (e.g., Robert Lewandowski).
2. Unknown-Real: Real but obscure entities (e.g., lower-league athletes) the model likely cannot describe.
3. Fabricated: Morphologically plausible but invented names (e.g., "Roman Lewandowicz").

Prompts were one-sentence questions ("Who is {entity}?" or "What is {entity}?"). The athletes domain served as the primary testbed for behavioral analysis, while the other three domains were used to test generalization.

Signal Extraction and Metrics

For each prompt, a single forward pass was performed. The study focused on post-SwiGLU MLP activations at the prompt point (the last token of the prompt, after the entity has been processed).

• Dispersion Metrics (Unsupervised):
  • Inverse Participation Ratio (IPR): Measures the concentration of activation (high IPR = localized).
  • Spectral Entropy: Measures the entropy of the normalized activation distribution (low entropy = localized).
  • Preprocessing: Activations were winsorized at the 99th quantile to prevent massive outliers from dominating the statistics.
• Supervised Upper Bound: An $\ell_2$-regularized logistic regression probe trained on hidden states to determine the maximum linearly decodable signal.
• Baselines:
  • Token Count: A lexical baseline based on entity length.
  • Semantic Entropy: A multi-sample baseline (5 samples + LLM clustering) measuring answer consistency.
  • First-Token Entropy: Output-side entropy baseline.

Evaluation Framework

• Selection-Aware Nulls: To prevent overfitting to layer selection, the authors used permutation tests (10,000 permutations) where labels were shuffled and the best layer/metric was re-selected for each permutation. The 95th percentile of these null maxima served as the performance floor.
• Held-Out Validation: Layers were selected on half the data and evaluated on the other half.
• Behavioral Audit: For the athletes domain, an LLM judge (Claude Opus 4.8) graded answers for factual correctness under strict and soft rubrics. A separate audit (Claude Haiku 4.5) counted refusals and hedged answers.

Key Contributions

1. Unsupervised Familiarity Detection: The study demonstrates that simple, closed-form dispersion statistics (IPR and spectral entropy) computed on a single forward pass can separate known from fabricated entities with AUROC 0.94–1.00 across all model scales (1.5B–11B). This signal clears selection-aware permutation floors (~0.70–0.74) and survives held-out layer selection.
2. Generalization Across Domains: The detector generalizes across athletes, cities, writers, and musicians. Cross-domain probe transfer yields mean off-diagonal AUROCs of 0.92–0.99. Performance drops observed when transferring to the "cities" domain were shown to be caused by prompt template differences ("What is" vs. "Who is"), not entity type, via a matched-template counterfactual.
3. Dissociation of Scaling Curves: The paper identifies two distinct scaling behaviors:
   • Representational Familiarity: The ability to distinguish known from unknown entities is already at ceiling (AUROC > 0.94) at the smallest scale (1.5B) and does not significantly improve with scale.
   • Behavioral Reliability: The ability to answer correctly about known entities scales sharply. For known athletes, fully correct answers under a strict judge rose from 0/42 (1.5B) to 19/42 (11B).
4. Negative Result on Factual Reliability: Within the set of known entities, the dispersion metrics fail to distinguish between correct answers and hallucinations (AUROC ~0.68–0.82, often indistinguishable from baselines). The signal detects familiarity, not truthfulness.
5. Complementarity with Semantic Entropy: While one-pass dispersion outperforms semantic entropy on the "known vs. fabricated" contrast (0.94+ vs. 0.71–0.83), semantic entropy outperforms dispersion on the "correct vs. hallucination" contrast (up to 0.87). The two metrics consume different information (prompt-side familiarity vs. generation-side consistency).
6. The Abstention Gap: Despite carrying a near-perfect internal signal of unfamiliarity at all scales, the models almost never refuse to answer. In 2,520 sampled athlete answers, there were only 2 refusals and 1 hedged answer (99.88% direct assertions). This gap exists even at 11B, where the internal signal is legible.

Results

• Separability: The best unsupervised dispersion metric achieved AUROCs of 0.936 (1.5B) to 0.984 (4.5B) for known vs. fabricated athletes. The supervised probe reached 0.99–1.00, indicating the information is present in the activations.
• Control Contrasts:
  • Known vs. Unknown-Real: Dispersion achieved 0.97–1.00, proving the signal tracks familiarity rather than just "weirdness" of fabricated strings.
  • Unknown-Real vs. Fabricated: Dispersion achieved 0.70–0.82, suggesting a residual lexical or plausibility component in fabricated strings, but this is far below the main signal.
• Depth Structure: The familiarity signal is present from early-middle layers (relative depth ~0.2) and persists through the network. Spectral entropy shows a consistent "knowledge band" at relative depth [0.48, 0.77] across models.
• Attention Diffuseness: The signal is not carried by a small, fixed set of attention heads. Per-head analysis shows the familiarity signal is diffuse, with only 1–2 heads shared across the top-20 heads of all four domains.
• Behavioral Scaling: Correctness on known entities improved monotonically with scale (0 → 2 → 10 → 19 correct out of 42 under strict criteria).
• Refusal Behavior: The models rarely abstain. Only the 11B model refused twice; the 1.5B and 4.5B models never refused, despite having the same internal familiarity signal.

Significance and Claims

The paper claims that entity familiarity and factual reliability are distinct phenomena moving on different scaling curves.

• Familiarity is a low-capacity property: Small models (1.5B) already possess a legible internal representation of what they do and do not know. This is a "coarse-grained, volumetric shadow" of the entity-recognition circuitry found in mechanistic studies.
• Reliability is a high-capacity property: The ability to retrieve facts without hallucination requires significantly more parameters and data repetition.
• The Alignment Gap: The most significant finding is the near-zero-refusal regime. The models possess a "perfect" internal signal indicating they do not know an entity, yet they almost never act on it behaviorally. The authors argue this is an alignment/tuning gap rather than a capability gap, as larger models (and different families like Gemma-4) can be induced to refuse under similar conditions.

The paper concludes that while activation dispersion provides a cheap, label-free gate for detecting unfamiliar entities (useful for routing to retrieval or abstention), it cannot be used as a hallucination detector for content about familiar entities. The internal state tells the model "I don't know this entity," but it does not tell the model "I am about to lie about this known entity."