💬 NLP

Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering

This paper demonstrates that language models, particularly those adapted for Polish, possess graded internal readouts of entity familiarity that are robust to prompt language and can be effectively steered to control refusal behaviors, though these pre-generation signals remain distinct from the policies governing final abstention decisions.

Original authors: Grzegorz Brzezinka

Published 2026-07-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Grzegorz Brzezinka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Inner Monologue of a Robot: When Does It Know It Doesn't Know?

Imagine you are asking a very smart robot a question. Sometimes, the robot knows the answer and speaks up confidently. Other times, it might make up a story because it doesn't actually know the truth, a mistake we call "hallucinating." Scientists have long wondered: Can the robot tell the difference between "I know this" and "I'm guessing" before it even starts typing?

To understand this, we need to look inside the robot's brain, which is made of layers of mathematical connections called "activations." Think of these activations like the electrical signals in a human brain when you see a face. If you see a friend, the signal is strong and specific; if you see a stranger, it's weaker or fuzzy. Researchers have found that these electrical signals often contain a hidden "confidence meter." If the robot is asked about something it has seen many times in its training data, the signal looks different than if it's asked about something totally made up. The big question is: Can we read this meter to stop the robot from lying before it says a single word?

The Polish Detective and the "Familiarity" Meter

This paper is like a detective story, but instead of solving crimes, the author, Grzegorz Brzezinka, is investigating how twelve different AI models "feel" about Polish names and places. The goal was to see if these models have a built-in "familiarity meter" that tells them how well they know an entity, and if that meter works even when the question is asked in a different language.

The Setup: A Library of Polish Names
To test this, the author built a special test set of 1,440 Polish entities. Imagine a library with four sections: athletes, cities, writers, and musicians. In each section, there are real people and places, ranging from super-famous stars (like the top 10% of most-viewed Wikipedia pages) to obscure locals (the bottom 10%). Crucially, the author also invented 240 fake names that looked and sounded exactly like real Polish names but didn't exist. This was the control group: if the robot gets confused by a fake name, it's a liar; if it knows the difference, it's honest.

Finding 1: The Meter is a Dial, Not a Switch
The first big discovery is that familiarity isn't just a light switch (On/Off). It's a dimmer switch. The study found that for models specifically adapted to Polish (like the Bielik and PLLuM families), the "familiarity score" rises smoothly as the entity becomes more popular.

The Analogy: Imagine a volume knob. For a super-famous singer, the knob is turned all the way up. For a local band, it's halfway. For a made-up name, it's at zero.
The Surprise: This "dimmer" effect was much stronger in the Polish-adapted models than in the giant, general-purpose models (like Gemma-4 or Qwen3). Even though the general models were bigger (had more "parameters"), they didn't have a better familiarity meter. It turns out that learning Polish specifically was more important for this skill than just making the model bigger. The Polish-adapted models showed a clear correlation (between 0.28 and 0.57) between popularity and their internal confidence, while the others were barely above zero.

Finding 2: The Meter Works Across Languages
The author then asked a tricky question: Is this meter reading the Polish words or the actual entity? To test this, they took a question about a famous Polish athlete and asked it in English ("Who is...?") instead of Polish ("Kim jest...?").

The Result: The meter barely blinked. The models retained 96% to 101% of their accuracy when switching languages. This suggests the robot isn't just recognizing Polish grammar; it's actually recognizing the person inside the question, regardless of the language used to ask about them.

Finding 3: We Can Hack the "Refusal" Button
This is the most dramatic part of the story. One of the models, Gemma-4-12B, had a habit of saying "I don't know" (refusing) when it was unsure. The author decided to see if they could control this behavior by "steering" the robot's brain.

The Experiment: They found a specific mathematical direction in the model's brain that represented "familiarity." By adding a tiny nudge in this direction, they could force the robot to change its mind.
The Magic:
- If they nudged the robot to feel less familiar with a famous person, the refusal rate jumped from 24% to 100%. The robot suddenly decided it didn't know the answer, even though it did.
- If they nudged the robot to feel more familiar with a fake name, the refusal rate dropped from 73% to 0%. The robot confidently started making up biographies for people who didn't exist.
The Takeaway: This proves that the familiarity signal isn't just a passive observation; it's a causal lever. If you push the "familiarity" button, the robot's decision to speak up or stay silent changes immediately.

Finding 4: The "Pre-Flight" Check
Finally, the paper asked: Can we use this meter to stop the robot from lying before it generates an answer?

The Comparison: The author compared their "one-pass" meter (which checks the question before the answer is written) against other methods that wait until the answer is finished to check for errors.
The Verdict: The pre-flight meter was excellent at spotting fake names, beating almost all other methods. However, predicting whether an answer would be factually wrong was harder. The meter worked well for the Polish models, but for the model that refused to answer (Gemma-4), the results were tricky because the model's own habit of saying "I don't know" messed up the error count.
The Big Picture: The study concludes that while the robot's brain does contain a graded familiarity signal, not all robots use it to decide whether to speak. The Polish models had the signal but never refused to answer. The Gemma model refused, but its signal was less precise. The paper suggests that we might need to build an external "gatekeeper" that reads this signal and decides for the robot whether to answer or look up the info, especially for obscure, long-tail questions.

Why This Matters

This research suggests that AI models have a hidden "gut feeling" about what they know, and we can read it. It's not just about making models bigger; it's about how they are trained. If we can teach models to trust this internal meter, or build tools that listen to it, we might be able to stop them from confidently making things up about things they've never seen. It's a step toward making AI not just smarter, but more honest about its own limits.

Technical Summary: Graded Entity-Familiarity Readouts in Language Models

Problem Statement

Language models (LMs) face a fundamental challenge when queried about entities they have not encountered: they must choose between abstaining or confabulating (hallucinating). While prior work has established that internal states contain information relevant to this choice, most detection methods operate post-generation (scoring the answer) or rely on binary distinctions (known vs. fabricated). This leaves open critical questions regarding the nature of the internal signal: Is it a binary bit or a graded quantity? Is it robust to the language of the prompt? Is the link between familiarity and refusal causal or merely correlational? Finally, can a pre-generation signal competitively gate generation to prevent hallucinations?

Methodology

The authors propose a unified measurement protocol applied at the "prompt point"—the final token of the input question, before any answer generation occurs. The study utilizes a new dataset of 1,440 Polish entities spanning four domains (athletes, cities, writers, musicians) and ten popularity deciles (based on Wikipedia pageviews), plus 240 fabricated control entities.

Experimental Design:

Models: Twelve instruction-tuned models from four families: Bielik (Polish-native), PLLuM (Polish continual pretraining on Llama-3.1 and Mistral-NeMo bases), Gemma-4, and Qwen3. These families span a "Polish-adaptation axis."
Readout Signals: Three signal classes are measured in a single forward pass:
1. Dispersion: Unsupervised metrics (inverse participation ratio, Shannon entropy) on MLP activation vectors.
2. Probe: A supervised logistic regression classifier on residual-stream hidden states.
3. First-token Entropy: Baseline entropy of the first generated token.
Research Questions (RQs):
- RQ1 (Language): Does the signal transfer from Polish to English prompts?
- RQ2 (Gradation): Does the score track continuous popularity?
- RQ3 (Causality): Can steering the familiarity direction causally alter refusal behavior?
- RQ4 (Application): Can the signal gate generation (abstain/retrieve) competitively with post-generation detectors?

Key Contributions and Results

1. Graded Familiarity and Polish Adaptation (RQ2)

The study demonstrates that familiarity is a graded quantity, not a binary bit.

Gradation: In Polish-adapted families (Bielik and PLLuM), familiarity scores correlate strongly with entity popularity (Spearman $\rho$ ranging from 0.28 to 0.57). In contrast, multilingual, non-Polish-centric models (Gemma-4 and Qwen3) show weak or negligible correlation ( $\rho \le 0.11$ ), despite having comparable or larger parameter counts.
Adaptation vs. Scale: The ability to track popularity is more strongly associated with Polish adaptation than with model scale. Controlled comparisons of PLLuM models against their non-Polish base architectures (Llama-3.1 and Mistral-NeMo) show that continual pretraining on Polish text significantly increases the gradation of the familiarity signal.
Behavioral Mirror: The internal gradient mirrors behavioral correctness in Polish-native models (Bielik-11B correctness rises from 6% to 68% across popularity deciles), whereas non-adapted models (Qwen3-14B) confabulate across the board.

2. Cross-Language Robustness (RQ1)

The familiarity signal is robust to the surface form of the question stem.

Transfer: Probes trained on Polish questions retain 96–101% of their within-language AUROC when applied zero-shot to English questions about the same entities (and vice versa) for Bielik and Gemma-4 models.
Entity Disjoint Transfer: When tested on held-out entities, the Polish-adapted family (Bielik) maintains 98–100% transfer performance, while the multilingual Gemma-4 degrades to 74–93%. This suggests the signal is tied to entity recognition features rather than template-specific language quirks.

3. Causal Steering of Refusal (RQ3)

Using Gemma-4-12B (the only model in the set that natively refuses), the authors demonstrate a causal link between the familiarity signal and refusal behavior.

Intervention: By adding a rank-one familiarity direction (derived from the difference between known and unknown/fabricated activations) to the residual stream at a specific layer (Layer 30), the authors can monotonically steer refusal rates.
Dose-Response:
- Pushing known entities toward "unfamiliar" ( $\alpha < 0$ ) increases refusal from 0.24 to 1.00.
- Pushing unknown/fabricated entities toward "familiar" ( $\alpha > 0$ ) suppresses refusal from 0.73 to 0.00, causing the model to fluently confabulate.
Separation: A separate "refusal direction" exists at a later layer (Layer 44) that can suppress refusal without altering the familiarity signal, indicating a dissociation between the representation of familiarity and the policy that converts it into abstention.

4. Pre-Generation Gating (RQ4)

The calibrated familiarity probe serves as an effective pre-generation gate.

Discrimination: The probe outperforms all adapted post-generation baselines (D2HScore, EigenTrack, MIND) and semantic entropy in distinguishing real from fabricated entities (AUROC 0.859–0.934 vs. baselines $\le 0.717$ ).
Cost Efficiency: As a one-forward-pass signal, it allows routing decisions (answer vs. retrieve vs. abstain) before any generation cost is incurred.
Limitations in Behavioral Prediction: While the probe predicts entity existence well, its ability to predict behavioral error (hallucination) is model-dependent. It performs well on models where behavior tracks knowledge (Bielik, Qwen3) but appears to invert on Gemma-4 because that model's refusal policy (which the probe does not control) is conflated with correctness in the evaluation metric.

Significance and Claims

The paper claims to establish a graded pre-generation entity-familiarity readout that is robust to prompt language and causally potent for steering refusal.

Representation vs. Policy: A key finding is the dissociation between representational familiarity (which exists in all families, but is graded only in Polish-adapted ones) and the abstention policy (which is present only in Gemma-4). Polish-adapted models possess the internal signal to know what they don't know but lack the policy to act on it; Gemma-4 has the policy but a less graded signal.
Deployment Implication: A calibrated, one-pass probe can supply the missing "gate" for models that possess the signal but lack the policy, routing long-tail queries to retrieval before generation begins.
Scope: The authors remain modest, noting that the causal link between the natural prompt-time computation and the refusal decision requires further mediation-style evidence, and that the current steering results are specific to one model and domain. The study does not claim to solve hallucination universally but provides a mechanism to detect and gate entity familiarity in specific contexts.