💬 NLP

Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model

This paper demonstrates that while Classical Chinese language models possess internal knowledge allowing them to distinguish known from unknown facts, they fail to externally express this uncertainty in their generated text, indicating that metacognitive capabilities like saying "I don't know" do not emerge from language modeling alone but require explicit training signals.

Original authors: Jiuting Chen, Yuan Lian, Hao Wu, Tianqi Huang, Hiroshi Sasaki, Makoto Kouno, Jongil Choi

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiuting Chen, Yuan Lian, Hao Wu, Tianqi Huang, Hiroshi Sasaki, Makoto Kouno, Jongil Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student who has memorized every book in a massive library. This student can recite history, poetry, and philosophy with perfect grammar and style. But there's a catch: this student has never been taught how to say, "I don't know."

That is essentially what this research paper discovered about Artificial Intelligence (specifically, Large Language Models).

Here is the breakdown of their experiment and findings, explained through simple analogies.

1. The Experiment: The "Pure" Scholar

The researchers built a special AI model from scratch. Instead of feeding it the messy internet (which has English, numbers, and modern slang), they fed it only 1.5 billion words of Classical Chinese (the ancient, literary language of China).

The Goal: To see if an AI, just by reading and predicting the next word in a sentence, naturally learns to admit when it is guessing or making things up.
The Test: They asked the AI about real historical events (e.g., "Did General Huo Qubing march in 121 BC?") and fake ones (e.g., "Did General Huo Qubing invent the airplane in 121 BC?").

2. The Internal Secret: The AI Knows It's Wrong

Inside the AI's "brain" (its mathematical calculations), it actually does know the difference between truth and fiction.

The Analogy: Imagine the AI is a musician playing a song. When it plays a real historical fact, the music flows smoothly. When it tries to play a fake fact, the music hits a sour note. The AI's internal "perplexity" (a measure of confusion) jumps up significantly when it encounters a lie.
The Result: The AI's internal alarm bells were ringing loud and clear. It knew, mathematically, that the story about the "flying airplane" didn't fit the pattern of history.

3. The External Lie: The AI Says It's Confident

Here is the shocking part: Even though the AI knew it was confused, it never said so.

The Analogy: Imagine that sour note from the musician. Even though the music sounded wrong, the musician kept playing with a huge smile and a confident bow, pretending everything was perfect.
The Result: When asked about fake events, the AI didn't say, "I'm not sure" or "This sounds made up." Instead, it confidently wrote a beautiful, grammatically perfect story about the flying airplane, acting as if it were 100% true.

4. The "Humility Paradox"

The researchers found something even stranger. The AI was actually more likely to say "I don't know" when talking about things it did know, compared to things it didn't know.

Why? In Classical Chinese literature, it was a polite custom for scholars to say, "This humble servant does not know..." before giving a very confident answer.
The Analogy: The AI learned the script of humility, not the feeling of uncertainty. It thought, "Oh, I'm talking about a famous general? I should use the polite 'I don't know' phrase!" But when asked about a totally fake event (which didn't match any polite script in its training), it just made up a story confidently.
The Takeaway: The AI was mimicking the words of uncertainty, not actually feeling uncertain.

5. The Universal Truth: It Happens Everywhere

To make sure this wasn't just a quirk of ancient Chinese, they tested the same idea on English (using GPT-2) and Japanese models.

The Result: The pattern was identical.
- English models (trained on Reddit/web text) would confidently make up facts about Mars populations.
- Japanese models (trained on Wikipedia) would confidently explain time travel.
- None of them naturally learned to say, "I don't know," just by reading books.

The Big Conclusion: "The Learned Scholar Without Self-Awareness"

The paper argues that being smart and being self-aware are two different things.

Language Modeling is like teaching a parrot to speak. The parrot can learn to say "I don't know" if it hears humans say it often enough, but it doesn't actually understand what ignorance means.
Self-Awareness (Metacognition) is the ability to look at your own thoughts and say, "Wait, I'm guessing here."

The Final Verdict:
AI models, no matter how big or how many languages they speak, are like brilliant actors who have forgotten they are acting. They can recite the script perfectly, even when the script is a lie. They only learn to say "I don't know" when humans explicitly train them to do so (a process called RLHF), not because they figured it out on their own.

Without that extra human training, the AI is a "Learned Scholar" who is fluent and knowledgeable but fundamentally unable to distinguish its own knowledge from its own hallucinations.

1. Problem Statement

The paper addresses a fundamental question in Large Language Model (LLM) research: Is the ability to recognize and express ignorance (metacognition) an emergent property of pure autoregressive language modeling, or does it require explicit external training signals (such as Reinforcement Learning from Human Feedback, RLHF)?

While modern LLMs (e.g., GPT-4) can say "I don't know," this behavior is typically attributed to RLHF. The authors hypothesize that without RLHF, a model may possess internal knowledge of its own uncertainty (encoded in probability distributions) but fail to externalize this state linguistically, leading to confident hallucinations that are mathematically indistinguishable from creative generation.

2. Methodology

The authors designed a controlled, multi-lingual experimental framework to isolate the effects of pure language modeling from RLHF.

A. Primary Experiment: Classical Chinese Model

Model Architecture: A 318M-parameter Transformer trained from scratch.
Training Data: A curated corpus of 1.56 billion tokens of pure Classical Chinese (文言文), containing zero English characters or Arabic numerals. Sources included historical texts (e.g., Zizhi Tongjian), Confucian classics, Buddhist sutras, and poetry.
Tokenizer: A custom SentencePiece BPE tokenizer (vocab size 16k) with byte-fallback enabled to ensure Out-of-Distribution (OOD) inputs (like English) are encoded as byte sequences rather than <unk> tokens.
Training: Trained for 15,000 iterations on a single NVIDIA RTX PRO 6000 GPU with no RLHF or fine-tuning.

B. Experimental Design

The study utilized six test categories ranging from in-distribution to fully OOD, with a specific focus on Test 5 (Factual OOD):

Real Events: Verifiable historical facts.
Fabricated Events: Plausible but fictional events using real dates/figures.
Semi-Fabricated Events: Real historical figures performing fictional actions (e.g., Su Shi presenting the art of flying).

Metrics:

Internal Uncertainty: Measured via Perplexity (PPL). A higher PPL indicates the model finds the input less probable (i.e., "knows" it is likely false).
External Uncertainty: Measured by the frequency of epistemic markers in generated text (e.g., "不知" (do not know), "未闻" (have not heard), "臣愚" (this foolish minister)).

C. Cross-Lingual Validation

To ensure generalizability, the authors replicated the experiment using:

English: GPT-2 family (124M to 1.56B parameters) trained on WebText.
Japanese: rinna models (110M to 1.3B parameters) trained on Wikipedia/CC-100.
Comparison: All models were tested on real vs. fabricated historical prompts without any RLHF.

3. Key Results

A. Internal Knowledge vs. External Expression (The Dissociation)

Internal Signal: The model successfully encodes factual knowledge. There is a significant Perplexity Jump between real and fabricated events.
- For the 318M Classical Chinese model: Fabricated events had 2.39× higher PPL than real events ( $p = 8.9 \times 10^{-11}$ ).
- Semi-fabricated events (real figure + fake action) showed the highest PPL (4.24×), indicating the model detects the mismatch between known entities and unknown contexts.
External Behavior: Despite high internal uncertainty, the model never learned to express uncertainty.
- The frequency of uncertainty markers remained flat (2–4%) across all training checkpoints, regardless of whether the input was known or unknown.
- Crucially: The model generated confident, fluent hallucinations for OOD inputs just as it did for in-distribution inputs.

B. The "Humility Paradox"

The study found a counterintuitive correlation: the model used more uncertainty markers for in-distribution questions (8.3%) than for OOD questions (3.5%).

Cause: In Classical Chinese texts, phrases like "臣愚不知" (This foolish minister does not know) are rhetorical conventions used to show humility before answering, not genuine expressions of ignorance.
Implication: The model learned the pattern of hedging for familiar topics but lacked the cognitive mechanism to apply it when it actually didn't know the answer.

C. Cross-Lingual Universality

The findings held across three languages and writing systems:

English (GPT-2): Showed no significant difference in uncertainty expression between known and unknown topics (approx. 4–5% for both).
Japanese (rinna): Showed near-zero uncertainty expression for both categories (<2%), reflecting the encyclopedic tone of Wikipedia training data.
Conclusion: The frequency of "I don't know" is determined entirely by training data conventions, not the model's epistemic state.

D. Scaling Laws

Factual Encoding: The ability to distinguish real from fake (PPL jump ratio) scales with model size (e.g., English GPT-2 jumped from 1.70× at 124M to 2.51× at 1.56B).
Metacognitive Expression: No scale threshold was found where metacognitive expression emerged spontaneously. Even the largest models (1.56B) failed to say "I don't know" when they didn't know.

4. Key Contributions

Proof of Internal/External Dissociation: Demonstrated that autoregressive models can internally "know" they are hallucinating (via PPL) while externally generating confident falsehoods.
The Humility Paradox: Identified that epistemic markers in training data are often rhetorical, causing models to hedge on known topics but hallucinate confidently on unknown ones.
Indistinguishability of Creativity and Hallucination: Argued that without external calibration, "creativity" and "hallucination" are mathematically identical processes (sampling from a probability distribution); the distinction only exists post-hoc via ground truth verification.
Cross-Lingual Validation: Confirmed that this lack of emergent metacognition is a universal property of language modeling, not specific to a language or script.

5. Significance and Implications

Role of RLHF: The paper provides direct evidence that the ability to refuse queries or express uncertainty is not an emergent property of language modeling but an externally imposed behavioral layer created by RLHF.
Safety and Reliability: Relying on a model's generated text to determine its confidence is dangerous. A model can be highly confident in a hallucination because its internal uncertainty signals are never mapped to linguistic output.
Future Directions: To build truly self-aware AI, explicit training signals (like RLHF or specific refusal training) are necessary to create a "mirror" that maps internal uncertainty states to external linguistic expressions. Pure language modeling alone produces a "learned scholar without self-awareness."