💬 NLP

Latent Knowledge as a Predictor of Fact Acquisition in Fine-Tuned Large Language Models

This study demonstrates that latent knowledge—biomedical facts stored in a model's weights but initially inaccessible—is the strongest predictor of how quickly large language models acquire new facts during fine-tuning and their ability to generalize to unseen ontology mappings, while also revealing that previously correct mappings are more prone to degradation when not reinforced during training.

Original authors: Daniel B. Hier, Tayo Obafemi-Ajayi

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel B. Hier, Tayo Obafemi-Ajayi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (like the one used in this study) as a giant, chaotic library that was built by reading millions of books. Before the researchers started their experiment, this library was already full of facts, but they were stored in a messy way. Some facts were written in bright, bold letters on the front cover (easy to find), while others were scribbled in invisible ink in the back of a dusty book (hard to find, but still there).

The researchers wanted to know: If we give this library a specific "study guide" (fine-tuning), how fast will it learn new facts? And what determines whether it learns them quickly or forgets old ones?

Here is the story of their findings, broken down into simple concepts:

1. The "Invisible Ink" Discovery (Latent Knowledge)

The researchers discovered that the library had a secret. Some facts were written in "invisible ink." If you asked the library a question normally, it would say, "I don't know." But if you asked it in a slightly different, more random way (using a technique called stochastic decoding), the library would suddenly say, "Oh! I actually do know that!"

They called this Latent Knowledge. It's like a student who knows a math formula but is too nervous to write it down during a test. The knowledge is there, just hidden.

The Big Finding:

Facts with "Invisible Ink" were learned super fast. When the researchers started the "study guide" (fine-tuning), the model learned these facts almost immediately. It was like the student just needed a little nudge to remember what they already knew.
Facts with NO "Invisible Ink" were slow learners. If the library had never seen a fact before (or it was very rare), the model had to learn it from scratch. This took much longer and happened at a slower pace.

2. The Three Outcomes of Studying

The researchers watched the library over 20 "study sessions" (epochs) and tracked three specific things:

Memorization (Learning what you see): The model learned the facts it was explicitly taught.
- Result: It learned about 72% of the facts it was taught. But again, the ones it "half-knew" (latent knowledge) were learned first and fastest.
Generalization (Guessing the unseen): The researchers gave the model a list of facts it had never seen before and asked if it could figure them out just by studying the similar ones.
- Result: This was very rare (only about 6% of the time). However, the model only succeeded at guessing these new facts if it already had that "invisible ink" (latent knowledge) about them. If it had zero prior exposure, it couldn't guess the new facts.
Degradation (Forgetting what you knew): This is the scary part. Sometimes, while the model was studying new things, it forgot facts it knew perfectly well at the start.
- Result: The model was much more likely to forget facts it had not been taught during the study sessions. The facts it was taught were protected, like a shield. It's like studying for a history test and accidentally forgetting your own phone number because you were so focused on the new material.

3. The "Popularity" Factor

The researchers also checked if how "popular" a fact was in the real world (how often it appeared in medical books) mattered.

The Surprise: How often a specific word appeared didn't matter much.
The Real Winner: How often the identifier (the specific code for the fact) appeared mattered a lot.
The Champion: The "Invisible Ink" (Latent Knowledge) was the single strongest predictor. It mattered more than popularity or frequency.

The Takeaway Metaphor

Think of the model as a gardener and the facts as seeds.

Latent Knowledge is like a seed that has already sprouted a tiny root underground. When you water it (fine-tuning), it shoots up into a plant instantly.
No Latent Knowledge is a seed that is completely dormant. You have to water it for a long time before it even thinks about growing.
Generalization is trying to grow a plant from a seed you've never seen before. You can only do this if the soil (the model's brain) already has the right nutrients (latent knowledge) to support it.
Degradation is like the gardener focusing so hard on planting new flowers that they accidentally step on the old ones. The flowers you are currently watering (the training set) stay safe, but the ones you ignore (the unseen set) get trampled.

Summary

The paper concludes that you can't just force a model to learn anything instantly.

If the model already has a "hint" of the knowledge (latent knowledge), it learns fast and can even guess similar new things.
If the model has no hint at all, learning is slow, and it won't be able to guess new things.
To stop the model from forgetting old facts, you must keep practicing those specific facts during the training.

This helps scientists understand when to train a model and what to expect, rather than just hoping it magically learns everything.

Technical Summary: Latent Knowledge as a Predictor of Fact Acquisition in Fine-Tuned Large Language Models

Problem Statement
Large language models (LLMs) encode biomedical facts with uneven strength following pretraining, often mirroring the long-tail distribution of their training corpora. While some facts are encoded but inaccessible under deterministic decoding (termed "latent knowledge"), others are minimally represented. Current research often treats fact acquisition during fine-tuning as a binary outcome (learned vs. not learned), failing to quantify the rate of acquisition, the dynamics of generalization to unseen facts, or the mechanisms of knowledge degradation. Furthermore, it remains unclear how pretraining-derived representations, specifically latent knowledge, influence the efficiency of fine-tuning in injecting structured ontology facts (term–identifier pairs) into LLMs.

Methodology
The authors employed a survival analysis framework to model fact acquisition as a time-to-event process, treating the epoch of fine-tuning as time and the first occurrence of a correct identifier output as the event.

Model and Data: The study utilized the Llama-3.1-8B-Instruct model. Two biomedical ontologies served as structured testbeds: the Human Phenotype Ontology (HPO) and the Gene Ontology (GO).
- HPO: 800 term–identifier pairs were used as training data (seen terms).
- GO: 800 pairs were split into 400 training (seen) and 400 withheld (unseen) pairs to assess generalization.
Latent Knowledge Detection: Latent knowledge was identified via stochastic decoding (sampling at temperature 1.0). A term possessed latent knowledge if at least one of 50 stochastic samples produced the correct identifier prior to fine-tuning.
Fine-Tuning Protocol: The model was fine-tuned for 20 epochs using Low-Rank Adaptation (LoRA). For HPO, all 800 pairs were trained. For GO, only the 400 seen pairs were trained, while the 400 unseen pairs were evaluated for generalization.
Analytic Framework:
- Acquisition/Generalization: Modeled using Kaplan–Meier estimators to generate accumulation curves ( $F(t)$ ) and velocity curves ( $V(t)$ ).
- Predictors: Cox proportional hazards models were used to quantify the impact of covariates on acquisition rates. Covariates included latent knowledge status, identifier frequency in PubMed Central (PMC), term frequency in PMC, and curated annotation counts.
- Degradation: Analyzed by tracking terms that were correct at baseline (epoch 0) but became incorrect during training.

Key Contributions

Rate-Based Framework: The paper introduces a survival analysis approach to characterize fact learning, moving beyond binary accuracy metrics to quantify the velocity of acquisition, the timing of convergence, and the risk of degradation.
Latent Knowledge as a Primary Predictor: The study establishes that latent knowledge (information present in weights but inaccessible deterministically) is the strongest predictor of fact acquisition speed and generalization success.
Asymmetric Dynamics of Fine-Tuning: The work delineates three distinct outcomes of fine-tuning:
- Memorization: Rapid acquisition of seen facts, heavily dependent on latent knowledge.
- Generalization: Rare (5.8% in this study) but predictable generalization to unseen facts, occurring almost exclusively when latent knowledge exists.
- Degradation: The loss of previously correct facts, which is significantly mitigated only by continued exposure during training, not by latent knowledge.

Results

Acquisition Velocity: At baseline, the model correctly recalled 2.8% of HPO pairs. After 20 epochs, 71.9% were acquired. Terms with latent knowledge were acquired significantly faster (mean time 2.9 epochs) compared to those without (11.4 epochs). Latent knowledge increased the hazard ratio (HR) of acquisition by 2.6 ( $p < 0.001$ ).
Dose-Response Effect: A graded relationship was observed; terms with "high" latent knowledge (stochastic accuracy $\ge$ 10%) converged earlier and at higher peak velocities than those with "moderate" latent knowledge.
Generalization: Generalization to unseen GO terms was uncommon (5.8% of 379 terms). However, latent knowledge was the dominant predictor, increasing the hazard of generalization by a factor of 13.67 ( $p < 0.005$ ). Identifier frequency in PMC also positively predicted generalization (HR = 3.39).
Degradation: Of 43 GO terms correct at baseline, 55.8% degraded during training. Degradation was significantly more likely for unseen terms than for seen terms (HR = 0.10 for seen terms, indicating a protective effect of training). Notably, latent knowledge did not protect against degradation for facts that were already correct.

Significance and Claims
The paper claims that pretraining-derived structure, specifically latent knowledge, fundamentally shapes the efficiency of fine-tuning. It posits that latent knowledge narrows the model's effective search space, allowing fine-tuning to adjust existing weak associations rather than discovering new ones from scratch. Consequently, fine-tuning is most efficient for injecting ontology facts where the model already possesses "half-knowledge."

The authors conclude that this framework offers a principled method for:

Identifying "ontology deserts" (terms with no latent knowledge) where fine-tuning is likely data-inefficient.
Predicting which unseen facts are amenable to generalization.
Designing curricula that prioritize the reinforcement of clinically important facts to prevent catastrophic forgetting, as resistance to degradation relies on training exposure rather than pretraining salience.

The study emphasizes that these findings are specific to the observed learning curves and do not claim to explain the internal mechanistic shifts (e.g., logit redistribution) driving these changes, suggesting that future work should integrate mechanistic analysis with this rate-based framework.

1. The "Invisible Ink" Discovery (Latent Knowledge)

2. The Three Outcomes of Studying

3. The "Popularity" Factor

The Takeaway Metaphor

Summary

More like this