💬 NLP

Text Corpora as Concept Fields: Black-Box Hallucination and Novelty Measurement

This paper introduces "Concept Fields," a black-box, corpus-attributable framework that models text as local drift fields in embedding space to provide interpretable, probabilistic scores for detecting hallucinations and measuring novelty without relying on model internals.

Original authors: Nicholas S. Kersting, Vittorio Castelli, Chieh Ting Yeh, Xinzhu Wang, Saad Taame

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicholas S. Kersting, Vittorio Castelli, Chieh Ting Yeh, Xinzhu Wang, Saad Taame

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of books, but instead of reading the words, you turn every sentence into a unique coordinate on a giant, invisible map. In this map, sentences that mean similar things are close together, and sentences that mean different things are far apart.

This paper introduces a new way to check if a sentence written by an AI (or a human) makes sense based on that library. They call this map the "Concept Field."

Here is how it works, broken down into simple analogies:

1. The "River of Ideas" (The Concept Field)

Imagine the library isn't just a static pile of books, but a flowing river.

The Flow: If you read a sentence about "buying a car," the river naturally flows toward the next likely sentence, like "driving it home."
The Current: The authors created a system that maps the "current" of this river. They don't just look at one sentence; they look at the change (the delta) between one sentence and the next.
The Map: They built a database (called a VSDB) that stores these sentences and the "current" that connects them. This creates a field where every point has a direction and a speed, representing how ideas usually flow in that specific library.

2. The "Compass Check" (Scoring Hallucinations)

Now, imagine an AI writes a new story. How do we know if it's lying (hallucinating) or staying true to the facts?

The Test: You take the AI's sentence and see if it follows the "current" of the river.
The Score (Zeta): The system calculates a score called Zeta. Think of this like a compass deviation.
- Low Score: The AI's sentence flows perfectly with the river. It's "grounded" (true to the source material).
- High Score: The AI's sentence is swimming upstream or jumping out of the river entirely. It's "ungrounded" (a hallucination or a wild creative leap).
The "Unsure" Zone: Sometimes the river is foggy or the map is incomplete. The system can say, "I'm not sure," rather than forcing a wrong answer. This is a "triage" system: Grounded, Ungrounded, or Unsure.

3. The "Physics Lab" (The 2D Ballistics Example)

To prove their idea works, the authors ran a simple test using physics, not text.

The Setup: They simulated 1,000 balls being thrown into the air at different angles. They mapped the path of every ball.
The Result: They created a "wind map" (the Concept Field) showing where the balls usually go.
The Test: When they threw a ball that followed the same physics, the map said, "Yes, that fits!" When they threw a ball with "air resistance" (a different physics rule), the map screamed, "No! That doesn't fit the pattern!"
The Point: This proved their math works for text, too. If the "flow" of ideas changes, the system detects it.

4. Two Different Libraries, One Tool

The authors tested this on two very different libraries:

The Law Library (CFR): A collection of US federal regulations. Here, "flowing against the current" is bad (it's a lie). They used the system to catch AI hallucinations in legal text.
The Story Library (Project Gutenberg): A collection of classic novels. Here, "flowing against the current" can be good (it's creativity). They used the system to detect when an AI was writing something new and creative, rather than just copying old stories.

The Magic: The same math worked for both. A high score meant "different from the library" in both cases. In law, that's a warning; in stories, that's a feature.

5. Why This is Special (The "Black Box" Advantage)

Most AI detectors try to peek inside the AI's brain (white-box) or ask the AI to judge itself (which is often unreliable).

This method is "Black Box": It doesn't care how the AI works. It only looks at the output and compares it to the library.
It's Fast and Cheap: It doesn't need expensive supercomputers. It just needs a standard computer and the library data.
It's Traceable: If the system says a sentence is a hallucination, it can point to the specific sentences in the library that prove it. It's like saying, "You said X, but the library says Y, and here is the page number."

6. Finding Hidden Patterns (The "Whirlpools")

The authors also looked at the shape of the river.

Sources and Sinks: They found places where many ideas seem to start (like a fountain) or end (like a drain).
Whirlpools: They found places where ideas seem to spin around a central topic without moving forward.
Why it matters: These aren't just numbers; they represent hidden patterns in how humans talk and write. For example, a "whirlpool" might be a topic people talk about a lot but never seem to resolve.

Summary

The authors built a map of how ideas flow in a specific collection of text. They created a compass that tells you if a new sentence is following that flow or drifting away.

If you are checking facts, a drift means a lie.
If you are checking creativity, a drift means a new idea.
It works fast, doesn't need to know how the AI thinks, and can explain why it made a decision by pointing to the original text.

Technical Summary: Text Corpora as Concept Fields

Problem Statement
Current methods for detecting LLM hallucinations or assessing groundedness often rely on white-box model internals, grey-box perplexity metrics, or black-box approaches that require heavy calibration (e.g., LLM-as-judge). These existing solutions are frequently expensive, architecture-dependent, or lack operational interpretability. Furthermore, many geometric approaches in embedding space are limited to pairwise comparisons (e.g., DTW) or lack a mechanism to account for local uncertainty within a corpus. The authors propose a shift from viewing hallucination as a binary failure to a broader concept of groundedness: whether an output is supported by a target corpus. This framing applies to both compliance (where deviation is negative) and creativity (where deviation is positive).

Methodology
The core contribution is the Concept Field, a local drift field with pointwise uncertainty estimated in sentence-embedding space. The methodology proceeds as follows:

Vector Sequence Database (VSDB): The authors introduce a storage layer that holds fixed-dimensional sentence embeddings alongside sequence-position metadata and "next-delta" information ( $\Delta_{i,j} = v_{p+1} - v_p$ ). This allows the system to treat a corpus not just as a bag of sentences, but as a collection of trajectories in embedding space.
Concept Field Construction: Using the VSDB, the authors compute a local probability density of sentence transitions. By treating the corpus as a "vector field," they define a deterministic drift where the "flow" represents the most likely next concept given a current sentence. This field is estimated using Inverse Distance Weighting (IDW) interpolation over the nearest neighbors in the embedding space.
Scoring Mechanism ( $\zeta$ ): To evaluate a candidate sentence transition, the system calculates the mean absolute z-distance ( $\zeta$ $ζ$ ) between the observed delta and the local Gaussian estimate of the Concept Field.
- A small $\zeta$ indicates statistical agreement with the corpus flow (grounded).
- A large $\zeta$ indicates a departure from the corpus flow (ungrounded or novel).
- The score is fully black-box (requiring no model internals), corpus-attributable (tracing back to specific corpus sentences), and admits a direct probabilistic interpretation.
Geometric Analysis: The authors explore the divergence and curl of the Concept Field on dense clusters. High divergence indicates "logic sources" (common starting points) or "logic sinks" (common conclusions), while high curl suggests "implicit topics" or regions where vectors circulate around a central concept without a clear linear flow.

Key Contributions

Concept Field via VSDB: A corpus-level representation of sentence-sequence flow with local uncertainty, implemented through a novel Vector Sequence Database.
Selective Groundedness Detection: A triage policy (Grounded / Ungrounded / Unsure) based on $\zeta$ thresholds. This approach achieves strong error rates on accepted cases while maintaining a "unsure" bucket for ambiguous transitions.
Cross-Domain Robustness: The same mechanism supports both compliance-style groundedness checks (detecting hallucinations in legal text) and novelty-style deviation checks (detecting creative departures in literature) without re-calibration of thresholds.
Corpus Geometry: Qualitative evidence that topological features (divergence/curl) in the Concept Field correspond to meaningful semantic patterns, such as common reasoning flows or avoidance regions.

Experimental Results
The authors evaluated the approach on two large-scale settings:

CFR (Code of Federal Regulations): A factual/legal corpus used to test hallucination detection. The system was tested on LLM-reworded sentence pairs, distinguishing between those that preserved meaning (grounded) and those that changed semantics (ungrounded).
Project Gutenberg: A general literature corpus used to test novelty detection, where "ungroundedness" (high $\zeta$ ) is a desired trait for creative writing.

Performance Findings:

Selective Classification: Under a triage policy (e.g., $\zeta_{low}=1, \zeta_{high}=3.75$ ), the method achieved an F1 score of 0.978 and an AUC of 0.984 on the CFR dataset with ~56% coverage (44% of examples marked "unsure").
Baseline Comparison: The Concept Field outperformed or matched baselines that used simple nearest-neighbor distance (Euclidean or Cosine) on the VSDB. Crucially, while distance-based baselines required corpus-specific threshold tuning, the Concept Field maintained consistent performance and threshold interpretation across both the CFR and Gutenberg domains.
Coverage-Risk Behavior: The method demonstrated similar coverage-risk trade-offs across both domains, supporting the claim that the probability-based interpretation of $\zeta$ transfers effectively between factual and creative corpora.
Geometric Insights: Qualitative analysis of divergence and curl revealed semantically coherent clusters (e.g., "GOOOOAAALL!!" as a logic sink in sports commentary, or specific regulatory phrasing as logic sources/sinks in CFR).

Significance and Claims
The paper claims that Concept Fields provide a fast, lightweight, and interpretable signal for groundedness and novelty that is complementary to LLM-as-judge and white-box detectors.

Interpretability: Unlike black-box judges, every score traces back to nearby corpus sentences, offering explainability.
Efficiency: Once indexed, the method is faster and cheaper (no API token usage) than LLM-based evaluation, running on commodity CPUs.
Language Agnosticism: The approach is not tied to specific model architectures, making it applicable to various world languages.
Modesty: The authors frame the geometric analysis (divergence/curl) as "hypothesis-generating" rather than a definitive quantitative result. They acknowledge that the quality of the Concept Field is data-dependent, noting that logically consistent corpora (like laws) yield more reliable fields than noisy internet data. The work is presented as a foundational step, leaving room for the community to improve embedding schemes and interpolation methods.