💬 NLP

A Geometric Profile of Semantic Information in Text: Frame-Conditional Uniqueness and a Trade-Off Triangle for Scalar Summaries

This paper introduces a geometric framework for measuring semantic information in text via a three-coordinate profile of novelty, breadth, and integration, proving a fundamental trade-off that prevents any single scalar summary from satisfying all desirable properties while demonstrating that a rank-normalized configuration effectively outperforms existing baselines.

Original authors: Dmitriy Kompaneets

Published 2026-06-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dmitriy Kompaneets

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of books, and you want to answer a simple question: "How much meaning is actually in this specific text?"

For decades, scientists have tried to answer this using math that counts how unpredictable the words are (like guessing the next letter in a word). But as this paper points out, that's like measuring a painting by counting how many different shades of blue are on the canvas. It tells you about the colors, but not the picture. Two paintings can have the exact same mix of colors but tell completely different stories.

This paper proposes a new way to measure meaning by looking at the "shape" of the ideas inside a text, using a tool called sentence embeddings (which turn sentences into points in a giant, multi-dimensional map).

Here is the core of their discovery, broken down into simple concepts:

1. The "Shape" of Meaning (The Profile)

Instead of trying to squeeze a whole book's meaning into a single number (like a test score), the authors say we need a three-part profile. Think of it like describing a person not just by their height, but by three distinct traits:

Novelty (The "Newness"): How far does this text wander from the "average" boring conversation? If you write a text that sounds exactly like a generic news template, its novelty is low. If you write something surprising and unique, it moves far away from the center of the map.
Breadth (The "Spread"): How many different distinct ideas are in there? Imagine a text that talks about cats, then rockets, then baking, then quantum physics. That has high breadth. A text that talks about cats for 10 paragraphs has low breadth.
Integration (The "Glue"): How well do those ideas stick together? A text that jumps randomly from cats to rockets to baking has low integration (it's a messy bag of facts). A text that weaves those topics together into a coherent story has high integration.

The Analogy:

Poetry is like a tight, beautiful knot. It has high novelty (unique words) and high integration (everything fits), but low breadth (it stays on one theme).
A chaotic dialogue is like a scattered pile of toys. It has high breadth (lots of different topics) but low integration (nothing connects).
Legal text is like a wide, flat net. It covers a lot of ground (high breadth) and is very structured (high integration), but it's not very "new" (low novelty).

2. The "Semantic Quantum" (The Pixel of Meaning)

The paper introduces a concept called the Semantic Quantum. Think of this as the "pixel" of meaning.

If you have a text with 100 sentences, but 90 of them are just rephrasing the same idea, the text isn't actually 100 ideas deep.
The authors use a "clustering" tool to group similar sentences together. If 10 sentences are almost identical, they count as one "quantum" (one idea).
This prevents the math from being tricked by repetition. It's like counting the number of unique ingredients in a soup, rather than counting every single grain of salt.

3. The "No-Go" Triangle (The Impossible Dream)

Here is the paper's most important warning. The authors tried to combine those three traits (Novelty, Breadth, Integration) back into a single number (a "Scalar") to make it easy to use.

They proved mathematically that you cannot have a perfect single number. It's a "Trade-Off Triangle." You can only pick two of the following three superpowers for your number:

Stability: If I rewrite the text slightly (paraphrase), the number stays the same.
Ranking Power: If I compare two different texts, the number correctly tells me which one is "richer."
Cross-Model Fairness: If I use a different AI tool to measure the text, the number stays comparable.

The Catch:

If you make a number that is stable (doesn't change with small edits), it becomes bad at ranking different texts.
If you make a number that is great at ranking, it becomes unstable (small changes make the score jump wildly).
If you try to make it fair across different AI tools, you lose the other two.

Because of this, the paper suggests we stop looking for one "magic number" and instead use two different tools for two different jobs:

Tool A (Sminmax): Good for theoretical math and checking if a summary is stable.
Tool B (Srank): Good for ranking documents (like a search engine) where you just need to know "which one is better," even if the exact score jumps around a bit.

4. The "Volume" Discovery

The authors found something cool about the "Breadth" part of the profile. They discovered that the "spread" of ideas in a text is mathematically identical to a concept from physics called a Determinantal Point Process (DPP).

Simple Analogy: Imagine you are picking a team for a project. You don't want three people who all think exactly the same way. You want a team where everyone brings a different perspective, but they still work well together.
The math shows that the "Breadth" of a text is essentially the volume of the best possible diverse team you could pick from that text's sentences. This gives the "Breadth" score a solid mathematical foundation, proving it's not just a guess.

Summary

The paper argues that meaning is too complex to be a single number.

It gives us a 3D map (Novelty, Breadth, Integration) to see the shape of a text.
It warns us that trying to flatten this map into one score always involves a compromise (the Trade-Off Triangle).
It provides two practical tools (one for stability, one for ranking) so we can measure text meaning in a way that actually makes sense for the job we are doing.

The authors tested this on 5 classic novels (like Moby Dick and Pride and Prejudice), 23 types of synthetic text, and various AI models, and found that this 3D approach works much better than previous methods at telling the difference between a coherent story and a random bag of facts.

Technical Summary: A Geometric Profile of Semantic Information in Text

Problem Statement

Current methods for quantifying information in text face fundamental limitations. Shannon's information theory measures uncertainty over symbol sequences but remains intentionally indifferent to semantic content; texts with identical token statistics can possess vastly different meanings. Conversely, contemporary pairwise metrics (e.g., BERTScore, ROUGE) measure similarity between two texts rather than characterizing the internal semantic richness of a single text. Furthermore, philosophical accounts of semantic information often lack computability on real-world data.

There is a need for a framework that measures semantic content based on the structure of meaning representations (embeddings) rather than syntactic unpredictability, while acknowledging that semantic information may not reduce to a single universal scalar.

Methodology

The paper proposes a geometric framework grounded in sentence embeddings. Let $T$ be a text partitioned into segments, mapped to a high-dimensional vector space via an embedding model $E$ . The framework operates through the following steps:

Deduplication and Semantic Quanta: To prevent repetition from distorting geometry, the framework clusters highly similar segments using agglomerative clustering with a cosine distance threshold $\tau$ . Each cluster is replaced by a centroid, forming a set of "semantic quanta" $\tilde{T}$ . The threshold $\tau$ acts as an explicit measurement-resolution parameter.
Frame-Conditional Uniqueness: The authors first derive a scalar measure, the Semantic Information Law (SIL), $I_E(T) = \|\mu_T - \mu_0\| \cdot \text{rank}(C_T)$ , based on six axioms (paraphrase invariance, redundancy non-increase, novelty monotonicity, idea additivity, orthogonal invariance, and continuity). However, empirical testing reveals this scalar is too coarse, as rank saturates in neural spaces and the novelty term dominates.
The Three-Coordinate Profile: To address the coarseness of the SIL, the framework defines a profile $P_E(T) = (N, B, I)$ $P_{E} (T) = (N, B, I)$ :
- Novelty ( $N$ ): The Mahalanobis distance of the text's mean embedding from a neutral baseline mean, log-compressed.
- Breadth ( $B$ ): The effective rank of the deduplicated centroid covariance, weighted by the mean radial cosine distance from the centroid.
- Integration ( $I$ ): The connectedness among ideas, measured via the second-nearest-neighbor cosine similarity (2-NN).
Scalar Summaries: Recognizing that different applications require different properties, the paper defines two practical scalars derived from the profile:
- $S_{minmax}$ : Uses min-max normalization, prioritizing analytic stability (Lipschitz continuity) for theoretical analysis.
- $S_{rank}$ : Uses rank normalization, prioritizing ordinal robustness for ranking and document comparison.

Key Contributions

Frame-Conditional Uniqueness Theorem: The paper proves that within a fixed embedding model and baseline, six natural axioms uniquely determine a scalar measure of semantic information up to a scale constant. This result establishes that semantic information is representation-indexed, not a universal law like Shannon's entropy.
The Semantic Profile and Quantum: It introduces a three-coordinate profile (Novelty, Breadth, Integration) and formalizes the "semantic quantum" as a discrete minimal unit of structure, making the resolution of measurement explicit via the clustering threshold $\tau$ .
The No-Go Theorem (Trade-Off Triangle): A central theoretical contribution is the proof that no scalar summary of the profile can simultaneously satisfy three desirable properties:
- Analytic Stability: Stability under paraphrase and closed-form composition.
- Ordinal Robustness: Robustness across text scales and bootstrap perturbations.
- Cross-Representation Comparability: Comparability across different embedding models without re-fitting.
  The theorem demonstrates that these properties are mutually exclusive; practical scalars must occupy distinct corners of this trade-off triangle.
Variational Characterization of Breadth: The paper connects the breadth coordinate to the log-determinant of a Determinantal Point Process (DPP) maximum-a-posteriori selection. This provides an optimization-theoretic foundation for breadth, showing it is equivalent to the log-volume of a diversity-maximizing subset of segments.

Empirical Results

The framework was validated across 23 synthetic categories, 5 Project Gutenberg novels, and 3 embedding models.

Ordinal Performance: The recommended configuration, $S_{rank}$ with weights $(0.5, 3.0, 1.0)$ and 2-NN integration, passed 25 of 28 ordinal checks as point estimates. After Benjamini–Hochberg correction, 21 of 28 checks remained significant. This outperformed seven baselines, including unigram entropy, raw Euclidean novelty, and a BERTScore-based novelty signal.
Trade-Off Validation: Empirical results confirmed the no-go theorem. $S_{minmax}$ achieved analytic stability but failed ordinal robustness (21/28 checks), while $S_{rank}$ achieved ordinal robustness but failed analytic stability (21% drift under paraphrase). No scalar satisfied more than one property simultaneously.
Breadth Correlation: On 507 Gutenberg chapters, the breadth coordinate $B$ showed a Spearman correlation of $\rho = 0.985$ with the DPP log-volume, validating the variational characterization.
Cross-Model Stability: The profile coordinates themselves were found to be more stable across different embedding models ( $\rho \in [0.92, 0.98]$ ) than any derived scalar ( $\rho \in [0.79, 0.84]$ ), supporting the claim that the profile is the fundamental object.

Significance and Claims

The paper argues that semantic information in text is best understood not as a single universal number, but as a representation-indexed structured profile.

Theoretical Shift: The work moves away from the search for a "Shannon-style" universal scalar for semantics. Instead, it posits that the profile is the theoretical object, while scalar summaries are practical conveniences whose specific forms are dictated by the impossibility of satisfying all operational requirements simultaneously.
Explicit Assumptions: By making the measurement apparatus (embedding model, segmentation, baseline, threshold, normalization reference) explicit, the framework allows practitioners to inspect and calibrate commitments, localizing disagreements to specific apparatus choices rather than the framework itself.
Interpretability: The three coordinates correspond to distinct semantic intuitions (departure from generic discourse, diversity of ideas, and connectedness), allowing for diagnostics (e.g., distinguishing faithful vs. off-topic summaries) that a single scalar would conflate.

The paper concludes that the trade-off triangle is a structural property of the map from profile to scalar, not a defect of any specific construction. The framework provides a rigorous, computable method for measuring semantic structure while acknowledging the inherent limitations of reducing that structure to a single value.