💬 NLP

Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering

This paper introduces Granuscore, a reference-free metric based on hierarchical embedding structures that effectively measures text granularity, validates its utility across discourse contexts, and applies it to analyze question-answering datasets and model behaviors.

Original authors: Lukas Ellinger, Alexander Fichtl, Miriam Anschütz, Georg Groh

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lukas Ellinger, Alexander Fichtl, Miriam Anschütz, Georg Groh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are describing a famous skateboarder named Tony Hawk. You could say, "Tony Hawk was born in San Diego." Or, you could say, "Tony Hawk was born in California." Or even, "Tony Hawk was born in the United States."

All three sentences are true, but they feel different. The first one is fine-grained (very specific, like a high-resolution photo). The last one is coarse-grained (broad, like a blurry map).

For a long time, computers have struggled to measure this "level of detail" automatically. They can count words or check grammar, but they can't easily tell if a sentence is being specific or vague without needing a human to look at it first.

This paper introduces Granuscore, a new tool that acts like a "detail detector" for text. Here is how it works and what the authors found, explained simply.

The "Zoom Lens" Analogy

Think of language as a giant, 3D map of the world.

Coarse concepts (like "furniture" or "the United States") are like the center of the map. They are broad and cover a lot of ground.
Fine concepts (like "a rusty wrench" or "San Diego") are far out on the edges. They are specific and detailed.

The authors used a special type of AI map (called a hierarchical embedding space) where the distance from the center tells you how specific a word is.

Granuscore measures how far out on the map a sentence's words sit.
Low Score: The words are far out on the edge = Very Specific (Fine-grained).
High Score: The words are near the center = Very Broad (Coarse-grained).

The clever part? It doesn't need a dictionary or a human to check the answer. It just looks at the "shape" of the words in this AI map.

What They Tested

The researchers put Granuscore through three main stress tests:

1. The "Sorting Game" (GRANOLA-EQ)
They gave the tool a list of sentences about the same thing but with different levels of detail (e.g., "Tony Hawk," "Skateboarder," "Sportsman").

Result: Granuscore successfully sorted them from most specific to most general, matching how humans would order them. It was much better than just counting words or using older dictionary methods.

2. The "Scientific Paper" Test
They looked at real scientific papers. They knew that the Introduction section usually talks about big, broad ideas, while the Related Work section dives into tiny, specific details of other studies.

Result: Granuscore correctly identified that the Introductions were "coarser" (higher scores) and the Related Work sections were "finer" (lower scores). It understood the context without being told what a paper looks like.

3. The "Specificity" Test
They checked if Granuscore could explain why some sentences feel more "concrete" than others, even if the sentences were the same length.

Result: Yes. A short sentence with very specific words got a "fine" score, while a long sentence with vague words got a "coarse" score. It proved that "specificity" isn't just about how many words you use; it's about what those words represent.

The Question Answering Discovery

The authors also used Granuscore to look at how AI models answer questions. They compared the questions, the correct answers, and the answers the AI actually gave.

The "Wrong Answer" Pattern: When the AI got an answer wrong, its response was often too specific (too fine-grained). It tried to guess a precise detail it didn't actually know.
The "Right Answer" Pattern: When the AI got it right, the level of detail in its answer matched the question and the correct answer much better.
The "I Don't Know" Pattern: When the AI refused to answer (abstained), it gave very broad, coarse statements (e.g., "I cannot answer this").

The Big Takeaway: The paper suggests that Granuscore can act as a "difficulty meter." If a question and its correct answer are very specific (low Granuscore), the AI is more likely to struggle or hallucinate a fake specific detail. If the topic is broad, the AI is more likely to be correct.

Summary

Granuscore is a new, automatic way to measure how "zoomed in" or "zoomed out" a piece of text is.

It doesn't need a human to grade it.
It works by looking at the "shape" of words in a smart AI map.
It helps us understand why AI sometimes fails: when it tries to be too specific about things it doesn't know, it makes mistakes.

The authors have released this tool as a free software package so others can use it to analyze text, improve AI, or study how humans communicate.

Technical Summary: Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering

Problem Statement

Natural language conveys information at varying levels of abstraction, ranging from fine-grained references (e.g., "Tony Hawk") to coarse descriptions (e.g., "a sportsman"). While granularity is fundamental to human communication and significantly impacts dialogue systems, accessibility, and reading comprehension, existing methods for measuring it are limited. Current approaches typically rely on surface-level features like sentence length or specificity, or they depend on manually constructed taxonomies (e.g., WordNet, Wikidata) which suffer from limited coverage and cannot handle free-form text effectively. There is a lack of a scalable, reference-free measure capable of quantifying granularity at the sentence level across diverse domains.

Methodology: Granuscore

The authors introduce Granuscore, a reference-free measure that leverages the structural properties of a hierarchical embedding space to estimate semantic granularity. Lower scores indicate finer-grained expressions, while higher scores indicate coarser ones.

Core Architecture

Hierarchical Embedding Space: The method builds upon the Hierarchy Transformer (HiT) model (Chen et al., 2024), which represents hierarchical structures in a hyperbolic embedding space (Poincaré ball). In this geometry, radial distance from the origin ( $Dist_0$ ) correlates with hierarchical depth: specific concepts lie farther from the center, while general concepts lie closer.
Feature Extraction:
- Global Signal: The model extracts $Dist_0$ as a primary signal for hierarchical depth.
- Relational Signals: To capture global structure beyond local similarity, the input embedding is compared against a set of anchor entities. The system computes pairwise cosine similarities and manifold distances between the input and these anchors.
Scoring Model: A gradient-boosted decision tree (LightGBM) is trained to map the raw similarity and distance features to a scalar granularity score.
Calibration: Raw scores are converted to percentile scores using a fixed calibration distribution based on the WordNet noun set (approx. 119k concepts), ensuring the scale is annotator-independent.
Multi-Word Handling: For sentences or longer texts, the system decomposes the input into referential units (noun phrases) using a SPACY-based splitter. The final sentence score is the mean of the bottom 80% of unit scores to mitigate the influence of outliers.

Training Data

The model is trained on GRANOLA-EQ, a dataset containing questions and sets of answers referring to the same entity at different granularity levels. The training targets are normalized to a continuous scale from 1 (finest) to 4 (coarsest). The training procedure enforces strict data splitting to prevent leakage, ensuring no granularity realization appears in multiple splits.

Key Contributions

Introduction of Granuscore: A novel, reference-free metric that exploits hierarchical embedding space properties to measure granularity without relying on predefined vocabularies or manual taxonomies.
Empirical and Conceptual Validation:
- GRANOLA-EQ: Granuscore reliably recovers human-annotated hierarchical orderings.
- Discourse Contexts: It captures expected granularity differences between scientific paper sections (e.g., coarser language in Introductions vs. finer language in Related Work).
- Sentence Specificity: It explains non-linear variation in sentence specificity across domains (movie reviews, tweets, news) beyond what sentence length alone can predict.
Application to Question Answering (QA): The authors analyze four QA benchmarks (FACTS PARAMETRIC, SIMPLEQA, SQUAD, TRUTHFULQA) across six language models. They identify consistent patterns where:
- Datasets with lower granularity (finer details) are harder for models.
- Incorrect model outputs tend to have lower granularity than correct outputs.
- The "granularity gap" (difference between model output and gold answer) correlates with response failure.
Reproducibility: The tool is released as a pip package to facilitate further research and production use.

Results

Granularity Ordering: On the GRANOLA-EQ test set, the best-performing variant (HiT Dist0 + Random Anchors) achieved 83.76% Global Pairwise Accuracy and 89.03% Intra-entry Pairwise Accuracy. This significantly outperforms baselines like Word Count (~50%), WordNet (limited by coverage), and standard sentence embeddings (MiniLM).
Discourse Analysis: Introduction sections in scientific papers exhibited significantly higher Granuscores (coarser) than Related Work sections (finer), aligning with rhetorical expectations ( $p \le 5.49 \times 10^{-37}$ ).
Sentence Specificity: Adding Granuscore to Generalized Additive Models (GAMs) improved the explained deviance of sentence specificity predictions by 7–50% across domains, demonstrating a non-linear relationship where lower Granuscore correlates with higher specificity.
QA Performance: A clear correlation was observed between dataset granularity and model correctness. Models performed better on datasets with higher Granuscores (coarser answers). Furthermore, models that failed to answer or answered incorrectly tended to produce outputs with lower granularity than their correct counterparts, suggesting a difficulty in adapting output granularity to confidence levels.

Significance and Claims

The paper positions Granuscore as a scalable, broadly applicable tool for analyzing text granularity. Its primary significance lies in providing a principled lens for:

Characterizing Dataset Difficulty: Granularity offers a metric for QA dataset difficulty that is distinct from superficial properties like answer length.
Analyzing Model Behavior: It reveals that models struggle to adjust answer granularity based on confidence, often producing overly fine-grained incorrect answers or failing to provide coarse, reliable alternatives when uncertain.
Future Applications: The authors suggest Granuscore could serve as a training signal to encourage models to align output granularity with their confidence (e.g., responding coarsely when uncertain) and support controlled language adaptation tasks like simplification and summarization.

The authors acknowledge limitations, noting the dependence on the WordNet hierarchy for the underlying embedding model, which may limit performance in domains poorly represented in WordNet. They also note that while empirical validation is robust across multiple settings, a dedicated human study comparing Granuscore scores directly against explicit human granularity judgments was not conducted.