💬 NLP

Floor, Ceiling, and the Fusion Gap: How Much of Crowd Reading Attention Can Machines Predict?

This paper establishes the performance bounds for predicting crowd-highlighted text in web documents, demonstrating that while frontier language models capture only a fraction of the signal, an unweighted fusion of diverse models significantly outperforms single models by leveraging document-level structure, a gain that is efficiently retained by a distilled student model.

Original authors: Kazuki Nakayashiki, Keisuke Watanabe

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kazuki Nakayashiki, Keisuke Watanabe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read a book the way a human does. You want the robot to know which sentences are the "good parts" so it can highlight them for you. This is a huge challenge in the world of Artificial Intelligence, specifically in a field called "Information Retrieval" or "Natural Language Processing." The big question isn't just "Can the robot guess?" but "How good can it possibly get?"

To answer this, scientists need two things: a Floor and a Ceiling. The Floor is the absolute worst, dumbest guess you could make—like just highlighting the first few sentences of a page because that's where people usually start reading. The Ceiling is the theoretical limit of perfection. It represents the best possible score anyone could ever get, even if they had magic powers, because human readers don't all agree on what is "good." If a robot gets a score of 100, but humans only agree with each other 80% of the time, the robot can't actually be perfect; the ceiling is lower than 100.

This paper is a detective story about finding that Floor and Ceiling for a specific task: predicting which sentences a crowd of regular people will highlight in web articles. The researchers wanted to know: Are our smartest AI models close to human perfection, or are they still stumbling in the dark? And if they aren't there yet, what is the cheapest, easiest way to get them closer?

The Great Highlighting Heist

The researchers set up a massive experiment using 120 real web articles. They looked at how a "crowd" of unpaid, untrained readers highlighted sentences for their own enjoyment. Then, they built a scoreboard to see how well different AI models could predict those highlights.

First, they established the Floor. They tried the dumbest strategy possible: just highlighting the top sentences (the "lead"). Surprisingly, this simple trick worked pretty well, scoring a 0.2410. This makes sense because most articles put the most important info at the start.

Next, they built the Ceiling. They split the crowd of readers in half. They asked: "If half the crowd tried to predict what the other half would highlight, how well would they do?" This is the best anyone could possibly do, because it accounts for the fact that humans are messy and don't always agree. The ceiling score was 0.4437.

The gap between the Floor (0.2410) and the Ceiling (0.4437) is the "Headroom"—the space where AI can actually improve. The gap is 0.2028. The big question was: How much of this gap can current AI fill?

The Results: AI is Halfway There

The researchers tested five of the world's most advanced AI models (the "frontier models"). They found that the best single model could only fill about 53% of the gap. In other words, the task is roughly half-solved. The AI is much better than the dumb "first sentence" guess, but it still has a long way to go to match the collective wisdom of the crowd.

Here is where it gets interesting. The researchers tried to figure out why the AI wasn't doing better.

Is it just about position? They tried to teach a simple computer program to guess based only on where a sentence is and how long it is. It managed to recover only 5% of the gap. This proves that the missing piece isn't just about "where" the sentence is; it's about what the sentence actually says (the semantics).
Is it about compressing text? They tried a fancy tool designed to shrink text (LLMLingua-2) to see if it could find the highlights. It failed miserably, scoring even worse than the dumb "first sentence" guess. This means that trying to summarize or compress text doesn't help an AI understand what humans find interesting.

The Magic of the "Fusion"

So, if one smart AI isn't enough, what if you ask five different ones?

The researchers created a Fusion. They took the rankings from five different top-tier AI models and averaged them together. They also added a tiny nudge to favor the beginning of the article (the "position prior").

The result? The Fusion scored 60% of the gap. This is a significant jump.

It beats the best single model: The Fusion beat the single best AI by a small but statistically solid margin.
It doesn't need the "star": Even if you remove the best-performing AI from the group and just average the other four, the group still beats the single best model. This suggests that different AIs make different mistakes, and when you average them out, those mistakes cancel each other.
It's not a fluke: The team ran this experiment again on a completely new set of 217 documents (a pre-registered replication). The Fusion still won, confirming the result wasn't a lucky accident.

The "Distillation" Surprise

Finally, the researchers asked: "Can we teach a smaller, cheaper AI to do this?" They took the "wisdom" of the five-model Fusion and tried to teach it to a single, smaller AI model (an 8-billion parameter model).

They tried two types of students:

The Local Student: A model that only reads a sentence and its immediate neighbors. This student only kept 63% of the Fusion's advantage.
The Whole-Document Student: A model that reads the entire article at once. This student kept 90% of the Fusion's advantage and performed just as well as the strongest single AI model.

This tells us something profound: The "signal" of what humans find interesting lives in the structure of the whole document, not just in local sentences. To be a good highlighter, an AI needs to see the big picture.

The Takeaway

This paper doesn't claim that AI has "solved" reading. Instead, it gives us a realistic bracket:

The task is about half-solved. We aren't there yet.
The unsolved half is about meaning, not position. Simple tricks won't work; the AI has to understand the text.
The cheapest upgrade is to ask multiple models. If you want the best results right now, don't just use one AI. Ask five different ones and average their answers. It's a simple trick that beats the best single model.

The authors also warn that as AI models get smarter and start agreeing with each other more, this "fusion advantage" might shrink. But for now, if you are building a "Highlight AI," the recipe is simple: don't rely on just one brain; use a committee.

Technical Summary: Floor, Ceiling, and the Fusion Gap

Problem Statement
Current benchmarks for predicting human reading attention (specifically, which sentences a crowd of readers will highlight) lack interpretability because they report scores in isolation. Without establishing a "floor" (the performance of a trivial baseline) and a "ceiling" (the theoretical maximum performance given the noise in the human labels), it is impossible to determine how much of the task is actually solved. Furthermore, prior research on this specific corpus has shown that individual language models agree with each other far more than they agree with individual human readers, and that same-model ensembling fails to close the gap to human performance. This paper addresses the aggregate "crowd" signal rather than individual reader behavior, asking: how much of the crowd's attention map can machines predict, and what is the most efficient way to improve that prediction?

Methodology
The study utilizes a naturalistic corpus of 120 English web documents, each annotated by a median of 19 unpaid, uninstructed readers who highlighted text for their own purposes without seeing others' marks.

Ground Truth Construction: For each document, readers are randomly split into two halves (A and B). The label is defined as the top 15% of sentences in Half B, ranked by the count of distinct readers who highlighted them.
The Floor: A truncation baseline using sentence position (the "lead" heuristic), as crowd highlights are front-loaded.
The Ceiling: A "split-half oracle" where Half A's sentence counts predict Half B's label. This represents the maximum achievable score given the inherent noise of the human label itself.
Evaluation Metric: Average Precision (AP), with all comparisons paired at the document level and corrected for domain clustering (74 domains) using Holm-adjusted permutation tests.
Model Arms: Five frontier models (GPT-5.4, GPT-5.5, Claude Sonnet 4.5, Gemini 3.1 Pro, Gemini 3.6 Flash) provided full sentence rankings. These were converted to normalized Borda scores to ensure output granularity matched the floor and ceiling.
Fusion Strategy: An unweighted cross-vendor fusion of the five model rankings, optionally combined with a position prior.
Validation: The primary findings were subjected to multiple robustness checks (ablation, sensitivity to label thresholds, prompt paraphrasing) and confirmed via a pre-registered replication on an independent corpus of 217 documents.

Key Results

The Bracket: The task is roughly half-solved. The gap between the floor (truncation) and the ceiling (split-half oracle) is +0.2028 AP. No single frontier model reaches 55% of this "headroom."
Semantic Nature of the Gap: Surface features (position, length) recover only 5% of the headroom. The unsolved portion is semantic, requiring models to read and understand the sentences.
Fusion Superiority: An unweighted fusion of the five frontier models plus a position prior achieves 60% of the headroom. This significantly outperforms the best single model (Gemini 3.1 Pro) by +0.0159 AP (Holm-adjusted $p=0.019$ ). Crucially, this gain persists even when the best model is excluded from the fusion, indicating the advantage comes from the diversity of the ensemble rather than a single dominant model.
Failure of Compression: A state-of-the-art prompt compressor (LLMLingua-2) performed below the floor, scoring indistinguishably from random selection, demonstrating that optimization for compression does not transfer to predicting human attention.
Distillation and Context: Distilling the fusion into a single 8B model reading the whole document retains 90% of the fusion's edge and achieves statistical parity with the strongest single frontier model. In contrast, a local-context student (reading only ±2 neighbors) retains only 63% of the edge. This suggests the crowd's signal resides in document-level structure rather than local sentence features.
Replication: The primary finding (fusion beats best single model) was confirmed on an independent corpus of 217 documents with a pre-registered protocol, yielding a gain of +0.0179 AP ( $p=0.042$ ).

Significance and Claims
The paper claims to provide a "bracket, not a number" for evaluating highlight AI. It establishes that:

The task is not saturated; there is substantial room for improvement.
The unsolved half of the task is semantic, not positional.
The cheapest known improvement is to average the rankings of several different models (heterogeneous fusion) rather than relying on a single model or same-model ensembling.
The residual disagreement between models from different vendors is signal, not noise, consistent with the finding that models from different families err on different sentences.
The "fusion gap" is a depreciating asset; as models converge in capability and agreement, this specific ensemble advantage is expected to shrink.

The authors emphasize that their findings are modest and specific to the constructed bracket: the task is roughly half-solved, and the path forward involves leveraging document-level structure through diverse model ensembles or distillation into large-context models.