💬 NLP

Measuring Alignment With Reader Highlights Net of Position and Length

This paper introduces a rigorous, non-circular metric for evaluating context compression by controlling for document position and sentence length, demonstrating that language models significantly outperform naive truncation and classical heuristics in predicting human reader highlights, while achieving performance comparable to advanced AI models and human readers.

Original authors: Kazuki Nakayashiki, Keisuke Watanabe

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kazuki Nakayashiki, Keisuke Watanabe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Text Filter: Why We Need a Better Way to Read

Imagine you are trying to teach a robot how to understand a massive library. The robot is smart, but it has a short attention span; it can only read a tiny slice of a book before it gets overwhelmed. So, before the robot reads a page, a "compressor" has to decide which sentences to keep and which to throw away. For a long time, the only way to check if the compressor was doing a good job was to ask another robot to grade the result. This is a bit like asking a student to grade their own homework; it's a circular logic trap where the judge might just be copying the student's mistakes.

To break this loop, scientists started looking at something more human: highlighting. When real people read a book, they naturally underline or highlight the parts they find most interesting or important. These highlights act as a "gold standard" of what humans actually care about, independent of any robot's opinion. However, there's a catch. Humans have bad habits too. We tend to highlight the first few sentences of a paragraph (the "lead") and we love long, dramatic sentences. If a computer program just keeps the first sentences or the longest ones, it might look like it's doing a great job matching human highlights, but it's actually just following a boring pattern, not understanding the content. The big question for this field is: Can an AI actually figure out what's important, or is it just good at guessing where the important stuff usually sits in a document?

The Paper's Big Discovery

This paper, written by researchers from Glasp Inc., sets out to answer that question with a very strict, fair test. They took 120 web documents that had been highlighted by at least 12 different people each. Their goal was to see if a modern language model (like the ones powering chatbots) could pick out the sentences that real humans highlighted, once they removed the "cheating" factors of position and sentence length.

The Main Finding
The researchers found that when you strip away the bias of "first sentences" and "long sentences," the language model still does a surprisingly good job. Specifically, the model managed to keep 38.4% of the sentences that the crowd had highlighted. In contrast, when the model looked at similar sentences that weren't highlighted (sentences of the same length and depth in the document), it only kept 19.9% of those.

This gap is the "enrichment" score. The model's score was +0.196, which is a significant jump. To put this in perspective, the researchers compared the AI to a single human reader doing the same task. A single human reader scored +0.182. The AI (specifically GPT-5.4) scored +0.184 against that same human. In other words, the AI predicted what a crowd of people would highlight about as well as a single person could. A stronger model, Claude Opus 5, even did slightly better than the single human at +0.230.

What the Paper Rules Out
The authors were very careful to prove that this wasn't just the AI cheating by following old tricks.

It's not just about position: If the AI was just keeping the first sentences, its score would drop to near zero once they fixed for position. In fact, a simple "keep the first 20%" rule scored only +0.003 after the fix.
It's not just about sentence length: The AI didn't just pick long sentences.
It's not just about word frequency: They tested an old-school method from 1958 (Luhn's heuristic) that counts how often words appear. That method recovered about half the effect (+0.088), proving that simple word counting helps, but the AI does roughly twice as well as the best simple method.
It's not about definitions: The AI didn't just pick sentences that sounded like dictionary definitions.

How Sure Are They?
The paper is very confident in these numbers because they used a "randomization test." Imagine shuffling the deck of cards (the sentences) over and over again to see if the result happens by chance. The chance of getting this result randomly was p = 0.0005. This means there is a 99.95% certainty that the AI is actually finding something real, not just guessing. They also tested this across two different AI vendors (GPT and Claude), and both got similar results, which makes the finding even stronger.

The "Gotchas" and Nuances
The paper is honest about its limitations and what it doesn't prove:

The "Compression" Tool didn't work: They tested a specific tool designed to compress text (LLMLingua-2), and it didn't show a clear signal. The authors suggest this might be because the tool was set up in a way that wasn't its intended use, so they don't claim the tool is useless, just that it didn't work in this specific test.
Prompting matters: If you ask the AI "What is important to keep?" it does much better than if you ask "What would a reader highlight?" The first question got a score of +0.158, while the second got +0.074. This shows that how you ask the AI matters a lot.
Not a perfect match: The AI isn't a magic mind-reader. It still misses a lot of highlights, and the researchers admit that their test data comes from a specific type of website (static HTML) and might not represent every kind of text in the world.

The Bottom Line
This paper suggests that modern AI models have learned to understand human interest in text beyond just simple tricks like "keep the first sentence." They can identify what humans find valuable, performing roughly as well as a single human reader. However, they aren't doing something completely magical; they are just doing about twice as well as the best old-fashioned word-counting methods. The study confirms that while AI is getting better at reading between the lines, it still has room to grow, and we need to be careful about how we ask it to summarize things.

Technical Summary: Measuring Alignment With Reader Highlights Net of Position and Length

Problem Statement
Current methods for evaluating context compression in large language models (LLMs) rely on downstream task accuracy, creating a circular evaluation loop where one model judges what "matters" for another. This paper proposes using naturalistic social highlighting—where independent readers mark passages on public web pages for their own use—as a non-circular reference for what readers choose to keep.

However, a naive metric (the fraction of crowd-marked sentences a compressor retains) is confounded by two strong biases:

Position Bias: Crowd marks are front-loaded; readers tend to highlight earlier sentences.
Length Bias: Crowd-marked sentences are significantly longer than unmarked ones.

Consequently, simple truncation (keeping the first $N$ sentences) or length-based selection achieves high scores on this metric without actually capturing reader intent. The core problem is to measure alignment with reader highlights net of these two confounds.

Methodology
The authors introduce a rigorous estimation framework designed to isolate the signal of reader choice from position and length.

Data: The evaluation set consists of 120 public web documents (78 domains), each marked by at least 12 independent readers. Sentences are labeled "crowd-marked" if they fall in the top 15% by mark count (with a minimum of two marks).
Matching Estimator: To remove confounds, the method matches each marked sentence against unmarked sentences within the same document based on:
- Relative Depth: Within $\pm 0.05$ of the marked sentence's position in the document.
- Within-Document Length Rank: Within $\pm 0.05$ of the marked sentence's length rank.
  The estimator calculates the enrichment ( $E_d$ ) as the difference between the fraction of marked sentences kept and the fraction of their matched unmarked neighbors kept.
Calibration and Null Testing: A critical contribution is the calibration of the estimator itself. The authors generate synthetic null datasets where the "keep" set is determined only by position and length (simulating a perfect confound). They find that standard depth-only stratification yields a false positive rate of 20–36% on these nulls, whereas their matched estimator maintains a false positive rate of 4.5–7.0%.
Randomization Test: Significance is assessed via an exact randomization test that reassigns treatment status within matched sets, assuming nothing about clustering.

Key Contributions

Calibration of Confound Controls: The paper quantifies the error rate of standard matching controls, demonstrating that depth-only stratification fails significantly (20–36% false positives) when synthetic confounds are present.
Robust Alignment Signal: A language model importance ranking retains 38.4% of crowd-marked sentences compared to 19.9% of their matched neighbors, yielding an enrichment of +0.196 ( $p=0.0005$ ). This result replicates across two different LLM vendors (GPT-5.4 and Claude Opus 5).
Baseline Comparison: Classical extractive methods are not null. Luhn's 1958 heuristic achieves an enrichment of +0.088, recovering roughly half the effect of the LLM. This suggests reader selection is partly recoverable by counting words, with LLMs roughly doubling the performance of the best cheap lexical method.
Prompt Framing Sensitivity: The result is highly sensitive to prompt framing. Asking a model "what is important to keep" yields significantly higher alignment (+0.158) than asking "what would a typical reader highlight" (+0.074).
Correction of Prior Work: The authors report a non-replication of their own prior work, which had claimed zero-shot models performed worse than a lead baseline. On this corpus, the published instruction shows real enrichment (+0.074) when evaluated with the new matched estimator.

Results

Main Effect: The LLM enrichment of +0.196 is statistically significant and stable across matching tolerances (0.03 to 0.10).
Surface Features: Conditioning on lexical centrality reduces the effect by only ~0.010 (5%), indicating the agreement is not merely due to the model identifying central vocabulary.
Contiguity: The effect is not an artifact of paragraph-level marking; isolated marks show strong alignment (+0.227 for GPT-5.4).
Compression Ratio: The magnitude of the effect varies by compression ratio, peaking at $r=0.30$ (+0.222) and dropping to +0.083 at $r=0.05$ . The headline result (+0.196) is reported at $r=0.20$ .
Human Baseline: When compared to a single human reader (whose marks are recomputed excluding themselves), the LLM (GPT-5.4) achieves an enrichment of +0.184, which is statistically indistinguishable from the human baseline (+0.182). A stronger model (Claude Opus 5) exceeds the human baseline (+0.230).
LLMLingua-2: A purpose-built compressor (LLMLingua-2) shows an enrichment of +0.038, which is not statistically significant ( $p=0.085$ ) and falls within the confidence interval of the Luhn baseline.

Significance and Claims
The paper claims that language models can align with unprompted human reading behavior beyond simple surface features like position and length. However, the authors are modest in their interpretation:

Not Categorical Superiority: The finding is not that LLMs are categorically different from cheap methods; rather, they double the signal recoverable by lexical heuristics.
Not "Human Attention": The alignment is measured as agreement with what readers chose to keep, not an objective measure of importance.
Non-Circular Evaluation: The study provides a non-circular reference for compression evaluation, avoiding the trap of using one model to judge another.
Limitations: The authors acknowledge limitations including the single-platform data source, potential pretraining contamination (though tests on mark recency suggest it is not the driver), and the fact that the effect size varies significantly with compression ratio. They explicitly state that the result licenses the claim that models agree with readers "beyond position, length, and the seven features in the balance table," but does not prove the mechanism is "human attention."

The Great Text Filter: Why We Need a Better Way to Read

The Paper's Big Discovery

Technical Summary: Measuring Alignment With Reader Highlights Net of Position and Length

More like this