💬 NLP

Not Worth Mentioning? A Pilot Study on Salient Proposition Annotation

This paper introduces a pilot study that operationalizes graded proposition salience by defining an annotation task, applying it to a multi-genre dataset, and preliminarily evaluating its relationship with discourse unit centrality in Rhetorical Structure Theory.

Original authors: Amir Zeldes, Katherine Conhaim, Lauren Levine

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amir Zeldes, Katherine Conhaim, Lauren Levine

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to summarize a massive, chaotic library for a visitor who only has five minutes. You know the visitor needs the "most important" parts of the story. But here's the tricky part: What actually counts as "important"?

For decades, computer scientists have been trying to teach machines how to pick these important parts (a task called extractive summarization). However, they've mostly focused on picking whole sentences. They haven't really figured out how to measure the importance of specific ideas (or "propositions") inside those sentences, especially when you have to deal with the messy reality of real human writing.

This paper is like a pilot test for a new way of grading those ideas. Here is the breakdown using some simple analogies:

1. The Old Way vs. The New Way

The Old Way (The "Yes/No" Switch): Previously, researchers looked at whether an idea was mentioned in a summary or not. It was like a light switch: either the idea was in the summary (ON) or it wasn't (OFF).
The Problem: Summaries are subjective. One person might write a summary focusing on the who, while another focuses on the why. If you only have one summary, your data is biased.
The New Way (The "Dimmer Switch"): The authors (Amir, Katherine, and Lauren) borrowed a trick from a previous study about people (entities). Instead of one summary, they used five different summaries for the same text.
- If an idea appears in all 5 summaries, it's a superstar (very salient).
- If it appears in 2 summaries, it's a supporting character (moderately salient).
- If it appears in 0 summaries, it's a background extra (not salient).
- This creates a gradient (a dimmer switch) rather than a simple on/off switch.

2. The Experiment: Breaking Text into "Bricks"

To do this, the team had to break down 32 different texts (ranging from news articles and court transcripts to vlogs and textbooks) into tiny, manageable chunks called propositions.

The Analogy: Think of a text as a wall made of bricks. Some bricks are whole sentences, some are just fragments (like a headline). The team used a pre-existing map (called Rhetorical Structure Theory) to decide where one "brick" ends and the next begins.
The Task: They asked human annotators to look at each "brick" and ask: "Does this specific idea show up in any of the five summaries?"
The Tricky Part: Sometimes a summary mentions the result of an idea without using the exact same words. The team had to decide if that counted as a "match" or just a "hint." It's like recognizing that "The King is dead" and "The throne is vacant" are talking about the same event, even though the words are different.

3. Did It Work? (The Agreement Test)

When you ask humans to judge importance, they often disagree.

The Result: The team found that while humans didn't agree 100% of the time (which is normal), they agreed much better than random chance.
The "Duplicate" Issue: The biggest source of disagreement was when two different sentences said the exact same thing. If one annotator picked Sentence A and another picked Sentence B, the system had to decide: Is this a disagreement, or just two people picking different "duplicates" of the same important idea? Once they accounted for this, agreement went up significantly.

4. The "Tree" Connection

The researchers also wanted to see if their new "importance score" matched up with how linguists usually analyze text structure (using something called RST trees).

The Analogy: Imagine a family tree. The "root" is the main topic at the top. The branches go down to smaller details.
The Finding: They found that ideas closer to the "root" (the main topic) were indeed more likely to be in the summaries. However, this wasn't the only factor. Long sentences, specific types of relationships between ideas, and where the sentence appeared in the text also mattered.
The Reality Check: Even with all these clues, their computer model could only predict importance about 74% of the time. This tells us that human judgment of importance is still very complex and hard to automate perfectly.

5. Why Does This Matter?

This paper is a prototype. It's like building a small model airplane to test the aerodynamics before building a real jet.

The Goal: They are building a dataset and a tool (an annotation interface) so that in the future, computers can learn to understand why an idea is important, not just that it is important.
The Future: They plan to use this data to train AI (Large Language Models) to become better at summarizing, so that when you ask an AI to "summarize this," it doesn't just grab random sentences, but actually understands the core narrative.

In a Nutshell

The authors are trying to teach computers to understand nuance. Instead of just asking "Is this sentence in the summary?", they are asking "How often does this specific idea appear across different summaries?" They've built a new measuring stick, tested it on a small group of texts, and found that while it's a promising tool, teaching machines to truly grasp human importance is still a work in progress.

1. Problem Statement

While extractive summarization has a long history of aiming to recover the most important propositions in a text, there is a significant gap in research regarding the operationalization and measurement of graded proposition salience in naturally occurring data.

The Gap: Unlike entity salience, which has been operationalized via summary-based metrics (e.g., frequency of mention across multiple summaries), proposition salience lacks a consistent, graded metric.
The Challenge: Existing methods often rely on binary labels (mentioned/not mentioned) or suffer from instability due to the subjectivity of single summaries. There is a need for a method that quantifies how salient a proposition is based on its presence across multiple human-generated summaries.

2. Methodology

The authors adapt a methodology previously used for Salient Entity Extraction (SEE) (Lin and Zeldes, 2025) and apply it to propositions.

Data Source

Corpus: The GUM corpus (Zeldes, 2017), specifically the test set comprising 32 documents across 16 genres (spoken and written English).
Scale: ~30,000 tokens, 3,779 Elementary Discourse Units (EDUs), and approximately 19,000 proposition-summary alignments.
Summaries: Each document has five distinct human-written summaries, allowing for a gradient metric.

Annotation Task

The annotation process involves three steps:

Defining Proposition Markables: The authors use Elementary Discourse Units (EDUs) from Rhetorical Structure Theory (RST) as the atomic units for propositions. This ensures a "tiling" approach where every word belongs to exactly one proposition. EDUs can range from full sentences to fragments (e.g., headings) and can be discontinuous.
Alignment: Annotators align propositions to the five summaries using a hierarchical judgment system:
- Strong Match: The summary explicitly refers to the same event/predicate or entity.
- Approximate Alignment: The proposition is not explicitly mentioned but is essential for the summary's content (e.g., a summary mentions a time, triggering a proposition containing that time).
  - Normal Trigger: Information carried directly.
  - Component Alignment: The summary aggregates information from multiple propositions (e.g., summarizing "disappointment" based on multiple specific statements in the text).
- Duplicates: Multiple propositions providing the same salient information (redundancy).
Scoring: Salience is calculated as a gradient score based on the number of summaries (out of 5) in which a proposition appears or triggers content.

Tools

A custom annotation interface was developed to facilitate this task, allowing annotators to highlight pre-defined EDUs and categorize alignments (match, approximate, component, duplicate) for each summary individually.

3. Key Contributions

Definition of Graded Proposition Salience: A novel framework for quantifying proposition importance based on multi-summary alignment, moving beyond binary classification.
Pilot Dataset: A new dataset of salient propositions aligned to summaries with rich metadata (approximate triggers, components, duplicates) across 16 genres.
Annotation Interface & Guidelines: Publicly released guidelines and a flexible annotation tool designed for sentence/span highlighting and custom field definitions.
Correlation Analysis: A preliminary study linking the new salience metric to RST Centrality (discourse unit nesting depth).

4. Results

Inter-Annotator Agreement

Strict Agreement: Initial agreement was moderate (Cohen's $\kappa \approx 0.65$ ).
Lenient Scenarios: Agreement improved significantly when accounting for the nature of the task:
- Treating "duplicate" units as interchangeable raised $\kappa$ to 0.84.
- Ignoring disagreements on "component" alignments (which are inherently subjective) further improved metrics.
Observation: Disagreements often stemmed from lexical overlap vs. identical reference (e.g., selecting a specific verb phrase vs. a general concept) and handling fragment components.

Salience Distribution

Unlike entity salience (which follows a U-shaped distribution), proposition salience follows a descending histogram.
The largest group of salient propositions appears in only one of the five summaries, indicating high inter-summary variance at the proposition level compared to the entity level.

Correlation with RST Centrality

The authors investigated if RST centrality (distance from the tree root) predicts salience:

Correlation: A significant negative correlation was found ( $r^2 = -0.287, p < 0.0001$ ), meaning units closer to the root (more central) tend to be more salient.
Confounding Factors: The correlation is influenced by:
- Position: Earlier units in a document are more likely to be salient.
- Length: Longer EDUs and full sentences overlap more with summaries than short clauses.
- Relation Type: Discourse roles (e.g., CAUSE, PURPOSE) influence salience.
Model Performance: A mixed-effects model confirmed centrality as the strongest predictor of salience. However, the model achieved only a weak fit (marginal $r^2 = 0.128$ ) and a binary classification accuracy of 73.98% (only slightly better than a majority baseline of 72.33%). This suggests that while centrality matters, it is not the sole determinant of salience.

5. Significance and Future Work

Theoretical Impact: This study challenges the assumption that discourse centrality (RST) is a perfect proxy for informational salience, highlighting the complexity of proposition selection in summarization.
Practical Application: The dataset and guidelines provide a benchmark for training and evaluating NLP models (including LLMs) on graded salience tasks.
Future Directions: The authors plan to annotate the full GUM corpus (using NLP/LLM assistance for the training set) and refine guidelines based on adjudication processes to improve consistency. The ultimate goal is to develop prompting strategies for LLM ensembles to approach human-level performance in identifying salient propositions.

Conclusion: The paper successfully operationalizes a gradient metric for proposition salience, demonstrating that while the task is challenging and subject to annotator variance, it is feasible and offers valuable insights into the relationship between discourse structure and informational importance.