💬 NLP

Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference

This paper argues that in graded health-literacy annotation, disagreement is structurally driven by task complexity rather than annotator error, and that preserving this perspectivist variance is statistically necessary to avoid obscuring or reversing key social-scientific findings.

Original authors: Olga Kellert, Sriya Kondury, Candice Koo, Nemika Tyagi, Steffen Eikenberry

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Olga Kellert, Sriya Kondury, Candice Koo, Nemika Tyagi, Steffen Eikenberry

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to grade a stack of essays about how to stay safe during a pandemic. In the old way of doing things (the "standard" method), you would have a team of teachers read each essay, argue about the grade, and then take the average score to decide if the student passed or failed. If the teachers disagreed, the system assumed someone made a mistake or the instructions were unclear.

This paper says: "Wait a minute. That disagreement isn't a mistake. It's actually a clue."

Here is the story of what the researchers found, explained simply:

1. The Big Idea: Disagreement is a Signal, Not Noise

Usually, when computers or humans can't agree on an answer, we think, "Oh no, the data is messy." We try to smooth it out to get one single "truth."

The researchers studied 6,323 open-ended answers from people in Ecuador and Peru about COVID-19. Instead of forcing a single "Right" or "Wrong" label, they let the teachers give a "partial credit" score (like 0.5 out of 1.0).

The Analogy: Think of the answers like a blurry photo.

Old Way: You try to sharpen the photo until it looks like a single, clear picture, throwing away the blur.
New Way: You realize the blur itself tells you something. Maybe the object in the photo is just hard to see, or maybe different people are looking at it from different angles.

2. Who is the Problem? The Question or the Teacher?

The researchers asked: "Why do the teachers disagree? Is it because the teachers are bad at grading, or is it because the questions are tricky?"

The Result: It wasn't the teachers. The teachers were actually quite consistent. The disagreement came from the questions themselves.

The Metaphor: Imagine a game of trivia.

Question A: "What is 2 + 2?" Everyone agrees the answer is 4. No disagreement.
Question B: "What is the best way to handle a complex emotional situation?" Everyone has a different, valid opinion. The disagreement isn't because the players are confused; it's because the question is inherently hard to answer with a single fact.

The study found that the "tricky" questions (like "How do masks actually stop the virus?") caused the most disagreement. This means the disagreement was structured by the difficulty of the topic, not by the teachers' errors.

3. The "Agreement Filter" Reveals Hidden Truths

This is the most surprising part. The researchers looked at the data in two different ways:

The "Blender" Method: They mixed all the answers together (aggregated data) to find general trends.
The "Sieve" Method: They separated the answers into three piles:
- High Agreement: Everyone basically agreed on the score.
- Medium Agreement: The teachers were kind of unsure or split.
- Low Agreement: The teachers totally disagreed.

The Shocking Discovery: When they looked at the "Medium Agreement" pile, the rules of the game changed completely!

Country Differences: In the "High Agreement" pile, Country A seemed much smarter than Country B. But in the "Medium Agreement" pile, Country B looked smarter than Country A. The direction of the result flipped!
Education: In the "High Agreement" pile, people with more education knew more. But in the "Medium Agreement" pile, education didn't matter at all.
City vs. Country: In the "High Agreement" pile, city people did better. In the "Medium Agreement" pile, country people did better.

The Analogy: Imagine you are looking at a chameleon.

If you look at it from far away (Aggregated/Blender), it looks green.
If you look at it up close under specific lighting (High Agreement), it looks blue.
If you look at it under different lighting (Medium Agreement), it looks red.

If you just say "It's green," you are missing the fact that the chameleon changes color based on its environment. Similarly, if you just average the data, you miss the fact that social trends (like education or location) change depending on how clear or confusing the question is.

4. Why Does This Matter?

The paper argues that we need to stop pretending there is always one single "Ground Truth" for complex human knowledge.

For AI and Computers: If you train a computer to just guess the "average" answer, it might learn the wrong patterns. It needs to understand why people disagree. Is the question hard? Is the knowledge unevenly distributed?
For Public Health: If a health official looks at the "average" data, they might think, "Oh, city people know more than rural people." But if they look deeper, they might realize that rural people actually know more about certain specific topics, but the data was hidden by the "blur" of disagreement.

The Takeaway

Disagreement isn't a bug; it's a feature.

When people disagree on a graded task (like health literacy), it's often because the topic is complex and knowledge is uneven. By ignoring the disagreement and just averaging the scores, we hide the real, messy, and important differences between groups of people. To understand the truth, we have to look at the disagreement, not just the final score.

1. Problem Statement

Traditional Natural Language Processing (NLP) annotation pipelines operate under the assumption that every data instance possesses a single "latent ground truth." Disagreement among annotators is typically treated as noise and resolved via aggregation methods (e.g., majority voting). However, this approach fails in graded interpretive tasks, such as health-literacy evaluation, where responses may contain partial knowledge or implicit reasoning.

The paper addresses a critical gap: When does disagreement reflect instability in the task itself (epistemic difficulty) rather than annotator unreliability? Specifically, the authors investigate whether aggregating graded health-literacy judgments obscures important social-scientific inferences (e.g., effects of education, geography, or country) and whether "perspectivist" modeling (treating disagreement as informative) is statistically necessary for valid inference in low-resource, multilingual settings.

2. Methodology

Dataset and Context

Source: 6,323 open-ended responses to COVID-19 health questions collected from Ecuador and Peru.
Demographics: Participants (N=299) were recruited from historically underserved Quechua/Kichwa-speaking communities (rural and urban).
Languages: Responses were in Spanish, Quechua, and Kichwa.
Annotation Scale: Responses were independently labeled by four annotators using a proportional correctness score on a 5-point scale: $\{0, 0.25, 0.5, 0.75, 1.0\}$ ${0, 0.25, 0.5, 0.75, 1.0}$ .
- Scores reflect the degree of alignment with normative guidelines (WHO and national health ministries).
- This yielded 17,305 annotation-level observations.
Subset: Seven specific open-ended questions were selected for inferential analysis based on semantic specificity and frequency of response.

Analytical Framework

The authors employed a Perspectivist approach, moving beyond simple aggregation to analyze the structure of disagreement:

Variance Decomposition: Used mixed-effects models to partition total variance into components attributable to Question-level (conceptual difficulty), Annotator-level (rater inconsistency), and Residual factors.
Baseline Modeling: Trained a logistic regression classifier using TF-IDF features to predict binary correctness ( $<0.5$ vs. $\ge0.5$ ) to establish lexical predictability.
Agreement-Stratified Inference: Questions were stratified by inter-annotator agreement levels (High: $\kappa > 0.60$ ; Medium: $0.40 \le \kappa \le 0.60$ ; Low: $\kappa < 0.40$ ).
Comparative Analysis: Sociodemographic effects (Country, Education, Urban-Rural, Gender) were re-evaluated within each agreement stratum to test for stability or reversal of trends.

3. Key Contributions

Empirical Validation of Structured Disagreement: The study demonstrates that in graded health-literacy tasks, disagreement is primarily driven by question-level conceptual difficulty rather than annotator inconsistency.
Epistemic Stability vs. Instability: It distinguishes between "epistemically stable" components (where agreement is high and inferences are robust) and "unstable" components (where disagreement reveals varying degrees of partial knowledge).
Agreement-Stratified Inference: The paper introduces a methodological framework showing that aggregating data across all agreement levels can lead to false or reversed conclusions regarding social factors.
Extension of Perspectivism: It extends perspectivist NLP theory beyond overtly subjective tasks (like toxicity detection) to "objective" graded tasks involving factual knowledge and reasoning.

4. Key Results

A. Variance Structure

Question Dominance: Variance decomposition revealed that 33.73% of the total variance was attributable to the Question (conceptual difficulty), whereas only 1.14% was attributable to the Annotator.
Intraclass Correlation (ICC): The ICC for questions ( $\approx 0.337$ ) was substantially higher than for annotators ( $\approx 0.011$ ).
Implication: Disagreement is a signal of task complexity, not rater noise. High-variance questions involved complex biomedical mechanisms (e.g., mask reuse, asymptomatic transmission).

B. Lexical Predictability vs. Epistemic Stability

A TF-IDF logistic regression model achieved 83.98% accuracy in predicting binary correctness.
Crucial Finding: High lexical predictability does not equate to inferential stability. Even with strong surface-level signals, the underlying social inferences remained unstable when stratified by agreement levels.

C. Agreement-Stratified Inference (The "Reversal" Effect)

The study found that social-scientific effects varied significantly in magnitude and direction depending on the level of inter-annotator agreement:

Interpretation: In "Medium Agreement" regimes, responses are ambiguous or inconsistently interpretable. Aggregating these with "High Agreement" data masks the instability, leading to conclusions that may be artifacts of the aggregation method rather than true population trends.

D. Multivariate Interactions

Education effects were context-sensitive: The gradient was stronger in rural areas than urban ones, and stronger in Ecuador than Peru. These patterns were most visible in the high-agreement subset.

5. Significance and Implications

For NLP Research:
- Modeling Strategy: Strong perspectivist modeling (incorporating disagreement as a feature rather than noise) is statistically necessary for valid inference in graded interpretive tasks.
- Evaluation: Standard accuracy metrics are insufficient; evaluation must account for the structure of disagreement to avoid masking epistemic uncertainty.
For Public Health & Social Science:
- Diagnostic Tool: Inter-annotator agreement serves as a diagnostic indicator of epistemic stability. Low agreement in specific demographic groups or question types signals where knowledge is unevenly distributed or where guidelines are unclear to the population.
- Policy Interpretation: Aggregated health literacy scores may obscure critical disparities. Policymakers must consider that "average" performance might hide reversed trends in specific subgroups (e.g., rural vs. urban) depending on the clarity of the questions.
Theoretical Contribution:
- The paper argues that disagreement in graded tasks is not merely "noise" but encodes differences in partial knowledge. It challenges the "single ground truth" paradigm, suggesting that for complex, real-world knowledge, the distribution of judgments is the data of interest, not just the mean.

Conclusion

The paper concludes that health-literacy annotation contains both epistemically stable and unstable components. Aggregating across these components without regard for disagreement structure risks obscuring meaningful social variation. The authors advocate for agreement-stratified and hierarchical modeling approaches to ensure that inferences drawn from NLP systems in public health are robust, context-aware, and statistically valid.