💬 NLP

When Certainty Is an Artifact: Keyword Lexicon Blindness and the (Mis)Measurement of Rhetorical Stance

This paper demonstrates that keyword-based lexicons can produce statistically significant but entirely artifactual findings regarding rhetorical stance by failing to capture semantic context, whereas LLM-based semantic classification reveals that negative discourse often pairs with hedging rather than the emphatic certainty falsely detected by traditional word-counting methods.

Original authors: Bo Chen

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand the personality of four famous people (a macro investor, a tech optimist, an academic economist, and a geopolitical strategist) by listening to their interviews. You want to know: When they talk about bad news (like a crisis or a crash), do they sound confident and certain, or do they sound unsure and cautious?

This paper is a detective story about how we measure that "confidence." It reveals that a very popular tool used by scientists to analyze language is actually broken in a specific way, leading them to believe something that isn't true.

Here is the story in simple terms:

1. The Broken Flashlight (The Keyword Method)

For a long time, researchers have used a "keyword flashlight" to measure confidence. This tool works like a simple counter:

If it hears words like "always," "certainly," "absolutely," or "never," it counts them as Confidence.
If it hears words like "maybe," "could," or "risk," it counts them as Uncertainty.

When the researchers used this tool on 85 interviews, they found a shocking pattern: Every single speaker sounded incredibly confident when talking about bad news.

The correlation was huge (around 0.85 to 0.93).
It looked like a universal law: When people talk about doom, they sound 100% sure.

The researchers initially thought this was a fascinating psychological discovery: "Oh, so when people are scared, they double down and sound super confident!"

2. The High-Definition Camera (The LLM Method)

But then, the researchers decided to check their work using a "High-Definition Camera"—a modern Large Language Model (LLM). Unlike the keyword counter, the LLM doesn't just count words; it understands context, grammar, and meaning, just like a human does.

They fed the exact same 32,000 sentences into the LLM. The result? The "Universal Law" vanished.

Dalio's "Confidence" dropped: His correlation plummeted from a massive 0.85 down to a weak, non-significant 0.20.
The Pattern Reversed: For two of the speakers, the LLM found that when they talked about bad news, they actually sounded uncertain and cautious (hedging), which is what common sense suggests.
The Conclusion: The "super-confident pessimists" were an illusion created by the broken flashlight.

3. Why Did the Flashlight Lie? (The Three Glitches)

The paper explains that the keyword tool failed because it is "blind" to three specific things. Think of it like a robot that counts words but doesn't understand how humans speak:

Blindness to Negation (The "Not" Trap):
- The Sentence: "I am never absolutely totally confident."
- The Keyword Tool: It sees "never," "absolutely," and "totally." It counts three "Confidence" points! It thinks the speaker is very sure.
- The Reality: The speaker is saying the exact opposite: "I am not sure at all." The tool missed the word "never" because it was looking for the presence of strong words, not the meaning of the sentence.
Blindness to Double Meanings (The "Certain" Trap):
- The Sentence: "You have a certain body..."
- The Keyword Tool: It sees "certain" and counts it as "Confidence."
- The Reality: Here, "certain" just means "a specific one" (like "a specific body"), not "I am 100% sure." The tool can't tell the difference between "I am certain" and "a certain thing."
Blindness to Softeners (The "Sort Of" Trap):
- The Sentence: "It is sort of clear."
- The Keyword Tool: It sees "clear" and counts it as "Confidence."
- The Reality: "Sort of" makes the statement weak. The tool doesn't know that "sort of" cancels out some of the strength of "clear."

4. The Real Discovery

The paper argues that the "strong correlation" the keyword tool found wasn't about the speakers' psychology. It was about how language works.

When people talk about scary, serious topics (like debt crises or wars), they naturally use big, dramatic words like "always," "never," and "everyone." The keyword tool counted these dramatic words as "confidence." But in reality, those words were just part of the drama of the topic, not a sign that the speaker was actually sure of the outcome.

The Analogy:
Imagine a weather forecaster who says, "It is absolutely going to be a disaster if it rains."

The Keyword Tool hears "absolutely" and says, "Wow, this person is very confident!"
The LLM (Human) hears the whole sentence and realizes, "They are actually very worried about a disaster."

The Bottom Line

This paper is a warning to scientists: Just because a computer counts words and finds a huge, statistically significant pattern, doesn't mean the pattern is real.

The "Certainty" the researchers thought they found was an artifact—a glitch in the measuring tool. When they fixed the tool (by using an AI that understands context), the "Certainty" disappeared, and the speakers looked much more like normal humans: cautious when things look bad, and confident when things look good.

Key Takeaway: Don't trust a word-counting machine to tell you what people mean. You need a tool that understands the story behind the words.

Technical Summary: When Certainty Is an Artifact

Problem Statement

Computational social science (CSS) frequently relies on keyword lexicons to measure rhetorical stance, specifically the relationship between affective valence (positive/negative) and rhetorical modality (certainty/hedging). A prevailing assumption in the literature is that negative discourse is delivered with hedging to manage reputational risk, while positive discourse is delivered with certainty. However, this paper investigates whether statistically significant findings in this domain are genuine rhetorical patterns or artifacts of the measurement instrument itself.

The authors identify a critical methodological gap: standard keyword-based instruments (bag-of-words) fail to resolve syntactic negation, polysemy, and modifier scope. Consequently, they may conflate surface-level lexical co-occurrence with deep semantic meaning. The central research question is whether the observed coupling between negative content and emphatic certainty is a speaker-specific rhetorical strategy or a universal measurement artifact caused by the limitations of keyword counting.

Methodology

The study employs a dual-instrument design to validate measurements against a "ground truth" derived from Large Language Models (LLMs).

1. Corpus Construction

The authors curated a longitudinal corpus of 85 YouTube interviews featuring four public intellectuals with distinct discourse profiles:

Ray Dalio (Macro investor): 35 videos (2016–2026), exhibiting a structural break in frequency post-2025.
Cathie Wood (Techno-optimist): 17 videos.
Kenneth Rogoff (Academic economist): 17 videos.
Peter Zeihan (Geopolitical strategist): 16 videos.

Total corpus size: 498,200 words and 32,625 sentences.

Preprocessing: Raw auto-generated subtitles were processed using LLM-assisted speaker diarization (DeepSeek-V4-Flash) to separate guest speech from interviewer interjections, a step critical for removing noise that previously obscured speaker-specific patterns.

2. Instrument A: Keyword Lexicon Scoring

A custom multi-dimensional sentiment lexicon was constructed to separate Affective Valence (Positive/Negative) from Rhetorical Modality (Emphatic/Certain vs. Hedged/Doubt).

Mechanism: The lexicon counts surface tokens (e.g., "always," "never," "crisis") and applies basic negation flipping rules.
Metric: Monthly normalized frequencies were calculated, and Pearson correlations ( $r$ ) were computed between Negative Valence frequency and Emphatic/Certain Modality frequency.
Robustness Check: The analysis was replicated using two independent, widely cited reference lexicons (LIWC-22 and Hyland's 2005 lists) to ensure findings were not specific to the authors' curated word list.

3. Instrument B: LLM-Based Semantic Classification

To provide a context-aware validation, the entire corpus (32,625 sentences) was processed using zero-shot semantic classification via Doubao-Seed-2.0-Lite.

Task: Each sentence was classified simultaneously on two axes: Valence (positive/negative/neutral) and Modality (emphatic/hedged/neutral).
Validation: A cross-model robustness check was performed using GPT-5.4 on 10 independent random samples (100 sentences each) to verify that findings were not artifacts of a single model's training data.

4. Error Analysis

A sentence-level audit was conducted on 547 sentences flagged as "high-certainty" by the keyword lexicon but classified differently by the LLM, categorizing failures into specific linguistic blind spots.

Key Results

1. The Keyword Artifact

Under keyword-based scoring, all four speakers exhibited a strong, statistically significant positive correlation between negative affect and emphatic certainty:

Dalio: $r = 0.851$ ( $p < 0.001$ )
Rogoff: $r = 0.917$ ( $p < 0.001$ )
Zeihan: $r = 0.932$ ( $p < 0.001$ )
Wood: $r = 0.725$ ( $p = 0.003$ )

This pattern was universal across speakers with radically different personas (optimists vs. pessimists, academics vs. investors) and persisted across three different keyword lexicons (Custom, LIWC-22, Hyland). This suggests the correlation is a structural property of keyword counting rather than a rhetorical strategy.

2. The LLM Reversal

When the same corpus was analyzed via LLM-based semantic classification, the strong positive correlations vanished or reversed:

Dalio: $r(neg, emph) = 0.206 $($ p = 0.371$, non-significant).
Wood: $r(neg, emph) = -0.514 $($ p = 0.060$).
Zeihan: $r(neg, emph) = -0.223 $($ p = 0.487$).
Rogoff: $r(neg, emph) = 0.063 $($ p = 0.864$).

Instead, the LLM revealed a negative-hedging coupling consistent with conventional linguistic theory (pessimists hedge):

Rogoff: $r(neg, hedged) = 0.875 $($ p = 0.001$).
Zeihan: $r(neg, hedged) = 0.722 $($ p = 0.008$).

3. Cross-Model Robustness

The LLM findings were confirmed to be independent of the specific model provider. Doubao-Seed-2.0-Lite and GPT-5.4 produced statistically indistinguishable correlation estimates for both hedging and emphatic metrics ( $p > 0.24$ for all comparisons), validating the substantive pattern.

4. Structural Failure Modes

Sentence-level error analysis identified three specific mechanisms by which keyword lexicons invert semantic meaning:

Syntactic Blindness: Failure to parse negation structures (e.g., "never absolutely confident" is scored as high certainty rather than low confidence).
Polysemy Blindness: Inability to distinguish context-dependent meanings (e.g., "a certain body" vs. "certainly").
Categorical Absence: Softening modifiers (e.g., "sort of," "almost," "I think") lack dedicated categories and fail to attenuate the scores of adjacent certainty markers.

Key Contributions

Demonstration of Measurement Artifacts: The paper provides empirical evidence that strong, statistically significant correlations in CSS (specifically the negative-certainty coupling) can be entirely artifacts of the measurement instrument. It shows that keyword counting measures a universal lexical co-occurrence tendency (serious topics attract serious language) that is orthogonal to, and can systematically invert, actual rhetorical stance.
Error Typology for Lexicon Auditing: The authors propose a reusable framework for auditing keyword-based instruments, identifying three structural failure modes (syntactic blindness, polysemy blindness, categorical absence) that explain why bag-of-words representations fail to capture modality.
Validation Framework for CSS: The study establishes a four-step validation template for future research:
- Test for generalizability across speakers/domains.
- Validate against full-population, context-aware semantic classifiers (LLMs).
- Replicate validation with independently trained models to rule out provider bias.
- Conduct sentence-level error analysis to diagnose specific linguistic failures.

Significance and Claims

The paper argues that treating keyword counts as direct measurements of epistemic certainty constitutes a category error. A finding that appears to be about a speaker's psychology (e.g., "Dalio is increasingly certain about doom") may be entirely about the counting of words.

The authors do not claim that keyword methods should be abandoned, acknowledging their interpretability and efficiency. However, they assert that strong keyword-based correlations must be interpreted as signals of lexical co-occurrence, not rhetorical stance, unless validated by context-aware methods. The study concludes that when a computational instrument produces a striking finding, the instrument itself deserves as much scrutiny as the speaker it claims to measure. The "Dalio effect" observed in the data was not about Dalio, but about the limitations of the counting mechanism.