💬 NLP

A New Semisupervised Technique for Polarity Analysis using Masked Language Models

This paper introduces an enhanced semisupervised polarity analysis technique that utilizes word2vec as a masked language model to assign more accurate and interpretable probabilistic polarity scores, demonstrating its superiority over traditional spatial models through an analysis of China Daily's COVID-19 coverage.

Original authors: Kohei Watanabe

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kohei Watanabe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand the "mood" or "focus" of thousands of news articles. Do they sound proud? Do they sound worried? Do they talk about success or failure?

For a long time, researchers had two main ways to do this:

The Dictionary Method: You make a giant list of "good" words and "bad" words. If an article uses many words from your list, it gets a high score. Problem: You need a massive, perfect list to get good results, and making that list takes forever.
The Old Computer Method (Spatial Models): You teach a computer to group words based on how often they appear together. If "win" and "victory" appear near each other, the computer thinks they are close friends. It then guesses the mood of an article based on how "close" the words in that article are to your starting "seed" words. Problem: This method is often messy, hard to interpret, and very sensitive to which starting words you pick.

The New Idea: A "Fill-in-the-Blank" Game

The author, Kohei Watanabe, proposes a new, smarter way to do this using a technique called Latent Semantic Scaling (LSS), but upgraded with a tool called word2vec.

Think of the old method like a map. You place a pin on "Success" and another on "Failure." The computer measures how far away every other word is from those pins. If a word is halfway between them, it's neutral. But maps can be tricky; if you move the pins slightly, the whole map changes, and the distances don't always make sense.

The new method is more like a game of "Mad Libs" or "Fill-in-the-Blank."

Instead of measuring distance on a map, the computer plays a guessing game. It looks at a sentence and asks: "If I hide the word 'success' here, how likely is it that this word would fit?"

The Seed Words: You give the computer a few starting words (seeds), like "win," "goal," or "champion."
The Game: The computer reads the news articles and tries to predict if those seed words would naturally appear in the context of other words.
The Score: If the computer is very confident that "victory" fits where "success" was hidden, it gives "victory" a high "polarity score" (a score of how much it relates to the concept).

Why is this better?

It's a Probability, Not a Distance: Instead of saying "this word is 5 miles away from success," the computer says, "There is a 90% chance this word belongs in a sentence about success." This is much easier to understand and compare.
It's Less Picky: In the old "map" method, your starting words had to be the most extreme examples (like "triumph" instead of just "win"). In this new "guessing game," you can use more moderate words, and the computer still figures out the extreme ones on its own.
It Self-Corrects: The computer has a built-in "scorecard" called perplexity. Think of this like a test grade. If the computer is good at predicting words, its "perplexity" score is low (it's not confused). If it's bad, the score is high. This allows researchers to tune the computer's settings automatically to get the best results without needing a human to check every answer.

The Test: China Daily and the Pandemic

To prove this works, the author tested it on China Daily, a newspaper controlled by the Chinese government, during the COVID-19 pandemic.

The Goal: See how the newspaper talked about "Achievement" and "Health" regarding China versus other countries.
The Comparison: The author compared the new "Guessing Game" method against the old "Map" method and a massive, manually created dictionary.
The Result: The new method was the winner. It matched the massive dictionary better than the old computer method did. It was also more consistent; no matter which starting words were picked, the results stayed reliable.

What the Analysis Found

Using this new tool, the author found some interesting patterns in the news:

Health Focus: At the very start of the pandemic (early 2020), the newspaper focused heavily on health issues within China. As time went on, the focus shifted to talking about health issues in other countries.
Achievement Focus: The newspaper's pride in China's "achievements" dipped at the start of the crisis but grew stronger again by the end of 2023.
The "Sweet Spot": When the author combined these two ideas (Achievement + Health), the newspaper only sounded truly proud of China's health achievements during two specific times: right at the beginning of the crisis and right after the strict lockdowns ended in late 2022. This suggests the government was trying to frame the end of the lockdown as a success story.

The Bottom Line

This paper introduces a new way for computers to read text that is more like a human guessing game and less like a rigid map. It is more accurate, easier to understand, and requires less manual work to set up. It shows that by using these "masked language models" (the fill-in-the-blank games), we can get clearer, more objective insights into how media portrays different topics.

1. Problem Statement

Polarity analysis (measuring the emphasis or sentiment of a text toward a specific concept) is crucial in social science and communication research. Existing methods face significant limitations:

Keyword Matching: Requires extensive, manually curated dictionaries which are often unavailable for niche domains or specific languages.
Supervised Machine Learning: Requires large, annotated corpora, which are costly and time-consuming to create.
Unsupervised Machine Learning: Often fails to capture pre-defined theoretical concepts without guidance.
Existing Semisupervised Methods (Latent Semantic Scaling - LSS): The original LSS relies on spatial modeling (using Singular Value Decomposition or SVD to create word vectors). This approach has four critical flaws:
1. High Measurement Error: Document scores often have large errors unless aggregated by other variables.
2. Lack of Interpretability: Scores are relative within a semantic space and lack substantive meaning (e.g., they can be negative in unipolar models).
3. Seed Word Sensitivity: Seed words must be "extreme" (highly polarized) to generate a useful range of scores; moderate seed words fail to produce meaningful results.
4. Optimization Difficulty: Hyperparameters (like vector size) cannot be easily optimized because the underlying algorithm (SVD) lacks a standard goodness-of-fit metric like perplexity.

2. Methodology

The author proposes a new version of Latent Semantic Scaling (LSS) that replaces spatial modeling with probabilistic modeling using word2vec as a masked language model.

Core Algorithmic Shift

From Spatial to Probabilistic:
- Original (Spatial): Calculates polarity based on the cosine similarity between a word's vector and seed word vectors.
- Proposed (Probabilistic): Calculates polarity as the predicted probability of a seed word occurring in the context of a target word.
Model Architecture:
- Utilizes word2vec (specifically Skip-gram and Continuous Bag-of-Words algorithms) trained on the target corpus.
- Unlike standard word2vec usage (which only uses input layer weights $V$ for vectors), this method utilizes both the input layer weights ( $V$ ) and the output layer weights ( $W$ ) to compute the probability $q_{ij}$ of a target word $j$ given a context word $i$ .
- The probability is calculated via a sigmoid function: $q_{ij} = \text{sigmoid}(\sum H_k W_{jk})$ .
Polarity Estimation:
- A word's polarity score ( $\hat{g}_i$ ) is the weighted average of the probabilities that the seed words occur in the context of that word.
- Advantage: This allows the model to assign extreme polarity scores to relevant words even if the seed words themselves are not "extreme," reducing sensitivity to user seed selection.
Optimization:
- Because the model is probabilistic, perplexity (cross-entropy between predicted probabilities and observed frequencies) can be used to optimize hyperparameters (e.g., vector dimension, training algorithm) without needing a "gold standard" labeled dataset.

Evaluation Design

Corpus: 91,709 English articles from China Daily (2019–2023) covering the COVID-19 pandemic.
Target Concepts: "Achievement" and "Health."
Benchmark: Full LIWC dictionaries (manually curated, containing ~200–300 words per concept).
Experimental Setup:
1. Randomly sampled 10 sets of 10 unipolar seed words from the LIWC dictionaries (simulating imperfect user input).
2. Trained four model types: Spatial LSS (SVD), Spatial LSS (word2vec), Probabilistic LSS (word2vec), and Mini-dictionaries (seed words only).
3. Correlated model outputs against the full LIWC dictionary scores.
4. Analyzed the relationship between perplexity and correlation coefficients.

3. Key Contributions

Probabilistic LSS Framework: Introduces a method to convert LSS from a spatial similarity measure to a probabilistic language model, making scores interpretable as probabilities (bounded 0–1) rather than arbitrary vector distances.
Robustness to Seed Selection: Demonstrates that probabilistic models are less sensitive to the "extremeness" of seed words. They can produce results comparable to large dictionaries even with moderately relevant seed words.
Hyperparameter Optimization: Provides a mechanism to optimize model hyperparameters (like vector size) using perplexity scores, solving a major limitation of SVD-based LSS.
Open-Source Implementation: The method is implemented in the LSX and wordvector R packages, ensuring reproducibility and transparency without reliance on proprietary black-box models.

4. Results

Performance Correlation:
- Word2vec-based Probabilistic models achieved the strongest correlation with the full LIWC dictionaries ( $r=0.51$ for Achievement, $r=0.66$ for Health).
- Skip-gram (SG) training generally outperformed Continuous Bag-of-Words (CBOW), though the gap was smaller in probabilistic models than in spatial ones.
- SVD-based models performed the worst ( $r \approx 0.34–0.48$ ), often underperforming even simple "mini-dictionaries" (seed words only).
Variance and Reliability:
- Probabilistic models showed the lowest variance in correlation coefficients across different seed word samples, indicating high reliability.
- Spatial models and SVD models showed high variance, meaning their results fluctuated significantly based on the specific seed words chosen.
Perplexity as a Metric:
- A strong negative association was found between perplexity and correlation. Lower perplexity scores reliably predicted higher accuracy against the benchmark, validating perplexity as a tool for model selection in the absence of ground truth.
Case Study (China Daily):
- The model successfully identified shifts in China Daily's framing. It detected a shift from focusing on China's health achievements early in the pandemic to focusing on other countries later.
- By combining "Achievement" and "Health" scores, the model highlighted specific periods (start and end of the "Zero-COVID" policy) where the state media framed the strict lockdowns as a success, a nuance missed by analyzing concepts in isolation.

5. Significance

Methodological Advancement: This study bridges the gap between traditional dictionary analysis and modern deep learning. It offers a "best of both worlds" approach: the interpretability of dictionaries (probabilistic scores) with the automation of machine learning.
Accessibility for Researchers: It lowers the barrier for semisupervised text analysis. Researchers no longer need massive annotated datasets or perfect seed words; they can use small seed sets and optimize the model using perplexity.
Transparency: By using a transparent, local word2vec implementation rather than proprietary Large Language Models (LLMs), the study ensures that political science and communication research remains reproducible and free from hidden biases inherent in commercial AI models.
Future Potential: The paper argues that this probabilistic framework can be extended to more complex masked language models (e.g., BERT, RoBERTa) to further enhance the capture of nuanced concepts in text analysis.