💬 NLP

A Comparative Evaluation of Structural Topic Models and BERTopic for Short, Open-Ended Survey Responses

This paper compares Structural Topic Models (STM) and BERTopic for analyzing short, open-ended survey responses, finding that while BERTopic with contextual augmentation produces more coherent and interpretable topics, STM remains superior for inferential covariate analysis, suggesting the two methods offer complementary strengths for applied social science research.

Original authors: Yan Jiang, Sihong Liu, Philip A. Fisher

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yan Jiang, Sihong Liu, Philip A. Fisher

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of interviewing suspects, you are reading thousands of tiny, messy notes left by people answering a survey question: "What are your biggest worries right now?"

Some notes are long and detailed. Others are just one word like "Money" or "Housing." Some have typos like "uncertanty" instead of "uncertainty." Your job is to sort these notes into piles (topics) so you can understand what people are really talking about.

This paper compares two different detective tools (methods) for doing this sorting job: STM and BERTopic.

The Two Tools

1. STM (The "Word Counter" Detective)
Think of STM as a very organized, rule-following accountant.

How it works: It counts how often words appear together. If "bills" and "rent" appear in the same notes often, it puts them in the same pile.
Its superpower: It is great at answering "Why?" questions. It can tell you if one group of people (like single parents) worries about money more than another group (like couples), and it can do this with statistical proof.
Its weakness: Because it just counts words, it sometimes gets confused by short notes. It might mix different themes together (like putting "money" and "health" in the same pile) or include vague words like "just" or "can't" that don't really mean much.

2. BERTopic (The "Meaning Reader" Detective)
Think of BERTopic as a modern, high-tech AI that reads the feeling and meaning of the words, not just the count.

How it works: It uses a massive library of language knowledge to understand that "car payment" and "auto loan" mean the same thing, even if the words are totally different. It groups notes based on their semantic "vibe."
Its superpower: It is excellent at sorting short, messy notes into very clear, distinct piles. It rarely mixes up different themes.
Its weakness: It is mostly descriptive. It can tell you what the piles are, but it's harder to use it to make strict statistical claims about why different groups worry differently.

The Experiment: Testing the Tools

The researchers took a real dataset of over 14,000 short survey responses from childcare providers. They tested both tools under different conditions to see which worked best:

The "Raw" Test: Did they use the notes exactly as written (with typos)?
The "Clean" Test: Did they fix the spelling mistakes first?
The "Stem" Test (for STM only): Did they chop words down to their root (e.g., changing "working" to "work")?
The "Context" Trick (for BERTopic only): Since the notes were so short, the researchers added the survey question to the front of every note. So, instead of just "Money," the note became: "Question: What are your worries? Answer: Money." This gave the AI more context to understand the note.

What They Found

1. BERTopic was the better sorter.
Across the board, BERTopic created cleaner, more logical piles of notes. The "Meaning Reader" understood the short notes much better than the "Word Counter."

The Magic Trick: The biggest boost for BERTopic came from the Context Trick. Adding the survey question to the short answers made the topics much clearer. It was like giving the detective a hint before they started reading.
The Surprising Result: Using a "smarter," more complex AI model (higher dimensions) didn't actually help. In fact, it sometimes made things worse and threw away more data. Sometimes, simpler is better for short notes.

2. STM was still useful, but messier.
STM's piles were often a mix of different themes (e.g., a pile containing "money," "covid," and "government" all at once). It struggled a bit with the short, noisy text. However, it kept more of the original words in its analysis.

3. The "Best of Both Worlds" Strategy.
The paper suggests these tools aren't enemies; they are teammates.

Step 1: Use BERTopic first to figure out what the main themes are. It's great at discovering the topics because it handles short, messy text so well.
Step 2: Once you know the topics, use STM to ask statistical questions about them. If you want to know if "Home-based" providers worry about rent more than "Center-based" providers, STM is the tool to give you the statistical answer.

The Bottom Line

If you are trying to make sense of short, messy survey answers:

Don't just rely on one tool.
Fix your data first (clean up typos).
Give the AI context (add the question to the answer) if you are using the modern method.
Use the modern method (BERTopic) to find the themes, and then use the traditional method (STM) to test your theories about those themes.

The paper concludes that while the modern tools are better at finding clear patterns in short text, the traditional tools are still essential for doing the heavy lifting of statistical analysis. You need both to get the full picture.

Technical Summary: A Comparative Evaluation of Structural Topic Models and BERTopic for Short, Open-Ended Survey Responses

Problem Statement

Topic modeling has become a standard tool in applied psychology and social sciences for analyzing open-ended survey responses. However, the evaluation of these methods has historically relied on benchmark corpora consisting of longer, cleaner documents (e.g., news articles). This leaves a significant gap in guidance for short, open-ended survey responses, which are often characterized by:

Extreme variability in length (ranging from single words to hundreds).
High levels of noise, including typographical errors and encoding artifacts.
Fragmented phrases that lack standalone semantic context.

The paper addresses the need to compare two dominant methodological traditions—probabilistic bag-of-words models (represented by Structural Topic Models, or STM) and embedding-based approaches (represented by BERTopic)—specifically under these challenging data conditions. It further investigates how preprocessing decisions (typographical correction, stemming) and modeling choices (embedding dimensionality, contextual augmentation) influence performance.

Methodology

Data Source

The study utilized a corpus of 14,238 open-ended survey responses from 4,873 early care and education (ECE) providers in the United States. The data was collected across 127 survey waves between March 2021 and October 2025. The focal question was: "What are the biggest challenges and concerns for you and your family right now?"

Data Characteristics: Responses averaged 15.6 words. 7.4% were single words, and 34.6% were 1–5 words. The raw text contained 1,163 identified typographical errors.

Experimental Design

The authors compared three STM conditions and five BERTopic conditions, varying preprocessing and model specifications. All models were estimated across a range of topic numbers ( $K = 10$ to $60$) and three random seeds, resulting in 144 total model runs.

STM Conditions:

Raw: Original text without correction.
Clean: Typographical errors corrected.
Clean + Stem: Cleaned text with stemming applied (reducing words to root forms).

BERTopic Conditions:

Raw: Original text.
Clean: Typographical errors corrected.
Clean + Emb: Cleaned text using a higher-dimensional embedding model (text-embedding-3-small, 1,536 dimensions) instead of the default (all-MiniLM-L6-v2, 384 dimensions).
Clean + Context (Ctx): Cleaned text with the survey question appended to each response to provide semantic context.
Clean + Ctx + Emb: Combined contextual augmentation and higher-dimensional embeddings.

Evaluation Metrics

Quantitative:
- Topic Coherence: Measured via Normalized Pointwise Mutual Information (NPMI).
- Topic Diversity: Measured via Inverted Rank-Biased Overlap (RBO).
- Data Retention: Proportion of words retained (STM) or documents assigned to topics (BERTopic).
- Stability: Word and topic overlap across preprocessing conditions and between models.
Qualitative: Human evaluation of topic interpretability, distinctness, and the presence of mixed themes.
Covariate Analysis: Examination of how topics vary by time and provider type (center-based vs. home-based).

Key Results

1. Topic Coherence and Quality

BERTopic Superiority: BERTopic consistently produced higher topic coherence (NPMI) than STM across all conditions. STM conditions yielded negative NPMI values (approx. -0.08), indicating modest word co-occurrence, whereas BERTopic achieved positive values (0.09–0.12).
Impact of Contextual Augmentation: The most significant performance gain for BERTopic came from contextual augmentation (appending the survey question). This strategy yielded the highest coherence scores (0.12), demonstrating that providing semantic context is critical for short texts.
Embedding Dimensionality: Contrary to the hypothesis that higher-dimensional embeddings capture more nuance, using the 1,536-dimensional model did not improve coherence and, in some cases, reduced it compared to the default 384-dimensional model. It was also associated with greater data loss.

2. Topic Interpretability and Structure

BERTopic: Generated topics that were more interpretable, stable, and sharply defined (e.g., distinct clusters for "basic needs," "financial stress," "housing"). It exhibited high stability across preprocessing conditions (86.7% word overlap between raw and clean).
STM: Topics were often broader, mixed, and less distinct. They frequently included general terms (e.g., "cant," "just") and combined multiple themes (e.g., mixing COVID concerns with financial hardship). STM showed lower stability across conditions (78.7% overlap).
Overlap: Cross-model overlap was modest (~40% word overlap), largely because STM relies on bag-of-words frequency while BERTopic relies on semantic similarity in embedding space.

3. Data Retention

STM: Retained 51.9%–57.9% of words after preprocessing (filtering rare/frequent terms).
BERTopic: Retained 59.4%–72.7% of documents. Documents that did not fit dense clusters were assigned to an outlier category.
Finding: Cleaner data generally improved retention. However, using higher-dimensional embeddings with limited context led to greater data loss in BERTopic.

4. Covariate Analysis

STM: Supports inferential analysis by incorporating covariates (e.g., time, provider type) directly into the estimation process, providing confidence intervals and statistical estimates.
BERTopic: Covariate analysis is post-hoc and descriptive. It compares topic prevalence across groups after topics are generated but does not offer the same inferential statistical framework.

Key Contributions

Contextual Augmentation Strategy: The paper introduces and validates "contextual augmentation" (appending the survey question to responses) as a highly effective strategy for improving embedding-based model performance on very short texts.
Empirical Evidence on Embedding Dimensionality: The study challenges the assumption that higher-dimensional embeddings are universally superior for short-text analysis, finding that they can reduce coherence and increase data loss in this specific context.
Complementary Framework: Rather than declaring a single winner, the paper establishes that STM and BERTopic offer complementary strengths:
- BERTopic is superior for topic discovery, generating coherent and stable structures from noisy, short data.
- STM is superior for inferential analysis, allowing researchers to test hypotheses about how topic prevalence varies by covariates.

Significance and Claims

The authors claim that their findings provide practical guidance for applied social science researchers who must analyze short, open-ended survey responses. They emphasize that:

Model performance in applied settings cannot be assumed to mirror results from benchmark corpora.
Data preparation is critical: Researchers must actively engage with data characteristics (e.g., noise, length) and make context-specific preprocessing decisions rather than relying on standardized pipelines.
Hybrid Workflows are Recommended: For research goals involving both discovery and inference, the authors suggest an iterative workflow: use BERTopic first to identify robust, interpretable topic structures, and then use those structures to inform STM analyses for hypothesis testing and covariate inference.
Evaluation must be Mixed: Quantitative metrics (coherence/diversity) are insufficient on their own; qualitative assessment and domain knowledge are essential for evaluating topic quality.

The paper concludes that STM and BERTopic should not be viewed as competing alternatives but as tools with distinct, complementary roles in the research lifecycle, particularly when dealing with the unique constraints of short-form survey data.