TopicENA: Enabling Epistemic Network Analysis at Scale through Automated Topic-Based Coding

This paper introduces TopicENA, a scalable framework that integrates BERTopic with Epistemic Network Analysis to automate concept coding, thereby enabling the structural analysis of large text corpora while providing practical guidance on optimizing topic granularity and inclusion thresholds.

The Gist

Imagine you are a detective trying to solve a mystery hidden inside a massive library containing millions of books. Your goal isn't just to count how many times the word "cat" appears; you want to understand how ideas connect. Do "cats" usually appear near "milk"? Do they appear near "dogs" or "space rockets"?

This is what Epistemic Network Analysis (ENA) does. It maps out how concepts in text (like student essays or chat logs) hang out together, creating a visual "friendship map" of ideas.

The Old Problem: The Manual Bottleneck

Traditionally, to create these maps, a team of expert detectives had to read every single sentence and manually tag the ideas.

• The Analogy: Imagine trying to map the entire internet by having one person read every single webpage and write down the topics by hand. It would take forever. You could only analyze a tiny, tiny slice of the data before you got exhausted. This meant ENA was great for small studies but useless for huge datasets.

The New Solution: TopicENA

The authors of this paper, Owen Lu and Tiffany Hsu, built a new tool called TopicENA. Think of it as hiring a super-fast, AI-powered robot librarian to do the tagging for you.

Instead of humans reading every sentence, they use a smart AI system called BERTopic to automatically group words into "topics" (like "Electric Cars," "Pollution," or "Space Exploration"). Then, they feed these AI-generated topics into the ENA system to build the maps.

The paper tests this new robot librarian in three different scenarios to see how to get the best results.

---

The Three Test Cases (The "Goldilocks" Experiments)

The researchers ran three experiments to figure out the "settings" for their robot. They realized that if you set the robot's brain too high or too low, the maps break.

Case 1: The Size of the "Bins" (Topic Granularity)

Imagine you are sorting a pile of mixed LEGO bricks into bins.

• Coarse Granularity (Big Bins): You have a few huge bins. One bin is "Red Stuff," another is "Blue Stuff."
  • Result: If you have a tiny pile of bricks, big bins are useless because everything ends up in the same bin, and you can't see any patterns. But if you have a mountain of bricks, big bins work great because they keep the chaos organized.
• Fine Granularity (Tiny Bins): You have hundreds of tiny bins. One is "Red 2x4 Brick," another is "Red 1x2 Brick."
  • Result: If you have a mountain of bricks, you end up with thousands of empty bins, and the map looks messy and broken. But if you have a tiny pile, tiny bins help you see the specific differences.

The Lesson: For huge datasets, use "big bins" (coarse topics). For small datasets, use "small bins" (fine topics).

Case 2: The Strictness of the "Guest List" (Topic Inclusion Threshold)

Imagine a party where a guest can belong to multiple groups (e.g., "The Science Club" and "The Art Club").

• Low Threshold (Too Lenient): You let everyone into every club. Suddenly, the "Science Club" and "Art Club" are the exact same group of people. The map becomes a giant, blurry blob where you can't tell anyone apart.
• High Threshold (Too Strict): You are so picky that you only let people in if they are 100% sure they belong. Most people get kicked out. The party is empty, and you can't see any connections.
• Medium Threshold (Just Right): You let people in if they have a decent connection. Now you can see that the "Science Club" has a few artists, and the "Art Club" has a few scientists. The map is clear and interesting.

The Lesson: You need to find a "sweet spot" for how strict you are about assigning topics. If you are too loose or too strict, the map becomes useless.

Case 3: The Stress Test (Scale)

Finally, they threw the entire library (24,000 essays, nearly half a million sentences) at the robot.

• The Result: The robot didn't crash! It successfully identified the 7 main themes of the 7 different writing assignments (like "Electoral College" or "Driverless Cars") without a human ever telling it what those themes were.
• The Discovery: It even found subtle differences between high-scoring students and low-scoring students. For example, high-scoring students connected "driverless cars" and "pollution" more often in their thinking than low-scoring students did.

---

Why This Matters (The Big Picture)

Before this, analyzing the "thinking patterns" of thousands of students was impossible because it required armies of human coders.

TopicENA changes the game:

1. It's Scalable: It can handle massive amounts of data (like a whole school district's essays) in a fraction of the time.
2. It Shifts the Human Role: Instead of humans doing the boring, repetitive work of tagging every sentence, humans now act as conductors. They set the parameters (like the bin sizes and guest list rules) and then interpret the beautiful, complex maps the AI generates.
3. It's More Accurate: Unlike older methods that just counted words, this AI understands context. It knows that "bank" in a river context is different from "bank" in a money context.

The Bottom Line

This paper introduces a way to use AI to map the "thought structures" of huge groups of people. It's like upgrading from a hand-drawn sketch of a single neighborhood to a satellite view of an entire continent, allowing educators and researchers to finally see the big picture of how people learn and think.

Technical Summary

Here is a detailed technical summary of the paper "TopicENA: Enabling Epistemic Network Analysis at Scale through Automated Topic-Based Coding."

1. Problem Statement

Epistemic Network Analysis (ENA) is a learning analytics method used to visualize and statistically compare the relational structures of concepts within discourse data. However, traditional ENA faces a critical scalability bottleneck:

• Reliance on Manual Coding: Standard ENA requires expert researchers to manually code text segments into specific conceptual categories.
• Limited Scale: This manual process restricts ENA to small datasets (e.g., hundreds of utterances or a few thousand posts), making it impractical for large-scale educational corpora (e.g., tens or hundreds of thousands of essays).
• Gap in Methodology: While previous attempts to automate ENA using topic modeling (e.g., LDA) exist, they often lack generalizable guidance on how specific modeling choices (granularity, thresholds) impact network outcomes across different data scales.

2. Methodology: The TopicENA Framework

The authors propose TopicENA, a framework that integrates BERTopic (a neural topic modeling approach) with ENA to automate semantic coding while preserving the ability to model structural associations.

Core Pipeline Stages:

1. Neural Topic Induction (BERTopic):

   • Instead of a "bag-of-words" approach (like LDA), TopicENA uses Sentence-BERT (SBERT) embeddings to capture contextual semantics.
   • It employs UMAP for dimensionality reduction and HDBSCAN for density-based clustering to identify latent topics.
   • Topics are summarized using class-based TF-IDF.
   • Multi-Topic Attribution: Unlike traditional one-hot encoding, documents are treated as probabilistic distributions across multiple topics, reflecting the multi-dimensional nature of discourse.

2. Topic-to-ENA Encoding:

   • Topic Inclusion Threshold (topic_inclusion_th): A critical parameter where a topic is considered "present" in a document if its probability score exceeds a set threshold. This allows a single text unit to co-occur with multiple topics, a prerequisite for building meaningful co-occurrence networks.

3. Epistemic Network Construction:

   • The encoded topic data is processed via an R-based ENA pipeline to generate network graphs representing the relational structure among topics.

Experimental Design

The study utilized the ASAP 2.0 dataset (24,728 argumentative essays, segmented into 457,002 utterances). The experiments were divided into three cases to test sensitivity and scalability:

• Case 1 (Granularity): Tested the effect of topic granularity (controlled by UMAP/HDBSCAN parameters like n_neighbors) on small vs. medium datasets (Assignments 4 & 5).
• Case 2 (Thresholds): Tested the effect of the topic inclusion threshold (Low: 0.01, Medium: 0.05, High: 0.10) on network density and interpretability.
• Case 3 (Scalability): Applied TopicENA to the full dataset (all 7 assignments) to test its ability to recover task-specific semantic structures and compare high-performing vs. low-performing student groups.

3. Key Contributions

1. Framework Proposal: Introduces TopicENA, a generalizable tool that replaces manual expert coding with automated, context-aware semantic units, enabling ENA on datasets orders of magnitude larger than previously possible.
2. Methodological Guidance: Systematically demonstrates how topic granularity and inclusion thresholds must be calibrated based on data scale and distribution, rather than using fixed parameters.
3. Shift in Researcher Role: Reorients the researcher's role from labor-intensive instance-level annotation to higher-level interpretation of emergent structural patterns and theoretical sense-making.
4. Validation of Neural Topic Modeling: Establishes BERTopic as superior to LDA for ENA in this context due to its ability to preserve contextual semantics and handle short texts (sentences/utterances) without losing coherence.

4. Key Results

Case 1: Sensitivity to Topic Granularity

• Small Datasets: Fine-grained topics (small n_neighbors) led to sparse co-occurrence and overlapping networks, reducing interpretability. Medium granularity provided the best balance.
• Large Datasets: Coarse-grained topics (large n_neighbors) were preferable. They maintained sufficient co-occurrence strength and stability. Overly fine granularity on large data resulted in noise and dense, uninterpretable networks.
• Conclusion: Granularity must align with data scale (Coarse for large data, Fine/Medium for small data).

Case 2: Sensitivity to Topic Inclusion Threshold

• Low Threshold (0.01): Caused excessive multi-topic assignment, resulting in overly dense networks that masked group differences.
• High Threshold (0.10): Resulted in sparse networks with insufficient structural information.
• Medium Threshold (0.05): Achieved the optimal balance, allowing clear structural differences between high- and low-performing groups to emerge.
• Conclusion: Thresholds should be determined by the distribution of topic probabilities (often in the middle range) rather than fixed arbitrarily.

Case 3: Scalability and Structural Recovery

• Task Recovery: BERTopic successfully identified 7 distinct topics that perfectly aligned with the 7 distinct writing assignments in the dataset (e.g., "Electoral College," "Driverless Cars," "Pollution") without using assignment labels during modeling.
• Group Differentiation: The framework successfully distinguished between high-performing (scores 4-6) and low-performing (scores 1-3) groups.
  • Example: High-performing groups showed stronger co-occurrence between "driverless" and "pollution" concepts, whereas low-performing groups did not.
• Conclusion: TopicENA is robust enough to handle ~457k utterances, recovering task-level semantic structures and highlighting subtle group differences.

5. Significance and Implications

• Scalability: TopicENA breaks the "manual coding bottleneck," allowing ENA to be applied to massive educational datasets (e.g., entire course cohorts) that were previously intractable.
• Contextual Fidelity: By using BERTopic, the method captures semantic nuances and context that traditional bag-of-words models (LDA) miss, leading to more accurate conceptual representations.
• Methodological Transparency: The study argues that knowledge structures in automated ENA do not "emerge" naturally but are the result of calibrated analytical conditions (granularity + threshold). This makes the process more transparent, reproducible, and falsifiable than manual coding, where theoretical assumptions are often embedded implicitly in the coding scheme.
• Future Application: The framework opens new avenues for large-scale learning analytics, AI-assisted education research, and cross-lingual discourse analysis, shifting the focus from "what concepts exist" to "how concepts relate at scale."

Limitation Note: The study acknowledges that because it does not use manual coding as a ground truth, traditional inter-rater reliability metrics are not applicable. Instead, validity is established through the stability of structural patterns across different parameter settings and the logical coherence of the recovered topics.