💬 NLP

WikiSTAR: A System for Shedding Light on the Hidden History of Scientific Wikipedia Articles

WikiSTAR is an interactive system that leverages an LLM classifier and a multi-label taxonomy to filter routine edits from Wikipedia's revision history, enabling experts to visualize and analyze the evolution of scientific knowledge at unprecedented granularity.

Original authors: Omer Ehrlich, Nitzan Barzilay, Rona Aviram, Tom Hope

Published 2026-07-15✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Omer Ehrlich, Nitzan Barzilay, Rona Aviram, Tom Hope

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Wikipedia as a giant, bustling library where millions of volunteers are constantly rewriting the books. For science articles, this library holds a secret history: a record of how our understanding of the universe, viruses, and computers has changed over time. But there's a catch. The library is so noisy that the important updates—like the discovery of a new star or a breakthrough in AI—are completely buried under a mountain of tiny, boring changes, like fixing a typo or changing a font color. It's like trying to find a specific new chapter in a book that's been rewritten 18,000 times, where 99% of the changes are just someone rearranging the furniture.

Enter WIKISTAR (Scientific Tracking of Article Revisions). Think of WIKISTAR as a super-smart, time-traveling detective that can sift through that mountain of noise to find the gold.

The Detective's Toolkit

WIKISTAR doesn't just read the text; it understands the science behind the edits. It uses a special set of rules (called a "taxonomy") designed by science experts. This rulebook has ten specific categories for what counts as a "scientific" change, such as:

Adding a new piece of scientific information.
Introducing a new technical term.
Changing the story or "narrative" of how a science concept is explained.
Adding or removing references to academic studies.

If an edit doesn't fit these scientific categories (like just fixing a spelling mistake), WIKISTAR ignores it. This is crucial because the paper explicitly argues against the idea that all edits are created equal. Previous tools looked at the whole page or even individual words, but WIKISTAR zooms in on sections (like specific chapters of a book). The authors found that looking at the whole page is too messy, and looking at single sentences loses too much context. The "section" is just the right size to see what's actually happening.

How It Works: The Magic Pipeline

Here is the step-by-step magic trick WIKISTAR performs:

The Split: It takes a Wikipedia article and breaks every single version of it into its individual sections.
The Match: It plays a high-stakes game of "spot the difference." It matches a section from today with the version of that same section from yesterday, even if the editors renamed the section or split it in half.
The Classification: It uses a powerful AI brain (an LLM) to read the "before" and "after" versions. It asks: "Did this change add new science? Did it remove a fact? Did it change the story?"
The Dashboard: Finally, it turns all those answers into colorful, interactive maps and charts.

What the Detective Found

The paper doesn't claim WIKISTAR has solved the mystery of science forever. Instead, it suggests that this tool opens up a whole new way to study history. To test this, the team built a "training gym" called WIKISTAR-BENCH. They gathered 1,387 real examples of section edits from Biology, Computer Science, and Neuroscience and had human experts label them.

When they tested their AI detective against these human experts, the results were promising but not perfect. The AI got about 82% of the big-picture classifications right (a score called macro-F1). It was really good at spotting obvious things, like adding a new academic reference (getting 88% right), but it struggled a bit more with tricky, interpretive changes, like deciding if the "story" of the science had changed (scoring 61%). This tells us that while the tool is powerful, it's not a magic wand that gets everything right every time; it still needs human guidance for the most complex judgments.

The "Time Machine" for Experts

To see if this tool was actually useful, the researchers invited three real experts to play with it: a Wikipedia editor who is also a physics PhD student, a science journalist, and a philosopher of science.

The results were glowing. The experts said WIKISTAR showed them patterns they could never have found by reading the history manually.

One expert looked at the "Artificial Intelligence" article and saw how the focus shifted from broad definitions to deep technical details, and finally to societal impacts, all in one glance.
Another looked at the "Chaos Theory" article and realized that the structure of the article had changed in ways that no other tool could show, calling it a "dream" for researchers.
A third expert, studying vaccines, found a specific moment where a huge amount of new scientific content was added. The tool let them zoom in, read a summary of those changes, and then dive straight into the raw text to verify it. They described it as "almost like guided reading."

The Bottom Line

WIKISTAR is a system that suggests we can finally see the hidden history of science on Wikipedia. It turns a chaotic mess of 18,000 edits into a clear story of how scientific ideas grow, change, and sometimes get corrected. It doesn't replace the need for human experts, but it gives them a super-powered microscope to see the evolution of knowledge in a way that was previously impossible. The paper releases the tool, the code, and the dataset so that anyone curious about how science is written can start exploring their own time machine.

Technical Summary of WIKISTAR: A System for Shedding Light on the Hidden History of Scientific Wikipedia Articles

Problem Statement
Wikipedia serves as a primary source for public scientific understanding, and its openly accessible revision history offers a unique longitudinal record of how scientific knowledge evolves. However, the sheer volume of routine edits (e.g., formatting, rephrasing) obscures scientifically meaningful changes. Isolating these significant edits manually is highly labor-intensive, leaving the scientific history of articles hidden and inaccessible. While prior work has classified general edits or visualized authorship conflicts, these approaches operate at the whole-page or token level and do not focus on the specific scientific significance of changes. There is a need for a system that can automatically detect, quantify, and contextualize scientific textual changes at a granular level to reveal how scientific concepts are introduced, debated, and refined over time.

Methodology
The authors present WIKISTAR (Scientific Tracking of Article Revisions), a system designed to extract, classify, and visualize section-level scientific edits. The methodology consists of three primary stages:

Section Identification and Linking:
- The system retrieves article revisions via the MediaWiki API and parses raw wikitext into constituent sections using regex-based parsers.
- To create "section edit pairs" (the current version and the immediately preceding version of a specific section), the system matches sections across revisions. Since sections are not stable (they may be renamed, split, or merged), the system searches backward through revisions and computes two similarity measures: title similarity (normalized Levenshtein distance) and content similarity (cosine similarity over embeddings). A section is matched to the most recent candidate exceeding a similarity threshold.
Scientific Edit Classification:
- The core task is defined as scientific edit classification: assigning labels to section edit pairs based on a taxonomy of ten expert-designed, scientifically meaningful edit types (e.g., New Scientific Information, Technical Terms Added, Change in Scientific Narrative, Academic References Added) plus a Non-Scientific Edit label.
- The system employs Large Language Models (LLMs) as classifiers. The authors evaluated seven models (including GPT-5 variants, LLaMA 3.3, and Qwen3-Next) against a human-annotated benchmark.
- Prompt engineering refinements, such as asking the model to first determine if an edit is scientific and flagging common definition mistakes, significantly improved performance.
Interactive Visualization:
- The classified data is presented through interactive views allowing users to explore history at varying granularities:
  - History Overview: An LLM-generated chronological summary of major scientific claims and narrative shifts.
  - Trends Over Time: An interactive line plot showing the frequency of edit types over time, with the ability to filter by section and generate text summaries of specific time periods.
  - Section Comparison: A heatmap comparing edit type trends across different article sections.
  - Section Evolution: A view showing the lifecycle of sections (appearance, renaming, editing).
  - Edit Table: A filterable table of all edits with metadata.

Key Contributions

WIKISTAR System: A novel system for tracking the evolution of scientific content in Wikipedia articles, bridging the gap between raw revision logs and actionable scientific insights.
Scientific Edit Classification Task: The introduction of a specific classification task for Wikipedia sections, grounded in a 10-label taxonomy designed by scientific Wikipedia experts.
WIKISTAR-BENCH: The release of the first human-annotated dataset for this task, containing 1,387 section-level edit pairs across three domains (Biology, Computer Science, and Neuroscience). The dataset demonstrates high inter-annotator agreement (mean pairwise Cohen's $\kappa = 0.89$ ).
Empirical Evaluation: A comprehensive comparison of seven LLMs as classifiers, establishing GPT-5-mini and GPT-5.4 as the top performers (Macro-F1 scores of 0.83 and 0.82, respectively).

Results

Classification Performance: On WIKISTAR-BENCH, the best-performing model (GPT-5-mini) achieved a Macro-F1 of 0.83 and a Micro-F1 of 0.86. Performance varied by label type: the model excelled on objective labels with explicit textual signals (e.g., Wikilink Added F1=0.89) but struggled with interpretive labels (e.g., Change in Scientific Narrative F1=0.61), where human agreement was also lower.
Domain Consistency: Performance remained stable across Biology, Computer Science, and Neuroscience, indicating the taxonomy and prompts generalize well without overfitting to a single domain.
User Study: In a study with three experts (a Wikipedia editor/physicist, a science journalist, and a philosopher of science), participants rated the system highly on a 5-point Likert scale (average ratings of 4.33–5.00). Participants reported that WIKISTAR surfaced patterns they could not easily discover manually, prompted new research questions, and enabled analyses previously considered impractical due to the volume of data.

Significance
The paper claims that WIKISTAR makes the scientific edit history of Wikipedia accessible to researchers, journalists, and scientists at a scale previously impossible. By moving from manual, labor-intensive close reading of revision histories to automated, section-level analysis, the system enables the study of knowledge evolution as a "rich longitudinal record." The authors position WIKISTAR as the first tool to operate at the section level specifically for scientific significance, allowing users to trace how scientific concepts develop, where attention concentrates, and how narratives shift over time. The release of the benchmark and code aims to facilitate further research into the representation of science in online encyclopedias.