💬 NLP

Prompt Stability Scoring for Text Annotation with Large Language Models

This paper addresses the vulnerability of large language model text annotation to prompt variations by proposing a general framework called the Prompt Stability Score (PSS), which adapts traditional reliability metrics to diagnose stability issues and includes a Python package for practical implementation.

Original authors: Christopher Barrie, Elli Palaiologou, Petter Törnberg

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christopher Barrie, Elli Palaiologou, Petter Törnberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Is Your AI "Fickle"?

Imagine you hire a very smart, but slightly nervous, assistant to sort a pile of letters into "Yes" and "No" envelopes. You give them a specific instruction: "If the letter mentions money, put it in the 'Yes' pile."

You ask them to do this 30 times.

Scenario A: They put the same letters in the same piles every single time.
Scenario B: On the 1st try, they put a letter about "savings" in the "Yes" pile. On the 2nd try, they put that same letter in the "No" pile. On the 3rd try, they put it in "Yes" again.

Even if the assistant is usually right, Scenario B is a problem. It means their decision-making is unstable. If you can't trust them to be consistent with the exact same instructions, you certainly can't trust them if you change the wording slightly (e.g., "If the letter talks about cash...").

This paper is about building a test to see if your AI assistant is stable (Scenario A) or fickle (Scenario B) before you let it do any real work.

The Problem: AI is Sensitive to "Word Salad"

The authors explain that Large Language Models (AI) don't work like a calculator. They don't look up a fixed answer in a book. Instead, they predict the next word one by one, like a person guessing what comes next in a sentence.

Because of this, two things can mess up their consistency:

The "Same Prompt" Problem: Even if you type the exact same instructions twice, the AI might give a slightly different answer because of tiny, invisible random factors in its computer brain (like a coin flip happening in the background).
The "Rephrasing" Problem: If you change your instructions just a little bit—like saying "Classify this" instead of "Sort this"—the AI might get confused and give a totally different answer.

The paper argues that accuracy isn't enough. An AI might get the right answer 90% of the time, but if it flips its answer every time you tweak the prompt, your research results are unreliable.

The Solution: The "Prompt Stability Score" (PSS)

The authors created a tool (a Python package called promptstability) that acts like a stress test for your AI instructions.

Think of it like testing a bridge:

Intra-Prompt Stability (The "Repetition" Test): You run the exact same instruction 30 times on the same data. The tool checks: Did the AI give the same answer every time? If the answers jump around, the bridge is shaky.
Inter-Prompt Stability (The "Rephrasing" Test): You take your instruction and ask a different AI to rewrite it in 10 different ways (e.g., "Sort these," "Categorize these," "Label these"). Then you run all 10 versions on the same data. The tool checks: Did the AI still agree with itself, even though the words changed?

The tool gives you a score (called Krippendorff's alpha, which is just a fancy way of saying "agreement score").

High Score (Close to 1.0): Great! Your instructions are robust. The AI is consistent.
Low Score (Below 0.8): Warning! Your instructions are too fragile. A tiny change in wording breaks the system.

What They Found (The "Stress Test" Results)

The researchers tested this on six different real-world datasets (like US political tweets, UK party manifestos, and news articles). Here is what they discovered:

Simple is Stable: When the task was easy and the data was clear (like identifying if a tweet was from a Republican or Democrat), the AI was very stable. It didn't matter if you rephrased the prompt; it gave the same answer.
Complex is Fragile: When the task was hard or the data was messy (like deciding if a tweet uses "populist" language, or sorting survey answers into 12 very similar categories), the AI became unstable. Small changes in the prompt caused the AI to flip-flop.
The "Format" Trap: Sometimes the AI wasn't actually confused; it just forgot to follow the formatting rules (like writing a paragraph instead of a number). When the researchers filtered out these "sloppy" answers, the stability scores went up. This taught them that sometimes the problem isn't the AI's brain, but its inability to follow strict formatting rules.
Model Matters: They compared a powerful AI (GPT-4) with a smaller, open-source one. The powerful one was much more stable. This means the "fickleness" isn't always your fault; sometimes you just need a smarter AI.

The "Playbook": What Should You Do?

The paper gives a simple guide for researchers:

Step 1: Run the Stability Test. Before you spend money or time validating your results, run the promptstability tool.
Step 2: Check the Score.
- If the score is high: You are good to go. Your pipeline is robust.
- If the score is low: Stop! Do not trust your results yet.
Step 3: Fix the Problem.
- Is the AI ignoring formatting? Fix the prompt to be stricter.
- Is the task too vague? Make your instructions clearer.
- Is the data too messy? Maybe this specific task is too hard for an AI to handle consistently.
- Is the AI too weak? Try a more powerful model.

The Bottom Line

You cannot assume that just because an AI gives you an answer, that answer is reliable. This paper provides a checklist to ensure that your AI isn't just guessing randomly or reacting too sensitively to how you phrase your questions.

Stability is the foundation of trust. If your AI can't agree with itself when you ask the same question in slightly different ways, you can't trust the results it gives you for your research.

Technical Summary: Prompt Stability Scoring for Text Annotation with Large Language Models

1. Problem Statement

Researchers increasingly utilize Large Language Models (LMs) for text annotation, often employing zero-shot or few-shot approaches where a "prompt" instructs the model to classify data. While these methods offer scalability, they introduce a critical reliability gap: LM outputs are often sensitive to minor, semantically equivalent variations in prompt wording or stochastic decoding parameters.

Current literature lacks a routine, automated reliability check for these pipelines. A single prompt yielding high accuracy does not guarantee that a slightly rephrased prompt will produce the same labels. If outputs shift across semantically similar prompts or repeated runs, the classification pipeline is "fragile," undermining reproducibility and rendering downstream estimates unreliable. The authors argue that stability is a necessary precondition for accuracy and reproducibility; a pipeline that is unstable under minor perturbations should not proceed to costly validation steps like manual coding.

2. Methodology

The paper introduces the Prompt Stability Score (PSS), an automated diagnostic framework that quantifies the agreement of an annotation pipeline under controlled perturbations. The framework treats repeated model runs and semantically equivalent prompt variants as analogues to human raters in inter- and intra-coder reliability studies.

Core Components

Intra-Prompt Stability (Intra-PSS): Measures the consistency of a model when the same prompt is run repeatedly on the same items. This captures run-to-run variability caused by decoding stochasticity (e.g., temperature sampling) and non-deterministic hardware execution.
Inter-Prompt Stability (Inter-PSS): Measures the consistency of a model when semantically equivalent prompt variants are used. The framework generates these variants using an external paraphrasing model (PEGASUS) at varying decoding temperatures. Low temperatures produce conservative rephrasings, while higher temperatures act as stress tests to reveal where the pipeline breaks down.

Statistical Operationalization

Metric: The authors use Krippendorff's $\alpha$ ( $K\alpha$ ) as the primary agreement statistic. This choice allows for the handling of nominal, ordinal, and interval data, and corrects for chance agreement.
Uncertainty Estimation: Nonparametric bootstrapping is employed to generate confidence intervals for stability estimates.
Implementation: The authors provide an open-source Python package, promptstability, which automates the generation of prompt variants, execution of repeated runs, and calculation of stability scores.

Experimental Design

The framework was validated across six datasets and twelve distinct outcomes, generating approximately 3.1 million annotations (roughly 300 million input tokens). The datasets span diverse social science contexts (e.g., US Senator tweets, UK party manifestos, NYT articles, survey responses) and vary in:

Outcome Complexity: Binary vs. multi-class vs. continuous scales; granular vs. coarse categories.
Data Complexity: Information-rich vs. information-poor inputs (e.g., short vs. long texts).
Model Capability: Comparisons between different LMs (e.g., GPT-3.5, GPT-4o, DeepSeek-R1).

3. Key Results

The empirical analysis maps instability to four primary sources: criteria ambiguity, outcome complexity, data complexity, and model capability.

General Stability Patterns: Most benchmark tasks exhibit high intra-PSS (repeatability under a fixed prompt), converging quickly to high agreement levels. However, inter-PSS is more variable; stability often declines as prompt paraphrases become more diverse.
Criteria Ambiguity: Several tasks (e.g., stance detection, "most important issue" classification) showed rapid drops in inter-PSS even at low paraphraser temperatures. This indicates that the coding criteria are fragile to minor wording changes, suggesting ambiguous prompt design or ill-defined constructs.
Data and Outcome Complexity:
- Data Complexity: Tasks with low signal-to-noise ratios (e.g., predicting voter choice from limited profile data) showed poor stability even with fixed prompts.
- Outcome Complexity: Expanding label sets to more granular, overlapping categories (e.g., 12 vs. 6 issue categories) reduced stability, even when the input data remained constant.
Model Capability: Comparisons between GPT-4o and DeepSeek-R1:8b revealed that model choice significantly impacts stability. The smaller model exhibited lower intra-PSS and faster degradation in inter-PSS, demonstrating that instability can stem from model limitations rather than just prompt design.
Diagnostic Utility: The authors demonstrate that filtering for correct output formats (e.g., removing malformed integers) can distinguish between formatting errors and genuine classification instability. For some datasets (e.g., "Most Important Issue"), filtering resolved instability; for others (e.g., "Synthetic Short"), it did not, indicating a fundamental task difficulty.
Cost Feasibility: The authors estimate that running these stability diagnostics on a typical sample size costs between $3 and $150 (using GPT-3.5), suggesting the approach is a feasible early-stage diagnostic rather than a prohibitive validation step.

4. Key Contributions

Conceptual Framework: The paper reframes LM annotation reliability through the lens of inter- and intra-coder reliability, defining "prompt stability" as a distinct, measurable property necessary for reproducibility.
Automated Diagnostic Tool: The introduction of the Prompt Stability Score (PSS) and the accompanying promptstability package provides researchers with a standardized, automated method to stress-test annotation pipelines before investing in ground-truth validation.
Empirical Characterization: By testing across 12 outcomes and 6 datasets, the paper provides a large-scale characterization of when prompt-based pipelines are stable versus fragile, identifying specific failure modes related to data signal, label granularity, and model choice.
Practical Playbook: The authors offer a "PSS Playbook" for interpreting results, advising researchers on when to refine prompts, clarify constructs, switch models, or adjust decoding settings based on stability scores.

5. Significance and Claims

The paper claims that high accuracy from a single prompt is insufficient for reproducibility. If an annotation pipeline is fragile to minor perturbations (rephrasings or repeated runs), any downstream estimates inherit this fragility, regardless of the model's average performance.

The authors position PSS not as a measure of truth or accuracy, but as a precondition for validity. They argue that researchers should treat stability diagnostics as an essential early step in the research workflow. A low PSS serves as a warning to revisit prompt wording, construct definitions, data quality, or model selection before committing resources to expensive validation procedures. The paper concludes that while stability does not guarantee correctness, it is the necessary foundation upon which reproducible LM-based social science research must be built.