💻 computer science

A Methodological Guide on Using Large Language Models for Text Annotation in the Social Sciences and Humanities with Python and R

This paper provides a comprehensive, step-by-step methodological guide with Python and R code snippets to help social science and humanities researchers effectively leverage large language models for text annotation while addressing challenges related to accessibility, error bias, prompt design, and statistical integration.

Original authors: Qixiang Fang, Javier Garcia Bernardo, Erik-Jan van Kesteren

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qixiang Fang, Javier Garcia Bernardo, Erik-Jan van Kesteren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a historian trying to understand the dreams and goals of thousands of students by reading their diary entries. In the old days, you would have to read every single entry yourself, highlight the important parts, and write a summary. This is text annotation. It's the process of labeling data so a computer can understand it later. But reading thousands of diaries by hand is slow, expensive, and you might get tired and make mistakes.

Enter Large Language Models (LLMs). Think of an LLM not as a magic robot that knows everything, but as a super-fast, incredibly well-read intern. This intern has read almost everything on the internet. If you give them a diary entry and ask, "Is this student's goal specific or vague?", they can answer in a split second.

However, this intern is new. They can be confused, they might hallucinate facts, and they might not understand your specific rules. This paper is a user manual for hiring this intern to do your research work without getting fired (or ruining your data).

Here is the guide broken down into simple, everyday concepts:

1. The Intern's Brain (How LLMs Work)

The paper explains that LLMs aren't actually "thinking" like humans. Imagine a giant game of "Guess the Next Word."

The Analogy: If you say, "The sky is...", the intern guesses "blue" because that's what usually comes next.
The Twist: This intern has read billions of books. So, if you ask them to "act like a teacher grading a student," they use all that reading to guess what a teacher would say next. They aren't calculating truth; they are calculating the most likely answer based on patterns.

2. The Job Interview (Setting Up the Project)

Before you hire the intern, you need to know if the job is even possible.

The Reality Check: If you ask the intern to guess a student's personality from a single sentence, they will fail. The data just isn't there. But if you ask them to count how many times a student mentions "math," that's easy.
The Gold Standard: You can't just trust the intern blindly. You need a Reference Key. This means you (or a human expert) must grade a small pile of diaries perfectly first. This is your "Gold Standard." You will use this to check if the intern is doing a good job.

3. Writing the Instructions (Prompt Engineering)

This is the most important part. If you tell the intern, "Grade these goals," they will be confused. You need to be a strict manager.

Role-Playing: Tell them, "You are a strict university professor."
Clarity: Don't say "be nice." Say "Give a score of 0, 1, or 2 based on these exact rules."
Examples: Show them a "bad" example and a "good" example before they start. This is called "Few-Shot Prompting" (giving them a cheat sheet).
Chain of Thought: Ask them to "think out loud" before giving the final score. It's like asking the intern to show their math work. This reduces mistakes.

4. The Trial Run (Evaluation)

Once the intern starts grading, you check their work against your Gold Standard.

The Score: You calculate an agreement score. Did the intern agree with the human expert?
The Loop: If the intern is wrong, don't fire them yet. Look at why they were wrong. Did they misunderstand the rule? Did the rule itself make sense? You tweak the instructions (the prompt) and try again.
The Trap (Overfitting): Imagine you practice a test with the exact same questions you will take on the real exam. You'll get a perfect score, but you haven't actually learned anything. The paper warns you not to use your "Gold Standard" data to train the intern too much. You need a separate "Test Set" that the intern has never seen to see if they are actually smart or just memorized the practice questions.

5. The Final Exam (Downstream Analysis)

Now you have thousands of graded diaries. You want to run a statistical analysis (e.g., "Do specific goals lead to better grades?").

The Problem: The intern isn't perfect. They make mistakes.
Random Noise: Sometimes they guess wrong randomly. This makes your results look "fuzzier" (harder to find a real pattern).
Systematic Bias: Sometimes they are consistently wrong in the same way (e.g., they always give high scores to students who use big words). This is dangerous because it can trick you into thinking there is a relationship where there isn't one.
The Fix: The paper introduces special math tools (like a "correction filter") that can look at the intern's mistakes and adjust your final results to account for them. It's like a detective correcting a witness's testimony based on known biases.

6. Scaling Up (Efficiency and Safety)

Batching: Instead of asking the intern one question at a time (which is slow and expensive), you give them a stack of 1,000 papers at once.
Privacy: If your data is secret (like medical records), you can't send it to a public cloud intern. You might need to run a "local" version of the intern on your own computer.
Documentation: Keep a diary of exactly what instructions you gave the intern and which version of the software you used. If you don't, no one else can repeat your experiment.

The Big Takeaway

Using an LLM for research is like hiring a very fast, very smart, but slightly unreliable assistant.

If you treat them like a magic oracle, you will get bad science.
If you treat them like a tool that needs clear instructions, constant checking, and statistical correction, you can do research faster and on a scale that was previously impossible.

The paper's main message: Don't just trust the machine. Measure the machine, correct for its mistakes, and always keep a human in the loop.

Based on the paper "A Methodological Guide on Using Large Language Models for Text Annotation in the Social Sciences and Humanities with Python and R" by Fang, Bernardo, and van Kesteren, here is a detailed technical summary:

1. Problem Statement

While Large Language Models (LLMs) offer a transformative potential for automating text annotation in Social Sciences and Humanities (SSH) research—replacing costly, slow, and inconsistent manual labeling—researchers face three primary barriers:

Accessibility: The rapid pace of model development and the technical requirements of APIs and programming languages (Python/R) intimidate non-technical SSH researchers.
Methodological Rigor: Researchers often lack a structured workflow to design prompts, evaluate quality, and avoid overfitting during the iterative development of annotation tasks.
Statistical Validity: A critical, often overlooked issue is the impact of annotation errors (both random and systematic) on downstream statistical analyses. Even high-accuracy annotations can bias regression coefficients, inflate/deflate p-values, and lead to incorrect substantive conclusions if measurement error is not accounted for.

2. Methodology

The paper proposes a comprehensive, end-to-end methodological framework for integrating LLMs into SSH research workflows. It utilizes a running example of annotating student-chatbot dialogues to evaluate "goal specificity" (scoring 0–2) to demonstrate technical implementation in both Python and R.

The methodology is structured into four main phases:

A. Project Setup and Conceptual Foundation

Construct Validity: Emphasizes that LLMs cannot resolve conceptual ambiguity; the research question and data must contain sufficient signal.
Gold-Standard Data: Requires a held-out subset of human-annotated data (gold labels) to serve as a reference for evaluation. This subset must be strictly separated from prompt development data to prevent data leakage.
Expectation Management: Researchers must set realistic agreement targets based on human inter-annotator agreement, acknowledging that LLMs are imperfect measurement instruments.

B. Prompt Engineering and Execution

Prompt Design: Introduces techniques to improve output quality, including:
- Role-based prompting: Assigning specific expert personas.
- Clarity: Using explicit verbs (e.g., "classify" vs. "analyze").
- Few-shot prompting: Providing labeled examples within the prompt.
- Chain-of-thought: Requiring step-by-step reasoning before final scoring.
Structured Output: Advocates for enforcing structured formats (e.g., JSON via pydantic in Python or ellmer in R) to extract numeric scores directly, avoiding error-prone text parsing.
Environment: Recommends starting with cloud notebooks (Google Colab, RStudio Cloud) and using APIs (e.g., OpenAI) for accessibility, while noting privacy considerations (GDPR) for local model deployment.

C. Evaluation and Iteration (The Explore–Select–Evaluate Framework)

To address overfitting (where prompts are tuned too specifically to a validation set, yielding optimistic performance estimates), the authors propose a three-stage data splitting framework:

Explore (10–20% of data): Iterative prompt design, error inspection, and refinement.
Select (40–60% of data): Comparing a fixed set of candidate prompts to choose the best performer. No further modifications are allowed here.
Evaluate (20–30% of data): Final, unbiased performance estimation on a completely held-out test set.

Metrics: Uses appropriate agreement statistics (Cohen's $\kappa$ , Krippendorff's $\alpha$ for categorical/ordinal data; ICC, MAE for continuous data) alongside qualitative error analysis.

D. Downstream Statistical Analysis

Error Classification: Distinguishes between Random Error (reduces reliability, attenuates effect sizes) and Systematic Error (biases estimates, distorts validity).
Correction Methods: Introduces statistical techniques to correct for measurement error in regression and other estimators. The paper highlights specific software packages:
- Python: ppi_py (Prediction-Powered Inference).
- R: postpi, pspa, dsl, ipd.
- These tools use the gold-standard subset to model the relationship between LLM labels and true labels, correcting downstream parameter estimates.

3. Key Contributions

Unified Workflow: Provides the first integrated guide covering the entire lifecycle of LLM annotation, from prompt design to statistical correction, specifically tailored for SSH researchers.
Code Implementation: Offers reproducible, side-by-side code snippets in Python and R for every step, including API interaction, structured output parsing, and error correction.
The Explore–Select–Evaluate Framework: A novel adaptation of machine learning validation principles to prompt engineering, specifically designed to prevent selection bias and overfitting in non-training-based LLM tasks.
Statistical Integration: Bridges the gap between NLP and quantitative social science by explicitly demonstrating how to correct for annotation error in regression models, preventing biased scientific conclusions.
Practical Best Practices: Offers concrete advice on cost management (batching, caching), reproducibility (version control, metadata), and ethical considerations (data privacy, leakage).

4. Results (Illustrative)

While the paper is a methodological guide rather than an empirical study with new dataset results, it demonstrates the efficacy of its framework through the running example:

Initial Performance: An initial prompt yielded a Krippendorff's $\alpha$ of 0.222 (low agreement) with the gold standard.
Iterative Refinement: By analyzing disagreement patterns (e.g., the LLM confusing "specificity" with "actionability") and refining the prompt instructions, the agreement improved significantly.
Final Performance: The refined prompt achieved a Krippendorff's $\alpha$ of 0.894, demonstrating that systematic prompt engineering can yield high-quality annotations comparable to human experts.
Statistical Impact: The paper illustrates via simulation that ignoring measurement error (specifically random error) can attenuate regression slopes (e.g., from $\beta=0.50$ to $\beta=0.29$ ), while systematic error can shift intercepts, validating the necessity of the proposed correction tools.

5. Significance

This paper is significant because it moves the discourse on LLMs in SSH from "can we use them?" to "how do we use them rigorously?"

Democratization: It lowers the barrier to entry for SSH researchers by providing clear, language-agnostic code and conceptual explanations that bridge the gap between computer science and social science.
Scientific Integrity: By emphasizing that LLMs are "measurement instruments" rather than perfect truth-seekers, it enforces a culture of transparency, error quantification, and statistical correction.
Future-Proofing: The framework is designed to be robust against model updates and API changes, ensuring that research workflows remain reproducible and valid as the technology evolves.
Paradigm Shift: It establishes a new standard for text-based research, where large-scale automation is paired with rigorous psychometric validation, allowing for the analysis of datasets previously too large for manual coding.