💬 NLP

Refining and Reusing Annotation Guidelines for LLM Annotation

This paper proposes and empirically validates an iterative moderation framework that systematically refines and reuses annotation guidelines to align Large Language Models with gold-standard benchmarks across biomedical named entity recognition tasks, confirming the efficacy of guideline integration, reasoning-optimized models, and moderation under minimal supervision.

Original authors: Kon Woo Kim, Jin-Dong Kim, Akiko Aizawa

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kon Woo Kim, Jin-Dong Kim, Akiko Aizawa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant, super-fast robot assistant how to read medical texts and highlight specific diseases. The robot is incredibly smart; it knows what a "heart attack" or "diabetes" is. However, when you ask it to do a specific job, it often makes tiny mistakes because it doesn't know the exact, strict rules your team uses.

In the world of human annotators (people who label data), we solve this by writing a Rulebook (an annotation guideline). If a human gets it wrong, a supervisor reviews their work, points out the mistake, and updates the Rulebook so everyone learns.

This paper asks: Can we do the same thing with AI? Can we take an existing Rulebook, let the AI try to follow it, find its mistakes, and then use the AI itself to rewrite the Rulebook to be clearer?

Here is how the authors tested this, explained through a simple analogy.

The Setup: The "Robot Intern" and the "Rulebook"

The researchers treated the Large Language Model (LLM) like a new Robot Intern.

The Task: The intern has to read medical abstracts and highlight disease names.
The Problem: If you just say "Find the diseases," the intern guesses. It might highlight a symptom instead of a disease, or miss a disease hidden in a list.
The Solution: You give the intern a Rulebook (the annotation guidelines).

The Three Big Questions (Hypotheses)

The team wanted to prove three things:

Does the Rulebook help? (Hypothesis 1)
- Analogy: Does giving the intern a manual make them better than just guessing?
- Result: Yes. The robot got much better when it had the specific rules to follow.
Do "Thinking" Robots work better? (Hypothesis 2)
- Analogy: Some robots just spit out answers fast (Non-Reasoning). Others pause, think step-by-step, and analyze logic (Reasoning). Which one follows the complex Rulebook better?
- Result: Yes. The "Thinking" robots were significantly better at understanding the nuances of the rules.
Can the robot fix its own manual? (Hypothesis 3)
- Analogy: Imagine the intern makes a mistake. Instead of a human boss fixing the manual, the intern looks at the mistake, figures out why it happened, and writes a new, clearer rule to prevent it next time.
- Result: Yes. The robot could successfully refine the rules, though the improvements were small but consistent.

The Process: The "Self-Correcting Loop"

The authors built a system that acts like a rehearsal loop before the real work begins. Here is how it works:

The Trial Run: The Robot Intern reads a small set of 10 medical documents using the current Rulebook.
The Grade: The system compares the Robot's highlights against the "Gold Standard" (the perfect human answers).
The Detective Work: If the Robot missed something or highlighted the wrong thing, the system groups these errors.
- Example: "Oh, the robot keeps missing diseases that are listed in a 'with/of' sentence structure."
The Rewrite (Moderation): The Robot acts as a Supervisor. It looks at the error, explains why it happened, and writes a new rule to fix it.
- New Rule: "If a disease is listed in a 'with' or 'of' phrase, don't ignore it! Highlight it."
Repeat: The Robot reads the 10 documents again with the new Rulebook. If it gets better, great. If not, it stops.

The Results: What Actually Happened?

The Rulebook Matters: Without the rules, the robot was okay but not great. With the rules, it became very good.
Thinking Helps: The "Thinking" models (like the ones that pause to reason) followed the rules much better than the fast, automatic ones.
The "Self-Correction" is Real but Small: The process of the robot rewriting its own manual worked. It fixed specific errors (like missing diseases in lists). However, the improvement wasn't a massive jump; it was a small, steady nudge in the right direction (about 1-3% better).

The Catch (Limitations)

The authors were careful to point out a few things:

The Sample Size: They only used 10 documents to teach the robot how to rewrite the rules. This is like trying to learn a whole language by studying only 10 sentences. It works for a quick test, but it might miss bigger patterns that only show up in 1,000 documents.
Cost vs. Speed: Some robots (like the "Thinking" ones) are expensive and slow. Others are cheap and fast but make more mistakes. There is a trade-off.
The "Gold" Requirement: This whole system needs a perfect "Gold Standard" answer key to check against. If you don't have a human-written rulebook to start with, this specific method doesn't work.

The Bottom Line

This paper shows that we can treat AI annotation like a training program. Instead of just hoping the AI gets it right, we can give it a rulebook, let it practice, let it analyze its own mistakes, and have it rewrite the rulebook to be clearer.

It's not a magic wand that makes the AI perfect instantly, but it is a proven way to systematically align a super-smart robot with the specific, boring, detailed rules that humans need it to follow.

Technical Summary: Refining and Reusing Annotation Guidelines for LLM Annotation

Problem Statement

While Large Language Models (LLMs) demonstrate strong zero-shot capabilities in general text labeling, they frequently struggle to adhere to the specialized, granular conventions required by gold-standard benchmarks. In traditional corpus annotation, human annotators rely on detailed guidelines to resolve ambiguities, define entity boundaries, and handle "gray-zone" cases. However, translating these complex, domain-specific rules into effective LLM prompts is non-trivial. Existing approaches often fail to capture the nuance required for consistent, high-quality annotation, leading to stochastic outputs that misalign with downstream application requirements.

Methodology

The authors propose an iterative moderation framework that simulates the early phases of human annotation projects to systematically refine and reuse existing annotation guidelines for LLMs. The process treats the LLM as a new annotator undergoing a training phase, where guidelines are iteratively improved based on performance discrepancies.

The Iterative Moderation Loop

The framework operates in three main stages per iteration ( $k$ ):

Annotation: An LLM annotator generates predictions ( $A_k$ ) on a development set using the current guidelines ( $G_k$ ).
Evaluation: Predictions are compared against gold annotations ( $A_g$ ) using strict span-and-type matching to calculate an F1-score (Inter-Annotator Agreement proxy). If performance meets a threshold or fails to improve, the loop terminates.
Moderation (Refinement): If performance is insufficient, a discrepancy analysis identifies the dominant error pattern. This triggers a three-step refinement process:
- Pattern Explanation: The model analyzes contrastive evidence between error cases and verified true positives to identify specific linguistic or structural cues causing the discrepancy.
- Principle Generation: The model synthesizes a generalized, actionable rule in a strict IF/THEN/EXCEPT format, ensuring it includes negative constraints to prevent over-correction.
- Guideline Refinement: The new principle is integrated into the existing guideline document ( $G_{k+1}$ ), maintaining the original document structure and formatting without summarization.

Experimental Setup

Tasks: Biomedical Named Entity Recognition (NER) on three datasets: NCBI Disease, BC5CDR (Chemical and Disease), and BioRED.
Models: Three LLM families (GPT, Gemini, DeepSeek), comparing non-reasoning vs. reasoning variants.
Data Strategy: A "low-resource" approach was adopted to simulate early-stage annotation. Guideline refinement was performed on a small set of 10 documents, while performance was evaluated on a distinct set of 100 documents.
Baselines: Simple prompting (S), original human guidelines (G), and the proposed guideline refinement via moderation (M).

Key Contributions

Empirical Validation of Hypotheses: The study provides evidence for three core hypotheses:
- H1: Incorporating explicit annotation guidelines significantly improves LLM annotation performance compared to simple prompting.
- H2: Reasoning-optimized models consistently outperform non-reasoning counterparts in guideline-driven tasks, likely due to the need for both deductive (applying rules) and inductive (deriving rules) reasoning.
- H3: Annotation guidelines can be effectively improved through an iterative moderation process even under minimal supervision (small development sets).
Iterative Moderation Framework: The introduction of a structured loop that automates the refinement of human-written guidelines, demonstrating its efficacy in aligning LLM outputs with gold-standard conventions.
Qualitative Analysis: A detailed examination of the refinement process, offering insights into how specific linguistic discrepancies (e.g., handling of feature-list contexts) are resolved and how refinements can propagate to related cases or introduce new false positives.

Results

Guideline Integration: Providing original guidelines substantially improved F1 scores across all datasets and models (e.g., GPT-5 on NCBI Disease improved from 0.46 to 0.73).
Reasoning Models: Reasoning variants consistently achieved higher performance than non-reasoning variants when using guidelines (e.g., DeepSeek improved from 0.29 to 0.55 on NCBI).
Moderation Efficacy: The iterative refinement process yielded further, albeit modest, performance gains (typically +0.01 to +0.03 in F1). These gains were consistent across datasets but varied in statistical significance depending on the model (e.g., statistically significant for Gemini, borderline for GPT).
Error Dynamics: Analysis revealed that moderation often involves a precision-recall rebalancing. While targeted error types (e.g., missed DiseaseClass mentions) were resolved, the process occasionally introduced new false positives, indicating that refinement is not a monotonic improvement but a targeted correction.

Significance and Claims

The paper claims that the systematic reuse and refinement of annotation guidelines serves as a viable alignment mechanism for LLMs. The authors argue that their approach bridges the gap between an LLM's internal knowledge and the specific, rigid conventions of gold-standard benchmarks.

The significance lies in demonstrating that high-level annotation principles can be induced from minimal supervision (10 documents) through an iterative, reasoning-driven process. This offers a potential alternative to large-scale statistical learning for adapting LLMs to specialized domains. However, the authors remain modest about the magnitude of the gains, acknowledging that while the framework shows potential, there is "substantial room for improvement" regarding the robustness of the moderation procedure and the stability of results derived from small sample sizes. The work positions itself not as a state-of-the-art performance booster, but as a proof-of-concept for a rationalistic approach to guideline evolution in the era of LLMs.