🤖 AI

OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering

This position paper proposes OLAF, a conceptual framework that treats LLM-based annotation in empirical software engineering as a rigorous measurement process by defining key constructs like reliability, calibration, and drift to enhance transparency and reproducibility.

Original authors: Mia Mohammad Imran, Tarannum Shaila Zaman

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mia Mohammad Imran, Tarannum Shaila Zaman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive library of messy, handwritten notes. In the past, you would have hired a team of human librarians to read every note, decide what category it belongs to (like "Bug Report," "Feature Request," or "Spam"), and write a label on it. This is slow, expensive, and sometimes two librarians might disagree on the same note.

Now, imagine you hire a super-smart robot (an AI or Large Language Model) to do the labeling for you. It's fast and cheap. But here's the problem: Is the robot actually doing a good job, or is it just guessing?

This paper, titled OLAF, argues that we need to stop treating these robots like magic black boxes that just "do the work." Instead, we need to treat them like scientific measuring tools, just like a thermometer or a scale. If you use a scale to weigh ingredients for a cake, you need to know if the scale is accurate, if it drifts over time, and if it gives the same result every time you use it.

Here is the breakdown of the paper's ideas using simple analogies:

1. The Problem: The "Magic Box" Trap

Right now, many researchers use AI to label data but don't tell us how they did it.

The Analogy: Imagine a baker says, "I used a special oven to bake this bread." But they don't tell you the temperature, the time, or the brand of the oven. If you try to bake the same bread, it might come out burnt or raw.
The Reality: Researchers often forget to mention small details like the exact words they typed to the AI (the "prompt"), the specific version of the AI, or the settings they used. Because of this, no one else can repeat their experiment to see if they get the same results.

2. The Solution: OLAF (The "Quality Control" Framework)

The authors propose a new framework called OLAF. Think of OLAF as a checklist for quality control that you must use before you trust the robot's labels. It treats the AI not as a worker, but as a measuring instrument that needs to be calibrated.

OLAF asks you to check six specific things (Constructs):

Reliability (The "Consistency" Check): If you ask the robot to label the same note five times, does it give the same answer? Or does it flip-flop?
Consensus (The "Group Hug" Check): If you use three different robots, do they all agree on the label? If they all say "Bug," that's a strong signal. If they disagree, the task might be too confusing.
Aggregation (The "Voting" System): How do you combine the answers? Do you just take a majority vote? Or do you use a fancy math formula to figure out which robot is the most reliable?
Transparency (The "Receipt" Check): Did you write down everything? The robot's name, the version number, the exact words you typed? Without this "receipt," the work is useless to others.
Calibration (The "Confidence" Check): If the robot says, "I am 99% sure this is a bug," is it actually right 99% of the time? Or is it just overconfident? This checks if the robot's confidence matches reality.
Drift (The "Shifting Goalposts" Check): AI models change over time. A robot that works perfectly today might act differently next month because the company updated its software. OLAF checks if the robot's behavior has "drifted" or changed unexpectedly.

3. How to Use the Robot (The Six Configurations)

The paper also explains that you don't always have to let the robot do everything. It suggests six ways to mix humans and robots, like different recipes:

Human-in-the-Loop: The robot does the first pass, but a human double-checks the ones the robot isn't sure about. (Like a student doing homework, and a teacher grading the hard parts).
Model-in-the-Loop: The robot suggests an answer, and another robot or a human checks it.
Verifier-in-the-Loop: A second robot acts as a "referee" to check if the first robot's answer is good before a human sees it.
Filter: The robot quickly throws away the obvious junk (like spam emails) so humans only have to look at the important stuff.
Judge: The robot acts as a referee, grading other AI's work based on a set of rules.
Annotator: The robot does the whole job alone. (This is fast, but risky if the robot makes a mistake).

4. The Catch (Limitations)

The authors admit that this isn't a perfect solution yet.

The Analogy: You can't fully trust a thermometer if you don't know how it was built or if the factory changes the design every week.
The Reality: Many popular AI models (like the ones from big tech companies) are "black boxes." We don't know their training data, and they might change without telling us. OLAF tries to measure how stable they are, but it can't guarantee they won't change tomorrow.

Summary

The paper says: "Stop treating AI labeling as magic. Start treating it like science."

If you want to use AI to label data in software engineering, you need a framework (OLAF) that forces you to check if the AI is consistent, if it's honest about its confidence, and if you've written down exactly how you used it. Without this, your research results might be unreliable, like a cake made with an unmeasured oven.

Technical Summary: OLAF – Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering

Problem Statement
Empirical Software Engineering (ESE) increasingly relies on Large Language Models (LLMs) to automate or assist in annotation tasks, such as labeling bug reports, classifying commit intents, or identifying self-admitted technical debt. However, the methodological rigor of these practices remains underdeveloped. Current studies often treat LLM-based annotation as a purely automated activity rather than a measurement process, leading to several critical gaps:

Lack of Standardization: There are no standardized measures for reliability, calibration, or drift in LLM annotation studies.
Fragmented Reporting: Essential configuration details (e.g., prompt templates, decoding parameters, model versions) are frequently omitted, hindering reproducibility.
Configuration Sensitivity: Minor changes in prompts, temperature settings, or model updates can drastically alter outcomes (a phenomenon termed "prompt drift" or "LLM hacking"), yet these sensitivities are rarely quantified.
Undefined Constructs: Abstract concepts like "label accuracy" lack formal operational definitions linking them to measurable procedures, threatening construct, internal, and external validity.

Methodology and Framework (OLAF)
The paper proposes the Operationalization for LLM-based Annotation Framework (OLAF). Rather than offering a new algorithm or a specific experimental result, OLAF is a conceptual framework designed to treat LLM annotation as a measurable, auditable process. It operationalizes annotation through six key constructs and maps them to specific metrics and reporting focuses (summarized in Table 1 of the paper):

Annotator Reliability: Measures agreement across individual annotators (human or model) using chance-corrected statistics (e.g., Cohen's $\kappa$ , Gwet's AC1, Krippendorff's $\alpha$ ).
Consensus: Reflects group-level agreement to determine if annotators converge on a shared judgment or if the task contains inherent ambiguity.
Aggregation: Defines methods for combining multiple annotators into a single label, specifically distinguishing between probabilistic aggregation (e.g., Dawid–Skene, GLAD, MACE) and majority voting.
Transparency: Mandates complete documentation of model identifiers, parameters, prompts, and settings (following practices like Datasheets and Model Cards) to ensure reproducibility.
Calibration: Evaluates the correctness of predicted probabilities by comparing confidence scores against empirical accuracy using metrics like Expected Calibration Error (ECE) and Brier Score.
Drift: Captures behavioral stability by measuring how much model behavior changes under prompt or context variations, quantified via activation distances (L2, cosine), output distribution metrics (Jensen-Shannon divergence), or prompt sensitivity indices (POSIX).

The framework also categorizes six common annotation workflows (Table 2) to contextualize where these metrics apply:

Human-in-the-Loop (HitL)
LLM/Model-in-the-Loop (MitL)
Verifier-in-the-Loop (VitL)
LLM-as-a-Pre-screen-Filter
LLM-as-a-Judge (LLMaaJ)
LLM-as-an-Annotator

Key Contributions

Conceptual Shift: The paper argues for reconceptualizing LLM annotation from an automation task to a measurement process, requiring the same validation rigor as traditional code metrics.
Operational Definitions: It provides the first structured set of operational definitions for reliability, consensus, drift, and calibration specifically tailored to LLM-based annotation in SE.
Guidelines for Practice: The authors provide concrete guidelines for researchers, including:
- Explicitly declaring the annotation configuration (e.g., HitL vs. LLM-as-an-Annotator).
- Documenting full model and prompt details.
- Selecting appropriate aggregation methods based on observed agreement.
- Implementing drift tracking using fixed calibration subsets (e.g., re-annotating 50–100 expert-labeled items when parameters change).
Resource Provision: The authors have established a companion website (se-llm-annotation-olaf.github.io) with detailed guidelines and examples.

Results and Claims
As a position paper, OLAF does not present new experimental data or empirical results validating the framework's efficacy. Instead, its "results" are the synthesis of existing literature (citing works by Ahmed et al., Baumann et al., Wagner et al., etc.) to demonstrate the current lack of rigor.

The paper claims that:

Current LLM annotation practices are insufficient for safety-critical or compliance-sensitive settings due to undefined reliability and calibration.
The proposed six constructs provide a necessary foundation for reproducible research, allowing researchers to select dimensions relevant to their specific task (e.g., deterministic tasks may not require all six dimensions).
The framework acknowledges limitations, particularly regarding proprietary models (e.g., GPT, Claude) where training data and update schedules are opaque, making "constrained stability" the realistic goal rather than deterministic behavior.

Significance
The significance of OLAF lies in its potential to elevate the methodological standards of Empirical Software Engineering. By providing a structured vocabulary and measurement approach, it aims to:

Motivate methodological discussions regarding the validity of LLM-generated labels.
Enable future empirical work to be more transparent and reproducible.
Prevent the "black box" nature of LLM annotation from undermining the validity of SE research findings.

The authors position OLAF as an initial step, with future work planned to empirically evaluate the framework's impact on reproducibility and to develop SE-specific operational examples.