💬 NLP

How Annotation Trains Annotators: Competence Development in Social Influence Recognition

This study demonstrates that the human data annotation process itself serves as a training mechanism that significantly enhances annotators' competence and confidence—particularly among experts—and that these improvements in annotator skill subsequently lead to better performance in Large Language Models trained on the resulting data.

Original authors: Maciej Markiewicz, Beata Bajcar, Wiktoria Mieleszczenko-Kowszewicz, Aleksander Szczęsny, Tomasz Adamczyk, Grzegorz Chodak, Karolina Ostrowska, Aleksandra Sawczuk, Jolanta Babiak, Jagoda Szklarczyk, Pr

Published 2026-04-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maciej Markiewicz, Beata Bajcar, Wiktoria Mieleszczenko-Kowszewicz, Aleksander Szczęsny, Tomasz Adamczyk, Grzegorz Chodak, Karolina Ostrowska, Aleksandra Sawczuk, Jolanta Babiak, Jagoda Szklarczyk, Przemysław Kazienko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of people to help you spot hidden traps in a maze. Some of your hires are professional maze-runners (experts), while others are just regular people who have never seen a maze before (non-experts).

This paper is about what happens to these people while they are doing the job of finding the traps. The researchers wanted to know: Does the act of looking for traps actually make the hunters better at hunting? And does this improvement change the quality of the map they draw for us?

Here is the story of their journey, broken down simply:

1. The Setup: The "Training Wheels" Phase

The researchers gathered 25 people from different walks of life: psychologists, teachers, parents, teenagers, and communication experts. They gave them a massive pile of 1,000+ conversations (mostly between an AI and a teenager) and asked them to find "social influence"—basically, spotting when someone is trying to manipulate or persuade another person.

To see if the people were learning, they played a clever trick:

Round 1 (The Warm-up): Everyone annotated 30 specific conversations.
Round 2 (The Grind): They spent two weeks annotating 175 new conversations.
Round 3 (The Test): They were given the exact same 30 conversations from Round 1 and asked to do them again, as if they had never seen them before.

2. The Discovery: The "Muscle Memory" Effect

The results were fascinating. It turns out that doing the work made them better at the work.

They got faster: Just like a gamer gets faster at a video game after playing it for a few hours, the annotators finished the second round of the same texts much quicker.
They got sharper: When they looked at the conversations the second time, they didn't just see the obvious stuff. They started spotting subtle details they missed before.
- Analogy: Imagine looking at a painting. The first time, you just see "a tree." The second time, after studying art, you see "the way the light hits the leaves and the specific type of bark." They started seeing the nuance.
They got more confident: The experts, in particular, felt much more sure of their answers. They could explain why they thought something was manipulation, using better words and clearer logic.

3. The Twist: The "Confused Expert" Paradox

Here is the most interesting part. Even though the annotators were getting better at their job, they felt more stressed and more tired the second time around.

The Metaphor: Think of a student learning to drive. On day one, they are relaxed because they don't know what they don't know. By day ten, they are hyper-aware of every other car, every speed limit sign, and every pothole. They are driving better, but they feel the weight of the responsibility more heavily.
The researchers call this "conscious incompetence" turning into "conscious competence." They realized how tricky the task actually was, which made them work harder and feel more mentally drained, even though their output was higher quality.

4. The AI Connection: Why This Matters for Robots

The researchers didn't just stop at human feelings. They asked: "Does this human growth actually help the AI?"

They took the data from the "beginning" (Round 1) and the "end" (Round 3) and used it to train a Large Language Model (an AI).

The Result: The AI trained on the "end" data (where the humans were smarter and more detailed) performed better.
The Analogy: If you teach a robot to recognize dogs using photos taken by someone who just learned what a dog is, the robot might miss a cat that looks like a dog. But if you teach the robot using photos taken by someone who has studied animal behavior for weeks, the robot learns the subtle differences much faster.

5. The Big Takeaway

The paper concludes that annotation is not just a task; it's a class.

When you ask people to label complex, subjective things (like "is this person being manipulated?"), they aren't just filling out a form. They are learning a new skill.

For the Humans: They leave the project more aware of how manipulation works in real life. They can spot it in ads, in conversations with their kids, and on social media.
For the AI: The data gets better as the humans get better.
For the Experts: The people who already knew a little bit about the topic (like the psychologists) improved the most, becoming the "super-annotators."

In short: You can't just hire people to label data and expect them to stay the same. The process of labeling changes them, and that change makes the final product (both the human's understanding and the AI's brain) smarter.

1. Problem Statement

Human data annotation is traditionally viewed as an objective process to establish "ground truth," particularly when involving experts. However, in subjective domains like social influence recognition, annotator judgments are fluid and can evolve over time. The paper addresses three critical gaps:

Subjectivity vs. Objectivity: The assumption that human annotation is static or purely objective is flawed; annotator competence and worldview shift during the annotation process.
Data Quality Dynamics: It is unclear whether changes in annotation labels over time represent random noise, a shift in worldview, or genuine competence development (learning).
Impact on AI: The extent to which these competence shifts affect the quality of datasets used to train Large Language Models (LLMs) remains under-explored.

The study aims to quantify how annotator competence evolves during a social influence recognition task and determine if data collected at different stages of the annotation process yields different model performance.

2. Methodology

2.1 Dataset and Task

Dataset: 1,021 AI-generated dialogues depicting social influence targeted at adolescents.
Annotation Schema: Annotators labeled texts for:
- Degree of social influence.
- One or more of 20 specific social influence techniques.
- RICs (Free-text): Intentions of the influencer, Reactions of the target, and Consequences of submission.
- Certainty levels and optional comments.
Participants: 25 annotators divided into five groups (5 each): Psychologists, Communication Experts, Adolescents, Parents, and Teachers (Educators). This mix included both domain experts and non-experts.

2.2 Experimental Design

The process was structured into three phases to enable a "Pre/Post" comparison:

Pre-Annotation: 30 texts annotated by all participants.
Calibration: A session to clarify definitions without using specific text examples to avoid bias.
Main Annotation: 174–175 unique texts annotated over two weeks.
Post-Annotation: The same 30 texts from the Pre-phase were re-annotated from scratch.

2.3 Evaluation Metrics

The study employed a multi-modal approach to measure competence and data quality:

Quantitative Analysis:
- Agreement: Krippendorff's $\alpha$ (using Jaccard distance) for inter- and intra-annotator agreement.
- Efficiency: Annotation time per text (Pre vs. Post).
- Label Stability: Retention rates of specific technique labels and shifts in free-text RICs (semantic analysis of breadth, detail, and thematic categories).
LLM Training & Evaluation:
- LLMs (DeepSeek-V3.2 and Llama-3.1-8B-Instruct) were trained on Pre-data and Post-data separately.
- Models were tested on the Main dataset (held-out data).
- Performance measured via Jaccard similarity for multi-label classification of influence techniques.
Psychometric & Qualitative Analysis:
- Surveys: SIR-SC (Perceived Competence & Self-Confidence) and NASA-TLX (Workload) administered Pre and Post.
- Interviews: Semi-structured interviews with 5 annotators (one per group) to analyze thematic shifts in interpretation.
- Content Analysis: Manual and LLM-assisted coding of free-text RICs to track changes in perspective, detail, and thematic coverage.

3. Key Results

3.1 Competence Development & Efficiency

Speed: Annotators became significantly faster, reducing average time per text from 6.56 min (Pre) to 5.86 min (Post) ( $p=0.0009$ ).
Self-Perception: Significant increases in Perceived Competence (Cohen's $d=0.567$ ) and Self-Confidence were observed.
Workload Paradox: Despite higher efficiency, NASA-TLX scores increased, indicating higher perceived mental demand and frustration. This suggests a state of "conscious incompetence" where annotators became more aware of the task's complexity and their own potential errors.

3.2 Data Quality & Label Evolution

Expert vs. Non-Expert: Expert groups showed significant increases in the number of RICs (Intentions, Reactions, Consequences) and improved inter-annotator agreement. Non-experts showed less change in label counts but still demonstrated qualitative improvements.
Qualitative Shifts:
- Richness: Post-annotations were more detailed, with broader thematic coverage.
- Nuance: Annotators shifted from viewing consequences as "definite" to "potential/conditional," showing greater interpretative caution.
- Technique Refinement: There was a systematic shift from broad, intuitive labels (e.g., "Liking") to specific, behaviorally defined techniques (e.g., "Labeling," "Door-in-the-face"). The retention rate for specific techniques was ~65%, with significant substitution patterns indicating refined discrimination.
Agreement: Intra-annotator agreement remained higher than inter-annotator agreement (typical for subjective tasks), but experts showed improved convergence over time.

3.3 Impact on LLM Performance

Superiority of Post-Data: Models trained on Post-annotations consistently outperformed those trained on Pre-annotations across all settings.
- ICL (DeepSeek-V3.2): The performance gap widened with more few-shot examples. At 30-shot, the Jaccard score improved by +0.0149.
- SFT (Llama-3.1): Fine-tuning on Post-data yielded a positive delta of +0.0069.
Conclusion: The "learning" phase of annotators resulted in higher-quality training data that directly improved downstream AI model performance.

4. Key Contributions

Empirical Evidence of Annotator Learning: The study demonstrates that the annotation process itself acts as a competence development intervention, particularly for experts, leading to more nuanced and reliable data.
Data Quality vs. Time: It challenges the static view of ground truth, showing that data quality is dynamic and often improves as annotators internalize task schemas.
LLM Performance Correlation: It provides direct evidence that training data collected after annotators have undergone the learning curve (Post-phase) yields better AI models than data collected at the start (Pre-phase).
Educational Value: The process serves as an educational tool, increasing participants' critical awareness of social influence and their ability to explain manipulation mechanisms.

5. Significance and Implications

For AI Development: Data collection pipelines should account for the "learning curve" of annotators. Relying solely on early-stage annotations may result in suboptimal model training. Iterative calibration and allowing annotators to "warm up" before finalizing datasets is recommended.
For Subjective Tasks: In domains like social influence, mental health, or ethics, treating annotators as active learners rather than static labelers improves the validity of the resulting datasets.
For Education: The study suggests that annotation tasks can be integrated into educational curricula (e.g., AI literacy programs) to help youth and adults recognize and resist persuasive manipulation.
Future Work: The authors propose expanding this to larger groups and non-subjective tasks to better quantify competence and refine agreement metrics.

In summary, the paper argues that annotation is a learning process. By acknowledging and leveraging this development, researchers can generate higher-quality datasets that significantly enhance the performance of AI models trained on them.