💬 NLP

Exploring and Testing Skill-Based Behavioral Profile Annotation: Human Operability and LLM Feasibility under Schema-Guided Execution

This paper proposes a skill-based framework for Behavioral Profile annotation, demonstrating that while LLMs like GPT-5.4 can reliably execute a subset of well-defined annotation skills, their performance is selective and best understood as an independent third voice rather than a direct human substitute, highlighting the need to evaluate automation feasibility at the skill level rather than the task level.

Original authors: Yufeng Wu

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yufeng Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to read a book and write a detailed report on every single sentence.

In the old way of thinking, we asked: "Can the robot read the whole book and write the report?" If the robot made a few mistakes, we'd say, "No, it's not ready yet."

But this paper suggests that question is too simple. It's like asking, "Can a human build a house?" The answer isn't just "yes" or "no." It depends on which part of the house. A human can easily lay bricks (easy task), but might struggle to design the electrical wiring without a manual (hard task), and might be completely lost trying to paint a ceiling while standing on a wobbly ladder (impossible task without help).

The authors of this paper argue that annotating language (labeling sentences with data) is exactly like building a house. It's not one big job; it's a bundle of many different, tiny jobs called "skills."

Here is the breakdown of their findings using simple analogies:

1. The "Bundle of Skills" vs. The "Monolith"

Think of a sentence like a complex recipe.

The Old View: You hand the recipe to a chef (the AI) and ask, "Is this dish good?"
The New View: You break the recipe down.
- Skill A: Chopping onions (Easy, the AI is great at this).
- Skill B: Tasting the sauce to see if it's too salty (Medium, the AI is okay, but needs a second look).
- Skill C: Deciding if the dish feels "sad" or "happy" (Hard, the AI gets confused).

The paper found that for language analysis, some "skills" are easy for AI, some are fixable with a little more focus, and some are just too vague for the current rules to handle.

2. The Human Test: "Cognitive Load" vs. "Broken Rules"

The researchers had two human experts label 300 sentences. They did this in two rounds:

Round 1: The humans did everything at once (chopping, tasting, judging). They got tired and made mistakes.
Round 2: They went back only to the sentences they disagreed on.

The Discovery:

The "Tired" Skills: For some tasks, the humans disagreed in Round 1 because they were overwhelmed. In Round 2, when they focused just on that one tiny task, they agreed perfectly. These are Recoverable Skills. The rules were fine; the humans just needed a break.
The "Broken" Skills: For other tasks, even when the humans focused just on that one thing in Round 2, they still couldn't agree. This means the Rules (Schema) are broken. The task is too vague, like asking someone to "guess the mood of a cloud." No amount of focus will fix a broken definition.

3. The AI Test: The "Independent Third Voice"

They then asked a super-smart AI (GPT-5.4) to do the same job.

The Big Surprise:
The AI and the Humans agreed on which tasks were hard and which were easy.

Analogy: Imagine a map of a mountain. Both the Human and the AI agree that the North Peak is steep and dangerous, and the South Path is flat and easy. They share the same map (Shared Taxonomy).

BUT, when they actually climbed the mountain:

The Human slipped on a specific rock on the North Peak.
The AI slipped on a different rock on the North Peak.
They didn't make the same mistakes on the same sentences. They climbed independently (Independent Execution).

Why is this good?
If the AI made the exact same mistakes as a human, it would be useless (it would just be a copy). But because it makes different mistakes, it acts like a third expert in the room.

Human A says: "This sentence is sad."
Human B says: "This sentence is neutral."
AI says: "I think it's sad, but for a different reason than Human A."

The AI isn't a replacement for humans; it's a new voice that helps the team see the sentence from a different angle.

4. The "Small Robots" (Open-Source Models)

The researchers also tested smaller, free AI models. These models failed, but not because they were "dumb."

The Problem: They couldn't follow the "format."
Analogy: Imagine you ask a small robot to sort red and blue balls. The robot understands red and blue perfectly. But instead of putting them in the bins, it starts drawing pictures of the balls or writing poems about them.
The failure wasn't in understanding the language; it was in following the strict rules of the game.

The Final Takeaway: How to Fix the Workflow

The paper suggests we stop trying to automate the whole job at once. Instead, we should use a Skill-Based Workflow:

Audit the Skills: Break the job down. Which parts are easy? Which parts are broken?
Filter: Throw away the "broken" rules (the ones humans can't agree on even with focus).
Deploy the AI: Let the AI handle the "easy" and "recoverable" skills.
Use the AI as a Partner: Don't let the AI replace the human. Let the AI be the "Third Voice" that helps humans make better decisions on the tricky parts.

In short: We shouldn't ask, "Can AI do this job?" We should ask, "Which parts of this job can AI do, and how can we use it to help humans do the rest?" The answer is: AI is a powerful, independent teammate, but only if we give it clear, specific instructions for specific tasks.

1. Problem Statement

Behavioral Profile (BP) annotation is a rigorous corpus-based method for lexical analysis that requires simultaneous coding of multiple linguistic dimensions (lexical, syntactic, semantic, collocational, and pragmatic) for every concordance line.

The Challenge: BP annotation is traditionally viewed as a monolithic task, making it difficult to automate. Large Language Models (LLMs) have shown promise in single-feature tasks, but their ability to handle the complex, multi-dimensional nature of BP annotation remains unclear.
The Gap: Existing research often evaluates LLMs at the "task level" (i.e., can the model do the whole job?), obscuring the fact that different parts of a schema vary significantly in their operational difficulty. There is a lack of granular analysis regarding which specific annotation operations (skills) are automatable and which are not.

2. Methodology

The study proposes a skill-based framework where BP annotation is decomposed into distinct "annotation skills" rather than treated as a single task.

Data:
- Corpus: 3,134 concordance lines of 30 Chinese metaphorical color-term derivatives (e.g., hēichē 'unlicensed taxi', hóngbāo 'red envelope').
- Schema: A 14-feature BP schema covering five linguistic levels (lexical, syntactic, semantic, collocational, pragmatic).
System Architecture (Skill-File-Driven Pipeline):
- Unlike standard prompt engineering, the study uses an external schema-driven system.
- Components: Schema files (defining labels, rules, examples), prompt templates (injecting definitions), and runner configurations (binding resources and decoding defaults).
- Execution: This ensures the workflow is auditable, reusable, and strictly constrained by the schema, rather than relying on ad-hoc prompting.
Human Annotation Protocol:
- Two human annotators performed a two-round schema-only protocol on a 300-instance validation subset.
- Round 1: Independent labeling to measure "cold-take" operability.
- Round 2: Focused re-annotation of only the disagreed cells to distinguish between cognitive-load difficulty (recoverable) and structural underspecification (irreversible).
Model Evaluation:
- Frontier Model: GPT-5.4 (OpenAI).
- Open-Source Models: Qwen2.5-7B, GLM-4-9B, Yi-1.5-9B.
- Evaluation Levels:
  1. Skill Level: Correlation of difficulty profiles between humans and models.
  2. Instance Level: Correlation of errors on specific concordance lines.
  3. Lexical-Item Level: Correlation of errors on specific target words.

3. Key Contributions

Conceptual Shift: Reframes BP annotation from a "single task" to a "bundle of annotation skills." This allows for a more precise diagnosis of automation feasibility.
Operational Framework: Introduces a skill-file-driven pipeline that externalizes decision rules and examples, moving beyond prompt-only approaches to create a structured, reproducible execution environment.
Classification of Skill Operability: Proposes a three-tier classification for annotation skills based on human pilot data:
- Directly Operable: Reliable in the first pass.
- Recoverable: Unstable initially but stabilizes after focused re-annotation (cognitive-load issue).
- Structurally Underspecified: Remain unstable even after re-annotation (schema definition issue).
New Interpretation of LLM Role: Introduces the concept of "Shared Taxonomy, Independent Execution," arguing that LLMs should be viewed as an "independent third skill voice" rather than a direct human substitute.

4. Key Results

Skill Heterogeneity: The 14 BP features were not uniformly difficult.
- 5 skills were directly operable.
- 4 skills were recoverable after focused re-annotation.
- 5 skills remained structurally underspecified.
GPT-5.4 Performance:
- Achieved substantial reliability on the retained 10 skills (Accuracy = 0.678, $\kappa$ = 0.665, Weighted F1 = 0.695).
- Performance was selective, not global; it excelled on some skills but struggled with others.
Correlation Analysis (The "Shared Taxonomy, Independent Execution" Pattern):
- Skill Level: Strong alignment between human and GPT difficulty ( $r = 0.881$ ). Both find the same features hard or easy.
- Instance Level: Near-zero alignment ( $r = 0.016$ ). They do not fail on the same specific sentences.
- Lexical Level: Negative/No alignment ( $r = -0.142$ ). They do not fail on the same specific words.
- Implication: GPT shares the human understanding of what is difficult but makes different errors on where and how.
Open-Source Model Failures:
- Failures were concentrated in schema-to-skill execution (e.g., mapping the wrong label inventory or format violations) rather than a total lack of linguistic competence.
- High rates of "off-schema" outputs (e.g., 76% invalid rate for F4 in Qwen2.5-7B) indicated a breakdown in constrained decoding.

5. Significance and Implications

Methodological Shift: The paper argues that automatic annotation should be evaluated based on skill feasibility rather than task-level automation. A "yes/no" answer to "Can LLMs do BP annotation?" is misleading; the answer is "Yes, for specific skills under specific conditions."
Workflow Redesign: Proposes a semi-automatic workflow:
1. Pilot human annotation to classify skills (Operable vs. Underspecified).
2. Screen out structurally underspecified skills.
3. Deploy LLMs only on the "retained" skills as an independent third voice to triangulate with human annotators, rather than replacing them.
Theoretical Insight: The finding that human and LLM difficulty aligns at the macro level but diverges at the micro level suggests that LLMs provide non-redundant annotation evidence. They are not merely "noisy human copies" but distinct agents operating within the same difficulty landscape.
Practical Application: For corpus linguistics and NLP, this suggests that schema design must be iterative. Schemas containing "structurally underspecified" skills cannot be automated simply by using a more powerful model; the schema itself requires revision.

In conclusion, the study demonstrates that while LLMs cannot yet fully automate complex, multi-dimensional linguistic annotation, they can significantly enhance the process when integrated as a skill-specific, independent executor within a rigorously structured, schema-driven workflow.