💬 NLP

On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

This paper demonstrates that Large Language Models' performance in annotation tasks is primarily constrained by the alignment between their internalized priors and task definitions rather than text-level memorization, revealing that nearly two-thirds of zero-shot errors remain resistant to prompt-based correction.

Original authors: Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Stubborn Intern"

Imagine you hire a highly educated intern (the AI) to sort a pile of letters. You give them a specific rule: "Mark any letter that is rude as 'Toxic'."

You might assume that if you explain the rule clearly, the intern will follow it perfectly. However, this paper argues that the intern isn't a blank slate. Before you hired them, they spent years reading millions of books, websites, and social media posts. They have already formed their own internal idea of what "rude" means.

The paper asks: If your rule clashes with the intern's internal idea, who wins?

The answer is surprising: The intern's internal idea usually wins. Even if you give them a perfect written rule, they often stick to their own gut feeling, and they will do so with total confidence.

The Three Main Experiments

The researchers tested this using 9 different AI models and 5 different types of "toxicity" datasets (like gaming chats, news comments, and social media). Here are the three things they discovered:

1. The "Memory" Myth vs. The "Concept" Match

The Question: Does the AI do better because it has memorized the specific letters you are asking it to sort, or because it actually understands the concept of "toxicity" the way you do?

The Analogy: Imagine a student taking a test.

Memorization: The student has seen these exact questions before and remembers the answers.
Concept Match: The student actually understands the subject matter and can apply the rules to new questions.

The Finding: The researchers found that memorization doesn't help. In fact, if an AI has memorized the text, it might actually do worse because it's just reciting old data instead of thinking.
Instead, performance depends on Definition-Specific Familiarity (DSF). This is a fancy way of measuring: "Does the AI's internal definition of 'toxic' match your written definition?"

If the AI's internal "rude detector" matches your rule, it does a great job.
If their internal detector is different (e.g., they think "rude" means "hate speech against a group," but you meant "any mean comment"), it fails, even if it's a very smart model.

2. The "Sticky" Mistake

The Question: If the AI makes a mistake, can you fix it by giving it more examples or better instructions?

The Analogy: Imagine the intern marks a letter as "Safe" when it's actually "Toxic." You say, "No, look at this example! This is toxic!"

The Finding: It's like trying to unstick gum off a shoe. Most mistakes (about 65%) cannot be fixed.
The researchers call this "Decision Stickiness." Once the AI makes a decision, it is very hard to change its mind.
The Confidence Trap: The worst part? The AI is often most stubborn when it is most confident. If the AI says, "I am 99% sure this is safe," and it's wrong, your instructions almost never change its mind. It's like a confident driver who refuses to look at the GPS even when they are clearly going the wrong way.

3. The "Confidence" Trap

The Question: If you give the AI a wrong rule (a "misaligned" definition), will it realize it's confused and lower its confidence score?

The Analogy: Imagine you tell the intern, "Actually, today we are sorting by color, not by rude-ness."

The Finding: The AI happily follows your new, wrong rule. But here is the scary part: It doesn't get confused. It still says, "I am 90% sure I am doing this right."
The AI does not lower its confidence score just because the instructions are weird or wrong. It just applies the new rule with the same high confidence it had before.
The Result: You cannot trust the AI's "confidence score" to tell you if it's following your instructions correctly. It might be following a completely wrong definition, but it will tell you it's 100% sure.

The Takeaway for Humans

If you want to use AI to label or sort data (like finding toxic comments), the paper suggests three simple rules:

Don't rely on the AI's "fame." Just because a model is huge or has seen a lot of data doesn't mean it will do your specific job well.
Check the "Concept Match" first. Before you start, check if the AI's internal idea of the task matches your definition. If they don't match, no amount of prompting will fix it.
Don't trust the confidence meter. If the AI says it's "very confident," it doesn't mean it's right. It might just be confidently following the wrong instructions.

In short: The AI is not a blank slate waiting for your instructions. It comes with its own "brain" and "habits." If your instructions don't align with those habits, the AI will likely ignore you, make mistakes, and confidently insist it's right.

Technical Summary: On the Limits of LLM Adaptability

Problem Statement

Large Language Models (LLMs) are increasingly deployed for zero-shot annotation and "LLM-as-a-judge" tasks, operating under the implicit assumption that user-provided prompts (task definitions) fully govern model behavior. However, LLMs are not blank slates; they possess model-internalized priors developed through pre-training, instruction tuning, and reinforcement learning from human feedback (RLHF). These priors form implicit understandings of concepts (e.g., "toxicity") that may not align with specific user definitions. The core problem investigated is the tension between these internalized concepts and user instructions: to what extent can prompting override a model's internal priors, and how do these priors constrain annotation performance and reliability?

Methodology

The authors evaluated 9 instruction-tuned LLMs (spanning dense and mixture-of-experts architectures, including Llama, Mistral, DeepSeek, and proprietary models) across 5 primary toxicity detection datasets (covering social media, gaming, news, and forums) and supplementary non-safety tasks.

The study addressed three Research Questions (RQs) using specific metrics and experimental conditions:

RQ1 (Familiarity vs. Alignment): Does performance correlate with text memorization or conceptual alignment?
- Text Familiarity: Measured via ROUGE-L, BERTScore, and embedding cosine similarity by prompting models to continue text instances (40% prefix) and comparing the output to the ground truth suffix.
- Definition-Specific Familiarity (DSF): A novel metric quantifying the semantic alignment between a model's internal concept and the dataset's definition. This was computed by prompting the model to describe the target concept in its own words and measuring the cosine similarity between this description and the dataset's official definition. DSF was calculated as a consensus across six diverse sentence encoders to ensure robustness.
- Analysis: Partial correlations were used to control for dataset-level confounds (e.g., difficulty, domain).
RQ2 (Decision Stickiness): Can zero-shot errors be corrected via prompting?
- Rescue Rate: Defined as $P(\text{Correct} | \text{Prompted}, \text{Zero-Shot Wrong})$ .
- Conditions: Zero-shot, Aligned Definition, Few-Shot, Few-Shot + Definition, and DSPy-optimized prompts.
- Confidence Analysis: Investigated the relationship between zero-shot confidence scores and the probability of error correction.
RQ3 (Misalignment Susceptibility): How do models behave under incorrect definitions?
- Concept Substitution: Definitions from different datasets (e.g., "Hate Speech" vs. "Gaming Toxicity") were applied to target datasets to create misaligned conditions.
- Metrics: Measured Change Rate, Rescue Rate, Corruption Rate, Prediction Bias, and self-reported confidence levels under misalignment.

Key Results

1. Conceptual Alignment Drives Performance (RQ1)

Memorization is not predictive: After controlling for dataset difficulty, three distinct text memorization metrics (ROUGE-L, BERTScore, embedding similarity) showed no positive association with annotation accuracy (partial $r \approx -0.15$ to $-0.19$). In some cases, higher memorization correlated with lower performance.
DSF is predictive: Definition-Specific Familiarity (DSF) showed a significant positive association with zero-shot accuracy (partial $r = +0.41$ , $p=0.003$ ). Models whose internal concepts aligned with the task definition performed significantly better. This finding held across safety (toxicity) and non-safety (irony, subjectivity) domains.

2. High Decision Stickiness (RQ2)

Limited Rescue: The overall Rescue Rate was only 34.8%, meaning nearly two-thirds of zero-shot errors persisted even when provided with aligned definitions, few-shot examples, or automated prompt optimization (DSPy).
Confidence Barrier: High-confidence errors were particularly resistant to correction. The rescue probability dropped to 20.8% for errors where the model expressed high confidence (>0.9), compared to a peak of 51.8% for moderate confidence.
Iterative Failure: Multi-turn prompting (adding examples, definitions, and reconsideration requests) only marginally improved rescue rates (from 7.5% to 18.7% over three turns), confirming that decision stickiness is robust to iterative correction.
Model Size: Steerability was not monotonic with model size; smaller models (e.g., Mistral-Small-24B) sometimes exhibited higher rescue rates than larger ones (e.g., Llama-3.1-70B).

3. Misalignment and Calibration Failure (RQ3)

Behavioral Shift: Models adjusted their prediction thresholds to match misaligned definitions (e.g., narrow definitions led to under-prediction, broad definitions to over-prediction), demonstrating they follow instructions literally.
Confidence Blindness: Crucially, models remained highly confident (84–91%) even when applying misaligned definitions. Confidence levels did not decrease under misalignment, representing a fundamental calibration failure. Practitioners cannot rely on confidence scores to detect when a model is applying an incorrect definition.
Performance Variance: The choice of definition induced larger performance swings (up to 17% accuracy variation) than the choice of model (approx. 5% variation).

Significance and Claims

The paper argues that for annotation tasks, conceptual alignment between a model's internalized priors and the task definition is the primary determinant of performance, superseding text-level memorization or model scale.

Limits of Prompting: Prompt-based steering has fundamental limits. "Decision stickiness" implies that prompting is more effective at consolidating correct answers than at rescuing errors, particularly high-confidence ones.
Reliability Risks: The inability of confidence scores to signal definition misalignment poses a critical risk for "LLM-as-a-judge" pipelines, as models may confidently apply incorrect criteria without triggering uncertainty-based safeguards.
Practical Implications: The authors propose that annotation pipelines should prioritize measuring definition alignment (e.g., via DSF-style checks) before large-scale deployment and avoid relying on confidence scores as a proxy for definition validity. They suggest that definition design is a first-order experimental factor, often more impactful than model selection.

The study concludes that while LLMs are powerful tools, their internalized priors anchor their behavior in ways that limit the effectiveness of prompt-based steering, necessitating rigorous validation of model-definition pairings rather than treating models as flexible, blank-slate annotators.