💬 NLP

Validity of LLMs as data annotators: AMALIA on authority

This paper demonstrates that while Portugal's national language model AMALIA achieves high agreement with human coders on the moral foundation of authority, it lacks construct validity because it relies on surface-level shortcuts rather than theoretical reasoning, highlighting the need for national LLM benchmarks to evaluate the evidential basis of model performance beyond mere agreement.

Original authors: Manuel Pita

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Manuel Pita

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a new, super-smart robot named AMALIA. It's Portugal's very own language robot, built with public money to understand and speak European Portuguese better than anyone else. The big hope was that AMALIA could act like a super-fast, super-cheap human assistant for researchers. Specifically, scientists wanted to use it to read thousands of social media posts and figure out if people were talking about "Authority"—a tricky idea that isn't just about the word "boss" or "police," but about whether someone is respecting or challenging the rules of the hierarchy.

Here is the twist: AMALIA is great at guessing the right answer, but it's terrible at showing its work.

The "Right Answer, Wrong Reason" Problem

Think of AMALIA like a student taking a math test.

The Goal: The teacher asks, "Solve for X using this specific formula."
The Result: AMALIA writes down the correct number for X.
The Catch: When the teacher asks, "Show me your steps," AMALIA can't. Instead of using the formula, it just guessed the answer because it saw a pattern, like "Oh, the word 'boss' is in the sentence, so the answer must be 'Authority'."

In the real world, this is dangerous. If a robot agrees with human experts 90% of the time, we usually say, "Great! We can trust it!" But this paper argues that for AMALIA, that agreement is a fake-out. It's like a parrot that learns to say "I love you" whenever it sees a human smile. The parrot isn't feeling love; it's just matching the smile to the phrase. AMALIA was matching surface clues (like seeing a police officer or hearing someone angry) to the label "Authority," rather than actually understanding the complex theory of power and respect that the researchers were trying to measure.

The "Black Box" Test

To prove this, the researchers didn't just ask AMALIA the big question. They broke the question down into tiny, simple pieces, like a detective breaking a case into clues.

The Big Question: "Is this text about Authority?"
The Tiny Clues: "Does it mention a leader?" "Does it judge that leader's behavior?" "Does it talk about tradition?"

If AMALIA truly understood the concept, it should get the same result whether you asked the big question or the tiny clues. But it didn't.

When asked the big question, AMALIA agreed with human experts about 71% of the time.
When asked the tiny clues and forced to put the pieces together, its agreement dropped to just 36%.

This huge drop is called the "recovery gap." It's like a magician pulling a rabbit out of a hat. If you ask, "Where is the rabbit?" and the magician points to the hat, that's good. But if you ask, "Show me the rabbit inside the hat," and the hat is empty, you realize the magician was just using a trick. The paper found that AMALIA's "trick" was relying on surface shortcuts, like seeing "moral outrage" near a powerful person and assuming it's about authority, even when it isn't.

The "Bigger Robot" Comparison

The researchers didn't just test AMALIA; they tested two other robots: Llama (a mid-sized one) and GPT-OSS (a giant one with 120 billion parameters, compared to AMALIA's 9 billion).

The Giant Robot (GPT-OSS): When asked the same tricky questions in Portuguese, it didn't have a gap. It got the right answer and could show its work. It understood the theory.
The Mid-Sized Robot (Llama): It had a small gap, but it was getting better.
AMALIA: It had a massive gap.

This proves that the problem wasn't the Portuguese language or the specific texts used. The problem was the size and training of the model. The paper suggests that AMALIA is simply too small to handle the complex "grain" of the theory. It's like trying to build a skyscraper with a hammer meant for a birdhouse; the tool just isn't big enough to do the job, even if it looks like it's working.

What This Means for National Robots

The paper explicitly rules out a few popular ideas:

It's not about the language: Even though AMALIA is Portugal's "home" robot, it didn't perform better than the big international robots. In fact, the big robots agreed with humans more than AMALIA did.
Agreement isn't enough: Just because a robot agrees with humans doesn't mean it's measuring the right thing. It might just be pattern-matching.
It's not a failure of the model's creators: The paper praises AMALIA for being open and transparent. The issue is that the current "gold standard" for testing robots (just checking if they agree with humans) is flawed.

The Bottom Line

So, is AMALIA useless? Not at all! The paper says AMALIA is excellent for screening and pre-coding. It can read a huge pile of text and say, "Hey, this looks interesting, humans should look at it." It's a great first filter.

But can it measure complex ideas like "Authority" on its own? No. Not yet.
The paper concludes that before we trust a national robot to measure what our citizens think and value, we need to ask a different question. We shouldn't just ask, "Do you agree with the humans?" We need to ask, "Can you show me your work?"

Until AMALIA (or any small national model) can show its work and prove it's using the right logic and not just guessing based on surface clues, it's not ready to be the final judge. The paper suggests that for now, AMALIA is a helpful assistant, but it's not the boss of the data.

Technical Summary: Validity of LLMs as Data Annotators

Problem Statement
The paper addresses a critical gap in the evaluation of Large Language Models (LLMs) as data annotators for social science research. While national "sovereign" LLMs (such as Portugal's AMALIA) are increasingly deployed to analyze local text, their evaluation has historically relied on reliability (agreement with human coders) rather than construct validity (faithfulness to the theoretical definition of the construct being measured).

The core problem is that high agreement between an LLM and human annotators does not guarantee the model is measuring the intended theoretical construct. Instead, the model may be achieving agreement through "surface correlates" or shortcuts (e.g., detecting moral outrage near an authority figure rather than the specific stance on hierarchy required by the theory). Because LLMs are black boxes, standard agreement metrics cannot distinguish between a genuine inference and a spurious correlation. This is particularly urgent for sovereign models, where the assumption is that a model trained on local data and language will naturally serve as a superior measurement instrument for that community.

Methodology
The study employs a grain calibration method to probe the inferential path of the LLM, moving beyond holistic agreement to test the validity of the measurement instrument.

Construct and Corpus: The study focuses on the "authority/subversion" moral foundation from Moral Foundations Theory. The dataset consists of 748 texts from the Moral Foundations Reddit Corpus (MFRC), transcreated into European Portuguese (pt-PT) via a rigorous "transcreation" pipeline. This pipeline ensures meaning, stance, and referents are preserved while adapting idiom and register, with blind generation and human adjudication to prevent code leakage.
Models: The primary subject is AMALIA-9B, a 9-billion-parameter model for European Portuguese. It is compared against two larger open-weight multilingual models: Llama-3.3-70B and GPT-OSS-120B.
The Recovery Gap ( $\Delta$ ): The central metric is the "recovery gap," defined as $\Delta = F^u_1 - F^d_1$ $Δ = F_{1}^{u} - F_{1}^{d}$ .
- $F^u_1$ : Performance (F1 score) of the undecomposed prompt (holistic instruction).
- $F^d_1$ : Performance of the decomposed prompt, where the construct is broken down into atomic clauses (detection, distinction, appraisal) and recombined via a strict Boolean integration rule derived from the theory.
- A $\Delta \approx 0$ indicates the model measures the construct as defined by the theory. A large $\Delta$ indicates the model relies on shortcuts not specified by the theory.
Experimental Design: The study tests AMALIA under both English and Portuguese instructions on the Portuguese corpus. It compares these results to the reference models on the same corpus and the reference models' performance on the original English corpus. The study is pre-registered, with hypotheses fixed before confirmatory testing on a held-out set of 448 texts.

Key Results

High Reliability, Low Validity for AMALIA: AMALIA achieves high agreement with human annotators (Holistic $F_1 \approx 0.71$ ), comparable to models 8–13 times its size. However, it exhibits a large recovery gap ( $\Delta \approx 0.36$ under Portuguese instructions and $0.44$ under English). This indicates that decomposition recovers only about half of AMALIA's holistic performance.
Surface Correlates vs. Theory: Error analysis reveals that AMALIA's false positives are largely driven by surface correlates, such as "generic moral outrage" near authority figures or the mere presence of an authority figure, rather than the required evaluative stance on hierarchy. In 78% of false positives, the model relied on surface bases rather than the construct's theoretical definition.
Model-Specific Failure: The failure is specific to AMALIA, not the corpus or the language.
- GPT-OSS-120B closes the gap on the Portuguese corpus ( $\Delta \approx 0.03$ ), performing similarly to its English results.
- Llama-3.3-70B shows a moderate gap ( $\Delta \approx 0.12$ ), which is larger than its English performance but significantly smaller than AMALIA's.
- This suggests the issue is the "construct–model instrument" (specifically the 9B model's capacity to execute the theoretical grain), not the Portuguese language or the transcreated corpus.
Instruction Language: The hypothesis that Portuguese instructions would narrow the gap was not statistically supported ( $\Delta_{pt} = 0.358$ vs. $\Delta_{en} = 0.436$ ; the confidence interval for the difference included zero). While the direction was consistent with the hypothesis, the effect was not significant.

Key Contributions

Operationalizing Construct Validity: The paper introduces and applies the "recovery gap" as a diagnostic tool to distinguish between reliable agreement and valid measurement in LLMs. It demonstrates that agreement alone is insufficient for validating LLMs as scientific instruments.
Sovereign Model Evaluation: It provides the first validity study of a national LLM (AMALIA) in its native language, challenging the assumption that "home-language" models automatically possess superior measurement validity.
Transcreation Pipeline: It details a rigorous, pre-registered pipeline for creating cross-lingual corpora for construct validation, ensuring that code transfer is justified by verification rather than assumed.
Granular Error Analysis: By decomposing prompts into atomic clauses, the study identifies specific failure modes (e.g., reliance on moral outrage as a proxy for authority) that holistic metrics obscure.

Significance and Claims
The paper concludes that while AMALIA is a reliable annotator (producing codes that match human consensus), it is not yet a valid instrument for measuring the theoretical construct of authority. It produces the "right" codes for the "wrong" reasons, relying on surface patterns rather than the theoretical inferences required by the construct's definition.

The authors argue that for sovereign LLM programs to be trusted with measurement tasks, they must move beyond agreement metrics. The critical question is not whether the model agrees with humans, but whether it can "show its work" by executing the specific inferential steps defined by the theory. The study suggests that smaller national models may lack the "executable grain" to perform complex theoretical decompositions without recalibration, and that validity evidence must be generated for specific measurement uses, not just general language benchmarks. The paper explicitly states that AMALIA is suitable for triage and pre-coding but not for standalone measurement of theoretical constructs.