Technical Summary: Quantifying Data Contamination in Psychometric Evaluations of LLMs

Problem Statement

Recent research has increasingly applied established psychometric questionnaires (e.g., Big Five Inventory, Portrait Values Questionnaire) to Large Language Models (LLMs) to assess high-level psychological constructs such as personality, values, moral foundations, and dark traits. However, a critical validity threat exists: data contamination. Because LLMs are trained on massive text corpora that likely include these widely available psychometric inventories and their scoring procedures, model responses may reflect prior exposure to the test items rather than genuine psychological characteristics.

While prior studies have raised concerns that such contamination threatens the reliability of psychometric evaluations, there has been no systematic attempt to quantify the extent of this phenomenon. Existing contamination detection methods often focus on standard NLP benchmarks and lack the specificity required for psychometric assessments, which involve unique response scales, reverse-coded items, and specific scoring protocols.

Methodology

The authors propose a comprehensive framework to systematically measure data contamination in psychometric evaluations across three distinct dimensions. This framework is applied to 21 models from major families (OpenAI, Qwen, GLM, Gemini, Claude, Llama) and four widely used inventories:

1. Big Five Inventory (BFI-44)
2. Portrait Values Questionnaire (PVQ-40)
3. Moral Foundations Questionnaire (MFQ)
4. Short Dark Triad (SD-3)

The three aspects of contamination measured are:

1. Item Memorization

This assesses whether models have memorized the specific wording of inventory items.

• Semantic Memorization: Models are prompted with an inventory name and item index to reconstruct the item text. The generated text is compared to the original using cosine similarity of embeddings (text-embedding-3-large).
• Key Information Memorization: Models are presented with an item where the most informative keyword (annotated by psychology experts) is masked. The model must generate the missing keyword. Success is measured by the rate of correct keyword recovery.

2. Evaluation Memorization

This assesses whether models understand the scoring logic and construct associations of the inventory, beyond simple text recall.

• Item-Dimension Mapping: Models are asked to map each inventory item to its corresponding psychological dimension (e.g., mapping an item to "Extraversion"). Performance is measured using the averaged F1-score across dimensions.
• Option-Score Mapping: Models are given an item, response options, and a target dimension, and asked to output the numeric scores defined by the official scoring protocol (including handling reverse-coded items). Performance is measured using Mean Absolute Error (MAE) between predicted and ground-truth scores.

3. Target Score Matching

This assesses whether models can strategically adjust their responses to achieve a specific desired score, indicating they have internalized the scoring rules well enough to manipulate outcomes.

• Models are prompted with an item and a target score (minimum, mean, or maximum) and instructed to select the response option that yields that score.
• Performance is measured using MAE between the target score and the score actually achieved by the model's selection.

Key Results

The study provides systematic evidence that contamination is widespread and often saturated across modern LLMs:

• Ubiquity of Contamination: Across all 21 models, contamination was evident in all three aspects.
• Evaluation Memorization Saturation: Models exhibit near-ceiling performance in understanding scoring protocols. The average F1-score for Item-Dimension Mapping was 0.94 across all models. For Option-Score Mapping, advanced models (e.g., GPT-5, GLM-4.5) showed very low MAE, indicating they have internalized complex scoring schemes, including reverse-coding.
• Item Memorization: Models demonstrated significant ability to reconstruct item text (average semantic similarity 0.31) and recover masked keywords (average success rate 0.39).
• Target Score Matching: Advanced models demonstrated the ability to steer responses to match specific target scores with high precision (MAE ≈ 0.1–0.2), suggesting they can manipulate responses to fit desired personality profiles.
• Inventory Differences: Contamination was more pronounced in BFI-44 and PVQ-40 compared to MFQ and SD-3. The authors correlate this with the higher citation counts and online availability of BFI and PVQ, suggesting greater exposure during pretraining.
• Model Scaling: Larger models within the same family generally exhibited lower MAE and higher F1-scores, suggesting that scaling amplifies contamination, particularly in evaluation memorization. However, smaller models (e.g., Llama variants) showed comparatively weaker contamination signals.

Significance and Claims

The paper claims to offer the first systematic experimental evidence supporting concerns that psychometric evaluations of LLMs are compromised by data contamination.

• Validity Threat: The findings suggest that when LLMs are tested on standard psychometric inventories, their results may reflect memorization of training data and the ability to strategically game the scoring system rather than revealing intrinsic psychological tendencies.
• Methodological Contribution: The proposed framework moves beyond simple "recall" checks to evaluate the deeper understanding of scoring protocols and the ability to manipulate outcomes, which is critical for psychometric validity.
• Implications for Research: The authors argue that the field must adopt "contamination-aware evaluation practices." Since popular inventories like BFI-44 and PVQ-40 are heavily contaminated, researchers should be cautious when interpreting LLM personality or value profiles derived from these tools.
• Baseline Establishment: The paper provides specific baselines (e.g., random guessing expectations for F1 and MAE) to help practitioners interpret contamination scores in future studies.

The authors conclude that while their study covers major inventories, many others remain unexplored, and the findings are currently limited to English items. Nevertheless, the results fundamentally challenge the reliability of using standard psychometric questionnaires to evaluate LLM characteristics without accounting for data contamination.