🤖 AI

Machine individuality: Separating genuine idiosyncrasy from response bias in large language models

This study applies crossed random-effects models to extensive psycholinguistic data to demonstrate that large language models exhibit stable, stimulus-specific behavioral differences termed "machine individuality," which are distinct from global response biases and stochastic noise.

Original authors: Valentin Kriegmair, Dirk U. Wulff

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Valentin Kriegmair, Dirk U. Wulff

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a room full of 10 different robots. You ask them all the same question: "How does the word 'apple' make you feel?"

Some might say, "It's sweet and happy!" Others might say, "It's crunchy and red." Some might be very enthusiastic, while others are very flat.

For a long time, scientists studying these robots (Large Language Models, or LLMs) thought that if two robots gave different answers, it was just because one robot was "grumpier" than the other, or maybe they were just guessing randomly. They treated the robots like identical twins who only differed in their mood on a given day.

But this new paper asks a deeper question: Are these robots actually different people with their own unique personalities, or are they just broken copies of each other?

Here is the story of how they found out, explained simply.

The Problem: The "Grumpy Robot" vs. The "Unique Robot"

When you ask a human, "How happy is the word 'sun'?" they might say "8 out of 10." If you ask another human, they might say "7."

The Old Theory: Maybe the first robot is just a "high scorer" (it always gives high numbers), and the second is a "low scorer." That's just a bias. It's like a scale that is always off by 5 pounds. It doesn't mean the scale has a personality; it just means it's broken in a specific way.
The New Question: What if the robots aren't just broken scales? What if they actually see the world differently? Maybe Robot A thinks "sun" is warm and cozy, while Robot B thinks "sun" is hot and dangerous. That would be true individuality.

The Experiment: A Massive Word Test

To figure this out, the researchers didn't just ask the robots a few questions. They put them through a massive, grueling test.

The Subjects: 10 different AI models (from companies like Google, Microsoft, OpenAI, and Alibaba).
The Test: They asked the robots to rate over 100,000 words on 14 different scales.
- Examples: How "moral" is the word "knife"? How "funny" is "banana"? How "loud" is "whisper"?
The Scale: They asked the same word 5 times to see if the robot was consistent or just guessing.

In total, they gathered 75 million ratings. That's like asking a billion people a question, but with robots.

The Magic Trick: The "Variance Cake"

The researchers used a fancy statistical tool (called a "crossed random-effects model") to slice up the answers like a cake. They wanted to see what ingredients made up the differences in the robots' answers.

They found the cake was made of four layers:

The Consensus (The Truth): Most robots agreed on the basics. Everyone agreed "fire" is hot and "ice" is cold. This was the biggest slice (32%).
The Bias (The Broken Scale): Some robots just liked to give high numbers, and others liked to give low numbers. This was a "grumpy" or "optimistic" bias.
The Noise (Random Guessing): Sometimes robots just made up an answer.
The Secret Ingredient (Machine Individuality): This is the big discovery.

The Result: Even after removing the "broken scales" and the "random guesses," there was still a 17% difference in how the robots rated specific words.

This means Robot A didn't just give higher numbers than Robot B. Robot A genuinely thought the word "butterfly" was more "magical" than Robot B did, even when they were both trying to be accurate.
This difference wasn't random. It was a coherent fingerprint. If a robot had a unique way of seeing "sadness," it also had a unique way of seeing "joy" and "funny." They weren't just glitching; they had a consistent, unique personality.

The Analogy: The Art Gallery

Imagine you are in an art gallery with 10 different critics.

The Consensus: They all agree the painting is "a picture of a boat."
The Bias: One critic always uses big, dramatic words. Another always uses small, quiet words.
The Individuality: But when you ask them how the boat makes them feel, one critic says, "It feels lonely and cold," while another says, "It feels adventurous and free."

The paper proves that these AI models aren't just "broken" versions of the same thing. They are like different critics with their own unique lenses. One might be the "optimist," another the "realist," and another the "dreamer."

Why Does This Matter?

This changes how we talk about AI.

Before: We thought if an AI was "rude," it was just a bug or a bad setting.
Now: We know that different AI models have inherent personalities. One model might naturally be more empathetic, while another is more blunt, not because of how we programmed them, but because of how they learned to see the world.

The Takeaway:
AI models are not identical clones. They are like a group of people who have read the same library of books but interpreted them in their own unique ways. They have Machine Individuality. If you want a robot to be your friend, a judge, or a therapist, you can't just pick "an AI." You have to pick the right AI, because they are all different people.

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed in high-stakes decision-making and companionship roles, understanding their behavioral dispositions is critical. However, existing literature attempting to profile LLM "personalities" using psychometric inventories and cognitive paradigms faces a fundamental methodological flaw: conflation.

Current approaches cannot distinguish between:

Stimulus-specific individuality: Stable, substantive traits where a model evaluates specific situations differently based on its internal semantic representation.
Response biases: Systematic directional shifts (e.g., acquiescence, extreme responding) or stochastic noise that affect all responses uniformly.

Previous studies (e.g., Sun et al., 2025) have shown that models can be distinguished by classifiers, but these signals often rely on surface-level stylistic regularities (response styles) rather than genuine situational evaluation. The authors ask: Does stable behavioral individuality, separable from consensus and bias, exist in LLMs?

2. Methodology

To disentangle these sources of variation, the authors applied crossed random-effects models, a standard tool in psychometrics, to a massive dataset of LLM ratings.

Experimental Design

Models: 10 open-weight LLMs were selected to ensure diversity in architecture (Dense, Mixture-of-Experts, Hybrid), parameter scale (20B–235B), and provider (Alibaba, Mistral, Google, OpenAI, IBM, Microsoft, TII).
Stimuli: 107,083 unique words drawn from the psychNorms metabase.
Tasks: Models were asked to rate these words across 14 psycholinguistic norms, covering:
- Affective dimensions (Valence, Arousal).
- Perceptual dimensions (Concreteness, Sensory norms: Visual, Auditory, Gustatory, Olfactory, Haptic).
- Social/Moral dimensions (Morality, Gender Association, Socialness, Humor).
- Acquisition dimensions (Age of Acquisition).
Data Collection:
- Zero-shot prompting: Models received instructions adapted from original human studies.
- Sampling: Each word was rated 5 times per model using stochastic sampling ( $T=1.0$ ) to capture variance, plus one deterministic pass ( $T=0$ ).
- Scale: Total observations: ~74.9 million ratings.

Statistical Framework

The authors decomposed the rating $y_{ijk}$ (for word $i$ , model $j$ , repetition $k$ ) using a linear mixed-effects model:
$y_{ijk} = \mu + \tau_i + \beta_j + \iota_{ij} + \epsilon_{ijk}$

Where:

$\mu$ : Grand mean.
$\tau_i$ (Trait): Shared consensus across all models for a specific word (semantic reality).
$\beta_j$ (Bias): Systematic directional offset for a specific model (e.g., a model that is generally more "positive" than others).
$\iota_{ij}$ (Idiosyncrasy): The Model $\times$ Word interaction. This captures how a specific model evaluates a specific word net of consensus and general bias. This is the operational definition of Machine Individuality.
$\epsilon_{ijk}$ : Residual noise.

Validation

Null Modeling: Parametric bootstrapping ( $N=100$ ) was used to generate synthetic data without interaction terms to ensure observed idiosyncrasy variance exceeded structural artifacts.
Specificity Analysis: Ridge regression was used to predict a model's idiosyncrasy on a "held-out" norm based on its idiosyncrasies on the other 13 norms. A Specificity Ratio (within-model $R^2$ / cross-model $R^2$ ) > 1.0 indicates a coherent, model-specific fingerprint.

3. Key Results

Variance Partitioning

Trait (Consensus): Dominated the variance at 32.1%, confirming that LLMs share a broad consensus on the semantic landscape.
Idiosyncrasy (Individuality): Accounted for an average of 16.9% of total variance (ranging from 4.8% for Valence to 34.0% for Gender Association).
- This value robustly exceeded the statistical null model (upper bound < 0.033%; $p < 0.01$ for all norms).
- This proves that models differ not just in how much they rate things (bias), but in how they evaluate specific words relative to the consensus.

Coherence and Fingerprinting

Cross-Norm Prediction: The Specificity Ratio for all models ranged from 1.74 to 3.43.
- This indicates that a model's unique deviation on one dimension (e.g., Morality) is best predicted by its own unique deviations on other dimensions, rather than by the patterns of other models.
- This confirms that machine individuality is a coherent, multi-dimensional fingerprint.

Human Alignment & Decoding

Stochastic vs. Deterministic: Stochastic aggregation (averaging 5 samples) correlated better with human norms ( $\Delta \bar{r} = 0.032$ ) than deterministic decoding.
Trade-off: Deterministic decoding maximizes replicability but sacrifices the variance structure (idiosyncrasy) and human alignment.
Sui Generis Nature: LLMs correlated moderately with human norms ( $\bar{r} = 0.48–0.67$ ), suggesting their semantic representations are partly unique rather than mere noisy approximations of human judgment.

4. Key Contributions

Conceptual Definition: Introduces the term "Machine Individuality" to describe stable, stimulus-specific behavioral differences in LLMs that are distinct from response bias and noise.
Methodological Advancement: Demonstrates the application of crossed random-effects models to LLM evaluation, providing a rigorous statistical framework to separate substantive traits from response styles.
Empirical Evidence: Provides the first large-scale evidence (74.9M ratings) that LLMs possess genuine, coherent individuality across diverse psychological dimensions.
Paradigm Shift: Argues that future evaluations of LLM dispositions must move beyond aggregate "personality scores" (which conflate bias and trait) to analyze stimulus-specific interactions.

5. Significance

For AI Safety and Alignment: Understanding machine individuality is crucial for deploying LLMs in sensitive contexts. If a model has a specific "fingerprint" that makes it harsh on moral judgments or overly sensitive to gender associations, this cannot be corrected by simple prompt engineering or bias mitigation that only targets global offsets.
For Model Selection: Users can now select models not just by capability (speed/accuracy) but by their specific behavioral "personality" or evaluation style, knowing these traits are stable and coherent.
For AI Psychology: The study establishes that LLMs are not monolithic "hiveminds" but possess distinct, measurable internal states that interact with the environment (prompts/words) in unique ways. This bridges the gap between psychometrics and AI, suggesting that "machine personality" is a real, quantifiable phenomenon.

Conclusion: The paper concludes that machine individuality is not only measurable but consequential. Future research must investigate whether these idiosyncrasies persist in contextualized, multi-turn interactions and whether they predict downstream behavioral differences in real-world applications.