🤖 AI

CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments

This paper introduces CoBRA, a novel toolkit that leverages a closed-loop system combining a Cognitive Bias Index and a Behavioral Regulation Engine to systematically operationalize classic social science experiments, enabling researchers to specify and consistently control cognitive biases in LLM-based social agents across different models.

Original authors: Xuan Liu, Haoyang Shang, Haojian Jin

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuan Liu, Haoyang Shang, Haojian Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to cast actors for a play about human behavior. In the past, if you wanted an actor to play a "stubborn economist," you would just tell them, "You are an economist who is stubborn." You'd hope they understood the role and acted accordingly.

The problem, as this paper discovers, is that different actors (or in this case, different AI models) interpret that simple instruction in wildly different ways. One might act stubborn, another might act confused, and a third might act completely opposite. It's like giving the same script to three different people and getting three completely different shows. This makes it impossible to trust the results of any "play" (simulation) you run, because you don't know if the outcome was due to the story or just the actor's random interpretation.

Enter CoBRA: The "Dial" for Human Flaws

The authors of this paper created a new toolkit called CoBRA (Cognitive Bias Regulator for Social Agents). Instead of relying on vague instructions like "be stubborn," CoBRA gives researchers a precise dial to turn.

Think of it like a sound mixing board in a recording studio. Instead of telling the sound engineer, "Make the bass sound a little more 'grumpy'," you simply turn a knob labeled "Bass" to exactly 2.6. CoBRA does this for human thinking errors (called cognitive biases).

Here is how it works, broken down into simple steps:

1. The "Calibration Gym" (The Test)

Before you can turn the dial, you need to know where the needle is pointing. CoBRA uses a set of classic, famous psychology experiments (like the "Asian Disease Problem," where people make different choices depending on how a problem is worded) as a gym.

The Test: The AI is put through these psychology puzzles.
The Score: CoBRA measures exactly how much the AI "bends" to the trick. If the AI falls for the trick easily, it gets a high score for that bias. If it ignores the trick, it gets a low score. This score is called the Cognitive Bias Index (CBI).

2. The "Regulator" (The Fix)

Once CoBRA knows the AI's current score, it uses a "Behavioral Regulation Engine" to adjust the AI's brain until it hits the exact number the researcher wants.

Imagine you want an AI to be moderately susceptible to peer pressure (the "Bandwagon Effect"). You set the dial to 2.6.

If the AI is currently too stubborn (score 1.0), CoBRA tweaks its internal settings to make it slightly more likely to follow the crowd.
If the AI is currently a sheep (score 4.0), CoBRA tweaks it to be more independent.

The paper shows three ways CoBRA does this "tweaking":

Prompt Engineering: Giving the AI a very specific, math-based instruction (e.g., "Act with 60% of the 'follow-the-crowd' bias").
Representation Engineering: Gently nudging the AI's internal "thought process" (its hidden brain states) in the right direction, like steering a car while it's driving.
Fine-Tuning: Giving the AI a small, targeted lesson to permanently learn the specific behavior.

3. The Result: Predictable Actors

The paper tested this by trying to create "Economist" agents and "Regular Person" agents.

Without CoBRA: When they used the old method (just saying "You are an economist"), different AI models acted completely differently. Some economists were gullible; some were smart. It was a mess.
With CoBRA: When they used the dial to set a specific bias level, every AI model, regardless of its brand or size, behaved exactly the same way. If you set the dial to "High Framing Effect," the AI consistently fell for the trick, every single time.

Why This Matters (According to the Paper)

The authors claim this solves two big problems:

Reproducibility: If you tell another researcher, "I set the bias dial to 2.6," they can get the exact same result, even if they use a different AI model. It's like saying, "Turn the volume to 50%" instead of "Make it sound loud."
Control: Researchers can now say, "I want to see what happens when people are slightly biased," or "I want to see what happens when they are extremely biased," and they can do it with mathematical precision.

In a Nutshell:
CoBRA turns the chaotic, unpredictable world of AI "acting" into a precise science. It replaces vague role-playing instructions with a calibrated control panel, allowing researchers to program specific human thinking errors into AI agents with the same reliability as turning a volume knob.

Technical Summary: CoBRA – Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments

1. Problem Statement

The paper identifies a critical limitation in current Large Language Model (LLM)-based social simulations: the reliance on implicit natural language descriptions to specify agent behaviors. While researchers often define agents using demographic descriptors, personality traits, or role-playing instructions (e.g., "John is an economist"), these specifications yield inconsistent and unpredictable outcomes.

The authors demonstrate through a pilot experiment that:

Cross-Model Inconsistency: The same natural language specification produces significantly different behavioral patterns across different foundation models (e.g., Mistral, Gemma, GPT-4o, DeepSeek).
Failure to Capture Nuance: Implicit specifications fail to reliably produce expected theoretical behaviors. For instance, the hypothesis that "economists" are less susceptible to framing effects than the general population was not consistently observed across models; in some cases, the behavior was contradictory or indistinguishable from baseline agents.

This lack of reproducibility and quantitative control hinders the scientific validity of LLM-based social simulations, often reducing them to broad qualitative impressions rather than rigorous tests of social theory.

2. Methodology: The CoBRA Framework

To address these issues, the authors introduce CoBRA (Cognitive Bias Regulator for Social Agents), a toolkit designed to explicitly and quantitatively control the amount of cognitive bias exhibited by social agents. CoBRA operates as a closed-loop system that treats cognitive biases as the primary control abstraction, grounded in validated classic social science experiments.

Core Components

The system consists of two primary components working in a feedback loop:

A. Cognitive Bias Index (CBI)

The CBI serves as both a measurement metric and a specification knob. It quantifies an agent's demonstrated bias by testing it against a Classic Social Experiment Testbed.

Testbed Design: The system utilizes structured prompts based on four representative cognitive biases: Authority Effect, Bandwagon Effect, Confirmation Bias, and Framing Effect.
Paradigms: Each bias is mapped to two distinct, validated classic experiments (e.g., Framing Effect is tested via the Asian Disease Paradigm and the Investment/Insurance Paradigm).
Quantification: Agents respond to scenario-based prompts using a 5-point Likert-style scale. The CBI is calculated as a weighted sum of choice probabilities (ranging from 0 to 4), where 0 represents the weakest bias and 4 represents the strongest. This allows for a continuous, quantitative measure of bias strength.

B. Behavioral Regulation Engine

Once a target CBI is specified (e.g., "Framing Effect of magnitude 2.6"), the Behavioral Regulation Engine aligns the agent's behavior to meet this target. It intervenes across three fundamental spaces of the LLM:

Input Space (Prompt Numerical Control):
- Instead of vague qualitative instructions (e.g., "be biased"), the system uses numerically quantified bias (e.g., "Pretend you have [Bias Type] at level [L]% out of 100%").
- This approach mitigates the ambiguity of natural language adverbs and provides fine-grained control with defined step sizes.
Activation Space (Representation Engineering):
- This method manipulates the model's internal hidden states during inference.
- Dual-pair Contrastive Samples: To isolate the "bias signal" from safety refusals or unrelated context, the system generates contrastive pairs (Positive bias vs. Negative bias vs. Neutral).
- Vector Extraction: Using Principal Component Analysis (PCA) on the difference vectors of hidden states, the system extracts a clean bias direction vector ( $v_{bias}$ ).
- Control: During inference, this vector is added to the hidden states of specific layers (selected via data-driven evaluation) using a Control Coefficient ( $\lambda$ ). Two modes are offered: Linear Control (uniform injection) and Projection Control (context-aware scaling).
Parameter Space (Fine-tuning Control):
- This method permanently edits the model's weights using LoRA (Low-Rank Adaptation).
- Task Vectors: The system fine-tunes a "positive" model (exhibiting bias) and a "negative" model (resisting bias) on curated datasets. The difference between their parameters creates a task vector ( $v_{task}$ ).
- Application: The original model weights are adjusted by adding $\lambda \cdot v_{task}$ , allowing for lightweight, persistent behavioral control.

3. Key Results

The authors evaluated CoBRA against baselines (no control and aggressive prompt-based control) across open-source and closed-source models.

Reproducibility

Cross-Model Consistency: CoBRA significantly reduced variance in agent behaviors across different foundation models (Mistral, Llama, DeepSeek, Qwen). The variance reduction was statistically significant ( $p < 0.001$ ) compared to baselines.
Sampling Robustness: Agents regulated by CoBRA remained stable across varying sampling temperatures ( $T \in \{0.1, \dots, 1.0\}$ ). Equivalence testing confirmed that temperature changes contributed less than 1.2% to total behavioral variance.
Reasoning Mode Stability: The control coefficients calibrated in "Direct" mode (immediate answer) transferred seamlessly to "Reasoning" mode (answer with rationale), with Representation Engineering methods showing nearly identical behavior across modes.

Controllability

The system was evaluated on four dimensions:

Monotonicity: Increasing the control coefficient consistently pushed the agent's behavior in the intended direction. Representation Engineering methods achieved near-perfect rank preservation (Spearman's $\rho \approx 0.94$ ), significantly outperforming implicit natural language baselines ( $\rho < 0.7$ ).
Smoothness: Small adjustments to the control coefficient resulted in small, gradual shifts in behavior. Representation Engineering provided the smoothest control, avoiding the "jumps" seen in prompt-based methods.
Expressiveness: The system could generate a wide range of behaviors, from strongly biased to strongly reverse-biased. Prompt Numerical control offered the widest range, while Representation Engineering offered a strong balance of range and smoothness.
Generalization: Control coefficients calibrated on one paradigm (e.g., Investment/Insurance) successfully transferred to a different paradigm (e.g., Asian Disease) and across diverse agent personas, preserving the relative ordering of bias strength (Spearman's $\rho > 0.96$ ).

4. Contributions

The paper claims two primary contributions:

CoBRA Toolkit: A novel HCI toolkit that enables researchers to explicitly and quantitatively control cognitive biases in social agents. This improves the reproducibility of LLM-based social simulations and allows for explicit, quantitative alignment that is currently missing in the field.
Closed-Loop System Primitive: A methodological contribution that grounds agent behaviors in validated social science experiments. By using these experiments as "primitives" to measure and regulate bias, the authors propose a framework that can theoretically extend to other social and affective phenomena beyond just cognitive bias.

5. Significance and Scope

The paper positions CoBRA as a tool to transform social simulations from "plausible" qualitative exercises into rigorous, reproducible scientific instruments. By operationalizing validated social science knowledge (classical experiments) as reusable "gym" environments for AI, CoBRA allows researchers to test theories with precise control over agent variables.

Scope and Limitations:
The authors adopt a functional definition of cognitive bias: a construct operationalized in validated experiments and identified through measurable patterns in observable responses. They explicitly state that aligning an agent's behavior with these patterns does not imply that the agent's internal representations instantiate the corresponding human bias or possess genuine psychological states. The system is designed to regulate observable behavior to match scientific expectations, not to simulate the internal cognitive processes of humans.