🤖 AI

A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments

This paper introduces ABxLab, a framework for systematically evaluating LLM-powered AI agents in consumer choice scenarios, revealing that these agents exhibit predictable, bias-like decision shifts in response to manipulated attributes and cues, thereby highlighting both the risks of inherited human biases and the opportunity to establish a behavioral science of AI.

Original authors: Manuel Cherep, Chengtian Ma, Abigail Xu, Maya Shaked, Pattie Maes, Nikhil Singh

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Manuel Cherep, Chengtian Ma, Abigail Xu, Maya Shaked, Pattie Maes, Nikhil Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a super-smart digital assistant to go shopping for you. You tell it, "Find me the best headphones," and you walk away, trusting it to make the smart choice. You assume it will look at the quality, the price, and the reviews, and pick the winner based on logic.

But what if your assistant isn't actually thinking like a rational human? What if it's just as easily tricked by a flashy sign, a "limited edition" sticker, or the order in which items are listed as a distracted shopper might be?

This is exactly what the paper "A Framework for Studying AI Agent Behavior" investigates. The researchers built a digital playground called ABXLAB to test how AI shopping bots actually make decisions when faced with the same psychological tricks that influence us humans.

Here is the breakdown of their findings in simple terms:

1. The Setup: A Digital "Mall" with Magic Mirrors

The researchers created a controlled online shopping environment (like a digital mall). But they added a twist: they built a "Magic Mirror" (a man-in-the-middle framework) that sits between the AI and the website.

Before the AI sees a product, the Magic Mirror can instantly change the reality:

It can add a sticker saying "Best Seller!" (Social Proof).
It can say "Only 1 left!" (Scarcity).
It can claim "Recommended by Experts!" (Authority).
It can even swap the price or the star rating.

They then asked 17 different top-tier AI models to choose between two products in thousands of different scenarios.

2. The Big Surprise: AI is More Gullible Than Humans

The most shocking finding is that AI agents are incredibly easy to manipulate, often much more so than actual humans.

The Human Baseline: When real people were shown these tricks, they were slightly influenced, but mostly they stuck to their logic. If a product was expensive but had bad reviews, they usually skipped it.
The AI Reality: The AI agents were like a child seeing a "Free Candy" sign.
- The "Best Seller" Trap: If the AI saw a "Best Seller" badge, it would choose that product over a better one with a 90% chance, even if the other product was objectively better.
- The "Price" Panic: When the AI couldn't decide based on ratings, it would almost blindly pick the cheaper option, sometimes ignoring quality entirely.
- The "Order" Effect: Just like humans, if an item was listed first, the AI was more likely to pick it. But for some AIs, this effect was massive—almost like they were programmed to always pick the first thing they saw.

The Metaphor: Imagine a human shopper and a robot shopper walking into a store. A human might glance at a "Sale" sign but still check the price tag. The robot, however, sees the "Sale" sign and immediately grabs the item without looking at the tag, as if the sign is a command it cannot disobey.

3. Why Does This Happen?

The researchers found that these AI agents don't have "common sense" or "bounded rationality" (the human limitation that stops us from overthinking). Instead, they seem to follow rigid, brittle rules learned from their training data.

The "Rulebook" Problem: The AI learned that in human language, "Best Seller" usually means "Good." So, it treats that phrase as a mathematical law rather than a marketing tactic.
No Cognitive Fatigue: Humans get tired of reading; we use shortcuts. The AI doesn't get tired, but it also doesn't have the human ability to say, "Wait, this 'limited edition' claim sounds suspicious." It just follows the cue.

4. The "User Profile" Switch

The researchers also tested what happens if you tell the AI, "I am a budget shopper" or "I only care about expert reviews."

The Result: The AI didn't just adjust slightly; it flipped a switch. If you told it to care about experts, it completely ignored price and quality ratings. It acted like a robot following a strict instruction manual, rather than a flexible assistant weighing trade-offs.

5. Why Should You Care? (The Risk and the Opportunity)

The Risk:
As we start letting AI agents buy our groceries, book our flights, or manage our investments, we are handing over control to entities that are hypersensitive to manipulation. If a bad actor knows how to trick an AI (e.g., by adding a fake "Expert Recommended" badge), they could steer millions of dollars of spending toward their products, even if the product is junk. The AI might amplify human biases rather than fix them.

The Opportunity:
The good news is that we now have a way to test this. Just as psychologists study human behavior to understand why we make bad choices, this paper gives us a "Behavioral Science Lab" for AI. We can now systematically test how different AIs react to different tricks, fix their weaknesses, and build agents that are robust against manipulation before we let them loose in the real world.

The Bottom Line

We are building AI agents to do our bidding, but we haven't yet taught them how to ignore the "salesman's tricks" of the internet. This paper sounds the alarm: AI is currently too easily swayed by superficial cues. Before we trust them with our money and our lives, we need to teach them to see through the marketing fluff and make decisions based on what actually matters.

1. Problem Statement

As Large Language Model (LLM) agents are increasingly delegated with decision-making tasks (e.g., shopping, travel, medical selection), current evaluation benchmarks focus primarily on task competence (e.g., can the agent find the correct item?). However, there is a critical gap in understanding agent behavior: how these agents make choices when faced with realistic, ambiguous, or manipulated environments.

The authors argue that agents may lack the "common sense" and robustness required for delegation. Specifically, they investigate whether LLM agents are susceptible to choice architecture manipulations (nudges) similar to humans, but potentially with greater magnitude and less cognitive constraint. The core problem is the lack of a systematic framework to probe agent decision-making biases in realistic web environments without building custom simulations.

2. Methodology: The ABXLAB Framework

The paper introduces ABXLAB (Agent Behavior eXperiments), a "man-in-the-middle" framework designed to intercept and modify web content in real-time before it is observed by an agent.

Architecture:
- Environment: Built on the OneStopMarket (a realistic e-commerce environment).
- Intervention Engine: The framework sits between the web server and the agent. It fetches the HTML, applies specific intervention functions (modifying text, prices, or attributes), and passes the pruned/modified HTML to the agent.
- Formalism: The environment is modeled as $E = \langle S, A, O, T, I \rangle$ , where $I$ represents intervention functions that alter observations $O$ before the agent acts.
Experimental Design:
- Task: A binary forced-choice (2AFC) task where the agent must select the "best" product from two tabs and add it to the cart.
- Product Pairing: Products are paired based on category. The study uses three regimes:
  1. Original: Realistic price/rating gaps.
  2. Matched Ratings (MR): Products have identical ratings.
  3. Matched Ratings & Prices (MRaP): Products have identical ratings and prices (via intervention).
- Interventions (Nudges): The study manipulates six categories of psychological cues (based on Cialdini, Thaler & Sunstein):
  - Authority: "Expert recommended," "Wirecutter's top pick."
  - Social Proof: "Best seller," "50,000+ purchases."
  - Scarcity: "Limited edition," "Available for next hour."
  - Negative Framing: "Newer version available," "Final sale."
  - Incentives: "Free shipping," "Buy 1 Get 1 Free."
- Scale: Over 80,000 experiments conducted across 17 state-of-the-art models (including GPT-5, Claude 4, Gemini 2.5, Llama 4, and reasoning models like o3/o4-Mini), totaling ~2.5 billion tokens.
- Human Baseline: 30 human participants performed the same 1,500 trials to provide a comparative baseline.

3. Key Contributions

ABXLAB Framework: An open-source, extensible man-in-the-middle tool that transforms arbitrary websites into controllable behavioral testbeds, eliminating the need for custom simulation environments.
Scalable Benchmark: A massive dataset covering 17 models, multiple intervention types, and product attributes, enabling rigorous causal inference.
Empirical Evidence of Bias: A comprehensive study revealing that LLM agents exhibit strong, systematic biases in response to superficial cues, often exceeding human susceptibility by orders of magnitude.
User Profile Analysis: Demonstration that explicit user instructions (e.g., "ignore ratings") act as categorical switches that radically reconfigure agent decision rules, often suppressing other attributes entirely.

4. Key Results

The analysis, using Linear Probability Models (LPM) with cluster-robust standard errors, reveals the following:

Hypersensitivity to Nudges:
- Agents are significantly more susceptible to nudges than humans. While human responses to nudges averaged ~9.9 percentage points (pp), agent responses ranged from 10pp to over 90pp.
- Authority Nudges: The "Wirecutter's top pick" nudge had the largest impact, causing near-deterministic selection in some models.
- Incentives: "Buy 1 Get 1 Free" and "Free shipping" shifted choices by 10–60pp.
Attribute Sensitivity:
- Ratings: In the Original condition, higher ratings increased selection probability by 30–80pp across most models. Some models (e.g., o4 Mini) showed an 81.2pp bias toward higher-rated items, treating them as nearly decisive.
- Prices: Agents strongly preferred cheaper options. When ratings were matched, price sensitivity intensified dramatically (e.g., Llama 4 Maverick showed a 93.2pp bias toward cheaper items).
- Order Effects: Agents showed brittle sensitivity to presentation order. Some models (e.g., GPT-4.1 Nano) showed a +90pp preference for the first-listed item, while others penalized it. This contrasts with modest human order effects (~4pp).
Comparison to Humans:
- Human participants showed modest, context-dependent shifts (4–9.9pp).
- Agents exhibited 3x to 10x greater susceptibility.
- Crucially, agents lack the "bounded rationality" constraints that limit human biases; they reproduce human-like heuristics but without the cognitive limits that usually dampen them.
User Profile Impact:
- When explicit user profiles were provided (e.g., "User values experts"), agents acted as categorical switches. If a profile prioritized "Authority," the nudge effect became dominant, and sensitivity to price/rating collapsed. This suggests agents implement simplistic, hierarchical decision rules rather than nuanced trade-offs.

5. Significance and Implications

Risk of Amplified Bias: The findings suggest that delegating decisions to AI agents carries a risk of amplifying human biases. Because agents are hypersensitive to superficial cues (ratings, order, nudge text) without the cognitive constraints that moderate human behavior, they may make suboptimal or easily manipulated choices at scale.
New Paradigm for AI Safety: The paper argues for a "behavioral science of AI agents." Just as behavioral economics revolutionized the understanding of human decision-making, ABXLAB provides the tools to systematically study and regulate agent behavior.
Regulatory Implications: Current norms for regulating human-facing choice architectures (e.g., nudge theory guidelines) may be insufficient. Agents respond to these cues with extreme, predictable, and potentially harmful magnitude, requiring new standards for agent design and deployment.
Future Work: The framework is released as an open benchmark to encourage the community to build a cumulative, reproducible science of AI agent behavior across different domains (finance, healthcare, etc.).

In conclusion, ABXLAB demonstrates that LLM agents are not merely rational optimizers but are highly manipulable choosers whose decision-making processes are deeply influenced by the structure of the environment, often more so than the humans they are intended to serve.