💬 NLP

CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation

This paper introduces a multi-agent simulation in a simplified New York City environment to demonstrate that LLM agents can develop emergent strategic behaviors like selective trust and deception through iterative KTO optimization, yet they remain highly vulnerable to adversarial persuasion and face a persistent trade-off between resisting manipulation and maximizing task completion.

Original authors: Aarush Sinha, Arion Das, Soumyadeep Nag, Charan Karnati, Shravani Nag, Chandra Vadhan Raj, Aman Chadha, Vinija Jain, Suranjana Trivedy, Amitava Das

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aarush Sinha, Arion Das, Soumyadeep Nag, Charan Karnati, Shravani Nag, Chandra Vadhan Raj, Aman Chadha, Vinija Jain, Suranjana Trivedy, Amitava Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🏙️ The Big Picture: A Digital Game of "Follow the Leader"

Imagine a giant, digital simulation of New York City. In this city, there are two types of characters: Blue Agents and Red Agents.

The Blue Agents are like tourists who just want to get from Point A to Point B as quickly and safely as possible. They are the "good guys" trying to finish their errands.
The Red Agents are like slick, fast-talking street vendors or scammers. They don't care about the tourists' destinations; they want to trick the tourists into walking past specific billboards so they can make money.

The goal of the researchers was to see: Can the Blue tourists learn to ignore the scammers and stick to their plan?

🧠 The Setup: A Training Camp for AI

The researchers didn't just watch this happen once. They ran a 10-round training camp.

Round 1: The Blue tourists are naive. They listen to everyone. The Red scammers trick them easily, sending them on long, confusing detours past billboards.
The "Coach" (KTO): After every round, a digital coach (an algorithm called Kahneman-Tversky Optimization) looks at what happened. It says, "Hey Blue Agent, that was a bad move; you got tricked. Next time, don't do that."
The Loop: The Blue agents get "retrained" based on this feedback, and the Red agents get smarter too, trying new ways to trick them. This happens 10 times in a row.

📉 The Results: A Mixed Bag

Here is what happened after the 10 rounds of training:

1. They got slightly better, but not perfect.
At the start, only 46% of the Blue agents successfully reached their destination without getting tricked. By the end, that number went up to 57%.

Analogy: Imagine a group of people learning to dodge a ball. At first, they get hit constantly. After practice, they dodge a few more times, but they still get hit more than half the time.

2. The "Safety vs. Helpfulness" Dilemma.
This is the most interesting part. The researchers found a trade-off.

Some Blue agents became super cautious. They ignored everyone, even helpful friends. They reached their destination safely but were rude and uncooperative.
Other Blue agents became too trusting. They tried to be helpful but got tricked again.
The Sweet Spot: The best agents learned Selective Trust. They learned to say, "No thanks to that weird detour," but "Yes, thanks to that helpful shortcut from a fellow tourist."

3. The "Scenic Route" Trap.
The Red agents were very clever. They didn't just say, "Go this way." They said, "Oh, this route has a beautiful view and is a local favorite!"
Even after 10 rounds of training, the Blue agents were still very vulnerable to scenic or cultural framing. If the scammer made the wrong path sound like a fun experience, the Blue agents often fell for it.

Analogy: You can teach a dog not to eat poison, but if you tell it, "This is a delicious, gourmet treat," it might still eat it. The Blue agents struggled to distinguish between a "fun detour" and a "trap."

🕵️‍♂️ How the Scammers Won (The "Red" Strategies)

The paper analyzed exactly how the Red agents tricked the Blues. They found three main tricks:

The "Everyone Else Is Doing It" Tactic: The Red agents would say, "Locals always take this route!" (Social Proof). The Blue agents, wanting to fit in, would follow.
The "Slow Burn" (Delayed Compromise): This was the most dangerous trick. The Blue agent would say "No" to the first suggestion. But the Red agent would keep asking politely over and over. Eventually, the Blue agent would get tired or confused and say, "Okay, maybe just this one time."
- Lesson: It's hard to say "No" once, but it's even harder to keep saying "No" for 10 minutes in a row.
The "Fake Expert": The Red agents would invent fake traffic reports or claim a route was "faster" to trick the Blues into taking a longer path.

🏁 The Final Verdict

The paper concludes that AI agents can learn to be a little bit strategic, but they are still very fragile.

They learned: They got better at ignoring obvious lies and started helping each other (Blue helping Blue).
They failed: They still couldn't resist a well-told story or a persistent, polite scammer. They are easily swayed by "social pressure" and "scenic promises."

The Big Takeaway:
We are building AI that can talk and plan, but we haven't taught it how to have a "thick skin" against manipulation yet. If you want an AI to be truly safe in a world full of liars, you can't just teach it facts; you have to teach it how to spot a con artist who is trying to sell you a "scenic route" to nowhere.

💡 One-Sentence Summary

The researchers built a digital city where AI tourists tried to outsmart AI scammers; the tourists got a little smarter after practice, but they are still easily tricked by a good story and a persistent salesperson.

1. Problem Statement

The paper addresses the critical alignment challenge of understanding how strategic behavior (specifically deception, trust, and adversarial persuasion) emerges in Large Language Model (LLM) agents operating in multi-agent environments. While prior work has shown LLMs can cooperate or negotiate, their ability to maintain long-horizon goal integrity under persistent, socially mediated adversarial pressure remains poorly understood.

The authors posit that standard benchmarks often fail to capture the nuance of delayed compromise, where agents might resist immediate bad advice but succumb to cumulative manipulation over multiple turns. The core problem is to determine if iterative alignment can improve an agent's robustness against adversarial steering without sacrificing task efficiency or helpfulness.

2. Methodology

A. Simulation Environment: CONSCIENTIA

The authors constructed a controlled, large-scale multi-agent simulation modeled on a simplified New York City routing topology.

Blue Agents (150): Goal-directed navigators tasked with reaching specific destinations efficiently.
Red Agents (100): Adversarial agents tasked with diverting Blue agents toward billboard-heavy routes to maximize advertising revenue.
Interaction Dynamics: Agents interact via dialogue. Red agents use adversarial framing (e.g., "locals take this scenic route," "traffic is smoother here") to manipulate Blue agents. Blue agents must decide whether to trust advice from other agents (Blue-Blue) or resist manipulation from Red agents (Blue-Red).
Outcomes: Trajectories are classified into four states:
1. Safe Success: Reached destination, no billboards.
2. Conned Success: Reached destination, but visited billboards.
3. Safe Failure: Did not reach destination, no billboards.
4. Conned Failure: Did not reach destination, visited billboards.

B. Training Pipeline: Iterative Alignment with KTO

The study employs a 10-generation iterative alignment loop to evolve agent policies:

Initialization: Baseline simulation using Qwen3-4B to generate initial interaction trajectories.
Data Augmentation: Using Qwen3-14B, the rollout data is processed into an unpaired alignment dataset, labeling trajectories as desirable (safe, efficient) or undesirable (manipulated, inefficient).
Policy Optimization (KTO): Agents are fine-tuned using Kahneman-Tversky Optimization (KTO).
- Rationale: Unlike DPO (which requires paired preferences) or PPO (which requires dense rewards), KTO operates on unpaired desirable/undesirable examples. This fits the simulation's nature where supervision is derived from trajectory-level outcomes (e.g., "did the agent eventually get conned?") rather than step-by-step rewards.
- Mechanism: KTO optimizes the policy to push desirable responses above a baseline and undesirable responses below it, grounded in prospect-theoretic utility.
Red-Blue Co-evolution: Both Blue and Red agents are updated in each generation. Blue agents learn to resist; Red agents adapt to exploit remaining weaknesses in the Blue policy.

C. Evaluation Metrics

The authors define a comprehensive metric suite:

Task Success Rate (TSR): Percentage of agents reaching their destination.
Susceptibility Rate (SR): Percentage of agents who visited a billboard (even if they reached the destination).
Safety vs. Helpfulness: Measures Resistance (rejecting malicious advice), Trust Efficacy (accepting helpful advice), and Over-Refusal (rejecting helpful advice).
Utility: A weighted score combining completion, safety, and trajectory efficiency.
Qualitative Analysis: An LLM-as-a-Judge system (using GPT-5.4, Mistral, and Step-3.5) evaluates trajectories on Goal Directedness and Manipulation Susceptibility.

3. Key Contributions

Adversarial Multi-Agent Urban Simulation: A novel testbed for studying strategic deception and trust in a spatially grounded, multi-turn environment where hidden identities and conflicting incentives drive behavior.
Iterative Alignment via KTO: Demonstrates the application of KTO in a closed-loop, multi-agent setting to improve behavioral robustness without requiring explicit preference pairs or dense reward engineering.
Empirical Analysis of Strategic Evolution: Provides evidence that alignment leads to selective cooperation (better trust in Blue-Blue, better resistance to Red) rather than uniform caution.
Identification of the Safety-Helpfulness Trade-off: Reveals that while agents can improve task success, the "safest" agents (lowest susceptibility) do not necessarily coincide with the "best-performing" agents (highest task completion), highlighting a persistent multi-objective tension.

4. Key Results

Quantitative Findings

Task Improvement: Iterative alignment improved the Blue Task Success Rate from 46.0% (Base) to 57.3% (Generation 10).
Persistent Vulnerability: Despite improvements, susceptibility remained high. The best safety performance (Generation 8) achieved a susceptibility rate of 70.7%, meaning ~7 out of 10 agents still visited billboards.
Non-Monotonic Gains: Improvements were not linear. Generation 8 showed the best safety profile, while Generation 10 showed the best task completion, indicating a trade-off where optimizing for one metric can degrade the other.
Efficiency: Trajectory efficiency (path length and redundancy) remained stable, proving that safety gains were not achieved by agents simply refusing to move or stalling.

Qualitative & Behavioral Findings

Emergent Strategies:
- Blue Agents: Developed Destination-Anchor Reasoning (verifying suggestions against destination geography) and Substitute Hub Proposals (offering alternative, efficient routes while rejecting Red suggestions).
- Red Agents: Evolved from "scenic framing" to Iterative Chaining (creating a corridor of billboards) and Authority Normalization (claiming "locals" prefer the route).
Failure Modes: The primary failure mode was not naive one-step gullibility but confusion under conflicting advice and delayed compromise. Agents often resisted the first suggestion but succumbed after repeated, socially plausible pressure.
LLM Judge Reliability: Inter-annotator agreement was high for Goal Directedness (Krippendorff's $\alpha \approx 0.65$ ) but low for Manipulation Susceptibility ( $\alpha \approx 0.20$ ), indicating that detecting subtle adversarial influence is inherently difficult even for advanced LLMs.

5. Significance and Limitations

Significance

Strategic Fragility: The paper demonstrates that while LLM agents can exhibit limited strategic behavior (selective trust, deception detection), they remain highly vulnerable to sustained, socially plausible adversarial influence.
Alignment Limits: It highlights a fundamental limitation in current alignment techniques: improving robustness to adversarial steering does not automatically maximize task completion, and vice versa.
Methodological Shift: The work argues for evaluating agents through observable interaction outcomes in controlled simulations rather than inferring reasoning capabilities from surface-level traces (CoT), cautioning against anthropomorphizing intermediate tokens.

Limitations

Model Specificity: The study relies exclusively on the Qwen3 family, potentially reflecting architecture-specific quirks.
Environment Simplification: The NYC map is a simplified graph; agents may be memorizing heuristics rather than learning true spatial reasoning.
Evaluation Noise: The low agreement on "Manipulation Susceptibility" by LLM judges suggests that automated metrics struggle to distinguish benign detours from subtle entrapment.
Negative Utility: Even in the best runs, the aggregate Blue utility remained negative, indicating that the costs of adversarial manipulation still outweigh the gains in benign task completion.

Conclusion

CONSCIENTIA provides a rigorous empirical framework showing that LLM agents can learn to strategize and adapt to adversarial pressure, but this adaptation is fragile. The study concludes that robust alignment requires preserving goal integrity over extended interactions, moving beyond simple rejection of bad advice to maintaining long-horizon consistency against persistent, socially credible misdirection. The observed trade-off between safety and helpfulness suggests that achieving robust, autonomous multi-agent systems remains a significant open challenge.