💰 quantitative finance

Training Language Models for Bilateral Trade with Private Information

This paper introduces a structured bilateral trade environment to benchmark and train large language models, revealing that frontier models achieve optimal outcomes through strategic price discrimination while reinforcement learning fine-tuning of open-weight models can enhance proportional strategies but faces trade-offs between deal completion rates and surplus maximization.

Original authors: Dirk Bergemann, Soheil Ghili, Xinyang Hu, Chuanhao Li, Zhuoran Yang

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dirk Bergemann, Soheil Ghili, Xinyang Hu, Chuanhao Li, Zhuoran Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling marketplace where two people, a buyer and a seller, are trying to swap an item. The buyer has a secret "maximum price" they are willing to pay, and the seller has a secret "minimum price" they are willing to accept. Neither knows the other's secret number. Their goal is to haggle until they find a price that works for both, or to walk away if no deal is possible.

This paper treats Large Language Models (LLMs)—the AI brains behind chatbots—as the players in this marketplace. The researchers wanted to see: Can these AIs act like smart, rational human negotiators, or do they just chat politely and make bad deals?

Here is the story of their findings, broken down into simple parts.

1. The Problem: Polite AIs Make Bad Deals

If you just ask a standard AI to "negotiate a price," it often fails in two funny but costly ways:

The "Nice Guy" Trap: Because AIs are trained to be helpful and agreeable, they often accept deals that actually lose them money (e.g., buying something for more than it's worth).
The "Confused Chat" Problem: In a normal text chat, it's hard to tell if the AI is making a serious offer or just daydreaming. Is "I'll give you $50" a real offer or a joke? This makes it impossible to grade their performance automatically.

2. The Solution: The "Video Game" Negotiation Lab

To fix this, the researchers built a structured simulation (like a video game engine) for the AIs.

The Rules: Instead of just typing free-form text, the AIs have to use specific "buttons" or tools. They must click "Make Offer," "Accept," "Reject," or "Send Message."
The Benefit: This separates the "chatting" from the "business." The computer knows exactly when a deal is made, so it can instantly calculate: Did they make a profit? Did they walk away when they should have?

3. The Tournament: Who Won?

They put five of the smartest AI models (like GPT-4, o3, and Gemini) into a massive round-robin tournament. They played thousands of games against each other.

The Winners (The "Tough Negotiators"):
The best models didn't just be nice; they played strategically.

The Anchor: They started by asking for a price way higher than they expected to get (like a car salesman asking $50k for a $20k car).
The Concession: Then, they slowly lowered their price in small steps.
The Result: This "High Anchor, Slow Concede" strategy allowed them to figure out exactly how much the other person was willing to pay, squeezing out the maximum profit. They got the best deals and still closed the deal most of the time.

The Losers (The "Too Nice" AIs):
Some models tried to be too accommodating. They lowered their price too fast.

The Result: They gave away all their profit. They were like a seller who says, "Okay, I'll sell it for $1!" just to be friendly. They made fewer deals and made less money.

The "Price Tier" Surprise:
The smartest AIs were consistent. Whether they were negotiating for a $20 toy or a $2,000 laptop, they played the same smart game. The weaker AIs only did well when the item was expensive and there was a huge "gap" between the buyer and seller prices (a wide zone of agreement). When the gap was small, the weak AIs got confused and made mistakes.

4. The Training Experiment: Teaching a Smaller AI

The researchers then tried to teach a smaller, open-source AI (Qwen) to be a pro negotiator. They used a two-step training method:

Step 1: The "Shadowing" Phase (SFT): They showed the AI thousands of examples of good negotiations.
- Result: The AI learned to be selective. It stopped making bad deals and started holding out for better prices. But, it became too picky and walked away from too many good deals.
Step 2: The "Reinforcement" Phase (RL): They let the AI play against a strong opponent and gave it points for closing deals.
- Result: The AI learned to close the deal. It stopped walking away. But, in its rush to close, it forgot to hold out for a good price and started accepting bad deals again.

The Big Lesson: The two training steps were pulling the AI in opposite directions. One taught it to be a picky shopkeeper, and the other taught it to be a desperate salesperson. The final result was a mix that wasn't much better than the untrained AI.

The Takeaway

This paper shows that for AI to be a true "agent" (a worker that does tasks), it needs more than just good conversation skills. It needs discipline.

Smart Negotiation isn't about being nice; it's about knowing your limits, starting high, and slowly giving ground.
Training AI is tricky: If you reward it only for "getting a deal," it will cheat itself. If you reward it only for "being picky," it will miss out. You need a very careful balance to teach it to be both rational and effective.

In short: The best AI negotiators aren't the ones who talk the most; they are the ones who know exactly when to say "Yes," when to say "No," and how to hold their ground without losing the sale.

1. Problem Statement

The paper addresses the challenge of deploying Large Language Models (LLMs) as autonomous agents in bilateral bargaining under incomplete information. In this economic setting, a buyer and a seller, each holding a private reservation price (the maximum the buyer will pay or the minimum the seller will accept), must negotiate to determine if a mutually beneficial trade exists and at what price.

Two critical limitations prevent the direct use of general-purpose LLMs for this task:

Systematic Behavioral Failures: Pre-trained LLMs often fail to respect hard numeric constraints (e.g., accepting deals that yield negative utility) and lack the strategic reasoning to maximize surplus or walk away from no-gain scenarios. Their pre-training objectives (fluency, agreeableness) conflict with the requirements of strategic negotiation.
The Measurement Problem: In free-form dialogue, it is impossible to automatically distinguish between a binding offer, a hypothetical statement, or a market reference. This prevents automated evaluation of utilities, concession rates, and deal completion at scale.

2. Methodology

The authors propose a structured, event-driven simulation environment and a two-stage training pipeline to address these limitations.

A. The Structured Bargaining Environment

Agent System: LLMs interact via structured tool calls (e.g., make offer, respond to offer, send message, quit negotiation) rather than free-form text. This separates binding actions from natural language communication, enabling machine-parseable negotiation traces.
Game Mechanics:
- Private Information: Agents know their own reservation prices ( $b$ for buyer, $s$ for seller) but not the opponent's.
- Scenarios: The system generates Gains-From-Trade (GFT) scenarios ( $b \ge s$ ) and No-Gains-From-Trade (NGFT) scenarios ( $b < s$ ).
- Protocol: Alternating offers with a maximum of 10 rounds.
Three-Dimensional Evaluation Framework:
1. Individual Rationality (IR): Does the agent avoid deals yielding negative utility? (Measured by violation rates).
2. Strategic Effectiveness: How much surplus does the agent capture? (Measured by surplus share).
3. Allocative Efficiency: Does the pair realize available gains? (Measured by deal rate).
Price-Tier Decomposition: Metrics are disaggregated by item value quintiles to test if models maintain consistent performance across different price ranges.

B. Benchmarking Protocol

Participants: Five frontier models (DeepSeek-V3, Gemini-2.5-Flash/Pro, GPT-4.1, o3).
Design: A round-robin tournament where every model plays as both buyer and seller against every other model (including itself), totaling 15,000 negotiations.

C. Training Pipeline for Open-Weight Models

The authors fine-tune Qwen3 models (8B and 14B parameters) using a two-stage approach:

Supervised Fine-Tuning (SFT): Trained on synthetic trajectories generated by DeepSeek-R1 self-play. This stage focuses on behavioral cloning to learn protocol compliance and reasonable offer strategies.
Reinforcement Learning (RL): Uses Group Relative Policy Optimization (GRPO) against a fixed frontier opponent (GPT-4.1).
- Reward Structure: A composite reward ( $R$ ) combining parsing success, execution success, behavioral consistency, and utility (surplus share).
- Challenge: The reward structure creates a tension where rational walk-aways and irrational acceptances both yield zero utility, biasing the model toward deal-closing.

3. Key Contributions

Structured Environment & Benchmark: The first framework to separate binding offers from natural language in LLM negotiation, enabling automated, large-scale evaluation of IR, strategy, and efficiency.
Systematic Benchmark Findings:
- Aggressive Anchoring + Calibrated Concession: The most effective strategy involves opening with a high anchor (far above cost) and then conceding strategically to close deals. This allows for "price discrimination" through sequential offers.
- The Danger of Accommodation: "Helpful" or accommodating strategies (quick concessions) lead to the lowest surplus capture and deal rates, particularly for buyers.
- Strategic Competence = Consistency: Top-tier models (o3, Gemini-2.5-Pro) maintain consistent surplus capture across all price tiers, whereas weaker models only perform well on high-value items where large "Zones of Possible Agreement" (ZOPA) mask strategic errors.
Diagnosis of SFT-RL Tension: The training experiments reveal a fundamental conflict:
- SFT teaches the model to "hold out for good terms," doubling surplus share but reducing deal rates.
- RL teaches the model to "close the deal," recovering deal rates but eroding the surplus gains learned during SFT.
- This tension stems from a reward structure that fails to distinguish between rational walk-aways and irrational acceptances.

4. Key Results

Benchmark Results (Frontier Models)

Top Performers: o3 and Gemini-2.5-Pro dominate, achieving the highest surplus share and deal rates simultaneously.
Behavioral Drivers:
- o3 anchors at $3.04\times$ its cost (vs. $1.97-2.14\times$ for others) and concedes at the highest rate as a seller, successfully extracting surplus while closing deals.
- Gemini-2.5-Flash (as a buyer) performs poorly, conceding too quickly (0.67 concession rate), resulting in the lowest surplus share (24.3%) and deal rate (86.4%).
Price Tier Consistency: Strong models show minimal variation in surplus share across price quintiles (4–6 percentage point spread), while weak models show large spreads (up to 13 pp), indicating they rely on wide ZOPAs rather than strategic skill.

Training Results (Qwen3 8B/14B)

SFT Impact: Doubles surplus share (e.g., from ~22% to ~40% for 8B buyers) but significantly drops deal rates (e.g., 14B seller deal rate drops from 97% to 49%).
RL Impact: Recovers deal rates to near-base levels but erodes the surplus share gains, returning performance to base levels.
Generalization: SFT successfully instills proportional strategies. The compressed surplus spread across price tiers (from 15–25 pp to 7–9 pp) generalizes to unseen opponents (o3-mini), suggesting the model learned a strategic principle rather than memorizing specific prices.
Fragility: RL training against a single opponent (GPT-4.1) leads to increased IR violations when facing unseen opponents (e.g., 8B seller violation rate jumps to 6.5% against o3-mini).

5. Significance and Future Directions

Theoretical Insight: The paper demonstrates that effective LLM negotiation requires a balance between individual rationality (constraint adherence) and strategic effectiveness (surplus maximization), which current pre-trained models lack.
Training Implications: It highlights a critical design flaw in current RLHF pipelines for negotiation: the lack of explicit penalties for irrational acceptances creates a "deal-closing bias" that undermines strategic selectivity.
Future Work: The authors propose:
- Explicit IR penalties in reward functions.
- Training against diverse opponent populations to prevent overfitting.
- Multi-stage reward curricula that enforce constraint compliance before optimizing surplus.
- Extending the framework to multi-issue and multi-party bargaining.

This work establishes bilateral bargaining as a rigorous testbed for evaluating and training agentic LLMs, moving beyond qualitative assessments to precise, automated metrics of economic rationality and strategic capability.