Technical Summary: CoffeeBench

Problem Statement

As Large Language Model (LLM) agents advance in long-horizon tasks, existing benchmarks primarily evaluate single agents interacting with passive environments or homogeneous multi-agent systems. However, real-world economic systems are inherently heterogeneous, involving distinct roles (e.g., farmers, manufacturers, retailers) that must communicate, negotiate, and transact autonomously over extended periods to pursue individual objectives. There is a lack of benchmarks capable of evaluating LLM agents in such complex, multi-agent economic ecosystems where strategic coordination, inventory management, and financial planning are required over long time horizons.

Methodology

The authors introduce CoffeeBench, a benchmark designed to evaluate LLM agents within a simulated, heterogeneous multi-agent economy.

Environment Design

• Structure: The simulation models a coffee supply chain over a 90-day horizon involving six autonomous firms: two farmers, two roasters, and two retailers.
• Roles & Heterogeneity: Each role has distinct economic functions, tool sets, and constraints. Farmers produce green beans; roasters process them into roasted products; retailers sell to consumers.
• Economic Dynamics:
  • Time Management: The environment uses an event-driven, asynchronous timeline. Agents operate within a daily business window (09:00–19:00). Tool calls advance local time by 30 minutes. Agents can be reactivated by external events (messages, trade offers) even while idle.
  • Marketplace: A shared marketplace allows unrestricted peer-to-peer trading. Transactions involve shipping lags, net-30 payment terms, and potential interest on late payments.
  • Constraints: Agents face fixed daily operating costs, inventory spoilage (0.5%/day), storage caps, and bankruptcy rules (negative cash balance leads to exit).
  • Demand: Retailers face a competitive consumer demand model influenced by pricing, brand loyalty, and stochastic noise, including a simulated demand surge (Days 40–53).
• Objective: Each agent is instructed to maximize cumulative net income (Revenue − COGS − OpEx − Interest + Interest Revenue) over the simulation.

Experimental Protocol

• Setup: In each run, the evaluated LLM controls Roaster A, while the remaining five firms are controlled by fixed reference agents (defaulting to Claude Sonnet 4.6).
• Models Evaluated: The study tests five frontier closed models (GPT-5.5, Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro) and two open-weight models (Kimi K2.6, GLM-5.1).
• Baselines: Two rule-based baselines are used: PassiveRoaster (always waits) and HeuristicRoaster (fixed inventory/pricing policy without communication).
• Context Management: To handle the 90-day horizon, intermediate trajectory history is summarized once the context exceeds 160k tokens, retaining the system prompt, initial trajectory, and recent steps.
• Metrics: Primary metric is Net Income. Secondary metrics include revenue, tool call volume, idle days, and message counts.

Key Contributions

1. First Heterogeneous Multi-Agent Economic Benchmark: CoffeeBench is the first benchmark to evaluate LLM agents in a multi-agent economy with distinct, heterogeneous economic roles (farmers, roasters, retailers) interacting autonomously.
2. Long-Horizon Evaluation: It extends evaluation beyond short-term tasks to a 90-day simulation requiring sustained planning, cash flow management, and strategic adaptation.
3. Identification of Failure Modes: The benchmark reveals specific behavioral pathologies in LLM agents, notably the "idle-drift" failure mode, where an agent maintains coherent reasoning but repeatedly chooses inaction.
4. Open Resources: The authors release code, agent trajectories, and full interaction logs to support future research.

Results

• Performance: All evaluated LLMs outperformed the passive baseline. Most achieved positive net income, with GPT-5.5 (+$3,109) and Claude Opus 4.7 (+$2,782) achieving the highest mean net incomes.
• Behavioral Drivers:
  • Communication: Higher-performing models (GPT-5.5, Claude Opus 4.7) communicated significantly more actively with counterparties (farmers and retailers) compared to lower-performing models.
  • Tool Usage: Profitability was not strictly correlated with the volume of tool calls. Kimi K2.6 had high tool usage but lower profitability, suggesting that effective coordination and pricing strategy matter more than execution volume.
  • Idle-Drift: Claude Haiku 4.5 exhibited a distinct failure mode, averaging 40 idle days out of 90. Despite generating coherent reasoning traces, it repeatedly selected wait_for_next_day(), leading to negative net income (-$630).
• Pricing and Inventory: Inventory hygiene alone did not guarantee profitability. Gemini 3.1 Pro maintained lean inventory and low spoilage but achieved mid-range income, whereas GPT-5.5 achieved higher income through better realized B2B pricing and active negotiation.
• Strategic Ceiling: Analytical estimates suggest current models achieve only ~13% of the theoretical net income ceiling under optimistic assumptions, indicating significant room for improvement in long-horizon strategic planning.

Significance and Claims

The paper claims that CoffeeBench provides a necessary framework for evaluating LLM agents in realistic, open-ended economic settings that go beyond simple tool use or single-agent planning.

• Strategic Coherence: The results suggest that current frontier models lack the long-horizon strategic coherence required for complex market behaviors, such as sustained coordination or manipulative trading, even under strong incentives (e.g., revenue-maximizing stress tests).
• Behavioral Analysis: The benchmark successfully identifies specific failure modes like "idle-drift," offering a new lens for understanding how LLM agents behave over extended periods when faced with asynchronous, multi-agent dynamics.
• Future Research: The authors posit that CoffeeBench can serve as a testbed for studying emergent strategic behaviors, including undesirable outcomes like collusion, and for developing agents capable of reliable decision-making in multi-agent economies.

The paper concludes that while current models show promise, substantial progress is needed in long-horizon planning, proactive communication, and strategic adaptation to fully realize the potential of LLM agents in economic systems.