Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce

Technical Summary: "Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce"

Problem Statement
As Large Language Model (LLM) agents transition from conceptual prototypes to deployed infrastructure in payment networks and retail platforms, a critical evaluation gap remains: can these agents price competently in real-world market conditions? Existing benchmarks typically evaluate product knowledge or negotiation on fixed instances, or study auctions under stylized, static rules. They fail to capture the core challenges of deployed agentic commerce: learning hidden customer preferences, managing profit margins in the presence of adaptive competitors, and revising beliefs when demand shifts unexpectedly. The paper posits that without systematic testing in dynamic, non-stationary environments, it is unclear whether LLMs can learn what customers value, choose profitable offers, and adapt to market shocks.

Methodology: The Bazaar Framework
The authors introduce Bazaar, a diagnostic framework for evaluating LLM merchants in dynamic multi-attribute sealed-bid auctions.

• Environment: The framework simulates a market with $M$ merchants and $I$ customers over $T$ global rounds. In each round, every customer runs a sealed-bid auction.
• Multi-Attribute Auctions: Merchants submit bids consisting of a product configuration (a vector of $K$ attributes with $L$ levels) and a price. Customers have hidden, additive valuation functions over these configurations. The customer selects the offer maximizing their utility ($V(x) - p$).
• Information Constraints: The focal LLM merchant observes only sparse feedback: the identity of the winner, the winning configuration, and the winning price. It does not observe customer valuations, competitor costs, or competitor bids. Losing merchants receive no information about the utility gap.
• Dynamic Shocks: To test adaptation, the framework introduces unannounced preference shifts midway through the game. For a subset of customers, the valuation curves for two attributes are swapped (e.g., $A \leftrightarrow B$), altering the optimal configuration without changing the underlying demand structure. The agent must detect this shift solely from the change in auction outcomes.
• Opponents: The focal LLM competes against three adaptive, rule-based "specialist" bots. Each bot has a structural cost advantage on one specific attribute and adapts its margin based on win/loss history (raising margins after wins, lowering after losses).
• Evaluation Metrics: The authors move beyond simple win rates to a diagnostic suite:
  • Total Profit: Cumulative earnings.
  • Margin Discipline: Profit per win.
  • Oracle Regret Decomposition: The gap between realized profit and a hindsight-optimal oracle is split into Loss Regret (rounds lost that the oracle would have won) and Underprice Regret (rounds won but priced below the optimal margin).
  • Shock Recovery: The ability to regain pre-shock performance levels after a preference shift.

Key Contributions

1. A Diagnostic Framework for Dynamic Pricing: Bazaar provides a closed-form utility environment that allows for exact evaluation despite dynamic competition. It separates performance into distinct failure archetypes: "Losers" (who fail to win profitable auctions) and "Underpricers" (who win but leave money on the table).
2. Empirical Evaluation of Frontier LLMs: The study evaluates 11 frontier LLMs from four providers (Anthropic, Google, OpenAI, xAI) across various thinking-effort configurations.
3. Discovery of Failure Modes and Trade-offs: The benchmark reveals that high win rates do not guarantee high profits. It identifies a "strong-learner/weak-adapter" phenomenon where models that learn quickly pre-shock often struggle to revise beliefs post-shock.
4. Regret-Based Analysis: By decomposing regret, the paper shows that thinking effort (inference compute) shifts models along a failure surface, often converting "Losers" into "Underpricers" by fixing configuration search but failing to optimize margin extraction.

Results

• Profit vs. Win Rate: Across 11 models, total profit correlates strongly with margin per win ($r=0.99$) but less so with win rate ($r=0.88$). For instance, Gemini 3.1 Pro achieved the highest win rate (77.9%) but earned less total profit than Opus 4.6 (67.8% win rate) because Opus 4.6 extracted significantly higher margins ($2.22 vs. $1.83 per win).
• The Impact of Thinking Effort: Increasing inference compute (thinking budget) dramatically alters performance. GPT-5.4 at default (no extended thinking) earned only $266 (Tier 4), but at high effort, it earned $1,936 (Tier 2), a 7.3x increase. However, this shift often changed the failure mode: high-effort GPT-5.4 reduced loss regret (winning more) but increased underprice regret (charging too little).
• Shock Adaptation: A "strong-learner/weak-adapter" pattern emerged. Models like GPT-5.3 and Opus 4.5 climbed rapidly in win rate before the shock but suffered significant degradation afterward (losing ~19 percentage points). Conversely, Gemini 3.1 Pro, which plateaued earlier pre-shock, recovered fastest after the shock, suggesting its concise, hypothesis-oriented reasoning allowed for better belief revision.
• Cost Efficiency: Gemini 3.1 Pro was the most cost-effective top performer, achieving high profit at 4.8x lower API cost than the top profit earner, Opus 4.6.
• Bandit Baselines: Classical bandit algorithms (Thompson Sampling, EXP4) set a non-trivial floor, outperforming several base LLMs in profit, though top LLMs surpassed them in margin extraction and adaptation speed.
• Version Recency: Newer model versions did not consistently outperform older ones. For example, Opus 4.7 underperformed Opus 4.6 and Opus 4.5 at provider defaults, and GPT-5.3 outperformed GPT-5.4 and GPT-5.5.

Significance and Claims
The paper claims that while current LLMs are progressing in agentic commerce, they leave "substantial headroom" for improvement. The strongest agent captured less than a third of the hindsight-optimal profit. The authors emphasize that static benchmarks are insufficient; the dynamic nature of Bazaar reveals failure modes—such as the inability to revise beliefs after market shifts or the tendency to underprice even when winning—that would remain hidden in static evaluations.

The work connects the fields of LLM agents in auctions, multi-agent benchmarks, e-commerce, and algorithmic pricing. It suggests that "margin discipline" is the discriminating skill for profitable agents and that the "thinking budget" is a primary lever for performance, though it must be balanced against the risk of shifting from a "Loser" to an "Underpricer." The authors conclude that while LLMs are becoming capable economic agents, their pricing strategies in competitive, non-stationary markets remain an open challenge requiring further refinement in belief revision and strategic reasoning.