Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation

Technical Summary: Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation

Problem Statement
Current evaluations of Large Language Models (LLMs) predominantly focus on isolated cognitive tasks—such as reasoning, knowledge retrieval, and economic rationality in stylized settings. These benchmarks fail to address the defining challenge of real-world executive decision-making: the integration of conflicting recommendations from specialized stakeholders under conditions of information asymmetry, organizational constraints, and temporal dependencies. Specifically, there is a lack of systematic testing regarding whether LLM agents can perform CEO-level strategic resource reallocation, a high-leverage decision requiring the synthesis of fragmented, often contradictory information streams into coherent action.

Methodology: CEO-BENCH
To address this gap, the authors introduce CEO-BENCH, a multi-agent benchmark designed to evaluate LLMs on strategic resource reallocation across business units.

• Task Definition: The core task involves an LLM agent acting as a CEO overseeing a multi-business-unit firm. The agent must generate a resource reallocation plan (shifting capital, talent, and attention) across multiple rounds. The action space is continuous, requiring the construction of a plan rather than selection from predefined options.
• Agent Architecture: The benchmark employs a role-conditioned multi-view architecture. The CEO agent receives structured inputs from four C-suite advisors (CFO, CTO, COO, CMO), each with distinct priorities, private signals, and constraints.
  • Information Asymmetry: Advisors share the company scenario but possess different private signals (e.g., CFO sees financing constraints; CTO sees infrastructure bottlenecks), forcing the CEO to reconcile locally rational but globally incompatible recommendations.
  • Multi-Round Dynamics: Scenarios evolve over time, where prior allocations affect future organizational states, enabling the evaluation of history-sensitive strategic consistency.
• Scenario Design: The benchmark includes 13 scenarios divided into four difficulty tiers (Easy, Tension, Fragile, Adversarial) to test performance under varying levels of cross-functional conflict and uncertainty. Unlike traditional benchmarks, CEO-BENCH avoids a single "correct answer," instead evaluating alignment with multiple acceptable strategic profiles (e.g., aggressive growth vs. sequenced rebalancing).
• Evaluation Metrics: Performance is measured across four dimensions:
  1. Role Integration: The ability to synthesize conflicting perspectives rather than collapsing into a single functional viewpoint.
  2. Conditional Boldness: The calibration of strategic aggressiveness to the specific organizational state (avoiding default conservatism or recklessness).
  3. History-Sensitive Judgment: The incorporation of prior organizational trajectories to avoid short-sighted optimization.
  4. Plan Validity: Adherence to hard constraints (mass balance, caps) and strategic fit.

Key Results
Experiments were conducted on five frontier models (Gemma-4-27B, GPT-OSS-20B, Qwen-Plus, Claude-3.5-Haiku, and Nemotron-9B) across the 13 scenarios.

• Structural Validity vs. Strategic Calibration: All models achieved high scores in Plan Validity (generating structurally feasible plans), but performance diverged sharply on Strategic Calibration (boldness and integration).
• Model Performance:
  • Gemma-4-27B and GPT-OSS-20B demonstrated the strongest overall performance, producing the highest number of valid plans and showing robustness in "Fragile" and "Adversarial" settings.
  • GPT-OSS-20B excelled in Role Integration (81/100), effectively articulating trade-offs between advisors.
  • Conditional Boldness was the weakest dimension across all models. Many models defaulted to conservative execution even when the scenario demanded bold action.
• Failure Modes: The study identified systematic failure patterns:
  • Single-Advisor Capture: Models often over-weighted one advisor's perspective (e.g., following the CMO's growth logic or CFO's risk control) while ignoring others.
  • Conservative Default: A tendency toward under-reallocation, particularly in models like Nemotron and Qwen.
  • Historical Amnesia: Weaker models treated each round as a fresh optimization problem, ignoring prior allocation paths and leading to inconsistent strategies (e.g., repeated over/under-funding).
  • Integration–Boldness Trade-off: Models that engaged deeply with conflicting perspectives tended to produce less decisive actions, while those that were decisive often failed to integrate diverse views.

Significance and Claims
The paper delineates the current capability boundary of LLMs as organizational decision-makers. The authors claim that while frontier models can generate structurally valid plans, they struggle with the integrative executive judgment required to reconcile cross-functional conflict and calibrate strategic boldness under dynamic constraints.

The primary contribution is the demonstration that meaningful performance differences in executive decision-making emerge only when role conflict, information asymmetry, and dynamic constraints are jointly imposed. The findings suggest that the bias toward caution and the difficulty in synthesizing conflicting functional logics are significant hurdles for AI-assisted executive systems. The benchmark provides a framework for future research to move beyond isolated reasoning tasks toward evaluating complex, hierarchical, and temporally dependent organizational behaviors.