$\texttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent Execution

1. Problem Statement

As Large Language Model (LLM) agents become more capable, a critical gap remains in their ability to maintain strategic coherence over long horizons. While agents can handle short-term tasks, they often fail when required to:

• Plan under uncertainty over hundreds of turns.
• Learn from delayed feedback (e.g., financial consequences appearing months later).
• Adapt when early mistakes compound into catastrophic failures (bankruptcy).
• Retain crucial facts across context window truncations.

Existing benchmarks (e.g., Vending-Bench) often feature immediate, tangible feedback loops or known adversaries. They fail to simulate the complexity of real-world business management, where risks are calculated, outcomes are delayed, and adversaries are hidden.

2. Methodology: YC-Bench

The authors introduce YC-Bench, a simulation-based benchmark designed to evaluate an agent's ability to run a simulated startup over a one-year horizon (spanning hundreds of turns).

Core Mechanics

• Formalism: The environment is modeled as a Partially Observable Markov Decision Process (POMDP). The agent starts with $200,000 and must maximize final funds.
• The Agent's Role: Acting as a CEO via a CLI interface, the agent must:
  • Accept Tasks: Choose contracts from a marketplace across four domains (Training, Inference, Research, Data Engineering).
  • Assign Employees: Match tasks to a fixed roster of 8 employees with varying, hidden skill profiles (productivity per domain).
  • Manage Finances: Handle recurring monthly payroll that grows with task completion and salary bumps.
• Adversarial Dynamics:
  • Hidden Adversaries: ~35% of clients are adversarial. They offer high rewards but secretly inflate work requirements, making deadlines impossible to meet. Their status is never explicitly revealed; the agent must infer it from repeated failures.
  • Trust System: Success builds trust with specific clients, unlocking higher-tier tasks and reducing future workloads. Trust decays if the agent switches clients too frequently.
• Memory Constraints:
  • The conversation history is truncated to the last 20 turns.
  • Scratchpad: The only mechanism for long-term memory is a persistent scratchpad injected into the system prompt. Agents must autonomously decide what information (e.g., "blacklisted clients," "employee strengths") to record to survive context truncation.

3. Key Contributions

1. Novel Benchmark Design: YC-Bench introduces a deterministic but unknown transition environment that tests long-horizon coherence through compounding financial dynamics, hidden information asymmetry, and adversarial client detection.
2. Comprehensive Evaluation: The study evaluates 12 frontier models (both proprietary and open-source) across 3 random seeds each, including GPT-5.4, Claude Opus 4.6, Gemini 3.1, and open-source models like GLM-5 and Qwen3.5.
3. Failure Mode Analysis: The paper identifies specific, distinct failure modes in frontier models, such as "over-parallelization" (assigning too many concurrent tasks) and the "reasoning-execution gap" (deriving correct strategies but failing to act on them).
4. Open-Source Release: The benchmark is released as a configurable, reproducible open-source tool to facilitate future research.

4. Experimental Results

The evaluation of 12 models revealed significant performance gaps:

• Profitability: Only 3 out of 12 models consistently surpassed the starting capital of $200K.
  • Top Performer: Claude Opus 4.6 achieved the highest average final funds ($1.27M).
  • Runner-up: GLM-5 achieved $1.21M but with 11× lower inference cost, demonstrating superior cost-efficiency.
  • Failure: 7 models ended below their starting capital, and several went bankrupt in at least one seed.
• Key Success Predictor: Scratchpad usage was the strongest predictor of success. Models that failed to record adversarial clients or strategic rules in the scratchpad repeated costly mistakes after context truncation.
• Adversarial Detection: This was the primary failure mode, accounting for 47% of bankruptcies. Top models accepted adversarial tasks at 1/4 the rate of the next-best models by inferring patterns of failure.
• Cost Efficiency: While Opus 4.6 had the highest raw revenue, Kimi-K2.5 and GLM-5 offered the best return on investment (revenue per API dollar).

5. Error Analysis & Failure Modes

The authors analyzed four representative models to understand why they failed:

1. Lack of Reflection (Gemini 3 Flash): Executed a rigid, repetitive loop without updating its strategy or checking task feasibility, accepting 12 adversarial tasks.
2. Reasoning-Execution Gap (Claude Sonnet 4.6): Derived correct strategies (e.g., "one task at a time") in the scratchpad but immediately ignored them, leading to over-commitment and understaffing.
3. Aware Inaction (Grok 4.20): Accurately diagnosed critical issues (e.g., "Runway down to 1 month") in the scratchpad but failed to alter behavior, continuing to accept tasks from known bad clients.
4. Sustained Coherence (Claude Opus 4.6): The only model that maintained a consistent pipeline of Perceive → Record → Retrieve → Act. It actively updated its "blacklist" and optimized trust, though it still occasionally violated its own rules.

6. Significance

• Beyond Short-Term Reasoning: YC-Bench proves that current frontier models still struggle with long-horizon planning, often collapsing into repetitive loops or hallucinated behaviors when the horizon extends beyond immediate feedback.
• The Memory Bottleneck: The study highlights that without explicit, persistent memory mechanisms (like a scratchpad), agents cannot maintain strategic coherence in long-running tasks.
• Real-World Applicability: The benchmark simulates the "compounding consequences" of real business decisions, revealing that models capable of high-level reasoning often fail at the execution and consistency required for autonomous operation.
• Cost vs. Performance: The results suggest that for production agents, the "best" model is not necessarily the one with the highest raw performance, but the one that balances performance with inference cost (e.g., GLM-5 outperforming Opus 4.6 in cost-efficiency).

In conclusion, YC-Bench establishes that long-horizon coherence is a distinct capability gap in current AI agents, requiring not just better reasoning, but robust memory management and the ability to close the loop between diagnosis and action over extended periods.