Technical Summary: STOCKTAKE – Measuring the Gap Between Perception and Action in LLM Agents

1. Problem Statement

Large Language Model (LLM) agents are increasingly evaluated on long-horizon decision tasks, such as supply chain management, where performance depends on inferring hidden states (e.g., demand shifts, supplier failures) from noisy observations over time. A critical limitation in current evaluations is the inability to distinguish between two distinct failure modes:

1. Perception Failure: The agent fails to correctly infer the hidden state (state estimation error).
2. Action Failure: The agent correctly infers the state but fails to act appropriately (control error), a phenomenon known as the "knowing-doing gap."

Existing benchmarks often rely on reference policies (oracles) that possess privileged information (ground truth) unavailable to the agent. This conflation makes it impossible to attribute cost to a specific failure mode; a shortfall could be due to the agent not knowing the truth or knowing it but failing to act. Furthermore, many benchmarks lack mechanisms to grade the agent's stated beliefs alongside its actions.

2. Methodology: The STOCKTAKE Benchmark

The authors introduce STOCKTAKE, a 26-week supply-chain replenishment benchmark designed to isolate and measure these failure modes separately.

2.1 Environment Design

• Task: The agent acts as a replenishment manager for an electronics importer, making weekly ordering decisions based on a dashboard of noisy symptoms.
• Hidden State (Factored POMDP): The environment is a Partially Observable Markov Decision Process (POMDP) with six independent hidden factor processes: demand regime, supplier health, freight market, port congestion, batch quality, and canal status.
• Observations: The agent receives noisy weekly symptoms (e.g., berth wait times, defect reports) but never sees the hidden state directly.
• Action Independence: Crucially, the hidden state dynamics are action-independent. The "event tape" (the sequence of hidden state changes and noise) is generated from a fixed seed before the agent acts. This ensures that any policy running on a seed faces the exact same shocks, allowing for fair comparison.
• Stress Profiles: The benchmark uses 50 curated seeds categorized into three stress profiles:
  • ISOLATED: Short, non-overlapping single-factor shocks.
  • PERSISTENT: Multi-week overlaps (e.g., port congestion coinciding with demand rise).
  • COMPOUND: Pileups of three or more concurrent factor episodes.

2.2 The Fair Oracle

To separate perception from action, STOCKTAKE employs a Fair Bayes Oracle:

• Conditioning: Unlike privileged oracles, this reference policy is conditioned on the identical observation stream the LLM agent receives. It never sees the hidden state.
• Mechanism: It utilizes an exact Bayes filter for each of the six factors to maintain a posterior belief over the hidden state. It then drives a rollout policy by sampling futures and selecting the cost-minimizing action.
• Asymmetry: The only disclosed asymmetry is that the oracle uses the true generative parameters (transition probabilities, regime means), whereas the LLM receives only a qualitative description. Thus, the oracle represents the Bayes-optimal policy given the true model, not an omniscient one.

2.3 Metrics

The benchmark introduces a suite of metrics to decouple state estimation from control:

1. Skill Score: A normalized score ranging from 0 to 1 (and potentially beyond).
   • 0 (Floor): A symptom-blind base-stock policy that ignores all symptoms.
   • 1 (Oracle): The mean cost of the Fair Bayes Oracle.
   • Interpretation: A score > 0 indicates the agent improved upon ignoring symptoms; a score < 0 indicates the agent performed worse than a blind rule.
2. Detection Lag: The number of weeks between the onset of a hidden stress regime and the first time the agent names that factor in its written rationale.
3. Knowing-Doing Rate: The probability of a stockout occurring in a week where the agent has correctly diagnosed a stress factor. This measures the failure to act on correct beliefs.

2.4 Experimental Setup

• Models: Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5.
• Protocol: Each model runs on all 50 seeds (200 total episodes) under a "rules-only" prompt with no strategy playbooks or examples.
• Belief Grading: An auxiliary LLM grades the agent's weekly written rationales to determine if specific factors were correctly identified.

3. Key Results

The evaluation of 200 episodes across four frontier models yields several counter-intuitive findings:

3.1 Uniform Perception, Divergent Action

• State Estimation: All four models demonstrated nearly identical state estimation capabilities. They detected 84–88% of hidden stress episodes, typically within 0.32–0.42 weeks of onset.
• Control Failure: Despite uniform detection, skill scores varied drastically, ranging from 0.62 (GPT-5.4) to −0.23 (DeepSeek-V4-Pro).
• The Floor Paradox: Two models (DeepSeek-V4-Pro and Grok 4.5) finished with negative skill scores. On roughly half their seeds, a symptom-blind base-stock rule would have been cheaper than acting on their largely correct diagnoses.

3.2 The Two Faces of the Gap

The paper identifies that the "knowing-doing gap" manifests in two opposing directions:

1. Under-Response (Stockouts): Even with correct diagnosis, models fail to prevent stockouts. On PERSISTENT seeds, 34–43% of correctly diagnosed stress weeks still ended in stockout.
2. Over-Response (Excessive Cost): Models may act on correct diagnoses but with excessive cost. The models with negative skill scores (DeepSeek, Grok) had the lowest stockout rates on diagnosed weeks but the highest total costs. They "over-dosed" on protection (e.g., locking freight rates, air-expediting) in ways that exceeded the value of the symptoms.

3.3 Stress Profile Sensitivity

• GPT-5.4 performed best on COMPOUND stress (Skill 0.86) but struggled with ISOLATED stress.
• Claude Sonnet 5 performed best on PERSISTENT stress (Skill 0.70) but weakest on COMPOUND.
• The models that performed worst overall (DeepSeek, Grok) showed the shortest detection lags, confirming that the failure was not in seeing the problem, but in the policy derived from that view.

4. Contributions

1. Fair Oracle Benchmark: Introduction of a six-factor supply-chain POMDP where the upper reference is a computable, fair Bayes-filter oracle conditioned on the agent's exact observation stream. This allows the attribution of cost specifically to the "acting" phase rather than the "knowing" phase.
2. Controlled Stress Axis: A methodology for generating seeds with fixed hidden stress profiles (ISOLATED, PERSISTENT, COMPOUND) as independent variables, holding the world model constant.
3. Decoupled Metrics: A framework that separately measures state estimation (detection lag) and control (skill score, knowing-doing rate), revealing that current frontier agents fail primarily in control, not perception.

5. Significance and Claims

The paper claims that STOCKTAKE provides the first rigorous evidence that the performance degradation of LLM agents on long-horizon tasks is concentrated in control (policy elicitation) rather than state estimation (inference).

• The Bottleneck: The primary bottleneck for current agents lies between a correct stated belief and the costly action it calls for.
• Limitations of Current Evaluations: The authors argue that benchmarks using privileged oracles conflate "not knowing" with "knowing but not acting," obscuring the specific nature of agent failures.
• Modest Scope: The authors acknowledge the benchmark covers a single domain (import replenishment), one SKU, and four models. They explicitly state that the results do not generalize to all agent tasks but highlight a specific failure mode in sequential decision-making under partial observability.
• Future Directions: The paper suggests that fixing these agents requires different interventions: better inference scaffolding is not the solution (since perception is already strong); instead, the focus must shift to better policy elicitation and cost-sensitive action selection.

The paper concludes that while agents are increasingly capable of "seeing" the hidden world, they remain inconsistent in "doing" what that vision requires, often oscillating between under-reacting (leading to stockouts) and over-reacting (leading to prohibitive costs).