Technical Summary: OpenPM – Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents

1. Problem Statement

The paper addresses three critical failures in the current evaluation of Large Language Model (LLM) trading agents, which often lead to inflated or non-deployable results:

1. Information Leakage: Agents frequently access data unavailable at the decision time (e.g., future observations, event-stamped records rather than availability-stamped records), creating artificial skill.
2. Optimistic Execution: Reported returns often ignore trading costs (spread, slippage, turnover), presenting upper bounds rather than realistic deployable estimates.
3. Unenforced Mandates: Natural-language risk constraints (e.g., "conservative," "max 20 names") are often treated as soft preferences for post-hoc scoring rather than hard constraints enforced on the executed portfolio.

Existing benchmarks often fail to control for these issues simultaneously, particularly lacking point-in-time (PIT) data gating and strict mandate enforcement.

2. Methodology: The OpenPM Framework

OpenPM is an auditable, point-in-time evaluation framework designed to treat credible evaluation as an engineering artifact.

Core Design Principles

• Point-in-Time Data Environment: The system enforces a strict "availability gate." A record enters the agent's state at decision time $T$ only if its ts_available $\le T$. This distinguishes between event time and availability time (e.g., a quarterly filing is only available after EDGAR acceptance). The environment covers an S&P 500 universe with 5-minute market-state observations.
• Mandate Enforcement: Natural-language risk mandates are parsed into typed constraints (e.g., max weight per name, max number of names). Crucially, these are enforced by a deterministic in-loop critic that projects the agent's proposed weights onto the feasible set before execution, rather than merely scoring the agent post-hoc.
• Cost-Aware Execution: Returns are calculated using side-aware half-spreads (buys at ask, sells at bid) with no market impact modeling, providing a conservative but realistic cost baseline.

The Tiered Allocator (Reference Agent)

The paper introduces a reference agent architecture to isolate specific components of the trading pipeline:

1. Mandate Compilation: Converts natural language to typed constraints.
2. Liquidity Gate: Filters the universe to tradable names (e.g., top 50 by liquidity).
3. Analyst Tier: Six parallel LLM analysts independently score candidates across typed channels (technical, regime, events, news, 8-K items, 10-K/10-Q narratives).
4. Constructor: A separate LLM proposes portfolio weights based on the aggregated analyst scores and spreads, but without access to raw prices or the composite score.
5. Deterministic Critic: Projects the constructor's proposal onto the feasible set (enforcing long-only, name caps, liquidity limits).
6. Fill Simulator: Executes trades at quoted sides.

Experimental Setup

• Dataset: A frozen, content-hashed snapshot of the S&P 500 (508 names) active between Feb 19, 2026, and May 1, 2026. The evaluation window is March 2, 2026, to May 1, 2026 (44 trading days).
• Isolation Strategy: To isolate the "constructor" capability, the authors run the analyst tier once to create a "frozen capture" of evidence. This same evidence is then replayed across different constructor models (gpt-5, Opus-4.7, DeepSeek-V3.2, Qwen3-Max, gpt-4o-mini) to determine if the constructor adds value independent of the signal quality.
• Modes: Experiments run in "once-then-hold" (single rebalance) and daily rebalancing modes.

3. Key Results

The paper reports structural findings rather than absolute alpha claims, emphasizing that results are upper bounds on a single frozen window.

Constructor Capability

• Conditional Gains: Stronger constructors (e.g., gpt-5, Opus-4.7) show modest, model-dependent gains over equal-weighting (EW) the same candidate pool, but only when the upstream analyst signal is strong.
• Analyst Dominance: The quality of the analyst tier is the primary driver of performance. When the analyst capture was weak (DeepSeek-V3.2), no constructor could beat the same-pool EW baseline. When the capture was strong (gpt-5), most constructors beat EW.
• Overlap: Strong constructors largely re-weight the EW roster (75–78% overlap) rather than replacing it entirely. Weaker constructors often deviate significantly and underperform.

Cost and Turnover

• Turnover is the Cost Driver: In daily rebalancing, turnover becomes the dominant cost factor. High-turnover models (e.g., Qwen3-Max, gpt-4o-mini) saw their net returns drop below the SPY benchmark due to cost drag, despite having similar gross returns to lower-turnover models.
• Discipline Matters: The winning pattern involves selecting well and trading sparingly. Low turnover alone is insufficient (gpt-5 had the lowest turnover but underperformed SPY due to poor selection in that specific regime).

Compliance and Auditing

• Mandate Adherence: All constructors successfully adhered to explicit numeric caps (e.g., 10% max weight) in their raw proposals.
• Critic Necessity: The deterministic critic was essential for enforcing liquidity constraints (hard clips) that were tighter than the mandate and invisible to the model.
• Contamination: All runs passed the contamination certificate, confirming no look-ahead leakage.

4. Significance and Claims

The paper positions OpenPM not as a validated alpha generator, but as a diagnostic tool for the LLM trading community.

• Engineering Artifact: It reframes evaluation as an engineering problem requiring strict data contracts, availability gating, and deterministic enforcement layers.
• Honesty in Claims: The authors explicitly state that all returns are "upper bounds on a single frozen window without market impact, not validated alpha." The goal is to keep claims honest by tying every reported number to the specific conditions (data fingerprint, cost model, mandate) that produced it.
• Diagnostic Insights: The framework reveals that "construction skill" is often a proxy for "analyst signal quality." The paper argues that compute should be prioritized for the analyst tier, using the cheapest capable constructor thereafter.
• Auditability: By generating artifacts like contamination certificates, cost-sensitivity curves, and constraint-adherence reports for every run, OpenPM allows researchers to verify that results are not artifacts of data leakage or optimistic execution assumptions.

In summary, OpenPM demonstrates that while LLMs can construct portfolios that outperform simple baselines under specific conditions, their performance is heavily contingent on upstream signal quality and is easily eroded by turnover costs. The framework provides the necessary rigor to distinguish genuine skill from evaluation artifacts.