Technical Summary: Closing the Feedback Loop in Verbal Reinforcement Learning

1. Problem Statement: The Retention-Forgetting Dilemma

The paper addresses a fundamental challenge in Training-Free Verbal Reinforcement Learning (RL) for Large Language Model (LLM) agents. In this paradigm, agents improve by extracting verbal rules from world feedback (e.g., market returns, task outcomes) and injecting them into their context, rather than updating model parameters.

The core problem identified is the retention-forgetting dilemma in non-stationary environments:

• Retention Failure: If an agent retains all accumulated experience, its context becomes cluttered with stale, contradictory, or regime-specific rules. This leads to "negative transfer," where the agent confidently applies outdated logic, degrading performance below zero-shot baselines.
• Forgetting Failure: If an agent discards rules that fail in the current context, it suffers "catastrophic forgetting" when conditions revert to a previous regime where those rules were valid, forcing the agent to re-learn from scratch.

Existing methods largely focus on experience extraction (how to generate rules) but underinvest in insight governance (how to manage, evaluate, and curate those rules over time).

2. Methodology: A Three-Layer Architecture with Feedback-Driven Curation

To navigate this dilemma, the authors propose a three-layer architecture connected by a feedback-driven curation loop. This system is designed to satisfy four specific requirements (R1–R4) that existing methods fail to meet simultaneously.

The Three Layers

1. Layer 1: Rules (Distilled Experience)
   • Captures patterns extracted from world feedback as trigger conditions and corrective actions.
   • Lifecycle: Rules are deprecated (stopped from firing) rather than deleted. This preserves the knowledge for potential future relevance while preventing immediate negative transfer.
2. Layer 2: Evidence (Persistent Structured Logs)
   • Each rule maintains an append-only log tracking every episode where it was evaluated, the outcome, and the specific conditions.
   • Unlike scalar confidence scores (which discard history), these logs preserve the full trail of why a rule succeeded or failed. This allows the system to distinguish between a rule that failed once (noise) and one that failed consistently (signal).
3. Layer 3: Skills (Compositional Governance)
   • A higher-order mechanism that governs which rules to apply, how to resolve conflicts between competing rules, and when to abstain.
   • Skills evolve based on accumulated evidence, encoding priority orderings and anti-patterns derived from the evidence logs.

The Curation Loop

The architecture operates via a Critic–Proposer–Curator pipeline processing batches of experience:

• Critic: Evaluates rule-augmented reasoning against a zero-shot baseline using observed world outcomes. It attributes performance changes to specific rules based on their impact on the chain of thought, not just final success/failure.
• Proposer: Appends critic evaluations to the per-rule evidence logs (never modifying rules in-place) and proposes new rules for uncovered error patterns.
• Curator: Reads cross-rule evidence to make lifecycle decisions. It deprecates rules with consistently negative evidence and evolves skills to manage the active rule set.

3. Key Contributions

The paper identifies four requirements necessary for effective learning from non-stationary world feedback and demonstrates that existing methods fail to satisfy all four:

1. Outcome-driven Evaluation (R1): Systematic evaluation of whether stored knowledge helped, based on observed outcomes.
2. Persistent Structured Evidence (R2): Accumulation of evidence across episodes linked to specific knowledge, surviving deprecation.
3. Non-monotonic Knowledge Lifecycle (R3): The ability to add and deactivate knowledge without deletion, preserving evidence to prevent catastrophic forgetting.
4. Compositional Governance (R4): A mechanism to resolve rule conflicts and determine application order based on evidence.

The authors contribute:

• The framing of the retention-forgetting dilemma as the central challenge for verbal RL.
• A three-layer architecture (Rules, Evidence, Skills) that closes the gap between extraction and governance.
• Empirical evidence showing that governance quality, not the quantity of experience, determines whether an agent improves or degrades.

4. Empirical Results

The authors validate their approach using financial forecasting (S&P 500 equities, 2013–2017), a domain characterized by abundant, noisy, delayed, and non-stationary feedback.

Findings:

• No Governance (Reflective Accumulation): Accumulated experience actively harms performance. Directional accuracy drops by 4.9 percentage points compared to zero-shot, and the Sharpe ratio becomes negative (-0.12 vs. 0.53).
• Partial Governance (Reflective Refinement): In-place modification and scalar scoring recover accuracy to near zero-shot levels but fail to improve risk-adjusted returns (Sharpe 0.36). Conflicting rules still accumulate.
• Full Governance (Proposed Method): The agent achieves genuine improvement:
  • Directional Accuracy: +5.3 percentage points over zero-shot (56.5% vs. 51.2%).
  • Risk-Adjusted Returns: Sharpe ratio nearly doubles (1.00 vs. 0.53).
  • Risk Management: Maximum drawdown is reduced by 60% (13.0% vs. 34.5%).

The results demonstrate that the same accumulated experience can either degrade or dramatically improve performance solely depending on the presence of the curation loop.

5. Significance and Claims

The paper posits that the primary bottleneck for agents learning from world feedback is not experience extraction (generating rules) but insight governance (managing them).

• Design Principle: The authors argue that any agent learning from non-stationary world feedback must treat the management of its own knowledge as a first-class design problem, satisfying the four identified requirements.
• Modest Scope: The authors explicitly state that their empirical validation is limited to a single domain (S&P 500 equities) and a predominantly bullish regime. They caution that the results should not be interpreted as evidence of live trading viability.
• Future Work: The paper suggests that the curation mechanism itself (critic criteria, deprecation thresholds) could eventually be learned from world feedback, creating a second-order loop for "learning how to learn," but notes this is currently hand-designed.

The core takeaway is that without a structured, evidence-based governance layer, accumulated experience in non-stationary environments is likely to be detrimental rather than beneficial.