Technical Summary: AgentOdyssey

Problem Statement

Current benchmarks and training paradigms for Large Language Model (LLM) agents are inadequate for studying test-time continual learning. Traditional evaluations implicitly assume that learning ceases after training, serving only as static inference-time assessments. Even when learning is considered, it typically occurs over many pre-evaluation episodes rather than as a continuous process interleaved with inference during deployment.

Real-world agents face a central challenge: after initial training, they must continuously acquire new knowledge and skills, retain episodic experiences, and plan over long horizons while interacting with a dynamic world. Existing text-game environments lack the necessary scalability, stochastic dynamics, and long-horizon complexity to evaluate these capabilities. Furthermore, they often conflate agent reasoning with low-level control or visual grounding, obscuring the specific cognitive abilities required for continual learning.

Methodology

The AgentOdyssey Framework

The authors introduce AGENTODYSSEY, a novel evaluation framework that procedurally generates open-ended text games. The framework is designed to test five key abilities hypothesized to be essential for test-time continual learning:

1. Exploration: Finding novel entities and actions.
2. Episodic Memory: Recalling past experiences (e.g., dropped items, visited locations).
3. World Knowledge Acquisition: Learning new rules, dynamics, and crafting recipes.
4. Skill Learning: Improving performance through procedural knowledge (e.g., note-taking).
5. Long-Horizon Planning: Decomposing objectives and managing goals over extended sequences.

Game Generation Engine:
The framework utilizes an LLM-based program synthesis engine grounded in a formal ontology. It consists of three generators:

• Entity Generator: Creates locations, objects, and NPCs with specific attributes.
• Dynamics Generator: Defines action rules (player-initiated) and step rules (environment-driven, e.g., day-night cycles, NPC patrols).
• Quest Generator: Creates main and side quests with linear or independent dependencies.

The engine includes an automated testing pipeline that runs generated games with random agents to verify soundness, detect runtime errors, and ensure the game is solvable. The environment is modeled as a Partially Observable Markov Decision Process (POMDP) where time advances in fixed increments, enabling time-dependent rules.

Evaluation Methodology

The authors propose a multifaceted evaluation approach beyond simple game completion:

• Game Progress: Measured via normalized cumulative rewards for main quests and supplementary metrics (side quests, exploration, crafting, defeat).
• Diagnostic Tests:
  • World Knowledge QA: Assesses understanding of game rules and dynamics.
  • Episodic Memory QA: Tests recall of specific past events and trajectory details.
  • Object/Action Exploration: Measures the proportion of unique objects and actions encountered.
  • Action Diversity: Quantifies behavioral variety using entropy.
• Model Cost: Tracks total token usage to evaluate efficiency.

Agent Paradigms Evaluated

The study evaluates diverse agent architectures across different LLM backbones (including GPT-5, Qwen-3, and LLaMA-3):

• Long Context (LC): Appends full history to the context window.
• Fixed Size Memory: Uses sliding windows or self-updating buffers.
• RAG (Retrieval-Augmented Generation): Retrieves relevant past tuples from an external database.
• SFT (Supervised Fine-Tuning): Updates model weights via LoRA based on experience.
• RL (Reinforcement Learning): Updates weights via PPO.
• Latent Memory: Compresses experience into learnable latent tokens.

Key Results

Experiment 1: Diagnosing Key Abilities

Using a complex game environment (18 areas, 83 objects, 13 NPCs, 24 quest stages), the authors found:

• Performance Scaling: The Long Context agent (specifically with GPT-5) achieved the highest performance, demonstrating that agent capability scales with both memory capacity (context size) and the backbone model's reasoning abilities. However, even the top agent remained far below human performance.
• Correlation: Diagnostic test scores (World Knowledge and Episodic Memory) strongly correlated with game progress.
• Limitations:
  • Exploration: Agents exhibited myopic strategies, failing to collect intermediate resources not immediately linked to the current quest.
  • Episodic Memory: Agents frequently fell into repetitive behavioral loops (local minima) and failed to recover from errors, even when corrective feedback was present in their context.
  • Skill Learning: Agents struggled to acquire procedural skills, such as externalizing recipes via note-taking, and failed to adapt to deterministic enemy patterns.
  • Long-Horizon Planning: Agents often lost track of primary objectives after completing subgoals.
  • Cost: Long Context agents incurred quadratic token costs, severely limiting their meaningful horizon under budget constraints.

Experiment 2: Effect of Mechanisms on Test-Time Training

In a simplified environment to test training dynamics:

• Short-Term Memory (STM): The addition of a short-term memory buffer (a fixed-size context of recent steps) consistently improved performance across RAG and SFT agents.
• SFT Performance: An SFT agent augmented with STM outperformed a vanilla STM agent, suggesting that parametric updates serve as an effective form of long-term memory.
• Catastrophic Forgetting: SFT agents showed a decline in World Knowledge QA accuracy and Episodic Memory QA after training, indicating a loss of general language capabilities due to fine-tuning.
• Reflection/Summarization: These mechanisms did not improve performance for reasoning-heavy models, suggesting they may already implicitly perform these functions.

Significance and Claims

The paper claims that AGENTODYSSEY is the first framework to systematically evaluate test-time continual learning agents in non-resettable, long-horizon environments. Its significance lies in:

1. Bridging the Gap: Moving beyond static inference evaluation to assess agents that must learn continuously during deployment.
2. Diagnostic Depth: Providing tools to isolate specific cognitive failures (e.g., memory hallucination vs. planning failure) rather than relying solely on task completion.
3. Identifying Bottlenecks: Revealing that while stronger base models improve performance, current agents lack robust mechanisms for episodic memory retention, procedural skill acquisition, and efficient long-horizon planning.
4. Memory Importance: Highlighting that short-term memory is a critical, often overlooked component for effective test-time training, even for agents with parametric updates.

The authors conclude that substantial headroom remains for improvement, particularly in developing architectures that support persistent memory and scalable test-time learning without incurring prohibitive computational costs.