💻 computer science

Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks

This paper introduces COSPLAY, a co-evolutionary framework where an LLM decision agent and a skill management agent collaboratively discover, refine, and retrieve structured skills from unlabeled rollouts, significantly improving long-horizon performance in game environments compared to frontier LLM baselines.

Original authors: Xiyang Wu, Zongxia Li, Guangyao Shi, Alexander Duffy, Tyler Marques, Matthew Lyle Olson, Tianyi Zhou, Dinesh Manocha

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiyang Wu, Zongxia Li, Guangyao Shi, Alexander Duffy, Tyler Marques, Matthew Lyle Olson, Tianyi Zhou, Dinesh Manocha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly forgetful, robot how to play a complex video game like Diplomacy or Super Mario.

The robot has a massive brain (a Large Language Model, or LLM) that knows a lot about the world. However, when faced with a long, complicated game, it often gets lost. It forgets what it did five minutes ago, it doesn't know when to switch strategies, and it keeps making the same mistake over and over. It's like a chef who knows how to chop vegetables but keeps trying to bake a cake with a hammer because they forgot the recipe.

This paper introduces a new system called COS-PLAY to fix this. Think of COS-PLAY not as a single robot, but as a team of two coworkers working together in a kitchen.

The Two Coworkers

The Chef (The Decision Agent): This is the robot playing the game. Its job is to look at the screen, decide what to do next, and press the buttons.
The Sous-Chef (The Skill Bank Agent): This is a second robot whose only job is to watch the Chef, take notes, and organize a "Recipe Book."

How They Work Together (The Co-Evolution)

In the old days, you would just tell the Chef, "Here is the recipe, now cook!" If the Chef failed, you'd have to rewrite the whole recipe from scratch.

With COS-PLAY, the process is a continuous loop of learning:

Step 1: The Chef Tries to Cook
The Chef plays a round of the game. Sometimes it does great; sometimes it burns the toast. It doesn't know why it failed, it just knows the score.

Step 2: The Sous-Chef Watches and Writes
While the Chef plays, the Sous-Chef is watching closely. It sees a pattern: "Oh, every time the Chef moves the army to the border, it usually takes 3 turns to scout the enemy, then 2 turns to set up an attack."
The Sous-Chef takes this messy sequence of actions and turns it into a clean, reusable "Skill Card" (like a recipe card).

The Skill Card says: "When you see the enemy at the border, do these 5 steps in order to scout them. Stop when you see a red flag."

Step 3: The Chef Gets a New Tool
The Sous-Chef puts this new Skill Card into the Recipe Book (the Skill Bank).
Now, when the Chef starts the next game, it doesn't have to figure everything out from scratch. It can open the Recipe Book, find the "Scout the Border" card, and follow the instructions.

Step 4: The Loop Continues

If the Chef uses the card and wins, the Sous-Chef says, "Great! That card works."
If the Chef uses the card and loses, the Sous-Chef says, "Hmm, that card didn't work here. Let's tweak the recipe."
The Sous-Chef might also notice the Chef is doing something brilliant on its own and add a new card to the book.

Over time, the Chef gets smarter because it has a better library of tricks, and the Recipe Book gets better because it's constantly being updated with what actually works. They "co-evolve"—they grow together.

Why This is a Big Deal

The Problem with "Just a Brain":
Big AI models are like geniuses with short attention spans. They can write a poem, but if you ask them to play a 50-step strategy game, they get confused halfway through. They forget their plan.

The Solution:
COS-PLAY gives the AI a memory of "chunks". Instead of remembering every single button press, it remembers "chunks" of behavior (Skills).

Analogy: Imagine trying to write a novel. If you try to remember every single letter you type, you'll go crazy. But if you remember "Write a chapter about the hero's journey," then "Write a chapter about the villain's plan," it's much easier. COS-PLAY teaches the AI to think in chapters, not letters.

The Results: The "Magic" of the System

The researchers tested this on six different games, from simple puzzles like 2048 to complex social strategy games like Diplomacy (where you have to negotiate with other players).

The Result: Even though they used a relatively small AI model (an "8B" model, which is like a standard laptop computer), it performed better than the world's biggest, most expensive AI models (like GPT-5 or Gemini) on these games.
Why? Because the big models were trying to "think" their way through every step, while the small model with COS-PLAY had a library of proven strategies to pull from. It was like a junior employee with a perfect handbook beating a genius who had to figure everything out on the fly.

The "Failure Mode" (A Funny Twist)

The paper also looked at how the system fails.

Old AI: When it fails, it often collapses. It gets confused, panics, and loses everything it had built up.
COS-PLAY: When it fails, it usually stagnates. It gets stuck doing the same thing over and over (like a hamster on a wheel), but it rarely loses everything. It has a "safety net" built into its skills that prevents total disaster.

In a Nutshell

COS-PLAY is a system where an AI learns to play games by teaching itself a library of tricks while it plays.

The Player uses the tricks to win.
The Librarian watches the player, writes down the tricks that work, and throws away the ones that don't.
Together, they get better and better, turning a confused robot into a master strategist.

It's the difference between a student trying to memorize a whole textbook in one night versus a student who builds a set of flashcards, reviews them, and improves their study guide every single day.

1. Problem Statement

Long-horizon interactive environments (such as complex video games) require agents to perform multi-step reasoning, chain skills over extended timeframes, and make robust decisions under delayed rewards and partial observability. While Large Language Models (LLMs) show promise as game-playing agents, they struggle with consistent long-horizon decision-making due to two main limitations:

Lack of Structured Memory: They often fail to discover, retain, and reuse structured skills across episodes, leading to repetitive or incoherent behavior.
Static vs. Dynamic Gap: Existing approaches often rely on curated human data or treat skills as static tools. There is a lack of frameworks where the decision agent (which acts) and the skill bank (which stores knowledge) co-evolve. The decision agent needs better skills to act effectively, but the skill bank needs high-quality trajectories from the agent to discover and refine new skills.

The paper addresses the question: Can a decision agent and a skill bank be learned jointly in a closed loop to improve long-horizon performance without human supervision?

2. Methodology: COS-PLAY Framework

COS-PLAY is a multi-agent co-evolution framework consisting of two primary components that interact in a closed loop:

A. The Decision Agent ( $A_D$ )

Role: An LLM-based agent that interacts with the game environment.
Mechanism:
- Skill Retrieval: At each timestep, it retrieves candidate skills from the current skill bank based on the observation and its internal intention state ( $z_t$ ).
- Intention Update: It updates its strategic focus (intention) conditioned on the selected skill.
- Action Execution: It executes primitive actions guided by the active skill's plan, current observation, and intention.
Training: Optimized using Group Relative Policy Optimization (GRPO) with specific LoRA adapters for action-taking and skill-retrieval. It is rewarded for progress, skill adherence, and minimizing unnecessary switching.

B. The Skill Bank Agent ( $A_S$ )

Role: An LLM-based pipeline that manages the "Skill Bank" (a structured library of reusable behavior protocols).
Mechanism: It processes unlabeled rollouts (trajectories) from the Decision Agent through a four-stage pipeline:
1. Boundary Proposal: Heuristically identifies potential skill-transition points using signals like predicate flips, intention changes, and reward spikes.
2. Infer Segmentation: Selects the optimal subset of boundaries to segment the trajectory into skill episodes and labels them against existing skills or marks them as "new."
3. Contract Learning: Aggregates state changes (added/deleted predicates) across multiple instances of a skill to learn a compact Effect Contract (pre-conditions, success criteria, and state changes). It filters out noise to ensure reliability.
4. Skill Bank Maintenance: Performs operations like Refine (update contracts), Materialize (promote new skills), Merge/Split (handle redundancy or over-broadness), and Retire (remove unused skills).
Training: Also optimized via GRPO with LoRA adapters for segmentation, contract learning, and curation.

C. The Co-Evolution Loop

The Decision Agent interacts with the environment using the current Skill Bank to collect trajectories.
The Skill Bank Agent analyzes these trajectories to discover, refine, and update the Skill Bank.
The updated Skill Bank is fed back to the Decision Agent, improving its future decision-making.
This cycle repeats, allowing both agents to improve simultaneously.

3. Key Contributions

Co-Evolution Framework: Proposes the first unified framework that couples LLM-based game decision-making with an agentic skill-bank pipeline for unsupervised skill discovery and continual refinement.
Structured Skill Abstraction: Introduces a rigorous skill representation including Summary, Pre-conditions, Plan, Success/Abort Criteria, and a verified Effect Contract (state changes), enabling reliable retrieval and execution.
Comprehensive Evaluation: Demonstrates that a relatively small model (8B parameters) equipped with this framework can outperform frontier LLMs (e.g., GPT-5.4, Gemini-3.1-Pro) in long-horizon tasks.
Data Efficiency: Achieves strong performance with few-shot adaptation (approx. 25 iterations) compared to the hundreds of steps required by traditional RL-based agents.

4. Experimental Results

The framework was evaluated on six game environments:

Single-Player: 2048, Candy Crush, Tetris, Super Mario Bros.
Multi-Player (Social Reasoning): Avalon, Diplomacy.

Key Findings:

Performance: COS-PLAY (based on an 8B Qwen3 model) achieved a 25.1% average reward improvement over GPT-5.4 on single-player benchmarks.
Social Reasoning: In multi-player games (Avalon and Diplomacy), COS-PLAY remained competitive with State-of-the-Art (SOTA) LLMs, outperforming Gemini-3.1-Pro by 8.8% in Diplomacy.
Skill Reusability: Analysis showed the system learns compact, reusable skill abstractions rather than memorizing isolated trajectories. The skill bank remained stable (55–70 active skills) while discovering over 100 unique skills, with periodic curation removing redundancy.
Generalization: The adaptation process preserved the base model's general reasoning capabilities (Math-500, MMLU-Pro), with negligible performance drops.

5. Significance and Implications

Bridging the Gap: COS-PLAY demonstrates that structured skill banks can act as a powerful inductive bias, allowing smaller models to compete with massive frontier models in complex, long-horizon tasks.
Autonomous Improvement: It moves beyond static tool use or one-off prompting, showing how agents can autonomously build a "memory" of procedural knowledge that evolves with experience.
Scalability: The framework is highly data-efficient, requiring minimal human annotation and few training iterations to reach high performance.
Future Direction: The authors note limitations in handling raw visual inputs (relying on text summaries) and propose extending the framework to multi-modal environments where skills can capture visual reasoning dynamics.

In conclusion, COS-PLAY establishes a new paradigm for autonomous agents where the policy (how to act) and the knowledge base (what skills exist) co-evolve, leading to robust, reusable, and adaptable long-horizon decision-making.