💬 NLP

Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning

The paper introduces SuperIgor, a self-guided framework that employs goal-conditional reinforcement learning and iterative co-training to enable language models to autonomously generate and refine high-level plans for instruction-following tasks, thereby reducing reliance on manual annotation while improving adherence and generalization.

Original authors: Zoya Volovikova, Nikita Sorokin, Dmitriy Lukashevskiy, Aleksandr Panov, Alexey Skrynnik

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zoya Volovikova, Nikita Sorokin, Dmitriy Lukashevskiy, Aleksandr Panov, Alexey Skrynnik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly clumsy, robot how to play a complex video game like Minecraft. You give it a simple command: "Build a stone furnace."

In the past, researchers tried two main ways to solve this:

The "Show Me" Method: They recorded thousands of hours of expert humans playing the game and forced the robot to copy them. This works well, but it's incredibly expensive and time-consuming to hire all those experts.
The "Trial and Error" Method: They let the robot play the game on its own, hoping it figures out the rules. But without a map or a guide, the robot often gets lost, confused, or gives up because the "reward" (winning) is so far away.

Enter SuperIgor.

SuperIgor is a new framework that acts like a collaborative coaching duo. It pairs a Language Coach (a Large Language Model, or LLM) with a Physical Player (a Reinforcement Learning agent). Instead of relying on expensive human recordings or blind trial and error, they teach each other.

Here is how the magic happens, broken down into simple analogies:

1. The Coach and the Player

The Language Coach (LLM): This is the "brain." It reads your instruction ("Build a furnace") and breaks it down into a checklist. It doesn't know the game perfectly yet, so it might guess the steps: 1. Get wood, 2. Make a table, 3. Get stone...
The Physical Player (RL Agent): This is the "hands." It tries to execute the checklist in the game world. It doesn't understand the big picture; it just knows how to move, chop, and craft.

2. The "Skill Curriculum" (Learning to Walk Before Running)

The biggest problem in these games is that the reward is too far away. If the robot tries to build a furnace immediately, it might fail 1,000 times before succeeding once. It gets discouraged.

SuperIgor uses a Skill Curriculum. Imagine a video game tutorial that unlocks levels as you master them.

Level 1: The robot is only allowed to try instructions like "Get wood." It learns this quickly and gets a "win" signal.
Level 2: Once it masters "Get wood," the coach adds a new step: "Get stone." Now the robot practices combining the two.
Level 3: Finally, it tries the full "Build a furnace" plan.

This prevents the robot from being overwhelmed. It builds a foundation of small wins before tackling the big boss.

3. The Feedback Loop (The "Self-Correction" Cycle)

This is the most unique part. In most systems, the coach writes the plan, and the player tries it. If the player fails, the coach just guesses again.

In SuperIgor, they have a conversation:

The Coach writes a plan.
The Player tries it in the game.
The Result: If the player succeeds, the Coach gets a "High Five." If the player fails, the Coach gets a "Gentle Nudge."
The Update: The Coach learns from the Player's experience. It realizes, "Oh, I told the player to get stone before making a table, but that's impossible! I need to change my plan."

Over time, the Coach gets better at writing realistic plans, and the Player gets better at following them. They get smarter together, without needing a human to step in and fix the mistakes.

4. Why is this a big deal?

No Human Experts Needed: You don't need to hire thousands of gamers to record data. The system generates its own training data by playing and learning.
It Handles "New" Instructions: If you ask the robot to "Build a diamond sword" (something it hasn't seen before), it can still do it. Because it learned the logic of the game (how to combine skills) rather than just memorizing a specific script.
It's Robust: Even if you change the wording of the instruction (e.g., "Craft a furnace" vs. "Make a smelting box"), the system understands the goal remains the same.

The Bottom Line

Think of SuperIgor as a self-improving apprenticeship. Instead of a master craftsman teaching an apprentice by hand, the apprentice (the robot) tries to build things, and the master (the AI coach) watches, critiques the blueprint, and rewrites it to be more practical. Eventually, they become a perfect team, capable of solving complex, long-term puzzles that neither could solve alone.

It turns the chaotic process of "trial and error" into a structured, efficient learning journey, making AI agents much better at following our complex human instructions.

1. Problem Statement

The paper addresses the challenge of instruction-following in complex, dynamic, and partially observable environments (specifically the CrafText benchmark, a Minecraft-like environment).

The Core Difficulty: Existing methods struggle to ground natural language instructions into long-horizon behaviors when rewards are sparse and delayed (only given upon full task completion).
Limitations of Prior Work:
- Demonstration-based methods (e.g., VPT, STEVE-1) require expensive, high-quality expert datasets and do not scale well.
- Experience-based RL methods often rely on predefined subtasks or fixed skill libraries. They fail in open-ended environments where low-level skills are not predefined, and they struggle with the "credit assignment" problem (determining which actions led to success).
- Planning-based methods often use massive LLMs (100B+ params) and rely on rigid, pre-defined subgoal sets, limiting flexibility and scalability.

Goal: Develop a framework that enables an agent to learn instruction-following from scratch in an environment without predefined low-level skills, using a self-learning mechanism to generate and refine its own plans.

2. Methodology: The SuperIgor Framework

SuperIgor introduces a self-supervised, iterative co-training loop between a Large Language Model (LLM) acting as a planner and a Reinforcement Learning (RL) agent acting as the executor.

A. Core Components

LLM Planner: Generates structured plans (sequences of subtasks) from natural language instructions.
RL Agent: Executes the generated plans in the environment using PPO (Proximal Policy Optimization).
Iterative Feedback Loop: The RL agent's performance provides feedback to refine the LLM's plan generation, creating a "self-guided" improvement cycle.

B. The Four-Stage Pipeline

The framework operates in cycles (Cycle 1 to Cycle N):

Plan Generation (Cycle 1 & Refinement):
- Ontology Construction: Instead of generating free-form text, the LLM first builds a Subtask Bank (a vocabulary of atomic skills) and an Ontology Graph (hierarchical dependencies between subtasks).
- Plan Expansion: Instructions are decomposed into plans based on this ontology, ensuring logical consistency and prerequisite ordering.
- Re-prioritization: In subsequent cycles, candidate plans are filtered and re-ranked based on the LLM's confidence (Negative Log-Likelihood) and the RL agent's historical success.
Policy Learning (Skill Curriculum Learning):
- To address sparse rewards, the authors introduce a Skill Curriculum.
- The agent starts by training on single-subtask plans.
- Once a subtask's success rate exceeds a threshold ( $\tau = 0.7$ ), it is marked as "mastered."
- The curriculum dynamically expands to include plans with mastered subtasks plus one new, unmastered subtask. This creates a smooth learning gradient.
Plan Validation:
- The RL agent executes candidate plans multiple times (across different random seeds) to calculate a reliable Success Rate (SR). This aggregates performance in stochastic environments.
LLM Fine-Tuning (DPO):
- The LLM is fine-tuned using Direct Preference Optimization (DPO).
- Preference Pairs: Plans with high SR are treated as "preferred," and low-SR plans as "dispreferred."
- Objective: The LLM learns to align its plan generation with the RL agent's actual capabilities, effectively learning which plans are executable rather than just what the plan should look like linguistically.

3. Key Contributions

Self-Supervised Training Paradigm: A novel framework where the LLM and RL agent co-evolve without requiring manually annotated datasets or expert demonstrations.
Skill Curriculum Learning: A specialized curriculum design that enables RL agents to master complex, compositional tasks under sparse reward conditions by incrementally building a "skill tree."
Ontology-Guided Plan Generation: A structured approach to plan extraction that enforces hierarchical dependencies and reduces synonymy, ensuring plans are logically sound and executable.
DPO for Plan Selection: Using DPO to align the planner with the executor's performance, acting as a lightweight mechanism to filter out unexecutable plans and prioritize effective strategies.

4. Experimental Results

Experiments were conducted on the FOCUS dataset within the CrafText benchmark, featuring Atomic (single goal) and Combo (multi-goal) instructions.

Effectiveness vs. Baselines:
- SuperIgor significantly outperformed instruction-only baselines (e.g., PPO-T, FiLM).
- On Atomic tasks, SuperIgor achieved 0.34–0.44 success rate vs. 0.10–0.19 for baselines.
- On Combo tasks, SuperIgor achieved 0.21 vs. 0.08 for baselines.
- While an "Oracle" (trained on human-written perfect plans) performed best, SuperIgor closed the gap significantly (only ~0.20 behind Oracle vs. ~0.30 for baselines).
Generalization:
- Paraphrase Robustness: SuperIgor maintained performance when instructions were rephrased (e.g., "Craft a furnace" vs. "Make a furnace"), showing strong linguistic generalization.
- Compositional Generalization: On New Objects tasks (combinations of goals never seen together), SuperIgor outperformed the Oracle. While the Oracle dropped sharply on unseen combinations, SuperIgor's performance was more stable. This is attributed to the agent learning from multiple alternative plans per instruction, exposing it to richer variability.
Iterative Improvement:
- Performance improved from Cycle 1 to Cycle 2 (e.g., Combo tasks rose from 0.06 to 0.21), demonstrating the efficacy of the self-guided refinement loop.
Ablation Studies:
- Curriculum + Ontology: The curriculum was ineffective without the ontology-structured plans (performance dropped to 0.06 without ontology).
- LLM Choice: Smaller models (Qwen-14B) outperformed larger, less specialized models (NextCoder-32B), highlighting that model specialization matters more than parameter count for plan generation.
- DPO vs. GRPO: DPO outperformed Group Relative Policy Optimization (GRPO) for planner tuning, as DPO's pairwise preference approach better matched the plan-ranking objective than direct reward optimization.

5. Significance and Conclusion

SuperIgor represents a significant step forward in embodied AI and instruction following:

Scalability: It removes the bottleneck of collecting expert demonstrations, relying instead on the synergy between LLMs and RL.
Robustness: It demonstrates that agents can learn complex, long-horizon tasks in dynamic, partially observable environments without predefined skill libraries.
Generalization: The framework achieves state-of-the-art results on out-of-distribution tasks, proving that self-guided plan extraction leads to more robust and flexible agents than static planning or pure RL approaches.

The work suggests that the future of instruction-following agents lies in co-adaptive systems where the planner learns from the executor's failures, and the executor learns from the planner's structured guidance.