LLM-assisted Semantic Option Discovery for Facilitating Adaptive Deep Reinforcement Learning

This paper proposes a novel LLM-driven closed-loop framework that maps natural language instructions to executable rules and semantically annotates options to enhance the data efficiency, interpretability, and cross-environment transferability of Deep Reinforcement Learning, with experimental validation showing superior performance in constraint compliance and skill reuse.

The Gist

Imagine you are teaching a robot to navigate a busy office.

The Old Way (Traditional AI):
Usually, we teach robots by letting them crash, fail, and try again millions of times until they finally get it right. It's like teaching a child to ride a bike by throwing them off a cliff and hoping they learn to balance before hitting the ground. This takes forever (low data efficiency), and even when they learn, they can't explain why they turned left instead of right (lack of interpretability). If you move them to a slightly different room with a new obstacle, like a printer, they often forget everything and have to start over.

The New Way (This Paper's Solution):
The authors propose a new system called LLM-SOARL. Think of this as giving the robot a smart, experienced human mentor (the Large Language Model or LLM) who speaks the robot's language but also understands human instructions.

Here is how it works, broken down into three simple parts:

1. The "Skill Library" (Semantic Option Discovery)

Imagine the robot learns a trick: "Pick up the coffee cup and walk to the desk."
In the old days, if you asked it to "Pick up the mail and walk to the desk," it would treat this as a completely new, scary task and start crashing into walls again.

In this new system, the LLM Mentor looks at the new task and says, "Hey, that's basically the same as the coffee task! It's just a different object."

• The Analogy: It's like realizing that "driving to the grocery store" and "driving to the gas station" use the same driving skills, even though the destinations are different.
• The Magic: The system automatically tags these skills with human-readable labels (like "Move Coffee to Office"). When a new task comes along, the robot checks its "Skill Library," sees the match, and instantly reuses the old, safe driving skills instead of relearning from scratch.

2. The "Safety Guardian" (Constraint Adaptation)

Sometimes, a human boss gives a vague warning: "Be careful not to bump into the plants or the new printer."
Traditional robots are bad at understanding vague language. They might need a rigid, pre-programmed map that says "No plants allowed."

Here, the LLM Mentor acts as a translator.

• The Analogy: It's like a human supervisor who hears "Don't hit the printer" and immediately draws a red "No-Go" zone on the robot's map.
• The Magic: The system turns that sentence into a strict rule. If the robot gets too close to a printer, it gets an immediate "penalty" (like a gentle electric shock or a time-out). This forces the robot to learn the new rule instantly without needing to crash into the printer first.

3. The "Loop" (Continuous Improvement)

The system works in a circle:

1. Listen: The robot gets a new goal and a new rule (e.g., "Deliver juice, avoid the printer").
2. Translate: The LLM Mentor translates the rule into a map and checks the Skill Library for existing tricks.
3. Act: The robot tries the trick.
4. Learn: If it works, the skill gets a gold star. If it hits the printer, the Safety Guardian stops it immediately.
5. Repeat: The robot gets smarter and faster every time.

Why is this a big deal?

The researchers tested this in two worlds:

1. Office World: A grid where a robot delivers coffee and mail. When they added a new obstacle (a printer) and a new rule, the robot adapted instantly using the LLM's help, whereas older robots had to relearn everything.
2. Montezuma's Revenge: A very hard video game with hidden traps and delayed rewards. The robot used the LLM to understand the game's logic and avoid traps it had never seen before.

The Bottom Line:
This paper introduces a way to make AI smarter, safer, and faster by letting it talk to humans in plain English. Instead of just crunching numbers, the AI uses a "brain" (the LLM) to understand the meaning behind instructions, reuse old tricks for new jobs, and strictly follow safety rules without needing millions of failed attempts. It's the difference between a robot that learns by trial-and-error and a robot that learns by listening and understanding.

Technical Summary

Here is a detailed technical summary of the paper "LLM-assisted Semantic Option Discovery for Facilitating Adaptive Deep Reinforcement Learning" (LLM-SOARL).

1. Problem Statement

Deep Reinforcement Learning (DRL) has achieved success in complex tasks but faces three critical barriers in practical applications:

• Low Data Efficiency: Traditional DRL requires massive interaction data, especially in environments with sparse rewards and delayed feedback.
• Lack of Interpretability: Learned policies are often "black boxes," making it difficult to ensure behavioral safety or understand decision logic.
• Limited Transferability: Policies trained in one environment often fail to adapt to new, slightly different scenarios (e.g., adding a new obstacle like a printer to an office) without retraining from scratch.

Existing hybrid approaches (combining symbolic planning and RL) struggle to formalize general knowledge or adapt to unstructured human instructions (natural language) regarding environmental changes.

2. Methodology: The LLM-SOARL Framework

The authors propose LLM-SOARL, a closed-loop framework that integrates Large Language Models (LLMs), Symbolic Planning, and Deep Reinforcement Learning. The system operates on a hierarchical structure with three core modules:

A. Planning-Meta-Control Module

• Function: Acts as the high-level controller. It autonomously learns action models, formulates dynamic goals, and selects optimal "options" (temporally extended actions).
• Mechanism: It uses a symbolic planner (Metric-FF) to generate plans based on a Planning Domain Definition Language (PDDL). It maps high-level symbolic states to low-level primitive actions using a meta-controller.
• Reward Shaping: It assigns intrinsic rewards to encourage the exploration of new symbolic state pairs and updates the policy based on the success rates of action models.

B. Semantic Skill Generation Module

• Function: Enables cross-task skill reuse by mining implied general skills across different tasks.
• Mechanism:
  1. Semantic Labeling: When an option (policy) is executed, the LLM analyzes the transition from the initial state to the goal state. It abstracts low-level numerical changes into structured "Predicate-Argument" semantic labels (e.g., act(coffee, office)).
  2. Skill Library: Successful policies are stored in a "Skill Library" indexed by these semantic labels.
  3. Retrieval & Reuse: When a new task arises, the system generates a semantic label for the required sub-goal. If a matching label exists in the library, the agent retrieves and reuses the pre-trained policy, bypassing the need for re-exploration.

C. Constraint Adaptation Module

• Function: Ensures behavioral safety and compliance by translating unstructured natural language instructions into executable constraints.
• Mechanism:
  1. Semantic Parsing: The LLM parses natural language constraints (e.g., "Do not bump into plants and printers") to extract abstract entities (e.g., {plant, printer}).
  2. Limitation Set Generation: These entities are converted into a set of atomic propositions (a "Limitation Set") recognizable by the environment.
  3. Real-time Monitoring: A Reward Machine monitors the agent's state in real-time. If the agent's state intersects with the Limitation Set, the system triggers a penalty signal or terminates the episode, effectively enforcing the constraint without needing to retrain the entire policy from scratch.

3. Key Contributions

1. LLM-Driven Semantic Skill Generation: A novel module that mines general skills across tasks by translating state transitions into interpretable semantic labels, enabling efficient skill reuse.
2. Constraint Adaptation via Natural Language: A mechanism that converts intuitive linguistic instructions into formal, executable safety rules (Limitation Sets) using LLMs, allowing agents to adapt to new environmental constraints dynamically.
3. Closed-Loop Integration: The framework successfully bridges the gap between unstructured human language, symbolic planning, and low-level DRL, providing a platform that is data-efficient, interpretable, and transferable.

4. Experimental Results

The framework was evaluated on two domains: Office World (grid-based navigation) and Montezuma's Revenge (complex Atari game with sparse rewards).

• Data Efficiency:
  • In Office World, LLM-SOARL converged significantly faster than the baseline (SORL) when switching tasks.
  • It demonstrated the ability to reuse prior knowledge, drastically reducing the number of samples required to learn new tasks (e.g., delivering coffee vs. mail).
• Constraint Compliance:
  • When new obstacles (e.g., printers) were introduced via natural language instructions, the Constraint Adaptation Module allowed the agent to learn to avoid them with significantly fewer violations compared to learning from scratch.
  • In Montezuma's Revenge, the agent successfully internalized the constraint "Do not touch the stone," reducing the violation rate to zero after initial exploration.
• Cross-Task Transferability:
  • The Semantic Label Matching mechanism allowed the agent to identify that a sub-goal in a new task was semantically identical to a pre-trained skill (e.g., act(RightLadder, MiddleLadder)), enabling immediate policy execution without redundant exploration.

5. Significance

This paper presents a significant step toward safe and adaptable AI agents. By leveraging the world knowledge of LLMs, LLM-SOARL addresses the "blindness" of learning from scratch. It offers a solution for real-world scenarios where:

• Safety is paramount: Constraints can be updated via natural language without retraining.
• Efficiency is critical: Agents can transfer skills across similar but distinct environments.
• Interpretability is required: The use of semantic labels and symbolic planning makes the agent's decision-making process transparent and understandable to humans.

The framework effectively demonstrates that combining LLMs with symbolic planning and RL can overcome the limitations of traditional DRL in complex, dynamic, and safety-critical environments.