🤖 machine learning

Leveraging Human Feedback for Semantically-Relevant Skill Discovery

The paper introduces Semantically Relevant Skill Discovery (SRSD), a human-in-the-loop approach that improves the efficiency and diversity of unsupervised skill discovery in reinforcement learning by using human semantic labels to guide the learning of more meaningful and relevant behaviors.

Original authors: Maxence Hussonnois, Thommen George Karimpanal, Santu Rana

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Maxence Hussonnois, Thommen George Karimpanal, Santu Rana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to play soccer.

If you just tell the robot, "Explore and do interesting things," it might spend three hours learning how to spin in circles or wiggle its toes. These are "diverse" behaviors, but they are useless for soccer.

If you try to teach it using "preferences"—asking the human, "Do you like this movement more than that one?"—the human gets exhausted. It’s like playing a game of "Hot or Cold" for ten hours straight. It takes forever to figure out that "kicking" is better than "spinning."

This paper introduces a smarter way to teach robots called SRSD (Semantically Relevant Skill Discovery).

The Core Idea: The "Labeling" Shortcut

Instead of playing "Hot or Cold," the researchers suggest a method called Semantic Labeling.

Think of it like this: Imagine you are a coach watching a player. Instead of constantly saying "better" or "worse," you simply point and give things a name. You see a player run and say, "That’s running." You see them kick and say, "That’s a goal kick." If they start picking their nose, you just say, "That’s irrelevant."

By giving these behaviors names (semantics), the human provides a massive amount of information with very little effort. The robot doesn't just learn that one move is "better"; it learns the concept of "running" versus "kicking."

How the "Brain" of SRSD Works

The researchers built a system that works in three main steps:

The Explorer (The Toddler Phase): First, the robot is allowed to act like a toddler. It moves around randomly to see what its body can do (this is the "unsupervised" part). It discovers it can jump, flip, and crawl.
The Teacher (The Labeling Phase): The human steps in. Instead of comparing two moves, the human looks at a clip and assigns a label: "Running," "Jumping," or "Irrelevant."
The Student (The Learning Phase): The robot builds a "Semantic Predictor"—essentially a mental dictionary. It starts to realize, "Ah, when I move my legs like this, the human calls it 'running.' I should do more of that!"

Why is this a big deal? (The "Swiss Army Knife" Effect)

The paper proves two major things:

It’s much faster: Traditional methods get "confused" as you add more types of skills. If you have 16 different sports to learn, a preference-based robot gets lost in the math. But the SRSD robot stays efficient because it’s just learning a list of names.
It’s more diverse: Because the robot is rewarded for discovering new names (new semantic categories), it doesn't just learn 10 different ways to run forward. It learns to run, to jump, to flip, and to balance. It builds a "Swiss Army Knife" of skills rather than just one very sharp, single-purpose knife.

The Bottom Line

The researchers showed that by treating human feedback like naming things rather than ranking things, we can teach robots a wide, useful, and organized repertoire of skills much more efficiently. It moves us away from robots that are just "randomly active" toward robots that are "meaningfully capable."

Technical Summary: Leveraging Human Feedback for Semantically-Relevant Skill Discovery

1. Problem Statement

Unsupervised skill discovery in Reinforcement Learning (RL) aims to motivate agents to learn a diverse repertoire of behaviors without explicit reward functions. However, traditional unsupervised methods often suffer from two major flaws:

Misalignment: They may discover behaviors that are diverse but unsafe, unethical, or contextually irrelevant (e.g., an agent in a football simulation learning to "sleep" instead of "kick").
Feedback Inefficiency: While recent "preference-based" approaches attempt to align skills with human desires, they rely on pairwise comparisons. As the number of distinct skill types (e.g., running, jumping, walking) increases, the number of comparisons required to distinguish between them grows exponentially, making them highly inefficient for complex, multi-skill environments.

2. Methodology

The authors propose Semantically Relevant Skill Discovery (SRSD), a human-in-the-loop framework that shifts from preference-based feedback to semantic labeling.

A. Semantic Labelling (The Feedback Mechanism):
Instead of comparing two trajectories, a human observer is presented with a trajectory segment and asked to:

Identify if the behavior is relevant to the context.
If relevant, assign it to a specific semantic category (e.g., "running").
If not relevant, mark it as "irrelevant."
The authors mathematically demonstrate that this approach scales $O(1)$ with respect to the number of semantic classes, whereas preference-based feedback scales $O(1/|C|)$ , making semantic labeling significantly more efficient for large skill spaces.

B. The SRSD Framework:
The framework optimizes a multi-objective reward function:
$r_{SRSD} = r_{exploration} + r_{diversity} + w_t \cdot r_{relevance}$

Exploration & Diversity ( $r_{exp}, r_{div}$ ): Uses Contrastive Skill Discovery (CSD) and particle-based entropy estimation to ensure the agent explores the state space and learns distinguishable skills.
Relevance ( $r_{rel}$ ): A learned Semantic Predictor ( $q_\psi$ ) estimates the probability that a trajectory belongs to a desired semantic class. This predictor is trained via cross-entropy loss on the human labels.
Dynamic Weighting ( $w_t$ ): The relevance reward is introduced gradually to allow the agent to explore before being constrained by semantic goals.

C. Distributional Critic:
To handle the complexity and noise introduced by combining multiple intrinsic reward functions, the authors incorporate Truncated Quantile Critics (TQC). This allows the agent to model the entire distribution of returns, effectively managing aleatoric uncertainty and preventing the overestimation of value common in multi-objective RL.

D. Active Sampling:
To maximize the utility of a limited human feedback budget, the authors implement an active sampling strategy. This ensures that the human is not overwhelmed by "irrelevant" samples and that the dataset remains balanced across all relevant semantic categories.

3. Key Contributions

Semantic Labelling: A novel, feedback-efficient mechanism that leverages human cognitive strengths in categorization rather than comparison.
SRSD Framework: A novel human-in-the-loop architecture that integrates semantic relevance into the unsupervised skill discovery loop.
Distributional Perspective: The application of distributional RL (TQC) to mitigate uncertainty when learning from multiple intrinsic reward signals.
Theoretical & Empirical Validation: Proof that semantic labeling scales better than preference-based methods and extensive testing across locomotion and navigation tasks.

4. Results

Superior Performance: In experiments using the DeepMind Control Suite (DMC) and URLB benchmarks, SRSD consistently outperformed baselines (LSD, CIC, ComSD, and HaSD) in zero-shot, few-shot, and fine-tuning evaluations.
Semantic Diversity: Qualitative results (e.g., in Cheetah and Walker environments) show that SRSD discovers a wide range of distinct, meaningful behaviors (e.g., backflips, running, walking) rather than just variations of a single movement.
Scalability: In 2D navigation tasks, SRSD maintained high precision and recall even as the number of semantic sectors increased to 16, whereas preference-based methods (HaSD) failed to cover the required semantic space.
Robustness: The method proved robust to "human irrationality" (mistakes, myopia, and amnesia) and remained effective even with a very limited human feedback budget (as few as 40 labels).

5. Significance

This work represents a significant step toward General-Purpose Agents. By moving away from rigid, pre-defined rewards and inefficient pairwise preferences, SRSD provides a scalable pathway for humans to guide the discovery of complex, diverse, and contextually appropriate behaviors. This has profound implications for robotics and embodied AI, where agents must learn to navigate unpredictable environments by interacting with human supervisors in a natural, efficient manner.