🤖 AI

Boosting Deep Reinforcement Learning with Semantic Knowledge for Robotic Manipulators

This paper proposes integrating Knowledge Graph Embeddings with Deep Reinforcement Learning to provide semantic context for robotic manipulators, achieving a 60% reduction in learning time and a 15% improvement in task accuracy without increasing computational complexity.

Original authors: Lucía Güitta-López, Vincenzo Suriani, Jaime Boal, Álvaro J. López-López, Daniele Nardi

Published 2026-01-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lucía Güitta-López, Vincenzo Suriani, Jaime Boal, Álvaro J. López-López, Daniele Nardi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot arm to pick up a mug, a bottle, or a cereal box. In the world of Deep Reinforcement Learning (DRL), the robot is like a baby learning to walk. It has to try, fail, fall, try again, and eventually figure out how to stand. The problem is that this "trial and error" process takes an enormous amount of time and computer power. The robot has to bump into thousands of objects before it finally learns the right way to grab them.

This paper proposes a clever shortcut: giving the robot a cheat sheet in the form of "semantic knowledge."

Here is how the researchers did it, explained simply:

1. The Problem: Learning by Blind Guessing

Usually, a robot learning to grab things only sees a picture (a video feed) and has to guess what to do. It doesn't "know" that a red cylinder is a bottle or that a handle belongs to a mug. It has to learn these facts from scratch by crashing into things millions of times.

2. The Solution: The "Knowledge Graph" Cheat Sheet

The researchers gave the robot a Knowledge Graph. Think of this as a giant, pre-written encyclopedia of common sense facts about the world.

The Graph: It contains facts like "A mug is red," "A bottle is cylindrical," and "You grab a mug by the handle."
The Embedding: They turned these facts into a special mathematical code (called an embedding) that the robot's brain can read instantly.

3. The New Architecture: Adding a Second Sense

The researchers built a new brain for the robot.

Old Brain: Only looked at the camera image (Visual).
New Brain: Looked at the camera image AND read the "cheat sheet" (Semantic Knowledge) at the same time.

Imagine you are trying to find a specific book in a dark library.

The Old Way (Baseline Robot): You turn on a flashlight and scan every single shelf, reading every spine, hoping to find the right one. It takes forever.
The New Way (This Paper's Robot): You have a flashlight, but someone also whispers, "The book is red and on the third shelf." You still use your flashlight, but now you know exactly where to look. You find the book much faster.

4. The Experiments: Two Robots, Two Scenarios

They tested this on two different robot arms (a mobile one called TIAGo and a fixed industrial one called IRB120) in two situations:

Fixed Colors: The mug was always red, the bottle always yellow.
Random Colors (The Hard Mode): The mug could be red or blue, the bottle yellow or purple. The robot had to adapt on the fly.

5. The Results: Faster and Smarter

The results were impressive:

Speed: The robots with the "cheat sheet" learned up to 60% faster than the ones without it. In the "Hard Mode" (random colors), the smart robot learned in half the time it took the others.
Accuracy: The smart robots were more accurate. They grabbed the objects correctly about 15% more often than the robots that had to guess everything from scratch.
Efficiency: The robots didn't just learn faster; they moved more smoothly. Instead of wobbling around and making many small, confused adjustments, they went straight to the object and aligned their gripper correctly almost immediately.

6. What This Means (According to the Paper)

The paper claims that by feeding the robot "common sense" facts alongside its camera vision, the robot doesn't need to waste time rediscovering basic facts about the world. It can focus its energy on learning the specific movements needed to grab the object.

Crucially, the paper states:

This method did not make the computer slower or more complicated to run.
The improvement came purely from giving the robot better information, not from making the robot "stronger."
The experiments were done in a computer simulation (a virtual world). The paper notes that moving this to a real physical robot would require extra steps to translate the computer images to real-world cameras, but the core idea works in the simulation.

In short, the paper shows that if you teach a robot what things are (using a knowledge graph) while it learns how to move, it learns the job much faster and does it better.

Technical Summary: Boosting Deep Reinforcement Learning with Semantic Knowledge for Robotic Manipulators

Problem Statement
Deep Reinforcement Learning (DRL) has emerged as a powerful framework for robotic control, particularly for complex sequential decision-making in changing environments. However, its practical deployment is frequently hindered by high sample complexity, requiring substantial amounts of experience (agent-environment interactions) to learn effective policies. This results in significant computational and time costs. While virtual environments mitigate some economic costs, the sheer volume of experience required remains a bottleneck. The authors posit that learning efficiency could be improved if the agent possesses contextual knowledge about the environment that complements its raw visual observations, thereby reducing the need for extensive exploration.

Methodology
The proposed approach integrates semantic knowledge into the DRL learning architecture using Knowledge Graph Embeddings (KGEs). The methodology involves the following components:

Knowledge Graph Construction and Embedding: A pre-built knowledge graph ( $G$ ) containing semantic relationships between entities (e.g., objects, colors, shapes) is utilized. For a given scene, a subgraph selector extracts relevant entities and their relationships (nodes at distance 1). This subgraph is converted into a textual representation and processed using a pre-trained GloVe model to generate a single continuous vector embedding representing the scene's semantic context.
DRL Architecture Integration: The authors propose an original architecture based on the Asynchronous Advantage Actor-Critic (A3C) algorithm. The state representation begins with convolutional layers (CNN) processing 64×64 RGB images. Crucially, the semantic embeddings are concatenated with the hidden activations from these visual layers before the policy approximation block (specifically, before the LSTM layer). This allows the agent to utilize both raw visual data and rich contextual information simultaneously.
Experimental Setup: Experiments were conducted in simulation using the MuJoCo platform with two robotic manipulators: the 7-DoF TIAGo mobile manipulator and the 6-DoF ABB IRB120 industrial arm. The task involved grasping three distinct targets (a mug, a bottle, and a cereal box) with specific orientation constraints.
Experimental Conditions: Two sets of experiments were performed:
1. Invariant Features: Targets had fixed colors.
2. Domain Randomization (DR): Target colors were randomized (two possible colors per object) to increase complexity.
  Three agent types were compared in each set: a Baseline Model (BM) with no KGE, a Partial KGE agent (object type only), and a Full KGE agent (object type and color).

Key Contributions

Novel Architecture: An original DRL agent architecture that successfully integrates KGEs during the learning process by modifying baseline layers to concatenate embeddings with visual hidden activations.
Quantitative Methodology: A framework to quantify performance improvements in terms of learning time and accuracy by comparing agents with full semantic knowledge, partial knowledge, and no semantic knowledge.
Exploration Analysis: An analysis of joint orientation distributions to evaluate how KGEs influence the agents' exploration and exploitation capabilities.
Comprehensive Evaluation: A quantitative and qualitative assessment of embedding influence across different environments and robot morphologies.

Results
The experimental validation demonstrated that integrating semantic knowledge significantly enhances learning efficiency without increasing computational complexity:

Learning Time Reduction:
- In environments with invariant features, the Full KGE agent achieved a learning time reduction of up to 25% (TIAGo) compared to the BM, with the best models converging earlier.
- In environments with Domain Randomization, the improvement was more pronounced, achieving up to a 60% reduction in learning time for both robots. The Full KGE agent reached its best performance in roughly half the training steps required by the BM.
Accuracy Improvements:
- Invariant Features: The Full KGE agent improved accuracy by approximately 12 percentage points over the BM for both robots (e.g., 72% vs. 60% for TIAGo; 92% vs. 80% for IRB120).
- Domain Randomization: The Full KGE agent showed a 16% accuracy improvement for TIAGo (79% vs. 63%) and a 10% improvement for IRB120 (86% vs. 76%) over the BM.
Statistical Significance: One-way ANOVA tests confirmed that the performance differences between the Full KGE agents and the BM were statistically significant across all conditions (p-values < 0.001).
Joint Distribution Analysis: Agents with Full KGEs exhibited lower standard deviations in joint angle distributions and reached grasping points more directly. For instance, in DR scenarios, the Full KGE agent required fewer steps (20 vs. 34) to approach and align with the target compared to the BM, indicating more efficient exploitation of the correct policy.

Significance and Claims
The paper claims that providing agents with semantic knowledge in the form of KGEs is a viable strategy to reduce sample complexity and improve the effectiveness of DRL in robotic applications. The authors emphasize that this approach introduces commonsense knowledge without incurring significant computational costs or architectural complexity. The results suggest that semantic context helps agents learn faster and achieve higher accuracy, particularly in complex scenarios with varying features (Domain Randomization). The study concludes that while the current approach relies on predefined knowledge graphs, the methodology demonstrates promising capabilities for any scenario where DRL agents can leverage semantic knowledge sources to bridge the gap between observation and action. Future work is identified as exploring optimal integration points for KGEs within the network and addressing sim-to-real transfer.