🤖 machine learning

Quantifying the Effect of Feedback Frequency in Interactive Reinforcement Learning for Robotic Tasks

This work investigates the influence of feedback frequency in interactive reinforcement learning for robotic tasks with continuous spaces and shows that no single optimal frequency exists and that feedback rates should be dynamically adapted to the agent's increasing competence.

Original authors: Daniel Harnack, Julie Pivin-Bachler, Nicolás Navarro-Guerrero

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Harnack, Julie Pivin-Bachler, Nicolás Navarro-Guerrero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a Robot to Move

Imagine you are trying to teach a robot arm to pick up a cup. The robot does not yet know how to do this. It must learn through trial and error, failing and trying again. This is called Reinforcement Learning.

Normally, the robot learns very slowly because it must make millions of mistakes before finding the right move. To speed this up, researchers let a "teacher" (such as a human or an intelligent computer program) provide feedback to the robot.

The big question this paper asks is: How often should the teacher correct the robot?

Should the teacher correct the robot every time it makes a mistake?
Should the teacher let the robot struggle a bit before intervening?
Or should the teacher only intervene at the very end?

The Experiment: A Gym for Robot Arms

The researchers set up a "gym" with various robot arms (some small, some large) and different difficulty levels.

The Easy Level: A simple arm with 2 joints (like a basic elbow and shoulder).
The Hard Level: A complex arm with 7 joints (like a full human arm with shoulder, elbow, wrist, and fingers).

The goal was simple: The robot had to move its hand to a specific point. If it got close enough, it received a "good job" signal. If it moved away, the teacher said, "Oops, undo that," and let the robot try again.

They tested various settings for the "Probability to Ask" (L). Imagine this as a dial in the robot's brain:

L = 0: The robot never asks for help. It learns alone.
L = 0.99: The robot asks for help almost every time it makes a mistake.

What They Found: There Is No One-Size-Fits-All Solution

The results were surprising because the "best" way to teach changed depending on how difficult the task was and how long the robot had already been training.

1. The Problem of the "Overprotective Parent"

With simple tasks (the small 2-joint arm), the robot learned fastest when the teacher helped a lot. It was like a student who learns quickly when a tutor is right there to correct every typo. The more help, the better the final result.

However, with complex tasks (the large 7-joint arm), too much help at the beginning was a disaster.

The Analogy: Imagine teaching someone to ride a bicycle. If you hold the handlebars so tightly that they never fall, they might learn to balance perfectly while you are holding them. But the moment you let go, they fall because they never learned how to recover from a wobble on their own.
The Result: With the complex robots, too frequent correction by the teacher at the start caused the robot to learn a "fake" version of the task. It got stuck in a dead end and could not master the harder parts of the task. It had to struggle a bit to learn how to correct its own mistakes.

2. The "Goldilocks" Shift

The paper discovered that the perfect amount of help is not a fixed number; it changes over time.

Early in training: The robot needs a "Goldilocks" amount of help—not too much, not too little. If the teacher helps too much, the robot becomes lazy and does not explore enough. If the teacher helps too little, the robot gets frustrated and learns too slowly.
Later in training: Once the robot has learned the basics, it actually benefits from more help to refine its movements and become super-precise.

3. The "Adaptive Trainer"

The researchers tried a new strategy: The Adaptive Trainer.
Instead of keeping the help level constant, they started with a moderate amount of help and slowly increased it as the robot improved.

The Result: This worked best. It combined the speed of early learning (by not over-correcting) with the precision of late learning (by correcting more frequently once the basics were established).

The Main Takeaway

There is no single "magic number" for how often a teacher should correct a robot.

Simple tasks: More help is usually better.
Complex tasks: You must start with less help so the robot learns to explore, and then gradually give more help as it becomes smarter.

The paper concludes that the best way to teach a robot is to be an adaptive teacher: Start by letting the robot struggle a bit to build a strong foundation, and then intervene more frequently to polish the details as the robot becomes more competent.

Summary in One Sentence

Teaching a robot is not about constant correction; it is about knowing when to let it stumble so it can learn to walk, and when to hold its hand so it can learn to run.

1. Problem Statement

Reinforcement Learning (RL) is widely used in robotics but suffers from low data efficiency, often requiring millions of interactions to solve complex problems. Interactive Reinforcement Learning (IRL) addresses this by incorporating teacher feedback (human or artificial) to guide the agent. However, the literature lacks a consensus regarding the optimal feedback frequency:

Some studies suggest that higher frequencies produce more robust agents.
Others argue that excessive feedback leads to "reward hacking" or delays the onset of learning.
A third group advocates for medium frequencies.

The authors hypothesize that these contradictions arise because previous studies primarily used low-dimensional, discrete grid-world environments. They argue that the effect of feedback frequency likely depends on task complexity and the agent's competence level (training stage), factors that have not been systematically isolated in continuous robotic control tasks.

2. Methodology

Experimental Setup

Robotic Platforms: The study utilized inverse kinematics learning tasks for two robots:
- NAO: 2 Degrees of Freedom (DoF) and 4 DoF configurations.
- KUKA LBR iiwa 14 R820: 2 DoF, 4 DoF, and 7 DoF configurations.
Task: The agent must control the robot arm in joint space to reach arbitrary target zones in the Cartesian task space.
Reward Function: A sparse reward function was used (Reward = 1 for reaching the target, 0 otherwise) to isolate the effect of feedback without distorting reward shaping.
Algorithm: Continuous Actor-Critic Learning Automaton (CACLA), implemented with Multi-Layer Perceptrons (MLPs) using PyTorch and Adam optimization.

Interactive Mechanism (Error Correction)

Feedback Type: The teacher provides binary feedback based on whether an action reduces the Euclidean distance to the target.
Mechanism: If an action increases the distance (an "error"), the teacher signals this, and the agent reverses the action and explores a new one. This steers the exploration process rather than directly shaping the reward function.
Control Variable (Ask Likelihood, $L$ ): The probability that the agent accepts teacher feedback.
- $L = 0.0$ : Vanilla RL (no feedback).
- $L = 0.99$ : Highly interactive (feedback at almost every step).
- The study tested $L$ values from 0.0 to 0.99 in steps.

Evaluation Metrics

Failure Rate: Percentage of missed targets (equivalent to $1 - \text{average cumulative reward}$ ).
Positioning Error: Mean Euclidean distance to the target.
Analysis Strategy: Performance was analyzed not only by final convergence but through failure-rate-based time thresholds. The authors compared how quickly different $L$ values reached specific performance thresholds (e.g., 50%, 10%, 5% failure rate) at various training stages.

3. Key Results

The experiments showed that the "optimal" feedback frequency is not static; it depends heavily on task complexity and the training phase.

A. Dependence on Task Complexity

Simple Tasks (NAO 2 DoF): High feedback frequencies ( $L \geq 0.7$ ) consistently improved performance and led to perfect convergence. Low frequencies diverged later in training.
Complex Tasks (KUKA 2, 4, 7 DoF):
- Early Training: High feedback frequencies were often detrimental, causing a delayed learning phase or significantly worse performance compared to low/medium frequencies.
- Late Training: As the agent became more competent, high feedback frequencies became advantageous, leading to more robust strategies and lower failure rates in the long term.
- Divergence: Agents with low frequency in complex tasks often reached a local optimum and then diverged (performance deteriorated), whereas agents with high frequency maintained stability.

B. The "Optimal Trajectory"

The study found that no single $L$ value is optimal for the entire training duration.

Early Stage: Medium or low $L$ values are often superior to prevent the agent from becoming overly dependent on simplified feedback, which hinders exploration of the entire state space.
Late Stage: High $L$ values become optimal to refine the strategy and ensure robustness against hyperparameter sensitivity.

C. Quantifying Task Complexity Reduction

The authors introduced a heuristic to predict the optimal initial $L$ . They measured the relative task complexity induced by feedback (how much the teacher simplifies the task for a random agent).

Result: Feedback frequencies that reduce task complexity too drastically (making the task too easy) are detrimental at the start.
Optimal Range: The most beneficial initial $L$ values correspond to a relative reduction in task complexity of approximately 0.78 to 0.95. This suggests the teacher should simplify the task slightly, but not so much that the agent fails to learn generalizable behaviors.

D. Proof-of-Concept for Adaptive Strategy

The authors demonstrated an adaptive interaction regime on the KUKA 2 DoF task:

Start with $L=0.0$ (no feedback) for rapid initial exploration.
Switch to $L=0.5$ at epoch 14.
Switch to $L=0.99$ at epoch 18.

Result: This adaptive agent achieved the rapid early convergence of low- $L$ agents and the long-term precision of high- $L$ agents, outperforming any fixed- $L$ strategy as measured by the area under the failure rate curve.

4. Key Contributions

Resolution of Contradictions: The paper unifies contradictory literature by showing that "more feedback is better" and "less feedback is better" are both true, depending on task complexity and training stage.
Validation in Continuous Space: Unlike previous grid-world studies, this work validates these dynamics in continuous state and action spaces using real robotic kinematics (NAO and KUKA).
Dynamic Optimal Frequency: It establishes that optimal feedback frequency is a temporal trajectory, not a static hyperparameter.
Heuristic for Initialization: It proposes a method to select the initial feedback frequency based on the relative reduction of task complexity, offering a practical guideline for practitioners.

5. Significance and Conclusion

This research fundamentally shifts the paradigm of Interactive RL from searching for a "one-size-fits-all" feedback frequency to introducing adaptive interaction regimes.

Practical Implications: For robotic applications, this suggests that human teachers or automated oracles should not provide constant, high-frequency feedback. Instead, they should begin with sparse or medium feedback to encourage exploration and gradually increase feedback frequency as the agent learns the task structure.
Robustness: High-frequency feedback in later stages is crucial to prevent strategy divergence and ensure robustness, especially in high-dimensional (7 DoF) tasks.
Future Directions: The authors suggest that future work should focus on automating the detection of optimal switching points for $L$ and investigating the relationship between feedback frequency and "capacity loss" in neural function approximators.

In summary, the paper argues that an adaptive feedback frequency, tailored to the agent's competence and the specific complexity of the robotic task, is key to maximizing data efficiency and performance in Interactive Reinforcement Learning.