💬 NLP

Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications

This paper introduces Hierarchical Reward Design from Language (HRDL) and its solution, Language to Hierarchical Rewards (L2HR), to translate nuanced human specifications into hierarchical reward functions that better align AI agent behavior with human expectations in complex, long-horizon tasks.

Original authors: Zhiqin Qian, Ryan Diaz, Sangwon Seo, Vaibhav Unhelkar

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhiqin Qian, Ryan Diaz, Sangwon Seo, Vaibhav Unhelkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook a complex meal, like a salad. You don't just want the robot to finish the salad; you want it to do it in a specific way. Maybe you want it to chop the tomatoes first, then the onions, and finally the lettuce. Or maybe you want it to avoid stepping on a specific spot on the floor while carrying the eggs.

This is the core problem the paper addresses: How do we teach AI not just what to do, but how to do it?

Here is a simple breakdown of the paper's ideas, using some everyday analogies.

1. The Problem: The "Flat" Instruction

Traditionally, when we program AI, we give it a "flat" set of instructions. Think of this like a single, giant to-do list where every item is just "Do X."

The Issue: If you tell a robot, "Make a salad, but chop tomatoes before onions," a flat instruction system struggles. It sees the whole task as one big blob. It doesn't understand that "chopping tomatoes" is a distinct step that happens before "chopping onions."
The Result: The robot might chop the onions first, then the tomatoes, and then get confused. It might also walk through a "danger zone" (like a hot stove) while carrying a fragile egg because it doesn't understand the context of carrying the egg.

The paper argues that for complex, long tasks, flat instructions are like trying to describe a symphony by just listing the notes without mentioning the rhythm or the sections. It misses the structure.

2. The Solution: The "Hierarchical" Manager

The authors propose a new way of thinking called Hierarchical Reward Design.

Imagine a construction site.

The Flat Approach: You tell every worker, "Build a house," and you give them a reward for every brick they lay. They don't know if they are building the foundation, the walls, or the roof. They just lay bricks randomly until the house is done.
The Hierarchical Approach: You have a Site Manager (High-Level) and Crews (Low-Level).
- The Manager decides the sequence: "First, pour the foundation. Then, build the walls. Then, put on the roof."
- The Crews execute the details: "Okay, we are pouring the foundation. Let's mix the cement and pour it carefully."

In this new system, the AI has two brains working together:

The High-Level Brain: Decides which sub-task to do next (e.g., "Go get the eggs").
The Low-Level Brain: Decides how to do that specific sub-task (e.g., "Walk carefully so I don't drop the eggs").

3. The Magic Tool: L2HR (Language to Hierarchical Rewards)

The paper introduces a tool called L2HR. This is the bridge between human language and robot code.

How it works: You talk to the AI in plain English. You say, "Please pick up the apples and eggs, but make sure you alternate between them, and don't walk near the stool when carrying eggs."
The Translation: L2HR uses a Large Language Model (like the one powering this chat) to translate your English sentence into two separate "reward codes":
- Code for the Manager: "Give a bonus if the robot picks up an egg after an apple."
- Code for the Crew: "Give a penalty if the robot gets too close to the stool while holding an egg."

4. Why This Matters (The Analogy of the "GPS")

Think of the old way (Flat Rewards) as a GPS that only knows your final destination. It says, "Get to the store." It doesn't care if you drive through a school zone or take a dangerous shortcut, as long as you arrive.

The new way (Hierarchical Rewards) is like a smart GPS with a co-pilot.

The Co-pilot (High-Level) says, "We need to stop at the gas station before the store."
The Driver (Low-Level) says, "Okay, I'm stopping at the gas station, and I'm going to drive slowly because there are kids playing nearby."

5. The Results

The researchers tested this in three different "worlds":

Rescue World: A robot delivering supplies in a disaster zone.
iTHOR: A robot in a virtual kitchen picking up apples and eggs.
Kitchen: A robot making a salad with a specific order of chopping.

The findings were clear:

Flat Rewards: The robots often failed to follow the "how" instructions. They might deliver the supplies but in the wrong order, or they might drop the eggs near the stool.
Hierarchical Rewards (L2HR): The robots followed the instructions perfectly. They alternated items correctly, avoided danger zones specifically when carrying fragile items, and chopped vegetables in the exact order requested.

Summary

This paper is about upgrading how we talk to AI. Instead of giving AI a single, vague goal, we are teaching it to break big goals into smaller, manageable steps and to follow rules for how to perform those steps.

By using Language to Hierarchical Rewards (L2HR), we can simply tell an AI, "Do it this way," and it will understand the structure of the task, acting more like a responsible human employee and less like a confused machine just trying to get the job done.

1. Problem Statement

As AI agents are deployed in complex, long-horizon environments (e.g., homes, hospitals), ensuring they adhere not just to task completion but also to nuanced human behavioral specifications (safety, order, style) is critical.

Limitation of Flat Rewards: Traditional Reinforcement Learning (RL) relies on flat reward functions $r(s, a)$ $r (s, a)$ that map state-action pairs to scalar values. The authors argue that flat rewards are fundamentally limited in capturing complex specifications for long-horizon tasks, particularly those requiring:
- Subtask Sequencing: Decisions based on the history of executed subtasks (e.g., "deliver all food items before switching to medical supplies").
- Context-Dependent Execution: Different execution strategies depending on the current high-level goal (e.g., "avoid obstacles only when carrying fragile items").
The Gap: While Hierarchical Reinforcement Learning (HRL) decomposes tasks into subtasks (options), the design of rewards for these hierarchical agents remains largely unexplored. Existing methods often force hierarchical agents to use flat reward signals, losing the structural benefits of the hierarchy.

2. Methodology

The paper introduces a formal framework and a specific algorithm to solve this problem.

A. Hierarchical Reward Design (HRD)

The authors formally define the Hierarchical Reward Design (HRD) problem, extending the classical Reward Design Problem (RDP).

Structure: Instead of a single flat reward, HRD utilizes two distinct reward functions:
1. High-Level Reward ( $r_H$ ): A function $r_H(o_{prev}, s, o_{curr})$ that guides the selection of the next subtask (option). It encodes specifications regarding what subtasks to perform and in what order.
2. Low-Level Reward ( $r_L$ ): A function $r_L(s, o, a)$ that guides the execution of primitive actions within a specific subtask. It encodes specifications regarding how to execute a subtask.
Theoretical Expressivity: The paper proves that HRD is strictly more expressive than flat reward design.
- Property 1: High-level rewards can encode dependencies on previous options (e.g., "if last option was A, next must be B"), which flat rewards cannot do because they lack access to the option history in the state $s$ .
- Property 2: Low-level rewards can encode execution constraints specific to the current option (e.g., "move slowly only when option is 'carry fragile'"), which flat rewards cannot distinguish.
Compatibility: The framework is shown to be compatible with standard decision-making models, inducing a Markov Decision Process (MDP) at the low level and a Semi-Markov Decision Process (SMDP) at the high level.

B. Hierarchical Reward Design from Language (HRDL)

To make HRD practical, the authors define HRDL, a specific instantiation where the reward functions are generated from natural language specifications.

C. Language to Hierarchical Rewards (L2HR)

L2HR is the proposed algorithm to solve HRDL using Large Language Models (LLMs).

Input: Natural language descriptions of task objectives and behavioral preferences, along with environment code context (state/action spaces).
Prompting Strategy: The LLM is prompted with structured instructions to generate code for two separate functions:
1. get_high_level_pref: Takes (state, prev_option, current_option) as input.
2. get_low_level_pref: Takes (state, current_option, action) as input.
Two-Stage Training Procedure:
1. Stage 1 (Low-Level): The LLM generates $k$ candidate low-level reward functions. These are used to train low-level policies. Only policies that successfully complete subtasks (based on pseudo-rewards) are retained.
2. Stage 2 (High-Level): The LLM generates $k$ candidate high-level reward functions. These are used to train high-level policies, conditioned on the valid low-level policies from Stage 1.
Filtering: The system filters out reward candidates that result in syntax errors or fail to preserve task feasibility (i.e., the agent fails to complete the primary task).

3. Key Contributions

Formal Problem Definition: The first formal definition of the Hierarchical Reward Design (HRD) problem, establishing a theoretical foundation for designing rewards for hierarchical agents.
Theoretical Proof: Proof that hierarchical rewards are strictly more expressive than flat rewards for capturing long-horizon behavioral specifications (subtask ordering and context-dependent execution).
Algorithm (L2HR): A novel algorithm that leverages LLMs to synthesize hierarchical reward code directly from natural language, bridging the gap between human intent and complex agent training.
Empirical Validation: Comprehensive experiments demonstrating that L2HR outperforms flat reward baselines in both simulated environments and human evaluations.

4. Experimental Results

The authors evaluated L2HR against a "Flat" baseline (LLM generating a single flat reward decomposed into hierarchical components) and a "Task-only" baseline across three domains: Rescue World, iTHOR, and Kitchen.

Syntactic Correctness (Q1): Hierarchical reward generation had significantly lower code syntax error rates than flat generation. Flat rewards often required the LLM to hallucinate state variables (like "previous object type") that were not available in the flat state space, whereas hierarchical prompts explicitly provided the necessary context (options).
Task Feasibility (Q2): Agents trained with hierarchical rewards (Hier) maintained high task completion rates. Flat reward agents often failed tasks because the generated rewards interfered with the core objective or relied on spurious heuristics to infer missing history.
Alignment with Specifications (Q3):
- Rescue World: Hierarchical agents achieved 76.92% expert-level alignment on high-level persistence (delivering all items of one type before switching), compared to 12.50% for Flat. Flat rewards could not represent the "previous option" dependency.
- iTHOR: Hierarchical agents achieved 87.50% high-level alignment (alternating object types) and 75.00% low-level alignment (avoiding a stool only when handling eggs). Flat agents failed to learn the alternating pattern effectively.
- Kitchen: Hierarchical agents achieved 92.86% alignment with a specific chopping sequence, while Flat agents only achieved 10.00%.
Human Evaluation: In a user study with non-expert participants, agents trained with hierarchical rewards were consistently rated significantly higher (average score > 4.0) for alignment with behavioral specifications compared to flat reward agents.

5. Significance

This work represents a significant step toward Responsible AI by providing a mechanism to translate complex human values and operational rules into actionable reward signals for sophisticated agents.

Scalability: It addresses the scalability issue of reward design; as tasks become more complex, manual reward engineering becomes intractable. L2HR automates this for hierarchical structures.
Alignment: It moves beyond simple "task completion" metrics to ensure agents behave in ways that are safe, efficient, and consistent with human expectations (e.g., safety protocols, workflow preferences).
Future Direction: The paper lays the groundwork for future research in combining hierarchical structures with other alignment paradigms (like Inverse RL or Learning from Demonstration) and applying these methods to real-world robotics.

In summary, the paper argues that to align AI with human needs in complex environments, we must move beyond flat reward functions. By formalizing Hierarchical Reward Design and solving it via L2HR, the authors demonstrate that agents can be trained to not only finish tasks but to do so in a manner that strictly adheres to nuanced human specifications.