🤖 AI

Action-Sufficient Goal Representations

This paper argues that value-based goal representations in hierarchical offline goal-conditioned reinforcement learning can fail to support optimal action prediction, and proposes "action sufficiency" as a necessary condition, demonstrating that actor-based representations satisfying this criterion empirically outperform traditional value-based approaches.

Original authors: Jinu Hyeon, Woobin Park, Hongjoon Ahn, Taesup Moon

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinu Hyeon, Woobin Park, Hongjoon Ahn, Taesup Moon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a giant, complex maze. You don't want to give the robot a step-by-step list of every single turn, because that would be overwhelming and impossible to memorize. Instead, you use a "hierarchical" approach: a smart "Manager" gives the robot a high-level goal, like "Go to the red door," and a "Worker" robot figures out the specific steps to get there. This is the world of Offline Goal-Conditioned Reinforcement Learning. It's a branch of artificial intelligence where robots learn from a massive library of past experiences (like a video archive) without needing to bump into walls in the real world to learn. The magic ingredient in this system is the "goal representation"—a compressed, shorthand code that the Manager sends to the Worker. Think of it like a text message: if the Manager sends "Red Door," the Worker needs to understand exactly what that means to move its legs correctly.

For a long time, scientists believed the best way to create this shorthand code was to focus entirely on "value." In robot terms, "value" is just a score that says, "How close are we to winning?" The idea was simple: if the code perfectly captures how close the robot is to the goal, the robot will naturally know what to do. It's like assuming that if a GPS tells you exactly how many miles are left to your destination, you automatically know which turn to take next. But what if that GPS is right about the distance, but completely wrong about the direction? What if it tells you "10 miles to go" for two different paths, one leading to a cliff and the other to a park? The robot would be confused, even though the "score" was accurate. This is the puzzle researchers have been trying to solve: does knowing the score guarantee you know the move?

This paper, titled "Action-Sufficient Goal Representations," dives into that exact question and flips the script. The authors, a team from Seoul National University and Trillion Labs, argue that the old way of thinking—focusing only on the "score" (value)—is actually a trap. They discovered that a robot can have a perfect understanding of how close it is to a goal, yet still fail miserably at choosing the right action. To fix this, they introduced a new concept called Action Sufficiency. Instead of just asking, "How good is this state?", they asked, "Does this code contain all the specific details needed to pick the right move?"

The researchers proved mathematically that knowing the "value" (the score) does not automatically mean you have the "action" (the move). They ran a clever experiment using a digital cube puzzle. They gave two robots the same perfect "score" information about where they were, but one robot used the old "value-only" code, while the other used a new "actor-based" code that was trained specifically to predict moves. The result was a shock: the robot with the "value-only" code stumbled and failed, while the robot with the "actor-based" code soared to success. It turns out that the old code was like a map that only showed distance but erased direction, while the new code kept the crucial directional clues.

The paper suggests that to build better robots, we need to stop training our goal codes just to be good at predicting scores. Instead, we should train them to be "action-sufficient"—meaning they must preserve the specific, tiny details that tell a robot whether to turn left, right, pick up a block, or put it down. By training the robot's "Worker" directly on the moves it needs to make, the system naturally learns a shorthand that works. In their tests, this new approach consistently beat the old methods, especially in tricky, high-difficulty tasks where the old "value-only" robots gave up entirely. The paper doesn't claim this is a magic bullet that solves every problem in the universe, but it does show that for robots to learn complex skills from past data, they need a map that tells them not just how far they are, but exactly how to get there.

Technical Summary: Action-Sufficient Goal Representations

1. Problem Statement

In offline Goal-Conditioned Reinforcement Learning (GCRL), particularly within hierarchical frameworks for long-horizon tasks, a critical design choice is the goal representation. This representation serves as the compressed interface between a high-level planner (which predicts subgoals) and a low-level controller (which executes actions).

Existing state-of-the-art methods, such as HIQL, derive goal representations primarily through value learning objectives. These approaches operate under the implicit assumption that features optimized for accurate value estimation inherently contain sufficient information for optimal action prediction.

This paper challenges that assumption. The authors demonstrate that value sufficiency (preserving information necessary to estimate the value function) does not guarantee action sufficiency (preserving information necessary to predict the optimal action). Specifically, value-centric representations may collapse distinct goals that require different optimal actions but share the same value, leading to a fundamental disconnect between accurate value estimation and successful control.

2. Methodology and Theoretical Framework

2.1. Definitions of Sufficiency

The authors formalize two distinct information-theoretic conditions for goal representations $Z = \phi(S, G)$ :

Value Sufficiency: A representation is value-sufficient if it preserves all information needed to predict the optimal value function $V(S, G)$ . Formally, the conditional mutual information $I(V; G | S, Z) = 0$ .
Action Sufficiency: A representation is action-sufficient if it preserves all information needed to predict the optimal action distribution $P(A | S, G)$ . Formally, $I(A; G | S, Z) = 0$ , implying conditional independence $A \perp G | (S, Z)$ .

2.2. Theoretical Analysis

The paper proves that value sufficiency does not imply action sufficiency.

The Conflict: In many environments, multiple distinct goals can share the same optimal value (e.g., goals equidistant from the current state but in opposite directions). A value-sufficient representation might compress these goals into a single scalar or embedding indistinguishable by value, yet they require opposite actions.
The Gap: The authors derive a decomposition of the action sufficiency gap ( $\Delta_A$ ) for value-sufficient representations:
$\Delta_A = I(A; G | S, V) - I(A; Z | S, V)$
This equation shows that even if a representation perfectly predicts $V$ (making the first term of the decomposition relevant), it fails to be action-sufficient unless it retains the specific information $I(A; Z | S, V)$ that disambiguates actions within the same value level set.

2.3. Proposed Solution: Actor-Based Representations

To address this, the authors propose learning actor-based goal representations ( $\phi_A$ ).

Mechanism: Instead of learning the representation solely via value estimation (e.g., Temporal Difference learning), the goal encoder is trained jointly with the low-level policy by minimizing the negative log-likelihood (NLL) of the optimal actions.
Theoretical Guarantee: The authors prove that minimizing the actor NLL loss is equivalent to minimizing the conditional KL risk between the optimal policy and the learned policy. Consequently, achieving near-optimal NLL guarantees that the learned representation is approximately action-sufficient ( $\Delta_A \leq \epsilon$ ).
Training vs. Deployment: During training, the low-level policy conditions on the full ground-truth goal $G$ to learn the encoder. At deployment, the encoder is detached, and the low-level policy conditions on the subgoal representation predicted by the high-level planner.

3. Experimental Results

3.1. Motivation and 1D Thought Experiment

The authors first demonstrate empirically that value-based representations fail in the OGBench cube tasks. Even when provided with oracle subgoals (perfect planning), low-level policies using value-based representations ( $\phi_V$ ) achieved significantly lower success rates than those using actor-based representations ( $\phi_A$ ). A 1D thought experiment confirmed that a representation preserving only distance (value-sufficient) fails to distinguish between left and right moves (action-insufficient), whereas a representation preserving direction succeeds.

3.2. Discrete Cube Environment

To perform exact information-theoretic analysis, the authors constructed a Discrete Cube environment (a 4x4 grid with pick-and-place mechanics).

Findings: They computed exact conditional mutual informations for various representations.
- Representations that were perfectly value-sufficient but collapsed action distinctions (e.g., "value_only" or "distances") resulted in high failure rates.
- Success rates correlated strongly with the action sufficiency gap ( $\Delta_A$ ) rather than the value sufficiency gap ( $\Delta_V$ ).
- Among value-sufficient representations, performance was governed by the within-level-set disambiguation term $I(A; Z | S, V)$ .

3.3. Offline GCRL Benchmarks (OGBench)

The authors integrated actor-based representations into the HIQL framework and evaluated them on the OGBench suite (including state-based and pixel-based tasks like cube-double, cube-triple, cube-quadruple, and scene).

Performance: Replacing value-based representations with actor-based ones yielded consistent and substantial improvements in success rates across all tasks.
High-Difficulty Tasks: The improvement was most pronounced in high-difficulty tasks (e.g., cube-quadruple) where value-based representations collapsed entirely (near-zero success).
Pixel-Based Observations: In visual tasks, where nuisance information is abundant, actor-based representations significantly outperformed value-based baselines, suggesting they are better at filtering noise while retaining action-relevant structure.
Comparison: The proposed method outperformed standard offline GCRL baselines (GCIQL, GCIVL) and the original HIQL with value representations.

4. Key Contributions

Formal Definition: Introduced the concept of action sufficiency as a necessary condition for goal representations in hierarchical control, distinct from value sufficiency.
Theoretical Proof: Proved that value sufficiency does not imply action sufficiency and provided a decomposition of the resulting control gap.
Methodological Proposal: Proposed a simple, principled method for learning actor-based goal representations via standard policy NLL training, which is theoretically guaranteed to be approximately action-sufficient.
Empirical Validation: Demonstrated through both exact information-theoretic analysis in a discrete domain and large-scale offline RL benchmarks that action-sufficient representations are critical for control success, particularly in complex, long-horizon tasks.

5. Significance and Limitations

The paper claims that its primary significance lies in providing a policy-oriented analytical framework for goal representations, correcting the implicit assumption that value-centric features are sufficient for control. By shifting the learning objective from value estimation to action prediction, the method ensures the representation retains the structural information necessary for the low-level policy to succeed.

Limitations noted by the authors:

Theory vs. Practice: The theoretical guarantees assume access to the true distribution, whereas offline learning relies on static datasets with potential coverage gaps.
Sufficiency vs. Predictability: While action sufficiency is necessary for the low-level policy, the representation must also be compact and predictable for the high-level planner. The paper notes a trade-off where raw goals are action-sufficient but hard to predict, and value representations are easy to predict but not action-sufficient. Future work is needed to balance these requirements.

The work is presented as a methodological advancement for hierarchical offline RL, specifically targeting robotic manipulation and complex decision-making tasks, without introducing capabilities beyond existing hierarchical RL paradigms.