🤖 machine learning

Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning

This paper investigates how different methods of incorporating action information into the state update function of recurrent neural networks affect reinforcement learning performance, offering empirical evaluations and discussing future design challenges.

Original authors: Matthew Schlegel, Volodymyr Tkachuk, Adam White, Martha White

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matthew Schlegel, Volodymyr Tkachuk, Adam White, Martha White

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a dark, confusing maze. The robot can't see the whole map; it only gets tiny, blurry glimpses of the walls right in front of it. To make good decisions, the robot needs a "memory" that remembers where it has been and what it just did. In the world of Artificial Intelligence, this memory system is called a Recurrent Neural Network (RNN).

This paper is like a mechanic's manual for that memory system. The authors asked a simple but crucial question: How should the robot's memory system "listen" to the robot's own actions?

When the robot moves (e.g., "Turn Left"), that action changes the world. The memory system needs to update itself based on that move. The paper tests different ways to feed that "Turn Left" information into the memory.

Here are the three main ways they tested, explained with analogies:

1. The "Additive" Approach (The Stack of Papers)

Imagine your memory is a stack of papers. When you take an action, you just add a new sticky note on top of the stack that says "I turned left."

How it works: The computer simply glues the action information next to the observation information and shoves it into the memory.
The Result: It works okay, but it's a bit clumsy. The memory gets cluttered, and it sometimes struggles to remember the exact sequence of events, especially if the maze is long.

2. The "Multiplicative" Approach (The Specialized Filter)

Now, imagine your memory isn't just a stack of papers, but a smart filter. When you take an action, the memory doesn't just add a note; it reshapes the entire memory based on that action.

How it works: If you "Turn Left," the memory system actively changes how it processes the past. It's like saying, "Because I turned left, the hallway I was in before must have looked different than if I had gone straight." It multiplies the action by the memory, creating a dynamic, shifting understanding of the past.
The Result: This was the winner in almost every test. The robot learned faster, made fewer mistakes, and could navigate much longer, trickier mazes than the "Additive" version. It was like the robot suddenly gained a sixth sense for how its actions changed the world.

3. The "No Action" Approach (The Amnesiac)

This is the control group. The robot tries to remember the maze but ignores what it just did.

The Result: As you might guess, this performed the worst. Without knowing what move it just made, the robot was often confused and got lost easily.

The "Secret Sauce": Why Multiplication Wins

The authors found that the "Multiplicative" method is superior because it treats the robot's actions as active ingredients rather than just extra data.

Analogy: Think of baking a cake.
- Additive: You put the flour, sugar, and eggs in a bowl and just stir them together. They sit next to each other.
- Multiplicative: You mix the ingredients so they chemically react. The flour changes because of the eggs; the sugar changes because of the heat. The result is a new substance (the cake) that is fundamentally different from the sum of its parts.
In the robot's brain, the "Multiplicative" method allows the memory to understand that Action A changes the meaning of Observation B in a way that Action C would not.

The Experiments (The Test Drives)

The authors tested these methods in several "mazes":

Ring World: A simple circular track. The Multiplicative robot learned the track perfectly and needed very little "training time" (truncation) to get it right. The Additive robot struggled to keep track of where it was.
TMaze: A long hallway with a T-junction at the end. The robot had to remember a clue from the start of the hall to know which way to turn at the end. The Multiplicative robot solved this easily. The Additive robot often forgot the clue.
Directional TMaze: A harder version where the robot's orientation matters. Here, the Additive robot failed completely, while the Multiplicative robot succeeded.
Lunar Lander: A complex video game-like environment where a robot must land on the moon. The Multiplicative robot learned to land much faster and more reliably than the others.

The Big Takeaway

The paper concludes that how you design the robot's memory matters more than we thought.

Many AI researchers have been using the "Additive" method (just gluing data together) because it's the standard way computers handle data. This paper shows that for robots learning to act in the real world, the "Multiplicative" method (reshaping memory based on action) is a much better fit. It's like upgrading from a simple notepad to a dynamic, thinking notebook that understands cause and effect.

In short: If you want a robot to learn from its mistakes and actions, don't just tell it what it did; teach its memory to react to what it did. The "Multiplicative" approach does exactly that.

Technical Summary: Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning

Problem Statement

Deploying reinforcement learning (RL) agents in real-world, partially observable environments requires the ability to build and maintain a state representation that serves as a sufficient statistic of the past. While Recurrent Neural Networks (RNNs) are a standard tool for modeling temporal dependencies, their application in RL involves critical design choices often overlooked in literature. Specifically, the paper addresses the strategy used to incorporate action information into the RNN's state update function. The authors note that while action plays a fundamental role in perception and state construction (citing Noë, 2004; Prinz, 1990; Clark, 2013), many existing RL implementations either omit action inputs or incorporate them via simple concatenation without rigorous analysis of the ramifications. The core problem is determining how the mechanism of action encoding (additive vs. multiplicative) impacts an agent's ability to predict value functions and learn control policies in Partially Observable Markov Decision Processes (POMDPs).

Methodology

The authors formalize the agent-environment interaction as a POMDP and evaluate various RNN architectures designed to update the hidden state $s_t$ based on the observation $o_t$ , previous action $a_{t-1}$ , and previous state $s_{t-1}$ . The study focuses on two primary categories of action incorporation:

Additive Encodings:
- Standard Additive: The action embedding is concatenated with the observation vector before being processed by the recurrent cell (e.g., $s_t = \sigma(W_x x_t + W_a a_{t-1} + b)$ ).
- Deep Additive: The action is passed through a feed-forward neural network ( $f_a$ ) to create an embedding before concatenation with the observation.
Multiplicative Encodings:
- Multiplicative (Second-Order): Inspired by second-order RNNs, this approach uses a tensor product to integrate the observation, state, and action. The update rule involves a weight tensor $W$ and $n$ -mode products: $s_t = \sigma(W \times_2 x_t \times_3 a_{t-1})$ .
- Factored Multiplicative: To mitigate the high parameter count of full-rank multiplicative updates, the authors employ a low-rank approximation using Canonical Polyadic (CP) decomposition. This factorizes the tensor into a sum of rank-1 components, reducing parameters while attempting to retain the benefits of multiplicative interaction.

The experiments utilize both standard RNNs and Gated Recurrent Units (GRUs). The authors employ off-policy semi-gradient TD(0) for prediction (using General Value Functions) and Q-learning with $\epsilon$ -greedy exploration for control. To improve sample efficiency, they incorporate Experience Replay (ER), storing hidden states within the replay buffer to initialize sequences during sampling.

Experimental Setup

The authors evaluate these architectures across several illustrative domains with varying observation types and dynamics:

Ring World: A cyclic environment where the agent must track distance from an active bit. Used to test prediction capabilities and sensitivity to truncation lengths.
TMaze: A classic corridor environment with a T-junction. Used to test control capabilities where action relevance is low (only East/West actions affect position significantly).
Directional TMaze: A modified TMaze where the agent's orientation is part of the state, making action critical for interpreting observations.
Image Directional TMaze: An extension using pixel-based observations (MNIST digits) to test performance with deep feature extractors.
Lunar Lander: A partially observable version of the OpenAI Gym environment to test performance in a continuous control setting with sensor noise.

Key Results

The empirical evaluation yields several distinct findings regarding the efficacy of action encodings:

Superiority of Multiplicative Updates: In domains where actions critically influence the underlying state dynamics (e.g., Ring World, Directional TMaze, Lunar Lander), the multiplicative update consistently outperforms additive and no-action variants. The multiplicative architecture achieves lower Root Mean Squared Value Error (RMSVE) in prediction tasks and higher success rates in control tasks, often requiring shorter truncation lengths ( $\tau$ ) to learn effectively.
Context-Dependent Performance: In the standard TMaze, where action relevance is limited to specific transitions, the multiplicative variant does not strictly outperform other methods; in some cases, standard GRUs without action inputs performed comparably or better. This suggests that the benefit of complex action encoding is tied to the specific dependency between actions and state transitions.
Parameter Efficiency: While full-rank multiplicative updates have significantly more parameters than additive ones, the factored (low-rank) multiplicative variants often achieve comparable performance with fewer parameters, though they generally underperform the full-rank multiplicative version.
State Representation Analysis: T-SNE visualizations of the learned hidden states reveal that multiplicative cells cluster hidden states according to the underlying environment state more effectively than additive cells. Additive cells tend to over-segregate states based on the previous action, whereas multiplicative cells maintain a more coherent representation of the environment state despite action variations.
Deep Additive Limitations: The "Deep Additive" variant, which learns an action embedding, generally performed similarly to or worse than the standard additive approach, suggesting that learning the embedding from scratch did not overcome the structural limitations of the additive update.

Significance and Claims

The paper claims that the choice of how to incorporate action into the state update function is a critical, often overlooked design decision that significantly impacts RL agent performance. The authors argue that:

Supervised Learning Assumptions May Not Transfer: Architectures and design choices successful in supervised learning (like simple concatenation) may not be optimal for the RL setting, where the agent must model its own agency on the data stream.
Multiplicative Interactions are Beneficial: For tasks requiring the agent to track how actions alter the state (common in partially observable control), multiplicative updates provide a structural inductive bias that facilitates better state construction and faster learning.
Need for Specialized RL Architectures: The results highlight a gap in current RL research, suggesting that "off-the-shelf" RNNs from supervised domains are insufficient. The paper calls for future work in developing recurrent cells specifically designed for the unique challenges of online, active data collection in RL, such as handling long temporal dependencies without relying solely on truncated backpropagation through time (BPTT).

The authors conclude that while multiplicative updates appear to be the "clear winner" in the tested domains, the optimal architecture is domain-dependent, and future systems may benefit from mechanisms that can learn or select the appropriate structural bias dynamically.