Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning

This paper introduces orthogonal bottlenecks, a lightweight, architecture-agnostic mechanism that constrains reinforcement learning representations to low-dimensional subspaces via fixed orthonormal projections, demonstrating both theoretically and empirically that task-relevant value functions can be preserved and often improved upon with minimal dimensionality while stabilizing feature geometry.

The Gist

Imagine you are trying to teach a robot to play a video game or walk across a room. Usually, we give these robots "brains" (neural networks) that are massive and over-engineered, like using a supercomputer to solve a simple math problem. They have millions of connections, processing huge amounts of data, even though the actual task might only require a few simple rules.

This paper asks a simple question: Do these robots actually need such huge brains, or are they just carrying around a lot of unnecessary baggage?

The authors found that the "thoughts" (representations) a robot needs to solve a task are often much simpler and smaller than we think. They discovered a way to force the robot's brain to think in a tiny, efficient space without losing its ability to learn.

Here is the breakdown of their discovery using everyday analogies:

1. The Problem: The Over-Cluttered Desk

Imagine a robot's brain is like a giant, messy desk with thousands of drawers. When the robot tries to figure out what to do, it has to search through all these drawers. Even though the robot only needs three specific tools (a hammer, a screwdriver, and a wrench) to fix a toy, the desk is so big that it wastes time and energy searching through empty drawers.

In technical terms, deep learning agents use high-dimensional representations (huge "desks") even when the task is intrinsically simple.

2. The Solution: The "Orthogonal Bottleneck"

Note on Prior Work: Other researchers have tried to shrink robot brains before, but most of those methods still train the full huge brain first and only compress it afterwards. This paper does something different — it forces the robot to LEARN DIRECTLY IN THE TINY SPACE from the start, so the huge brain never has to be built and trained at all.

The authors propose a clever architectural trick they call an Orthogonal Bottleneck.

Think of this as placing a special, rigid funnel between the robot's eyes (the encoder that sees the world) and its brain (the part that decides what to do).

• The Funnel: This funnel is fixed; it doesn't move or change shape. It is designed perfectly (mathematically "orthogonal") so that it doesn't squish or distort the information passing through it.
• The Effect: It forces all the robot's thoughts to pass through a very narrow channel. If the robot's brain was a 1,000-dimensional room, this funnel shrinks it down to a 2-dimensional hallway.

Why "Orthogonal"?
Imagine trying to pour water through a funnel. If the funnel is crooked or lumpy, the water splashes, spills, or gets stuck. But if the funnel is perfectly smooth and straight (orthogonal), the water flows through cleanly without losing any volume or changing its shape. This ensures the robot doesn't lose important information just because the channel is narrow.

3. The Big Discovery: "Small is Enough"

The paper proves two main things:

• The Theory: If a task has a "true" complexity of, say, 5 dimensions (like needing 5 specific tools), then as long as your funnel is at least 5 units wide, the robot can still solve the task perfectly. It doesn't matter how big the original desk was; the robot can do everything it needs to do inside that small hallway.
• An Important Caveat: This "small is enough" guarantee only holds because the funnel is ORTHOGONAL. If the funnel were crooked or lumpy (non-orthogonal), the information would get squished and distorted on the way through, and the robot would no longer be able to actually learn the task in the small hallway — even if the hallway is technically wide enough to fit the task. The orthogonality from section 2 isn't a nice-to-have polish; it's what makes the whole theorem work.
• The Reality Check: They tested this on many different games and robot tasks (from simple balance beams to complex video games like Atari and robot walking simulations).
  • Result: In almost every case, they could shrink the robot's brain down to a tiny size (sometimes just 2 or 3 dimensions!) and the robot performed just as well as the giant-brained version.
  • The "Tipping Point": There is a specific "minimum size" for each task. If the funnel is too small (smaller than the task's true complexity), the robot fails. But as soon as the funnel gets just a little bit bigger than that minimum, the robot's performance snaps back to 100%.

4. Why This Matters: Stability and Clarity

The authors also noticed something interesting about how the robot thinks with this funnel.

• Without the funnel: The robot's internal "thoughts" can get messy. Some parts of the brain might get huge and loud, while others go silent. This is like a choir where one person is screaming and everyone else is whispering; it's unstable.
• With the funnel: The robot's thoughts stay balanced. Every part of the small hallway is used equally. This makes the learning process more stable and prevents the robot from "breaking" or forgetting things.

They also tried making the funnel learnable (teaching the robot to build its own funnel), but found that a fixed, pre-made funnel was actually more reliable. It's like giving the robot a pre-fabricated, perfect hallway rather than asking it to build its own while it's trying to walk.

Summary

The paper shows that deep learning agents are often carrying around massive, unnecessary brains. By inserting a simple, fixed, and mathematically perfect "funnel" that forces the agent to think in a tiny, low-dimensional space, we can:

1. Keep performance high: The robot learns just as well.
2. Stabilize learning: The robot's internal thoughts stay organized and balanced.
3. Reveal the truth: It proves that the "true" complexity of many tasks is surprisingly small, hidden inside the massive neural networks we usually build.

Essentially, the authors found a way to tell the robot: "You don't need a mansion to live in; a perfectly designed tiny apartment works just fine."

Technical Summary

Technical Summary: Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning

Problem Statement
Deep reinforcement learning (RL) agents typically employ highly over-parameterized neural networks to represent policies and value functions. However, growing evidence suggests that the intrinsic structure of task-relevant value and policy manifolds is often low-dimensional, even when the ambient state space or network capacity is high. This mismatch between network capacity and task complexity raises the question of whether standard deep RL architectures allocate representational capacity far beyond what is necessary. While the "manifold hypothesis" posits that high-dimensional data concentrates near low-dimensional manifolds, existing approaches to recovering this structure often rely on auxiliary objectives, contrastive losses, or generative modeling to discover these manifolds post-hoc.

Methodology
This work proposes a simple, architecture-level inductive bias to enforce low-dimensional structure without auxiliary objectives or changes to the underlying RL algorithm. The core mechanism is the insertion of a fixed orthonormal projection between the encoder and the downstream policy/value heads.

• Architecture: Given an encoder $\phi_\theta$ that maps states $s$ to high-dimensional features $z \in \mathbb{R}^D$, the method projects these features onto a fixed $k$-dimensional subspace using a matrix $B \in \mathbb{R}^{D \times k}$ where $B^\top B = I_k$. The compressed representation is $h = B^\top z \in \mathbb{R}^k$, which is then fed to the policy and value heads.
• Fixed vs. Learned: The projection matrix $B$ is initialized via QR decomposition of a Gaussian matrix and remains fixed throughout training. The authors contrast this with trainable projections to assess the stability of the representation.
• Theoretical Framework: The analysis relies on the linear realizability assumption, a standard concept in RL theory (Du et al., 2020; Weisz et al., 2023). This assumes the optimal value function $V^\star$ can be expressed as a linear map in the feature space: $V^\star(s) = \Theta^\star \phi(s)$, where $\Theta^\star$ has an intrinsic rank $r$.

Key Contributions

1. Theoretical Guarantees on Expressivity and Dynamics:
   The authors prove that under the linear realizability assumption, a fixed orthogonal bottleneck of dimension $k \geq r$ (where $r$ is the rank of the optimal value function) preserves the expressivity of the original feature space.

   • Representational Sufficiency: If $k \geq r$, there exist encoder and head parameters such that the network exactly realizes $V^\star$. The fixed bottleneck does not reduce the capacity to represent the optimal value function.
   • Optimization Equivalence: The gradient dynamics of training the encoder and head parameters with the fixed bottleneck are identical to training a direct $k$-dimensional parameterization, provided the initialization is equivalent. The orthogonality condition ($B^\top B = I_k$) ensures that the projection does not act as a preconditioner that distorts gradient updates, unlike non-orthogonal fixed projections which can lead to unstable scaling.

2. Empirical Validation of Low-Dimensional Compressibility:
   The paper empirically demonstrates that deep RL representations can be compressed into very low-dimensional orthogonal subspaces across diverse benchmarks (Classic Control, MinAtar, Atari, Brax MuJoCo, and Meta-World) and algorithms (DQN, PPO, PQN).

   • Recovery Threshold: Performance typically recovers to baseline levels once the bottleneck dimension $k$ exceeds a small, task-dependent threshold. Beyond this threshold, increasing $k$ yields diminishing returns.
   • Encoder Width Independence: In experiments on the Humanoid task, varying the encoder width $D$ while keeping $k$ fixed showed that performance is largely insensitive to encoder capacity once the bottleneck dimension is sufficient, suggesting the bottleneck dimension is the primary factor governing expressivity.

3. Analysis of Representation Geometry:

   • Stability: Fixed orthogonal bottlenecks stabilize feature norms and prevent the "explosion" of feature scales often observed with non-orthogonal fixed projections (e.g., random Gaussian).
   • Effective Rank: Fixed orthogonal projections maintain a high effective rank relative to their dimensionality, indicating uniform usage of the subspace. In contrast, trainable projections can suffer from rank collapse and instability, particularly in larger bottleneck dimensions.
   • Manifold Visualization: In small domains (e.g., Acrobot, Freeway), the authors visualize the bottleneck activations, revealing that representations concentrate on thin, low-dimensional manifolds with smooth value gradients, rather than filling the ambient space.

Results

• Small Domains: For Classic Control and MinAtar, a bottleneck of size $k=2$ (or even $k=1$ in some cases) is sufficient to match baseline performance. Visualizations confirm that value manifolds are effectively 1D or 2D.
• Large-Scale Benchmarks: In Atari and MuJoCo tasks, performance recovers once $k$ exceeds a modest threshold (e.g., $k=8$ for Humanoid, $k=128$ for Phoenix). The minimal sufficient dimension correlates with environment complexity rather than encoder width.
• Multi-Task Learning: In the Meta-World MT10 benchmark, a fixed orthogonal bottleneck ($k=24$) modestly improved performance over the baseline, suggesting that constraining agents to a shared low-dimensional subspace can mitigate negative transfer and representation interference.
• Trainable vs. Fixed: While trainable projections offered slight benefits in specific small-bottleneck regimes, they exhibited instability and performance collapse in other settings (e.g., Phoenix with large $k$), whereas fixed orthogonal projections remained robust across all tested configurations.

Significance and Claims
The paper claims that deep reinforcement learning representations are often amenable to faithful compression into low-dimensional orthogonal subspaces. The significance of this work lies in:

• Simplicity: It offers a lightweight, architecture-agnostic mechanism (a fixed linear layer) to shape representation geometry without modifying the RL algorithm or adding auxiliary losses.
• Theoretical-Practical Bridge: It provides a principled justification for constraining representations via fixed orthogonal subspaces, linking the empirical success of small bottlenecks to the theoretical concept of linear realizability. The fact that performance is preserved when $k$ exceeds the intrinsic rank serves as an empirical falsification test for the presence of low-rank linear structure in learned value representations.
• Stability: It highlights that orthogonality is crucial for stable training dynamics in constrained subspaces, distinguishing fixed orthogonal bottlenecks from other dimensionality reduction techniques that may introduce instability or rank collapse.
• Alignment with Recent Conjectures: These findings empirically support the recent conjecture of Tenedini et al. (2026), who argued that the intrinsic manifold dimensionality of representations is driven primarily by environment complexity rather than network size. While Tenedini et al. framed this conjecture for policy manifolds, our results extend this picture to value representations: the minimum bottleneck dimension required for successful learning depends on the task and environment, not on the size of the underlying network.

The authors conclude that these findings support a representation-space interpretation of the manifold hypothesis in RL and suggest that future work could explore connections to object-centric learning to align these geometric low-dimensional manifolds with semantically meaningful factors.