🤖 machine learning

Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes

This paper introduces the "stable quotient" as a minimal, exact Markov sufficient statistic for holonomy-cover decision processes, enabling a reinforcement learning framework that achieves optimal memory compression and perfect decision accuracy by tracking hidden modes through structured permutation dynamics.

Original authors: Zuyuan Zhang, Yongshan Chen, Mahdi Imani, Tian Lan

Published 2026-07-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zuyuan Zhang, Yongshan Chen, Mahdi Imani, Tian Lan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to play a game, but the robot has a very strange limitation: it can only see the surface of the board, not the hidden gears turning underneath. In the world of Artificial Intelligence, this is called a "Partially Observable" problem. The robot sees a light turn green, but it doesn't know why—maybe the traffic light changed, or maybe a hidden timer just finished. To make smart decisions, the robot needs to remember its history. But here's the tricky part: if the robot tries to remember everything that ever happened, its brain gets too full and it freezes. If it remembers too little, it gets confused and makes bad moves. Scientists have been trying to find the "Goldilocks" memory: the smallest, most efficient way to remember just enough to act perfectly, without carrying around useless baggage. This paper dives into a specific, structured type of game where the hidden gears follow strict, predictable rules, asking a simple question: What is the absolute smallest memory a robot needs to win?

The researchers, Zuyuan Zhang and his team, studied a special kind of game they call a "Holonomy-Cover Decision Process." Think of it like a maze where the walls you see (the visible part) are always the same, but the floor beneath you is made of invisible, rotating platforms. Every time you take a step, the visible wall might stay the same, but the hidden platform rotates you to a different spot. If you walk in a circle, you might end up back at the same wall, but on a different hidden platform. The problem is that two different paths can look identical to your eyes but lead to completely different rewards or dangers because of how those hidden platforms twisted and turned.

The paper's main discovery is a method to find the "minimal Markov sufficient statistic." In plain English, this is the smallest possible "cheat sheet" the robot needs. Instead of remembering the entire history of every step it took, the robot only needs to track its current "stable class." Imagine the hidden platforms are grouped into teams. The robot doesn't need to know exactly which specific platform it's on; it just needs to know which team it belongs to. The authors proved that if the robot knows its current team, it can predict the future perfectly, just as if it knew the entire history. They call this the "stable quotient." It's like realizing that even though the maze has millions of paths, there are only a few distinct "types" of endings, and knowing which type you are in is all that matters.

The paper also tackles a common misconception: that simply counting how many times you went left or right is enough to solve these puzzles. The authors show that this "counting" approach fails miserably when the hidden gears don't play nice with each other (a concept called "non-abelian"). It's like trying to solve a Rubik's Cube by just counting how many times you twisted the top layer; the order of the twists matters just as much as the number. If you twist top-then-right, you get a different result than right-then-top. The paper proves that any memory system that ignores this order will fail to find the best path.

To test their ideas, the team built a digital playground. In one experiment, they took a game with 216 different hidden states and compressed it down to just 25 "stable classes" without losing any ability to win. In another, more complex game involving non-ordered twists, their new method (called HMRL) achieved a perfect 100% success rate using only three memory states. In contrast, other methods that tried to remember the whole history or just count twists either failed or needed thousands of memory slots to get the same result.

The researchers also figured out how to teach the robot this cheat sheet from scratch. They showed that if the robot can occasionally "reset" and check its position (like a checkpoint in a video game), it can learn the hidden rules and the correct memory groups very quickly. They proved that once the robot learns these groups, it can use standard, proven AI techniques to master the game, just as if it were playing a simple, fully visible game. However, they also warned that without these "checkpoints," the robot might never figure out the hidden rules just by watching passively, because different hidden realities can look exactly the same from the outside.

In short, this paper provides a mathematical map for finding the smallest, most efficient memory for a specific type of complex, hidden-world game. It proves that by grouping hidden states into "stable classes" and respecting the order of events, an AI can be both incredibly smart and incredibly efficient, using a tiny fraction of the memory that other methods require. It's a step toward building AI agents that don't just guess their way through the dark, but carry the perfect, minimal flashlight to see exactly what they need to know.

Technical Summary: Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes

1. Problem Statement

Reinforcement learning (RL) fundamentally relies on the Markov property, where future rewards and transitions depend solely on the current state. In Partially Observable Markov Decision Processes (POMDPs), the current observation is often insufficient to predict the future, necessitating the maintenance of a history-dependent statistic (e.g., a belief state) to restore the Markov property. However, the minimal such statistic—the smallest recursively updateable representation that preserves the Markovian property—is generally unknown and difficult to characterize.

Existing approaches often focus on expressive or efficient history representations (e.g., RNNs, Transformers) or belief compression, but they do not guarantee that the resulting memory-augmented process is strictly Markov, nor do they identify the minimal sufficient statistic. This paper addresses the specific challenge of identifying the minimal Markov sufficient statistic for a structured class of POMDPs where hidden dynamics are governed by permutation-based transport.

2. Methodology

2.1 Holonomy-Cover Decision Processes (HCDPs)

The authors introduce Holonomy-Cover Decision Processes (HCDPs) as a structured family of POMDPs. In an HCDP:

The latent state consists of an observed component ( $o$ ) and an unobserved hidden mode ( $i$ ).
The visible dynamics are Markov: the transition probability of the observation $o'$ given $o$ and action $a$ is independent of the hidden mode.
Hidden Transport: Whenever a transition occurs from $o$ to $o'$ via action $a$ , the hidden mode is deterministically permuted by a fixed permutation $\sigma_{o,a,o'}$ .
Path Dependence: While the visible path is Markov, the accumulated hidden mode depends on the ordered composition of these permutations. Consequently, two histories ending at the same observation may induce different hidden modes, leading to different rewards or optimal decisions.

2.2 Stable Quotient Construction

The core methodological contribution is the construction of the stable quotient, which serves as the minimal exact observation-wise Markov state.

Partition Refinement: The authors define a partition family over the hidden modes for each observation. They introduce a monotone operator $T$ $T$ that refines these partitions based on two criteria:
1. Reward Consistency: Modes must yield identical immediate rewards for all actions.
2. Successor Consistency: For every feasible transition, the transported modes must land in the same successor partition class.
Stabilization: Starting from a reward-based partition, the operator $T$ is iterated. Theorem 3.5 proves that this process stabilizes in finite steps to a coarsest stable family $\Pi^\star$ .
Exactness: Theorem 3.8 establishes that the stable quotient is the coarsest exact observation-wise abstraction. Any other exact abstraction must refine this stable quotient.
Markovization: Theorem 3.9 proves that the pair of the current observation $o$ and the stable class $c = [i]_o$ forms an exact, finite, value-preserving Markov state. The transition dynamics and rewards on this quotient state space are well-defined and independent of the specific hidden representative within the class.

2.3 Holonomy Memory Reinforcement Learning (HMRL)

To utilize this theoretical result in practice, the authors propose HMRL, an algorithmic framework:

Representation: Memory is represented by the current stable class.
Update: The class is updated via ordered edge transports (permutations) as the agent traverses the observation graph.
Diagnostics and Recovery: Since the initial class is often unknown, HMRL employs a calibrate-then-restart strategy. Under a resettable diagnostic interface (allowing repeated probes of the latent state), the algorithm infers local class prototypes and edge transports.
Theoretical Guarantees: Theorem 4.6 shows that class inference error decays exponentially with diagnostic repetitions. Theorem 5.2 and Corollary 5.4 demonstrate that once the local class tracking is synchronized (via calibration), the learned process is isomorphic to the canonical quotient MDP, allowing standard finite-MDP RL guarantees to transfer to the recovered state.

2.4 Non-Abelian Memory Barrier

The paper analyzes the necessity of ordered memory. Theorem 6.4 and Corollary 6.5 prove that for non-abelian transport groups (where the order of operations matters), memory representations based solely on commutative loop counts (e.g., counting how many times a loop was traversed) are insufficient. Such "abelianized" memories merge distinct histories that require different optimal actions, leading to suboptimal performance. In contrast, ordered stable-class tracking avoids this loss with finite memory.

3. Key Contributions

Formulation and Identification: The paper formulates HCDPs and identifies the stable quotient as the minimal Markov sufficient statistic among exact observation-wise abstractions. It provides a matching memory characterization ( $\max_o |C_o|$ symbols) for exact known-class tracking.
Diagnostic Recovery: It separates passive non-identifiability from diagnostic recovery. Under resettable calibrated diagnostics, the method identifies classes and transports up to observation-wise relabeling, providing conditions under which finite-MDP RL guarantees transfer to the recovered state.
Non-Abelian Lower Bound: The authors prove a non-abelian lower bound for count-factorized memories, demonstrating that ordered stable-class tracking is necessary to avoid information loss in non-commutative transport settings.
Algorithmic Framework: The proposal of HMRL, which integrates structural quotient discovery with standard RL backbones via a calibration-and-restart protocol.

4. Experimental Results

The authors validate their theory using two controlled environments: ChainCover and LoopGuess.

ChainCover (Structural Exactness):
- An acyclic depth-6 construction with 216 raw latent states.
- The algorithm successfully compressed the state space to 25 exact quotient states without any reward, transition, or optimal-value inconsistency.
- The refinement process stabilized at round 7, matching analytical predictions.
LoopGuess (Non-Abelian Control):
- A non-abelian environment where visible loops generate the symmetric group $S_3$ .
- Performance: HMRL achieved 1.000 final success and perfect paired-order accuracy (distinguishing histories with identical loop counts but different orders).
- Memory Efficiency: HMRL used only 3 decision-time memory states, matching the performance of the "Quotient-Q oracle" (which has access to the true stable class).
- Baselines:
  - Count-Q (commutative loop counts) failed, remaining near chance level (0.34) on same-count, different-order histories.
  - History-Q (full loop word) achieved high accuracy but required over $10^3$ visited memory states, demonstrating inefficiency compared to the minimal quotient.
  - Raw-Q oracle (full latent state) achieved perfect success but required 48 states and showed slower learning.

5. Significance and Claims

The paper claims to provide a principled solution to the "minimal Markov sufficient statistic" problem for a specific, structured class of POMDPs. Its significance lies in:

Theoretical Rigor: It moves beyond heuristic approximations (like RNNs) to provide exact, finite-state abstractions that preserve the Markov property and optimal value functions.
Efficiency: It demonstrates that complex history dependencies can be compressed into a minimal set of memory symbols (the stable classes) without loss of optimality, provided the underlying dynamics follow the HCDP structure.
Order Sensitivity: It explicitly characterizes the failure of commutative summaries in non-abelian settings, justifying the need for ordered transport tracking.
Practical Reduction: It offers a concrete "calibrate-then-restart" reduction that allows standard finite-MDP RL algorithms to be applied to partially observable settings once the structural quotient is identified.

The authors remain modest, noting that their results apply to HCDPs and that extending quotient discovery to noisy, continuous, or purely passive-data settings (without resettable diagnostics) remains an open direction. The experiments are presented as falsifiable checks of the theory under known model assumptions rather than evidence that arbitrary POMDP benchmarks admit such exact finite quotients.