🤖 machine learning

Path-dependent Discrete Amortized Inference

This paper proposes "Path-dependent Discrete Amortized Inference," a method that enhances discrete sampling from unnormalized posteriors by replacing the standard Markovian assumption with a learnable latent dynamical system, thereby enabling policies to utilize full trajectory history to overcome state aliasing and improve convergence and exploration.

Original authors: Tiago da Silva, Esmeralda S. Whitammer, Salem Lahlou

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC0 1.0

Original authors: Tiago da Silva, Esmeralda S. Whitammer, Salem Lahlou

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to build complex structures, like a LEGO castle or a DNA strand, piece by piece. The robot has a "goal sheet" (a mathematical map) that tells it which finished structures are the most valuable. The challenge is that the robot doesn't just pick the best final castle; it has to make millions of tiny decisions along the way to get there. In the world of artificial intelligence, this is called "sampling from a distribution." For smooth, continuous things (like drawing a curve), computers have powerful tools to do this. But when the task involves building discrete, blocky things (like graphs, sentences, or chemical molecules), it gets messy. The space of possibilities is so huge and jagged that standard methods often get stuck, confused, or fail to find the best designs. This is where a newer method called "GFlowNets" comes in. Think of GFlowNets as a smart construction crew that learns to build these objects by treating the building process like a game, where every step is a move in a Markov Decision Process (MDP). In this game, the robot only looks at the current state of the build to decide the next move, ignoring the history of how it got there.

However, there is a catch. Just like a human builder might forget they took a wrong turn three steps ago and keep making the same mistake, a robot that only looks at the current state can get confused. This is called "state aliasing," where two very different building histories look exactly the same to the robot, causing it to make the wrong choice. The paper you are about to read tackles this specific problem. The authors, Tiago da Silva and colleagues, argue that the rule "only look at the current state" is too limiting. They propose a new way to teach these builders: give them a memory. Instead of just seeing the current LEGO tower, the robot should also remember the entire path it took to build it. By adding a "latent dynamical system"—a fancy way of saying a built-in memory that updates as the robot builds—they show that the robot can learn much faster and build better, more complex structures. They prove mathematically that this "path-dependent" approach can solve problems that the old "memory-less" approach simply cannot, and they show through experiments that it works better on standard tests.

The Problem: The Robot with Amnesia

Imagine you are playing a game where you have to build a tower of blocks. You start at the bottom, and at every step, you can add a block to the left, the right, or stop. Your goal is to build a tower that matches a specific, complex pattern of colors.

In the old way of doing this (called a Markovian approach), the robot making the tower only looks at the tower as it is right now. It doesn't remember if it added a red block first or a blue block first; it only sees the current shape. This works fine for simple towers. But imagine a tricky situation: you have two different ways to build a tower that look identical at step 10, but one of those paths leads to a beautiful masterpiece, and the other leads to a wobbly mess. Because the robot only sees the identical shape at step 10, it can't tell the difference. It's like having amnesia. In the paper, the authors call this state aliasing. The robot gets confused because two different histories look the same, so it can't learn the right strategy to build the masterpiece.

The authors show that this isn't just a small glitch; it's a fundamental limit. Even if you give the robot a super-smart brain (a deep neural network), if it's forced to only look at the current state, it literally cannot learn to solve certain complex puzzles. They proved this with math, showing that the "memory-less" robot is stuck in a box of possibilities, while a robot with memory has a much bigger box to play in.

The Solution: Giving the Robot a Diary

To fix this, the authors introduced a new method they call Path-Dependent Discrete Amortized Inference. Instead of just looking at the current tower, the robot now carries a diary (or a "latent dynamical system").

Every time the robot adds a block, it doesn't just update the tower; it also updates its diary. The diary records the entire journey of how the tower was built. When the robot has to decide what to do next, it looks at both the tower and its diary.

Think of it like a detective solving a mystery. A memory-less detective only looks at the crime scene right now. A path-dependent detective looks at the crime scene and the timeline of events that led to it. With the diary, the robot can tell the difference between the "masterpiece path" and the "wobbly path," even if the towers look the same at that moment. The robot can say, "Ah, I know this shape! But in my diary, I see I took a left turn three steps ago, so I know I need to add a blue block now, not a red one."

The authors didn't just guess this would work; they built a specific type of "diary" using a clever mathematical trick called a Self-Referential Weight Matrix (SRWM). This is a special kind of memory that updates itself as the robot builds, rotating and shifting its internal state to keep track of the unique history. It's like a diary that rewrites its own pages in a secret code every time you write a new entry, ensuring no two histories ever get mixed up.

What They Found: Faster and Smarter Builders

The team tested their new "path-dependent" robot against the old "memory-less" robot on several standard challenges, like building sets of numbers, designing DNA sequences, and navigating grid worlds.

Solving the Unsolvable: In some experiments, the memory-less robot completely failed to learn the correct pattern. It kept building the wrong things because it couldn't distinguish between different paths. The path-dependent robot, however, learned the pattern perfectly. The authors showed mathematically that for certain types of problems, the memory-less robot is impossible to train to get the right answer, while the path-dependent one can.
Speeding Up: Even when the memory-less robot could eventually learn the answer, it took a very long time. The path-dependent robot learned much faster. In one test, the memory-less robot needed about 100 times more training steps to figure out the difference between two similar states that the path-dependent robot figured out almost immediately.
Better Results: When they measured how close the robot's output was to the perfect target, the path-dependent robot was consistently closer. Whether they were generating sets of numbers, DNA sequences, or navigating a grid, the robot with the diary produced higher-quality results.

The Takeaway

The paper suggests that when we are teaching AI to build complex, step-by-step objects, forcing it to forget its past is a bad idea. By giving the AI a "memory" of its entire journey, we unlock a much higher level of intelligence. The authors proved that this isn't just a nice-to-have; it's a necessary upgrade to solve certain problems that were previously out of reach. They didn't just say "it might work"; they showed through rigorous math and computer simulations that the path-dependent approach is strictly more powerful and efficient than the traditional method.

So, the next time you see an AI trying to build something complex, remember: it's not just about what it sees right now. It's about remembering how it got there. And with a little bit of memory, it can build wonders that were previously impossible.

Technical Summary: Path-dependent Discrete Amortized Inference

Problem Statement

The paper addresses the challenge of sampling compositional and discrete objects (e.g., graphs, sequences, phylogenetic trees) from a given unnormalized posterior distribution. While recent advances in Generative Flow Networks (GFlowNets) have framed this as learning a Markov Decision Process (MDP) that constructs objects step-by-step, the authors identify critical limitations in the standard Markovian assumption. Specifically, the Markovian property—where the policy depends solely on the current state—can hinder signal propagation during training and lead to state aliasing. In state aliasing, distinct states become representationally indistinguishable to the agent, causing the learned sampler to fail in distinguishing optimal actions for different trajectories that converge to the same state. This issue is particularly acute in environments with sparse rewards or complex structural dependencies, where standard deep neural networks may lack the expressivity to represent the optimal policy.

Methodology

To overcome these limitations, the authors propose Path-dependent Discrete Amortized Inference. The core methodology involves "lifting" the original state space of the MDP by introducing a learnable, state-conditional latent dynamical system $\{W_t\}_{t \ge 1}$ .

1. Lifted Pointed DAG

Instead of a standard state graph $G = (S, X, E)$ , the method operates on a lifted pointed DAG $G^\star$ . The state space is extended from $S$ to $S^\star \subseteq S \times \Omega$ , where $\Omega$ is a latent space (typically $\mathbb{R}^d$ ).

Latent Dynamics: The latent variable $W_t$ evolves deterministically based on the current state $s_t$ and the previous latent state $W_{t-1}$ via a learnable vector field $\phi$ :
$W_t = W_{t-1} + \phi(s_t, W_{t-1})$
Policy Dependence: The forward policy $p_F$ becomes path-dependent, conditioned on the pair $(s_t, W_t)$ . This allows the policy to distinguish between different trajectories leading to the same state $s_t$ , effectively breaking the Markovian constraint of the original state space.

2. Architecture: Self-Referential Weight Matrix (SRWM)

To implement the latent dynamical system efficiently, the authors utilize a modified Self-Referential Weight Matrix (SRWM).

The update rule for the latent state (represented as a weight matrix $W_t$ ) follows a specific recurrence:
$q_t, k_t, v_t, \beta_t = W \psi(s_t) + b$
$\bar{v}_t = W_{t-1} \zeta(k_t)$
$W_t = W_{t-1} R + \sigma(\beta_t) \cdot (v_t - \bar{v}_t) \otimes \zeta(q_t)$
Key Mechanism: The inclusion of a fixed, ergodic rotation matrix $R$ is crucial. It rotates the eigenvectors of the weight matrix throughout the trajectory rollout. This mechanism is designed to enforce diversity in the policy's hidden representations, thereby mitigating state aliasing more effectively than standard recurrent architectures like LSTMs or GRUs.

3. Learning Objectives

The authors prove that existing learning algorithms for GFlowNets can be extended to this lifted setting. They demonstrate that the Trajectory Balance (TB), Subtrajectory Balance (SubTB), and Contrastive Balance (CB) conditions remain valid for the lifted DAG.

The learning process minimizes the expected log-squared difference between the left and right sides of these balance equations.
Crucially, the vector field $\phi$ and the policy network are jointly learned via stochastic gradient descent on a single loss function.
Theoretical results (Proposition 4.8) indicate that if the backward policy $p_B$ is Markovian (independent of $W_t$ ), the optimal forward policy $p_F$ will also be Markovian with respect to the unlifted state. However, when $p_B$ is path-dependent, $p_F$ can leverage the full history to achieve better expressivity.

Key Contributions

Path-Dependent Algorithm: The introduction of a general-purpose path-dependent algorithm for amortized inference in discrete and compositional spaces, explicitly addressing the expressivity limits of Markovian samplers.
Theoretical Expressivity Proofs:
- Linear Policies: The authors prove (Proposition 3.1) that for linear policies, the space of learnable distributions in a path-dependent setting is strictly larger than in a Markovian setting.
- GNN Policies: They extend this to Graph Neural Networks (Proposition 3.2), showing that path-dependent parameterizations can distinguish states that 1-WL GNNs cannot (e.g., distinguishing edges added at different times), thus strictly improving expressivity.
Extension of Learning Theory: The paper provides a formal proof (Proposition 4.5 and Corollaries 4.6, 4.7) that standard GFlowNet learning objectives (TB, SubTB, CB) are sufficient to ensure correct sampling in the lifted, path-dependent setting.
Empirical Validation: Extensive experiments on standard benchmarks (Set Generation, Sequence Design, Grid World, Ising Models, etc.) demonstrate that the path-dependent approach achieves faster convergence and better goodness-of-fit (measured by Total Variation distance and Flow Consistency Score) compared to similarly parameterized Markovian models.

Results

The empirical analysis highlights several consistent findings:

Expressivity: In the "LINES" environment and graph-structured tasks, Markovian samplers failed to learn non-uniform distributions or distinguish between structurally similar states, whereas path-dependent models achieved near-perfect fits.
Convergence: Path-dependent models consistently converged faster. For instance, in the sparse LINES environment, a Markovian ReLU network took roughly 100 times more iterations to separate policies for distinct states compared to the path-dependent counterpart.
Architecture Efficiency: The SRWM with a rotation matrix outperformed standard gated RNNs (LSTM, GRU) in terms of convergence speed and final performance, attributed to its ability to mitigate state aliasing via the rotation mechanism.
Robustness: The method showed superior performance across diverse tasks, including biological sequence design (DNA binding affinity), preference learning, and Ising model simulations.

Significance and Claims

The paper claims that lifting the state space with a latent dynamical system is a necessary strategy for discrete amortized sampling, analogous to how Hamiltonian Monte Carlo (HMC) augments continuous spaces with momentum variables to accelerate convergence. The authors argue that:

State Aliasing is a Fundamental Bottleneck: The inability of Markovian samplers to distinguish states based on trajectory history is a primary cause of poor performance in complex compositional spaces.
Path-Dependence is Provably Superior: Theoretical results confirm that path-dependent parameterizations strictly expand the hypothesis class of learnable distributions.
Practical Applicability: The proposed method is a drop-in replacement for standard GFlowNet training, requiring only the modification of the state representation and the use of specific learning objectives that remain valid in the lifted space.

The authors conclude that algorithms designed for partially observable MDPs (POMDPs) are well-suited for discrete amortized sampling. They note that while they focused on deterministic latent dynamics, the feasibility of stochastic dynamics remains an open question for future research. The work does not claim to solve all discrete sampling problems but offers a general-purpose framework to mitigate the specific limitations of current Markovian approaches.