Topological Necessities: Mechanism-Invariant Strategic Subgoals for Cross-Embodiment Goal-Conditioned Control

Technical Summary: Topological Necessities for Cross-Embodiment Goal-Conditioned Control

Problem Statement
Offline goal-conditioned reinforcement learning (GCRL) aims to learn task policies from static datasets recorded by other agents, often with different embodiments (e.g., a point-maze agent vs. a humanoid robot). Existing approaches typically define subgoals as implicit byproducts of value functions, latent actions, or policy outputs specific to the executor that generated the data. Consequently, these subgoals absorb executor-specific variations (noise), degrading transfer performance when the executor or embodiment changes. The paper posits that a task-independent structure exists: a route-conditioned order of unavoidable stages that every successful executor must traverse, regardless of its specific dynamics. The challenge is to recover this structure from offline data without relying on the downstream executor's parameters.

Methodology
The authors propose a framework that treats offline datasets as inducing a geometric-topological representation of the strategy space. The core object is a "data-induced carrier," a transport-weighted graph built from successful trajectories, equipped with a goal-relative geometry.

1. Carrier Construction:

   • A $k$-nearest neighbor (kNN) graph is constructed over successful trajectories ($D^+$).
   • A transport-weighted metric is applied, where dimensions are weighted by a "straightness" score (net displacement over total variation) to filter out nuisance motion (e.g., gait oscillations).
   • A rectification embedding ($\psi_\theta$) is learned to map raw observations to a space where local distances match graph edge lengths and norms match a goal-relative radial coordinate ($r$).

2. Topological Readout (The "Gate" Discovery):

   • Goal-Relative Geometry: A multi-source Dijkstra algorithm assigns each node a radial coordinate $r(u)$ representing the graph geodesic distance to the goal set.
   • Shell-Measure Profile: The carrier states are binned into shells of constant $r$. A cross-sectional measure (volume or entropy) is computed for each shell in the rectified space, yielding a 1D profile $\mu(r)$.
   • Persistent Homology (H0): A sublevel-set filtration is applied to $\mu(r)$. Persistent valleys (bottlenecks) are identified where the measure drops significantly.
   • Certification: Under specific sampling and shape-regularity assumptions, surviving valleys correspond to separating sets (Theorem T3). These are "topological necessities": narrow neck regions that every admissible path must cross. The framework provides a certificate (persistence gap) that these gates are structural, not artifacts.
   • Route Bifurcation (H1): For tasks with loops in free space, a multiscale cubical persistent homology readout (H1) detects bounded holes. This assigns a winding-number signature to route classes, allowing the planner to commit to a route without switching mid-execution.

3. Recursive Gate Hierarchy:

   • The certified gates are not direct low-level targets (which may be out-of-distribution for the executor). Instead, the framework recursively subdivides the segments between gates using the same shell-measure filtration.
   • This creates a hierarchy of "sub-necks," densifying the path into in-distribution waypoints that the low-level executor can track.

4. Deployment:

   • The certified gate chain serves as a strategic backbone. The planner monitors progress via the radial coordinate and re-anchors if stalls or drifts occur.
   • Crucially, the gate set is discovered once on source data and frozen for deployment on a different embodiment, requiring no retraining or adaptation.

Key Contributions

• Task-Level Subgoals: The definition of subgoals as "topological necessities"—an order of unavoidable stages derived from the support of successful trajectories rather than a specific policy. This structure is falsifiable and survives executor replacement.
• Certified Construction: A transport-weighted carrier and a minimal detector (persistent homology on shell measures) with provable enumerability, separation certificates, and mechanism invariance. Theorems T1–T6 provide finite-sample and finite-step guarantees.
• Dimension-Matched Decomposition: The framework decomposes task structure by homology dimension: H0 for bottleneck ordering (separating sets) and H1 for route bifurcation (winding numbers).
• Evidence Chain: The paper provides causal validation of subgoal emergence, mechanism invariance across four regimes, and successful zero-label transfer across embodiments.

Experimental Results
The method was evaluated on the OGBench suite (PointMaze, AntMaze, Kitchen) and D4RL FrankaKitchen.

• PointMaze: The method achieved 100% success ($100 \pm 0$) across all sizes (medium, large, giant), outperforming strong baselines (QRL, HIQL) by margins of +14 to +32.
• AntMaze:
  • Medium: Matched the strongest baseline (HIQL) at parity (+0.5).
  • Large: Surpassed HIQL by +3.4.
  • Giant: Surpassed the strongest baseline (CFHRL) by +22.9 (90.9 vs. 68.0).
• Kitchen: Outperformed HIQL by +15.8 (partial) and +12.6 (mixed). Notably, the unsupervised PH-discovered necks matched the performance of a supervised event-label upper bound (96.5 vs. 94.3) with zero labels.
• Cross-Embodiment Transfer (Frozen):
  • A gate set discovered on PointMaze data and frozen transferred to Humanoid and Ant executors without retraining.
  • On Humanoid, it achieved the highest aggregate success rate (96.1) among all methods, including a map-privileged direct reference (85.3).
  • The margin was most significant on multi-route tasks (+36.0 over the direct reference, $p=1.4 \times 10^{-5}$).
  • Gates discovered on Ant data also transferred to Humanoid (91.0% aggregate), demonstrating that the structure is independent of the source embodiment.
• Robustness: Layout perturbations that invalidated the data caused the frozen sequence to fail (7.3% success), but re-identifying the gates on the surviving data restored performance (92.0%), confirming the gates are a function of the data topology, not the map.

Significance and Claims
The paper claims that the "structure worth recovering" from offline data is not the executor's action patterns or value function, but the route-conditioned stage order inherent to the task's topology.

• Mechanism Invariance: The framework demonstrates that strategic bottleneck structure is stable across dynamically distinct embodiments (PointMaze $\to$ Humanoid) as long as the free-space geometry (maze layout) remains fixed.
• Zero-Label Sufficiency: The topological readout can replace supervised event labels for subgoal definition, achieving performance comparable to or exceeding supervised baselines.
• Task-Conditioned Transferability: The authors modestly claim "task-conditioned transferability" rather than "unrestricted invariance." The bottleneck structure remains stable when the maze geometry is held fixed, but the framework does not claim to handle arbitrary changes in the environment's topology without re-identification.
• Structural vs. Statistical: The paper argues that the detected bottlenecks are "semantic facts" of the task (e.g., contact-commitment instants in Kitchen) rather than statistical artifacts, validated by their causal role in task completion and their persistence under mechanism changes.

The work establishes that a "topological necessity" approach can extract executor-independent task knowledge from offline data, enabling robust cross-embodiment transfer without the need for adaptation or retraining.