Technical Summary: TACO – Temporal Consensus Optimization for Continual Neural Mapping

1. Problem Statement

Neural implicit mapping has become a powerful paradigm for robotic navigation and scene understanding, offering compact, expressive, and high-fidelity scene models. However, existing online neural mapping systems face significant challenges when deployed in real-world, dynamic environments under strict memory and computation constraints.

Current state-of-the-art methods generally fall into two categories, both of which struggle with continual adaptation to changing scenes:

• Replay-based methods (e.g., Co-SLAM): These preserve consistency by storing and replaying historical observations. While effective in static scenes, they fail in dynamic environments where past observations conflict with new data, leading to persistent "ghost" artifacts (reconstructing objects in both old and new locations).
• Regularization-based methods (e.g., MAS, EWC): These prevent forgetting by penalizing changes to parameters deemed important for past data. However, in dynamic scenes, this often results in over-constrained optimization, preventing the map from adapting to genuine geometric changes and causing distorted or fragmented geometry.

The core problem is achieving a balance between retaining reliable past knowledge and adapting to new observations without storing or replaying historical data.

2. Methodology: TACO

The authors propose TACO (TemporAl Consensus Optimization), a replay-free framework that reformulates continual neural mapping as a temporal consensus optimization problem.

Core Concept

Instead of treating past data as raw observations to be replayed, TACO treats past model snapshots (frozen parameters from previous time steps) as "temporal neighbors." The current model is updated by enforcing a weighted consensus with these historical snapshots. Intuitively, the current map consults its own past knowledge, aligning with reliable historical regions while remaining free to revise outdated or conflicting regions.

Technical Formulation

The method builds upon the Co-SLAM framework (a real-time neural implicit mapper using multi-resolution hash grids and decoders for geometry and color) and utilizes the Method of Multipliers for constrained optimization.

1. Consensus Target: The optimization seeks to minimize the reconstruction loss on new observations ($R_t$) subject to a consensus constraint with the previous model state ($\Theta_{t-1}$). The constraint is defined as a parameter-wise weighted average:
   $$z_{t,t-1}(\Theta_t) = (W_t + W_{t-1})^{-1} (W_t \Theta_t + W_{t-1} \Theta_{t-1})$$
   Here, $W_t$ and $W_{t-1}$ are importance-aware weight matrices.

2. Importance-Aware Weighting: Not all parameters are equally reliable. TACO introduces a mechanism to dynamically balance stability and adaptation:

   • Sensitivity Estimation: Parameter importance is estimated by accumulating the sensitivity of the model's rendered outputs (color and depth) to parameter perturbations over iterations. This is computed via a proxy objective ($L_{proxy}$) without requiring ground-truth supervision.
   • Weight Construction: Parameters with high accumulated sensitivity (indicating they consistently influence reliable geometry) are assigned higher weights, enforcing stronger consensus. Parameters with low sensitivity (indicating noise or outdated regions) are assigned lower weights, allowing them to adapt freely to new observations.
   • Normalization: A global scaling factor aligns the average magnitude of weights with the penalty parameter $\rho$ to ensure numerical stability.
   • Masking: A thresholding mechanism suppresses consensus constraints on parameters with extremely low importance (e.g., in early training stages or unobserved regions) to prevent unnecessary constraints.

3. Optimization Loop: The system iteratively updates the primal variables (model parameters) and dual variables (accumulated disagreement signals) using the augmented Lagrangian. This allows the map to converge toward a state that minimizes error on new data while respecting the weighted consensus of the past.

3. Key Contributions

The paper outlines three primary contributions:

1. TACO Framework: A novel, replay-free framework that reformulates continual neural mapping as a temporal consensus problem, utilizing historical model snapshots rather than raw data.
2. Importance-Aware Mechanism: A dynamic weighting scheme that quantifies parameter reliability based on output sensitivity. This enables the system to preserve stable, reliable geometry while selectively updating regions that conflict with new observations.
3. Empirical Validation: Extensive experiments in both simulated and real-world environments demonstrating that TACO robustly adapts to dynamic scene changes while avoiding the artifacts common in replay-based and regularization-based baselines.

4. Experimental Results

The authors evaluated TACO against strong baselines (Co-SLAM with/without replay, MAS, EWC, CNM, UNIKD) on static and dynamic benchmarks.

• Static Scenes (Replica, ScanNet, Gibson): TACO achieves performance comparable to Co-SLAM with full replay (the upper bound) and outperforms or matches the best continual learning baselines (like MAS). It successfully preserves global geometry without catastrophic forgetting, demonstrating scalability to larger, complex environments.
• Dynamic Simulated Scenes (Habitat): In scenarios with staged object motion (moving furniture, scattering items), TACO successfully updates the map to reflect new configurations.
  • vs. Replay (Co-SLAM): TACO avoids "ghost" artifacts where objects are reconstructed in both old and new positions.
  • vs. Regularization (MAS): TACO avoids mesh tearing and fragmentation caused by over-constraining updates.
  • Metrics: TACO consistently outperforms MAS in dynamic settings, showing fewer artifacts, higher completion ratios, and better F1 scores.
• Real-World Experiments (Turtlebot): Tests involving moving a stool and reshuffling multiple cubes confirmed that TACO can adapt to sequential, heterogeneous changes in a physical environment, producing coherent reconstructions where baselines fail.
• Memory Efficiency: TACO significantly reduces CPU memory usage (RSS) compared to full replay methods (reducing usage by >50% relative to Co-SLAM with replay), making it suitable for resource-constrained robotic deployment.

5. Significance and Claims

The paper claims that TACO provides a principled solution for scalable, long-term neural mapping in dynamic robotic environments.

• Replay-Free Adaptability: By shifting the consensus paradigm from the spatial domain (multi-robot) to the temporal domain (single robot over time), TACO enables continual learning without the memory overhead of storing historical data.
• Balanced Optimization: The importance-aware mechanism resolves the trade-off between stability and plasticity. It treats past representations as temporally weighted constraints rather than immutable supervision, allowing the system to distinguish between reliable history and outdated information.
• Robustness: The method effectively mitigates the specific failure modes of existing approaches: the ghosting of replay-based methods and the rigidity of regularization-based methods.

The authors conclude that temporal consensus is an effective mechanism for achieving robust adaptation to scene changes while maintaining geometric consistency, offering a viable path for deploying neural mapping systems in real-world, non-static environments.