Zero-Shot Adaptation to Robot Structural Damage via Natural Language-Informed Kinodynamics Modeling

1. Problem Statement

High-performance autonomous mobile robots operating in unstructured, "in-the-wild" environments are prone to structural damage (e.g., punctured tires, broken suspensions, bent axles). Such damage fundamentally alters the robot's kinodynamics (the relationship between control inputs and resulting motion) by changing force distribution and physical constraints.

• The Challenge: Current state-of-the-art methods for handling kinodynamic shifts rely on online adaptation (e.g., fine-tuning models or meta-learning). These approaches require collecting new state-action data after the damage occurs.
• The Risk: Collecting data with a damaged robot using an incorrect kinodynamic model is inherently risky, potentially leading to catastrophic failures (e.g., rollovers, falling off cliffs) before the model can adapt.
• The Gap: Existing methods struggle to represent the heterogeneous nature of structural damage (ranging from scalar parameter changes to complex geometric deformations) and lack a mechanism for zero-shot adaptation (instant adaptation without new data).

2. Methodology: ZLIK Framework

The authors propose Zero-shot Language Informed Kinodynamics (ZLIK), a framework that uses natural language descriptions of damage to instantly adapt the kinodynamic model without online retraining.

A. Core Concept

ZLIK posits that while structural damages are physically diverse, they can be semantically described using natural language. The system learns to map these language descriptions into a latent space that is kinodynamically grounded, meaning the semantic representation directly correlates with the physical behavioral changes.

B. Architecture & Learning Process

The approach consists of two main stages:

1. Self-Supervised Alignment (Damage Representation):

   • Input: Pairs of natural language damage descriptions ($x_t$) and corresponding kinodynamic state-action trajectories ($\tau_{t,H}$).
   • Encoding:
     • Language is encoded using a pre-trained Sentence Transformer (EmbeddingGemma).
     • Trajectories are encoded using a Transformer Encoder.
   • Alignment Loss (VICReg): The model uses Variance-Invariance-Covariance Regularization (VICReg) to align the language embedding ($z_x$) and the trajectory embedding ($z_\tau$) in a shared latent space.
     • Invariance: Minimizes the distance between the language and trajectory embeddings (forcing them to predict each other).
     • Variance: Ensures embeddings maintain diversity (preventing collapse to a single point).
     • Covariance: De-correlates embedding dimensions to capture independent factors of variation.

2. Downstream Kinodynamic Modeling:

   • Model: A Transformer Encoder-Decoder architecture with Spatiotemporal Attention.
   • State Decomposition: Unlike monolithic models, ZLIK decomposes the 6-DoF state vector into dimension-specific tokens. This allows the attention mechanism to isolate how specific damages affect specific degrees of freedom (e.g., a broken suspension affecting vertical $z$ and pitch, but not yaw).
   • Conditioning: The frozen language embedding ($z_x$) is injected into the latent space of the encoder.
   • Prediction: The model predicts future states ($s_{t+1:t+P}$) based on the history of states/actions and the damage description, without updating weights during deployment.

3. Key Contributions

1. Novel Self-Supervised Approach: A method to align semantic language embeddings with physical kinodynamic responses, enabling the robot to "understand" damage descriptions in terms of motion.
2. Instant Zero-Shot Adaptation: A framework that allows a robot to switch its internal kinodynamic model immediately upon receiving a damage diagnosis, eliminating the need for dangerous online data collection.
3. Spatiotemporal Attention with State Decomposition: An architecture that explicitly handles the independence of state dimensions, crucial for modeling complex, non-linear damage effects.
4. Rigorous Validation: Extensive experiments demonstrating generalization across:
   • Different damage types (single and combined).
   • Sim-to-Real transfer (BeamNG.tech to physical robot).
   • Scale transfer (Full-size simulation to 1/10th scale physical robot).

4. Experimental Results

Experiments were conducted using the high-fidelity soft-body simulator BeamNG.tech and a physical 1/10th scale Verti-4-Wheeler (V4W).

• Performance vs. Baselines:

  • ZLIK significantly outperformed a non-adaptive "CleanTransformer" and the state-of-the-art adaptive baseline ANYCAR.
  • Zero-Shot Accuracy: ZLIK achieved up to an 81% reduction in kinodynamic error compared to baselines in zero-shot scenarios.
  • Comparison with Fine-Tuning: Even after fine-tuning ANYCAR with 10 minutes of new data (involving risky driving), ZLIK (zero-shot) still achieved lower prediction errors.

• Ablation Studies:

  • State Decomposition: Removing dimension-specific tokenization (Monolithic Transformer) resulted in significantly higher errors, proving the necessity of disentangling state dimensions.
  • Semantic Grounding: A confusion matrix experiment showed that prediction error spiked when the input language description did not match the actual physical damage, confirming the model relies on the semantic embedding for conditioning.

• Cross-Embodiment Generalization:

  • ZLIK successfully transferred from a full-size simulated vehicle to a 1/10th scale physical robot with different mass and dimensions.
  • It maintained superior performance over baselines in zero-shot mode on unseen physical damage (e.g., punctured tire, removed wheel), proving the semantic embeddings capture the fundamental nature of damage rather than overfitting to specific vehicle parameters.

5. Significance

• Safety: By eliminating the need for online data collection on damaged hardware, ZLIK prevents the "dangerous loop" where a robot must drive unsafely to learn how to drive safely.
• Scalability: The use of natural language allows the system to generalize to a vast array of potential damages without needing to pre-train on every specific failure mode.
• Robustness: The ability to bridge the Sim-to-Real and Full-to-Scale gaps suggests that semantic grounding provides a robust prior for physical dynamics that transcends specific hardware configurations.

In summary, ZLIK represents a paradigm shift from data-driven adaptation (requiring new interactions) to knowledge-driven adaptation (leveraging semantic understanding of physics), offering a safer and more efficient solution for deploying autonomous robots in harsh, unpredictable environments.