🤖 machine learning

Coupled Control and Wireless World Models for Resilient Remote Robotic Control

This paper proposes a resilient remote robotic control framework that integrates coupled control with wireless Joint Embedding Predictive Architecture (JEPA) world models to jointly forecast robot dynamics and channel conditions, thereby enabling adaptive communication scheduling and robust navigation under limited bandwidth and environmental disturbances.

Original authors: H. P. Madushanka, Sumudu Samarakoon, Mehdi Bennis

Published 2026-09-07

📖 6 min read🧠 Deep dive

CC0 1.0

Original authors: H. P. Madushanka, Sumudu Samarakoon, Mehdi Bennis

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot sent to explore a distant warehouse or a disaster zone, relying entirely on a wireless signal to receive instructions from a human operator miles away. For this robot to move safely, it must constantly send back a stream of high-definition images so the operator can see what the robot sees. However, sending these massive image files consumes a great deal of energy and clogs the wireless connection, much like trying to shout a long story across a windy field; if the wind picks up or the signal fades, the message gets lost, and the robot might crash or get stuck. The challenge for engineers is to keep the robot moving reliably even when the wireless connection is shaky, the battery is low, or the environment changes unexpectedly. To solve this, researchers are turning to a concept called a "world model," which is essentially a mental map the robot builds in its own computer. Instead of just reacting to what it sees right now, the robot learns to predict what will happen next and how the wireless signal will behave, allowing it to make smart decisions without needing to send a constant stream of data.

A team of researchers has developed a new system that puts this idea into practice by teaching a robot to understand both its physical surroundings and the invisible waves that carry its messages. They created a sophisticated framework where the robot uses two linked learning systems to build a shared understanding of the future. One system learns how the robot moves and what the world looks like based on camera images, while the other learns how the wireless signal changes over time. By combining these two streams of knowledge, the robot can predict when it is safe to stop sending images and when it must send them urgently. This approach allows the robot to skip unnecessary transmissions, saving energy and keeping the connection clear for the moments that truly matter. The researchers tested this system in a highly realistic virtual environment that mimics real-world physics, lighting, and wireless interference, finding that the robot could navigate complex paths with far fewer data transmissions than traditional methods while remaining stable even when the signal was disrupted.

The core of this innovation lies in how the robot processes information. Rather than simply reacting to every new image it captures, the robot uses a predictive architecture to imagine the next few seconds of its journey. It learns to recognize patterns in the visual data, such as the shape of a wall or the texture of the floor, and simultaneously learns to recognize patterns in the wireless signal, such as how the signal strength fluctuates as the robot moves past certain objects. When the robot predicts that the wireless channel will be clear and its own path is safe, it holds back on sending new images, relying instead on its internal prediction of where it should be. This "predictive scheduling" means the robot only breaks its silence to send an update when something unexpected happens or when the signal quality is good enough to handle a large file. This strategy drastically reduces the amount of data the robot needs to transmit, which in turn lowers the energy required to keep the connection alive.

To ensure this system works in the real world, the researchers built a detailed simulation that combined a robot physics engine with a wireless signal simulator. They created a digital twin of a robot navigating a closed course, complete with realistic lighting, obstacles, and a wireless network that behaves exactly like a real one, including signal reflections and interference. In this virtual testbed, they pitted their new predictive system against three older methods: a standard feedback controller that reacts instantly to errors without planning ahead, a learning-based system that tries to learn from trial and error but does not predict the future, and a predictive approach based on Vision Transformers (ViTs) that processes images using attention mechanisms. The results showed that the new system completed the navigation task faster and with greater stability. It finished the course in about 57 seconds, compared to nearly 78 seconds for the standard controller, and it used significantly less energy to transmit its data. Most importantly, when the researchers introduced sudden changes to the environment, such as shifting the wireless frequency or adding new obstacles that blocked the signal, the new system adapted quickly and kept moving, while the other systems struggled or failed entirely.

The researchers also discovered that the way the robot represents the wireless signal matters greatly. Instead of just using raw numbers from the signal, they tested converting the signal data into structured images that show how the signal changes over time and space. One method turned the signal history into a visual map of frequencies, while another used a mathematical technique to map the signal's shape into a topological image. They found that these structured representations allowed the robot to predict future signal conditions much more accurately than using raw data alone. This improved prediction meant the robot could schedule its transmissions with greater confidence, further reducing the need for constant communication. The system also included a safety mechanism that detects when the robot's internal prediction no longer matches what its camera actually sees, such as when a light changes or an object blocks the view. When this mismatch occurs, the system automatically adjusts its perception to realign with reality, allowing the robot to recover from the disturbance without needing to be retrained from scratch.

In the end, this work demonstrates that a robot can be both smarter and more efficient by learning to anticipate the future of both its body and its connection. By building a shared mental model of the physical world and the wireless network, the robot learns to communicate only when necessary, preserving its energy and maintaining control even in chaotic conditions. The experiments, conducted entirely within a synchronized simulation of a robot and its wireless environment, showed that this approach outperforms traditional methods in speed, energy efficiency, and resilience. While the system was tested in a virtual setting, the underlying principles suggest a path toward more autonomous robots that can operate reliably in the real world, where wireless signals are often unpredictable and resources are limited. The researchers plan to take these findings from the simulation to physical robots operating on real networks, aiming to create a digital twin that can continuously learn and adapt alongside its physical counterpart.

Technical Summary: Coupled Control and Wireless World Models for Resilient Remote Robotic Control

Problem Statement

Remote robotic systems operating over wireless networks face a critical trade-off between control reliability and communication efficiency. Continuously transmitting high-dimensional sensory data (e.g., camera images) to remote controllers creates significant communication overhead, latency, and energy consumption, which is unsustainable under limited bandwidth or unreliable connectivity. Furthermore, traditional control policies often fail to account for dynamic wireless channel conditions, packet losses, and environmental disturbances, leading to degraded performance or system failure. Existing world model (WM) approaches typically focus on physical environment dynamics but neglect the stochastic nature of wireless communication, while current communication-aware strategies often treat control and communication as independent problems rather than jointly optimizing them.

Methodology

The paper proposes a resilient, communication-aware remote robotic control framework that integrates Coupled Control Joint Embedding Predictive Architecture (C-JEPA) and Wireless Joint Embedding Predictive Architecture (W-JEPA). This approach utilizes a unified latent representation space to jointly model robot dynamics and wireless channel evolution.

Core Architecture

Coupled World Models:
- C-JEPA: Learns predictive latent representations of robot dynamics from visual observations using a Recurrent State Space Model (RSSM). It encodes visual inputs into deterministic and stochastic latent states to predict future robot states and rewards.
- W-JEPA: Learns the evolution of wireless communication conditions. It encodes wireless observations (Channel State Information or RF-derived representations) and conditions its predictions on the latent control states generated by the C-JEPA. This allows the system to forecast future channel quality based on predicted robot behavior.
Predictive Communication Scheduling:
- The framework employs a binary scheduling variable ( $\rho_t$ ) to decide whether to transmit an uplink observation.
- By jointly forecasting future robot states and wireless conditions, the controller can predict favorable channel states and omit transmissions when the latent dynamics are predictable, thereby reducing uplink traffic and energy consumption.
Structured RF Representations:
- To improve wireless prediction accuracy, the study evaluates three input representations for the W-JEPA: raw Channel State Information (CSI), spectrogram images (time-frequency representation), and Persistence Images (PIs) derived from Topological Data Analysis (TDA).
Adaptive Resilience Mechanism:
- The system detects discrepancies between predicted and observed latent representations ( $\Delta_t$ ) to identify environmental changes.
- Global Illumination Changes: The system selects optimal HSV color transformation parameters to align incoming observations with the learned latent space without retraining the policy.
- Local Visual Obstruction: A lightweight Variational Autoencoder (VAE) is adapted online using a subset of historical data to reconstruct corrupted visual inputs before latent encoding.

Implementation and Simulation

The framework is evaluated in a synchronized simulation environment integrating Gazebo (robot dynamics and perception), ROS (Robot Operating System), and Sionna (physics-based ray-tracing for wireless channel modeling).

Transfer Learning: To address the high computational cost of training directly in Gazebo, the C-JEPA is first pretrained in a lightweight Gym CarRacing environment and then fine-tuned in Gazebo using transfer learning.
Action Mapping: A discrete-to-continuous action mapping bridges the gap between the pretrained discrete action space and the continuous control commands required by the Gazebo robot.

Key Contributions

Realistic Synchronized Framework: Development of a unified robot-wireless simulation environment integrating Gazebo, ROS, and Sionna Ray Tracing, enabling joint evaluation of control and communication under realistic propagation conditions.
Extended C-JEPA/W-JEPA: Adaptation of the coupled JEPA framework from lightweight RL environments to realistic robotic simulators via an efficient transfer learning strategy, reducing training costs.
Robustness Evaluation: Comprehensive assessment of the framework under diverse wireless disturbances, including carrier frequency shifts, scatterer variations, access-point handovers, and intermittent uplink failures.
Adaptive Resilience: Proposal of a mechanism that detects perception mismatches and triggers online adaptation (via HSV tuning or VAE reconstruction) to recover from environmental changes without full policy retraining.
Structured RF Analysis: Systematic investigation of raw CSI versus structured representations (spectrograms and PIs) for predictive wireless world modeling, analyzing the trade-off between prediction accuracy and computational complexity.

Experimental Results

The proposed framework was evaluated against a Proportional Integral Derivative (PID) controller and a Deep Q-Network (DQN) baseline under various scenarios (E1–E8).

Performance Efficiency: In standard conditions, the proposed method achieved the shortest task completion time (57.1s) and highest cumulative reward (775.4) compared to DQN and PID.
Communication Efficiency: The framework significantly reduced uplink transmission energy (4.74 J vs. 9.21 J for DQN and 11.19 J for PID) and latency (0.48 ms uplink) by utilizing predictive scheduling.
Robustness to Wireless Perturbations: Under carrier frequency shifts (1.9 GHz and 2.6 GHz) and added scatterers, the proposed framework maintained stable navigation performance with minimal degradation. In contrast, the DQN baseline showed significant performance drops, while the PID controller remained relatively unaffected as its control policy does not rely on learned communication-aware representations.
Resilience to Connectivity Loss: The framework sustained stable performance for approximately 13 consecutive missing uplink transmissions, compared to only 8 for a Vision Transformer (ViT) baseline, demonstrating superior resilience to intermittent connectivity.
Structured Representations: Spectrogram and PI-based inputs consistently reduced wireless prediction errors compared to raw CSI, with PIs achieving the lowest error distribution despite higher generation costs.
Adaptive Recovery: The adaptive mechanism successfully restored navigation performance under global illumination changes and local visual obstructions, whereas non-adaptive baselines failed.

Significance and Claims

The paper claims that the proposed framework effectively balances navigation performance, communication latency, and energy efficiency in dynamic wireless environments. By jointly modeling control and wireless dynamics, the system enables predictive communication scheduling that reduces unnecessary transmissions while maintaining reliable control.

The authors highlight that the framework demonstrates resilience by adapting to both wireless perturbations and visual perception degradations without retraining the complete control policy. The integration of high-fidelity simulation tools (Gazebo and Sionna) provides a practical platform for evaluating communication-aware robotic systems before physical deployment.

The study acknowledges practical trade-offs, noting that while predictive modeling and structured representation generation introduce computational overhead, the execution times remain suitable for real-time operation on a single GPU. The authors conclude that while the current study is limited to single-robot simulation, the framework lays the groundwork for future extensions to multi-robot systems and physical validation on real-world hardware.