⚡ electrical engineering

JEPA-MSAC: A Joint-Embedding Predictive Architecture for Multimodal Sensing-Assisted Communications

This paper proposes JEPA-MSAC, a self-supervised joint-embedding predictive architecture that leverages multimodal sensing and communication data to learn a unified, transferable latent representation, enabling efficient and accurate multi-task physical-layer predictions with minimal adaptation costs in dynamic wireless environments.

Original authors: Can Zheng, Jiguang He, Guofa Cai, Nannan Li, Mehdi Bennis, Henk Wymeersch, Merouane Debbah

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Can Zheng, Jiguang He, Guofa Cai, Nannan Li, Mehdi Bennis, Henk Wymeersch, Merouane Debbah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car in a busy city. To drive safely and efficiently, you don't just look at the road directly in front of you; you use your eyes to see traffic lights, your ears to hear sirens, and your GPS to know where you are. You combine all these senses to predict what will happen next: "That car is turning," "The light is about to change," or "There's a pedestrian stepping out."

This paper introduces a new "super-brain" for future wireless networks (like 6G) called JEPA-MSAC. It tries to give cell towers and self-driving cars that same kind of predictive, multi-sense intelligence.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Specialist" vs. The "Generalist"

Currently, most AI systems for wireless networks are like specialist doctors.

One AI is trained only to find your location.
Another AI is trained only to point the antenna (beam) in the right direction.
A third AI is trained only to guess signal strength.

If you want to add a new task (like predicting traffic jams), you have to build a whole new AI from scratch. This is slow, expensive, and inefficient. Also, these specialists often miss the "big picture" of how the environment is changing.

2. The Solution: The "World Model" (JEPA-MSAC)

The authors propose a Generalist AI that learns how the world works first, then uses that knowledge to do any specific job.

Think of this AI as a student learning to drive.

The Input (The Senses): Instead of just looking at the radio signal, the system looks at everything:
- Cameras (Vision): Seeing the street.
- Radar/LiDAR (Touch/Depth): Feeling the shape of cars and buildings.
- GPS (Location): Knowing where the car is.
- Radio Signals (Hearing): Listening to the current connection quality.
The "Token" Language: The system takes all these different types of data (images, numbers, 3D points) and translates them into a single, common language (called "tokens"). It's like translating English, French, and Chinese all into a universal code so the brain can process them together.

3. The Secret Sauce: "Playing Hide-and-Seek" (Self-Supervised Learning)

How does the AI learn without a teacher telling it the answers? It plays a game of Hide-and-Seek with itself.

The Game: The AI looks at a sequence of events (e.g., the last 8 seconds of driving data). It then hides (masks) a chunk of the future data.
The Guess: The AI has to guess what the hidden future looks like based only on the past.
The Lesson: It doesn't try to guess the exact pixel of a picture (which is too hard and useless). Instead, it guesses the underlying dynamics: "If the car was moving left and the building is blocking the view, the signal will likely drop."

By playing this game millions of times, the AI builds a mental map (a "latent space") of how the world evolves. It learns the rules of physics and movement rather than just memorizing data.

4. The Payoff: The "Frozen Brain" and "Lightweight Hats"

Once the AI has learned this mental map, something cool happens:

The Brain is Frozen: The main part of the AI (the "backbone") stops learning. It has already figured out how the world works.
Lightweight Hats: To do specific jobs, we just put a tiny, cheap "hat" (a small task head) on top of the frozen brain.
- Need to find the car? Put on the Location Hat.
- Need to point the antenna? Put on the Beam Hat.
- Need to check signal strength? Put on the RSSI Hat.

Because the brain already understands the world, these hats are very small, fast, and cheap to train. You don't need to retrain the whole brain for every new job.

5. Why is this better?

Efficiency: It's like having one master chef who knows how to cook everything, rather than hiring a separate chef for every single dish.
Prediction: Because it learned the "rules of the world," it can predict the future more accurately, even in tricky situations where the signal gets blocked by a building.
Adaptability: If you add a new sensor (like a new type of radar), you just teach the brain to understand that new input, and it instantly becomes better at all tasks.

Summary

JEPA-MSAC is a smart system that teaches a wireless network to understand its environment by combining cameras, radar, GPS, and radio signals. Instead of memorizing specific answers, it learns how the world moves and changes. Once it understands the world, it can instantly solve many different problems (like finding your location or connecting your phone) with very little extra effort. It's the difference between a robot that just follows a script and a robot that actually understands the world around it.

1. Problem Statement

Future 6G wireless systems require predictive and transferable representations to support multiple physical-layer (PHY) tasks (e.g., localization, beamforming, signal strength prediction) in dynamic environments. Current approaches face two main limitations:

Single-Task Supervision: Most existing deep learning methods are designed for specific tasks, leading to high adaptation costs when deploying new tasks.
Limited Temporal Modeling: While Large AI Models (LAMs) offer better generalization, they often focus on static representation learning or raw data reconstruction rather than explicitly modeling the temporal evolution of the physical environment.
Multimodal Fusion Challenges: Existing sensing-assisted methods often fuse multimodal data (vision, radar, LiDAR, GPS, RF) solely to optimize a single downstream target, failing to capture the shared, underlying scene dynamics that drive communication performance.

The core challenge is to learn a shared, predictive latent representation of the environment that captures cross-modal dependencies and temporal dynamics, enabling efficient adaptation to multiple PHY tasks with minimal retraining.

2. Methodology: JEPA-MSAC

The authors propose JEPA-MSAC (Joint-Embedding Predictive Architecture for Multimodal Sensing-Assisted Communications), a self-supervised framework that learns a predictive latent space without reconstructing raw inputs.

A. System Model & Data

The framework operates in a Vehicle-to-Infrastructure (V2I) mmWave scenario. It ingests time-synchronized multimodal data:

Vision: RGB images from the Base Station (BS).
Radar: Raw FMCW radar data (converted to range-angle maps).
LiDAR: 3D point clouds (projected to depth images).
GPS: User Equipment (UE) coordinates.
RF: Historical beam-level Reference Signal Received Power (RSRP).

B. Architecture Components

Multimodal Tokenization:
- Heterogeneous inputs are preprocessed and mapped into a unified token space of dimension $D$ .
- Vision: EfficientNet extracts features.
- Radar/LiDAR: Lightweight 2D CNNs process spatial maps.
- State (GPS/RF): Linear projections map low-dimensional vectors to tokens.
- Positional Encoding: A factorized scheme is used to encode time steps, modality types, and intra-frame token positions separately, allowing the model to understand complex spatiotemporal relationships.
Pretraining (Temporal Block-Masked JEPA):
- Objective: Instead of reconstructing raw pixels (which is computationally expensive and less relevant for PHY tasks), the model predicts latent representations of missing future data.
- Mechanism: A "Temporal Block Masking" strategy masks contiguous segments of time for specific modalities.
- Process:
  - Context Encoder: Processes visible (unmasked) tokens.
  - Predictor: Takes the context and masked tokens (represented by a learnable mask token) to predict the latent state of the missing parts.
  - Target Encoder: A momentum-updated (EMA) teacher network processes the full sequence to generate the ground-truth latent targets.
- Loss: The model minimizes the distance between the predicted latent state and the target latent state using a Smooth $\ell_1$ loss.
Downstream Adaptation (Frozen Backbone):
- After pretraining, the backbone is frozen.
- Lightweight Task Heads are trained on top of the frozen predictive latent state ( $S_{Pred}$ $S_{P r e d}$ ) for three tasks:
  - Localization: Predicts future UE trajectory using a coarse-velocity estimate plus a residual correction layer.
  - Beam Prediction: Uses a classification head. Crucially, it employs Localization-Guided Cascading, fusing the predicted trajectory with the latent state to improve beam selection accuracy.
  - RSSI Prediction: Predicts future signal strength using a regression head, also leveraging the fused latent state.

3. Key Contributions

Predictive Latent State: Introduction of a JEPA-based framework specifically for wireless communications that learns a shared, predictive latent representation of environment dynamics rather than task-specific features or raw data.
Efficient Adaptation: A "freeze-and-adapt" paradigm where a single pretrained backbone serves multiple PHY tasks via lightweight heads, drastically reducing adaptation costs compared to training separate models.
Localization-Guided Cascading: A novel design where the output of the localization head (predicted trajectory) is injected into the beam and RSSI prediction heads, exploiting the geometric correlation between user position and signal quality.
Comprehensive Evaluation: Extensive experiments on the DeepSense 6G dataset demonstrating superior performance in accuracy, stability, and computational efficiency compared to state-of-the-art baselines.

4. Experimental Results

Experiments were conducted on the DeepSense 6G dataset (Scenario 32, urban street, 60 GHz mmWave).

Localization: JEPA-MSAC achieved centimeter-level accuracy and demonstrated slower error growth over long prediction horizons compared to AR, GRU, Transformer, and Kalman filter baselines.
Beam Prediction:
- Outperformed generic baselines and matched specialized models like M2BeamLLM in Top-1/Top-3 accuracy.
- Key Finding: JEPA-MSAC achieved the lowest L1-RSRP difference (power loss). Even when it did not select the exact optimal beam, it selected beams very close to the optimal, preserving signal power better than M2BeamLLM, which suffered from larger mismatches.
RSSI Prediction: Achieved competitive RMSE and MAE, showing stability across prediction horizons.
Ablation Studies:
- Masking: Temporal block masking (contiguous time segments) outperformed random or checkerboard masking, proving that learning temporal continuity is critical.
- Embedding Dimension: An intermediate dimension ( $D=128$ ) provided the best balance between latent space quality and task transferability.
- Training Paradigm: The proposed JEPA approach significantly outperformed reconstruction-based SSL and multi-task supervised learning, confirming that predicting latent dynamics is more effective than reconstructing raw signals for PHY tasks.
Efficiency: The frozen backbone approach resulted in significantly lower inference latency and parameter count compared to end-to-end supervised models or large LLM-based baselines (e.g., M2BeamLLM).

5. Significance

This work represents a paradigm shift in AI for wireless communications:

From Reconstruction to Prediction: It moves away from the standard "reconstruct the input" self-supervised learning (common in computer vision) toward "predict the future state," which is more aligned with the goals of communication systems (e.g., anticipating blockages or beam changes).
Unified World Model: It establishes a "wireless world model" that understands the joint evolution of geometry, motion, and signal propagation, enabling a single model to solve diverse tasks.
Scalability: By freezing the backbone and only training small task heads, the framework offers a scalable solution for 6G networks where new tasks may frequently emerge without the need for expensive retraining of the entire system.
Practical Impact: The ability to predict beam alignment and signal strength with high accuracy and low latency using multimodal sensing is critical for maintaining reliable mmWave links in high-mobility scenarios.