🤖 AI

WiFi2Cap: Semantic Action Captioning from Wi-Fi CSI via Limb-Level Semantic Alignment

The paper introduces WiFi2Cap, a privacy-preserving framework that generates fine-grained natural language captions from Wi-Fi CSI signals by leveraging a vision-language teacher-student alignment with mirror-consistency loss to resolve directional ambiguities, validated on a newly proposed synchronized CSI-RGB-sentence dataset.

Original authors: Tzu-Ti Wei, Chu-Yu Huang, Yu-Chee Tseng, Jen-Jee Chen

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tzu-Ti Wei, Chu-Yu Huang, Yu-Chee Tseng, Jen-Jee Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a room, but you want to know what someone is doing without ever turning on a camera or seeing them. You want to know if they are waving their left hand or their right hand, or if they are jogging forward or backward.

This is the challenge the researchers at National Yang Ming Chiao Tung University tackled with a new system called WiFi2Cap.

Here is the story of how they did it, explained simply.

The Problem: The "Silent" Signal

We all know Wi-Fi. It's the invisible waves that carry your internet. When you walk around, your body bumps into these waves, changing them slightly. This is called CSI (Channel State Information).

Scientists have figured out how to read these changes to guess if someone is standing, sitting, or walking. But until now, the "translation" from these radio waves to human language was very basic. It was like a robot saying, "Action: Walking." It couldn't say, "The person is jogging slowly to the left while facing the window."

Why is this hard?

The Language Gap: Radio waves are just numbers; human language is full of meaning. Bridging that gap is like trying to translate a song written in math into a poem.
The Mirror Trap: If someone waves their left hand, the radio waves look almost identical to when they wave their right hand (just mirrored). Old systems got confused and often guessed the wrong side.

The Solution: A Three-Act Play

The researchers built a three-stage "school" to teach the Wi-Fi system how to speak.

Act 1: The Visionary Teacher (The Video Expert)

First, they trained a "Teacher" AI using thousands of videos and their descriptions (like a movie with subtitles). This Teacher is an expert at understanding that a video of a person waving their right hand matches the sentence "waving right hand."

The Analogy: Think of this Teacher as a film critic who has seen every movie ever made and knows exactly how to describe the action.

Act 2: The CSI Student (The Wi-Fi Learner)

Next, they introduced the "Student." This is the system that only sees Wi-Fi signals.

The Lesson: The Student doesn't have eyes, so it can't learn from videos directly. Instead, it watches the Teacher. The Teacher says, "This video looks like this sentence." The Student then tries to make its Wi-Fi signal look like that same video in the Teacher's mind.
The Mirror Fix: Here is the magic trick. The researchers realized the Student kept mixing up left and right. So, they added a special rule called Mirror-Consistency Loss.
- The Analogy: Imagine the Student is learning to dance. If the Teacher says, "Step with your left foot," and the Student steps with the right, the Teacher doesn't just say "Wrong." The Teacher holds up a mirror, shows the Student the reflection, and says, "See? In the mirror, that's your right foot. You need to feel the difference." This forces the Wi-Fi system to learn the specific "feeling" of a left vs. right movement.

Act 3: The Storyteller (The Writer)

Finally, once the Student understands the Wi-Fi signal well enough to match the Teacher's visual concepts, it passes that understanding to a "Writer" (a language model like GPT-2).

The Analogy: The Wi-Fi signal is now a set of notes. The Writer takes those notes and turns them into a fluent sentence: "A person raises their left hand and waves."

The New Dictionary: The WiFi2Cap Dataset

To teach this system, the researchers couldn't just use old data. They created a brand new library called the WiFi2Cap Dataset.

They set up a room with Wi-Fi antennas and a camera.
People performed 100 different actions (like squatting, jogging, waving).
They recorded the Wi-Fi signal, the video, and a human-written sentence for every single action.
This is the "textbook" the Student used to learn.

The Results: From "Walking" to "Jogging Left"

When they tested the system, the results were amazing.

Old Way: Just guessed the action type (e.g., "Walking").
WiFi2Cap: Generated detailed sentences like "The person is jogging slowly to the left."

In fact, the system was so good at telling left from right that it outperformed all previous methods by a huge margin. It proved that you can understand human activity in a private room (like a bedroom or hospital) without ever using a camera, preserving privacy while still getting the details right.

The Big Picture

WiFi2Cap is like giving Wi-Fi a voice and a sense of direction. It takes the invisible, messy radio waves bouncing off a person and turns them into a clear, private story about what that person is doing, solving the tricky problem of "left vs. right" along the way. It's a big step toward smart homes that understand us without ever watching us.

1. Problem Statement

The paper addresses the challenge of semantic action captioning using Wi-Fi Channel State Information (CSI). While Wi-Fi sensing is privacy-preserving and robust to occlusion/lighting compared to cameras, existing systems are limited to low-level outputs (e.g., joint coordinates, segmentation masks) or predefined action classification.

The authors identify two primary barriers to generating free-form natural language descriptions from Wi-Fi signals:

Semantic Gap: There is a significant disconnect between raw radio signals (CSI) and natural language, exacerbated by a lack of large-scale paired CSI-text datasets.
Directional Ambiguity: Wi-Fi signals often suffer from "mirror ambiguity," making it difficult to distinguish between left/right limb movements or mirrored actions (e.g., waving the left hand vs. the right hand), which is critical for accurate semantic descriptions.

2. Methodology: The WiFi2Cap Framework

The authors propose WiFi2Cap, a three-stage framework designed to bridge the modality gap between Wi-Fi signals and language while enforcing direction-sensitive alignment.

Stage 1: Vision-Language Teacher

Goal: Learn a robust, shared semantic space between video and text.
Mechanism: Uses a frozen CLIP model (Visual and Text encoders) as a teacher.
Training:
- Contrastive Learning: Aligns video frames and their corresponding text captions using InfoNCE loss.
- Mirror-Consistency Loss: A novel objective where the teacher is trained to distinguish between an action and its horizontal mirror (e.g., "raising left hand" vs. "raising right hand"). This is achieved by flipping video frames and swapping directional words in captions, enforcing a margin where the correct pair scores higher than the mirrored pair.

Stage 2: CSI-Text Student

Goal: Align Wi-Fi CSI features to the semantic space established by the teacher.
Architecture:
- CSI Encoder: Processes amplitude and phase sequences from multiple receivers using dual ResNet-18 backbones. A gating module fuses these features to create a CSI embedding.
- Alignment: The student is trained in two sub-steps:
  1. Vision-CSI Alignment: The CSI embedding is aligned to the teacher's visual embeddings (distillation).
  2. CSI-Text Alignment: The CSI embedding is aligned directly to the teacher's text embeddings using contrastive learning.
Mirror-Consistency: The student also applies the Mirror-Consistency Loss to ensure the CSI embedding correctly distinguishes between left/right limb actions, preventing the model from learning mirrored ambiguities.

Stage 3: Prefix-Guided Language Generation

Goal: Generate natural language captions from the aligned CSI embeddings.
Mechanism: Uses a frozen autoregressive language model (e.g., GPT-2, Qwen).
Technique: Prefix Tuning. Instead of fine-tuning the entire LLM, a lightweight MLP maps the CSI embedding into learnable "key/value" prefixes injected into every layer of the Transformer attention mechanism. This conditions the frozen LLM to generate text specific to the Wi-Fi input.

3. Key Contributions

WiFi2Cap Framework: The first end-to-end system to generate fine-grained, free-form action captions directly from Wi-Fi CSI, utilizing visual knowledge transfer and contrastive learning.
Mirror-Consistency Loss: A novel regularization technique specifically designed to resolve left/right limb and mirrored-action ambiguities, a critical failure mode in cross-modal action understanding.
WiFi2Cap Dataset: A new synchronized benchmark containing 100 action classes with 5-second clips. Each sample includes:
- Wi-Fi CSI (1 Tx, 3 Rx).
- Synchronized RGB video.
- Sentence-level action descriptions (human-verified and LLM-refined).

4. Experimental Results

The authors evaluated the system on the new WiFi2Cap Dataset and transferred it to the public Person-in-WiFi 3D dataset.

Performance Metrics: The full WiFi2Cap pipeline significantly outperformed a baseline (CSI $\to$ $\to$ LM without teacher alignment) across all standard captioning metrics:
- BLEU-4: Improved from 14.85 to 51.78 (on WiFi2Cap).
- METEOR: Improved from 25.86 to 57.48.
- ROUGE-L: Improved from 32.38 to 64.32.
- CIDEr & SPICE: Showed consistent and substantial gains.
Transferability: The model generalized well to the Person-in-WiFi 3D dataset, improving BLEU-4 from 12.15 to 47.07, demonstrating that the learned alignment is robust across different capture setups.
Ablation Studies:
- Stages: Removing the teacher (Stage 1) or the prefix tuning (Stage 3) caused significant performance drops, confirming the necessity of the full pipeline.
- Mirror-Consistency: Removing this loss reduced BLEU-4 from 51.78 to 35.25, proving it is essential for directional accuracy.
- Language Models: Testing different decoders (GPT-2, phi-2, Qwen) showed that model capacity and pretraining data significantly impact the quality of the generated captions.

5. Significance

Privacy-Preserving Sensing: WiFi2Cap enables fine-grained semantic understanding of human activities in private indoor spaces (e.g., hospitals, bedrooms) without the privacy risks associated with camera-based surveillance.
Solving Directional Ambiguity: By explicitly addressing the left/right confusion through Mirror-Consistency, the paper solves a fundamental limitation in RF-based sensing that has previously hindered high-level semantic tasks.
Bridging Modalities: The work demonstrates that knowledge distillation from vision-language models can effectively transfer semantic understanding to the wireless domain, even with scarce paired data.

In conclusion, WiFi2Cap establishes a new state-of-the-art for privacy-friendly, fine-grained semantic sensing, moving beyond simple classification to rich, descriptive language generation from Wi-Fi signals.