⚡ electrical engineering

Data-Free Knowledge Distillation for LiDAR-Aided Beam Tracking in MmWave Systems

This paper proposes a data-free knowledge distillation framework for LiDAR-aided mmWave beam tracking that synthesizes LiDAR-like sequences via a generator aligned with teacher feature statistics, enabling a student model to be trained effectively on synthetic data without access to real samples.

Original authors: Abolfazl Zakeri, Nhan Thanh Nguyen, Ahmed Alkhateeb, Markku Juntti

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abolfazl Zakeri, Nhan Thanh Nguyen, Ahmed Alkhateeb, Markku Juntti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting the Future Without the Past

Imagine you are driving a car on a highway at night. To keep your connection to a cell tower strong (like keeping a video call clear), your car needs to constantly aim a "laser beam" of data at the tower. If the car turns or a building blocks the view, the signal breaks.

Usually, the car uses a high-tech sensor called LiDAR (which acts like a super-precise 3D camera) to see the road and predict where the tower will be in the next few seconds. This helps the car aim the beam perfectly before it even moves.

The Problem:
LiDAR data is huge. It's like trying to carry a library of 3D maps in your glovebox. Storing all this data to train a computer to be good at aiming is expensive, slow, and sometimes impossible because of privacy rules (you can't just upload everyone's driving data to a central server).

The Solution:
The authors of this paper created a system called Data-Free Knowledge Distillation (DF-KD). Think of it as a way to teach a student how to drive without ever showing them the actual road or the original driving manual.

How It Works: The Teacher, The Ghost, and The Student

The process happens in three main steps, involving three characters:

1. The Teacher (The Expert)

First, they train a super-smart computer model (the Teacher) using real LiDAR data. This Teacher is like a master driver who has memorized every twist and turn of the road. It knows exactly which way to point the beam.

Note: Once the Teacher is trained, the real LiDAR data is thrown away. The Teacher keeps the "knowledge" inside its brain, but we don't have the raw data anymore.

2. The Ghost Generator (The Imagination Machine)

This is the clever part. Since they can't use the real data to train a smaller, cheaper model, they need to make up fake data that looks like the real thing.

They use a Generator (a "Ghost") that starts with random noise (static on a TV screen).
The Ghost tries to create fake LiDAR images.
How does it know if it's doing a good job? It doesn't look at the images. Instead, it asks the Teacher: "If I show you this fake image, does it make your brain feel the same way as a real image would?"
The Teacher checks its own internal "feelings" (mathematical statistics) and tells the Ghost, "No, that feels wrong. Try again."
Eventually, the Ghost gets so good at making fake data that the Teacher can't tell the difference. The Ghost has successfully recreated the essence of the real data without ever seeing a single real photo.

3. The Student (The Lightweight Driver)

Now, they train a tiny, simple computer model (the Student) using only the Ghost's fake data.

The Student watches the Teacher predict the beam direction based on the Ghost's fake images.
The Student tries to copy the Teacher's answers perfectly.
Because the Ghost's data was made to trick the Teacher, the Student learns the same skills as the Teacher, but it never needed to see the original, heavy LiDAR data.

The Two Ways to Teach the Student

The paper tested two different ways for the Student to learn from the Teacher:

The "Soft" Way (KL Divergence): The Teacher says, "I'm 90% sure it's beam A, but maybe 10% beam B." The Student tries to match those exact percentages. This is like a teacher explaining the reasoning behind a guess.
The "Direct" Way (MSE Loss): The Teacher just gives a raw number score for each beam, and the Student tries to match those numbers exactly. The paper found this method works just as well as the "Soft" way but is simpler because it doesn't require tuning extra settings (like a "temperature" knob).

The Results: Small and Mighty

The researchers tested this on a dataset called DeepSense (which simulates cars driving in a city). Here is what they found:

The Teacher Matters: They tried different types of Teachers. A Teacher that used a special mix of "spatial" and "time" sensors (called CNN-GRU) was much better than a simple time-only sensor. It was like having a teacher who understands both the shape of the road and the speed of the car.
The Student Wins: The tiny Student model ended up being 23 times smaller (in terms of computer memory) than the original heavy models used in previous research.
Performance: Even though the Student was tiny and trained on "fake" data, it performed almost as well as models trained on real data. It could predict the correct beam direction for the current moment and even a few seconds into the future.

Why This Matters

This approach solves a major headache for 5G and 6G networks. It allows us to have super-smart, predictive beam tracking on small devices (like cars or phones) without needing to store massive amounts of sensitive driving data or wait for huge datasets to be collected.

In short: They taught a small, cheap computer to be a master driver by having it mimic a genius teacher, using a "hallucination machine" to create fake practice roads, so they never had to use the real roads again.

Technical Summary: Data-Free Knowledge Distillation for LiDAR-Aided Beam Tracking in MmWave Systems

Problem Statement
Millimeter-wave (mmWave) communication systems rely heavily on precise beam training to overcome high path loss and blockages. While multimodal sensing-aided communications, particularly using Light Detection and Ranging (LiDAR), have shown promise in reducing beam training overhead and predicting optimal beams for connected vehicles, existing machine learning (ML) approaches face significant deployment barriers. These approaches typically require large-scale training datasets, which are often unavailable due to storage constraints at the User Equipment (UE), privacy limitations at the Base Station (BS), or the high cost of data collection. Furthermore, complex models required for high accuracy are difficult to deploy on resource-constrained platforms. The paper addresses the challenge of training a lightweight beam tracking model without access to the original LiDAR training data after the initial teacher training phase.

Methodology
The authors propose a Data-Free Knowledge Distillation (DF-KD) framework based on knowledge inversion. The process operates in three distinct steps:

Teacher Training and Metadata Collection: A high-capacity teacher model is first trained on a real LiDAR dataset. During this phase, the teacher's internal feature statistics (specifically the mean and variance of the last hidden layer outputs) are collected to form a "metadata" set. No raw LiDAR data or ground-truth labels are retained for subsequent steps.
Synthetic Data Generation (Knowledge Inversion): A generator model ( $G$ ) is trained to synthesize LiDAR-like sequences from random noise. The generator is optimized using a metadata loss, which minimizes the discrepancy between the statistical distribution of the teacher's internal features when processing synthetic data versus the collected metadata. This ensures the synthetic data elicits similar internal representations in the teacher as real data would, without requiring access to the original samples.
Student Training: A lightweight student model is trained exclusively on the synthetic data generated by $G$ $G$ . The student learns to mimic the teacher's behavior using either:
- Kullback-Leibler (KL) Divergence: Aligning the student's output probability distribution with the teacher's softened logits.
- Mean Squared Error (MSE): A proposed alternative that directly minimizes the error between the raw logits of the teacher and the student, eliminating the need for temperature hyperparameter tuning.

Key Contributions

DF-KD Framework for Beam Tracking: The paper introduces a framework that eliminates the need for real LiDAR training data after the initial teacher training, addressing privacy and storage constraints.
CNN-GRU Teacher Architecture: The authors propose a teacher model combining Convolutional Neural Networks (CNN) for spatial feature extraction from LiDAR frames and Gated Recurrent Units (GRU) for temporal modeling. This architecture is shown to be more expressive than GRU-only alternatives.
MSE Loss for Distillation: The paper proposes using MSE loss between raw logits as a simpler alternative to standard KL divergence, requiring fewer hyperparameters while maintaining comparable performance.
Significant Complexity Reduction: The resulting student model achieves an approximate 23-fold reduction in parameters and a 38-fold reduction in FLOPs compared to the teacher, making it suitable for resource-constrained vehicular deployment.

Results
Simulations were conducted using the DeepSense 6G dataset (Scenario 8), representing a vehicle-to-infrastructure (V2I) scenario with vehicle speeds ranging from 3 to 18 km/h. Key findings include:

Generator Capacity: A generator with 128 hidden neurons provided the optimal balance between expressive capacity and generalization, outperforming smaller (32, 64) and larger (256) configurations.
Architecture Superiority: The proposed CNN-GRU teacher achieved the highest beam tracking accuracy (59.1% Top-1 current-slot accuracy) compared to GRU-only baselines. Crucially, the DF-KD student trained under this teacher outperformed students trained under GRU-only teachers, demonstrating that a stronger teacher directly translates to better student performance in the data-free setting.
Loss Function Comparison: The proposed MSE loss achieved performance comparable to the standard KL divergence loss but without the need for temperature tuning.
DF-KD vs. Standard KD: While standard KD (trained on real data) achieved higher accuracy (57.98% vs. 54.12% Top-1 current-slot), the DF-KD framework significantly outperformed a student trained without any distillation. The performance gap between DF-KD and standard KD narrowed at future beam prediction slots, validating DF-KD as a viable alternative when real data is unavailable.

Significance and Claims
The paper claims that the proposed DF-KD framework effectively addresses the computational and data-storage burdens of LiDAR-aided beam tracking. By synthesizing data that faithfully reflects the teacher's internal feature distribution, the method enables the deployment of highly compact models on resource-constrained platforms without compromising the ability to predict current and future beam indices. The authors emphasize that their approach differs from prior work (e.g., [10]) by introducing a more powerful CNN-GRU teacher, eliminating the need for real data post-training via knowledge inversion, and achieving a drastic reduction in model complexity. The work is presented as a step toward practical, privacy-preserving, and efficient sensing-aided communications, with future work suggested for Non-Line-of-Sight (NLoS) environments and broader mobility conditions.