💻 computer science

Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification

This paper presents a comprehensive benchmark of 130 federated learning and knowledge distillation combinations for 3D point cloud classification, revealing that while knowledge distillation effectively compresses models, standard evaluation methods can misleadingly inflate performance by leaking private proxy labels, thus necessitating label-free distillation metrics to accurately reflect federated teacher quality.

Original authors: Aizierjiang Aiersilan

Published 2026-07-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aizierjiang Aiersilan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Two-Part Problem

Imagine you are trying to teach a robot how to recognize different 3D objects (like a chair, a car, or a human head) using a cloud of dots (a point cloud). You face two big hurdles:

The Privacy Wall: You can't gather all the data in one place because it belongs to different hospitals or companies who are not allowed to share it.
The Tiny Computer: Even if you train a super-smart robot, it's too heavy and slow to run on a small, portable device (like a handheld scanner).

This paper tests two solutions working together:

Federated Learning (FL): A way to train the robot by sending instructions back and forth without ever moving the private data.
Knowledge Distillation (KD): A way to shrink the "super-smart" robot into a "compact" robot that fits on the small device but still remembers what it learned.

The authors built a massive benchmark (a giant test track) to see how well these two methods work together on 3D data. They didn't just test one thing; they tested 13 different training strategies combined with 10 different shrinking strategies, creating 130 different combinations to see what works and what fails.

The Two Datasets: The "Gym" and the "Hospital"

To test their ideas, they used two very different training grounds:

ModelNet40 (The Gym): A standard dataset with 40 types of objects (chairs, planes, etc.). It's a good general test, but the data is messy and unevenly distributed.
Craniosynostosis Dataset (The Hospital): A real-world medical dataset of patient head shapes. This is the "high stakes" test. Here, privacy is critical because it involves real patients, and the devices need to be small enough to use at a doctor's bedside.

The Setup: They simulated a "worst-case scenario" where the data is split unevenly. Imagine five students (clients) trying to learn together. Student A only sees chairs, Student B only sees planes, and so on. They never see the full picture. This is called Non-IID (data is not independent and identically distributed).

Finding #1: The "Broken Team" Problem (Federated Learning)

When the students try to learn together without sharing their private notes, the team performance drops significantly.

The Result: Even the best team strategy only reached about 76% accuracy on the Gym dataset and 75% on the Hospital dataset. A single teacher with access to all the data (Centralized) got 92% and 100% respectively.
The Analogy: Imagine a group of chefs trying to invent a new recipe. Chef A only has salt, Chef B only has sugar, and Chef C only has flour. They try to combine their ideas without ever seeing each other's ingredients. The final dish is okay, but it's nowhere near as good as if one master chef had access to the whole pantry.
The Surprise: Some strategies that work well for images (like 2D photos) completely collapsed on 3D data. Four specific "server-side" strategies (which try to be extra smart about how they combine the students' work) failed so badly they were barely better than random guessing.
Takeaway: There is no "one size fits all" strategy. What works for chairs might fail for head shapes.

Finding #2: The "Shrinking Machine" Works (Knowledge Distillation)

Once they had a trained model (even a slightly imperfect one), they tried to shrink it.

The Result: They successfully compressed the large model into a tiny one that was 74.5% smaller and twice as fast.
The Analogy: Think of the large model as a massive, heavy library. The shrinking process is like condensing that library into a single, lightweight pocket guide. The pocket guide is much faster to carry and read, yet it still knows almost everything the library knew.
Takeaway: Shrinking the model works great if the original teacher model was good.

Finding #3: The "Magic Trick" Illusion (The Big Warning)

This is the most important discovery of the paper. The authors found a trap in how people usually test these combined systems.

The Trap:
When testing the "shrunk" model, many researchers use a labeled cheat sheet (a set of data with the correct answers) to help the student learn.

The Illusion: If the "teacher" model (the one trained via the privacy-preserving method) was terrible (collapsed to near-random guessing), the "student" model could still get 99% accuracy.
Why? The student wasn't actually learning from the terrible teacher. It was ignoring the teacher and just memorizing the cheat sheet (the labels) provided during the shrinking process.
The Metaphor: Imagine a student taking a test. The teacher is asleep and gives bad advice. But the student is allowed to peek at the answer key on the desk while taking the test. The student gets 100% on the test, but it's not because they learned anything from the teacher; it's because they cheated using the answer key.
The Danger: In a real-world privacy setting (like the hospital), if you use this method, you might think your privacy-preserving system is working perfectly (100% accuracy), but in reality, your privacy system failed completely, and the model is just memorizing the answers it was supposed to protect.

The Solution:
The paper recommends a "Label-Free" test. When shrinking the model, do not give the student the answer key. Force the student to rely only on the teacher.

If you do this, the student's score drops to match the teacher's score. If the teacher failed, the student fails. This gives you a honest report card on whether the privacy system actually worked.

Summary

Privacy + 3D is hard: Training 3D models without sharing data causes a big drop in performance, and some popular methods fail completely.
Shrinking works: You can make the models much smaller and faster without losing much accuracy.
Don't be fooled: If you test the shrunk model using the correct answers (labels), you might think the system is perfect even when the privacy training failed. You must test without the answer key to see the truth.

The authors have made their code and all 504 test runs available online so others can verify these results and build better, safer 3D AI systems.

Technical Summary: Benchmarking Federated Learning & Knowledge Distillation for Point Cloud Classification

Problem Statement

The deployment of 3D point cloud analysis in privacy-sensitive (e.g., medical imaging) and resource-constrained (e.g., edge devices) environments faces two coupled barriers:

Data Privacy: Raw sensor data cannot be centralized due to regulatory or competitive constraints, necessitating Federated Learning (FL) where models are trained across clients without exchanging raw data. However, FL performance often degrades significantly under statistical heterogeneity (non-IID data), particularly when clients possess disjoint subsets of class labels.
Computational Efficiency: Full-scale 3D models (e.g., PointNet++) are often too large for edge hardware. Knowledge Distillation (KD) offers a solution by compressing a large "teacher" model into a smaller "student" model.

While both FL and KD are well-studied individually, there is a lack of systematic, reproducible benchmarks evaluating their combined pipeline for 3D point cloud classification. Specifically, it is unclear how KD objectives interact with federated teachers under extreme non-IID conditions, and whether the reported accuracy of a compressed student truly reflects the quality of the underlying federated model or merely the proxy labels used during distillation.

Methodology

The authors present a multi-seed benchmark evaluating the cross-product of 13 Federated Learning algorithms and 10 Knowledge Distillation objectives across two datasets:

Datasets:
- ModelNet40: A 40-class benchmark used to characterize federated degradation and isolate pipeline pitfalls against a strong centralized reference.
- Clinical Craniosynostosis Dataset: A real-world dataset of patient head shapes (4 classes) representing a privacy-critical scenario where the centralized task is essentially solved (100% accuracy), allowing for unambiguous measurement of teacher quality.
Non-IID Setup: A worst-case label-skew partitioning strategy is employed where training data is sorted by class and divided into five disjoint slices, assigning each client a unique subset of classes.
Architectures:
- Teacher: PointNet++ with Single-Scale Grouping (SSG).
- Student: A compact variant (SmallPointNet2) with 74.51% fewer parameters.
Pipeline: A two-stage process:
1. Stage 1 (FL): Train a federated teacher on the non-IID partition using one of 13 algorithms.
2. Stage 2 (KD): Distill the best federated checkpoint into the student model using a centralized proxy dataset and one of 10 KD objectives.
Evaluation Scale: The benchmark comprises 504 training runs across three random seeds, covering the full 130-pair teacher-objective grid on the clinical dataset.

Key Contributions

Benchmark Design: The first systematic evaluation of the full cross-product of FL algorithms and KD objectives for 3D point clouds, including a standardized multi-seed protocol and open-source framework.
Quantified Degradation under Label Skew: The study demonstrates that standalone FL degrades sharply under extreme label skew. On ModelNet40, the best FL method (FedNova) reaches only 76.32% compared to a 92.26% centralized baseline. On the clinical dataset, the best (FedProx) reaches 75.83% against 100%. Notably, four server-side optimizers (FedAvgM, FedAdam, FedYogi, FedAdagrad) collapse to near-chance levels on both datasets. No single algorithm is universally robust; rankings are dataset-dependent.
Effective Compression: Distillation successfully compresses the teacher into a student that is ~74.5% smaller and roughly twice as fast at inference. Under a strong centralized teacher, five of seven evaluated objectives match or surpass the teacher's accuracy.
Diagnosis of the Combined Pipeline Pitfall: The paper identifies a critical evaluation artifact:
- The Illusion: KD objectives that retain a hard-label cross-entropy term (using ground-truth labels from the proxy dataset) can recover student accuracy to near-centralized levels (e.g., ~92-100%) even when the federated teacher has collapsed (e.g., to 8.5% accuracy).
- The Mechanism: The hard-label term allows the student to learn directly from the proxy labels, effectively bypassing the federated teacher's knowledge. This masks the failure of the FL stage and reuses the very labels that federation was intended to protect.
- The Solution: Objectives without a hard-label term (pure transfer objectives like Feature KD, Attention Transfer, RKD, and SP) track the teacher's quality perfectly (correlation $r \approx 0.99$ ). If the teacher collapses, the student collapses.

Results

FL Performance: Under extreme non-IID skew, hierarchical 3D feature extraction is highly sensitive. Server-side adaptive optimizers are particularly fragile, likely due to ill-conditioned second-moment estimates on heterogeneous updates.
KD Performance:
- With Hard Labels: Student accuracy is decoupled from teacher quality. A collapsed teacher (8.5%) paired with Logit-MSE yields a 92.94% student, an 84.4-point gap driven entirely by the proxy labels.
- Without Hard Labels: Student accuracy correlates linearly with teacher quality. A collapsed teacher yields a collapsed student.
Clinical Validation: The "recovery illusion" persists on real patient data. A hard-label distilled student achieved ~100% accuracy from a federated teacher performing at chance level, while label-free distillation correctly reported the teacher's failure.

Significance and Claims

The paper claims that the combined FL-KD pipeline introduces a specific evaluation pitfall where reported accuracy reflects the proxy labels rather than the federated model's capability. The authors argue that:

Current evaluations are misleading: Reporting high accuracy for a compressed student does not guarantee the federated teacher is functional if the distillation loss includes a hard-label term.
Recommendation: FL-KD pipelines should be evaluated using label-free distillation (removing the cross-entropy term or using an unlabeled proxy split). This ensures that reported accuracy genuinely reflects the quality of the federated teacher.
Broader Impact: This benchmark provides a robust foundation for advancing privacy-preserving, edge-deployable 3D point cloud systems, highlighting that algorithm selection for FL is dataset-dependent and that compression strategies must be carefully designed to avoid masking federated failures.

The authors explicitly state that their study is limited to specific architectures (PointNet++ SSG) and one extreme non-IID regime (disjoint label skew), leaving the behavior of transformer-based backbones and softer partitioning strategies as open questions for future work.