💻 computer science

DKD-KAN: A Lightweight knowledge-distilled KAN intrusion detection framework, based on MLP and KAN

This paper proposes DKD-KAN, a lightweight intrusion detection framework that employs decoupled knowledge distillation to transfer complex feature representations from a high-capacity Kolmogorov-Arnold Network (KAN) teacher to a compact multilayer perceptron (MLP) student, achieving superior accuracy with significantly fewer parameters for deployment in resource-constrained environments.

Original authors: Mohammad Alikhani

Published 2026-03-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohammad Alikhani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a security guard how to spot a thief in a massive, high-tech factory.

The Problem: The "Super-Expert" vs. The "Tiny Guard"

In the world of cybersecurity, we have two types of "guards" (computer models):

The Super-Expert (The Teacher): This is a massive, incredibly smart model called a KAN (Kolmogorov–Arnold Network). It's like a genius professor who has read every book in the library. It can spot the tiniest, sneakiest cyber-attacks with near-perfect accuracy. But, this professor is huge, heavy, and requires a supercomputer to think. You can't carry them around in your pocket or put them on a small sensor in a factory.
The Tiny Guard (The Student): This is a small, lightweight model called an MLP (Multi-Layer Perceptron). It's like a new recruit. It's fast, light, and can run on a tiny device (like a smart thermostat or a factory sensor). But, on its own, it's not very smart and misses a lot of thieves.

The Challenge: We need the Tiny Guard to be as smart as the Super-Expert, but we can't just make the Tiny Guard bigger because it needs to stay small and fast.

The Solution: "Decoupled Knowledge Distillation" (DKD)

This paper proposes a clever training method called DKD-KAN. Think of it as a special tutoring session.

Instead of just telling the Tiny Guard, "You got this right, you got that wrong," the Super-Expert (Teacher) teaches the Tiny Guard (Student) in a very specific way using Decoupled Knowledge Distillation.

Here is the analogy:

Old Way (Standard Teaching): The teacher says, "The answer is 'Attack'." The student learns to guess "Attack." If the student is already pretty good at guessing "Attack," the teacher stops giving them extra help on the other details.
The DKD Way (The New Method): The teacher splits the lesson into two separate parts:
1. The "Target" Lesson: "Look at this specific thief. Here is exactly what they look like."
2. The "Non-Target" Lesson: "Look at all the innocent people who are not thieves. Here is exactly how they look different from the thief."

By separating these lessons, the Tiny Guard learns to spot the subtle differences between a "normal worker" and a "sneaky thief" much better. It doesn't just memorize the answer; it learns the nuance.

The Results: A Super-Compact Genius

The researchers tested this on two real-world industrial datasets (SWaT and WADI), which are like digital twins of water treatment plants and chemical factories.

Size: The "Teacher" (KAN) was huge. The "Student" (MLP) trained with DKD became tiny. It ended up with less than 2% of the teacher's brain power (parameters).
- Analogy: It's like shrinking a 100-story skyscraper down to a small, efficient treehouse, but the treehouse still has the same security system as the skyscraper.
Performance: Even though the student was tiny, it didn't lose its smarts. In fact, it got better at catching attacks than it would have been on its own.
- On the WADI dataset, the tiny student improved its success rate by 3% just by listening to the teacher.
- On the SWaT dataset, it improved by 4%.

Why Does This Matter?

Imagine you have a smart sensor on a water pipe in a remote village. It has a tiny battery and a weak processor.

Before: You couldn't put a smart security system there because the "Super-Expert" was too heavy. You had to use a "dumb" guard that missed attacks.
Now: You can use the DKD-MLP. It's light enough to fit on the sensor, runs on a small battery, but it thinks like the Super-Expert. It can catch cyber-attacks in real-time without needing a massive server farm.

Summary

The paper introduces a way to teach a small, fast computer model to think like a giant, slow, super-smart model. By splitting the teaching process into "what to look for" and "what to ignore," they created a tiny security guard that is incredibly accurate. This means we can finally put high-level cyber-security on small, everyday devices in our factories, power grids, and hospitals.

1. Problem Statement

Cyber-Physical Systems (CPS) and Industrial Control Systems (ICS) operate in resource-constrained environments (e.g., edge devices, real-time monitoring) where computational power and memory are limited. Traditional Deep Learning (DL) models for Intrusion Detection Systems (IDS) often suffer from:

High Computational Cost: Large parameter counts make deployment on edge devices difficult.
Preprocessing Bottlenecks: Many methods rely on complex preprocessing (e.g., PCA, windowing) that slows down real-time inference.
Data Imbalance: Intrusion datasets often have severe class imbalance (few attack samples vs. many normal samples), leading to poor detection of rare attacks.
Transparency: Many "black-box" DL models lack interpretability, which is critical for industrial trust.

The paper addresses the need for an IDS that is lightweight, fast, interpretable, and highly accurate without relying on complex feature selection or data augmentation techniques.

2. Methodology

The proposed framework, DKD-KAN, utilizes a Teacher-Student architecture combining Kolmogorov-Arnold Networks (KAN) and Multi-Layer Perceptrons (MLP) via Decoupled Knowledge Distillation (DKD).

A. Teacher Model: Kolmogorov-Arnold Network (KAN)

Architecture: Unlike traditional MLPs where learnable weights are on edges and activation functions are fixed on nodes, KANs place learnable 1D activation functions (spline-based) on the edges and aggregate sums at the nodes.
Theoretical Basis: Based on the Kolmogorov-Arnold Representation Theorem (KART), which states any multivariate continuous function can be represented as a finite sum of univariate functions.
Role: A high-capacity KAN is trained first to act as the "Teacher." It captures complex, non-linear feature relationships in the data with high accuracy and interpretability (due to visualizable spline functions).
Configuration: Uses high grid sizes ( $G=50$ ) and low spline orders ( $K=1$ or $3$) for optimal performance.

B. Student Model: Lightweight MLP

Architecture: A standard Multi-Layer Perceptron (MLP) with significantly fewer parameters.
Role: The "Student" model is designed for deployment on resource-constrained devices. It learns to mimic the teacher's output rather than just the ground truth.

C. Training Strategy: Decoupled Knowledge Distillation (DKD)

Standard Knowledge Distillation (KD) often couples the learning of target and non-target classes, which can lead to suboptimal learning for the student. The authors employ DKD, which separates the distillation loss into two independent components:

Target Class Knowledge Distillation (TCKD): Focuses on the probability of the correct class.
Non-Target Class Knowledge Distillation (NCKD): Focuses on the relative probabilities of incorrect classes.

The total loss function is defined as:
$L_{total} = (1-\lambda) \cdot L_{hard} + \lambda \cdot (\alpha \cdot TCKD + \beta \cdot NCKD)$

$\alpha$ and $\beta$ : Independent hyperparameters that allow fine-grained control over how much the student learns from target vs. non-target classes.
Warm-up: A linear warm-up strategy is applied to the DKD loss to prevent the student from overfitting to the teacher too early in training.

D. Data Preprocessing

Minimal Preprocessing: The approach avoids windowing, PCA, and feature selection (except removing non-varying features).
Standardization: Only basic standard scaling ( $X_{scaled} = \frac{X - \mu}{\sigma}$ ) is used to ensure fast inference.
Robustness: Instead of GANs or SMOTE for imbalance, the authors use a masking approach (randomly obscuring data) to improve model robustness.

3. Key Contributions

DKD Training Methodology: Successfully applied Decoupled Knowledge Distillation to compress a complex KAN teacher into a lightweight MLP student, achieving significant model compression without accuracy loss.
Lightweight & Fast Detection: The resulting models require minimal preprocessing and have extremely low parameter counts, making them suitable for edge deployment.
Performance Retention: Despite extreme compression (student models have <2% of the teacher's parameters), the framework maintains high detection performance.
Validation on Industrial Datasets: The method was rigorously tested on two standard ICS datasets: SWaT (Secure Water Treatment) and WADI (Water Distribution).

4. Experimental Results

The framework was evaluated on SWaT and WADI datasets against existing state-of-the-art methods (including USAD, DAGMM, LSTM, and standard KAN).

Model Sizes

SWaT: Student model has 1,622 parameters (vs. Teacher's ~22k).
WADI: Student model has 2,522 parameters (vs. Teacher's ~198k).

Performance Metrics (F1-Score Improvements)

Compared to a bare student MLP trained without distillation:

SWaT Dataset: F1-score improved by 4.18% (from 91.27% to 95.45%).
WADI Dataset: F1-score improved by 3.07% (from 95.38% to 98.45%).

Comparative Analysis

The DKD-MLP model achieved competitive F1-scores (95.45% on SWaT, 98.45% on WADI) compared to much larger models in the literature.
t-SNE Visualization: The student model's decision boundaries closely resembled the teacher model, particularly in separating "Normal" vs. "Attack" clusters, confirming effective knowledge transfer.
Confusion Matrices: Showed a significant reduction in False Negatives compared to the baseline MLP, indicating better attack detection.

5. Significance and Conclusion

The paper demonstrates that Kolmogorov-Arnold Networks can serve as powerful, knowledge-rich teachers for training highly efficient MLP students in the context of industrial cybersecurity.

Practicality: The solution bridges the gap between high-accuracy deep learning and the strict resource constraints of edge devices in CPS.
Efficiency: By decoupling the distillation process, the model learns subtle class distinctions that standard KD misses, leading to superior performance in imbalanced datasets.
Future Work: The authors note that hyperparameter tuning (specifically $\lambda$ and warm-up constants) for the SWaT dataset remains challenging and requires systematic optimization to ensure training stability.

In summary, DKD-KAN offers a robust, lightweight, and interpretable solution for real-time intrusion detection in critical infrastructure, proving that small models can achieve state-of-the-art performance when guided by advanced distillation techniques.