💻 computer science

Implicit spatial-frequency fusion of hyperspectral and lidar data via kolmogorov-arnold networks

This paper proposes IFGNet, a novel framework leveraging Kolmogorov-Arnold Networks and a LiDAR-guided implicit aggregation module to effectively fuse hyperspectral and LiDAR data in both spatial and frequency domains, thereby achieving superior classification performance on complex scenes compared to existing methods.

Original authors: Zekun Long, Judy X. Yang, Jing Wang, Ali Zia, Guanyiman Fu, Jun Zhou

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zekun Long, Judy X. Yang, Jing Wang, Ali Zia, Guanyiman Fu, Jun Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify different types of trees in a dense forest just by looking at a photograph. Sometimes, two different trees look almost identical in color (spectral data), making them hard to tell apart. However, if you also had a 3D map showing exactly how tall they are and the shape of their branches (LiDAR data), the job would become much easier.

This paper is about teaching a computer to do exactly that: combine a colorful "flat" photo (Hyperspectral Image) with a "3D height map" (LiDAR) to classify what objects are in a scene with extreme accuracy.

Here is how the authors solved the problem, explained simply:

The Problem: Old Tools Were Too Rigid

Previous methods tried to combine these two types of data using standard "neural networks" (the brain-like software computers use to learn). The authors argue these old tools are like rigid plastic molds. They have fixed shapes and can only stretch a little bit. When the data gets messy—like a tree branch that suddenly stops, or a building with a weird roof—these rigid molds can't bend enough to capture the details. They also struggle to look at the "big picture" (global patterns) and the "fine details" (local shapes) at the same time.

The Solution: IFGNet (The "Smart Clay" Network)

The authors created a new system called IFGNet. Instead of using rigid plastic molds, they used something they call Kolmogorov-Arnold Networks (KANs).

The Analogy: Think of KANs as smart, shape-shifting clay. Instead of being stuck in one shape, this clay can mold itself perfectly around any curve or corner. It learns the specific shape of the data as it goes, rather than forcing the data into a pre-made box. This allows the computer to understand the complex, wiggly relationships between the colors of the image and the height of the objects.

How It Works: Two Ways of Looking at the World

The system doesn't just mash the two data types together; it fuses them in two different "domains" (ways of looking at the data) simultaneously:

The Spatial Domain (The "Local Neighborhood" View):
- Imagine you are standing on a street corner. You look at a building and ask, "What is right next to me?"
- The system uses the LiDAR (height map) as a guide. If the height map shows a sharp drop (like a cliff or a building edge), the system knows not to blend the colors from the other side of that edge. It uses the height information to keep the boundaries sharp, preventing the "blur" that happens in older methods.
The Frequency Domain (The "Music" View):
- Imagine listening to a song. You can hear the individual notes (high frequency) and the overall melody or rhythm (low frequency).
- The system translates the image into "mathematical music" (using a process called Fourier Transform). This helps it see global patterns—like the repeating rhythm of a row of houses or the overall shape of a park—that might be missed if you only looked at individual pixels. It fuses the "music" of the colors with the "music" of the heights.

The Result: A Perfect Blend

By combining these two views (the local neighborhood guide and the global music pattern) using the "smart clay" (KANs), the system creates a unified picture that is much clearer than the sum of its parts.

What the paper claims they achieved:

Better Accuracy: When tested on two real-world datasets (one from Houston and one from Gulfport), their new method got significantly higher scores than all the previous "rigid mold" methods.
Efficiency: Despite being smarter, the system isn't necessarily a giant, slow monster; it remains lightweight and efficient.
Robustness: It works well even in tricky urban scenes where objects have similar colors but very different shapes.

In short, the paper says: "Stop trying to force complex 3D and color data into rigid boxes. Use flexible, shape-shifting math (KANs) to blend height and color together in both local and global ways, and you get a much smarter classifier."

Technical Summary: Implicit Spatial-Frequency Fusion of Hyperspectral and LiDAR Data via Kolmogorov-Arnold Networks

Problem Statement

Hyperspectral image (HSI) classification in complex scenes faces significant challenges due to spectral ambiguity, spatial heterogeneity, and the strong coupling between material properties and geometric structures. While Light Detection and Ranging (LiDAR) data provides complementary elevation and structural information to mitigate spectral ambiguity, existing fusion methods exhibit limitations:

Modeling Constraints: Most current approaches rely on Convolutional Neural Networks (CNNs) or Multi-Layer Perceptrons (MLPs) with fixed activation functions and linear weights. These struggle to model the structural discontinuities of LiDAR, intricate spectral features of HSI, and their complex non-linear interactions.
Fusion Strategy: Existing methods typically treat HSI and LiDAR as independent branches, lacking a mechanism where LiDAR geometry actively guides the sampling and refinement of spectral context.
Domain Coverage: While fusing data in both spatial and frequency domains shows preliminary value, it remains under-explored in the literature, particularly without predefined heuristics.

Methodology: IFGNet

To address these issues, the authors propose the Implicit Frequency-Geometry Fusion Network (IFGNet). This framework leverages Kolmogorov-Arnold Networks (KANs) to perform implicit aggregation across spatial and frequency domains. The architecture consists of four core components:

1. KAN-Based Encoder

Instead of traditional MLPs, the network employs KANs as encoders for both HSI and LiDAR data.

Mechanism: KANs parameterize non-linear mappings using learnable B-spline functions rather than fixed activation functions and static weight matrices.
Advantage: This allows for high-fidelity non-linear fitting of subtle spectral variations and enhanced local sensitivity to geometric discontinuities in LiDAR elevation data, providing a shared latent space for subsequent fusion.

2. Spatial Implicit Aggregation Unit (SIAU)

This module performs geometry-aware aggregation in the latent feature space.

Process: For a query spatial coordinate $q$ , the unit identifies a local neighborhood. It concatenates the hyperspectral latent feature of a neighbor, the LiDAR latent feature at the query position, and the relative coordinate offset.
Aggregation: This input is fed into a KAN-based implicit function to generate candidate features and aggregation weights. A softmax normalization is applied to the weights, and a weighted summation produces the final spatially guided representation.
Function: This acts as a geometry-conditioned implicit operator, allowing the model to prioritize structurally consistent regions and preserve sharp elevation boundaries often blurred by standard convolutions.

3. Frequency-Domain Implicit Aggregation

Recognizing that urban scenes exhibit strong correlations between object geometry and frequency characteristics, the framework extends implicit aggregation to the frequency domain.

Process: A 2D Discrete Fourier Transform (DFT) is applied to the encoded latent features to extract real and imaginary components.
Aggregation: The same implicit geometric aggregation operator (SIA) is applied component-wise to the frequency-domain representations of both HSI and LiDAR.
Reconstruction: An Inverse Discrete Fourier Transform (IDFT) reconstructs the frequency-guided representation, capturing global structural patterns and long-range dependencies.

4. Final Fusion and Classification

The spatially guided representation ( $F_{Spa}$ ) and the frequency-guided representation ( $F_{Freq}$ ) are integrated via an additive fusion strategy ( $\hat{z} = F_{Spa} + F_{Freq}$ ). The unified multi-modal feature is then passed to a lightweight classification head.

Key Contributions

The paper outlines three primary contributions:

KAN-Driven Nonlinear Feature Modeling: The authors pioneer the use of KANs for HSI-LiDAR fusion, replacing rigid linear weights with learnable spline-based functions. This enables the parameterization of fusion operators that adaptively capture complex, non-linear cross-modal interactions conditioned on both spectral content and spatial coordinates.
LiDAR-Guided Spatial Implicit Aggregation: A guided aggregation mechanism is introduced that uses LiDAR-derived geometric cues as a structural reference. By sampling neighboring hyperspectral features through an implicit neural representation, the model achieves area-aware fusion that preserves sharp elevation boundaries.
Dual-Domain Implicit Fusion Framework: The method models both spatial-domain geometric aggregation and frequency-domain structural dependencies simultaneously. This avoids predefined heuristics and captures a comprehensive representation of high-frequency phase information and global spectral consistency.

Experimental Results

The proposed IFGNet was evaluated on two benchmarks: the Houston 2013 and MUUFL Gulfport datasets. The framework was compared against state-of-the-art methods including CALC, GLTNet, M2FNet, MS2CANet, S2ENet, MFT, and ExViT.

Performance: IFGNet consistently outperformed all existing methods across Overall Accuracy (OA), Average Accuracy (AA), and Cohen's Kappa coefficient.
- Houston 2013: Achieved 99.37% OA, 99.50% AA, and 99.32% Kappa.
- MUUFL: Achieved 92.67% OA, 94.47% AA, and 90.45% Kappa.
Ablation Studies: Experiments confirmed that the spatial-only and frequency-only variants are individually effective but complementary. The full model, integrating both, yielded the best performance, validating the necessity of dual-domain implicit fusion.
Parameter Analysis: A patch size of 5×5 was determined to be optimal for the MUUFL dataset, balancing sufficient spatial-spectral context against redundant background information.

Significance and Claims

The paper claims that IFGNet offers a significant advancement in HSI-LiDAR fusion by moving away from rigid, linear-weighted architectures toward adaptive, spline-based function approximation. By decoupling spatial-geometric aggregation from frequency domain modeling and unifying them under an implicit representation framework, the method achieves effective multimodal fusion while maintaining a lightweight architecture. The authors assert that this design avoids the computational heaviness of deep transformer networks and heavy attention modules, making it suitable for practical remote sensing scenarios. The consistent performance gains across two distinct benchmarks suggest strong robustness and generalization capabilities for complex urban scenes.