Hyperspectral Image Land Cover Captioning Dataset for Vision Language Models

Technical Summary: HyperCap: Hyperspectral Land Cover Captioning Dataset for Vision Language Models

Problem Statement
Hyperspectral Imaging (HSI) has become a transformative technology for remote sensing, offering fine-grained spectral resolution for tasks like vegetation health assessment and mineral prospecting. However, current HSI research faces three critical bottlenecks:

1. Lack of Semantic Understanding: Existing deep learning models achieve high classification accuracy but operate as "black boxes," lacking the transparency and human-interpretable logic required for high-stakes domains like disaster response and precision agriculture.
2. Scarcity of Large-Scale Labelled Data: The annotation process for HSI is labor-intensive and costly, leading to limited, imbalanced datasets that hinder generalization.
3. Absence of Textual Annotations: Traditional HSI datasets rely solely on pixel-wise numerical labels. They lack natural language descriptions that capture semantic land cover information, limiting the application of Vision-Language Models (VLMs) and hindering domain generalization. While some recent works have attempted HSI captioning, they are often constrained by small scale, patch-level (rather than pixel-level) granularity, or reliance on template-based descriptions that lack diversity.

Methodology
The authors introduce HyperCap, the first large-scale, pixel-wise hyperspectral captioning dataset. The construction and evaluation process involves:

• Dataset Construction: HyperCap aggregates four widely used benchmark HSI datasets: Botswana, Houston13, Indian Pines, and Kennedy Space Center (KSC).
• Hybrid Annotation Strategy: To generate pixel-level captions, the authors employed a hybrid approach:
  1. Automated Generation: Large Language Models (LLMs), specifically ChatGPT-4o and Mistral Large, were prompted with a structured template containing spectral signatures and environmental context. The prompts explicitly constrained the models to avoid using class names, functional descriptions, or speculative details, forcing outputs based on observable physical attributes.
  2. Expert Refinement: Three remote sensing experts manually reviewed and refined the LLM outputs. They removed any remaining class names, functional interpretations, or hallucinations, ensuring the final captions were visually grounded and physically plausible.
  3. Granularity: Unlike previous patch-level approaches, HyperCap enforces a strict one-to-one alignment between individual pixels and unique, expert-refined captions.
• Experimental Framework: The authors evaluated the dataset using state-of-the-art HSI feature extractors (3D-RCNet, 3D-ConvSST, DBCTNet, FAHM) combined with text encoders (BERT-Large-Uncased, T5). They tested five fusion techniques: Cross Attention (CA), Concatenation (CONCAT), Multi-Head Attention (MHA), Pixel-Wise Addition (PWA), and Pixel-Wise Multiplication (PWM).
• Ablation Studies: The study included rigorous checks for "label leakage" (ensuring captions do not simply encode the class label) and data efficiency (testing performance with only 3% of training data).

Key Contributions

1. HyperCap Dataset: The creation of the first large-scale, pixel-wise HSI captioning dataset containing 21,237 annotated samples across four benchmarks.
2. Semantic Integration: The successful integration of spectral data with human-readable textual annotations, enabling models to generate explanations and enhancing semantic understanding beyond simple classification.
3. Benchmarking: The establishment of a baseline for vision-language learning in HSI, demonstrating that textual descriptions provide complementary semantic information rather than acting as direct proxies for class labels.

Results
Empirical evaluations across the four datasets demonstrate significant performance improvements when textual information is integrated:

• Classification Accuracy: Integrating text encoders with vision models yielded substantial gains in Overall Accuracy (OA), Precision, F1-Score, and Kappa. For instance, on the Indian Pines dataset, the DBCTNet model improved from 76.01% OA (vision-only) to 99.37% OA (with text fusion), a gain of over 23%. Similar dramatic improvements were observed for weaker architectures like DBCTNet on KSC (70.50% to 97.58%).
• Robustness to Data Scarcity: In low-annotation scenarios (using only 3% of labeled pixels), vision-only models suffered significant performance drops (e.g., FAHM dropped to ~94% OA). In contrast, multimodal fusion models maintained robust performance, often exceeding 98-99% OA, indicating that captions provide effective regularization and semantic guidance.
• No Label Leakage: Ablation studies confirmed that text-only models performed significantly worse than multimodal models, proving that the captions contain genuine semantic information and do not merely leak target labels.
• Retrieval and Captioning: The dataset supports multimodal retrieval tasks (Image-to-Text and Text-to-Image) and caption generation, with models like GIT and mPLUG showing strong performance in generating semantically aligned descriptions.

Significance and Claims
The paper positions HyperCap as a foundational resource for advancing remote sensing research. Its primary significance lies in bridging the gap between high-dimensional spectral data and human semantic understanding. By providing fine-grained, pixel-level textual descriptions, HyperCap:

• Enhances the interpretability and trustworthiness of AI-driven HSI classification.
• Democratizes state-of-the-art performance, allowing weaker architectures to achieve high accuracy through semantic fusion.
• Improves model robustness in real-world scenarios where labelled data is scarce.
• Establishes a new benchmark for future research in vision-language learning, domain generalization, and multimodal tasks within the hyperspectral domain.

The authors conclude that while current results are promising, future work will focus on scaling the dataset to include more diverse benchmarks and developing foundational captioning models specifically tailored for HSI data.