🤖 machine learning

ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data

ZAYAN is a self-supervised, feature-centric contrastive Transformer framework that utilizes zero-anchor dynamic feature encoding and redundancy minimization to learn disentangled, robust representations for tabular remote sensing data, achieving superior accuracy and generalization over existing baselines across diverse environmental benchmarks.

Original authors: Al Zadid Sultan Bin Habib, Tanpia Tasnim, Md. Ekramul Islam, Muntasir Tabasum

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Al Zadid Sultan Bin Habib, Tanpia Tasnim, Md. Ekramul Islam, Muntasir Tabasum

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world using a giant spreadsheet filled with data from satellites, weather sensors, and environmental surveys. This is the world of tabular remote sensing data. But here's the catch: these spreadsheets are messy. They have too many columns (features), many of which repeat the same information (redundancy), and they often lack clear labels telling the robot what everything means.

Enter ZAYAN (Zero-Anchor dYnamic feAture eNcoding), a new method designed to clean up this mess and teach the robot how to learn on its own.

The Problem with Old Methods

Most previous AI methods tried to learn by looking at entire rows of data (samples) and comparing them to each other, like a teacher pointing at two students and saying, "You are similar, you are different." This approach often gets stuck because it relies on picking specific "anchor" examples to compare against, and it struggles when the data is noisy or when there aren't enough labels to guide the teacher.

The authors of this paper argue that this sample-by-sample approach is missing the point for tabular data. They suggest that instead of looking at the whole row, we should focus on the individual columns (features) themselves.

The ZAYAN Solution: A Feature-Level Party

ZAYAN flips the script. Instead of comparing whole rows, it treats every single column in the spreadsheet as its own character in a story.

The "Zero-Anchor" Trick: Imagine you have a list of ingredients for a recipe. Instead of comparing your list to someone else's list (which requires a "reference" or anchor), ZAYAN takes each ingredient, shakes it up a little bit (adding noise or hiding parts of it), and asks, "Is this still the same ingredient?" It does this without needing a teacher to say "Yes" or "No." This is called zero-anchor contrastive learning.
The "Disentangled" Dance: The goal is to make sure each ingredient (feature) has its own unique voice. If two ingredients sound exactly the same (redundant), ZAYAN pushes them apart. It wants a "disentangled" space where every feature is distinct and useful, minimizing the noise and repetition.
The Transformer Brain: Once the features have learned to be unique and robust through this self-supervised dance, ZAYAN passes them to a Transformer (a type of AI brain famous for understanding context). This brain uses the cleaned-up features to make predictions, like classifying land cover or predicting floods.

The Results: How Well Did It Work?

The researchers tested ZAYAN on eight different datasets, ranging from urban land cover maps to flood prediction tables derived from satellite and GIS products.

The Scoreboard: ZAYAN didn't just play well; it dominated. It achieved the highest or tied-for-highest accuracy on seven out of eight datasets.
- On Urban Land Cover, it hit 84.80% accuracy.
- On Wilt (a plant disease dataset), it reached 99.69%.
- On Forest Type Mapping, it scored 97.21%.
- Even on the tricky Pluvial Flood dataset (which had 50% noise injected to make it harder), it managed 93.61%.
The Ranking: When you rank all the models tested (including old-school methods like Random Forests and modern deep learning models), ZAYAN took the top spot with an average rank of 1.06. The next best models, like TabICL and TANDEM, trailed behind with ranks of 3.38 and 6.06, respectively.
Robustness: The paper suggests that ZAYAN is particularly good at handling messy data. When they shuffled or dropped features, ZAYAN's performance stayed stable up to 25% shuffled features, only dropping significantly when half the features were removed. It also showed it could be "calibrated," meaning its confidence levels matched its actual accuracy reasonably well (with an expected calibration error of 0.151).

What It's NOT (And What's Still Hard)

The paper is careful to rule out a few things and admit limitations:

No Magic Bullet for Everything: While ZAYAN is great, it isn't a "breakthrough" that solves every problem instantly. The authors note that on some simpler tasks, older methods like K-Nearest Neighbors (KNN) or Gradient Boosted Trees (like CatBoost) still perform very well.
The Cost of Complexity: The method has a "quadratic complexity" issue. This means as the number of features grows, the computer work required grows much faster (specifically O(m²)). The paper explicitly states that for very high-dimensional inputs (like 2048-D image embeddings) or massive datasets (like the 325,834 rows in the Crop Mapping dataset), the method might run out of memory (OOM) or take too long.
Self-Supervised Limits: Because ZAYAN learns without labels, it doesn't use the "cheat sheet" of known answers during the training phase. The authors suggest that future work could mix in some labeled data to see if that helps even more.

The Verdict

ZAYAN suggests that the best way to learn from messy, high-dimensional environmental spreadsheets is to stop looking at the whole picture and start focusing on the individual parts, teaching each part to be unique and noise-resistant before putting them together. While it faces challenges with massive datasets due to its computational cost, the experiments show it is a highly effective recipe for learning from tabular sensing data, consistently outperforming both classical and modern deep learning baselines across a wide variety of remote sensing tasks.

Technical Summary: ZAYAN – Disentangled Contrastive Transformer for Tabular Remote Sensing Data

Problem Statement
Learning informative representations from tabular data in remote sensing and environmental science is hindered by three primary challenges: feature heterogeneity, scarce labels, and significant redundancy among features. While Self-Supervised Learning (SSL), particularly Contrastive Learning (CL), has shown promise for label-efficient representation learning, existing frameworks predominantly operate at the instance or patch level. These methods rely on explicit anchor selection and view augmentations of entire samples, often failing to address the specific structural redundancies and feature interactions inherent in tabular remote sensing data. Furthermore, standard Transformer models for tabular data typically treat all features as uniform tokens without explicitly disentangling redundant dimensions or minimizing feature redundancy.

Methodology
The authors propose ZAYAN (Zero-Anchor dYnamic feAture eNcoding), a self-supervised, feature-centric framework designed to learn robust, disentangled embeddings for tabular remote sensing data. The framework consists of two distinct modules:

ZAYAN-CL (Feature-Level Contrastive Pretraining):
- Zero-Anchor Objective: Unlike traditional CL that contrasts samples, ZAYAN-CL performs contrastive learning at the feature level. It generates two stochastic augmented views for each individual feature vector (using Gaussian noise, quantile warping with jitter, and random masking) without requiring explicit anchors or class labels.
- Loss Function: The module optimizes a feature-level InfoNCE loss ( $L_{ZAYAN-CL}$ ) to maximize agreement between augmented views of the same feature while pushing apart embeddings of different features. This is formulated to maximize the mutual information lower bound between views.
- Redundancy Minimization: To explicitly address feature redundancy, a penalty term ( $R(\theta)$ ) is introduced, which minimizes the pairwise cosine similarity between distinct feature embeddings. This is implemented efficiently via a matrix formulation ( $\|Z^\top Z - I\|_F^2$ ) to enforce orthogonality and encourage a disentangled embedding space.
- Output: The encoder produces normalized, diversity-maximized feature embeddings on a unit hypersphere.
ZAYAN-T (Downstream Transformer):
- Architecture: A Transformer backbone that consumes the pretrained feature embeddings. It incorporates "ZAYAN-aware" positional encodings to preserve the structural and angular relationships established during pretraining.
- Fine-Tuning: The model is fine-tuned for supervised classification using a total loss ( $L_{total}$ ) that combines standard cross-entropy with a structure-preserving loss ( $L_{preserve}$ ). This preservation term ensures the downstream Transformer outputs remain close to the pretrained feature embeddings, maintaining the disentangled geometry learned during the self-supervised phase.

Key Contributions
The paper outlines four primary contributions:

C1 (ZAYAN-CL): A novel feature-level, zero-anchor contrastive pretraining scheme that yields robust, disentangled, and redundancy-minimized embeddings for tabular remote sensing data without relying on labels or explicit anchors.
C2 (ZAYAN-T): A Transformer backbone designed to leverage these pretrained embeddings while preserving their structural and angular geometry for improved supervised classification.
C3 (Comprehensive Benchmarks): An extensive evaluation across eight diverse datasets, including six remote-sensing tabular benchmarks (e.g., Urban Land Cover, Forest Type Mapping) and two flood-prediction tables derived from satellite and GIS products.
C4 (Robustness and Deployment): Demonstrated consistent gains over traditional machine learning and modern tabular deep learning baselines in accuracy, robustness, and generalization, particularly under conditions of label scarcity, distribution shifts, and noise.

Experimental Results
The framework was evaluated on eight datasets spanning urban land cover, forest mapping, crop mapping, and flood risk prediction.

Performance: ZAYAN achieved the highest or tied-highest mean accuracy across all eight datasets. Notable results include 84.80% on Urban Land Cover, 99.69% on Wilt, 97.21% on Forest Type, and 93.61% on Pluvial Flood.
Ranking: ZAYAN achieved an average rank of 1.06 across the benchmarks, significantly outperforming the next-best models (TabICL, TANDEM, TabM, and TabR), which had average ranks ranging from 3.38 to 9.25.
Robustness: The model demonstrated superior performance in noisy and imbalanced settings (e.g., Pluvial Flood with 50% injected noise), where other methods suffered significant accuracy drops.
Ablation Studies: Experiments confirmed the necessity of the redundancy penalty, contrastive loss, and specific augmentations (noise, warping, masking). Removing any component resulted in a 4–9 percentage point drop in accuracy.
Deployment Diagnostics: Inference-time analysis showed ZAYAN maintains stability under feature perturbations (up to 25% shuffling), exhibits reasonable calibration (ECE = 0.151), and supports effective selective prediction and triage decisions.

Significance and Claims
The paper claims that ZAYAN provides an effective recipe for learning from tabular sensing data by shifting the contrastive paradigm from the sample level to the feature level. By explicitly tackling redundancy and heterogeneity through zero-anchor dynamic feature encoding, ZAYAN addresses the limitations of instance-centric SSL methods and uniform token-based Transformers. The authors assert that feature-level contrastive learning coupled with dynamic feature encoding enables the learning of representations where environmental signals are amplified and noise is suppressed, leading to superior generalization in remote sensing applications.

Limitations
The authors acknowledge that the redundancy penalty incurs quadratic computational complexity ( $O(m^2d)$ time, $O(m^2)$ space) with respect to the number of features, which may lead to memory constraints (OOM) for very high-dimensional inputs (e.g., 2048-D embeddings). Additionally, the purely self-supervised nature of the augmentations omits potentially beneficial label information, suggesting future work could explore sparse attention mechanisms or hybrid supervised-contrastive objectives to improve scalability.