📊 statistics

iStructTab: Structured Feature Sequencing for Multimodal Learning of Image and Tabular Data

The paper introduces iStructTab, a multimodal learning framework that employs Graph-Enhanced Descriptor Sequencing (GEDS) to optimize feature ordering via similarity graphs and integrates this structure into an order-aware transformer, thereby reducing feature dispersion and significantly improving predictive performance on image and tabular data benchmarks.

Original authors: Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Gyawali, Gianfranco Doretto, Donald A. Adjeroh

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Gyawali, Gianfranco Doretto, Donald A. Adjeroh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world. You give it two very different types of information: a photograph of a scene and a spreadsheet full of facts about that scene. This is called "multimodal learning," and it's how computers try to see and think like humans, combining what they see with what they know. But here's the tricky part: photos have a natural order. The sky is usually at the top, the ground at the bottom, and trees have roots below branches. The computer knows exactly where to look. Spreadsheets, however, are a different story. A list of facts—like a person's age, their favorite color, and their shoe size—doesn't have a built-in map. The computer doesn't know if it should look at the age first or the shoe size first. For a long time, scientists just threw these facts into the computer in whatever order they happened to be written down, hoping the machine would figure it out. But often, the computer got confused, mixing up important clues and missing the big picture.

This paper tackles that confusion by asking a simple question: "Does the order of facts matter?" The authors suggest that just like arranging books on a shelf by genre helps you find them faster, arranging data columns by how they relate to each other helps the computer learn better. They propose a new method to sort these facts before the computer even starts learning, turning a messy pile of information into a neat, logical story. By doing this, they hope to build smarter AI that can combine images and data without getting lost in the noise.

The Big Idea: Sorting the Messy Pile

Meet iStructTab, a new AI tool designed to be the ultimate organizer for computers that look at both pictures and spreadsheets. The researchers behind this tool realized that when you mix image data (like a photo of a car) with tabular data (like the car's year, color, and mileage), the computer often gets overwhelmed. It's like trying to read a book where the pages are shuffled randomly; you might get the gist, but you'll miss the plot.

The core problem is that spreadsheets are "unordered." The computer treats the first column and the last column as if they are just random neighbors. But in reality, some facts are best friends, while others are strangers. The authors argue that if we can figure out the best order to present these facts to the computer, it will learn much faster and make fewer mistakes.

The Detective Work: GEDS

To solve this, the team invented a clever sorting algorithm called GEDS (Graph-Enhanced Descriptor Sequencing). Think of GEDS as a super-smart detective who walks into a room full of scattered clues.

Gathering Clues: First, GEDS looks at every column of data. It doesn't just read the numbers; it calculates a "statistical fingerprint" for each one. It asks, "How much does this column vary? What is its average?"
Drawing the Map: Next, it draws a map (a graph) connecting the columns. If two columns are very similar or related, it draws a strong line between them. If they are totally different, the line is weak.
The Great Shuffle: Using this map, GEDS figures out the perfect order. It rearranges the columns so that the most related facts sit next to each other, creating a smooth, logical flow. It's like organizing a playlist so that songs with similar vibes play back-to-back, rather than jumping from heavy metal to lullabies.

The paper shows that this isn't just a guess; it's a mathematical solution to a complex puzzle known as the "Column Permutation Problem." While finding the perfect order is incredibly hard for computers (so hard that it's considered a "NP-hard" problem), GEDS finds a very good, practical solution that works well in real life.

The Classroom: OEMT

Once GEDS has sorted the data, it hands the neatly arranged list to a second part of the system called OEMT (Order-Aware Efficient Transformer with Memory Augmentation).

Imagine a student taking a test. If the questions are jumbled, the student might get confused. But if the questions are in a logical order, the student can use their memory to connect the dots. OEMT is that smart student. It has a special "memory token"—a little sticky note it keeps in its mind—to remember the global context of the whole dataset. Because the data is already sorted by GEDS, the student can focus on learning the deep connections between the image and the facts, rather than wasting energy trying to figure out the order.

The system is trained with a special rule: it gets a "punishment" (a loss function) if it tries to learn the data in a different order than what GEDS suggested. This forces the AI to respect the structure and learn the relationships properly.

The Results: Smarter and Faster

The researchers tested iStructTab on six different real-world datasets, ranging from identifying car models and pet adoption speeds to diagnosing medical conditions from X-rays and skin images.

Better Accuracy: On almost every test, iStructTab beat the previous best methods. For example, on a dataset of car images and attributes, it achieved a top ranking that was significantly better than other top-tier models. It didn't just win by a tiny bit; it consistently placed at the very top of the leaderboard.
Handling Noise: Real-world data is messy. Sometimes labels are wrong (like a photo of a dog labeled as a cat). The paper shows that iStructTab is very tough against this kind of noise. Even when 60% of the labels were messed up, it still performed better than other methods, suggesting it learned the true patterns rather than just memorizing mistakes.
Efficiency: You might think that sorting data and using a fancy transformer would make the computer slow. Surprisingly, iStructTab is actually very efficient. It uses fewer computing resources (like energy and memory) than many of its competitors while still getting better results. It's like getting a faster car that also uses less gas.

What This Means

The paper suggests that the way we feed data to AI matters just as much as the AI itself. By treating the ordering of data columns as a solvable puzzle, the authors have shown that we can build models that are more accurate, more robust against errors, and more efficient.

They didn't just say, "Order matters." They built a tool that figures out the order automatically and proved it works across different types of problems, from medical imaging to pet adoption. While the paper doesn't claim to have solved every problem in AI, it strongly suggests that ignoring the structure of our data is a mistake we can no longer afford to make. With tools like iStructTab, we are one step closer to AI that can truly understand the complex stories hidden in our pictures and spreadsheets.

Technical Summary of iStructTab: Structured Feature Sequencing for Multimodal Learning of Image and Tabular Data

1. Problem Statement

Multimodal learning involving images and tabular data faces significant challenges due to the structural mismatch between modalities. Images possess strong spatial structures, whereas tabular features are inherently unordered. This lack of ordering in tabular data often leads to ineffective representations characterized by redundancy, feature dispersion, and poor generalization. While existing fusion methods utilize attention mechanisms, contextual embeddings, and contrastive pretraining, they largely treat fused descriptors as unordered vectors, overlooking the potential impact of feature sequencing on cross-modal alignment and prediction performance. The authors posit that determining an optimal feature order is a critical, yet neglected, factor in hybrid image-tabular settings.

2. Methodology

The proposed framework, iStructTab, addresses this challenge by formulating multimodal feature ordering as a Column Permutation Problem (CPP). The architecture consists of two primary components:

A. Graph-Enhanced Descriptor Sequencing (GEDS)
GEDS is a structured feature sequencing algorithm designed to determine an optimal permutation ( $\pi$ ) of features that minimizes inter-feature dispersion. The process involves:

Feature Extraction: Images are encoded into vectors using a CNN (e.g., ResNet), and tabular data is encoded via a Transformer. These are concatenated to form a unified feature matrix $F \in \mathbb{R}^{N \times m}$ .
Descriptor Computation: For each feature column, statistical descriptors (mean and variance) are computed.
Graph Refinement: These descriptors are normalized, and a cosine similarity matrix is constructed. A graph convolution operation is applied to refine the descriptors, propagating similarity information across the feature graph.
Sequencing: An ordering score is derived from the Euclidean norm of the refined descriptors. Features are sorted based on these scores to generate the optimized permutation $\pi_{GEDS}$ . This process effectively treats the sequencing task as a learnable CPP, seeking a permutation that reduces dispersion and enhances structural coherence.

B. Order-Aware Efficient Transformer with Memory Augmentation (OEMT)
OEMT is the backbone network designed to process the reordered features while preserving their structural sequence:

Order-Aware Pooling: The reordered input features are projected into token embeddings and compressed into a smaller set of summary tokens ( $k \ll m$ ) using a learnable pooling matrix. This step enforces the feature sequencing structure.
Memory Augmentation: Learnable memory tokens are prepended to the pooled tokens to facilitate global reasoning and capture interactions across the ordered features.
Efficient Encoding: A Linformer-based encoder processes the sequence, reducing computational complexity from quadratic to linear with respect to sequence length.
Sequencing Loss: A dedicated loss function is introduced to align the learned representations with the target permutation. This loss penalizes deviations between the predicted sequencing scores and the target sequence derived from GEDS, encouraging consistent feature semantics and interpretability.

The model is trained end-to-end using a combined loss function comprising cross-entropy for classification and the sequencing regularization term.

3. Key Contributions

The paper outlines four primary contributions:

Formulation of CPP in Multimodal Learning: The authors frame multimodal fusion as a Column Permutation Problem, demonstrating how feature ordering impacts representation learning in hybrid tabular-image settings.
Introduction of GEDS: A novel, graph-enhanced descriptor sequencing algorithm that utilizes similarity propagation to generate task-aware feature permutations.
Design of OEMT: An efficient transformer architecture that explicitly enforces feature sequencing through learnable memory tokens and a sequencing-aware loss function.
Empirical Validation: Extensive experiments across diverse benchmarks show that the GEDS-OEMT framework significantly improves classification accuracy, robustness, and efficiency compared to state-of-the-art baselines.

4. Experimental Results

The authors evaluated iStructTab on six image-tabular datasets, including DVM (car images with attributes), Deep Lesion, HAM10000, CheXpert, Pokémon, and Pet Finder.

Performance: iStructTab achieved the best overall performance across all benchmarks, securing the top average rank (1.50) and the lowest average regret (2.21). It outperformed strong multimodal baselines like STiL and TIP, as well as image-only and tabular-only classifiers.
Robustness: The model demonstrated superior robustness to label noise, outperforming prior methods (e.g., CUFIT) across various noise rates, particularly under severe corruption (60% noise).
Efficiency: On the DVM dataset, iStructTab achieved the best accuracy-efficiency trade-off, lying on the low-complexity/high-accuracy Pareto frontier. It utilized fewer FLOPs and parameters while maintaining near-maximum accuracy compared to other multimodal methods.
Ablation Studies:
- Removing feature sequencing caused a significant performance drop (from ~99% to ~81%), confirming the necessity of the ordering signal.
- Removing memory tokens resulted in a substantial decline, highlighting their role in capturing global context.
- The sequencing loss provided useful regularization, though its removal had a smaller impact than removing the sequencing itself.
Representation Quality: t-SNE visualizations indicated that iStructTab learns coherent manifolds where semantically related cases cluster together, with clear cross-modal alignment, particularly in the HAM10000 dataset.

5. Significance and Claims

The paper claims that structured feature sequencing is a critical, previously overlooked factor in multimodal learning. By treating feature ordering as a solvable optimization problem (CPP) rather than an arbitrary assumption, iStructTab effectively minimizes feature dispersion and improves the alignment between image and tabular modalities.

The authors emphasize that their approach does not merely fuse data but organizes it structurally before processing. This structural prior is shown to be essential for stable and robust deep learning in hybrid settings. The work suggests that future multimodal systems should consider explicit feature sequencing to enhance predictive performance and generalization, moving beyond standard unordered fusion strategies. The paper concludes that iStructTab offers a scalable, efficient, and highly effective solution for image-tabular learning, with potential for extension to larger-scale pretraining and additional modalities.