🤖 machine learning

Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification

This paper introduces a multi-modal co-learning framework designed to handle missing arbitrary modalities in classification tasks by prioritizing inter-modal collaboration over fusion, offering two complementary approaches that demonstrate significant robustness gains across varying degrees of modality absence.

Original authors: Francisco Mena, Dino Ienco, Roberto Interdonato, Cassio F. Dantas, Simon Besnard

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Francisco Mena, Dino Ienco, Roberto Interdonato, Cassio F. Dantas, Simon Besnard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant jigsaw puzzle, but instead of just one picture, you have a team of friends, each holding a different piece of the puzzle. One friend has the sky, another has the ocean, and a third has the mountains. If you put all their pieces together, you get a perfect, complete picture. This is how "multi-modal" AI works: it combines different types of data—like photos, sound, and sensor readings—to make smarter decisions. But here's the catch: in the real world, things don't always go according to plan. Maybe the camera breaks, the microphone gets wet, or a sensor runs out of battery. Suddenly, your team is missing a few friends, and the puzzle is incomplete. If your AI is only trained to work when everyone is present, it will crash and burn the moment a piece goes missing. This is a huge problem for robots, self-driving cars, and medical devices that need to work reliably even when their tools fail.

The paper you are about to read tackles this exact headache. It asks: How can we teach an AI to stay smart and accurate even when it's missing some of its data sources? The authors propose a clever new way to train these systems, not by forcing them to "glue" the missing pieces together, but by teaching the different data sources to talk to each other and help one another out. They call this "co-learning." Think of it like a study group where the student who is good at math helps the one who is good at history, so that if the history student is absent, the math student can still explain the whole story. The researchers tested their ideas on two very different challenges: identifying crop types from satellite data and recognizing human activities from wearable sensors. They found that their new methods make AI much more resilient, keeping it on track even when the data gets messy or incomplete.

The Problem: When the Team Loses a Member

In the world of artificial intelligence, "multi-modal classification" is like having a detective who uses multiple senses to solve a case. Instead of just looking at a crime scene photo, the detective also listens to audio recordings, reads witness statements, and checks weather reports. By combining all these clues, the detective (or the AI) can make a much better guess about what happened. However, real life is messy. In the field of Earth Observation, a satellite camera might be blocked by clouds, or a radar might fail. In human-centered studies, a wearable sensor might run out of battery or get disconnected.

The big issue is that most AI models are trained assuming all the clues will always be there. If you train a model with five types of data and then try to use it with only three, it often gets confused and makes terrible mistakes. Previous attempts to fix this usually focused on "robust fusion," which is like trying to build a table that stays standing even if you remove one of its legs. The authors of this paper argue that this approach has limits, especially when you have many different types of data and any combination of them might go missing. They wanted to know: What if we didn't just try to patch the hole, but instead taught the remaining data sources to cover for each other?

The Solution: A Study Group for Data

The authors introduce a framework called "Co-Learning for Missing Arbitrary Modalities." Instead of forcing the AI to fuse all data into one giant blob, they set up a team of specialized models, one for each type of data (like one for optical images, one for radar, one for weather). These models are trained to work together, sharing what they know.

They developed two specific strategies, or "methods," to make this teamwork work:

Co-Miss (Co-learning for Missing): This method is like a rigorous drill. During training, the AI is forced to practice with missing data constantly. It's as if the teacher randomly tells the history student, "You're absent today," and the math student has to explain the whole lesson anyway. The AI learns to "imitate" what the full team would have said, even when parts of the team are missing. This is called "missing distillation." It works incredibly well when only one or a few pieces of data are missing.
FullCo (Full Co-learning): This method is designed for the worst-case scenarios, where almost the entire team is missing. It uses a technique called "mutual distillation," where every single data model tries to learn from the others and from the final group decision. It's like a round-robin study session where everyone teaches everyone else. This approach shines when the AI is left with very little information, such as when only one sensor is still working.

Both methods rely on two levels of learning. At the feature level, the models learn to identify what is "shared" (like the general shape of a tree, which both a photo and a radar can see) and what is "specific" (like the texture of the bark, which only the photo can see). At the decision level, they use knowledge distillation, where the models try to agree on the final answer, helping each other stay on the right track even when data is scarce.

The Results: Stronger in the Storm

The researchers tested their ideas on two real-world datasets. The first was Multi-CropHarvest, which involves identifying crop types using four different sensors: optical images, radar, weather data, and topographic maps. The second was HL-Opportunity, a dataset with 19 different sensors on a person's body to recognize activities like "making a sandwich" or "cleaning up."

The results were promising. When the AI was tested with all its data present, it performed very well. But the real test came when they started removing data:

In the "Minimal Missing" scenario (where just one sensor failed), the Co-Miss method was the star. It barely lost any accuracy, proving that its "drill" approach prepared it perfectly for small glitches.
In the "Extreme Missing" scenario (where almost all sensors failed, leaving only one), the FullCo method took the lead. While other methods crashed and dropped their performance significantly, FullCo managed to hold its ground, showing that its "mutual teaching" strategy helped it survive the storm.

For example, in the crop recognition task, when the AI was left with only one sensor, traditional methods saw their performance drop by about 40 points. The authors' methods dropped much less, with FullCo only dropping about 24 points. In the activity recognition task with 19 sensors, the FullCo method maintained a high level of accuracy even when most sensors were gone, outperforming many other state-of-the-art approaches.

What This Means

The paper suggests that by shifting the focus from just "gluing" data together to teaching data sources to collaborate, we can build AI systems that are much more robust. The authors found that their methods consistently outperformed or matched the best existing techniques across different missing-data scenarios.

However, they also noted a trade-off. The Co-Miss method, while excellent for small missing data problems, is computationally expensive to train because it has to simulate every possible combination of missing data. If you have 10 sensors, that's over 1,000 combinations to practice! For this reason, they recommend FullCo for situations where data might be severely missing or when you have a large number of sensors.

Ultimately, this work shows that the future of reliable AI isn't just about having more data; it's about teaching the data we have how to help each other when things go wrong. Whether it's a satellite trying to see through clouds or a smartwatch trying to track your workout when the battery dies, these co-learning strategies offer a path to AI that doesn't give up when the lights go out.

Technical Summary: Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification

1. Problem Definition

Multi-modal classification leverages complementary information from diverse data sources to enhance predictive performance. However, real-world applications often face inconsistent modality availability between training and inference due to operational constraints (e.g., sensor failures, privacy restrictions, environmental factors like cloud cover).

While prior research has largely addressed bimodal setups or focused on designing robust fusion processes (e.g., Modality Dropout), a significant gap remains in handling missing arbitrary modalities. This scenario occurs when any subset of the training modalities may be absent at inference time, ranging from the loss of a single modality (minimal) to the loss of all but one (extreme). Existing methods often fail to generalize beyond specific missing patterns or bimodal constraints.

2. Methodology

The authors propose a multi-modal co-learning framework that prioritizes inter-modal collaboration over traditional multi-modal fusion. The core philosophy is to encourage each modality to contribute to and benefit from others, rather than relying solely on a central fusion mechanism.

2.1. Architecture Overview

The framework employs decision-level (late) fusion. It utilizes modality-dedicated encoders ( $E_m$ ) to process each input $X_m$ . These encoders explicitly learn two types of feature representations:

Shared Features ( $z^{sha}_m$ ): Class-relevant features invariant across modalities.
Specific Features ( $z^{spe}_m$ ): Class-relevant features unique to a specific modality.

The final prediction is obtained by averaging per-modality probability estimates ( $\hat{y}_m$ ).

2.2. Learning Criteria

The model is optimized using a combination of feature-level and decision-level loss functions:

A. Feature-Level Learning (Co-learning)

Modality Discriminant Loss ( $L_{mod}$ ): An auxiliary classifier predicts the modality source from specific features ( $z^{spe}_m$ ), forcing the model to learn modality-unique representations.
Volume-Based Contrastive Loss ( $L_{cont}$ ): Based on GRAM, this loss minimizes the volume of the Gramian matrix formed by shared features across all modalities. This enforces the shared features of different modalities to be closer in the embedding space, promoting a unified representation.

B. Decision-Level Learning (Knowledge Distillation)
The authors introduce two distinct strategies to handle missing modalities, resulting in two proposed methods:

Co-Miss (Co-learning for Missing):
- Strategy: Uses Missing Distillation.
- Mechanism: During training, the model exhaustively simulates all possible missing modality combinations ( $2^M - 1$ cases) using Modality Dropout (ModDrop).
- Objective: The prediction derived from a missing subset ( $\hat{y}_{miss}$ ) is forced to imitate both the ground truth and the consensus derived from full-modality data ( $\hat{y}_{full}$ ).
- Loss: $L_{miss} = L_{CE}(y, \hat{y}_{miss}) + \lambda \cdot L_{KD}(\hat{y}_{full}, \hat{y}_{miss})$ .
FullCo (Full Co-learning):
- Strategy: Uses Mutual Distillation.
- Mechanism: Designed for extreme missing conditions. Each individual per-modality prediction ( $\hat{y}_m$ ) is trained to imitate the full-modality consensus ( $\hat{y}_{full}$ ) and the ground truth.
- Objective: This ensures that even a single available modality can approximate the performance of the full set.
- Loss: $L_{mut} = \frac{1}{M} \sum L_{CE}(y, \hat{y}_m) + \lambda \cdot L_{KD}(\hat{y}_{full}, \hat{y}_m)$ .

3. Key Contributions

Generalization to Arbitrary Missing Patterns: The work addresses the general scenario of "missing arbitrary modalities," covering minimal, moderate, and extreme missing conditions, rather than restricting analysis to bimodal setups.
Dual-Strategy Framework: The introduction of two complementary methods:
- Co-Miss: Optimized for minimal missing conditions (single modality absent).
- FullCo: Optimized for extreme missing conditions (all-but-one modalities absent).
Feature-Decision Synergy: The integration of volume-based contrastive learning (feature-level) with knowledge distillation (decision-level) to create robust representations that function effectively even when data is incomplete.
Comprehensive Evaluation: Validation on two diverse benchmarks: Multi-CropHarvest (4 modalities: optical, radar, weather, topographic) and HL-Opportunity (19 sensor modalities for human activity recognition).

4. Experimental Results

The methods were evaluated against seven state-of-the-art competitors and standard baselines (unimodal, feature-fusion, decision-fusion).

Multi-CropHarvest (4 Modalities):
- Co-Miss achieved the best performance in full-modality and minimal missing conditions.
- FullCo outperformed all methods in moderate and extreme missing conditions.
- Both methods tied for the best average ranking (1.5) across all conditions, significantly outperforming competitors (e.g., DSensD+, DML) with a macro missing F1 score improvement of ~4 points over the closest competitor.
HL-Opportunity (19 Modalities):
- Due to high redundancy among 19 sensors, performance degradation was minimal when only one modality was missing.
- FullCo achieved the second-best results across all conditions (statistically similar to the best in full and moderate cases) and the best average ranking (2.4).
- In extreme missing conditions, FullCo remained robust, while baseline methods suffered significant performance drops.
Ablation Studies:
- The contrastive loss ( $L_{cont}$ ) was identified as the most critical component for robustness.
- Decision-level distillation was crucial for handling severe missing conditions, while feature-level collaboration was key for minimal/moderate conditions.
- The volume-based contrastive loss proved superior to standard pairwise contrastive losses for missing scenarios.

5. Significance and Claims

The paper claims that the proposed co-learning strategies effectively enhance the robustness of multi-modal classification models in real-world scenarios where data availability is unpredictable.

Complementary Strengths: The study highlights that no single method dominates all scenarios; Co-Miss excels when data loss is minor, while FullCo is superior when data loss is severe.
Practical Applicability: The methods provide a viable solution for domains like Earth Observation (where cloud cover or satellite gaps cause data loss) and human-centered sensing (where sensors may fail or batteries deplete).
Limitations: The authors acknowledge that Co-Miss has a scalability limitation ( $O(2^M)$ training complexity) due to the exhaustive simulation of missing cases, making FullCo the recommended approach for scenarios with many modalities ( $M \ge 10$ ) or extreme missing conditions.

The work concludes that combining co-learning (feature sharing) with knowledge distillation (decision alignment) offers a synergistic approach to building resilient multi-modal systems capable of handling arbitrary data incompleteness.