💻 computer science

CandidateFusion-MKAN: A Unified Framework for Robust Multimodal Crisis Classification across Diverse Supervision and Evidence Conditions

CandidateFusion-MKAN is a unified framework that achieves robust single-output multimodal crisis classification by integrating modality-supported candidate-set learning, candidate-set likelihood, and bounded adaptive soft targets to effectively handle diverse supervision, missing or degraded evidence, and conflicting or complementary cues.

Original authors: Zequn Wang, Shanshan Li, Qingjie Liu, Zhian Pan, Guan Li

Published 2026-09-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zequn Wang, Shanshan Li, Qingjie Liu, Zhian Pan, Guan Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the chaotic hours following a disaster, emergency managers rely on a flood of information from social media to understand what is happening on the ground. A single post might contain a text message describing a collapsed bridge and a photograph showing a family stranded on a roof. Ideally, these two pieces of evidence would point to the same urgent need, such as "rescue efforts" or "infrastructure damage." However, the reality of human communication is often messier. Sometimes the text and the image tell different stories, perhaps because the writer is focused on the people while the photo captures the destruction, or because the image is blurry and the text is the only clear clue. In these moments, a computer system designed to sort these posts into categories faces a difficult choice: should it trust the text, the image, or try to guess a single answer that might ignore the truth in one of them? If the system forces a single, rigid decision too early, it risks discarding vital information that could save lives.

This is the specific challenge addressed by a new framework called CandidateFusion-MKAN, developed by researchers at the University of Emergency Management and the Big Data Center of the Ministry of Emergency Management. The team set out to build a system that could handle the messy reality of crisis data without losing the nuance of the original evidence. Instead of forcing the text and image to agree on a single label before the computer makes a decision, their system keeps both possibilities alive during the learning process. It treats the text and the image as two separate witnesses, each offering their own version of the truth, and only combines them at the very end to produce one final, reliable answer. This approach allows the system to remain robust even when the evidence is missing, degraded, or contradictory.

The researchers tested their system on a massive dataset of real-world disaster posts, known as CrisisMMD, which includes over 18,000 image-text pairs from seven different disasters. They found that in more than half of these posts, the text and the image actually supported different humanitarian categories. For example, a post might have text about "affected individuals" while the image showed "infrastructure damage." Traditional systems often struggle here, either ignoring one side or getting confused. The new framework, however, learned to preserve both categories as valid candidates. It uses a method that concentrates the computer's confidence on the set of options supported by the evidence, rather than forcing it to pick a single winner too soon. If the text and image agree, the system behaves like a standard classifier. If they disagree, it keeps both options in play, ensuring that the final decision is grounded in the actual evidence provided by both sources.

To handle the fact that real-world data is often incomplete, the system was also trained to cope with missing or poor-quality inputs. In the real world, a photo might be too dark to see, or a text message might be cut off. The researchers simulated these conditions by deliberately removing text or blurring images during testing. They found that their system could adapt seamlessly, shifting its reliance to the remaining clear evidence. When the text was missing, the system leaned heavily on the image; when the image was gone, it trusted the text. Crucially, it did this without needing to be retrained for every specific type of missing data. The system also learned to recognize when the text and image offered complementary clues—where neither was enough on its own, but together they painted a complete picture. In these cases, the system successfully combined the distinct insights from both sources to reach a more accurate conclusion than if it had treated them as redundant.

The results of the study were measured across thousands of test cases, including those where the text and image disagreed. The new framework achieved an accuracy of 92.60% on posts where the text and image agreed, matching the performance of the best existing systems. More importantly, on the difficult posts where the text and image offered conflicting information, the system maintained a high level of reliability, correctly identifying a valid category in over 90% of cases. It also significantly reduced the uncertainty in its predictions compared to older methods that simply averaged the results of text and image analysis. By keeping the evidence separate until the final moment, the system avoided the "averaging out" problem where a strong signal from one source gets diluted by a weak or contradictory signal from the other.

The researchers also looked at how the system handles rare categories, such as specific types of injuries or damage that appear in very few posts. They discovered that standard methods often lose these rare cases because they force a single label that might not be supported by both the text and the image. By keeping the candidate options open, the new system preserved the ability to recognize these rare but critical situations, even when they were only visible in one part of the post. This is vital for emergency response, where missing a rare but severe condition can have serious consequences. The study showed that the system could transfer its learning to new types of disasters it had never seen before, provided it had seen enough examples of the general types of events, such as floods or earthquakes, during training.

In the end, this work demonstrates that robust crisis classification does not require the evidence to be perfect or consistent. It requires a system that can respect the complexity of human communication and the limitations of real-world data. By treating the text and image as independent sources of truth and only merging them when necessary, the CandidateFusion-MKAN framework provides a more reliable tool for emergency managers. It ensures that when a decision is made, it is based on the full weight of the available evidence, whether that evidence is clear, conflicting, or incomplete. This approach offers a practical path forward for using artificial intelligence in high-stakes environments where the cost of a wrong guess is too high to ignore the details.

Technical Summary: CandidateFusion-MKAN

Problem Statement
Multimodal crisis classification in real-world emergency settings faces significant uncertainties where text and image evidence may support the same humanitarian category, support different categories, provide complementary cues, or become missing/degraded. Existing approaches often study these conditions in isolation or collapse inconsistent modality annotations into a single post-level label before training, thereby discarding supervision directly present in one modality. Furthermore, standard classifiers struggle to maintain robustness when inputs are incomplete or when modalities provide conflicting signals, yet the deployed system must still produce a single, reliable Top-1 prediction. The challenge is to unify the handling of label-consistent supervision, modality-supported label disagreement, complementary evidence, and missing/degraded modalities within a single-output framework without altering the final decision space.

Methodology: CandidateFusion-MKAN
The authors propose CandidateFusion-MKAN, a unified framework designed to preserve modality-supported supervision and adaptively route predictive paths based on evidence quality. The architecture and learning strategy are defined by four core components:

Modality-Supported Candidate-Set Learning (MSCSL): Instead of forcing a single ground-truth label, the framework represents the supervision for each instance as the union of categories supported by the text annotation ( $y^t_i$ ) and the image annotation ( $y^v_i$ ).
- If $y^t_i = y^v_i$ , the candidate set is a singleton (standard hard-label supervision).
- If $y^t_i \neq y^v_i$ , the candidate set contains two observed categories, preserving both signals without collapsing them.
Loss Functions (CSL and BAST):
- Candidate-Set Likelihood (CSL): The primary objective maximizes the total probability assigned to the observed candidate set $S_i$ . This concentrates probability mass on the valid categories without enforcing a specific distribution between them.
- Bounded Adaptive Soft Target (BAST): To guide the allocation of probability within a dual-candidate set, BAST provides a secondary, bounded soft target. It uses detached unimodal predictions to estimate the relative evidence strength of text vs. image, generating a target distribution $\rho_i$ that is clipped to a safe range. This guides the model to allocate probability based on modality-specific confidence without changing the label space.
Modality-Preserving Experts: The model employs three distinct predictive paths:
- Text Expert: Supervised solely by text annotations.
- Image Expert: Supervised solely by image annotations.
- Joint Expert: Supervised by the candidate set, utilizing cross-context representations (bidirectional token-level interaction), multiplicative interactions, and semantic similarity cues.
- All experts utilize frozen CLIP ViT-L/14 encoders and learnable KAN (Kolmogorov-Arnold Network) refinement layers.
Reliability-Adaptive Routing (RAR): A router dynamically combines the outputs of the three experts based on instance-level cues, including modality availability, expert confidence, entropy, predictive disagreement (Jensen–Shannon divergence), and semantic similarity. This allows the system to reallocate influence when a modality is missing or when evidence is degraded.

Key Contributions

Unified Framework: The paper organizes robust multimodal crisis classification around four recurring conditions (Label-Consistent, Label-Disagreement, Complementary Evidence, Missing/Degraded Modalities) and connects them to a single problem formulation and evaluation protocol.
Novel Supervision Mechanism: The introduction of MSCSL, CSL, and BAST allows the model to retain modality-supported categories during disagreement while maintaining a standard Top-1 interface.
Adaptive Architecture: The construction of separately supervised experts combined with reliability-adaptive routing ensures that modality-specific and cross-modal evidence remain available as input quality changes.
Comprehensive Evaluation: The framework is evaluated across label-consistent compatibility, complete-test disagreement, development-locked complementary evidence, 43 controlled missing/degraded conditions, and cross-event/disaster-type transfer.

Experimental Results
Experiments were conducted on the CrisisMMD dataset (18,082 instances), covering 973 label-consistent and 1,264 label-disagreement test instances.

Label-Consistent Supervision (Protocol A): The framework achieves 92.60% accuracy, demonstrating compatibility with conventional single-label classification.
Modality-Supported Label Disagreement (Protocol B): On the complete test set, the model achieves 90.84% AnyAcc (accuracy where the Top-1 prediction falls within the candidate set). On the specific disagreement subset, it reaches 90.74% AnyAcc. Compared to text–image probability averaging, it reduces Set-NLL by 0.048 and reduces admissibility Expected Calibration Error (ECE) from 0.214 to 0.047.
Complementary Evidence: On a subset of 463 instances with distinct but jointly useful cues, the model matches the best AnyAcc of 85.53% while reducing Set-NLL from 0.431 to 0.363.
Robustness to Missing/Degraded Modalities: The system maintains operational performance across 43 conditions. Explicit modality removal triggers clear reallocation by the router (e.g., shifting weight from text to image when text is missing), whereas continuous feature degradation affects performance more than routing weights.
Transfer Learning: In Leave-One-Event-Out (LOEO) and Leave-One-Disaster-Type-Out (LODO) settings, performance drops (e.g., 80.76% AnyAcc when holding out the hurricane family), highlighting sensitivity to source-domain coverage.

Significance and Claims
The paper claims that CandidateFusion-MKAN demonstrates robust single-output classification across diverse supervision and evidence conditions. Its significance lies in:

Preserving Supervision: It prevents the loss of minority or modality-specific evidence that occurs when inconsistent pairs are collapsed into a single label, which is particularly critical for long-tailed categories (e.g., "affected individuals" where 43.29% of modality-supported instances would be lost under standard consolidation).
Unified Handling: It addresses the gap where previous works studied disagreement, complementarity, or missing modalities separately, providing a single framework that handles all four conditions without altering the final five-class output space.
Calibration and Reliability: By concentrating probability on the observed candidate set and utilizing adaptive routing, the model produces better-calibrated predictions (lower ECE) and higher candidate mass than standard fusion baselines, even when the final Top-1 decision is correct.

The authors conclude that the framework provides a single-output interface while retaining more of the supervision and predictive evidence available in realistic multimodal crisis conditions, though they note that cross-domain transfer remains sensitive to the coverage of event families in the training data.

More like this