🤖 AI

Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models

This paper proposes a domain-knowledge-free, neurosymbolic fusion framework that leverages Label Vector Pools and consistency-based abduction to robustly combine pre-trained ViT detectors, achieving superior performance over majority voting baselines under both distributional shifts and coordinated adversarial attacks.

Original authors: Mario Leiva, Yue Ma, Qinru Qiu, Gerardo Simari, Paulo Shakarian

Published 2026-08-06

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mario Leiva, Yue Ma, Qinru Qiu, Gerardo Simari, Paulo Shakarian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship navigating through a stormy, alien galaxy. You have a crew of six expert sensors, each trained to spot specific landmarks like rocks, ships, or space stations. But here's the catch: your sensors were all trained in a calm, sunny solar system. Now, you are flying through a chaotic nebula filled with weird fog, swirling dust, and strange lighting. Your sensors are confused. They start shouting out conflicting reports: "That's a rock!" "No, it's a ship!" "Actually, it's a giant space-bear!"

This is the real-world problem scientists face with Artificial Intelligence. We have built incredibly smart computer programs (called "perception models") that can identify things in photos, like cars or people. But these programs are like our space sensors: they are experts at what they've seen before, but they get easily confused when the world changes. If you take a self-driving car trained on sunny California roads and drive it through a blizzard in Minnesota, it might start seeing snowflakes as pedestrians or stop working entirely.

To fix this, engineers usually try to get multiple AI models to work together, like a committee. The most common way to make a committee decision is Majority Voting: if three out of six models say "It's a car," the system decides it's a car. It sounds fair, right? But there's a big flaw. If a sneaky hacker (or just a really bad storm) tricks a few of the models into agreeing on the wrong answer, the whole committee gets fooled. It's like if three friends in a group chat all decide to prank you by saying the sky is green; if you just go with the majority, you'll believe the sky is green.

This paper introduces a smarter way to run that committee. Instead of just counting votes, the new system acts like a detective who checks for logical consistency. It asks: "Does this story make sense?" If one model says "It's a car" and another says "It's a cloud," the system doesn't just count heads; it looks at the evidence to see which story holds together without contradictions. The authors show that this method is much harder to trick and works better when the environment is totally new.

The Detective's New Toolkit: Learning Without a Reference Guide

The researchers, Mario Leiva and his team, wanted to solve two big problems. First, previous methods needed a "reference guide" of human knowledge to spot errors. For example, a human might tell the computer, "Pedestrians are usually on the sidewalk, not in the sky," or "Cars are bigger than this." But what if you are in a totally new environment where you don't know the rules? You can't write a reference guide for a place you've never seen.

Second, they wanted to stop the "Majority Vote" from being easily hacked.

The Solution: The "Label Vector Pool"
To solve the first problem, the team invented a way for the AI to learn its own reference guide without any human help. They used a clever trick involving the "memory" of each AI model.

Imagine each AI model has a mental library of every object it has ever seen during its training. When it sees a "car," it doesn't just store the word "car"; it stores a unique mathematical fingerprint (an embedding) of what that car looked like. The researchers gathered all these fingerprints for each type of object and grouped them into little clusters called Label Vector Pools.

Think of it like a "mood ring" for objects. If a model sees a new object, it checks: "Does this object's fingerprint look like the 'car' mood ring, or does it look like the 'tree' mood ring?" If the object is supposed to be a car but its fingerprint is way off—like a car that looks like a cloud—the system flags it as suspicious. This happens automatically, using only the math inside the model, with no need for human rules about sidewalks or sizes.

The Solution: The "Consistency Detective"
Once the system knows which predictions are suspicious, it doesn't just throw them away. Instead, it uses a method called Abductive Reasoning. This is a fancy way of saying "inference to the best explanation."

Imagine you are a detective trying to solve a mystery with six witnesses.

Majority Voting would just ask, "Who agrees with the most people?" If three witnesses say "The butler did it," the detective arrests the butler.
This New System asks, "Who is telling a story that doesn't contradict the others?"

If three witnesses say "The butler did it," but one witness says "The butler was in the kitchen at the time," and another says "The butler is allergic to the weapon," the system realizes that the "butler did it" story has holes in it. It might decide to trust the witnesses who are consistent with each other, even if they are in the minority. It builds a logical puzzle where every piece must fit perfectly. If a piece (a prediction) makes the whole picture fall apart, it gets rejected, even if many people voted for it.

The Big Test: Storms and Sneaky Hackers

The team tested this idea on a tough challenge: aerial images of a city under 15 different weather conditions, from heavy rain to dust storms. They used six different AI models, each trained on just one type of weather. When they threw them all into the mix, the weather was a mess.

The Clean Data Result
First, they tested the system in normal, non-hacked conditions. They found that their new "Consistency Detective" performed just as well as the best "Majority Vote" teams. It didn't lose any accuracy; it was just as good at spotting cars and people as the old methods.

The Attack Test
Then, they brought in the bad guys. They simulated a coordinated attack. Imagine a hacker who can't break the AI models individually, but can trick a small group of them into all shouting the same wrong answer at the same time.

The Majority Vote Team: When the hacker made 90% of the objects get a "flipped" label (a wrong label forced by the attack), the Majority Vote system collapsed. Its accuracy dropped to 0.35 on average across the test sets. It was completely fooled because the hackers just needed to control a few models to sway the majority.
The Consistency Detective: The new system didn't budge as much. Even with the same 90% attack, it kept an average accuracy of 0.42 across all 15 test sets. That might not sound like a huge difference, but in the world of AI, that's a 22% improvement over the old way.

The reason? The hackers tried to force a majority, but the Consistency Detective looked at the logic. If the hackers forced three models to say "It's a cloud," but the other three models (and the internal logic of the system) said "That doesn't make sense," the detective ignored the hackers. It trusted the story that made sense, not the number of voices.

Why This Matters

The paper shows that you don't need to be a human expert to build a safety net for AI. By letting the AI learn its own "fingerprint" rules and by checking if the stories it hears are logically consistent, you can build a system that is:

Portable: It works in new, weird environments without needing a manual.
Tough: It can't be easily tricked by hackers who try to rig the vote.

The researchers ran these tests on a standard computer server, and the system was fast enough to be practical. The "Consistency Detective" (specifically their "IP+TB" method) took a bit longer to solve the puzzle than the simple vote counter, but it was still quick enough to be used in real-time.

In the end, this paper suggests that when we are sending AI into the unknown—whether it's a disaster zone, a remote planet, or a city under attack—we shouldn't just listen to the loudest voice. We should listen to the voice that makes the most sense.

Technical Summary: Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models

Problem Statement

Deploying pre-trained perception models (specifically Vision Transformers) in novel environments—such as disaster zones or remote regions with no representative training data—leads to significant accuracy degradation due to distributional shifts. While assembling multiple models is a common strategy, simple combiners like majority voting (MV) are brittle: they trade recall for precision and are highly susceptible to coordinated failures where an adversary manipulates a minority of models to flip the collective vote.

Prior metacognitive approaches attempt to mitigate this by learning logical rules to flag errors. However, these methods rely on hand-authored domain knowledge (e.g., object-size priors, segmentation masks) that do not transfer to genuinely novel scenes. Furthermore, previous work did not adequately address scenarios where perceptual models are subject to adversarial perturbations designed to defeat voting-based ensembles.

Methodology

The proposed framework addresses these limitations by framing the fusion of imperfect detectors as a consistency-based abduction problem solved at test time. The system operates in two distinct phases:

1. Offline Training: Domain-Knowledge-Free Metacognitive Layer

Instead of relying on hand-crafted domain predicates, the authors learn an error-detection signal using only the internal embeddings of each pre-trained model.

Label Vector Pools (LVP): For each model $f_i$ and class $c$ , the system clusters the model's training detection embeddings (using $k$ -means, $k=3$ ) to create a pool of prototype vectors $\{ \mu_{i,c}^1, \mu_{i,c}^2, \mu_{i,c}^3 \}$ .
Error Detector: For a new detection with embedding $e$ , the system computes Euclidean distances to all prototypes in the pool, forming a feature vector $\phi(e)$ . A Random Forest classifier, trained per (model, class) pair, maps these distances to an error probability $s_{i,c}(e)$ .
Rule Learning: These probabilities are thresholded to generate Error Detection Rules (EDR). A rule fires if the error probability exceeds a learned threshold $\tau$ , which is controlled by a strictness hyperparameter $\epsilon$ . This layer requires no external scene artifacts (masks, priors) and is transferable to any new model or scene.

2. Online Inference: Consistency-Based Abduction

At test time, $\eta$ models perceive a novel scene. Their predictions, combined with the per-model EDR programs, form a logical program $\Pi$ .

Logical Formulation: The system seeks a subset of accepted predictions $H$ $H$ that maximizes coverage (recall) while minimizing logical inconsistency.
- Integrity Constraints: The only domain knowledge used is the mutual exclusivity of class labels (an object cannot have two different classes).
- Objective: Maximize $Pred(H)$ (the number of accepted assignments) subject to $Inc(H) \le \delta$ , where $Inc(H)$ is the normalized count of violated integrity constraints.
Solvers: The problem is solved via:
- Exact Integer Program (IP): Solves the optimization globally.
- Heuristic Search (HS): A polynomial-time greedy algorithm that iteratively selects the best (model, class) pair to accept based on coverage gain vs. inconsistency budget.
- Tie-Breaker (TB): An optional step to resolve ambiguous labels by selecting the prediction with the highest confidence.

Key Contributions

Domain-Knowledge-Free Metacognition: The paper demonstrates that error-detection rules can be learned entirely from the geometry of a model's own training embeddings (via LVP). This removes the need for hand-authored cues, making the system portable to novel environments where segmentation masks or size priors are unavailable. The LVP-based approach achieves parity with domain-knowledge baselines (within 0.002 F1) without using any domain knowledge.
Robustness to Coordinated Attacks: By shifting from vote-counting (majority voting) to consistency-based acceptance, the framework is inherently more robust to coordinated label-flip attacks. While majority voting fails when a minority of models are forced to agree on a wrong label, the abductive approach accepts predictions only if they are consistent with the logical constraints and other models' outputs.
Scalable Abductive Fusion: The authors provide both an exact IP formulation and a scalable polynomial-time heuristic (HS) for solving the abduction problem, demonstrating that consistency-driven fusion is computationally feasible for real-world detection scenarios.

Experimental Results

The framework was evaluated on the MDS-A (Multiple Distribution Shift – Aerial) dataset, comprising 15 weather-shifted test sets and six ViT-based detectors, as well as a multimodal VisDrone-DroneVehicle dataset (RGB + IR).

Clean Data Performance: On unattacked data, the proposed methods (IP+TB and HS+TB) perform statistically on par with the strongest majority-vote baseline (MV-Plurality), achieving comparable F1 scores (within 0.005 F1) while outperforming single-model baselines.
Adversarial Robustness: Under a coordinated label-flip attack (where an adversary forces a subset of models to agree on a wrong label):
- Majority voting degrades significantly as the flip rate ( $p$ ) increases.
- The abductive approach degrades gracefully. At a 90% flip rate, the proposed method achieves an average F1 of 0.42, compared to 0.35 for MV-Plurality (a 22% relative gain).
- Once the flip rate exceeds 0.4, the proposed method attains the highest F1 on every test set.
Runtime: The Heuristic Search (HS) variant processes scenarios in 1.9–19.7 seconds with a per-object solve time of 1.4–3.0 ms, making it suitable for tight latency budgets. The Exact IP is slower but remains practical for offline or less time-critical applications.

Significance and Claims

The paper claims that the metacognitive layer of a multi-model perception system can be effectively learned without domain knowledge, relying solely on the geometric properties of training embeddings. By framing model integration as a consistency-based abduction problem, the authors demonstrate a combiner that is:

Competitive on clean data: Matching the performance of standard ensemble methods without relying on voting.
Resilient to adversarial coordination: Maintaining performance under attacks that specifically target the independence assumptions of majority voting.

The authors emphasize that this approach allows the same abduction machinery to transfer across domains without re-engineering error-detection rules, as the rules are derived from the models' own internal representations rather than external scene semantics. While the LVP and domain-knowledge (DK) rules catch different types of errors (embedding anomalies vs. geometric/contextual mistakes), the paper notes that combining them did not yield a significant F1 improvement in their experiments, though it remains a direction for future work.