🤖 machine learning

BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis

This paper introduces BTI-Net, a multi-task medical image analysis framework that enhances segmentation and classification performance by establishing bidirectional decoder-level interactions gated through an uncertainty-aware mechanism to dynamically filter task-specific features without requiring additional inference passes.

Original authors: Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a medical mystery using two different tools at the same time: a microscope (to find the exact edges of a tumor) and a detective's intuition (to guess what kind of disease it is).

In most computer systems that do this, the "microscope" and the "detective" work in separate rooms. They both look at the same photo, but once they start their specific jobs, they stop talking to each other. The paper argues this is a waste. The detective's guess about the disease could help the microscope find the edges better, and the microscope's sharp edges could help the detective make a smarter guess.

The authors created a new system called BTI-Net to fix this. Here is how it works, using simple analogies:

1. The Problem: The "Silent Partners"

Think of a standard medical AI as a factory with one boss (the encoder) who gives instructions to two workers: a Boundary Worker (who draws the outline) and a Classification Worker (who names the disease).

The Old Way: The boss gives them a general briefing, and then they go to separate rooms to do their work. They never share what they find. If the Boundary Worker sees a fuzzy edge, the Classification Worker doesn't know about it. If the Classification Worker thinks, "This looks like a specific type of cancer," the Boundary Worker doesn't get that hint to look harder.

2. The Solution: The "Two-Way Talk" (TIM)

BTI-Net builds a two-way walkie-talkie between the two workers at every single step of their work.

The Boundary-to-Name Talk: The Boundary Worker sends a "spatial summary" (like, "Hey, the edge here is very jagged") to the Classification Worker. This helps the classifier refine its guess.
The Name-to-Boundary Talk: The Classification Worker sends a "global hint" (like, "Based on the shape, this is likely a benign cyst") to the Boundary Worker. This helps the boundary worker ignore confusing background noise and focus only on the relevant parts.
Progressive Refinement: They don't just talk once. They talk four times, starting with a rough sketch and ending with a high-definition drawing. Each conversation makes the next step sharper.

3. The Safety Valve: The "Uncertainty Gate" (UPA)

Here is the clever part: Sometimes, the two workers might disagree, or the image might be so blurry that their conversation is confusing. If they keep talking blindly, they might mess up the final result.

BTI-Net introduces a smart gatekeeper called Uncertainty Proxy Attention (UPA).

The Gatekeeper's Job: Before the workers share their new ideas, the gatekeeper checks three things:
1. Do they agree? (Are their new ideas moving in the same direction?)
2. How messy is the scene? (Is the image full of noise and hard to see?)
3. How confident are they? (Do they seem sure of their answer?)
The Decision:
- If the image is clear and they agree, the gate opens wide, and they swap information freely.
- If the image is messy or they are confused, the gate closes (or opens only a crack). This stops them from sharing "bad advice" that could ruin the result.
No Extra Cost: The best part is that this gatekeeper doesn't need a second look at the image or a special teacher to tell it what to do. It figures out the difficulty instantly while doing its normal job.

4. The Results: A Better Team

The authors tested this system on three different types of medical images:

Ultrasound (breast tissue)
Dermoscopy (skin lesions)
MRI (brain tumors)

In all three cases, BTI-Net performed better than previous methods.

Better Outlines: It drew the edges of tumors more accurately (higher "IoU" scores).
Better Guesses: It identified the type of disease more correctly (higher accuracy).
The "Gate" Matters: When they turned off the smart gatekeeper and forced the workers to talk constantly (even when confused), the results got worse. This proved that knowing when to talk is just as important as talking.

Summary

BTI-Net is like a medical diagnosis team where the experts don't just work in silos. They constantly check in with each other, but they also have a smart system that knows when to pause the conversation if the situation is too confusing. This allows them to produce a more accurate diagnosis and a clearer picture of the problem than systems that force them to work alone or talk blindly.

Technical Summary of BTI-Net

Problem Statement
Current Multi-Task Learning (MTL) architectures for medical image analysis predominantly rely on encoder-sharing designs, where a shared feature extractor feeds independent decoder heads. While efficient, this approach restricts cross-task communication to a coarse bottleneck, permanently discarding valuable signals once the decoders diverge. Specifically, segmentation tasks generate precise boundary cues that could disambiguate classification, while classification tasks accumulate global semantic priors that could resolve ambiguous spatial regions. Existing decoder-interaction methods (e.g., MTI-Net, DenseMTL) attempt to bridge this gap but suffer from two critical limitations: they often propagate information unidirectionally or apply fixed, per-sample blending weights. These fixed strategies fail to account for the inherent variability in medical imaging, where the reliability of cross-task interaction fluctuates significantly based on pathology type, imaging conditions, and lesion morphology. Furthermore, many existing interaction mechanisms incur high computational costs or require external annotations to adapt to sample difficulty.

Methodology
The proposed BTI-Net framework addresses these limitations by establishing bidirectional communication at every decoder level, regulated by an adaptive gating mechanism. The architecture augments an EfficientNet-B4 encoder with two primary components:

Task Interaction Modules (TIM):
TIM facilitates bidirectional information flow across four decoder resolutions ( $D_1$ to $D_4$ ), progressing from coarse semantics to fine boundary details.
- Seg $\to$ Clf Pathway: Spatial boundary context recovered by the decoder is injected into the classification vector via a gated residual. A 1 $\times$ 1 convolution reduces the decoder map, followed by global average pooling and a learned projection to generate a spatial summary. This is gated by a sigmoid function to select relevant cues per sample.
- Clf $\to$ Seg Pathway: Global semantic priors from the classification branch multiplicatively modulate the decoder feature map. The classification vector is projected to the decoder's channel space and combined with a self-gate derived from the decoder features to amplify only channels consistent with the semantic prior.
- Both pathways operate in parallel on pre-interaction features to prevent double-counting, with refined features propagating progressively through the decoder stack.
Uncertainty Proxy Attention (UPA):
Recognizing that cross-task interaction is not uniformly beneficial (e.g., on low-contrast or ambiguous inputs), UPA introduces a per-instance, per-level adaptive gate.
- Uncertainty Signals: UPA utilizes three interpretable signals derived without external annotations or additional inference passes:
  1. Cross-task alignment ( $a_\ell$ ): Measures cosine similarity between the latent shifts of both branches.
  2. Segmentation gradient energy ( $E_{seg}$ ): Captures scene complexity via the mean absolute spatial gradient of the decoder map.
  3. Classification activation spread ( $E_{clf}$ ): Distinguishes confident from uncertain predictions using the log-variance of the classification vector.
- Adaptive Gating: A two-layer MLP maps these signals to independent sigmoid weights ( $w_{seg}, w_{clf}$ ) for each task. Unlike softmax, independent sigmoids allow both tasks to be enhanced simultaneously when interaction is reliable.
- Training: The gates are trained using a difficulty-aware auxiliary loss ( $L_{gate}$ ) that aligns the gate weights with per-sample task performance (derived from soft-IoU and prediction confidence), ensuring the gate learns to suppress interaction on difficult samples.

Key Contributions

TIM: A novel bidirectional decoder-level interaction mechanism that propagates refined spatial features and semantic priors across all four decoder resolutions via channel-wise gated operations.
UPA: A per-instance, per-level adaptive gating strategy driven by three interpretable uncertainty proxies. It enables dynamic interaction strength without external annotations or Bayesian sampling overhead.
Cross-Modality Validation: Comprehensive evaluation on three diverse medical benchmarks (ultrasound, dermoscopy, and brain MRI), including the first decoder-interaction MTL evaluation on the BRISC dataset.

Experimental Results
The framework was evaluated on three public benchmarks: BUSI (breast ultrasound), HAM10000 (dermoscopy), and BRISC (brain MRI).

Performance: BTI-Net consistently outperformed both encoder-sharing baselines (e.g., MTAN, MTANet) and prior decoder-interaction methods (e.g., MTI-Net, DenseMTL).
- On BUSI, it achieved a segmentation IoU of 74.50% and classification accuracy of 93.16%.
- On HAM10000, it reached 89.80% IoU and 87.84% accuracy.
- On BRISC, it achieved 76.74% IoU and a classification accuracy of 99.10%.
Ablation Studies:
- Adaptive gating (UPA) contributed an additional +2.36 IoU over fixed bidirectional interaction.
- The complete model improved classification accuracy by up to +2.26 points over the strongest multi-task baseline.
- Removing the difficulty-aware regularization caused a drop of -0.53 IoU, confirming the necessity of direct supervision on sample difficulty.
Uncertainty Proxy Efficacy: UPA served as a reliable single-pass failure signal. On the BUSI dataset, the gate weights outperformed 30-pass Monte Carlo Dropout in both AUC-ROC and AUPRC for segmentation failure detection.

Significance and Claims
The paper claims that BTI-Net addresses a critical gap in medical image analysis by enabling cross-task refinement during spatial reconstruction, a phase previously neglected by encoder-sharing models. The significance lies in the demonstration that adaptive, bidirectional interaction is superior to fixed or unidirectional approaches, particularly in heterogeneous medical datasets where pathology and imaging conditions vary widely. The authors emphasize that their method extracts complementary task signals (boundary cues and semantic priors) without the computational overhead of stochastic sampling or the need for external difficulty annotations. The results suggest that decoder-level interaction, when gated by instance-specific uncertainty, can significantly enhance both segmentation precision and classification robustness across multiple imaging modalities.