💻 computer science

OBBSeg: Irregular Lesion Segmentation under Oriented Bounding Box Annotations

OBBSeg is a novel weakly supervised segmentation framework that leverages Oriented Bounding Box annotations, a Mask-to-OBB loss for geometric consistency, and prompt-driven semantic guidance to achieve performance comparable to fully supervised methods while significantly reducing annotation costs across diverse medical imaging modalities.

Original authors: Jun Wei, Xinchang Liu, Yu Liu, Chuhua Yang, Shuhui Wang, Hui Huang

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jun Wei, Xinchang Liu, Yu Liu, Chuhua Yang, Shuhui Wang, Hui Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to find and outline specific shapes inside a medical image, like a tumor in an X-ray or a polyp in a colon scan. To do this, the computer needs "training," which usually means a human has to draw a perfect, pixel-by-pixel outline around every single object. This is like asking a student to color in every single square of a coloring book page perfectly. It's incredibly accurate, but it takes forever and is very expensive.

The paper introduces a new method called OBBSeg that tries to find a "sweet spot" between doing nothing (which is too vague) and doing everything (which is too slow).

Here is how it works, broken down into simple concepts:

1. The Problem: The "Box" vs. The "Shape"

Usually, when people want to save time, they draw a simple square box around the object they want the computer to find.

The Flaw: Imagine trying to fit a long, skinny snake into a square box. The box covers the snake, but it also covers a lot of empty space (the background). If you tell a computer "the snake is somewhere in this square," the computer gets confused. It might draw a square-shaped snake, or it might get lost in the empty corners. This is especially bad for medical lesions that are long, thin, or tilted.

2. The Solution: The "Rotatable Box" (OBB)

The authors propose using Oriented Bounding Boxes (OBBs).

The Analogy: Instead of a rigid square, imagine a flexible, rotatable frame (like a picture frame you can tilt). You can rotate this frame so it fits snugly around that long, skinny snake.
The Benefit: This frame tells the computer two things at once: where the object is and which way it is pointing. It's much tighter than a square box, so the computer has less guessing to do. The paper found that drawing these tilted frames takes only slightly longer than drawing a square box, but gives the computer much better clues.

3. The Magic Trick: The "Mask-to-OBB" Loss

There is a catch: Even though the frame is tilted, it's still a rectangle. If you just tell the computer "draw a rectangle," it will still draw a rectangle, not the actual wiggly shape of the disease.

The Fix: The authors invented a special rule called Mask-to-OBB (M2O) loss.
The Analogy: Think of this like a "shadow puppet" game.
1. The computer tries to draw the actual shape (the puppet).
2. The M2O rule takes that puppet, flattens it, and rotates it to match the tilted frame (the shadow).
3. It checks if the shadow matches the frame the human drew.
4. Crucially, it ignores the tiny details of the puppet's edges during this check. It only cares if the puppet is the right size and facing the right way.
Why it works: This stops the computer from getting confused by the rectangular shape of the frame. It learns the orientation and size from the frame, but it's free to draw the actual, irregular shape of the disease.

4. The "Hint" System (Prompts)

To make the computer even smarter, the system uses "prompts."

The Analogy: Imagine you are playing a game of "Hot and Cold." The human gives the computer a tiny hint, like a dot or a scribble on the screen saying, "The object is right here."
The Innovation: Most systems only look at these hints at the very end of the process. OBBSeg uses these hints from the very beginning. It uses the hints to tell the computer, "Ignore the background noise; focus only on this area." It also uses a special module to sharpen the contrast between the "hot" (the disease) and the "cold" (the healthy tissue), making the disease pop out more clearly.

5. The Results

The authors tested this on 13 different medical datasets (including colon scans, skin images, and organ scans).

The Outcome: OBBSeg performed almost as well as the "perfect" pixel-by-pixel methods, but it required much less human effort to train.
The "Long Snake" Test: When looking at very long, thin lesions, OBBSeg was significantly better than older methods that used square boxes. It could handle the weird shapes much more accurately.

Summary

OBBSeg is a new way to teach computers to find medical problems. Instead of forcing humans to draw perfect outlines (too hard) or just drawing square boxes (too vague), it uses tilted, rotatable frames to give the computer a better idea of the shape's direction. It then uses a clever "shadow-checking" rule to ensure the computer learns the right size and direction without getting stuck drawing squares. Finally, it uses tiny "hints" to help the computer ignore background noise. The result is a system that is fast to train but still very accurate at finding irregular medical shapes.

Technical Summary: OBBSeg

Problem Statement

Medical image segmentation is critical for computer-aided diagnosis but faces a fundamental trade-off between annotation accuracy and efficiency. Fully supervised models (e.g., U-Net, U-Net++) require dense pixel-level masks, which are expensive and time-consuming to produce, particularly for lesions with irregular or ambiguous boundaries. Conversely, existing weakly supervised methods utilizing coarse annotations (points, scribbles, or standard axis-aligned bounding boxes) significantly reduce labeling effort but often fail to provide sufficient geometric constraints. Specifically, standard bounding boxes lack orientation information, making them inadequate for elongated or anisotropic lesions and leading to ambiguity in closely packed regions.

Methodology

The authors propose OBBSeg, an intermediate supervision paradigm that bridges the gap between weak and full supervision by leveraging Oriented Bounding Boxes (OBBs). The framework is built upon a Vision Transformer (ViT) backbone (specifically utilizing SAM2 or Res2Net) and integrates geometric constraints with prompt-driven semantic guidance.

1. Oriented Bounding Box (OBB) Supervision

Unlike standard boxes, OBBs are defined by four corner points, encoding both spatial extent and orientation. This allows for a tighter fit around elongated or tilted lesions, reducing background overlap and providing stronger geometric priors with only a marginal increase in annotation cost compared to standard boxes (17.8s vs. 13.3s per instance).

2. Mask-to-OBB (M2O) Loss

To address the inherent rectangular bias of OBB annotations, the authors introduce the Mask-to-OBB (M2O) loss. This is a differentiable, end-to-end formulation that enforces geometric consistency without iterative refinement. The process involves:

Splitting: Decomposing the predicted mask into instance-level sub-masks based on OBBs.
Affine Transformation: Rotating sub-masks to align with the OBB's orientation.
Projection: Performing max-projection along horizontal and vertical axes to eliminate fine-grained shape details, retaining only spatial extent and dimension.
Minimum Fusion & Inverse Transformation: Reconstructing a 2D mask via element-wise minimum fusion of projected vectors and rotating it back to the original orientation.
Supervision: Computing Binary Cross-Entropy and Dice loss between this transformed prediction and the OBB ground truth. This effectively removes shape bias while preserving geometric constraints.

3. Prompt-Driven Semantic Guidance

OBBSeg injects prompt information (points, scribbles, boxes, or circles) into the encoder stage, rather than just the decoder, to explicitly guide feature learning. Two complementary modules enhance this process:

Prompt-assisted Foreground Enhancer (PAFE): Uses prompt-derived binary masks to filter background noise and highlight lesion-relevant regions via element-wise multiplication and residual connections.
Differential-based Foreground Enhancer (DBFE): Explicitly amplifies foreground-background contrast by computing a differential factor between the foreground and pooled background features, thereby improving feature discriminability.

4. Loss Function

The total training objective combines three components:

$L_{M2O}$ : The proposed geometry-consistent loss.
$L_{PS}$ (Prompt Supervision Loss): Enforces hierarchical consistency between intermediate predictions and final outputs using prompt masks.
$L_{SC}$ (Scale Consistency Loss): Adopted from WeakPolyp to ensure prediction consistency across different input scales.

Key Contributions

OBBSeg Paradigm: A new segmentation framework utilizing OBBs to provide geometry-aware supervision that balances annotation efficiency with geometric precision.
M2O Loss: A noise-free, differentiable supervision strategy that mitigates the rectangular bias of box annotations, enabling geometry-consistent learning.
Prompt Integration: A unified architecture that integrates prompt-guided learning with OBB supervision via PAFE and DBFE modules to enhance foreground representation and robustness.
Comprehensive Evaluation: Extensive validation across 13 datasets and 5 imaging modalities (colonoscopy, dermoscopy, MRI, CT, ultrasound).

Experimental Results

The authors evaluated OBBSeg on 13 public datasets, including standard polyp datasets (Kvasir, ClinicDB, etc.), skin lesion datasets (ISIC), and multi-organ datasets (Synapse).

Performance vs. Weakly Supervised Methods: OBBSeg consistently outperforms existing weakly supervised methods (e.g., WeakPolyp, BoxInst, PointSup). For instance, on the Kvasir dataset, OBBSeg with OBB+Circle prompts achieved a Dice score of 95.7%, compared to 84.8% for WeakPolyp.
Performance vs. Fully Supervised Methods: The framework achieves performance comparable to, and in some cases exceeding, fully supervised models. On the Kvasir dataset, OBBSeg (95.7%) surpassed fully supervised U-Net (71.0%) and PraNet (89.8%), and was competitive with MedSAM2 (94.4%).
Boundary Accuracy: OBB supervision resulted in significantly lower HD95 (Hausdorff Distance) values compared to standard box supervision, indicating more precise boundary localization.
Elongated Lesions: The method showed particular strength in segmenting elongated lesions (aspect ratio > 2), improving performance by 5.3% over baseline weakly supervised methods.
Robustness: The system demonstrated tolerance to annotation errors, with only a 1.5% performance drop under realistic human OBB annotation deviations, compared to a 12.2% drop for standard box-based methods under similar perturbations.

Significance and Claims

The paper claims that OBBSeg offers a scalable and efficient solution for medical image segmentation by reducing the reliance on expensive pixel-level annotations. By combining the geometric fidelity of OBBs with the semantic guidance of prompts, the method effectively bridges the performance gap between weak and full supervision.

The authors emphasize that while OBBSeg achieves accuracy approaching fully supervised models, the segmentation quality is still inherently limited by the coarse nature of OBB annotations, particularly for lesions with complex contours or indistinct boundaries. The work suggests that OBBs provide a favorable balance between labeling efficiency and target coverage, making them a viable alternative for large-scale medical imaging applications where dense annotation is prohibitive. Future directions identified include adaptive label refinement and extending the framework to 3D volumes and temporal dependencies in medical videos.