💻 computer science

OpenWell: Energy-Based Interactive Discovery for Open-World Medical Segmentation

OpenWell is an energy-based framework that models anatomical classes as probabilistic energy wells to enable robust out-of-distribution detection and real-time, one-shot interactive adaptation for open-world medical segmentation without requiring gradient-based retraining.

Original authors: Luisa Vargas, Zdravko Marinov, Eleonora Poeta, Maria A. Zuluaga

Published 2026-10-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Luisa Vargas, Zdravko Marinov, Eleonora Poeta, Maria A. Zuluaga

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes world of medical imaging, computers have become remarkably skilled at reading scans. They can trace the outline of a liver, count the chambers of a heart, or map the complex network of blood vessels in a brain with a precision that often rivals human experts. These systems work by learning from thousands of examples, memorizing what a healthy kidney or a specific tumor looks like. However, this strength is also a limitation. Because these models are trained on a fixed list of known structures, they operate under a closed assumption: they only recognize what they have been taught to see. When a scan contains a rare anomaly, an unusual anatomical variation, or a pathological region that was not part of the training data, the computer often fails. It might confidently mislabel a strange new growth as a known organ, or worse, ignore it entirely, leaving a dangerous gap in the diagnosis.

This problem is particularly acute in the real world, where patient anatomy varies wildly and medical conditions are unpredictable. Doctors need tools that can not only identify the familiar but also recognize when something is different. The challenge lies in teaching a machine to say, "I do not know what this is," with the same confidence it uses to say, "This is a liver." Furthermore, if the machine does spot something unknown, it should be able to learn from a doctor's quick input and immediately understand how to segment that new structure in future scans, without needing to be completely retrained from scratch. This is the frontier of open-world medical segmentation: building systems that can discover the unknown and adapt on the fly.

A team of researchers has introduced a new approach called OpenWell to tackle these challenges. Instead of forcing the computer to memorize rigid, fixed shapes, they have designed a system that treats anatomical structures as flexible, probabilistic regions in a high-dimensional space. Imagine the computer's understanding of anatomy not as a list of static templates, but as a landscape of energy wells. In this landscape, familiar organs like the heart or lungs sit in deep, narrow valleys where the computer feels most confident. These valleys are not just single points; they are shaped to account for the natural variation in how a liver might look from one person to another. When a new voxel, or tiny 3D pixel, from a scan enters this space, the system measures how much energy it would take to fit it into one of these known valleys. If the voxel sits comfortably in a valley, it is identified as a known organ. If it lands in a high-energy area, far from any known valley, the system flags it as a potential novelty.

The researchers tested this idea on a variety of complex medical datasets, including scans of the abdomen, the pancreas, and whole-body magnetic resonance images. They found that their method was significantly better at spotting unknown structures than previous models. While older systems often struggled to distinguish between a rare tumor and normal tissue, or between a new anatomical variation and background noise, OpenWell successfully identified these anomalies. It did this by calculating a score that reflects how well a piece of tissue fits the learned patterns of known organs. If the fit was poor, the system raised an alert. Crucially, the researchers also built in a safety mechanism to ensure the system didn't get confused by empty space or air, which can also look strange to a computer. They added a filter that checks if the strange-looking area is actually biological tissue before flagging it as a new discovery.

The most significant breakthrough, however, is how the system learns from a doctor. In traditional models, if a computer encounters a new type of tumor it has never seen, it cannot learn from it without a lengthy and expensive retraining process that might cause it to forget what it already knew. OpenWell changes this dynamic. When the system flags a region as unknown, a doctor can simply click on that spot. The computer then instantly analyzes the features of that specific area and creates a new energy well for it. This happens in a fraction of a second, without any complex mathematical retraining or updates to the computer's core brain. The new structure is immediately added to the system's vocabulary. In tests, this "instant enrollment" allowed the system to segment these newly discovered structures across entire scans with high accuracy. Moreover, once a new structure was learned from one patient, the system could apply that knowledge to other patients, recognizing the same new structure in different scans without needing another click.

The researchers validated their findings across four major medical imaging benchmarks, comparing their system against the best existing models, including large foundation models designed for general image recognition. OpenWell consistently outperformed these competitors in both identifying known organs and detecting unknown ones. On datasets containing thousands of scans, the system achieved higher accuracy in spotting unseen anomalies, such as rare tumors or small glands, than any other method tested. It also proved robust when doctors clicked on different parts of a structure, whether in the center or near the edge, maintaining high performance regardless of where the interaction occurred. The study suggests that by modeling anatomy as flexible energy landscapes rather than rigid templates, and by allowing for immediate, interactive learning, medical AI can move closer to the adaptability required for real-world clinical practice.

While the results are promising, the researchers acknowledge that the system is not yet perfect. A single click might not always capture the full complexity of a scattered or highly irregular pathology. Additionally, the system's performance depends on a specific setting that controls how broad the background "noise" is allowed to be, though they found the system remains stable across a wide range of these settings. The current experiments focused heavily on abdominal anatomy, and the team notes that further work is needed to see how well this approach translates to other parts of the body or different imaging technologies. Nevertheless, OpenWell represents a meaningful step forward, offering a blueprint for medical AI that can learn continuously alongside clinicians, discovering the unknown and adapting to the unique realities of every patient it encounters.

Technical Summary: OpenWell: Energy-Based Interactive Discovery for Open-World Medical Segmentation

1. Problem Statement

Volumetric medical segmentation currently relies heavily on closed-set assumptions, where the set of semantic categories is fixed and fully known during training. This paradigm fails in real-world clinical scenarios where images contain unseen anatomical variations, rare pathologies, or artifacts not present in the training distribution. When confronted with Out-of-Distribution (OOD) data, standard segmentation networks often fail or produce unsafe predictions.

Existing approaches to handling novel structures typically involve retraining or fine-tuning networks with new annotated data, a process that is computationally expensive, time-consuming, and prone to catastrophic forgetting of previously learned classes. Furthermore, many existing OOD detection methods operate as post-hoc scoring mechanisms on representations learned for closed-set tasks, leading to semantic overlap where embeddings of unseen classes fall within regions occupied by known classes. There is a critical need for systems that can identify previously unseen anatomical structures while continuing to segment known ones, and adapt to these new structures without gradient-based retraining.

2. Methodology: OpenWell

OpenWell is an energy-based framework that bridges the gap between open-world discovery and interactive adaptation. It reformulates anatomical representation learning by modeling classes as probabilistic energy wells on a unit hypersphere rather than deterministic prototypes in Euclidean space.

2.1 Hyperspherical Feature Representation

OpenWell employs a hybrid architecture (Swin-UNETR backbone with a U-Net decoder) that projects voxel embeddings onto a unit hypersphere ( $S^{F-1}$ ). This eliminates the feature norm degree of freedom, encoding voxel identity purely by direction (cosine similarity), which is invariant to global contrast and intensity perturbations common in multi-site medical imaging.

Dual-Head Design: The framework uses a dedicated binary foreground head ( $g_\phi$ ) trained to separate anatomy from air (using intensity thresholds) independently of the vMF embedding space. This prevents background semantics from contaminating the energy landscape used for class discrimination.

2.2 Probabilistic Energy Wells (vMF Distributions)

Instead of deterministic mean vectors, each anatomical class $c$ is modeled as a von Mises-Fisher (vMF) distribution:
$p(z | \mu_c, \kappa_c) = C_F(\kappa_c) \exp(\kappa_c \mu_c^\top z)$

$\mu_c$ (Mean Direction): Represents the semantic center of the class.
$\kappa_c$ (Concentration): Encodes the inverse variance (intra-class variability). High $\kappa$ indicates morphologically consistent organs (deep, narrow wells), while low $\kappa$ indicates heterogeneous structures (shallow, broad wells).
Dynamic Memory Bank: The sufficient statistics $(\mu_c, \kappa_c)$ are maintained in a memory bank updated via Exponential Moving Average (EMA). This allows the system to learn class compactness adaptively from data without end-to-end prototype optimization.
Background Constraint: A hard upper bound $\kappa_0 \le \tau_{bg}$ is imposed on the background class to prevent "semantic collapse," ensuring the background well remains shallow and broad so that true anatomical novelties are not erroneously absorbed into it.

2.3 Learning and Training

The framework utilizes a two-stage training curriculum:

Segmentation Warm-up: The model converges using only Dice-Cross Entropy (Dice-CE) loss to establish a stable segmentation solution.
Joint Geometric Fine-tuning: A vMF-Contrastive Loss is introduced with a cosine-annealing weight. This loss minimizes the energy of known classes while explicitly modeling intra-class variability, shaping the hyperspherical geometry to separate known manifolds from potential novelties.

2.4 Open-Set Detection via Gated Free Energy

To distinguish unknown organs from general background noise, OpenWell introduces a Gated Free Energy scoring mechanism:

Raw Free Energy ( $F(z)$ ): Measures the incompatibility of a voxel with the manifold of known organs.
Adaptive Threshold: A dynamic threshold $\lambda(t)$ is set based on the 95th percentile of Free Energy values for known-class voxels.
Gating: The final novelty score $S(z_v)$ is the product of the foreground probability $p_{fg,v}$ and the rectified Free Energy:
$S(z_v) = p_{fg,v} \cdot \text{ReLU}(F(z_v) - \lambda(t))$
This gate suppresses high-energy noise in regions confidently classified as background, isolating true anatomical novelties.

2.5 Interactive One-Shot Enrollment

Upon detecting a novel region, a clinician provides a single user click. OpenWell treats adaptation as a landscape modification problem:

Instant Enrollment: The system aggregates local sufficient statistics from the clicked region (weighted by a Gaussian guidance map and foreground probability) to compute a new $(\mu_{new}, \kappa_{new})$ .
No Retraining: This new tuple is instantly appended to the memory bank, "digging" a new energy well. The model can then segment this new structure globally in the current and future scans without any gradient-based retraining or backbone parameter updates.

3. Key Contributions

Probabilistic Energy-Based Framework: OpenWell models anatomical classes as probabilistic energy wells on a hypersphere using vMF distributions, capturing both semantic identity and intra-class variability.
Energy-Based Representation Learning: Introduces a vMF-contrastive training objective that explicitly models intra-class variability to support Free Energy-based novelty detection.
Thermodynamic Novelty Detection: Proposes a unified Gated Free Energy scoring mechanism that replaces heuristic distance-based thresholding with a probabilistically grounded compatibility measure, effectively filtering background noise.
Interactive One-Shot Adaptation: Develops an Instant Enrollment mechanism that converts a single user interaction into a global segmentation of unseen structures by injecting a new energy well into the memory bank, eliminating the need for retraining.

4. Experimental Results

OpenWell was evaluated on four 3D medical imaging benchmarks (AMOS, BTCV, MSD-Pancreas, TotalSegmentator-MRI) covering CT and MRI modalities.

Closed-Set Segmentation: OpenWell achieved competitive Dice scores on known structures (e.g., 91.85 on AMOS, 89.45 on BTCV), outperforming the previous deterministic prototype-based method (MedOpenSeg) by 1.3–1.7 points. This suggests the geometric constraints act as an effective regularizer.
Open-Set Detection: OpenWell significantly outperformed closed-set baselines, open-set baselines, and foundation models (MedSAM, SAM-Med2D) in detecting unseen structures.
- On AMOS, it achieved 80.40 AUROC, exceeding MedOpenSeg by 1.2 points and MedSAM by over 10 points.
- On BTCV, it achieved 71.20 AUROC, a +3.07 improvement over MedOpenSeg.
- On MSD-Pancreas, it achieved 77.50 AUROC for tumor detection.
Interactive Enrollment: After a single simulated click, OpenWell achieved Dice scores ranging from 44.8 to 57.6 for unseen structures without gradient updates.
Cross-Scan Transfer: Enrolled wells retained 91.5% to 94.6% of their source-volume Dice performance when applied to new target volumes without further interaction.
Ablation Studies:
- Removing the hyperspherical projection and vMF loss caused a catastrophic drop in AUROC (from 71.20 to 50.25).
- Removing the background concentration clamp ( $\kappa_0 \le \tau_{bg}$ ) reduced AUROC to 55.30, confirming the necessity of preventing semantic collapse.
- The Gated Free Energy mechanism significantly outperformed raw Free Energy and cosine distance-only baselines.

5. Significance and Claims

The paper claims that OpenWell fundamentally transitions medical segmentation from deterministic prototype mapping to a thermodynamic energy landscape. By explicitly shaping the embedding space during training and utilizing a memory bank of probabilistic distributions, the system can:

Detect OOD structures with high fidelity, distinguishing true anatomical novelties from background noise.
Adapt instantly to new classes via a single user interaction, bypassing the computational cost and risks of catastrophic forgetting associated with retraining.
Bridge the gap between open-set recognition and continual learning, providing a blueprint for segmentation systems that learn continuously alongside clinicians.

The authors note limitations, including potential struggles with complex, scattered pathologies using a single click, the hyperparameter nature of the background cap (though robust across tested ranges), and the current concentration on abdominal anatomy. However, the work establishes a new paradigm for open-world medical image segmentation that is both statistically principled and practically interactive.