💻 computer science

Retrieval-Augmented Visual Prompting: Guiding Foundation Models in Two-Photon Imaging

This paper introduces Retrieval-Augmented Visual Prompting (RAVP), a framework that enhances zero-shot neuron detection and instance segmentation in two-photon calcium imaging by injecting retrieved annotated exemplars as visual prompts into foundation models like SAM 3, offering an effective inference-time alternative to traditional fine-tuning.

Original authors: Salvatore Calcagno, Marco Finocchiaro, Giovanni Bellitto, Daniela Giordano, Concetto Spampinato, Federica Proietto Salanitri

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Salvatore Calcagno, Marco Finocchiaro, Giovanni Bellitto, Daniela Giordano, Concetto Spampinato, Federica Proietto Salanitri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet, dark world inside a living brain, scientists use a powerful microscope to watch neurons fire. This technique, known as two-photon calcium imaging, captures the electrical activity of individual brain cells as bright, glowing spots against a dark background. It is a cornerstone of modern neuroscience, allowing researchers to see how populations of cells work together to create thought and movement. However, turning these glowing videos into useful data is incredibly difficult. The images are messy; some cells shine brightly while others are faint, and the appearance of the cells changes drastically depending on the animal, the depth of the brain being filmed, or the specific equipment used. To study these cells, scientists must first draw a precise outline around each one, a task that is slow, expensive, and prone to human error. For years, the hope has been that artificial intelligence could learn to do this automatically, but the sheer variety of how these cells look has made it hard for computer programs to generalize from one experiment to the next.

Recently, a new generation of artificial intelligence models has emerged that can understand images with very little instruction. These are called foundation models, and they are trained on massive collections of everyday photos, learning to recognize shapes and objects without needing to be taught every single detail. One such model, designed to segment anything in an image, has shown promise in medical imaging. Yet, when researchers tried to apply this powerful tool directly to the complex, shifting world of brain cell videos, it struggled. The model, trained on cats, cars, and trees, did not naturally understand the subtle, glowing blobs of a neuron. It needed help, but the traditional way to help an artificial intelligence is to retrain it, feeding it thousands of new examples until it learns the new rules. This process is slow, requires vast amounts of labeled data, and often locks the model into a specific way of seeing the world, making it less flexible for future experiments.

A team of researchers at the University of Catania in Italy asked a different question. Instead of trying to change the brain of the artificial intelligence, they wondered if they could change the eyes through which it sees. They proposed a method called Retrieval-Augmented Visual Prompting. Imagine you are trying to describe a specific type of cloud to a friend who has never seen it. You could try to explain the shape and color with words, or you could simply show them a picture of that cloud. The researchers realized that the best way to guide the artificial intelligence was to show it a picture. They built a system where, before the model looks at a new brain image, it is shown a small, carefully chosen example of a neuron from a library of previously annotated images. This example is placed right next to the new image, acting as a visual hint. The model is then asked to find all the other neurons that look like the one in the example.

The researchers tested this idea on a massive public dataset of brain recordings from mice. They found that simply showing the model a single, well-chosen example dramatically improved its ability to find and outline neurons, even when the model had never seen that specific type of brain recording before. The key was not just showing any example, but showing the right one. They developed a smart way to pick the best example from their library by matching the brightness and texture of the example to the specific image being analyzed. When they used this method, the model's performance jumped significantly, closing the gap between a brand-new, untrained model and one that had been extensively retrained on the specific data. In fact, using a single, perfectly matched example was more effective than showing the model several different examples at once. The study suggests that for specialized tasks like analyzing brain scans, the most efficient path forward might not be to rebuild the artificial intelligence from the ground up, but to simply give it the right visual context at the moment it needs to make a decision.

The results were clear and consistent across different types of brain recordings. When the model was left to guess on its own, it often missed faint cells or drew messy outlines. But when the researchers provided a single, relevant example from their library, the model suddenly understood what to look for. It became much better at finding the faint, dim cells that usually go unnoticed and at separating cells that were touching each other. The researchers also discovered that the specific way they chose the example mattered. A random example helped, but an example chosen to match the specific conditions of the new image helped even more. They trained a small, lightweight computer program to act as a selector, learning to predict which example would be most useful for a given image. This selector allowed the system to adapt instantly to new conditions without ever changing the core settings of the main artificial intelligence model.

This approach offers a compelling alternative to the standard method of retraining artificial intelligence. Retraining requires significant computing power and time, and it often results in a model that is excellent at one specific task but forgets how to do others. The new method keeps the original model frozen and unchanged, preserving its general knowledge while allowing it to adapt to new, difficult tasks through simple visual cues. The researchers found that this method recovered a large portion of the performance gains usually achieved by full retraining, but at a fraction of the cost. It suggests that in fields where data is scarce and conditions vary wildly, the future of artificial intelligence might lie less in teaching the machine new facts and more in teaching it how to look at the world through the right lens. By injecting visual memory directly into the input, the researchers have shown that a foundation model can be guided to perform expert-level tasks in specialized biomedical imaging without the heavy burden of parameter adaptation.

Technical Summary: Retrieval-Augmented Visual Prompting (RAVP)

Problem Statement

Two-photon calcium imaging (2PCI) presents a significant challenge for foundation models due to substantial variability in image appearance across recordings, experimental conditions, and animal subjects. Key difficulties include the wide range of neuronal fluorescence (where active neurons appear as bright blobs and silent ones are nearly indistinguishable from background), domain shifts caused by different microscope setups and cortical depths, and the scarcity of expert annotations required for training. While large-scale vision foundation models like the Segment Anything Model (SAM) family offer generalizable segmentation capabilities, their few-shot prompting mechanisms do not reliably transfer to 2PCI. Specifically, defining concepts via bounding boxes within the target image requires specialized manual annotation and fails to capture the full appearance variability of the target domain. Furthermore, it is unclear which specific examples are most informative for a given target image.

Methodology

The paper proposes Retrieval-Augmented Visual Prompting (RAVP), a framework designed to guide SAM 3 at inference time by injecting external visual memory directly into the model's input, without modifying model weights or architecture.

Core Mechanism

RAVP employs a retrieval-augmented image composition strategy:

Library Construction: An external annotated library $L$ is built from a held-out split of data, containing image crops centered on annotated neurons, their ground-truth masks, and mean fluorescence values.
Image Composition: For each target tile $T_p$ from a 2PCI max projection, a composite image $\tilde{I}_p$ is constructed. The target tile is placed at the center of a canvas, and a border is tiled with exemplar crops retrieved from $L$ .
Visual Prompting: The bounding boxes of the annotated neurons in the border region (the retrieved exemplars) serve as concept prompts for SAM 3's Promptable Concept Segmentation (PCS) head. The model is tasked with detecting and segmenting all instances matching the concept defined by the exemplar across the entire canvas.
Output Restriction: Predictions are restricted to the original target region via an inverse coordinate map.

Exemplar Selection Strategies

Recognizing that not all exemplars are equally informative due to fluorescence variability, the authors study several selection strategies:

Random: Sampling exemplars uniformly from the library.
Fixed Fluorescence: Selecting exemplars based on a fixed target fluorescence value (empirically set to low values to avoid bias toward only high-signal structures).
Image-Adaptive: Selecting exemplars whose mean fluorescence matches the mean intensity of the current target tile.
Recall Predictor ( $\pi_\theta$ ): A lightweight predictor trained offline to estimate the recall SAM 3 would achieve for a given target-exemplar pair. The predictor uses visual features extracted by a frozen ResNet-18 backbone and is supervised by actual SAM 3 inference outcomes on the training split. At inference, it selects the exemplar maximizing the predicted recall.

Implementation Details

Sliding Window: Due to SAM 3's constrained prediction capacity, images are processed using a sliding window approach with 50% overlap, followed by Non-Maximum Suppression (NMS).
Configuration: Experiments utilize a single exemplar crop per tile, placed randomly on the top or left border, with the rest of the border filled with black padding.

Key Contributions

RAVP Framework: A retrieval-augmented image composition strategy that transforms an external exemplar library into visual memory for SAM 3, enabling inference-time adaptation without weight updates.
Systematic Selection Analysis: A comparative study of exemplar selection strategies, demonstrating that matching exemplars to the target's fluorescence regime improves performance.
Lightweight Recall Predictor: The introduction of $\pi_\theta$ , a model trained to select informative exemplars at inference time without requiring ground-truth annotations for the target, using SAM 3's own inference outcomes as a supervision signal.
Comprehensive Evaluation: Extensive experiments on the Allen Brain Observatory (ABO) showing that visual memory injection consistently strengthens zero-shot performance, offering a competitive alternative to parameter adaptation.

Experimental Results

Experiments were conducted on the Allen Brain Observatory dataset, split into in-domain, out-of-domain (OOD), and STNeuroNet subsets.

Zero-Shot vs. Adaptation: Zero-shot SAM 3 achieved modest precision (AP50 ≈ 0.28) but high recall. Full fine-tuning, encoder-only fine-tuning, and LoRA significantly improved performance (AP50 ≈ 0.68–0.69).
RAVP Performance: In the zero-shot regime, RAVP significantly outperformed the baseline.
- Even random selection improved AP50 from 0.28 to 0.45.
- The Recall Predictor achieved the best results among zero-shot methods, reaching AP50 = 0.53 and a harmonic mean (H) of 0.62 on the in-domain test set.
- On the OOD split, RAVP improved the harmonic mean from 0.20 (zero-shot) to approximately 0.58–0.60, demonstrating robustness to domain shift.
Comparison to Fine-Tuning: While RAVP did not fully close the gap to fine-tuned models, it recovered a large fraction of the performance gains at a significantly lower adaptation cost (estimated at 42% of full fine-tuning and 55% of LoRA adaptation).
Ablation Studies:
- Exemplar Count: A single carefully selected exemplar ( $k=1$ ) was found to be more effective than multiple exemplars ( $k > 1$ ), which led to performance degradation.
- Library Size: Increasing the library size from 1 to 50 candidates yielded most of the gains, with diminishing returns beyond that.
- Selection Strategy: Under strong domain shift (OOD), the specific selection policy mattered less than the presence of the retrieval mechanism itself, though the Recall Predictor generally performed best in-domain.

Significance and Claims

The paper positions inference-time visual memory injection as a simple and effective alternative to parameter adaptation for foundation models in specialized biomedical imaging. The authors claim that RAVP allows for adaptation through input design alone, leveraging the native Promptable Concept Segmentation interface of SAM 3.

The work suggests that in domains where annotated corpora are rare and appearance variability is high, injecting retrieved exemplars directly into the image plane can substantially improve zero-shot detection and instance segmentation. The authors note that while the Recall Predictor requires an offline training phase using SAM 3 inference outcomes, simpler selectors (like image-adaptive or fixed fluorescence) avoid this cost while remaining competitive. The approach is presented as a flexible alternative to weight-based adaptation, with potential applicability beyond 2PCI to other biomedical domains where foundation models struggle to generalize.