💻 computer science

ProtoRefine: Prototype-conditioned refinement of strawberry instance masks with native-area decoding

ProtoRefine is a lightweight, prototype-conditioned refinement method that corrects strawberry instance mask errors caused by occlusion by reusing existing instance representations to reallocate pixel locations while preserving the original detection's count and confidence, thereby significantly improving mask accuracy on the StrawDI dataset with minimal additional parameters.

Original authors: Minhui Kang, Yiyao Zhang, Jiayun Zheng, Peizhe Long, Jiafeng Qi

Published 2026-11-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minhui Kang, Yiyao Zhang, Jiayun Zheng, Peizhe Long, Jiafeng Qi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet complexity of a strawberry patch, the fruit rarely presents itself in isolation. Nestled beneath broad green leaves, tangled among twisting stems, and crowded by neighboring berries, a single strawberry is often only partially visible to the human eye. For a computer trying to see the same scene, this clutter creates a significant challenge. The goal is not merely to find the fruit, but to draw a precise outline around the visible part of it, separating the red skin from the green foliage that touches it. This task, known as instance segmentation, is essential for modern agriculture, where farmers rely on accurate images to estimate fruit size, count the harvest, and determine ripeness. When a computer program draws a mask around a strawberry, it is essentially coloring in the pixels that belong to that specific berry. However, in the messy reality of a canopy, these digital outlines often go wrong: they might include a stray leaf as part of the fruit, or they might fail to capture a small, exposed patch of red that is clearly visible. Correcting these mistakes without losing the fruit's original location is the central problem researchers are trying to solve.

A team of researchers has developed a new method called ProtoRefine to fix these imperfect outlines. Their approach starts with a computer vision system that has already successfully located the strawberries. The system has done the hard work of finding the fruit and drawing a rough box around it, but the detailed shape inside that box is still flawed. Instead of building a new, heavy system from scratch to re-examine the entire image, the researchers created a lightweight add-on that acts like a fine-tuning tool. This tool takes the information the original system has already gathered—specifically, the general features of the image and the specific characteristics of the strawberry it just found—and uses them to redraw the mask with much greater care. The process is efficient because it does not need to look at the whole picture again; it only focuses on the small, local area where the fruit sits, using the existing data to make a better decision about which pixels belong to the fruit and which belong to the background.

The core of this improvement lies in how the system decides which pixels to keep. The researchers found that simply letting the computer guess the best shape often leads to errors in the total amount of fruit it thinks it sees. To solve this, they introduced a rule called native-area decoding. This rule forces the refined mask to keep the exact same number of "fruit pixels" as the original, rough mask. Imagine the original mask has a budget of 7,578 pixels to spend on the fruit. The new system cannot add more pixels or remove any from that total; it can only move them around. It uses a learned sense of what the fruit looks like to decide that a pixel currently covering a leaf should be moved to a spot where a piece of fruit is actually visible. By rearranging the pixels within a fixed budget, the system corrects the shape of the outline, removing the green intrusions and filling in the missing red spots, all while keeping the total size of the detected fruit exactly the same as before.

When tested on a standard set of strawberry images, this method proved highly effective. On a dataset containing 200 images with over 1,100 annotated strawberries, the refined system achieved a mask accuracy score of 84.03. This was an improvement of 1.32 percentage points over the original system that did not use the refinement tool. The improvement was even more noticeable when looking at the strictest measures of accuracy, where the outlines must match the ground truth almost perfectly, showing a gain of nearly three points. The system achieved this with a very small addition to the computer's memory, requiring only about 25,000 extra adjustable settings, which is a tiny fraction of the total size of the main system. This demonstrates that significant gains in precision can be made by carefully reusing information the computer has already computed, rather than by building larger, more complex models.

The researchers also discovered that the way the system interprets its own predictions matters greatly. They tested whether giving the system more detailed information about the image features would help, but found that the benefit depended entirely on the rule used to draw the final line. When the system was allowed to change the total number of pixels in the mask, the extra details helped improve the result. However, when the system was forced to keep the pixel count fixed, as in the native-area decoding method, those extra details provided very little additional benefit. This finding suggests that the value of complex image features is tied to how the final shape is constructed. If the goal is to correct the shape without changing the size, the simpler information the system already has is often sufficient. The study confirms that for strawberries hidden in a canopy, the best path forward is not necessarily to see more, but to rearrange what is already seen with greater precision.

Technical Summary: ProtoRefine

Problem Statement
In strawberry canopy images, instance segmentation often suffers from errors where leaves, stems, or neighboring fruits obscure contours. Even when a detector correctly localizes a fruit, the resulting mask may incorrectly include surrounding vegetation or miss visible fruit regions. Correcting these errors requires refining foreground–background assignments around already localized instances without altering the detection outputs (bounding boxes, classes, or confidence scores). While existing methods like YOLO26s-seg utilize prototype-based representations to generate coarse masks, there is a need to determine if these existing representations can be reused for compact local refinement without requiring additional image encoders.

Methodology: ProtoRefine
The authors propose ProtoRefine, a compact local refinement branch integrated into the native YOLO26s-seg mask path. The method operates as follows:

Input Representation: The system reuses the native segmenter's existing representations: spatial prototypes ( $P$ ), instance coefficients ( $c_i$ ), and coarse foreground logits ( $z_i$ ). For selected small instances, the predicted bounding box is expanded, and these features are sampled onto a shared 64×64 local grid.
Shared Local Predictor: A shared convolutional predictor combines the spatial prototypes, instance coefficients, coarse logits, and normalized local coordinates into a single input vector. This predictor uses a small architecture (two 3×3 convolutions with ReLU and a 1×1 projection) to predict dense foreground logits. Crucially, it shares weights across instances and requires no additional image encoder.
Native-Area Decoding: A key innovation is the decoding strategy. Instead of applying a free threshold to the predicted logits (which allows both foreground quantity and location to change), native-area decoding preserves the foreground pixel count ( $K_i$ ) of the original native mask. It reallocates these $K_i$ pixels to the locations with the highest predicted scores within the admissible domain (the union of the local support and the original foreground). This separates spatial reallocation from changes in foreground quantity.
Training and Inference: The native network is frozen; only the local predictor is trained using binary cross-entropy and a soft Dice loss. During inference, routing rules select specific candidates (e.g., small instances) for refinement, while other candidates retain their native masks.

Key Contributions

Compact Refinement from Native Representations: The paper demonstrates that spatial prototypes, instance coefficients, and coarse-mask evidence can be combined in a shared local predictor to refine masks without an additional image encoder or changes to detection outputs.
Reconstruction with Fixed Foreground Count: The introduction of "native-area decoding" provides an explicit rule for correction at a fixed count, preserving the native mask's foreground budget while using learned scores to optimize pixel placement.
Analysis of Representation and Decoding Dependence: Through controlled comparisons across three local-head initializations and varying input configurations (coarse vs. full inputs), the study establishes that the additional value of explicit prototype factors is conditional on the decoding rule used.

Experimental Results
Evaluated on the official StrawDI test set (640-pixel input), ProtoRefine achieves the following:

Performance: It achieves 84.03 mask Average Precision (AP), an improvement of 1.32 percentage points over the native YOLO26s-seg baseline.
Strict Overlap: It improves AP90 by 2.90 points and small-object AP (APS) by 2.04 points.
Efficiency: The refinement adds only 25,144 trainable parameters (approx. 0.22% of the native model).
Geometric Quality: On matched development instances, native-area decoding improves mean mask IoU (86.15% to 87.28%) and boundary F1 (87.95% to 89.77%) compared to native masks.
Ablation Findings:
- Explicit prototype factors provide larger AP gains under free decoding than under native-area decoding, suggesting their utility depends on how scores are reconstructed.
- Native-area decoding reduces omission (missed fruit) compared to free decoding but retains more spill (background inclusion) because it cannot correct the total foreground count if the native mask overestimates the area.
- While higher-resolution native inference (896px) yields higher AP (84.81), ProtoRefine improves quality at the lower 640px resolution without increasing input size.

Significance and Claims
The paper claims that existing instance representations in prototype-based segmenters can support compact local mask correction without additional encoders. It highlights that the incremental benefit of exposing explicit prototype inputs is linked to the reconstruction method (decoding rule). The authors modestly note that while the method improves mask overlap and boundary quality, it inherits quantity errors from the native mask (e.g., if the native count is wrong, the refinement cannot fix the total area). Furthermore, the study does not claim that improved masks automatically translate to better size estimation or harvesting decisions, noting that such applications require task-specific validation. The work establishes a framework for refining masks within a fixed detection pipeline, emphasizing the interplay between representation reuse and binary reconstruction strategies.

Technical Summary: ProtoRefine

More like this