💻 computer science

Cross-view agreement rescoring improves tiled small-object detection without retraining

This paper introduces a training-free, CPU-based cross-view agreement rescoring method that significantly improves small-object detection in tiled inference by leveraging discarded overlap signals to correct confidence score rankings, achieving consistent gains across diverse detectors and datasets without additional forward passes.

Original authors: Hyun-Woo Na, Eunkyung Kim, Haneol Jang

Published 2026-09-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyun-Woo Na, Eunkyung Kim, Haneol Jang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a drone flying high above a disaster zone, its camera capturing a vast landscape of rubble and debris. The mission is to find survivors, but the people they are looking for might appear as nothing more than a few pixels in a photograph containing millions of pixels. This is the daily reality for many automated vision systems used in conservation, security, and disaster response. To see these tiny details, computers often use a technique called tiled inference. Instead of trying to analyze the entire massive image at once, the software chops the picture into many smaller, overlapping pieces, examines each piece closely, and then tries to stitch the findings back together. It is a standard approach, but it has a hidden flaw. When the computer looks at the same person from different overlapping pieces, it assigns a different level of confidence to each sighting. Sometimes it is very sure, sometimes less so. When the system tries to merge these results, it simply ranks them by those confidence numbers, often discarding the correct sightings because they happened to get a slightly lower score in one of the views. The boxes that contain the people are already there, but the computer is looking at them in the wrong order.

Researchers at Hanbat National University have discovered a way to fix this sorting error without needing to retrain the computer model or add any new processing steps. They found that the information needed to correct the ranking was already being calculated and then thrown away. When a real object, like a person or a vehicle, appears in multiple overlapping views, the computer detects it in almost every piece of the puzzle where it is visible. In contrast, false alarms caused by confusing textures, like a patch of road that looks like a person, usually appear in only one view and vanish in the others. The team realized that by counting how many different views successfully re-detected the same object, they could boost the confidence score of the real sightings and lower the score of the false ones. They call this method cross-view agreement rescoring. It acts like a simple filter that sits between the detection phase and the final merging phase, adjusting the scores based on how consistently an object appears across the different camera angles.

The results of this approach are striking because they require no extra training data and no additional time for the computer to look at the image again. The researchers tested their method on a standard benchmark dataset used for drone photography, which contains thousands of images of small objects. Using a common detection model, they found that simply adding this agreement check improved the system's ability to find small objects by a significant margin. In their most rigorous test, the improvement was consistent and measurable, with the system finding more correct objects and fewer false alarms. The method worked so well that it outperformed more complex, recently developed techniques that try to change how the boxes are merged. The key insight was that the problem was not about finding better boxes or merging them more cleverly, but about trusting the right boxes in the first place. By normalizing the score based on the number of views that could have actually seen the object, the system learned to ignore the noise and focus on the signal.

This solution is remarkably efficient. It does not slow down the main processing unit that does the heavy lifting of image analysis. Instead, it runs as a quick, secondary step on a standard processor, adding only a fraction of a second to the total time it takes to process an image. The researchers tested this on a wide variety of scenarios, including different types of cameras, different sizes of objects, and even different models of detection software. In nearly every case, the method improved the results. It worked on images of tiny satellites, large aerial maps, and open-vocabulary systems that can detect objects it has never seen before. The improvement was not limited to a specific type of object; it helped wherever the same scene was viewed from multiple angles. The researchers noted that the method is particularly good at cleaning up the edges of the image, where objects are often cut off and confused with the background.

The study also clarified what this method cannot do. It cannot create new detections if the original software missed the object entirely. It also cannot fix false alarms that are actually real objects, such as a bicycle that is mistaken for a motorcycle; if the computer sees a bicycle in every view, the method will correctly boost its score, even if the label is wrong. However, for the vast majority of cases where the computer is confused by the way the image is chopped up, this simple check provides a powerful correction. The researchers demonstrated that the gain in accuracy comes from re-ordering the list of potential detections, not from finding new ones. By giving more weight to the sightings that are confirmed by multiple views, the system naturally pushes the true objects to the top of the list and pushes the confusing background noise to the bottom.

This work suggests that the future of small-object detection may not lie in building bigger, more complex models, but in better understanding the data that is already being produced. The researchers showed that a single, fixed setting for this agreement check worked across all their tests, from drone footage to satellite imagery, without needing to be adjusted for each new situation. This robustness indicates that the problem of mis-ranked scores is a fundamental part of how tiled inference works, and that the solution is a universal property of overlapping views. For anyone relying on automated vision to find small things in large spaces, this offers a practical and immediate way to improve performance. It turns the redundancy of overlapping views from a waste of computing power into a reliable tool for accuracy, ensuring that the computer trusts what it sees most often.

Technical Summary: Cross-View Agreement Rescoring for Tiled Small-Object Detection

Problem Statement

Tiled inference is the standard approach for detecting small objects in large images (e.g., drone or satellite imagery), where objects may occupy only a dozen pixels within a tens-of-millions-pixel frame. The standard pipeline involves slicing a large image into overlapping crops, running a detector on each, and merging the results. While methods like Slicing-Aided Hyper Inference (SAHI) have refined where to crop and how to merge boxes (e.g., Soft-NMS, weighted box fusion), this paper identifies a critical, overlooked defect in the pipeline: score mis-calibration.

Because the same object is viewed at different effective resolutions (full image vs. individual tiles), the confidence scores produced are not mutually comparable. Since Average Precision (AP) is a ranking metric, the merging stage's reliance on a single axis of confidence scores leads to significant accuracy loss. The paper argues that the primary bottleneck is not the selection of bounding boxes, but the ranking of those boxes.

Methodology: Cross-View Agreement Rescoring (CAR)

The authors propose Cross-View Agreement Rescoring (CAR), a post-processing technique that repairs score rankings without retraining, additional labels, or extra forward passes.

Core Mechanism

CAR operates between the detection stage and the merging stage. It modifies the confidence score $s$ of a detection $d$ by multiplying it with an agreement factor derived from overlapping views:
$\hat{s}(d) = s(d) \cdot (1 + \lambda \cdot \text{agr}(d))$
Where:

$\lambda$ is a fixed hyperparameter (set to 5.0).
$\text{agr}(d)$ is the agreement ratio, defined as the fraction of the effective view set $C(d)$ that re-detects the object.

The Effective View Set ( $C(d)$ )

A critical design choice is the definition of the denominator in the agreement calculation.

Eligibility: A view is in $C(d)$ only if the detection's bounding box lies fully inside the view's crop with a specific margin ( $m \cdot \sqrt{wh}$ ) from the crop edges. Views that clip the box or fail the margin test are excluded.
Matching: A view contributes to the numerator if it contains a same-class detection with an Intersection-over-Union (IoU) $\ge \tau_a$ (0.75) with the original detection.
Rationale: Dividing by the total number of views (as in standard ensemble methods) penalizes detections that appear in fewer views simply due to geometry, driving performance below baseline. Dividing only by the effective view set (views that could have seen the object) correctly normalizes the signal.

Implementation Characteristics

Zero Training: Requires no ground-truth labels for the target domain.
Zero Forward Passes: Uses detections already generated by the tiled inference pipeline.
CPU-Only: The computation is a lightweight post-processing step.
Frozen Setting: A single global configuration ( $\lambda=5$ , specific thresholds) is tuned once on a validation split and applied universally across different detectors, datasets, and merge rules.

Key Results

Primary Performance Gains

On the VisDrone test-dev split using YOLOv8s:

Small-Object AP ( $AP_s$ ): Improved from 13.53 (SAHI + NMS) to 14.58 (+1.05 points).
Overall AP: Improved from 20.83 to 22.53.
Statistical Significance: The gain of +1.05 $AP_s$ has a 95% confidence interval of [+0.89, +1.23] based on paired bootstrap resampling.

Generalization and Robustness

The method was evaluated across 1,095 pairings (110 detector-dataset configurations $\times$ 5 merge rules) spanning five benchmark splits (VisDrone, DOTA-v1.5, AI-TOD-v2) and three detector families (YOLO, RT-DETR, Faster R-CNN).

Universal Improvement: Rescoring improved AP in 93% of all configurations.
Merge Rule Agnostic: It improved performance on top of the recent CMA-NMS (crop-margin-aware) in 96% of cases.
Failure Reversal: In cases where tiling itself degraded performance (e.g., RT-DETR on VisDrone test-dev), CAR reversed the regression, turning the largest losses into the largest gains.
Open-Vocabulary: The method transferred successfully to open-vocabulary detectors (YOLO-World, Grounding DINO), improving performance despite their notoriously mis-calibrated text-similarity scores.

Cost Analysis

Latency: Adds approximately 10 ms per image (median) on CPU.
GPU Cost: Unchanged, as no additional forward passes are required.
Scalability: Overhead scales with detection density rather than image size.

Significance and Claims

The paper claims that tiled inference loses accuracy primarily due to ranking errors, not box selection errors.

Oracle Decomposition: An analysis replacing scores with ground-truth quality showed that 74% of the recoverable performance gap could be closed by re-ordering scores alone, without changing the set of detected boxes.
Structural Defect: The mis-calibration of scores across different framings is a structural property of tiled inference, not a quirk of specific models.
Efficiency: The "signal" needed to fix this ranking (the agreement of overlapping tiles) is already computed during standard inference but discarded. CAR recovers this signal at zero inference cost.
Practical Priority: The authors suggest that practitioners gain more from switching the default merge rule to standard NMS and applying CAR than from adopting more complex, crop-aware merging strategies, as score repair largely neutralizes differences between suppression geometries.

Limitations

The authors acknowledge three specific regimes where the mechanism weakens:

Sub-pixel Sizes: For extremely small objects (e.g., 6px), the agreement match becomes a test of localization jitter rather than existence, limiting gains on datasets like AI-TOD-v2.
Full-Image View: Including the full-image view in the effective set is necessary; removing it degrades performance, even if the object appears small in that view.
Semantic Confusion: The method cannot suppress false positives that overlap with real objects of neighboring classes (e.g., a "bicycle" detected where a "motorcycle" exists), as these are re-detected consistently across views just like true positives.

In summary, the paper presents a simple, label-free, and computationally cheap post-processing layer that significantly improves small-object detection by leveraging the redundancy inherent in tiled inference to repair score rankings.