💻 computer science

Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers

This paper revisits Human-in-the-Loop Object Retrieval by leveraging pre-trained Vision Transformers to address key design challenges in multi-object datasets, ultimately offering practical insights into representation strategies and active learning loops for effective interactive object class retrieval without prior labels.

Original authors: Kawtar Zaher, Olivier Buisson, Alexis Joly

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kawtar Zaher, Olivier Buisson, Alexis Joly

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking for a specific type of item in a massive, messy attic filled with thousands of boxes. You tell a robot, "Find me pictures of fire hydrants." The robot shows you a few images, but they are wrong: one shows a whole street (too much context), and another shows just a blurry red cylinder (too little context). You have to tell the robot, "No, that's not it," and point to the specific hydrant in the picture.

This paper is about teaching that robot how to learn from your corrections much faster and smarter, especially when the attic is full of other objects (like cars, dogs, or trees) mixed in with the fire hydrants.

Here is the breakdown of their approach using simple analogies:

1. The Problem: The "Whole Room" vs. The "Specific Toy"

Traditional search engines often look at the whole picture (the whole room) to guess what's inside.

The Flaw: If you are looking for a tiny toy car in a photo of a busy street, looking at the whole photo might just tell the robot, "This is a street scene." It misses the small detail.
The Alternative: If you only look at a tiny, zoomed-in square of the photo, you might see the toy car perfectly, but you lose the context that tells you it's actually a car and not just a random red block.

The authors wanted to find the perfect balance: a way to see the whole room (context) while also zooming in on the specific toy (detail).

2. The Solution: The "Human-in-the-Loop" Game

The researchers set up a game where a computer and a human work together:

The Start: The computer starts with a huge pile of unsorted photos.
The Guess: You give it a starting example (e.g., "Show me fire hydrants").
The Feedback: The computer shows you a few photos. You say, "Yes, that one is a hydrant," and "No, that one is a fire truck." Crucially, you also point to exactly where the hydrant is in the "Yes" photos.
The Learning: The computer learns from your "Yes" and "No" answers.
The Smart Selection (Active Learning): Instead of showing you random photos, the computer uses a special trick to pick the photos it is most confused about. It asks you to label those specific ones because they will teach it the most. This saves you time.

3. The Secret Sauce: "Patchwork" Descriptions

The paper tests different ways to describe an image to the computer. They used a modern "brain" (a Vision Transformer) that can look at an image in different ways:

Global Only (The Wide-Angle Lens): The computer looks at the whole image as one big blob.
- Result: Good for big scenes, bad for small objects hidden in clutter.
Local Only (The Magnifying Glass): The computer cuts the image into small squares (patches) and looks at them individually.
- Result: Great for finding small details, but sometimes it gets confused about what the object actually is because it lacks the big picture.
Hybrid (The Best of Both Worlds): This was the winner. The computer looks at the whole image and the small squares at the same time, stitching the information together.
- Analogy: It's like looking at a map of a city (Global) while also holding a street-level photo of a specific building (Local). You know exactly where the building is and what it looks like.

4. The Results: Finding the Right "Zoom Level"

The researchers tested this on two different "attics" (datasets):

Attic A (PascalVOC): Had fewer objects, and they were generally larger. Here, cutting the image into 4 big squares worked best.
Attic B (COCO): Was much messier with many tiny objects. Here, cutting the image into 16 tiny squares worked much better.

Key Finding: The "Hybrid" approach (combining the wide view and the zoomed-in view) consistently beat the other methods. It allowed the computer to find the right objects faster and with fewer corrections from the human.

5. Why This Matters

The paper concludes that by using this "Hybrid" strategy and asking the human to only label the most confusing pictures (Active Learning), you can build a system that finds specific objects in messy, crowded photos very quickly. It doesn't need to be trained on millions of labeled examples beforehand; it learns on the fly just by talking to you.

In short: They figured out how to make a search assistant that doesn't just look at the whole picture or just the tiny detail, but combines both to understand exactly what you are looking for, even in a cluttered room, while asking you for help only when absolutely necessary.

1. Problem Definition

The paper addresses Human-in-the-Loop Object Retrieval (HITL-OR), a task aimed at iteratively retrieving images containing objects of a specific "class-of-interest" from a large, unlabeled dataset.

The Challenge: Traditional Object Retrieval often fails in specialized domains (e.g., historical archives) where a single query yields too many irrelevant results. Standard deep learning retrieval methods usually require prior training on labeled data for specific categories, which is unavailable in this zero-shot, interactive setting.
The Multi-Object Complexity: The task is particularly difficult in multi-object datasets (e.g., COCO, PascalVOC) where the target object occupies only a small region of a cluttered scene. Global image descriptors often fail because they capture the dominant background context rather than the specific small object.
The Goal: To rapidly identify diverse instances of an object category using only an initial user query (with a region of interest) and iterative Relevance Feedback (RF), guided by Active Learning (AL) to minimize user annotation effort.

2. Methodology

The authors propose a framework that integrates an Active Learning loop with pre-trained Vision Transformer (ViT) representations.

A. Framework Workflow

Initialization: A large unlabeled dataset $D$ is encoded using a pre-trained ViT (specifically dinoV2) to extract features.
Query & Labeling: The user provides a query image and marks a Region of Interest (RoI). A small set of negative samples is randomly sampled.
Classification: A shallow Linear SVM is trained to distinguish the class-of-interest (positive) from the rest (negative). The authors chose SVM over fine-tuning deep layers because it generalizes better with very few labeled samples (low annotation budget).
Active Learning Loop:
- The classifier scores all unlabeled images.
- An Uncertainty Criterion ( $s_{AL} = 1 - |0.5 - s_i|$ ) selects the top- $b$ most informative images for the user to annotate.
- The user confirms relevance and selects the specific object bounding box in positive images.
- The training set is updated, and the loop repeats.

B. Representation Strategies

The core technical contribution lies in how image features are constructed from the ViT to handle multi-object scenes. The authors compare three strategies:

Global-Only (GO): Uses the single global descriptor of the entire image.
Local-Only (LO):
- Images are divided into a grid of $M$ patches (e.g., $2\times2$ or $4\times4$ ).
- Features are extracted either by feeding cropped patches into the network or averaging ViT patch tokens.
- LO-One: Uses a single representative patch per image (e.g., the patch with the largest overlap with the user's RoI).
- LO-All: Treats every patch in the image as a separate training sample. Positive images generate both positive (patches overlapping the object) and negative (background patches) samples.
Global-Local (GL) / Hybrid: Fuses global and local contexts.
- Concatenation: $[g_i, l_i]$ (preserves separate contexts).
- Pooling: $\frac{g_i + l_i}{2}$ (merges contexts).
- Note: For the "LO-All" strategy, the authors found that concatenating the global descriptor with all patches (including negative background patches) confuses the SVM. Therefore, they only concatenate the global descriptor with the positive local patches.

C. Inference

For Local and Hybrid strategies, the image score is determined by the maximum score among all its constituent patches ( $s_i = \max_m s_{i,m}$ ), ensuring that if the object is present in any patch, the image is retrieved.

3. Key Contributions

Revisiting HITL-OR with ViTs: The first comprehensive study applying pre-trained, self-supervised ViTs (dinoV2) to the Human-in-the-Loop Object Retrieval task without fine-tuning on the target dataset.
Representation Analysis: A systematic comparison of Global, Local, and Hybrid representations. The paper demonstrates that Hybrid (Global-Local) strategies are superior, as they capture both the fine-grained details of the object (via local patches) and the scene context (via global descriptors).
Active Learning Integration: The framework effectively uses an uncertainty-based AL loop to select informative samples, allowing high retrieval performance with minimal user input (simulated with a budget of 10 images per iteration).
Patch Granularity Insights: The study reveals that the optimal patch size depends on the dataset's object size distribution. Smaller patches ( $4\times4$ ) are better for datasets with many small objects (COCO), while larger patches ( $2\times2$ ) balance performance for more balanced datasets (PascalVOC).

4. Experimental Results

Experiments were conducted on PascalVOC2012 and COCO2017 using dinoV2 features.

Metrics: Mean Average Precision (MAP) for retrieval accuracy and Coverage (diversity of retrieved instances) for user satisfaction.
Key Findings:
- Hybrid Superiority: Hybrid representations (specifically GL-concat-All and GL-pool-All) consistently outperformed Global-Only and Local-Only baselines.
  - On PascalVOC2012, Hybrid strategies improved MAP by ~8-9% and Coverage by ~3-6% over Global-Only.
  - On COCO2017, improvements were even more significant (MAP +13-18%).
- LO-All Strategy: The "Local-All" approach (using all patches as samples) provided the best trade-off between performance and coverage, as it offered the classifier a richer, more diverse set of training examples.
- Patch Size:
  - COCO2017: $4\times4$ patches (16 total) performed best, likely because 74.7% of objects in this dataset are "small."
  - PascalVOC2012: $2\times2$ patches (4 total) offered the best balance, as the dataset has a more balanced distribution of object sizes.
- ViT Registers: Experiments with Register-based ViTs showed that standard ViTs performed better in this specific retrieval context, possibly because the "entangled" global information in standard patch tokens aids diversity.

5. Significance and Conclusion

This work provides a practical blueprint for building interactive retrieval systems for unlabeled, multi-object datasets.

Efficiency: It proves that high-performance retrieval can be achieved with minimal user effort (few-shot learning) by leveraging pre-trained foundation models and Active Learning.
Design Guidelines: The paper offers concrete design choices for future systems:
- Use Hybrid (Global-Local) representations to balance precision and diversity.
- Use Concatenation if retrieval precision is the priority; use Pooling if covering diverse object instances is more important.
- Adapt patch granularity based on the expected size of objects in the target dataset.
Future Work: The authors suggest developing adaptive multi-scale patch selection frameworks to dynamically handle varying object sizes without manual configuration.

In summary, the paper successfully modernizes the Human-in-the-Loop retrieval paradigm by replacing handcrafted features with pre-trained ViTs and demonstrating that a hybrid representation strategy significantly outperforms traditional global or local-only approaches in complex, multi-object scenarios.