🤖 machine learning

Detect in Any Scene: An Agentic Framework for Object Detection with Experience-Aware Reasoning

This paper introduces DetAS, an agentic framework that leverages a Multimodal Large Language Model to dynamically compose adaptive image restoration and multi-expert detection workflows, enhanced by a self-evolving experience harvesting mechanism to significantly outperform existing detectors in challenging, real-world scenarios.

Original authors: Wenlun Zhang, Jun Yin, Kentaro Yoshioka

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenlun Zhang, Jun Yin, Kentaro Yoshioka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find specific people in a crowded, messy room. Sometimes the room is foggy, sometimes it's pitch black, sometimes it's raining, and sometimes the people are wearing disguises.

The Problem with Old Detectors
Traditional computer vision systems are like a security guard who has only ever trained in a bright, clean office. If you put that guard in a dark, rainy basement, they get confused. They might miss people because the lighting is bad, or they might get stuck because they only know how to look for "office workers" and not "construction workers."

Current methods try to fix this in two ways:

Specialized Training: They train a guard specifically for "rainy days" and another for "foggy days." But if the weather changes slightly, or a new type of object appears, the guard fails.
Fixed Pipelines: They build a machine that always cleans the image first (like a photo editor) and then looks for objects. But sometimes, cleaning the photo actually makes it harder to see the object (like over-editing a photo until faces look weird).

The Solution: DetAS (Detect in Any Scene)
The authors propose a new system called DetAS. Instead of a single guard or a fixed machine, they created a smart team of agents led by a "Brain" (a Multimodal Large Language Model). Think of this Brain as a highly experienced project manager who can look at a messy situation and instantly decide what tools to use.

Here is how the team works, broken down into three simple steps:

1. The "Fix-It" Team (Self-Adaptive Image Restoration)

When the Brain looks at a picture, it asks: "Is this picture hard to see? Do we need to fix it?"

The Old Way: Always run the picture through a "De-haze" or "Brighten" filter, even if it's not needed.
The DetAS Way: The Brain checks the image. If it's foggy, it picks a "De-haze" tool. If it's dark, it picks a "Brighten" tool.
The Smart Twist: Sometimes, fixing the image makes it worse (like blurring a face while trying to remove rain). The Brain is smart enough to say, "Actually, this looks fine as is," and skips the fix. It only cleans the image if it helps the search.

2. The "Expert" Team (Multi-Expertise Detection)

Once the image is ready (or if it didn't need fixing), the Brain doesn't just use one detector. It has a toolbox full of specialist detectives:

One detective is great at finding tiny, crowded objects (like birds in a tree).
One is an expert at finding faces in the dark.
One knows how to spot cars on a highway.
One is trained for underwater scenes.

The Brain looks at the scene and says, "This is a dark street with faces. I'll call the Face-in-Dark expert and the General Street expert." It ignores the underwater expert because that would be a waste of time.

3. The "Referee" (Instance Grouping)

Since multiple experts might spot the same object (e.g., the Face expert and the Street expert both see a person), their answers might be slightly different. The Brain acts as a referee. It looks at all the suggestions, groups the ones that point to the same person, and picks the best, most accurate description. If an expert is hallucinating (seeing a person where there is none), the Brain can reject that guess.

4. The "Memory Book" (Self-Evolving Experience Harvesting)

This is the upgrade to DetAS-X.
Imagine the team makes a mistake on a specific type of foggy image. Instead of forgetting it, the system writes down a note in a Memory Book: "Hey, for this specific kind of fog, don't use the De-haze tool; it made things blurry. Just use the original image."

Next time the system sees a similar foggy image, it checks the Memory Book first. It learns from its past decisions, getting smarter and faster with every new image it processes. It doesn't just follow a rigid rule; it uses experience to make better choices.

The Results

The paper tested this "Agentic Framework" on six very difficult challenges (like finding faces in the dark, cars in heavy rain, and objects underwater).

The Outcome: The DetAS-X system was significantly better than all the other top AI models tested.
The Numbers: It improved detection accuracy by an average of 28% across all tests. In the hardest test (finding faces in the dark), it improved by 37%.

In Summary
Instead of forcing a computer to be a "one-size-fits-all" detector, DetAS acts like a flexible, thinking team. It decides when to clean the image, picks the right expert for the job, and learns from its past mistakes to get better over time. It turns object detection from a rigid, broken machine into a smart, adaptable agent.

Technical Summary: DetAS – An Agentic Framework for Object Detection with Experience-Aware Reasoning

Problem Statement

Object detection in real-world scenarios faces significant challenges due to diverse image degradations (e.g., haze, rain, low-light, underwater distortion) and heterogeneous object distributions. Existing approaches generally fall into two paradigms, both of which exhibit inherent limitations:

Representation Learning: These methods rely on curated datasets with fixed degradation types and object categories to learn specialized representations. While effective within their training domains, they fail to generalize when degradation types or scene distributions shift slightly.
End-to-End Pipelines: These approaches combine a standard detector with a dedicated image restoration module. However, they rely on predefined restoration strategies and manually designed pipelines. They often lack the flexibility to determine whether restoration is necessary or which specific restoration–detector combination is optimal for a given input, leading to suboptimal performance or performance degradation when restoration alters object boundaries or background colors.

Consequently, traditional closed-vocabulary detectors struggle to adapt to dynamically changing real-world environments where a single static pipeline cannot simultaneously handle varying environmental conditions and object distributions.

Methodology

The paper proposes DetAS (Detect in Any Scene), an agentic framework that reformulates object detection as a dynamic decision-making process. Instead of a static pipeline, DetAS leverages a Multimodal Large Language Model (MLLM) as a central agent to adaptively compose workflows by selecting from a toolbox of restoration modules and specialized detectors.

The framework consists of three core components:

1. Self-Adaptive Image Restoration (SAIR)

SAIR dynamically determines whether and how to enhance an image before detection.

Scene Perception: An LLM perception module analyzes the input image to predict a degradation-aware scene profile (e.g., normal, fog, rain, underwater, low-light, noise).
Tool Selection: Based on the predicted label, SAIR selects a corresponding restoration tool (e.g., dehazing, deraining, denoising, brightness enhancement) from a pool.
Image Selection Mechanism: Crucially, SAIR compares the original image with the restored version. It selects the version that maximizes detection-oriented cues (object visibility, boundary clarity, structural integrity) rather than visual aesthetics. If restoration degrades detection performance, the original image is retained.
Super-Resolution (SR): A final SR step is applied to increase spatial resolution for small or distant objects, with the upscaling factor selected automatically based on a target resolution.

2. Multi-Expertise Detection (MED)

MED integrates multiple domain-specialized detectors to handle diverse object distributions.

Detector Pool: The system maintains a pool of detectors with diverse expertise, including general-purpose, dense/small-object, autonomous-driving, drone-view, underwater, and face detectors.
Adaptive Selection: Given the image content and a target category list, the LLM agent selects the top- $K$ most suitable detectors from the pool.
Instance Grouping: To resolve duplicate predictions from multiple detectors, candidate bounding boxes are grouped based on spatial overlap (IoU) and visual similarity of cropped regions.
Instance-Level Reasoning: For each grouped instance, the system extracts the region and provides it to the LLM for final decision-making. This allows the LLM to focus on specific objects, reject illusory false predictions, and select the best bounding box–label pair, improving token efficiency and accuracy for small objects.

3. Self-Evolving Experience Harvesting (SEEH)

To address the rigidity of rule-based decisions and capture fine-grained image characteristics, the framework is extended to DetAS-X via SEEH.

Experience Collection: Before deployment, the system performs exploratory runs on a small set of annotated data. It records node-level decision outcomes (restorer selection, SR selection, detector selection) and their corresponding performance metrics (e.g., F1 scores) for specific scene profiles.
Experience Retrieval: During inference, the system generates a fine-grained profile key for the input image. It queries the experience memory to retrieve the top- $K$ most similar historical profiles and their associated optimal decisions.
Experience-Aware Reasoning: The LLM agent uses this retrieved historical experience alongside current visual evidence to make more informed decisions, allowing the system to progressively refine its policy and adapt to diverse scenarios without retraining.

Key Contributions

Agentic Detection Framework (DetAS): A novel framework that formulates detection as a dynamic decision process, utilizing an MLLM agent to adaptively select restoration tools and specialized detectors (SAIR and MED) rather than relying on fixed pipelines.
Experience-Aware Reasoning (DetAS-X): The introduction of Self-Evolving Experience Harvesting (SEEH), which accumulates node-level decision experience from a small set of annotated data. This enables the system to learn from past observations and improve decision quality under fine-grained conditions.
Comprehensive Evaluation: Extensive experiments demonstrating that the proposed framework significantly outperforms existing MLLM-based detectors across six challenging benchmarks involving diverse degradations.

Experimental Results

The framework was evaluated on six benchmarks: HazyDet (foggy drone scenes), MARIS (underwater), DarkFace (low-light face), B-Night and B-Rainy (BDD100K subsets), and COCO (standard detection).

Performance Gains: DetAS-X achieved an average improvement of 28.36% in F1 score across the six datasets compared to existing MLLM baselines.
Specific Improvements: Notable gains were observed in challenging conditions, with a 37.01% improvement on the DarkFace dataset for low-light face detection.
Comparison: While baseline MLLMs (e.g., Qwen3-VL-8B, GPT-4.1) struggled with severe degradations (often achieving F1 scores below 10% on specific degraded datasets), DetAS-X consistently achieved high performance (e.g., 44.98% on DarkFace, 52.35% on HazyDet).
Ablation Studies: Removing SAIR or MED caused significant performance drops (e.g., F1 on DarkFace dropped from 44.98% to 25.84% without SAIR). The optimal number of selected detectors ( $K$ ) was found to be 2; increasing beyond this led to slight degradation due to out-of-domain false predictions.

Significance and Claims

The paper claims that DetAS establishes a solid foundation for agentic detection, demonstrating that dynamic, input-aware decision-making is superior to static pipelines for robust object detection in complex and dynamic environments.

Paradigm Shift: The work suggests that robustness under diverse degradations requires adaptive reasoning rather than just improved representations or fixed restoration pipelines.
Scalability: The modular nature of the framework allows for the integration of additional tools and detectors to support broader applications, such as medical imaging or infrared detection, though these were not explicitly explored in the current work.
Future Potential: The authors posit that as MLLM grounding capabilities improve, the limitations of MLLM-based detectors in fixed-scene settings will diminish, further validating agentic detection as a promising paradigm for scalable real-world deployment.

The authors note that the code will be open-sourced upon acceptance.