💻 computer science

Agentic Discovery with Active Hypothesis Exploration for Visual Recognition

The paper introduces HypoExplore, an agentic framework that treats neural architecture discovery as a hypothesis-driven scientific inquiry using large language models, evolutionary branching, and multi-agent feedback to iteratively refine and validate lightweight vision architectures, achieving state-of-the-art performance on benchmarks like CIFAR and MedMNIST while building a transferable understanding of the design space.

Original authors: Jaywon Koo, Jefferson Hernandez, Ruozhen He, Hanjie Chen, Chen Wei, Vicente Ordonez

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaywon Koo, Jefferson Hernandez, Ruozhen He, Hanjie Chen, Chen Wei, Vicente Ordonez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to invent a new, super-efficient recipe for a cake.

The Old Way (Traditional AI Research):
Most scientists today act like a chef who has a specific, famous cake in mind (like a Chocolate Fudge). They tweak the recipe slightly: "What if I add a pinch more salt?" or "What if I bake it for 2 minutes longer?" They try thousands of these tiny tweaks. Sometimes they get a better cake, but often they just end up with a slightly different version of the same old cake, or they get stuck in a loop of bad ideas without realizing it.

The New Way (HypoExplore):
The paper introduces a system called HypoExplore. Instead of just tweaking a recipe, HypoExplore acts like a team of brilliant, curious scientists who treat architecture design like a scientific expedition.

Here is how it works, broken down into simple metaphors:

1. The Expedition Map (The Trajectory Tree)

Imagine the scientists are exploring a vast, foggy jungle. Every time they try a new path, they draw a map.

The Tree: They don't just wander aimlessly. They build a giant family tree of their ideas. If "Idea A" leads to a dead end, they mark it on the map so no one wastes time going there again. If "Idea B" leads to a beautiful waterfall, they note exactly how they got there.
The Benefit: This prevents them from walking in circles or forgetting what they already learned.

2. The Hypothesis Bank (The "Idea Ledger")

Instead of just saying "This cake tastes good," the scientists write down why they think it tastes good.

The Hypothesis: They write a rule like: "If we use vanilla instead of chocolate, the cake will be lighter."
The Confidence Score: Every time they test this rule, they update a "confidence score."
- If the cake is lighter, the score goes up.
- If the cake is dry, the score goes down.
- Over time, the system learns which rules are actually true and which are just lucky guesses.

3. The Team of Specialists (The Multi-Agent System)

HypoExplore isn't just one robot; it's a team of different experts working together:

The Idea Generator: The dreamer. It comes up with wild new concepts based on the research direction (e.g., "Let's try connecting the ingredients in a circle instead of a line!").
The Builder (Coder): The engineer. It takes the dreamer's idea and writes the actual code to build the model. If the code breaks, it fixes it immediately.
The Judge (Redundancy Filter): The bouncer. Before anyone tries a new idea, the Judge checks the map. "Hey, we tried something like this last week. It didn't work. Let's not waste time."
The Analysts (Feedback Agents): After the model is trained, four different experts look at the results:
- The Statistician: Looks at the numbers (accuracy, speed).
- The Visual Artist: Looks at the images the model got wrong to see why (e.g., "It confused a cat with a dog because it was looking at the fur, not the face").
- The Detective: Figures out exactly which change caused the improvement or failure.
- The Troubleshooter: Fixes errors if the code crashed.

4. The Strategy: "Exploration vs. Exploitation"

This is the smartest part. The system has to decide: Do we stick with what works, or do we try something risky?

Exploitation (The Safe Bet): If a hypothesis has a high confidence score (e.g., "Adding global tokens works 90% of the time"), the system tries to build on that success.
Exploration (The Wild Card): If a hypothesis has a low or uncertain score, the system says, "We don't know yet. Let's test it!" This prevents the team from getting stuck in a local trap where they only find "okay" solutions and miss the "amazing" ones.

The Results: What Did They Find?

The team put this system to the test on a standard computer vision challenge (CIFAR-10).

The Starting Point: They began with a very basic, low-performing model (18.91% accuracy).
The Journey: Over 50 rounds of "experiments," the system iterated, learned from its mistakes, and refined its hypotheses.
The Finish Line: They discovered a new architecture called GSTN (Global Shape Token Network).
- It achieved 94.11% accuracy.
- It is incredibly small and efficient (using very few "parameters," or brain cells).
- It worked so well that it also solved harder puzzles (medical images) without needing to be retrained from scratch.

The Big Takeaway

The paper argues that we shouldn't just let AI randomly guess and check. Instead, we should give AI a scientific method. By forcing the AI to form hypotheses, track evidence, and learn from its own "scientific papers," it can discover better solutions faster and, more importantly, understand why those solutions work.

It's the difference between a monkey hitting random keys on a piano and a composer who understands music theory, writes down their ideas, and iterates until they create a masterpiece.

1. Problem Statement

Designing effective neural architectures for computer vision remains a significant challenge. While automated methods exist, they often rely on fixed search spaces, hyperparameter tuning, or incremental modifications to seed architectures. This approach frequently leads to:

Redundancy: Repeatedly discovering similar design patterns.
Myopia: Getting stuck in local optima without exploring fundamentally new ideas.
Lack of Interpretability: Difficulty in understanding why a specific architecture works or fails.
Domain Specificity: High human effort is still required to adapt architectures to specialized domains (e.g., medical imaging).

The authors argue that automated architecture discovery should be reframed not merely as a search problem, but as autonomous scientific discovery. This requires a system that explicitly formulates, tests, and refines hypotheses based on empirical evidence, managing the trade-off between exploiting known principles and exploring uncertain ones.

2. Methodology: HypoExplore

The paper introduces HypoExplore, a memory-grounded multi-agent framework that treats neural architecture design as a hypothesis-driven scientific inquiry. The system operates without a predefined seed architecture or fixed search space, starting instead from a human-specified research direction.

Core Components

Multi-Agent Pipeline:
- Idea Agent: Generates novel architectures based on a research direction and parent context. In "evolution mode," it conditions on selected hypotheses to propose targeted modifications.
- Coding Agent: Implements the architecture in PyTorch (model.py) and configuration (config.py). It includes loops for error recovery (fixing code bugs) and hyperparameter refinement.
- Redundancy Filtering Agent: Uses an LLM judge and embedding similarity to compare new proposals against the history, rejecting duplicates to enforce novelty.
- Executor: Trains the model within a time budget, recording performance and diagnostics.
- Feedback Agents: Four specialized agents analyze results from different perspectives:
  - Quantitative: Analyzes metrics (accuracy, loss, efficiency).
  - Qualitative (VLM): Examines misclassified images and attention maps to detect biases (e.g., texture vs. shape).
  - Causal: Attributes performance changes to specific structural modifications between parent and child.
  - Diagnostic: Analyzes failures (timeouts, OOM) to extract implementation lessons.
- Hypothesis Synthesis Agent: Consolidates feedback from all agents, resolves conflicts, and updates the memory bank with evidence.
Structured Memory System:
- Trajectory Tree: Records the full lineage of the discovery process (parent-child relationships, hypotheses used, and outcomes), preserving the history of exploration branches.
- Hypothesis Memory Bank: Tracks specific architectural hypotheses (e.g., "Adding global tokens improves shape sensitivity"). For each hypothesis, it maintains:
  - Confidence Score ( $c_t$ ): A value in $[0, 1]$ updated based on evidence (supports or contradicts).
  - Evidence Logs: Supporting/contradicting data, failure modes, and implementation notes.
  - Usage Statistics: Number of times tested.
Dual Selection Strategy:
To guide the search, HypoExplore uses a two-stage selection process:
- Parent-Node Selection: Selects which branch of the Trajectory Tree to expand. It balances Branch Quality (performance + efficiency) with Availability (remaining unexplored hypotheses in that branch).
- Hypothesis Selection: Selects which specific hypotheses to test within the chosen parent. It uses a dual strategy:
  - Exploitation: Uses Thompson Sampling over weighted evidence to select high-confidence hypotheses.
  - Exploration: Uses Epistemic Value (uncertainty) to select hypotheses with confidence near 0.5 (maximum uncertainty), ensuring the system probes unknown areas.

3. Key Contributions

HypoExplore Framework: A novel multi-agent system that frames architecture discovery as explicit, evidence-driven scientific inquiry rather than blind search.
Explicit Hypothesis Management: The introduction of a Trajectory Tree and Hypothesis Memory Bank allows the system to track the lineage of ideas and accumulate reusable knowledge, preventing redundancy and enabling interpretable reasoning.
Dual Selection Mechanism: A strategy that dynamically balances exploiting validated principles with exploring uncertain mechanisms, avoiding the "exploitation collapse" seen in other evolutionary methods.
From-Scratch Discovery: The system does not rely on a fixed seed backbone, allowing it to discover fundamentally new architectural paradigms.

4. Experimental Results

The framework was evaluated on image classification tasks across general and specialized domains.

CIFAR-10:
- HypoExplore discovered GSTN (Global Shape Token Network), a lightweight architecture with only 0.9M parameters.
- Performance: Achieved 94.11% accuracy, starting from a root baseline of 18.91%.
- Comparison: Outperformed manually engineered lightweight models like MobileNet V3 and ShuffleNet V2 while using significantly fewer parameters.
- Ablation: Removing any core component (hypothesis memory, multi-agent feedback, or dual selection) resulted in significantly lower performance or early saturation.
Generalization:
- The discovered GSTN architecture transferred robustly to CIFAR-100 (72.6% Top-1) and Tiny-ImageNet (58.1% Top-1), demonstrating that the learned principles are not overfit to a single dataset.
Domain-Specific Discovery (MedMNIST):
- An independent discovery run on DermalMNIST yielded an architecture achieving 82.1% accuracy, setting a new state-of-the-art (SOTA) for that specific task.
- It also achieved competitive results on TissueMNIST and BreastMNIST without initializing from a medical-specific seed.
Analysis of Hypotheses:
- Predictive Power: The system's confidence scores were found to be highly predictive; hypotheses with higher confidence scores were significantly more likely to result in accuracy improvements (up to 80% prediction accuracy for high-confidence hypotheses).
- Cross-Lineage Transfer: Hypotheses learned in one evolutionary lineage successfully improved architectures in independent lineages, proving the discovery of transferable design principles.

5. Significance

Shift in Paradigm: The paper moves the field from "searching for parameters" to "learning design principles." By explicitly managing hypotheses, the system builds a genuine understanding of the design space.
Efficiency and Scalability: The discovered architectures are highly parameter-efficient, challenging the notion that high performance requires massive models.
Applicability: The framework's ability to discover SOTA models in specialized domains (medical imaging) without human priors suggests a path toward fully autonomous AI scientists capable of tackling diverse scientific problems.
Interpretability: Unlike black-box NAS methods, HypoExplore provides a clear "paper trail" of why a design was chosen, what hypotheses were tested, and what evidence supported the final architecture.

In conclusion, HypoExplore demonstrates that autonomous systems can effectively conduct scientific discovery in deep learning by combining multi-agent collaboration with rigorous hypothesis tracking and evidence-based memory, leading to the discovery of novel, efficient, and high-performing neural architectures.