💻 computer science

Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning

This paper demonstrates that for multi-image medical reasoning, defining a simple and robust agentic decision rule (specifically an order-vote policy) yields significantly better generalization than extending the evolutionary search budget or employing more complex strategies.

Original authors: Site Li, Jianyi Hao, Xiaofeng Liu

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Site Li, Jianyi Hao, Xiaofeng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but instead of looking at a single clue, you have a whole stack of photos that tell a story in a specific order. Maybe it's a series of X-rays showing a bone healing, or a sequence of satellite images tracking a storm. In the world of artificial intelligence, this is called "multi-image reasoning." It's not just about recognizing what's in one picture; it's about understanding how the story changes from frame to frame.

For a long time, scientists thought the best way to get a computer to solve these puzzles was to give it a really long, detailed set of instructions (a "prompt") and then let it try millions of different variations of those instructions to see which one worked best. They used a method called "evolutionary search," which is like a digital version of natural selection: you create many versions of a program, let the best ones survive, and mix them to create even better ones. The big question was: Is the secret to success just having a bigger, more powerful computer that can try more variations? Or is the secret actually in how the computer decides to look at the clues in the first place? This paper dives into that exact question, testing whether it's better to have a smarter strategy or just a longer search.

The Great AI Detective Contest

In this study, researchers set up a high-stakes contest for AI agents (smart computer programs) to solve medical puzzles using a dataset called MedFrameQA. Think of this dataset as a giant library of medical cases where each question comes with a sequence of 2 to 5 images. The AI has to look at the whole sequence and pick the right answer from a list of options.

The researchers didn't just throw random instructions at the AI. Instead, they used a powerful engine called ShinkaEvolve to "evolve" the instructions. They gave every contestant the exact same amount of time and computing power (50 generations of evolution) to improve their strategy. The goal was to see which type of "decision rule" worked best.

Here are the five contestants they tested:

The Fixed Baseline: The AI looks at all the images and the question once, then guesses immediately. It's a "one-and-done" approach.
The Reasoning Scaffold: The AI is told to "think step-by-step" and explain its logic before guessing. It's like asking a student to show their work.
Order-Vote: This is the clever one. The AI looks at the images, but it shuffles the order of the answer choices (A, B, C, D) and asks the question multiple times. If the AI keeps picking "B" no matter how you shuffle the list, that's a strong vote. It then counts up all the votes to make a final decision.
Order-Rerank: This is the heavy hitter. It does the voting thing, but if the votes are close, it goes back and does a super-detailed, second-round comparison of the top two choices. It's like a judge holding a second trial.
Order-Vote+: A mix-and-match version that only does the second-round check if the first vote is unsure.

The Surprise Winner: Simplicity Wins

After running the contest five times to make sure the results weren't just luck, the researchers found a clear winner. It wasn't the AI that tried the hardest or used the most complex logic.

The Order-Vote strategy took the gold medal, achieving a final accuracy of 57.89 ± 0.65%.

It beat the simple "Fixed" baseline, which only got 52.73 ± 0.42%.
It also beat the "Order-Rerank" strategy, which was more complicated and expensive to run, scoring 55.79 ± 0.43%.

Why did the simple voting method win? The researchers suggest that medical images are tricky. Sometimes, the order in which you list the answers (A, B, C, D) can accidentally trick the AI into picking the wrong one. The Order-Vote strategy is like a wise detective who doesn't trust their first gut feeling. Instead, they ask the same question in different ways to see if the answer stays the same. By "voting" across these different perspectives, the AI becomes much more robust and less likely to make a silly mistake just because the answer choices were shuffled.

In contrast, the Order-Rerank strategy, which tried to do a deep, second-stage analysis, actually performed worse. It was like a detective who spent so much time re-examining the evidence that they got confused or made a new mistake. The study found that this complex method was "brittle"—it worked well in some cases but failed more often overall, and it required significantly more computing power (about 417.2 seconds for the final test compared to 331.5 seconds for the winner).

The "More is Better" Myth is Busted

Here is the most surprising part of the story. The researchers asked: "What if we just let the AI evolve for longer? Maybe 100 generations instead of 50?"

They expected that more time would lead to a smarter AI. Instead, the opposite happened. When they let the Order-Vote strategy evolve for 100 generations, its performance on the final test actually dropped from 57.89% to 56.02%.

This suggests that giving the AI more time to search didn't make it smarter; it just made it better at memorizing the practice tests (the "holdout" set) while forgetting how to handle the real, unseen test questions. It's like a student who studies so hard for a specific practice exam that they memorize the answers but fail to understand the concepts when the real test comes. The researchers concluded that defining the right decision rule is far more important than just searching longer.

What This Means for the Future

The paper doesn't claim to have solved all medical mysteries or that these AI agents are ready to replace real doctors. In fact, the authors are very careful to say this is a "benchmark" study, not a clinical one. However, the findings offer a crucial lesson for anyone building smart AI systems.

If you want an AI to reason through a sequence of images, don't just throw more computing power at it or ask it to "think harder" with longer instructions. Instead, design a system that is robust to confusion. The Order-Vote method proved that a simple, smart strategy that checks its own work by shuffling the options is far more effective than a complex, expensive system that tries to over-analyze every detail.

In the end, the paper suggests that in the world of AI, quality of strategy beats quantity of search. A clever, simple rule that stays steady no matter how the clues are presented is the true key to solving complex medical puzzles.

Technical Summary: Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning

Problem Statement
The paper addresses the challenge of Multi-Image Medical Visual Question Answering (VQA), specifically using the MedFrameQA benchmark. Unlike single-image tasks, this domain requires agents to reason over ordered sets of images (e.g., longitudinal scans or multi-view studies) where clinical meaning evolves across frames. The authors argue that the primary bottleneck in this setting is not merely prompt length or verbosity, but the inference-time agentic decision rule. Existing approaches often fail to robustly aggregate evidence across ordered images, remain sensitive to answer-order perturbations, or overfit to noisy search-time feedback. The central question is whether extending the optimization search budget (e.g., evolutionary generations) is more impactful than defining the correct agentic decision policy.

Methodology
The study employs a controlled experimental protocol using ShinkaEvolve, a prompt-program evolution framework, as a shared search engine. The core methodology involves:

Dataset and Split: The authors utilize MedFrameQA, where each question is grounded in 2–5 images. To prevent selection bias, they constructed a reproducible internal frozen split (2,851 total examples) preserving video IDs:
- Evolution Pool: 1,331 examples (used for evolutionary search feedback).
- Selection Holdout: 665 examples (used to select the best generation within a run).
- Independent Final Test: 855 examples (frozen split used exclusively for final reporting).
Agentic Strategies: Five distinct inference-time strategies were compared, all evolved under the same high-budget ShinkaEvolve configuration (50 generations, 5 islands, 40-entry archive). The strategies differ in their decision logic rather than just prompt text:
- Fixed: A reactive, one-shot decision using a single prompt with all images.
- Reasoning: Adds a structured textual scaffold for stepwise inspection but retains a one-shot decision rule.
- Order-Vote: Perturbs the order of answer options across multiple local views, maps predictions back to the original label space, and aggregates results via voting. This aims to reduce answer-order sensitivity.
- Order-Rerank: Extends Order-Vote by adding a conditional second-stage pairwise reranking stage when initial votes disagree.
- Order-Vote+: A selective top-2 correction variant (treated as an appendix exploration).
Evaluation Protocol: The study runs five independent repeated runs for the main strategies. Performance is measured on the independent final test set. The authors also conducted a budget-sensitivity analysis by extending the search to 100 generations for the Order-Vote strategy to test if longer search yields better generalization.

Key Results
The experiments yield three primary findings:

Decision Rule Dominates Search Budget: The Order-Vote strategy achieved the highest final-test accuracy at 57.89 ± 0.65%, significantly outperforming the Fixed baseline (52.73 ± 0.42%) and the more complex Order-Rerank variant (55.79 ± 0.43%). Paired bootstrap analysis confirmed these gains are statistically significant ( $p < 0.01$ ).
Efficiency vs. Complexity: While Order-Vote requires more inference calls than the Fixed baseline, it is substantially cheaper than Order-Rerank. Order-Rerank, despite its higher computational cost and deeper deliberation, failed to surpass the simpler voting aggregation, suggesting that robust aggregation is more effective than heavy arbitration for this task.
Diminishing Returns of Extended Search: Extending the evolutionary budget from 50 to 100 generations for Order-Vote resulted in a marginal increase in holdout performance but a decrease in final-test accuracy (dropping from 57.89% to 56.02%). This indicates that longer search budgets can amplify selection noise rather than improve true generalization.

Significance and Claims
The paper claims that for multi-image medical reasoning, defining the correct agentic decision rule is substantially more impactful than expanding the optimization search budget.

Methodological Contribution: The study demonstrates that "more search" is not a panacea. If the underlying policy (e.g., a one-shot selector vs. a robust aggregator) is misaligned with the task's requirements (handling ordered evidence and answer-order sensitivity), increasing search depth yields diminishing or negative returns.
Practical Implication: For developing Multimodal Foundation Model (MFM) agents in clinical settings, the design of the inference-time policy—specifically mechanisms that ensure robustness to input perturbations like answer ordering—should be prioritized over simply increasing model scale or search iterations.
Limitations: The authors modestly note that conclusions are drawn from a single benchmark (MedFrameQA) using an internal frozen split rather than an official hidden test server. The study focuses on the optimization of prompt programs rather than the training of base models, and the results are not claimed to be ready for direct clinical deployment without further validation.

In summary, the paper argues that the "inductive bias" of the agent's decision rule (in this case, order-robust voting) is the critical factor for success, outweighing the marginal gains obtainable from deeper evolutionary search or more verbose reasoning scaffolds.

The Great AI Detective Contest

The Surprise Winner: Simplicity Wins

The "More is Better" Myth is Busted

What This Means for the Future

Technical Summary: Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning

More like this