🤖 AI

VESTA: Visual Exploration with Statistical Tool Agents

The paper introduces VESTA, a framework that enhances statistical modeling by equipping vision-language models with a dynamically growing toolkit of data transformations and diagnostic visualizations, which significantly outperforms existing agent-based systems on complex tasks like those found in the new DAWN benchmark.

Original authors: William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of fingerprints, your clues are numbers and graphs. Your job is to figure out the "rule" or "formula" that created those numbers. In the world of science, this is called fitting a model to data.

For a long time, computers have been getting better at doing this, but they often get stuck. They might guess a rule, look at the graph, say "that looks okay," and move on, even if the rule is actually wrong. They lack the ability to truly look at the data and ask, "Wait, why does this part look weird?"

This paper introduces VESTA, a new kind of AI detective that doesn't just guess; it builds its own magnifying glasses.

The Problem: The "One-Size-Fits-All" Detective

Previous AI systems tried to solve this by asking a smart computer (a Large Language Model) to write code, look at the result, and then write new code based on a text description of what went wrong.

Think of it like a student taking a math test who is only allowed to read the teacher's written comments. If the teacher says, "Your graph looks a bit off on the right side," the student has to guess how to fix it without being able to zoom in, highlight the specific curve, or run a special test to see if the curve is actually a straight line or a wave.

The Solution: VESTA's "Toolbelt"

VESTA (Visual Exploration with Statistical Tool Agents) changes the game. Instead of just reading text, VESTA is given a dynamic toolbelt.

Here is how it works, using a creative analogy:

Imagine VESTA is a chef trying to recreate a secret recipe from a photo of a finished dish.

The First Guess: VESTA looks at the photo and guesses, "Maybe it's a chocolate cake." It writes the recipe and bakes a test cake.
The Taste Test (The Critique): The AI tastes the cake. It realizes, "Hmm, this is too dry, and the texture is wrong."
The Old Way: An old AI would just say, "Okay, I'll try adding more milk next time."
The VESTA Way: VESTA realizes it needs a better way to check the cake. So, it builds a new tool.
- It might build a "Moisture Meter" tool to check the texture scientifically.
- It might build a "Crumb Analyzer" tool to see if the cake is too dense.
- It might build a "Flavor Profile" tool to check if it's actually chocolate or just cocoa powder.

Crucially, VESTA saves these tools. If it builds a "Moisture Meter" for the first cake, it keeps it in its belt. If it tries a different recipe later, it can use that same meter again. It doesn't just throw the tool away; it accumulates a growing library of custom-made diagnostic tools.

The "DAWN" Benchmark

To test if this new detective was any good, the researchers built a test called DAWN (Dataset for Automated Workflows and Numerical Modeling).

Think of DAWN as a series of increasingly difficult puzzles:

Easy Puzzles: Simple shapes, like a straight line or a simple wave.
Hard Puzzles: Complex shapes, like a wave that gets louder and faster at the same time, or a mix of two different patterns.
Astro Puzzles: Real-world astronomy challenges. For example, figuring out the "Initial Mass Function" (how many big stars vs. small stars are born) or analyzing "chirp" signals from crashing black holes. These are the "final boss" levels of data puzzles.

What They Found

The researchers ran VESTA against other AI systems (like BoxLM and PyVision) on these puzzles.

The "No Tools" vs. "Dynamic Tools" Test:
- When VESTA had no tools, it was okay at the easy puzzles but struggled with the hard ones.
- When VESTA was allowed to create its own tools on the fly, it got significantly better. It could look at a weird curve, build a specific tool to measure that curve, and then use that measurement to fix its recipe.
- The Result: VESTA with dynamic tools beat the other AI systems, especially on the difficult and astronomy puzzles.
The "Expert" Comparison:
- The researchers also gave VESTA a set of tools written by human statisticians (the "Expert Toolkit").
- The Surprise: VESTA didn't just copy the human tools; it often built tools that were smarter. It would combine three different human tools into one super-tool that could check three things at once.
- However, the human-written tools were still slightly better overall. This suggests that while VESTA is great at inventing tools, human experts still know a few tricks the AI hasn't figured out yet (like how to perfectly interpret a very complex, multi-panel graph).

The "Visual" Advantage

The paper highlights that VESTA's superpower is vision.

Old AIs read text descriptions of graphs.
VESTA looks at the graphs.
When VESTA builds a tool, it often creates a multi-panel image (like a dashboard with 6 different charts). It then looks at that dashboard to decide what to do next.

The authors found that VESTA creates tools that are much more sophisticated than other AI systems. For example, in the astronomy puzzles, VESTA realized it needed to look at the "tails" of the data (the extreme values) and built specific tools to zoom in on those tails, something the other AIs missed.

The Limitations

The paper is honest about where VESTA still struggles:

The "Blind Spot": Sometimes VESTA builds a very complex, multi-panel graph, but the AI looking at it (the "critic") gets confused by the complexity. It's like building a super-complex dashboard but then having a driver who can't read all the gauges at once.
Speed: Because VESTA keeps trying, building tools, and re-testing, it takes longer to solve a problem than a system that just guesses once.
Synthetic Data: The tests were done on computer-generated data where the "right answer" was known. The paper notes that we don't know yet how well this works on real-world, messy data where we don't know the answer.

Summary

VESTA is an AI that learns to solve complex data puzzles by building its own magnifying glasses. Instead of just guessing and checking, it invents new ways to measure and visualize the data, saves those inventions, and uses them to get closer to the truth. It proved that giving an AI the ability to create its own diagnostic tools makes it much better at understanding complex patterns, especially in fields like astronomy where the data is tricky.

Technical Summary: VESTA (Visual Exploration with Statistical Tool Agents)

Problem Statement

Fitting quantitative models to data is a fundamental step in scientific workflows but remains one of the least automated processes. While recent agent-based systems leverage Large Language Models (LLMs) and Vision-Language Models (VLMs) to iteratively propose and refine statistical models, they struggle with complex modeling tasks. Existing approaches often rely on iterative critique alone, lacking the capacity to actively explore data through hypothesis-driven visualizations and robust statistical tests. Furthermore, current benchmarks often lack the complexity and domain specificity required to evaluate agents on real-world scientific phenomena, such as astronomy tasks involving initial mass functions (IMFs) or gravitational-wave chirp signals.

Methodology: The VESTA Framework

The authors introduce VESTA (Visual Exploration with Statistical Tool Agents), a framework designed to equip VLMs with a dynamically growing toolkit to guide model refinement. VESTA operates within the PyMC probabilistic programming framework, utilizing Markov Chain Monte Carlo (MCMC) for model fitting.

Core Workflow

The VESTA workflow (Algorithm 1) follows an iterative loop of four steps:

Propose: Starting with an initial visualization of the data, the agent proposes $p$ diverse candidate statistical models. These are translated into executable PyMC code to ensure faithfulness to the specified parameters.
Generate Tools: This is the central contribution of VESTA. At each iteration, the agent decides whether to select an existing diagnostic tool from a registry or dynamically code a new Python-based diagnostic tool. These tools accept data and current model parameters, returning visual or statistical diagnostics (e.g., residual plots, QQ plots, tail transforms). Newly created tools are persisted in the registry for reuse within the current run.
Critique: The output of the selected or newly created tool, combined with the history of model fits, is passed to a VLM critic. The critic analyzes the visual evidence to identify where the current model fails (e.g., poor tail fit, missed periodicity) and proposes $p$ revised models. The loop can terminate early if the critic deems the fit satisfactory.
Summarize: At the end of each iteration, the visual descriptions, model parameters, metrics, and tool outputs are compressed into a structured summary. This summary is injected into the next iteration, allowing the system to reason over the full refinement trajectory without incurring prohibitive context growth.

Key Distinctions

Unlike prior systems that rely on static tool sets or single-use code generation, VESTA's tools are persistent and compositional. The agent can accumulate a toolkit of specialized diagnostic functions that test multiple hypotheses simultaneously, often composing several expert-level functions into a single, sophisticated visualization.

The DAWN Benchmark

To evaluate VESTA, the authors introduce DAWN (Dataset for Automated Workflows and Numerical Modeling), a benchmark targeting distribution fitting and time series modeling with varying difficulty tiers:

Easy: Unimodal distributions and simple linear/seasonal time series.
Hard: Mixtures of distinct distributions and complex non-linear dynamics (e.g., ARIMA, S-curves, ECG signals).
Astro: Real-world astronomy tasks, including fitting stellar Initial Mass Functions (Salpeter, Kroupa, Chabrier, and freeform variants) and modeling gravitational-wave chirp signals with frequency sweeps and amplitude decay.

Evaluation metrics include Jensen-Shannon Divergence for distribution fitting and Expected Log Predictive Density under Leave-One-Out (ELPD-LOO) for time series.

Key Contributions

Dynamic Tool Creation: VESTA introduces a mechanism for agents to autonomously generate, persist, and reuse diagnostic tools tailored to specific modeling challenges.
Visual Exploration: By utilizing VLMs to interpret complex, multi-panel visualizations generated by these tools, VESTA enables hypothesis-driven refinement that goes beyond textual critique.
DAWN Benchmark: A comprehensive benchmark covering synthetic and real-world astronomy tasks, designed to stress-test agents on complex, domain-specific modeling problems.
Tool Composition Analysis: The paper demonstrates that dynamically generated tools often compose multiple expert-level functions, covering more diagnostic categories per function than existing visual tool-creation systems.

Results

The authors evaluated VESTA against two baselines: BoxLM (an iterative PyMC model proposal system) and PyVision (a multi-turn VLM framework for code execution). Experiments were conducted across three toolkit configurations: no tools, static expert-written tools, and dynamic model-written tools.

Performance Gains: VESTA with dynamic tool creation consistently outperformed both baselines, with the most significant gains observed on the Hard and Astro splits.
Comparison to Experts: While VESTA equipped with expert-written tools achieved the strongest overall performance (establishing an upper bound), VESTA with dynamic tool generation approached this upper bound without access to any expert tools. It successfully recovered expert tools at a higher rate than PyVision and created sophisticated tools that often composed several expert functions to test multiple hypotheses simultaneously.
Tool Reuse: Within single problem instances, approximately 79.5% of dynamically generated tools were re-invoked across subsequent steps, demonstrating the value of tool persistence.
Limitations: The paper notes that while dynamic tools are effective, they do not yet fully outperform expert-written tools, likely due to current VLM limitations in reasoning over complex multi-panel diagrams.

Significance and Claims

The paper claims that VESTA represents a significant step forward in automating scientific modeling by addressing the gap in agents' capacity for iterative, evidence-driven model refinement. By enabling agents to actively explore data through a dynamically growing toolkit, VESTA allows for the discovery of complex statistical structures that static or single-use tool approaches miss.

The authors position VESTA as a framework that not only recovers expert knowledge but also generates novel, sophisticated diagnostic tools capable of handling domain-specific phenomena like gravitational wave chirps and stellar mass distributions. The work suggests that the ability to create and persist tools is a critical capability for agents operating in complex scientific domains, moving beyond simple code generation to genuine statistical exploration.