SciVisAgentSkills: Design and Evaluation of Agent Skills for Scientific Data Analysis and Visualization

Technical Summary: SciVisAgentSkills

Problem Statement
Recent advances in agentic visualization have enabled the translation of natural language into executable scientific visualization (SciVis) workflows. While general-purpose coding agents (e.g., Codex, Claude Code) demonstrate strong capabilities in executing commands and writing code, they frequently lack the specific tool expertise required for complex SciVis tasks. These agents often struggle with environment assumptions, tool-specific API usage patterns, and domain heuristics, leading to unnecessary exploration, API misuse, and output generation errors (e.g., capturing the wrong GUI viewport). Although general-purpose agents can handle long-horizon tasks, they require procedural guidance to operate reliably across diverse scientific tools such as ParaView, napari, VMD, and TTK.

Methodology
The authors introduce SciVisAgentSkills, a collection of reusable agent skills designed to augment general-purpose coding agents without modifying model parameters. The methodology involves:

1. Skill Design: The authors constructed four distinct agent skills corresponding to four representative SciVis environments:
   • ParaView: For volume rendering, isosurface extraction, and flow visualization.
   • napari: For bioimage visualization.
   • VMD: For molecular visualization.
   • TTK: For topology visualization.
2. Construction Protocol: Each skill is a self-contained, version-pinned procedural module following a unified design process:
   • Environment Specification: Fixing software versions, dependencies, and runtime constraints (e.g., enforcing headless rendering) to reduce ambiguity.
   • Documentation Alignment: Distilling official documentation into structured usage patterns and API summaries.
   • Exemplar Integration: Incorporating representative code snippets and function usage patterns from existing SciVis agents (e.g., ParaView-MCP, BioImage-Agent) to ground execution.
   • Failure-Aware Refinements: Encoding empirical fixes for common failure modes observed during agent interaction.
   • Format: Skills utilize a YAML frontmatter for discovery and Markdown for guidance, supporting progressive disclosure where metadata is checked before loading full bodies.
3. Evaluation Framework: The skills were evaluated on SciVisAgentBench, a benchmark comprising 108 expert-designed, multi-step tasks across five suites: ParaView Visualization, Molecular Visualization, Bioimage Visualization, Topology Visualization, and Object Identification.
4. Experimental Setup: The study compared two general-purpose agents, Codex and Claude Code, performing tasks with and without the SciVisAgentSkills. Evaluations were conducted over three independent trials using multimodal LLM judges and deterministic metrics to assess completion rates, overall scores, and token efficiency.

Key Contributions

• SciVisAgentSkills Collection: The release of a curated set of agent skills specifically tailored for scientific data analysis and visualization across four major tools.
• Design Protocol: A documented methodology for constructing agent skills that combines environment specification, documentation distillation, and empirical failure analysis.
• Benchmark Evaluation: A comprehensive evaluation of how these skills impact general-purpose coding agents on a realistic, multi-step SciVis benchmark, analyzing performance across different task types and agent harnesses.

Results

• Performance Improvement: Incorporating agent skills consistently improved the mean overall scores across all five task suites for both Codex and Claude Code. The most significant improvements were observed in Topology Visualization (where Claude Code's score increased by approximately 60%) and Object Identification.
• Task-Type Variance: Gains were most pronounced in tasks requiring multi-step reasoning and involving APIs or workflows less common in foundation model training data (e.g., TTK, ParaView). Improvements were more modest for Molecular Visualization (VMD), particularly for Claude Code, suggesting that strong foundation models may already possess substantial knowledge of mature, well-documented tools.
• Token Efficiency: Token usage effects were mixed and dependent on the agent harness and model behavior.
  • For Claude Code, output tokens decreased consistently, and input tokens decreased for Bioimage, Topology, and Object Identification tasks.
  • For Codex, token usage increased for Bioimage and Molecular Visualization tasks.
  • The authors conclude that token cost is a property of the interaction among skill content, model behavior, and context management, rather than skill verbosity alone. No clear correlation was found between token usage and performance scores.
• Completion Rates: While scores generally improved, completion rates did not always follow the same trend. In some cases (e.g., Codex on Object Identification), skills improved output quality but introduced execution paths that occasionally failed, lowering the completion rate.

Significance and Claims
The paper positions SciVisAgentSkills as an empirical study demonstrating that structured procedural knowledge is critical for enabling reliable, long-horizon SciVis workflows. The authors claim that:

1. Skills as a Reusable Layer: Agent skills serve as an effective, low-cost, and extensible layer between foundation models and specialized application domains, injecting task- and domain-specific expertise without retraining models.
2. Harness Interdependence: The effectiveness of skills is not isolated; it depends heavily on the execution harness (e.g., how the agent retrieves, caches, and reuses skill content). The authors argue that skills and harnesses must be studied jointly.
3. Generalizability: The consistent gains across diverse tools (ParaView, napari, VMD, TTK) suggest that agent skills can generalize across different scientific software ecosystems.
4. Call to Action: The work serves as a starting point for building a broader, reusable skill ecosystem for scientific data analysis, inviting community collaboration to expand the library of skills and evaluate their interaction with various agent harnesses in real-world scenarios.