💬 NLP

SciDataSailor: Deep Scientific Data Exploring

The paper introduces SciDataSailor, a framework that leverages Monte Carlo Tree Search to synthesize tool-interactive trajectories for deep scientific data exploration, accompanied by a fine-tuning dataset and a benchmark to evaluate agents' ability to navigate complex, hierarchical scientific repositories and answer domain-specific questions.

Original authors: Jiyong Rao, Yicheng Qiu, Chi Zhang, Chunfeng Song, Runkai Zhao

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiyong Rao, Yicheng Qiu, Chi Zhang, Chunfeng Song, Runkai Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a single crime scene, the evidence is scattered across a massive, dusty warehouse filled with millions of boxes. Some boxes are locked, some are written in code, and others contain riddles that only make sense if you open three different boxes at once. This is the daily reality for scientists working with "scientific data." Unlike a simple story you can read in a book, scientific data lives in huge, messy digital folders containing everything from raw numbers and strange file formats to hidden notes and complex charts. To find the truth, a researcher has to know exactly which box to open, how to read the code inside, and how to connect the dots between files that look nothing alike. It's a job that usually requires a human expert with years of training, because if you open the wrong box or misread a number, your whole theory could collapse.

Recently, we've given computers a new superpower: "AI agents." Think of these as digital detectives that can read, reason, and use tools like a human. They are great at searching the internet or answering questions from a textbook. But there's a catch: most of these AI detectives have never been trained to walk into a real, messy scientific warehouse. They are used to clean, organized libraries, not the chaotic, interdependent file systems where real science happens. The big question scientists have been asking is: Can we teach these AI agents to actually explore real data, figure out what's inside, run the necessary math, and draw a conclusion based only on what they find, without making things up?

This is exactly what the paper "SciDataSailor" tackles. The authors introduce a new way to train AI agents to become expert data explorers. They realized that to teach an AI how to navigate a complex scientific warehouse, you can't just give it a list of questions and answers. You need to show it the journey—the specific steps of opening a file, checking its format, running a calculation, and realizing, "Oh, this file doesn't match that one, I need to try a different path." To do this, they built a framework called SciDataSailor. Imagine a master coach who doesn't just tell the student the answer, but simulates thousands of different exploration paths. The coach uses a smart search method (called Monte Carlo Tree Search) to try out different strategies: "What if we check the file sizes first?" or "What if we look for the metadata before the data?" The coach then rewards the paths that find real evidence and discards the ones that lead to dead ends or make up facts.

Using this "coach," the team created two major tools. First, they built a massive training set called SciDataSailor-SFT-2K, which contains over 2,000 verified "exploration stories" where an AI successfully navigated a dataset. Second, they built a test ground called SciDataSailor-Bench, featuring 627 tasks to summarize data and 586 questions to answer, all based on real datasets from biology, earth science, and physics. When they tested their trained AI (a smaller, open-source model) against these challenges, the results were surprising. Before training, the AI was like a detective who kept knocking on the wrong doors, getting confused, and running out of time. After training with the SciDataSailor method, the same AI became much sharper. It started solving problems with fewer steps, making fewer mistakes, and actually finding the right answers more often.

The paper also highlights a fascinating gap between different types of AI. The "big name" proprietary models (the expensive, closed-source ones) were already quite good at this, acting like seasoned detectives who knew the warehouse layout intuitively. However, the smaller, open-source models (the ones anyone can download) struggled, often getting stuck in loops of trial and error. But here's the exciting part: once the smaller model was trained on the SciDataSailor data, it closed that gap significantly. It learned to be efficient, stopping its "trial and error" and moving straight to the evidence. The authors suggest that while these AI agents are getting better, they still need this kind of specific, evidence-based training to truly handle the messy, real-world complexity of scientific discovery. They aren't just guessing anymore; they are learning to sail through the data, one verified step at a time.

Technical Summary: SciDataSailor: Deep Scientific Data Exploring

Problem Statement

Scientific discovery increasingly relies on large, hierarchical repositories containing heterogeneous and interdependent files (e.g., raw measurements, metadata, annotations, and derived artifacts). Unlike natural language corpora, these datasets lack explicit schemas and standardized query interfaces, requiring researchers to navigate complex directory structures, infer implicit data organization, align cross-file evidence, and validate conclusions. This process is labor-intensive, error-prone, and scales poorly.

While Large Language Model (LLM) agents have advanced in planning and tool use, existing research largely overlooks their ability to interact directly with real scientific data assets in executable environments. Most agentic systems operate over web documents or textual corpora via structured queries, failing to address the specific challenges of inspecting raw files, inferring schemas, and grounding conclusions in observed tool outputs. Furthermore, manually constructing diverse, reliable trajectories for training agents in this domain is prohibitively expensive.

Methodology

The authors introduce Deep Scientific Data Exploring, a task paradigm where agents must navigate unfamiliar repositories, interpret heterogeneous files, execute analyses, and produce conclusions grounded in executed observations. To operationalize this, they present SciDataSailor, a framework for synthesizing tool-interactive trajectories by balancing broad exploration with targeted exploitation.

Core Framework: SciDataSailor

SciDataSailor instantiates trajectory synthesis as Monte Carlo Tree Search (MCTS) enhanced with four task-specific mechanisms:

Difficulty-Stratified Exploration Seeds: Exploration intents (seeds) are assigned to four difficulty levels ( $L_0$ to $L_3$ ) based on the reasoning and evidence required. $L_0$ focuses on dataset mapping, while $L_3$ targets scientific usability and robustness checks.
Dual-Feedback First-Play Urgency (DF-FPU): To address the inefficiency of conventional MCTS in treating unvisited children uniformly, SciDataSailor uses a pre-execution LLM-Judge to estimate a probe's potential utility ( $\rho_w$ ) and a post-execution judge to evaluate realized utility ( $y_w$ ). This allows the system to prioritize promising first visits and calibrate node values based on observed outcomes, filtering out low-evidence branches.
Hierarchical Strategy-to-Tool Action Generation: Before instantiating a decision as an executable probe, the system proposes and scores high-level strategies (e.g., schema inspection, statistical analysis, cross-file integration). This hierarchy reduces redundant probes and promotes complementary evidence gathering.
Entropy-Guided Branching: An adaptive expansion mechanism that widens the branching budget for uncertain states (high entropy) and narrows it for obvious ones. This combines prior entropy from prospective scores with token-level predictive entropy to optimize branch-budget utilization.

Quality Control

The framework incorporates rigorous quality control to ensure trajectories are executable and evidence-grounded:

Execution Validity: Rule-based validation rejects traces with unresolved errors, missing blocks, or malformed observations.
Step-Level Hallucination Detection: An LLM verifier checks that every action follows logically from the state and that reasoning is grounded in the observation, removing unsupported transitions.
Uncertainty Handling: Ambiguous instances are withheld or discarded to prevent training on unreliable data.

Key Contributions

The paper makes three primary contributions:

Task Formulation and Synthesis Methodology: The authors formulate deep scientific data exploration as long-horizon, tool-calling interactions with real repositories. They introduce SciDataSailor, which synthesizes executable, evidence-grounded trajectories using the mechanisms described above.
Executable Training and Evaluation Resources:
- SciDataSailor-SFT-2K: A supervised fine-tuning corpus containing 1,102 meta-information exploration trajectories and 1,225 dataset-targeted QA pairs, constructed from heterogeneous datasets across life, earth, and physical sciences.
- SciDataSailor-Bench: A benchmark suite comprising 627 meta-information summarization tasks and 586 scientific question-answering tasks across 27 raw datasets. It evaluates agents on six dimensions, including data-format heterogeneity, open-ended exploration, and raw-repository grounding.
Empirical Characterization of Scientific Data Agents: The authors conduct extensive comparisons under a unified executable-agent protocol, revealing systematic differences between proprietary and open-weight models.

Experimental Results

The evaluation was conducted on SciDataSailor-Bench using a unified ReAct-style agent scaffold with Python execution, testing models under 12 and 24 ReAct-step budgets.

Proprietary vs. Open-Weight Models: Proprietary models (e.g., GPT-5.4, Claude-Opus-4.7) demonstrated significantly higher reliability and efficiency. For instance, on the meta-information track with a 12-step budget, GPT-5.4 achieved a Pass@1 of 65.22 and a Success Rate (SR) of 98.07, whereas the strongest open-weight model (GPT-OSS-120B) achieved 27.54 Pass@1. Open-weight models often required extended trial-and-error exploration to reach successful outcomes.
Effect of Supervised Fine-Tuning (SFT): Fine-tuning a compact open-weight model (Qwen3.5-9B) on SciDataSailor-SFT-2K yielded substantial improvements.
- Meta-Information Track: Pass@1 increased from 14.01 to 28.99, and SR from 49.76 to 96.14. The average number of ReAct steps dropped from 10.21 to 6.24.
- Efficiency: The SFT-trained model matched the efficiency of the much larger GPT-5.4, completing tasks in approximately 5 tool calls regardless of budget, while consuming fewer total tokens than larger open-weight baselines.
Behavioral Analysis: The SFT-trained model exhibited "monolithic solver" behavior, generating comprehensive, self-contained code blocks with high structural complexity (e.g., deeper nesting, more control-flow nodes) rather than relying on iterative, small-step debugging. This strategy reduced error chains and improved first-attempt completeness.

Significance and Claims

The paper claims that Deep Scientific Data Exploring presents a distinct challenge for LLM agents because evidence is distributed across heterogeneous files and implicit schemas that cannot be reconstructed from text retrieval alone.

Behavioral Gap: The study identifies a pronounced gap where proprietary models explore scientific repositories more consistently and efficiently, while open-weight models often depend on additional trial-and-error interactions.
Supervision Value: The authors demonstrate that token-level supervision over verified agentic trajectories effectively transfers decision patterns, significantly improving the accuracy, completion rate, and interaction efficiency of compact open-weight models, particularly under constrained budgets.
Limitations: The paper modestly notes that while SFT improves efficiency, token-level imitation provides limited support for recovering from unforeseen execution failures. They suggest that combining trajectory supervision with verifier-guided reinforcement learning is a promising direction for strengthening adaptive self-correction capabilities.

The work establishes a new benchmark and training paradigm for agents that must discover, execute, verify, and integrate evidence in open-ended scientific environments, moving beyond static QA to dynamic, evidence-grounded exploration.