← Latest papers
🧠 neuroscience

ASTEROID, A Validated Annotated Dataset of Analytical Decisions in Graph Theory Based fMRI

ASTEROID is a validated, structured dataset comprising 156 graph theory-based fMRI studies with 125 methodological and experimental labels, created through an LLM-assisted and expert-reviewed workflow to enable systematic comparison of analytical decisions, robust multiverse analysis, and the development of automated extraction tools.

Original author: Imam, S. H.

Published 2026-11-19
📖 7 min read🧠 Deep dive

Original author: Imam, S. H.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine trying to understand how the human brain works by looking at a map of its electrical activity. Scientists use a powerful imaging technique called functional magnetic resonance imaging, or fMRI, to take pictures of the brain while people think, feel, or move. To make sense of these pictures, researchers often treat the brain like a vast network of cities connected by roads. In this view, specific brain areas are the cities, and the lines connecting them are the roads that carry information. This way of looking at the brain, known as graph theory, helps scientists measure how well different parts of the brain talk to each other and how organized the whole system is.

However, building this map is not as simple as taking a single photograph. It requires a long, complex series of steps to clean up the raw images, connect the dots, and calculate the final numbers. At almost every step, the researcher has to make a choice. They must decide which software to use, which mathematical rules to apply, and how to handle the inevitable noise in the data. These choices are like forks in a road; taking one path might lead to a map showing a highly connected brain, while taking another might show a disconnected one. Because the literature is filled with these different paths, and because scientists often describe their choices in messy, unstandardized sentences, it has been very difficult to compare studies or understand why results sometimes disagree.

To solve this problem, a team of researchers has created a new resource called ASTEROID. This is a massive, organized collection of information drawn from 156 published studies that used graph theory to analyze brain scans. The team did not just read these papers; they built a structured system to pull out every specific decision the authors made, turning vague descriptions into clear, comparable data. They found that while many studies follow similar basic steps, the details of how they build their brain networks vary wildly. By organizing this information, the researchers have created a tool that allows anyone to see exactly how different choices lead to different results, making it possible to test the stability of scientific findings in a way that was previously impossible.

The journey to create this dataset began with a simple but difficult task: reading 156 scientific papers and extracting the hidden instructions within them. The researchers knew that if they tried to do this by hand, it would take years and might still miss important details. Instead, they used a modern type of artificial intelligence, specifically a large language model, to act as a first pass at reading the text. They taught this computer program a specific set of rules, or a schema, that defined exactly what to look for. This schema included 125 different categories, ranging from basic details about the study participants to the complex mathematical steps used to build the brain network.

The computer was instructed to find the sentences in each paper that described a specific step, such as how the brain images were cleaned of motion blur or how the connections between brain areas were calculated. It then recorded the software used, the specific settings chosen, and the exact words the authors wrote. This process generated a first draft of the data for every single study. However, because scientific writing can be vague or inconsistent, the computer sometimes missed things or misunderstood the context. To fix this, the researchers built a safety net. They used automated checks to spot obvious errors, like missing steps or conflicting numbers, and then brought in human experts to review the results.

These experts, who are specialists in brain imaging and statistics, looked at the computer's work alongside the original papers. They corrected mistakes, filled in missing details, and ensured that the final data accurately reflected what the authors had actually done. This "human-in-the-loop" approach meant that the final dataset was not just a machine translation of text, but a carefully verified record of scientific practice. The team also refined their instructions for the computer as they went along. They realized that while the computer was good at finding basic steps, it struggled with the more complex parts of the analysis. By adding more specific guidance about these difficult steps, they improved the accuracy of the extraction significantly, especially for the parts of the study that involved building the network itself.

The result is a dataset that captures the full diversity of how scientists analyze brain networks. The researchers found that the choices made in the early stages of analysis, such as cleaning the images, were usually reported clearly and consistently. However, the choices made later, when actually building the network and measuring its properties, were often described in very different ways. Some studies used one method to define the connections, while others used a completely different one, and the papers often did not explain why they made those choices. This variation is not necessarily a mistake; it reflects the fact that there are many valid ways to approach the problem. But without a structured way to compare them, it is hard to know if a finding is solid or if it only appears because of a specific choice of method.

ASTEROID changes this by making these choices visible and comparable. The dataset includes not just the final numbers, but the specific path taken to get there. It links every decision back to the exact sentence in the original paper where it was described, allowing anyone to trace the logic of a study. The researchers have also built an interactive tool that lets users explore this data. A scientist can use it to see how many studies used a particular method, or to compare the results of studies that used different methods on similar questions. This helps in designing "multiverse analyses," a technique where researchers run their data through many different possible paths to see if the result holds up no matter which path they choose.

The team also tested how well their system worked by comparing the computer's initial guesses with a list of steps they had manually collected in a previous study. They found that the computer missed many details at first, particularly in the complex graph-analysis sections. But after they refined their instructions, the computer's accuracy improved dramatically. It began to find almost all the relevant information that the human experts had found, and in some cases, it even found details the humans had missed. This suggests that with the right guidance, artificial intelligence can be a powerful partner in organizing scientific knowledge, provided that human experts remain in the loop to verify the results.

This work does not claim to have solved the problem of inconsistent reporting in science, nor does it say that one method is better than another. Instead, it provides the map and the compass needed to navigate the existing landscape. By making the analytical choices of 156 studies visible and structured, ASTEROID allows the scientific community to move beyond simply comparing final results. It enables them to understand the process that created those results. This transparency is crucial for building trust in scientific findings and for teaching the next generation of researchers how to design robust studies. The dataset has been deposited as a draft on Zenodo and is currently under review; it will be made publicly available upon completion of the necessary release requirements, offering a new way to explore the complex decisions that shape our understanding of the human brain.

Drowning in papers in your field?

Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.

Try Digest →