💻 computer science

Automated Meta-Analysis of Transcranial Magnetic Stimulation Based on GraphRAG: A Methodology Study

This study proposes a GraphRAG-based methodology for automated meta-analysis of Transcranial Magnetic Stimulation (TMS) literature, which utilizes title-based chunking, paper-level knowledge graph construction, and hybrid retrieval to achieve high-accuracy parameter extraction and significantly reduce the time required for evidence synthesis.

Original authors: Yu Zhong, Jingna Jin, Xin Wang, Wang He, Tao Yin

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Zhong, Jingna Jin, Xin Wang, Wang He, Tao Yin

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery: Which specific settings for a "brain magnet" (Transcranial Magnetic Stimulation, or TMS) actually work best for different diseases?

The problem is that there are thousands of research papers, each a tiny puzzle piece. Some say "use a 5Hz frequency," others say "10Hz," and they all use different words to describe the same thing. Traditionally, a team of human experts has to read every single paper, copy the numbers into a spreadsheet, and do the math. It's like trying to build a giant Lego castle by hand, one brick at a time, while the instructions keep changing. It takes months, it's exhausting, and it's easy to miss a piece.

This paper suggests a new way to build that castle using a super-smart robot librarian (a Large Language Model) and a special filing system called GraphRAG.

The Old Way vs. The New Way

The authors argue that the usual way of using AI to find information (called standard RAG) is like trying to find a specific sentence in a library by just grabbing random pages of text. If you ask, "What was the pulse count in the Parkinson's study?", the old system might just hand you a paragraph that mentions pulses but doesn't give you the exact number, or it might mix up two different studies. It's too messy for a precise math problem like a meta-analysis.

Instead, this team built a GraphRAG system. Think of this not as a pile of loose pages, but as a giant, glowing spiderweb.

The Nodes: Instead of tiny text snippets, each "node" on the web is an entire research paper.
The Threads: The web connects papers that are similar, grouping them into "communities" (like a neighborhood of papers all about depression, or another neighborhood about stroke).
The Super-Search: When you ask a question, the robot doesn't just scan text; it looks at the whole web structure, checks the specific "neighborhood" of the answer, and pulls out the exact data points it needs.

How the Robot Reads the Papers

Reading a scientific paper is hard for computers because they are long, messy, and full of tables. The authors taught their robot a special trick called "Title-Based Chunking with Sliding Windows."

Imagine the paper is a long movie. If you try to watch the whole thing at once, your brain (or the computer's memory) gets tired and forgets the middle part. This is called the "lost-in-the-middle" problem.

The Fix: The robot cuts the movie into scenes based on the titles (like "Methods," "Results," "Discussion").
The Safety Net: If a scene is still too long, it uses a "sliding window" to overlap the edges, making sure no important detail falls through the cracks.
The Translator: The robot then acts like a translator, turning the messy movie script into a neat, organized JSON list (a digital spreadsheet) with specific fields like "Stimulation Frequency," "Sample Size," and "Risk of Bias."

Did It Work?

The team tested this system on a dataset they built called TMS-MAD, which included 26 meta-analyses and 250 supporting papers covering diseases like depression, Alzheimer's, and stroke. They ran the test on four different super-smart AI models (GPT-5.4, DeepSeek-V4-flash, Gemini-3.1-flash-lite, and Claude-haiku-4.5).

Here is what they found:

Speed: The robot was fast. It took an average of 86.71 seconds to extract all the necessary parameters from a single paper. That's a huge time-saver compared to humans.
Accuracy on Facts: For fixed facts (like the title of the paper or the study design), the robot was incredibly accurate, getting the exact match over 90% of the time.
Accuracy on Meaning: For open-ended descriptions (like "what was the treatment?"), the robot didn't always use the exact same words as the human experts, but the meaning was very close, with a similarity score above 0.8.
The "Forest Plot" Test: The ultimate test was seeing if the robot could recreate the famous "Forest Plot" graphs used in meta-analyses. When they asked the robot to calculate the results for specific questions (like "Does rTMS help with cognitive function after a stroke?"), the graphs it generated looked very similar to the ones in the original human-written papers.

Where It Stumbles (The Reality Check)

The authors are honest about where the robot isn't perfect yet.

The "Image" Problem: If a paper puts its data in a picture (like a chart or a graph image) instead of text, the robot cannot see it. It missed data in those cases because it's a text-reader, not an eye.
The "Math" Confusion: Sometimes the robot got confused about numbers. For example, it might mix up the total number of people in a study with the number of people actually analyzed, or confuse "pulses per session" with "total pulses."
The "Missing Data" Blind Spot: When checking if a study was biased (like, did they hide bad results?), the robot was good at spotting obvious things (like how they picked participants) but struggled with complex questions about missing data that required reading the whole story of the study.

The Bottom Line

This paper suggests that we can use this GraphRAG method to automate the heavy lifting of meta-analysis. It doesn't claim to have solved the problem completely or replaced human experts. Instead, it shows a feasible pathway where a robot can do the boring data extraction in under two minutes per paper, leaving humans to do the final thinking and decision-making. It's a powerful new tool in the detective's kit, but the detective still needs to be there to solve the case.

Technical Summary: Automated Meta-Analysis of Transcranial Magnetic Stimulation Based on GraphRAG

Problem Statement
The therapeutic efficacy of Transcranial Magnetic Stimulation (TMS) is highly dependent on specific stimulation parameters (e.g., frequency, intensity, site). However, different parameter combinations for the same disease often yield heterogeneous clinical outcomes, necessitating quantitative evidence synthesis via meta-analysis to optimize protocols. Traditional meta-analysis is resource-intensive, requiring months of manual labor by multiple experts to extract data from unstructured literature, and is prone to human error and difficulty in updating with new literature. While Large Language Models (LLMs) show promise in extracting information from text, they lack dynamic retrieval capabilities and are limited by training cutoffs. Furthermore, conventional Retrieval-Augmented Generation (RAG) systems, which retrieve text chunks, fail to satisfy the structured data aggregation requirements of meta-analysis, as they cannot maintain systematic associations among medical entities (e.g., treatment methods, outcome measures) across multiple papers.

Methodology
The study proposes a GraphRAG-based methodology workflow comprising three core stages: parameter extraction, knowledge graph construction, and multi-level querying.

Parameter Extraction (Long-Text Processing):
- Chunking Strategy: To address LLM context window limitations and the "lost-in-the-middle" problem, the authors employ a title-based text chunking strategy combined with a sliding window algorithm. This divides full texts into semantic blocks (abstract, methods, results, tables). Special "table protection" processing ensures the integrity of tabular data.
- Prompt Engineering: The system uses specific prompt templates combining these semantic blocks to extract structured data. Templates are designed for:
  - Basic Information: Study design, population, sample size.
  - Outcome Measures: Primary/secondary outcomes, scales, and time points.
  - TMS Parameters: Frequency, intensity, target, coil type, and sham methods.
  - Risk of Bias: Assessment using the RoB2 tool (five domains).
- Dynamic Storage: Extracted data is stored in JSON format. The schema is dynamic, adapting to experimental designs (e.g., two-arm RCT, multi-arm RCT, crossover) to handle specific data structures like carryover effects or pairwise comparisons while maintaining consistent core attribute naming.
Knowledge Graph Construction:
- Entity Definition: Unlike standard RAG which uses tokens or text chunks as nodes, this method uses entire papers as entity nodes.
- Hybrid Storage: The system integrates three information representations:
  - Keyword Storage: Extracted via PICO principles (Population, Intervention, Comparison, Outcome) and general keywords, utilizing synonym and parent-child mapping tables (e.g., TBS as a subtype of rTMS).
  - Structured Storage: Direct reuse of the JSON files from the extraction phase for fine-grained parameter queries.
  - Semantic Storage: Paper abstracts are converted to dense vectors using the BAAI/bge-base-en-v1.5 embedding model.
- Community Clustering: The Leiden algorithm is applied to partition the node graph into semantically cohesive communities, establishing entity associations based on weighted similarity scores.
Multi-Level Querying:
- A hybrid retrieval mechanism calculates a total score ( $Score_{total}$ ) by weighting keyword matching, structured field matching (using TF-IDF principles), and semantic vector cosine similarity.
- Keyword Query: Uses Boolean logic with synonym and hierarchical mapping to handle natural language diversity.
- Structured Query: Applies BM25-inspired weighting to suppress high-frequency generic terms (e.g., "TMS") and prioritize discriminative features (e.g., "deep TMS").
- Semantic Query: Uses cosine similarity between query and document vectors to rank relevance.

Dataset Construction
The authors constructed a specific dataset, TMS-MAD (TMS Meta-Analysis Dataset), comprising 26 TMS-related meta-analyses and their 250 supporting original studies (277 documents total). The dataset covers neurological disorders including depression, stroke, Alzheimer's disease, and Parkinson's disease, with a mix of study types (RCTs, crossover trials, etc.). All data was manually annotated to serve as a ground truth for evaluation.

Results
The method was evaluated across four LLMs: GPT-5.4, DeepSeek-V4-flash, Gemini-3.1-flash-lite, and Claude-haiku-4.5.

Extraction Accuracy:
- Fixed Fields: Exact Match (EM) values for fields like titles and study design types exceeded 90% across all models.
- Open Fields: While exact match scores were lower (e.g., BLEU-4 for intervention measures ranged from 12.4% to 32.5%), semantic similarity (Cosine Similarity) was high (>0.8 for most fields, >0.89 for TMS parameters), indicating the models captured the correct meaning despite phrasing differences.
- Numeric Fields: Mean Absolute Percentage Error (MAPE) for sample sizes was generally between 7–12%, though errors were higher for specific metrics like male percentage (up to 22%) and pulse counts (up to 39%).
- Risk of Bias (RoB2): Claude-haiku-4.5 achieved the highest accuracy (79.34%) and Kappa scores for overall bias risk. Performance was strongest on randomization domains and weakest on missing data domains.
Efficiency: The average time to extract parameters from a single paper was 86.71 seconds, significantly reducing the manual effort required for meta-analysis.
Retrieval Performance: The hybrid retrieval strategy outperformed unimodal approaches (keyword-only, structure-only, semantic-only) in terms of coincidence rate (the ratio of retrieved relevant papers to the ground truth set). While semantic-only retrieval returned a high volume of papers with lower precision, the hybrid approach effectively balanced recall and precision.
Forest Plot Replication: The system successfully reproduced forest plots for specific queries (e.g., rTMS for PSCI, cerebellar ataxia), though discrepancies occurred when data was presented solely in images, which the text-based LLM could not process.

Key Contributions

GraphRAG Paradigm for Meta-Analysis: The study introduces a novel application of GraphRAG where papers serve as entity nodes rather than text chunks, enabling the systematic retrieval and synthesis of entire studies required for meta-analysis.
Hybrid Storage and Retrieval: It proposes a three-modality storage strategy (keyword, structured, semantic) that balances retrieval granularity with information completeness, addressing the limitations of standard RAG in structured data aggregation.
TMS-MAD Dataset: The creation of a specialized, structured dataset for TMS meta-analysis to facilitate the evaluation of automated extraction and retrieval methods.
Automated Workflow: A complete pipeline from long-text chunking and prompt-based extraction to knowledge graph construction and effect size calculation, demonstrating feasibility for automating parts of the meta-analysis process.

Significance and Claims
The paper claims that the proposed method effectively supports data extraction and retrieval tasks for meta-analysis in the TMS domain. By leveraging LLMs for structured parameter extraction and GraphRAG for systematic literature synthesis, the study provides a feasible solution for automated meta-analysis. It demonstrates that while LLMs may not perfectly match human extraction in every exact string detail, they achieve high semantic fidelity and significantly reduce the time and resource costs associated with traditional meta-analysis. The authors conclude that this approach offers a practical pathway toward automating evidence synthesis, though they acknowledge limitations regarding image-based data extraction and the need for multimodal capabilities to fully resolve all data sources.