💬 NLP

Graphs of Research: Citation Evolution Graphs as Supervision for Research Idea Generation

The paper proposes "Graphs of Research" (GoR), a supervised fine-tuning method that leverages structured citation-evolution graphs derived from reference neighborhoods to significantly improve large language models' ability to generate novel research ideas, achieving state-of-the-art performance over existing baselines.

Original authors: Songyang Gao, Yinghui Xia, Siyi Liu, Hui Xiong

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Songyang Gao, Yinghui Xia, Siyi Liu, Hui Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: How Do We Invent New Science?

Imagine you are a chef trying to invent a brand-new dish. You have a cookbook full of old recipes (scientific papers).

The Old Way: Most current AI chefs just grab a few random recipes that sound similar to what they are trying to make. They read the ingredients list, but they don't really understand how the recipes relate to each other, who copied whom, or how the flavors evolved over time. They just mash the text together and hope for the best.
The Problem: This often leads to ideas that sound fancy but don't make sense, or they just repeat old tricks without adding anything new.

The New Solution: "Graphs of Research" (GoR)

The authors of this paper built a new way to teach an AI how to invent new research ideas. They call it GoR (Graphs of Research).

Instead of just handing the AI a pile of papers, they give it a map.

1. The Map Analogy

Imagine the old recipes aren't just a stack of paper; they are a family tree.

Recipe A was written in 1990.
Recipe B was written in 2000 and improved Recipe A.
Recipe C was written in 2010 and combined Recipe A and B.

The "GoR" system draws this family tree. It doesn't just list the recipes; it draws lines connecting them to show:

Who copied whom? (Predecessor links)
How popular was the idea? (Citation frequency)
When did it happen? (Publication time)
Where in the book was it mentioned? (Section placement)

2. Teaching the AI (The "Supervision" Part)

The authors took this "family tree" map and turned it into a structured story (a prompt) that they fed into a smart AI (a 7-billion-parameter model called Qwen2.5).

They didn't just ask the AI to guess; they trained it. They showed the AI: "Here is the map of these old ideas, and here is the new idea that was actually invented based on this map."

By studying thousands of these examples, the AI learned a secret: "To come up with a great new idea, I need to look at the connections between old ideas, not just the words."

The Experiment: Did It Work?

The team put their trained AI (called GoR-SFT) to the test against other famous AI methods and even a much larger, super-smart AI (GPT-4o).

The Test: They gave all the AIs a "seed" paper (a new starting point) and asked them to generate a new research idea.
The Judges: They used a panel of AI judges and human experts to score the ideas on things like:
- Novelty: Is it new?
- Feasibility: Can it actually be done?
- Clarity: Is it easy to understand?
- Significance: Does it matter?

The Results: The Small AI Beat the Big AI

Here is the surprising part:

The Small AI Won: The authors' trained AI (which is relatively small and cheap to run) consistently beat the much larger, more expensive GPT-4o.
Why? It wasn't about having a bigger brain; it was about having better training. The small AI had learned to read the "family tree" of science. The big AI, even when shown the same map, didn't know how to use it because it hadn't been trained on it.
The "Flat" vs. "Structured" Difference: When the authors removed the "map" (the graph structure) from the training, the AI's performance dropped. This proved that the structure of the connections was the secret sauce.

The Human Verdict

When real human experts (PhD students) looked at the ideas:

They liked the GoR-SFT ideas better for being clearer, more feasible, and more sound.
Interestingly, the "flat" methods (the old way) sometimes sounded more "creative" or "exciting," but they were often less practical. The GoR-SFT ideas were the ones that felt like they could actually work in a real lab.

Summary

Think of this paper as teaching an AI to be a scientific detective instead of just a word processor.

Old Way: "Here are 10 papers. Write a new idea." (The AI guesses).
GoR Way: "Here is a map showing how these 10 papers connect, influence each other, and evolved over time. Now, use this map to find the missing piece and invent the next big thing." (The AI learns the pattern).

The paper claims that by giving AI this "map" and training it specifically to understand these connections, we can generate better, more practical, and more innovative scientific ideas, even with smaller, cheaper computer models.

Technical Summary: Graphs of Research (GoR) for Research Idea Generation

Problem Statement

Automated scientific research increasingly relies on Large Language Models (LLMs) to close the loop between literature review, ideation, and execution. However, the ideation phase remains a bottleneck. Existing methods for generating research ideas typically rely on static retrieval of relevant literature or complex prompt engineering. These approaches treat references as a "flat text bag," discarding the structural relations (e.g., citation position, temporal evolution, predecessor links) that human researchers use to synthesize new ideas. Consequently, current systems fail to leverage the citation graph as a supervision signal, limiting the originality and feasibility of generated ideas.

Methodology: Graphs of Research (GoR)

The authors propose Graphs of Research (GoR), a supervised fine-tuning (SFT) framework that converts citation evolution structures into structured-text prompts for LLM training. The methodology operates in three stages:

1. Automated Citation-Graph Extraction

The system constructs a 2-hop reference neighborhood for each seed paper, forming a directed acyclic graph (DAG) where the seed is the sink and references are nodes.

Data Sources: The pipeline processes accepted papers from five major venues (NeurIPS, ICLR, CVPR, ICML, ACL) from 2020–2024.
Subgraph Construction:
- Expansion: Retrieves explicit references ( $R(v)$ ) and their references ( $R(r)$ ) to form a 2-hop neighborhood.
- Temporal Cone: Enforces strict temporal ordering, retaining only nodes $u$ where $t(u) < t(v)$ .
- Pruning: Selects a budget $K$ (between 12 and 30 references) using a two-pass scoring policy. The score combines seed-internal citation frequency, section weight, influence flags (Semantic Scholar), and a "sibling-boost" term to surface foundational papers referenced by many selected neighbors.
Annotation: Every edge in the DAG is annotated with eight features across five categories:
- Role: Which seed sections cite the reference and their weight.
- Influence: Global citation counts and local centrality within the subgraph.
- Recency: Year arithmetic between seed and reference.
- Topology: Hop distance (direct vs. indirect).
- Provenance: Parser confidence flags.
Target Extraction: An LLM extracts a structured five-field idea schema (Problem, Existing Methods, Motivation, Proposed Method, Experiment Plan) for both the seed and its references.

2. Structured-Text Prompt Serialization

The annotated DAG is serialized into a structured-text prompt. The format includes blocks for:

Seed Metadata: Venue, year.
Citation Subgraph: A temporally ordered list of references.
Edge Features: [EDGE] blocks detailing the eight features for each connection.
Predecessors: [PREDECESSORS] blocks marking explicit or parallel predecessor relations.
Reference Content: Titles, abstracts, and the five-field ideas of the references.
Task: A request to predict the seed's five-field idea in strict JSON format.

3. Supervised Fine-Tuning (GoR-SFT)

Model: Qwen2.5-7B-Instruct-1M.
Training Objective: Standard completion-only cross-entropy loss, masking all prompt tokens so gradients flow only through the generated JSON idea.
Baseline Comparison: A matched baseline (Refs-SFT) is trained on the same data but with all structural annotations ([EDGE] and [PREDECESSORS] blocks) stripped, leaving only flat reference lists. This isolates the contribution of graph structure.

Key Contributions

GoR Framework: The paper characterizes citation graph structure as an underused supervision signal. It proposes a method to serialize citation subgraphs with edge features and predecessor relations into structured text for LLM fine-tuning.
Automated Pipeline: The authors constructed a pipeline generating 498 training papers, 50 validation papers, and a leak-free 50-seed test set (from 2025 papers) with annotated citation subgraphs and structured idea targets.
Empirical Validation: Extensive experiments demonstrate that injecting citation-evolution graph structure into the supervision pipeline significantly improves idea quality compared to flat-reference baselines and zero-shot prompting.

Experimental Results

The evaluation uses a 50-seed test set from 2025 papers across the same five venues. Metrics include an LLM-judge tournament (Novelty, Significance, Feasibility, Clarity, Effectiveness), surface metrics (SPECTER2, ROUGE-L, BERTScore), and a blinded human evaluation.

Head-to-Head vs. Published Baselines

GoR-SFT was compared against three gpt-4o-driven baselines: Si baseline (retrieval-augmented), CoI-Agent (chronological chains), and ResearchAgent (multi-agent with academic graph).

Performance: GoR-SFT achieved SOTA, ranking first on 31, 40, and 48 of the 50 seeds against the three baselines, respectively.
Dimensions: GoR-SFT showed decisive wins in Feasibility, Clarity, and Effectiveness. While it slightly trailed the Si baseline in Novelty and Significance, it outperformed CoI-Agent and ResearchAgent in these creative dimensions as well.

Ablation Studies

SFT vs. Zero-Shot: The supervised fine-tuning itself was the dominant driver, with GoR-SFT (23.42 Mean Elo) significantly outperforming the zero-shot Qwen2.5-7B (14.12 Mean Elo).
Graph vs. Flat Reference: Adding graph supervision to the SFT process provided focused gains. GoR-SFT (23.42) outperformed the flat-reference SFT baseline (22.46), with specific improvements in Significance (+0.105) and Clarity (+0.070).
Supervision vs. Capacity: When compared to GoR-Agent (using the same graph prompt but with gpt-4o in zero-shot mode), GoR-SFT (7B model) won 32 of 50 seeds. This indicates that training on the graph structure is more effective than simply prompting a larger model with the same structure.

Human Evaluation

A blinded study with 5 PhD raters on 5 seeds confirmed the automated rankings. GoR-SFT won 5 of 10 metrics, including Overall, Feasibility, Clarity, Soundness, and Reproducibility. The raters noted that GoR-SFT's ideas were more grounded and executable, though the Si baseline retained an edge in pure novelty.

Significance and Claims

The paper claims that citation-evolution graphs serve as a powerful, previously overlooked supervision signal for LLM-based idea generation.

Efficiency: A 7B open-source model trained with graph supervision (GoR-SFT) can surpass flagship closed-source models (gpt-4o) in generating high-quality, feasible research ideas at near-zero inference cost.
Grounding: The method shifts ideation from "flat text bag" retrieval to structural synthesis, producing ideas that are more grounded in the specific evolution of the field.
Future Work: The authors plan to scale the extraction pipeline to larger datasets to equip LLMs with a deeper grasp of scientific evolution, aiming to accelerate automated scientific innovation.

The authors emphasize that while the method improves feasibility and clarity significantly, the trade-off in raw novelty compared to the most divergent baselines (like Si) suggests that structural grounding and creative divergence may require different optimization strategies.