How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description

Technical Summary: How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description

Problem Statement
Large Language Models (LLMs) are increasingly utilized for scientific literature synthesis, including related-work generation and automatic literature reviews. However, generating a literature review from scratch remains problematic due to the models' propensity for hallucinating references, providing uneven coverage, and imposing thematic organizations unsupported by the scholarly record. While retrieval-augmented generation (RAG) improves grounding, it does not fully resolve the difficulty of structuring a literature map. Bibliometric analysis offers a structured alternative, using metadata (citations, references) to organize papers into clusters representing research streams. The central research question addressed is: How much external bibliometric structure must be provided to an LLM to generate reliable cluster descriptions? Specifically, the authors investigate whether LLMs can autonomously infer bibliometric structure or if they function best as interpreters of algorithmically defined structures.

Methodology
The study evaluates six distinct pipelines representing a continuum of decreasing LLM responsibility and increasing bibliometric structure. The benchmark is constructed from 100 peer-reviewed bibliometric review papers. For each source paper, the authors manually extract the original search query and human-authored cluster descriptions, then reconstruct the corresponding Scopus corpus.

The six pipelines tested are:

1. Blind: The LLM receives only the search query and the target number of clusters. It must infer themes, references, and cluster structures from scratch.
2. Corpus: The LLM receives the query, the target number of clusters, and the full reconstructed Scopus corpus (titles and abstracts). It must induce cluster structure and describe them in a single pass.
3. Corpus-select: A two-stage process where the LLM first selects relevant papers from the corpus for each cluster, then synthesizes descriptions from this filtered evidence.
4. Labeled: The LLM receives the corpus pre-grouped by a standard bibliometric clustering algorithm (Louvain community detection based on bibliographic coupling or direct citation). The LLM's role is restricted to interpreting and describing these pre-defined clusters.
5. Labeled-select: Similar to Labeled, but in a two-stage process where the LLM first selects the most representative papers within the pre-defined clusters before generating descriptions.
6. Ranked: The LLM receives the pre-defined clusters but is provided only with a compact, top-ranked subset of papers (based on link strength) from each cluster to summarize.

Evaluation Metrics
The outputs are evaluated across five dimensions:

• Human Alignment: Semantic similarity (BERTScore) between generated and human-authored descriptions.
• Semantic Coverage: The extent to which generated descriptions cover the semantic content of the entire reconstructed corpus.
• Clustering Quality: Measured by Adjusted Rand Index (ARI) and Silhouette score, assessing whether the generated descriptions induce paper assignments that recover the original bibliometric structure.
• Graph-Structural Quality: Modularity scores evaluating if the induced clusters align with dense regions of the underlying citation or bibliographic coupling graph.
• Reference Grounding: Validation of generated references against real bibliographic records (OpenAlex) and checking if descriptions reflect the cited evidence.

Key Results

1. LLMs as Interpreters vs. Generators: LLMs produce descriptions that are semantically close to human-written ones (high BERTScore) but are unreliable when asked to infer bibliometric structure from scratch. The Blind pipeline performs poorly across all structural metrics and exhibits significant reference hallucination.
2. Impact of Structure: Performance improves significantly when bibliometric algorithms define the clusters and the LLM interprets them. Pipelines with algorithmic structure (Labeled, Labeled-select, Ranked) consistently outperform corpus-only pipelines (Corpus, Corpus-select) on clustering quality, graph modularity, and reference grounding.
3. Task-Dependent Structure: The optimal level of structure depends on the bibliometric relation:
   • Bibliographic Coupling: Benefits from full cluster context (Labeled performs best), as these clusters often reflect broad intellectual backgrounds distributed across many papers.
   • Direct Citation: Benefits from compact, link-ranked evidence (Ranked performs best), as citation structures often concentrate around influential central works.
4. Human vs. Structured LLMs: While human descriptions are written for scholarly interpretation, structured LLM pipelines often achieve higher scores on corpus-level, clustering, and graph-based metrics. This indicates that structured pipelines better preserve the reconstructed bibliometric or citation structure, though this does not necessarily imply greater domain insight or nuance than human experts.

Significance and Claims
The paper claims that LLM-assisted bibliometric synthesis is most promising as a hybrid workflow. In this model, algorithms provide auditable, precise structural tasks (such as clustering papers by citations or shared references), while LLMs handle the interpretive task of translating that structure into readable descriptions.

The authors position this work as a test case for broader LLM system design, suggesting that LLMs are most reliable when external systems provide structure and the model performs synthesis, explanation, or translation. The study explicitly cautions that its results demonstrate "structural fidelity" rather than a general superiority of LLMs over human experts in scholarly insight. The findings support a shift in literature review research from isolated single-task applications toward structured, multi-stage workflows where the division of labor between algorithmic precision and generative synthesis is carefully managed.