How Do LLMs Encode Scientific Quality? An Empirical Study Using Monosemantic Features from Sparse Autoencoders

1. Problem Statement

While Large Language Models (LLMs) are increasingly used to evaluate, generate, and synthesize scientific work, their internal mechanisms for understanding "scientific quality" remain opaque. Current research confirms LLMs can mimic human evaluation to some extent, but they function as "black boxes," making it difficult to understand how they encode concepts like rigor, relevance, and impact. This lack of interpretability hinders the trustworthiness of AI in research evaluation and limits our understanding of the model's internal representation of scientific knowledge.

2. Methodology

The authors propose a systematic framework to extract and analyze monosemantic features (interpretable, distinct concepts) from LLM activations using Sparse Autoencoders (SAEs). The methodology consists of four main stages:

• Data Preparation:

  • Source: A corpus of 38,639 Computer Science papers from the Academia/Industry DynAmics Knowledge Graph (AIDA KG).
  • Metrics: Three bibliometric proxies for quality were used: Citation Count, Journal SCImago Journal Rank (SJR), and Journal h-index.
  • Task Formulation: The continuous metrics were converted into quartiles. The study focused on binary classification tasks comparing the highest (1st) vs. lowest (4th) quartiles for each metric to create balanced datasets.

• Feature Extraction (SAE):

  • Models: Experiments utilized Gemma 2 2B and Gemma 2 9B-it.
  • SAE Configurations: Three settings were tested using different layers and SAE variants (e.g., gemma-scope-2b-pt-res and gemma-scope-9b-it-res at layers 20 and 31).
  • Process:
    1. The LLM generated a standardized three-sentence summary for each paper to control token length.
    2. SAEs were applied to the model's hidden states to extract sparse, monosemantic feature vectors for each token.
    3. Token-level features were averaged to create a single, interpretable vector representation per paper.

• Classification & Analysis:

  • Classifier: A Decision Tree was trained on the extracted monosemantic features to predict the quality quartiles. Decision trees were chosen for their high interpretability and ability to perform feature subset selection.
  • Baseline: A standard BERT classifier trained on tokenized text was used as a benchmark for performance comparison.
  • Interpretation: The top features from the decision trees were analyzed using Neuropedia (a tool for feature behavior analysis) and manual inspection of high-activation papers to assign semantic meaning to the feature indices.

3. Key Contributions

• First Systematic Investigation: This is the first study to explicitly map how LLMs encode the abstract concept of "scientific quality" using monosemantic features.
• Feature Discovery: The authors identified four recurring types of features that LLMs use to distinguish high-quality research:
  1. Research Methodologies: Features detecting rigorous protocols (e.g., randomized controlled trials, specific experimental procedures).
  2. Publication Type: Features distinguishing surveys and literature reviews (often associated with higher impact) from other types.
  3. High-Impact Fields & Technologies: Features signaling emerging or high-prestige domains (e.g., big data, IoT, specific communication standards).
  4. Scientific Jargon & Style: Features related to specialized terminology, academic conventions, and disciplinary identity.
• Cross-Model Consistency: The study demonstrates that these quality-related features are robust across different model sizes (2B vs. 9B) and SAE configurations.

4. Results

• Predictive Performance:
  • The Decision Tree classifier using monosemantic features achieved fair to good accuracy (ranging from ~0.71 to 0.75 average accuracy) across all three tasks.
  • Notably, for Task 1 (Citation Count), the monosemantic feature-based tree outperformed the BERT baseline (0.6788 vs. 0.6522), suggesting these features capture highly discriminative information about citation potential.
  • For Tasks 2 (SJR) and 3 (h-index), the BERT baseline performed slightly better, which is expected as BERT optimizes directly on raw text, whereas the SAE approach prioritizes interpretability over raw predictive power.
• Feature Specificity:
  • Citation Prediction: Heavily influenced by publication type (surveys/reviews) and topical keywords (emerging technologies), aligning with Information Foraging Theory (discoverability drives citations).
  • SJR Prediction: Strongly correlated with disciplinary fields (e.g., communication networks, electrical grids) and methodological rigor (clinical trials, ML techniques). This suggests SJR is linked to the prestige of the specific subfield and the perceived rigor of the study design.
  • h-index Prediction: A hybrid of the previous two, combining field-specific technologies with methodological norms and academic language markers.

5. Significance

• Demystifying LLMs: The study moves beyond treating LLMs as black boxes, providing concrete evidence that they internalize latent cues (methodology, field trends, style) that humans use to assess research quality.
• Research Evaluation: The findings suggest that LLMs can be made more transparent in peer review and grant evaluation by exposing the specific "monosemantic" concepts driving their decisions.
• Future Directions: The authors highlight that while current features are interpretable, future work is needed to refine these definitions, extend the analysis to other model architectures, and explore how these features can be used to improve LLM performance in scientific discovery and text generation tasks.

In conclusion, the paper establishes that LLMs do not merely process text statistically but encode a multidimensional, structured representation of scientific quality that aligns with established scientometric indicators.