CRISP: Characterizing Relative Impact of Scholarly Publications

The paper introduces CRISP, a method that leverages large language models to jointly rank and characterize the relative impact of all cited papers within a citing document through randomized ordering and majority voting, thereby outperforming prior state-of-the-art classifiers in accuracy and efficiency.

The Gist

Imagine you are a judge on a talent show. You have a contestant (let's call them "Paper P") who has just performed. To decide if they are a star, you look at the other performers they mentioned in their act.

The Old Way: The "Solo Interview"
Traditionally, researchers tried to figure out how important a reference was by interviewing the contestant about one specific person at a time.

• Judge: "You mentioned 'The Jazz Singer.' How important were they to your act?"

• Contestant: "Oh, they were great! I used their rhythm!"

• Judge: "Okay, that's a 10/10!"

• Judge: "You also mentioned 'The Opera Star.' How important were they?"

• Contestant: "Well, I mentioned them because they are famous, but I didn't really use their style."

• Judge: "Okay, that's a 10/10 too!"

The Problem: The judge is getting confused. They are treating the "Jazz Singer" (who was essential) and the "Opera Star" (who was just a name-drop) as equally important because they are judging them in isolation. They miss the fact that the Jazz Singer was the real foundation of the act, while the Opera Star was just a background decoration.

The New Way: CRISP (The "Group Ranking" Party)
The paper introduces a new method called CRISP. Instead of interviewing people one by one, CRISP invites all the people the contestant mentioned to sit in a room together and asks the AI judge to rank them all at once based on how much they actually helped the performance.

Here is how CRISP works, using simple analogies:

1. The "Group Photo" Analogy

Instead of looking at a single photo of the contestant and one reference, CRISP takes a group photo of the contestant and all their references.

• The Insight: When you see everyone together, it becomes obvious who is the "Main Character" and who is just "Extra."
• The Result: The AI can say, "This paper is the High Impact MVP because the whole act revolves around it," while that other paper is just Low Impact background noise. By comparing them side-by-side, the AI understands the relative importance much better.

2. The "Seating Chart" Trick (Fixing the Bias)

Large Language Models (the AI judges) have a weird quirk: they sometimes prefer the people sitting at the top of the list, just because they are at the top. It's like a teacher who accidentally gives the first student on the roll call a better grade just because they were first.

To fix this, CRISP plays a game of musical chairs:

1. It asks the AI to rank the list of references.
2. Then, it shuffles the list (like mixing up a deck of cards) and asks the AI to rank them again.
3. It does this three times with different orders.
4. Finally, it takes a majority vote. If the AI says "Paper A is the best" in two out of three shuffled lists, then Paper A is definitely the best, regardless of where it was sitting.

3. The "Efficiency" Bonus

You might think, "Wait, ranking 50 people at once sounds like a lot of work for the AI!"

• The Old Way: The AI had to read the whole paper 50 times (once for each reference). That's like hiring 50 different judges to interview 50 people individually. Expensive and slow!
• The CRISP Way: The AI reads the paper once, sees the whole list, and ranks everyone in one go. It's like hiring one super-judge to look at the whole group photo and point out the stars. This saves a massive amount of money and time.

Why Does This Matter?

In the world of science, we often count how many times a paper is cited (like counting how many people clapped). But a clap from a friend doesn't mean the same thing as a standing ovation from a critic.

• Old Method: Counts every clap equally.
• CRISP: Distinguishes between a polite nod (background info) and a standing ovation (core methodology).

The Bottom Line:
CRISP is a smarter, cheaper, and faster way to figure out which scientific papers are the real game-changers and which ones are just passing mentions. It stops us from treating a footnote the same as a foundation stone. By looking at the whole picture rather than isolated pieces, we get a much clearer view of what truly matters in science.

Technical Summary

1. Problem Statement

Current methods for assessing the scientific impact of a paper typically rely on citation counts, which are noisy proxies because they treat all citations as equal. More advanced approaches attempt to classify citation intent (e.g., background vs. methodology) or impact by analyzing the citation context in isolation.

The authors identify a critical limitation in these isolated approaches:

• Loss of Relative Context: Evaluating a citation $(p \to \tilde{p})$ based solely on the text surrounding it in paper $p$ ignores the relative importance of that reference compared to other works cited in the same paper.
• Inefficiency: Existing state-of-the-art (SOTA) methods often require one Large Language Model (LLM) call per citation edge. In a graph with $n$ citing papers and $m$ citation edges (where $m \gg n$), this results in $O(m)$ LLM calls, which is computationally expensive and difficult to scale.

2. Methodology: CRISP

The authors propose CRISP (Characterizing Relative Impact of Scholarly Publications), a framework that jointly ranks all cited papers within a single citing paper to determine their relative impact.

Core Workflow

1. Corpus Retrieval: Given a target paper $p^*$, retrieve all papers that cite it ($N_{in}(p^*)$).
2. Context Extraction: For each citing paper $q$, extract its full reference list $N_{out}(q)$ and the associated citation contexts $Ctx_{all}(q)$.
3. Joint Ranking & Labeling: Instead of scoring each citation independently, an LLM judge is prompted to rank the entire list of references in $q$ based on their impact on $q$.
   • Label Space: The model assigns impact categories: Low (0), Medium (1), and High (2).
   • Calibration: By seeing all references together, the LLM can calibrate its judgment against the specific citation conventions of paper $q$ (e.g., distinguishing between a paper that superficially cites many works vs. one that relies heavily on a few core references).

Mitigating Positional Bias

LLMs are known to exhibit positional bias (preferring items at the start or end of a list). To address this, CRISP employs Permutation Self-Consistency (PSC):

• The list of references is randomized and processed three times independently.
• The resulting impact labels are aggregated via Majority Voting to determine the final impact category for each citation.
• Alternative: The paper also explores aggregating the three rankings using Reciprocal Rank Fusion (RRF) and predicting labels via an Ordinal Regression model to ensure monotonicity in impact scores.

Efficiency Advantage

• Call Reduction: CRISP makes $O(n)$ calls (one per citing paper, multiplied by 3 for permutations), whereas isolated methods make $O(m)$ calls (one per citation edge). Since $m \gg n$, CRISP is asymptotically more efficient.
• Token Overhead: While the total token volume is similar, CRISP reduces prompt repetition overhead.

3. Experimental Setup

• Dataset: The authors utilized a human-annotated dataset from Arnaout et al. (2025) containing 442 citing papers and 1,338 cited papers.
• Models Tested:
  • Closed-source: GPT-5.1, o4-mini.
  • Open-source: Qwen3-30B-A3B-Instruct-2507-FP8.
• Baselines:
  • Random Classifier.
  • UKP (Arnaout et al., 2025): The prior SOTA method that evaluates citations independently.
• Evaluation Metrics: Accuracy, Precision, Recall, and F1-score (specifically for the "impact-revealing" class).

4. Key Results

CRISP consistently outperformed the prior SOTA (UKP) across all tested models:

Model	Metric	UKP (Baseline)	CRISP	Improvement ($\Delta$)
GPT-5.1	Accuracy	66.7%	78.6%	+11.9%
	F1	55.7	67.7	+12.0
o4-mini	Accuracy	63.0%	65.4%	+2.4%
	F1	56.7	65.3	+8.6
Qwen3	Accuracy	60.8%	75.1%	+14.3%
	F1	56.3	60.5	+4.2

• Average Improvement: Across all models, CRISP improved accuracy by +9.5% and F1 by +8.3%.
• Open-Source Competitiveness: The open-source Qwen3 model achieved competitive performance (75.1% accuracy) compared to the closed-source GPT-5.1, demonstrating cost-effectiveness.
• Qualitative Analysis: Confusion matrices indicate that CRISP significantly reduces false positives (misclassifying non-impactful citations as impactful) while maintaining comparable recall.

5. Key Contributions

1. Joint Ranking Approach: Introduced a method that leverages the full citation context of a paper to assess relative impact, proving that comparative analysis yields better results than isolated evaluation.
2. Performance Gains: Demonstrated a significant boost in accuracy and F1 scores over existing SOTA methods.
3. Scalability & Efficiency: Showed that joint ranking reduces computational costs (LLM calls) by orders of magnitude, making large-scale citation impact analysis feasible.
4. Resource Release: Released rankings for 1,338 cited papers, impact labels, and the full codebase to support future research.

6. Significance and Limitations

Significance:
CRISP shifts the paradigm from "how is this specific sentence citing a paper?" to "how important is this paper relative to the others in this bibliography?" This provides a more nuanced, calibrated, and scalable metric for scientific impact, useful for funding agencies, hiring committees, and researchers.

Limitations:

• Domain Scope: The study focused on Psychology, Medicine, and Computer Science; results may vary in fields with different citation norms.
• Language: Only English papers were analyzed.
• Context Window Constraints: While models have large context windows (200k+ tokens), performance degrades when reference lists exceed ~40 items, with models failing to rank the entire list (missing references).
• API Dependency: The pipeline relies on the Semantic Scholar API; missing metadata breaks the pipeline.
• Strategic Behavior: There is a risk that authors might strategically frame citations to game impact scores if this metric becomes widely adopted.