Technical Summary: Designing LLM Prompts to Extract Scores from Messy Text

Problem Definition

The paper addresses a specific challenge in Natural Language Processing (NLP): designing prompts for Large Language Models (LLMs) to accurately extract numerical research quality scores from unstructured, "messy" text reports. While LLMs are frequently used to generate text evaluations (e.g., grading essays or assessing research quality), their outputs are often inconsistent. They may embed scores within extensive explanations, omit scores entirely, provide invalid formats (e.g., percentages or non-standard scales), or fail to follow strict output constraints.

The specific task involves processing reports generated by LLMs that evaluate journal articles based on the UK Research Excellence Framework (REF) scale (1* to 4*). The core difficulty lies in two simultaneous requirements:

1. Extraction Accuracy: Correctly identifying the overall score or calculating an average of component scores (Originality, Significance, Rigour) when an overall score is absent.
2. Output Constraint: Ensuring the LLM returns only the numerical score (and an optional asterisk) or a specific missing value code (-1), without any explanatory text, punctuation errors, or formatting deviations.

Methodology

To facilitate research into this problem, the author introduces a shared dataset and a structured challenge.

Dataset Construction

The dataset consists of 1,446 short texts derived from reports generated by three medium-sized LLMs: Magistral, Llama 4 Scout, and Gemma3 27b. These models were tasked with scoring journal articles based on titles and abstracts.

• Source Filtering: The process began with hundreds of thousands of raw responses. Duplicate removal was performed first within each model's output and then across models.
• Manual Curation: To maximize difficulty and variety, the remaining large set was manually filtered in Excel. Similar reports were removed to ensure the final set contained a wide variety of styles, including texts with missing scores, invalid scores, and ambiguous formatting.
• Gold Standard: A human expert (the author) manually judged the correct score for each text. This "gold standard" includes the specific score (1*–4*) or -1 if the score is unclear or missing. In cases where three component scores were present without an overall score, the average was calculated as the gold standard.

The Challenge Task

Participants are tasked with designing an LLM prompt to process the 1,446 texts. The output must strictly adhere to a specific format:

• Valid Outputs: A number (integer or decimal) optionally followed by an asterisk (e.g., 3*, 2.667, 4 *) or the code -1.
• Invalid Outputs: Any response containing characters other than digits, spaces, asterisks, or the minus sign (e.g., "Score: 3*", "World-leading", or "3* because...").
• Scoring Logic: The system must return the reported overall score if available. If not, and if Originality, Significance, and Rigour scores are all present, it must return their average. If the text lacks a clear score or contains invalid data, it must return -1.

Evaluation

The paper provides a baseline solution using a specific prompt with ChatGPT 4.1-mini via API. The results are evaluated by comparing the LLM's output against the gold standard, allowing for a small rounding tolerance (within 0.005 of the exact value).

Key Results

The baseline experiment yielded the following statistics:

• Total Items: 1,446
• Correct Answers: 1,050
• Accuracy: 72.61%

The paper notes that the baseline prompt often fails in two ways:

1. Format Violations: The model includes explanatory text or incorrect symbols.
2. Extraction Errors: The model fails to correctly average component scores or misinterprets invalid scales (e.g., treating a 5-point scale as valid for the 4-point REF system).

Significance and Contributions

The paper makes three primary contributions to the field of information science and NLP:

1. A Shared Resource: It provides a curated, "messy" dataset of 1,446 research quality reports with a verified gold standard, enabling direct performance comparison between different prompt engineering strategies without the overhead of dataset creation.
2. A Benchmark Challenge: It establishes a clear, measurable task for the community to improve upon the 72.61% baseline. The challenge encourages the development of prompts that can handle complex numerical extraction and strict formatting constraints simultaneously.
3. Insight into LLM Capabilities: By focusing on a task that appears transparent (extracting a number) but contains hidden complexities (ambiguous scales, missing values, and formatting noise), the challenge aims to deepen the understanding of LLM limitations and the nuances of effective prompt design for numerical tasks.

The author positions this work as a practical strategy to increase efficiency in research, allowing multiple teams to generate solutions for a common task, and suggests the challenge is suitable for both researchers and student projects.