Research Entity Extraction and Topic Detection from UKRI Grant Proposals

Technical Summary: Research Entity Extraction and Topic Detection from UKRI Grant Proposals

Problem Statement
Publicly funded research is central to national innovation and economic growth, prompting governments to seek methods for identifying and prioritizing emerging research areas early in their development. While "ex-ante" funding proposals (research yet to be funded) contain valuable signals of future excellence and "unicorn" domains, systematically extracting these signals at scale presents significant technical challenges. Traditional methods often rely on fixed vocabularies or keyword co-occurrence, which struggle to capture novel concepts and emerging terminology that have not yet been formalized in existing taxonomies. Furthermore, the reliability and comparability of different Large Language Model (LLM) approaches when applied to complex, domain-specific grant texts remain under-explored. This paper addresses the need to evaluate automated methods for extracting and classifying research entities to inform public investment strategies.

Methodology
The study, part of the UKRI-funded "Tracking Stars and Unicorns" project, employs a modular three-stage pipeline to convert unstructured grant proposal text into structured data. The evaluation was conducted on a randomized dataset of 42 funded proposals from seven UK statutory research councils and a specialist national centre, ensuring broad disciplinary coverage.

1. Entity Extraction: The pipeline utilizes the Mistral language model, primed via iterative prompt engineering with structured instructions and exemplar-based guidance. This approach identifies discrete semantic units (research entities) including thematic topics, disciplines, methodologies, software, datasets, and significant events. The model is instructed to preserve original surface forms and decompose complex entities while avoiding the extraction of non-substantive elements.
2. Topic Classification: Extracted entities are mapped against the OpenAlex Topics taxonomy, a four-level hierarchy (domain → field → subfield → topic). The study compares a bespoke algorithm, DSIT-Taxonomies (developed by the Innovation Growth Lab and DSIT), against LLM-based approaches. The DSIT-Taxonomies tool uses standard libraries (e.g., DBpedia Spotlight, RAKE, YAKE, KeyBERT) for extraction and semantic similarity for classification.
3. Refinement: To address granularity inconsistencies in the raw DSIT-Taxonomies output, the authors implemented a refinement procedure. This enforces a mandatory downward-traversal constraint, ensuring every proposal is mapped to a leaf "topic" node within the OpenAlex hierarchy, thereby providing uniform resolution for statistical analysis.
4. Evaluation Framework: The study compares three approaches: Mistral, GPT-4o, and the DSIT-Taxonomies algorithm. Evaluation metrics include:
   • Entity Extraction: Statistical volume analysis and pairwise Jaccard similarity (using semantic matching via Sentence-BERT embeddings) to measure overlap.
   • Topic Classification: Accuracy rates determined by manual human-in-the-loop assessment of 42 proposals, resolving disagreements through consensus.

Key Results

• Entity Extraction Performance: Mistral and GPT-4o produced comparable, high-quality entity sets with significant semantic overlap (Jaccard similarity of 0.374). In contrast, the DSIT-Taxonomies approach yielded a significantly higher volume of entities (averaging ~25 per proposal vs. ~15 for LLMs) but showed low alignment with the LLM approaches (Jaccard similarity ~0.19).
• Quality and Hallucination: Content analysis revealed that DSIT-Taxonomies often fragmented complex compound nouns and extracted non-substantive elements (verbs, quantifiers), introducing noise. While both LLMs exhibited occasional hallucinations, Mistral was more conservative, producing fewer non-existent entities (mean error rate of 1.56 per proposal vs. 2.84 for GPT-4o) and achieving zero hallucinations in 16 out of 42 cases.
• Topic Classification Accuracy: The Mistral-based approach significantly outperformed the full DSIT-Taxonomies pipeline in topic classification accuracy. When using entities extracted by Mistral, the classification accuracy reached 90.5% (38/42 correct), compared to 71.4% (30/42 correct) for the standard DSIT-Taxonomies pipeline.

Key Contributions and Significance
The paper contributes a comparative assessment of LLM-based versus traditional algorithmic approaches for analyzing sensitive grant data. Its primary significance lies in demonstrating that Mistral offers a high-performance, operationally efficient, and secure solution for large-scale analysis. Unlike GPT-4o, Mistral's lightweight architecture allows for deployment within secure, internal government platforms, ensuring data sovereignty for confidential grant applications while maintaining analytical depth.

The study concludes that while LLMs (specifically Mistral and GPT-4o) generate comparable entity sets, the Mistral-based pipeline, when coupled with a refined classification logic, provides superior accuracy in mapping proposals to standardized research topics. This validates the use of generative models for identifying early signals of emerging research areas, supporting the project's goal of informing public investment to catalyze significant downstream societal and economic impacts. The authors note that these findings are preliminary and will be applied to a larger corpus of 350,000 funded and unfunded proposals to analyze long-term trends and benchmark domestic progress against global competitors.