💻 computer science

Demystifying Funding: Reconstructing a Unified Dataset of the UK Funding Lifecycle

This paper presents a reconstructed, unified dataset of the UK funding lifecycle that integrates previously disconnected sources—specifically the Gateway to Research database, funding opportunities, and competitive decision records—to enable a holistic analysis of the entire process from opportunity announcement to research outcomes.

Original authors: William Thorne, Rupert Shepherd, Diana Maynard

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: William Thorne, Rupert Shepherd, Diana Maynard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the UK's research funding system as a massive, bustling airport.

Every day, thousands of scientists (the passengers) want to board planes (get funding) to fly to new discoveries. The UK government (UKRI) runs the airport, publishes the flight schedules (funding opportunities), checks the tickets (panel reviews), and tells everyone who made it to the gate (funded projects).

The Problem:
Until now, the public could only see the arrival board showing which planes successfully landed (the funded projects). We could see who got on the plane, but we couldn't see:

The original flight schedule (the opportunity).
The list of people who tried to buy tickets but were turned away (unfunded proposals).
The secret meeting where the gate agents decided who got to board (the panel decisions).

Because these pieces of the puzzle were scattered in different rooms, locked in different filing cabinets, or hidden behind "Do Not Disturb" signs, it was impossible to see the whole picture. Researchers couldn't tell if the system was fair, or why some people got on the plane while others with similar tickets were left behind.

The Solution: The "Flight Tracker" Project
The authors of this paper, William, Rupert, and Diana, decided to build a super-powerful flight tracker. They went into the airport's back offices and stitched together three separate, messy databases into one giant, unified map.

Here is how they did it, using simple analogies:

1. Gathering the Scattered Clues

They had to hunt down three types of data that didn't talk to each other:

The Flight Schedules (Opportunities): These were written in long, confusing paragraphs on websites. Some were short, some were huge.
The Gate Agent Decisions (Panel Outcomes): This was the hardest part. Some airports (Research Councils) posted their decisions in neat spreadsheets. Others hid them in messy PDFs that looked like ancient scrolls. Two major airports even locked their data behind a digital fence (Tableau) where you couldn't download the list, claiming it was for "safety" but really just making it hard to see who got picked.
The Arrival Board (Funded Projects): This was the existing database (Gateway to Research), which was full of holes and missing links.

2. The "Robot Librarian" (AI Extraction)

Since the flight schedules were written in messy human language, the team built a Robot Librarian (an AI system).

The Challenge: If you asked the librarian, "How much money can I get?" and the answer was buried in a 5,000-word document, a normal search might miss it.
The Trick: Instead of reading the whole book at once, the Robot Librarian learned to chop the documents into logical "chapters" (like "Eligibility," "How to Apply," "Money"). It then used a smart search to find the exact chapter that held the answer, rather than getting lost in the noise.
The Result: This robot became incredibly good at pulling out specific numbers (like "Max Award: £50,000") from the messy text, achieving about 87% accuracy.

3. Connecting the Dots (Data Linking)

Now they had three piles of paper: Schedules, Gate Decisions, and Arrivals. But the names didn't match perfectly. One list said "Dr. Smith," another said "Smith, J."

The team acted like detectives, using fuzzy matching to say, "Hey, these two names are probably the same person."
They linked the Application (the ticket request) to the Opportunity (the flight) and the Panel Decision (the gate agent's nod).
The Big Win: For the first time, they could see the entire journey. They could see that 100 people applied for Flight A, 20 got rejected, and only 5 got on the plane. Before, we only knew about the 5 who got on.

Why Does This Matter?

Think of it like a game of cards.

Before: You could only see the winning hand. You didn't know what cards everyone else played, or if the dealer was cheating.
Now: You can see every card played by every player. You can finally ask: "Was the game fair? Did the rich players get better cards? Did the gate agents have a favorite team?"

The "Elephant in the Room"
The authors also pointed out a frustrating truth: Some of the airport's most important data (the panel decisions) is currently locked behind a paywall or a "no-download" sign, even though the government says everything should be open. They argue that if we want to know if the funding system is fair, we need to be allowed to see the gate agents' notebooks.

In Summary
This paper is about demystifying the magic box. The authors took a broken, scattered, and partially hidden system and rebuilt it into a clear, transparent window. They released the code and the data so that anyone—journalists, scientists, or curious citizens—can now look at the whole funding process and ask the hard questions about fairness and transparency.

1. Problem Statement

The paper addresses critical gaps in the UK Research and Innovation (UKRI) Gateway to Research (GtR) database, which is the primary public record of UK-funded research. The authors identify three main issues:

Data Quality & Completeness: GtR suffers from missing links, duplicate entries, inconsistent classifications, and a lack of unique identifiers for organizations.
Lifecycle Fragmentation: GtR primarily records funded projects and their outcomes but lacks a public link to the funding opportunities that solicited them or the panel meeting outcomes that determined funding. This creates a "black box" where the criteria for rejection and the decision-making process remain opaque.
Access Barriers: Panel decision data is published inconsistently (mix of PDFs, spreadsheets, and restricted dashboards). Two major councils (EPSRC and MRC) publish data exclusively via third-party dashboards (Tableau) with data exports disabled, hindering transparency and bias analysis despite CC BY-NC-SA 4.0 licensing.

The authors argue that this lack of transparency prevents rigorous scrutiny of funding fairness, peer review bias, and strategic investment management.

2. Methodology

The authors constructed a unified database by integrating three previously disconnected data sources and developing novel extraction pipelines.

A. Data Sources Integration

Gateway to Research (GtR): Extracted via all available API endpoints (projects, persons, organizations, outcomes). Data was ingested into a PostgreSQL schema using joined table inheritance for outcomes.
Funding Opportunities: Retrieved from the UKRI opportunity finder. Raw HTML was parsed into a tree-structured representation where the summary is the root and sections (e.g., "How to apply") are children. This structure facilitates hierarchical retrieval.
Panel Meeting Outcomes: Aggregated from council websites, the Government Web Archive, and spreadsheets. This involved significant manual effort to normalize inconsistent formats (e.g., color-coded Excel cells vs. unstructured PDF tables).

B. Metadata Extraction (Opportunities)

Since funding details (award values, duration, contribution %) are often buried in unstructured text, the authors framed this as a closed-domain Question Answering (QA) task.

Hierarchical Retrieval: Instead of simple sliding windows, they exploited the document tree structure. Each node (section) and its descendants formed a "chunk."
Hybrid Scoring: Chunks were scored using a weighted combination of BM25 (lexical) and Dense Cosine Similarity (semantic, via all-MiniLM-L6-v2).
- Formula: $S_{hybrid} = \alpha \cdot S_{BM25} + (1-\alpha) \cdot S_{\theta}$
Re-ranking: Top- $k$ candidates were re-ranked using a cross-encoder (ms-marco-MiniLM-L-6-v2).
Branch Deduplication: If an ancestor-descendant pair was selected, the lower-scoring chunk was discarded to prevent redundancy.

C. Entity Linking & Alignment

To unify the three sources, the authors performed conservative entity linking (prioritizing precision over recall):

Application $\to$ Project: Linked via exact case-insensitive matching of application_id/award_id against grant_reference.
Application $\to$ Opportunity: Linked via fuzzy title matching, constrained by council agreement, date ordering, and award range validation.
Panel Attendance $\to$ Person: A two-phase process involving clustering by (surname, initial) and aligning clusters to GtR persons based on project membership and organization similarity.

3. Key Contributions

Unified Dataset: The first public dataset linking UKRI funding opportunities, competitive panel decisions, and resulting research outcomes, covering the entire funding lifecycle (including unfunded proposals).
Technical Pipeline: A robust pipeline for extracting structured metadata from unstructured funding documents using hierarchical retrieval and hybrid scoring, outperforming standard baselines.
Transparency Advocacy: Documentation of specific barriers to data access (e.g., Tableau restrictions) and the release of code and data to enable independent audit of the funding process.
Expanded Scope: Inclusion of 38,862 applications (many unfunded) and 1,449 panel meetings, providing context for competitiveness and bias analysis that was previously impossible.

4. Results

Dataset Scale (Table 3):
- Projects: 173,220
- People: 141,833
- Organizations: 89,884
- Outcomes: 2,619,446
- Opportunities: 2,053
- Applications: 38,862 (Only ~27% linked to GtR projects, reflecting the volume of unfunded proposals).
Extraction Accuracy (Table 2 & 5):
- The Hierarchical Chunking strategy with $\alpha=0.25$ (25% BM25, 75% Dense) and no re-ranker achieved the best performance: 87.7% overall accuracy.
- This was a 2.4 percentage point improvement over the full-document baseline (85.3%).
- Error Analysis: 78% of errors were false positives, primarily due to field confusion (e.g., confusing "total funding" with "maximum award") and hallucinations. The model struggled most with "maximum award" (66.7% accuracy) due to ambiguous phrasing in documents.
Linking Success:
- 27.1% of applications linked to GtR projects (expected, as GtR only lists funded ones).
- 29.6% of applications linked to opportunities via fuzzy matching.
- 64.4% of panel attendance records linked to specific persons.

5. Significance

Democratizing Research Policy: By making the "black box" of panel decisions visible, the dataset enables researchers to study funding fairness, bias (gender, racial, institutional), and the impact of peer review on novel vs. conventional proposals.
Methodological Benchmark: The hierarchical retrieval approach for structured metadata extraction from complex government documents offers a replicable framework for similar information extraction tasks.
Policy Impact: The work highlights the contradiction between UKRI's open licensing and its restrictive data export practices, providing evidence to support calls for greater transparency in public research funding.
Holistic Analysis: Researchers can now analyze the full pipeline from opportunity announcement to rejection/funding, allowing for a deeper understanding of why certain proposals fail and how the system allocates resources.

The authors conclude that while the dataset inherits some of GtR's historical data quality issues, it represents a significant leap forward in the transparency and analytical potential of UK research funding data.