Automating the Open Data Maturity Index with a Multi-Agent LLM System
This dissertation presents a multi-agent LLM system that automates the Open Data Maturity Index assessment by achieving 70.1% accuracy against official 2025 responses through a collaborative research, verification, and adjudication workflow, while highlighting the need to reformulate the questionnaire to prioritize discoverable evidence over self-reported practices to overcome the limitations of open-web data.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every year, a team of experts in Brussels sits down to judge how well thirty-six European countries are sharing their public data. They ask a long list of questions: Does the government have a plan for open data? Is the national website easy to use? Do the datasets actually work? For each country, the national team answers these questions themselves, and then a human checker reviews the answers to make sure they are true. This process is slow, expensive, and relies heavily on trust. The question researchers at King's College London asked was simple: could a team of artificial intelligence agents do this job instead? They wanted to know if machines could scour the open internet, find the proof, and grade the countries without human help, while still being fair and accurate enough to be trusted.
The researchers built a system that works like a small committee of three digital agents. The first agent, the Researcher, reads the question and searches the web for evidence. If it finds a document that looks like an answer, it proposes a score. The second agent, the Verifier, acts as a skeptic. Its only job is to try to prove the Researcher wrong, searching for evidence that contradicts the claim. If the two agents cannot agree, a third agent, the Adjudicator, steps in to make the final call. Crucially, this system has a way to say "I don't know." If the agents cannot find enough proof on the public web, they are programmed to decline the question rather than guess. This is vital because the internet is full of gaps; just because a website does not mention a practice does not mean the practice does not exist.
When the team tested this system on a sample of eight countries, answering over a thousand questions, they found that the machines could do a good job, but only under specific conditions. The system successfully answered about half of the questions, matching the human experts' answers roughly seven times out of ten for the questions it chose to answer. However, it refused to answer the other half. The reason for this refusal was not a lack of intelligence, but a lack of evidence. The system was very good at finding proof that something exists. If a country had a policy or a feature on its website, the agents could find the document and quote it. But the system struggled immensely with questions that asked if something was missing. If a country did not have a certain practice, there was often no public document saying "we do not do this." The agents could not distinguish between a country that had no such practice and a country that simply had not published the proof that it did.
This limitation revealed a deeper problem with how the assessment is currently designed. The researchers discovered that the system's accuracy was not the same for every country. It performed best on the countries that were already doing the most to share their data. These nations had rich, detailed websites and many public documents, which gave the AI plenty of material to work with. The countries with less open data, which had fewer websites and fewer documents, were the ones the system struggled with most. In fact, the system was so unsure about the countries with less open data that it often declined to answer, or worse, guessed incorrectly because it mistook unrelated information for proof. The study showed that the more open a country was, the easier it was for the machine to grade it, creating a cycle where the system was most accurate for the countries that were already doing well.
The researchers also tested whether having a "skeptical" agent was necessary. They swapped the Verifier's role from trying to disprove the answer to trying to confirm it. Surprisingly, this change made almost no difference to the final score. The system did not become more accurate, nor did it become less accurate. The benefit of having multiple agents came not from their opposing views, but simply from the fact that they searched the web multiple times, gathering more evidence. The debate between the agents did not solve the core problem: the internet simply does not contain the evidence needed to prove that a practice is absent.
In the end, the study suggests that a fully automated assessment is not yet possible for the entire questionnaire, but a hybrid approach could work. The researchers propose rewriting the questions to ask for public proof rather than internal promises. Instead of asking a government team "Do you monitor search terms?", the new questions would ask "Show us the report you published about what you learned from monitoring search terms." This shift would force the evidence to exist on the open web, making it possible for the machines to verify it. For the parts of the assessment that still rely on internal knowledge, human experts would remain necessary. The machine could handle the half of the questions that can be answered with public documents, flagging the rest for human review. This would not only speed up the process but also make the assessment more honest, ensuring that a country's score is based on what it has actually published, not just what it claims to have done.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.