From Subjective Judgments to Auditable Standards:Protocol-Guided AI Auditing of Website Redundancy
This paper introduces CORA, a protocol-guided auditing framework that decomposes website redundancy into distinct, auditable metrics (repetition load, normal-use tax, and failure-domain recovery reserve) using versioned vision-language models and strict validation gates to demonstrate improved predictive accuracy over scalar baselines while emphasizing its current status as a controlled benchmark candidate rather than a general standard.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
We live in an age where websites are built with the help of artificial intelligence, and those same tools are increasingly asked to judge the quality of the very pages they help create. But measuring the quality of a digital interface is rarely as simple as counting errors. A feature that feels like helpful backup to one person might feel like distracting clutter to another, depending on what they are trying to do and what might go wrong. In the field of usability, researchers have long struggled to turn these shifting, subjective feelings into something solid and repeatable. The challenge is not just to ask a computer to look at a screen and say "this is messy," but to define exactly what "messy" means, to prove that the computer is looking at the right things, and to ensure that the answer doesn't change just because the software version was updated. Without a strict method, a computer's judgment is little more than a guess, no matter how confident it sounds.
A team of researchers at Beihang University and the Beijing Institute of Technology has proposed a new way to handle this problem, treating the evaluation of website redundancy not as a single opinion, but as a rigorous, three-part measurement. They call their method CORA. Instead of asking a model to give a single score for how cluttered a page is, CORA breaks the concept down into three distinct questions. First, it asks what is being repeated: is it the visual decoration, the information text, or the way a user interacts with the page? Second, it asks what that repetition costs a user during normal use, such as the extra clicks or time required to navigate past unnecessary elements. Third, and perhaps most importantly, it asks what remains useful if something breaks. If a primary button fails, does a backup route exist? If the main menu crashes, is there a landmark to guide the user? By separating these three elements—what is repeated, what it costs, and what survives a failure—the researchers aim to create a measurement that is both precise and honest about its own limits.
To test this idea, the researchers built a transparent laboratory environment where they could control every detail of the website and the task. They created thousands of simulated scenarios where they deliberately added or removed repeated elements, knowing exactly what the "correct" answer should be. They then asked two different artificial intelligence models to act as auditors. These models were not allowed to simply guess; they were required to provide specific evidence for every claim, pointing to exact parts of the screen that supported their judgment. Before any score could be released, a strict set of independent checks had to pass. The system verified that the model's evidence actually matched the screen, that the logic followed a consistent order, and that the model wasn't just repeating the same answer for every page. If even one of these checks failed, the system would withhold the score entirely, refusing to release a result that could not be fully verified.
The results of this experiment revealed a crucial distinction between a computer that can repeat an answer and a computer that can actually measure a concept. In one set of tests, the researchers found that their three-part method successfully separated the "cost" of repetition from the "reserve" of backup options. When they introduced failures into the simulated websites, the system correctly identified that pages with more backup routes were more robust, even if they looked more cluttered. This separation allowed the system to predict how well a user would succeed in a broken scenario far better than methods that simply counted total clutter. However, when the researchers turned to the artificial intelligence models to do the actual judging, the results were sobering. Both models produced answers that were highly consistent and repeatable, but neither was able to pass the strict verification checks. One model gave answers that looked correct on the surface but failed to point to the right evidence on the screen. The other model simply copied its own observations back to the system instead of providing the requested analysis. Because the verification gate was working as designed, it refused to release a single automated score from either model, even though they had generated thousands of responses.
This refusal to release a score is not a failure of the experiment, but rather its primary success. The study demonstrates that repeatability is not the same as validity. A machine can consistently produce the same wrong answer, or a consistent answer that lacks any real grounding in the visual evidence. By building a system that checks the evidence before releasing the judgment, the researchers showed that it is possible to catch these failures automatically. The study did not find a magic bullet that allows computers to perfectly judge website quality today. Instead, it provided a blueprint for how such a system could be built: one that treats the computer as a versioned instrument that must be calibrated and checked, rather than an oracle that simply speaks the truth. The researchers conclude that while their method works well in a controlled, simulated environment, it is not yet ready for the real world. It has not been tested on actual websites used by real people, nor has it been compared against human experts to see if the computer's "withheld" scores align with human intuition. For now, the system stands as a proof of concept: a way to ensure that when we do eventually let machines judge our digital spaces, we know exactly what they saw, what they missed, and why they decided to stay silent.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.