EViSE: An Efficient Framework for Multi-Criteria Summarization Evaluation
The paper proposes EViSE, an efficient multi-criteria summarization evaluation framework that combines cardinal-ordinal regression with within-source ranking to achieve high agreement with human judgments across faithfulness, coherence, and relevance while enabling fast, reference-free inference and effective use as a reward signal for policy optimization.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, machines have become remarkably good at reading a long text and writing a shorter version of it. This process, known as summarization, is useful for everything from digesting news articles to condensing medical reports. However, a significant problem remains: how do we know if the summary is actually good? For years, researchers have relied on simple computer programs that count how many words the summary shares with the original text or a human-written example. While these tools are fast and consistent, they miss the deeper qualities that make a summary useful. They cannot easily tell if the summary invented facts that were never in the source, if the sentences flow together logically, or if it captured the most important information while leaving out the trivial details. As artificial intelligence models become more sophisticated, the need for a smarter way to judge their work has become urgent, especially when these models are being trained to improve themselves.
A team of researchers from universities in Vietnam has developed a new system called EViSE to solve this problem. Instead of relying on simple word-matching, EViSE acts like a careful human editor who checks a summary against the original document for three specific qualities. First, it checks for faithfulness, ensuring the summary does not make up facts or misrepresent the source. Second, it evaluates coherence, looking at whether the sentences make sense together and form a smooth story. Third, it measures relevance, determining if the summary kept the most vital information and discarded the noise. The researchers built this system using a massive collection of 80,856 pairs of Vietnamese news articles and their summaries, where human experts had already graded each summary on these three distinct criteria.
The core innovation of EViSE lies in how it learns from this data. Rather than just trying to guess a single overall score, the system learns to predict the three specific scores simultaneously. It does this by looking at groups of summaries that all come from the same original article. Imagine a teacher grading five different student essays about the same history chapter; the teacher knows that if one essay is better than another, it should receive a higher score, even if the absolute numbers are hard to pin down. EViSE uses this same logic. It learns the exact numerical score for each summary while also learning the correct ranking order of the summaries within each group. This dual approach allows the system to be both precise in its scoring and consistent in its comparisons. The result is a tool that can evaluate a summary in a single, quick pass without needing to see a human-written "perfect" example to compare it against.
When the researchers tested their new framework, the results were clear and robust. They created two versions of the system: one that was small and fast, and another that was larger and more powerful. Both versions showed a strong ability to agree with human judgments. On a test set of news articles, the larger version achieved a correlation score of 0.806 with human ratings, a significant improvement over older methods that rely on word overlap, which struggled to reach scores above 0.58. The smaller, more efficient version was also highly effective, scoring 0.737, and it could process a summary in just 55 milliseconds on standard hardware. This speed is crucial because it means the system can be used repeatedly to guide artificial intelligence models as they learn, acting as a reliable coach that gives instant feedback without slowing down the training process.
The team also tested whether their system could handle topics it had never seen before. They took the models trained on news articles and asked them to evaluate summaries of technical textbooks about information technology. Despite the shift from general news to specialized academic content, the system maintained its ability to judge quality accurately. The larger model retained a correlation of 0.720 with human experts, and the smaller model held steady at 0.691. This suggests that the framework learned general principles of good summarization rather than just memorizing the style of news writing. Furthermore, the researchers demonstrated that this system could be used as a "frozen" reward signal in a reinforcement learning setup. When they swapped a previous evaluator for their new one to train a language model, the resulting summaries were of similar high quality, proving that the system is stable enough to be trusted with the delicate task of teaching other AI models.
The study explicitly challenges the idea that a single score based on word matching is sufficient for modern evaluation. The data showed that traditional tools like ROUGE and BERTScore failed to capture the nuances of factual accuracy and logical flow, often giving high scores to summaries that were fluent but factually incorrect. By contrast, EViSE's multi-criteria approach successfully separated these different dimensions. The researchers found that by combining the learning of exact scores with the learning of relative rankings, the system became much better at identifying the best summaries. They also ruled out the notion that a complex, slow system is the only way to achieve high accuracy; their compact model proved that high-quality evaluation can be achieved with minimal computational cost.
Ultimately, this work provides a practical and efficient path forward for evaluating how well machines summarize information. It moves the field beyond simple counting of shared words toward a more nuanced understanding of what makes a summary trustworthy and useful. By treating evaluation as a problem of learning multiple criteria at once, the researchers have built a tool that is not only accurate but also fast enough to be used in real-time training loops. The success of this framework in Vietnamese, a language with fewer resources than English, suggests that the approach is flexible and can be adapted to other languages and domains. As artificial intelligence continues to generate more text, tools like EViSE will be essential for ensuring that the output remains faithful to the truth, coherent in its delivery, and relevant to the reader's needs.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.