A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization

Imagine you are reading a story written by a very talented, but slightly mischievous, robot. This robot is an expert at "abstractive summarization," which is just a fancy way of saying it reads a long article and writes a short, punchy version of it in its own words. It's great at sounding natural and fluent; the sentences flow like a river. But here's the catch: sometimes, in its quest to sound cool, the robot makes things up. It might keep the names of the people right but swap who did what to whom. It's like a news anchor saying, "The President signed the treaty," when the document actually says, "The treaty was signed by the Secretary." The words are there, but the truth is twisted. This is called a "hallucination," and in the world of artificial intelligence, it's a big problem because we need our summaries to be not just pretty, but true.

For a long time, scientists trying to catch these lying robots used tools that were like checking a grocery list. They counted how many words the robot used that were also in the original article. If the robot used the word "apple" and the original had "apple," that was a point in its favor. But this is like judging a chef by how many ingredients they used, without tasting the dish. You could have a bowl full of raw, uncooked ingredients that matches the list perfectly, but it's still a terrible meal. The old tools were good at checking if the robot remembered the names, but they were terrible at checking if the robot remembered the relationships between those names. They couldn't tell the difference between a robot that told the truth and one that told a very convincing lie.

This is where a team of researchers from IIIT Bhubaneswar and some industry partners steps in with a new, sharper pair of glasses. They realized that to catch a liar, you don't just count words; you have to understand the story's structure. They built a new framework to evaluate summaries that acts like a detective looking for "relation-level hallucinations." Instead of just asking, "Did you use the right words?" they ask, "Did you connect the right dots?"

The researchers created a new scoring system called the Relation Hallucination Index (RHI). Think of the original text as a complex web of strings connecting different characters and events. A good summary should keep those strings intact. A hallucinating summary might cut a string and tie it to the wrong person. The old tools would miss this because the characters (the words) were still there. The new RHI, however, is designed to trace every single string. It uses a sophisticated process to break the text down into tiny building blocks called "triples"—basically, a Subject (who), a Verb (did what), and an Object (to whom).

To make sure these triples are accurate, the team didn't just use a basic scanner. They built a "dependency-aware" extraction engine. Imagine this engine as a very careful editor who doesn't just read the words but understands the grammar deep down. It knows how to handle tricky sentences where the subject is hidden (like in passive voice, "The ball was thrown" becomes "Someone threw the ball"). It knows how to ignore words that just report speech ("He said that...") and focus on the actual facts. It even cleans up the words by converting them to their base form (changing "running" to "run") so it doesn't get confused by different tenses. This ensures that when they compare the robot's summary to the original, they are comparing apples to apples, not apples to apple-pies.

The team tested this new system on a massive playground of data. They used three different types of news datasets: XSUM (which has very short, highly creative summaries), XLSUM (which has news from many different countries and languages), and the CNN/DailyMail dataset (which has longer, more detailed stories). They asked four famous AI models—BART-large-CNN, PEGASUS, T5-large, and GPT-3.5—to write summaries of 800 different news stories. They also included a "human reference" summary to see how close the robots could get to a perfect human score.

The results were eye-opening. When they used the old, word-counting tools, PEGASUS often looked like the winner because it used a lot of the same words as the original. But when the researchers applied their new RHI score, the picture changed. BART actually scored the highest in preserving the true relationships between events, even though it didn't always use the exact same words. GPT-3.5, while good at some things, showed a lot of "relation insertion variability," meaning it sometimes made up new connections that weren't there. The study found that the new RHI score was much better at spotting these subtle lies than the old methods.

One of the most interesting things they found was that the new score is "normalized." This means it works fairly well whether you are looking at a short, snappy summary or a long, detailed one. They broke down the errors into different categories: things the model got right, things it made up, things it missed, and things it got wrong. By looking at these pieces, they could see exactly how a model was failing. For instance, they found that while some models were good at keeping the main facts, they were terrible at handling negative statements (like "The company did not go bankrupt").

The researchers ran statistical tests to make sure their findings weren't just luck. They found that the differences in scores between the models were significant, meaning the new RHI score is a reliable way to tell a good, truthful summary from a bad, hallucinating one. They also showed that their new method is better than the old "Entity Hallucination Index" (which only checked if names were right) because it checks the actions and connections between those names.

In the end, this paper suggests that if we want AI to be trustworthy, we need to stop just counting words and start checking the story's logic. The new Relation Hallucination Index offers a way to do exactly that. It doesn't just tell us if a summary sounds good; it tells us if the story it tells is actually true. The authors hope that in the future, this kind of score can be used to train AI models to be more honest, helping them learn that being fluent isn't enough—you have to be faithful to the facts, too.