CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility - Semantic Metrics and Convergence Analysis
The paper introduces Credence, a claim decomposition framework that replaces token-overlap metrics with a semantic similarity-based evaluation (Semantic-F1) and provides formal convergence guarantees for repair pipelines, demonstrating significant improvements in decomposition accuracy and fact-checking reliability across multiple domains and models.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital age, machines are increasingly tasked with verifying the truth of statements found in news articles, social media posts, and official reports. This field, known as automated fact-checking, relies on computers to read a claim, search for evidence, and decide if the statement is true or false. However, a fundamental hurdle exists: human language is rarely simple. A single sentence often bundles multiple distinct facts together, such as a report stating that a leader announced a new policy and that unemployment numbers subsequently fell. To a computer, this is a tangled knot. If the machine tries to verify the entire sentence at once, it often fails because the evidence might support the policy announcement but contradict the economic data. To succeed, the machine must first untangle the sentence, breaking it down into its smallest, independent pieces of information. This process is called claim decomposition. For years, researchers have struggled to measure how well computers perform this task. The standard tools used to grade these systems were flawed, often penalizing a computer for using different words to say the same thing, and the methods used to fix mistakes were unproven, sometimes making the output worse before making it better.
A team of researchers from Vietnam has introduced a new framework called CREDENCE to solve these specific problems. Their work focuses on two main improvements: a better way to grade the quality of the broken-down sentences and a more reliable method for fixing errors when the computer gets stuck. The researchers argue that the old grading systems were too rigid. They relied on counting how many words overlapped between the computer's answer and a human's answer. This approach failed when a computer correctly identified a fact but phrased it differently, such as changing "he" to a full name like "Albert Einstein." The old system would mark this as a failure because the words did not match, even though the meaning was identical. The new framework replaces this word-counting method with a system that understands the meaning of the sentences. By analyzing the semantic essence of the text, the new metric can recognize that two different sentences convey the same truth, giving the computer credit for its understanding rather than punishing it for its vocabulary.
Beyond just grading, the team addressed the chaotic process of fixing errors. Previously, systems would often run in a loop where a computer would try to fix a mistake, only to introduce a new one, creating a cycle of degradation. The researchers applied a formal analysis to this repair process, proving mathematically that a specific type of rule-based fix always moves the system closer to a correct answer and eventually stops. They also demonstrated that when a computer tries to fix its own work using a more advanced language model, it can sometimes make things worse. To prevent this, they built a safety mechanism that stops the computer from trying to fix itself unless the initial, simpler rules have failed. This ensures that the system does not over-correct and ruin a good answer.
The team tested their new framework on a wide variety of text, including social media posts, encyclopedia entries, and news articles. They compared their system against several existing models, ranging from smaller, open-source programs to powerful, commercial artificial intelligence systems. The results showed that their new grading system consistently gave a much higher and more accurate score to correct decompositions than the old methods, with improvements ranging from 15 to 32 percentage points depending on the dataset. In practical terms, this means the system is far better at recognizing when a computer has successfully untangled a complex sentence. Furthermore, when they applied their repair process, the number of sentences that still contained multiple facts dropped significantly, often by nearly half or more, without losing any of the important details like names, dates, or numbers.
Perhaps most importantly, the researchers showed that this improved decomposition directly helps the final goal of fact-checking. When they fed the cleanly broken-down sentences into a verification system, the accuracy of the truth-checking improved dramatically, especially for complex scientific claims where multiple facts are often bundled together. In one instance involving scientific assertions, the accuracy jumped by nearly 26 percentage points. The study also revealed an interesting nuance: while the most powerful commercial models were already very good at breaking down sentences, the new framework was essential for smaller, local models to reach a similar level of performance. By using a conditional approach that only applies repairs when necessary, the system preserves the high quality of advanced models while lifting the performance of smaller ones.
The researchers also released three new sets of test data to help the community evaluate these systems in the future. These datasets cover different styles of writing, from the casual language of social media to the formal structure of news reports, ensuring that the tools work across the diverse landscape of human communication. While the work does not claim to have solved every problem in automated fact-checking, it provides a solid, mathematically grounded foundation for the next generation of truth-verifying tools. By fixing how we measure success and how we correct errors, the CREDENCE framework offers a clearer path toward machines that can reliably separate truth from falsehood in our complex information world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.