SourceMinds at CheckThat! 2026: NLI-Grounded Citation Auditing in a Multi-Agent Pipeline for Full Fact-Checking Article Generation

The Detective's Dilemma: Why We Need Better Fact-Checking Robots

Imagine the internet as a massive, chaotic library where anyone can shout out a "fact" from the stacks. Sometimes these shouts are true, but often they are wild guesses, half-truths, or outright lies. For a long time, computers have been good at playing a simple game of "True or False" with these shouts. They act like a referee, raising a green flag for truth or a red flag for lies. But in the real world, a simple "False" isn't enough. If someone spreads a rumor, we don't just want a referee's whistle; we want a full detective story. We need to know why it's false, what the evidence actually says, and exactly which page of the newspaper proved the liar wrong. This is the challenge of "fact-checking article generation." It's not just about finding the answer; it's about writing a clear, trustworthy story that connects the dots between a claim and the proof, citing every single source so no one can say, "Where did you get that?"

This is where the paper "SourceMinds at CheckThat! 2026" comes in. The authors, a team from the University of North Texas, realized that asking a super-smart computer to write a whole fact-checking article in one giant leap often leads to messy results. The computer might make things up or forget to cite its sources. So, they built a team of digital specialists—a "multi-agent pipeline"—to tackle the job step-by-step. Instead of one robot trying to do everything at once, they created a workflow where one robot finds the clues, another organizes the plan, a third writes the draft, and a final "auditor" checks every citation to make sure the story holds up. Their goal wasn't just to write faster, but to write better, ensuring that every sentence in the final article is backed by a real, verifiable source.

The Team of Digital Detectives

The paper describes a system called SourceMinds, designed to solve a very specific problem: how to turn a messy claim, a verdict (like "False"), and a pile of evidence documents into a clean, professional fact-checking article. The authors suggest that the best way to do this is to break the process down into a relay race, where different "agents" (specialized computer programs) pass the baton to one another.

Step 1: The Evidence Hunter
First, the system has to find the right clues. Imagine you are looking for a specific quote in a library with thousands of books. If you just grab the first few books that look similar, you might miss the best one. The SourceMinds team uses a "dense retrieval" agent, which is like a librarian who understands the meaning of words, not just the spelling. It scans the evidence documents and finds the most relevant sentences. But it doesn't stop there. It uses a "reranker" to double-check the best matches and a "source-balanced selector" to make sure it doesn't just pick ten quotes from the same single website. It wants a diverse team of sources, ensuring the evidence isn't biased toward just one voice.

Step 2: The Architect (Fact Planner)
Once the clues are gathered, the system doesn't just start writing. That would be like trying to build a house without a blueprint. The "Fact Planner" agent takes the selected evidence and creates a structured JSON plan. Think of this as the architect drawing the floor plan. It decides what the main argument is, lists the supporting points, and notes any "caveats" (warnings) if the evidence doesn't perfectly match the verdict. Crucially, if the evidence is weak, the planner is programmed to admit it rather than inventing a fake reason to make the story fit.

Step 3: The Journalist (Article Writer)
With the blueprint in hand, the "Article Writer" agent steps in. This agent acts like a professional journalist. Its job is to write a 250-to-450-word article that explains the claim, states the verdict, and walks the reader through the evidence. The rule here is strict: every factual sentence must have a "citation," which looks like (source: URL). It's like a student writing a research paper where every claim must have a footnote. The writer is told to use only the URLs provided in the evidence index, preventing it from making up fake sources.

Step 4: The Editor (Self-Critique)
Before the article goes to print, it goes through a "gated self-critique." This is a smart editor that only wakes up if the draft looks suspicious. The system checks the draft: does it have at least three citations? Does it use at least two different sources? If the draft looks "weakly grounded" (meaning it's missing proof), the editor gets to work. It reads the article sentence by sentence. If a sentence makes a claim but the citation doesn't actually support it, the editor rewrites the sentence or swaps the citation for a better one. If the draft is already good, the editor skips the work to save time. This step ensures that the article doesn't just look like it has sources, but that the sources actually back up the words.

Step 5: The Auditor (NLI Citation Auditor)
Finally, the "NLI Citation Auditor" performs a rigorous, automated background check. This agent uses a technique called "Natural Language Inference" (NLI). Imagine a logic test where the computer asks, "If this evidence is true, does it necessarily mean this sentence is true?" The auditor runs three passes:

1. The Cleanup: It throws out any citations pointing to URLs that aren't in the original evidence list.
2. The Repair: If a sentence has no citation, the auditor hunts for the best matching evidence and attaches a citation only if the evidence logically proves the sentence.
3. The Prune: If a sentence has too many citations, the auditor removes the redundant ones, keeping only the essential sources.

What They Found

The team tested their system on the official CLEF 2026 CheckThat! Lab test set, which contained 1,158 claims and an average of 7.8 evidence articles per claim. They compared their multi-agent pipeline against a standard baseline system.

The results were a mix of success and a clear lesson for the future. The SourceMinds system achieved a mean score of 0.329, beating the baseline score of 0.272. The authors suggest that breaking the task into specialized stages really helped. Specifically:

• Evidence Coverage: The system scored 0.394, the highest of all metrics. This means the system was very good at finding relevant evidence and spreading citations across different sources, rather than relying on just one document.
• Citation Quality: The system improved significantly in Citation Precision (0.337) and Citation Recall (0.339) compared to the baseline (0.223 and 0.240). This suggests that the "writer" and "auditor" agents worked well together to ensure that citations were present and actually necessary.

However, the paper also highlights a significant bottleneck. The Entailment Score—which measures whether the generated article actually matches the logic and reasoning of the ground-truth reference—was the lowest score at 0.245. In fact, this was the only metric where their system scored lower than the baseline (0.298).

The authors explain this gap by noting that having a citation doesn't guarantee the story is right. A sentence can have a perfect-looking citation like (source: example.com) but still misrepresent what that website actually said. The system was good at attaching tags to sentences, but it struggled to construct the deep, logical argument that connects the evidence to the final verdict in the same way a human expert would. The paper suggests that while their "sentence-by-sentence" validation is helpful, the next big challenge is to improve the "evidence-to-claim alignment"—making sure the whole story makes sense, not just the individual parts.

The Bottom Line

The SourceMinds team concludes that their multi-agent pipeline is a promising, modular way to generate fact-checking articles. It proves that you don't need to train a massive, specialized robot from scratch; instead, you can use a lightweight team of specialized agents to get better results. The system is effective at selecting the right evidence and producing articles that are well-cited and traceable. However, the lower entailment score suggests that the "magic" of fact-checking isn't just about finding sources; it's about weaving them into a coherent, logical narrative. The authors suggest that future work should focus on better reasoning stages to bridge the gap between "having a citation" and "telling the whole truth."