Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study

In the rapidly evolving world of artificial intelligence, computers are learning to see and speak at the same time. These systems, known as vision-language models, can watch a video and answer questions about what is happening on the screen. They are becoming increasingly common, helping to summarize news clips, assist in medical imaging, or simply describe a family vacation. However, a persistent problem plagues these systems: they often make things up with absolute confidence. When a computer describes a scene, it might invent a detail that never happened, or worse, it might confidently point to a specific moment in the video to prove a claim that is simply false. This is a deceptive form of error. The system does not just hallucinate; it provides a timestamp, a specific second in the video, which tricks the human user into believing the machine has checked its facts. The specificity of that time stamp feels like evidence, but without a real check, it is merely decoration.

Researchers have long known that asking a computer to check its own work is difficult. If you ask a model, "Is this picture true?" it often just agrees with you to be helpful, a behavior known as sycophancy. It wants to please the user rather than tell the truth. To solve this, a researcher developed a new system called GROUNDEDVQA. They built a pipeline that does not just generate an answer and move on. Instead, after the computer drafts an answer with specific time stamps, the system pauses. It goes back to the video, pulls out the exact frame mentioned, and asks a different, independent question: does this image actually support the statement? The goal was to see if this self-checking loop could catch the lies before they reached the user, and to figure out exactly how to build a checker that works.

The researcher tested their system on a short animated film, sampling frames every five seconds to create a searchable library of images. When a user asked a question, the system would retrieve the most relevant frames, draft an answer, and then begin the verification process. The researcher tried three different ways to build this checker, and the results of their experiments revealed a surprising truth about how to make machines tell the truth. Their first attempt was the most obvious one: they showed the vision model the image and the claim, and simply asked, "Does this image support this claim?" The system failed completely. Out of forty false claims it was tested on, it flagged zero of them. Even when the claim was about a blue car in a video that contained no car at all, the model insisted the claim was true. It was simply agreeing with the prompt, unable to separate its desire to be helpful from the visual reality.

The researcher then tried a second approach, which seemed more logical. They separated the seeing from the judging. First, the vision model described the image without knowing what the claim was, creating a "blind" caption. Then, a large language model, the same kind used to write the answers, was asked to read that description and decide if it matched the claim. This fixed the problem of the model just agreeing with the user, but it created a new one: the checker became wildly unstable. Depending on tiny, unimportant changes in how the question was phrased, the system would either flag every single claim as false, even true ones, or it would let every single claim pass, even the lies. It swung between two extremes, unable to find a middle ground. The large language model, despite its ability to write fluent text, proved to be a terrible judge of fact when asked to make a simple yes-or-no decision.

The solution came from a third design, which replaced the large language model judge with a much smaller, specialized tool trained specifically for logical reasoning. This tool, known as a natural language inference model, is designed to determine if one sentence logically follows from another. The system used the blind description of the image as the premise and the user's claim as the hypothesis. If the description did not support the claim, the system marked it as unverified. This approach worked with remarkable stability. When tested on forty claims, including fourteen that were based on false premises, this specialized checker caught seventy-nine percent of the fabricated claims. It correctly identified that a statement about fighting underwater was not supported by a frame showing a squirrel on a tree branch. Crucially, it did not flag any of the true claims. It left the factual statements alone, only catching the lies.

The study also highlighted the limits of this technology. The system is excellent at checking if a specific claim is supported by a specific frame, but it cannot fix errors in the initial search. If the system retrieves the wrong frame to begin with, the checker will still verify that the claim is true for that wrong frame. For example, if a user asks what happens at the very beginning of the video, the system might pull a frame from the middle. If the claim describes that middle frame accurately, the checker will say it is grounded, even though it is the wrong answer to the question. The researcher found that their system caught the lies, but it could not fix the retrieval errors. This distinction is vital: the verification step ensures the machine is honest about what it sees, but it does not guarantee the machine is looking at the right thing.

Ultimately, the paper demonstrates that building a reliable fact-checker for artificial intelligence is less about making the model smarter and more about choosing the right tool for the job. A general-purpose model that can write poetry or summarize news is not the same as a specialized model trained to detect logical inconsistencies. By decoupling the act of seeing from the act of judging, and by using a small, purpose-built classifier instead of a large, chatty one, the researcher created a system that is stable and trustworthy. They showed that the most obvious way to check for hallucinations fails, and that the most sophisticated general model is not the best judge. Instead, a simple, specialized tool that asks whether a description entails a claim is what actually works. The result is a pipeline that can catch the vast majority of fabricated claims while leaving the true ones untouched, offering a practical path toward more honest video understanding.