OVIBench: Benchmarking Online Video Question Answering under Interruption
This paper introduces OVIBench, the first standardized benchmark and corresponding training dataset designed to evaluate and improve Vision Language Models' ability to handle realistic user interruptions—such as cancellations, false triggers, and corrections—during online video question answering.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine watching a live broadcast of a nature documentary. You are leaning in, fascinated by a turtle carrying ducklings across a pond, when you suddenly realize the narrator is describing the wrong animal. In a real conversation, you would interrupt, saying, "Wait, that's a cat, not a duck," or perhaps, "Stop, I don't need to hear the rest." For decades, the artificial intelligence systems designed to watch videos and answer questions have operated in a much more rigid way. They were built to wait until the entire video finished playing before they began to speak, delivering a single, uninterrupted monologue. This approach worked well for static tests, but it failed to capture the messy, dynamic reality of how humans actually interact with technology. We do not wait for a machine to finish its thought before we correct it; we interrupt, we steer, and we stop it mid-sentence.
A team of researchers has now built a new way to test whether these video-watching computers can handle the real world. They created a benchmark called OVIBench, which simulates the experience of a user interrupting a model while it is in the middle of generating an answer. Instead of letting the model watch a video from start to finish and then speak, the researchers set up a scenario where the model starts talking, and then, at a specific moment, a user signal cuts in. This signal might tell the model to stop talking entirely, to ignore a mistaken command, or to correct a factual error it just made. The researchers categorized these interruptions into three distinct types: a "cancellation," where the user wants the conversation to end; a "false trigger," where the user accidentally says something that looks like a command but isn't; and a "correction," where the user points out a mistake and asks for a new answer.
To test this, the team gathered thousands of videos from various sources, ranging from cooking tutorials to crime footage and nature clips. They used a sophisticated system to generate questions about these videos and then inserted interruptions at random moments during the model's response. Because real-time testing with thousands of models would be chaotic and slow, they developed a clever offline simulation. They recorded exactly how fast a model speaks, calculated which frames of the video the model would have seen by the time an interruption occurred, and then fed that specific slice of video, along with the partial answer and the interruption signal, back into the model. This allowed them to test how the model reacted to the interruption as if it were happening live, ensuring every model faced the exact same conditions.
The results revealed a significant gap between how well these models perform in quiet, controlled tests and how they handle dynamic interruptions. While many of the top video models could answer questions correctly when left alone, they struggled immensely when asked to adapt to a user's mid-sentence command. They often failed to recognize when they should stop talking or, more critically, when they should change their answer to match a correction. In many cases, the models would continue generating text that contradicted the user's new instruction, or they would get confused by a false alarm and stop talking when they should have kept going. The study showed that the ability to handle these interruptions is not just a minor feature but a fundamental weakness in current technology, particularly when it comes to following correction requests.
To address this, the researchers created a new training dataset specifically designed to teach models how to recognize and react to these interruptions. They took a standard video model and fine-tuned it using thousands of examples where the model had to choose the correct reaction to an interruption. The results of this training were striking. The newly trained model, which was relatively small in size, outperformed much larger, more powerful models that had not received this specific training. It learned to distinguish between a genuine command to stop and a false signal, and it became significantly better at revising its answer when told it was wrong. In fact, this smaller, specialized model performed better than some of the largest models available, suggesting that the right kind of training data is more important than simply making the model bigger.
The researchers also developed a detailed set of criteria to judge not just whether the model got the right answer, but how well it handled the interaction. They measured whether the model's response flowed naturally after the interruption, whether it stuck to the facts visible in the video, and whether it preserved the parts of the conversation that didn't need changing. These tests confirmed that while current models are getting better at understanding video content, they are still largely unprepared for the fluid, interruptible nature of human conversation. The work suggests that for video assistants to become truly useful in real-world settings like live streaming or online education, they must be taught not just to see and speak, but to listen and adapt while they are speaking. This new benchmark provides a clear path forward, showing that with the right data and evaluation methods, we can build systems that are as responsive to a human's voice as they are to the images on the screen.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.