Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
This paper introduces Video-IFBench, a comprehensive benchmark featuring 1.5K samples and a diverse instruction taxonomy to evaluate the under-explored capability of Multimodal Large Language Models to follow complex, user-specified constraints in video understanding scenarios, revealing that current models struggle significantly with multi-constraint and conditional instructions.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, a new generation of systems has emerged that can see, hear, and speak. These multimodal models are trained on vast amounts of text, images, and video, allowing them to describe a scene, answer questions about a movie, or summarize a news clip. For years, researchers have measured their success by asking a simple question: is the answer correct? If a model describes a video of a cooking show and gets the ingredients right, it receives a high score. However, in the real world, a correct answer is often not enough. A user might ask a model to describe a video but insist on specific rules: "Only tell me what happens after the water boils," "List the steps in order," "Do not mention the color of the pots," or "Speak only in Spanish." These requests require the system to not only understand the content but also to follow a complex set of instructions, filtering out information it knows is true but was told to ignore. Until now, the ability of these advanced systems to follow such detailed, human-like constraints in video scenarios has remained largely untested.
A team of researchers has addressed this gap by creating a new testing ground called Video-IFBench. This benchmark is designed specifically to see if video-understanding artificial intelligence can actually do what it is told when the instructions are tricky. The researchers gathered over 700 videos from public sources, covering ten different domains such as sports, science, news, and daily life. These clips range from ten seconds to ten minutes in length, totaling about 49 hours of footage. From this collection, they constructed 1,500 unique test cases. Each test case pairs a video with a specific request that includes a mix of tasks and constraints. Some requests ask for a single piece of information, like "What is the text on the screen?" while others demand multiple steps, such as "List the actions, but only those lasting longer than two seconds, and present them as a numbered list." The most difficult tests involve conditional logic, where the model must first watch the video to decide which path to take. For example, a prompt might say, "If the video starts with a title about fixing a laptop, then describe the repair steps; otherwise, describe the person's actions in the last ten seconds." The model must correctly identify the starting condition to choose the right set of instructions to follow.
To ensure these tests were fair and accurate, the researchers built a semi-automated system to generate the questions and the rules for checking the answers. They used artificial intelligence to extract facts from the videos, such as what objects were present and when specific events occurred, and then used those facts to write complex instructions. Crucially, every instruction came with a checklist. Some items on the checklist could be checked by a computer program, such as verifying if the answer was written in JSON format or if it contained exactly five bullet points. Other items, which required understanding the meaning of the response, were checked by a separate, highly capable language model acting as a judge. This hybrid approach allowed the researchers to evaluate whether the models satisfied every single constraint in a request, not just the main task.
The results of this evaluation revealed a significant weakness in even the most advanced video models available today. When tested on the entire benchmark, the best-performing model achieved an overall score of just 54.5 percent. This means that in nearly half of the cases, the model failed to follow the full set of instructions, even if it understood the video content correctly. The study found that models struggled most when the instructions contained many constraints at once, or when the request required the model to make a decision based on the video content before answering. For instance, when asked to choose between different paths based on what happened in the video, models frequently picked the wrong path or ignored the condition entirely. The difficulty increased as the instructions became more complex; when a request involved a deep chain of "if-then" logic, the success rate dropped sharply.
The researchers also discovered that the type of constraint mattered greatly. Models were relatively good at following format rules, such as using a specific language or limiting the word count. However, they performed poorly on semantic constraints, which required them to understand the meaning of the video to filter information. For example, a model might be asked to describe only the actions of a specific person while ignoring everyone else, or to list events that happened within a precise time window. In these cases, the models often included information they were told to exclude or missed details they were supposed to find. The study showed that simply making the models larger or giving them more computing power did not solve the problem. While larger models generally performed better, they still failed to consistently follow strict instructions, suggesting that the ability to understand video content and the ability to follow complex rules are two different skills that current technology has not yet fully combined.
This work suggests that for artificial intelligence to be truly useful in real-world applications, it must learn to listen as carefully as it sees. The findings indicate that current systems are not yet reliable enough to be deployed in scenarios where users expect precise adherence to their specific needs. The researchers hope that by providing a clear measure of this capability, Video-IFBench will guide future development toward models that can not only see the world but also follow the nuanced, often contradictory, instructions of the people who use them. The path forward requires a shift in focus from simply getting the right answer to getting the right answer in the exact way the user asked for it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.