ViBR: Automated Bug Replay from Video-based Reports using Vision-Language Models
This paper introduces ViBR, a lightweight and fully automated framework that leverages CLIP-based action segmentation and Vision-Language Models to successfully reproduce software bugs directly from video-based reports without requiring explicit instrumentation or pre-constructed UI graphs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a software developer, and a user sends you a bug report. Instead of writing a long, confusing email saying, "I clicked here, then that, and then the app crashed," they send you a video of their screen.
This video is great because it shows exactly what happened. But here's the problem: You can't just watch the video and instantly make the bug happen on your own phone.
Why? Because your phone might be different. Maybe your screen is bigger, the app looks slightly different, or the buttons are in different places. Trying to manually re-watch the video and click the same spots on your own device is like trying to follow a treasure map drawn for a different island. It's slow, frustrating, and often leads to mistakes.
Enter ViBR (Video-based Bug Replay). Think of ViBR as a super-smart, robotic assistant that watches the user's video and automatically tries to recreate the bug on your phone, even if your phone looks different.
Here is how ViBR works, broken down into three simple steps using everyday analogies:
1. The "Scene Cutter" (Action Boundary Segmentation)
The Problem: A video is just a stream of hundreds of frames. The robot needs to know: "When did the user actually do something?"
The Analogy: Imagine watching a movie and trying to find the exact moment the hero punches the villain. The movie has thousands of frames where nothing happens, then a split-second action, then more nothing.
How ViBR does it: ViBR uses a special "eye" (called CLIP) that looks at two consecutive frames of the video. It asks, "Do these two pictures look basically the same, or did something big happen?"
- If the screen is just static, it ignores it.
- If the screen suddenly changes (like a new page loading after a click), it says, "Aha! That's an action!" and cuts the video into separate "scenes." It also figures out if the user tapped, scrolled, or typed.
2. The "Spotter" (GUI State Comparison)
The Problem: Now the robot knows the user clicked a "Submit" button in the video. But on your phone, the "Submit" button might be blue instead of green, or it might be at the bottom instead of the top. If the robot just looks for a green button, it will fail.
The Analogy: Imagine you are looking for a specific friend in a crowded room. In the photo (the video), your friend is wearing a red hat. In the real room (your phone), they are wearing a blue hat. A simple robot would say, "No red hat, I give up."
How ViBR does it: ViBR uses a Vision-Language Model (VLM)—basically an AI that can "see" and "read" at the same time.
- It doesn't just look for a "red hat." It understands the concept of the button.
- It looks at the whole screen and asks: "Where is the thing that acts like the 'Submit' button in this video?"
- It uses a technique called GroundingDINO to highlight potential buttons (like drawing a box around them) and then asks the AI: "Which of these boxes is the one the user clicked?"
- It checks if the current screen is "functionally the same" as the video, even if the colors or layout are slightly different.
3. The "Mime Artist" (Bug Replay on Device)
The Problem: Once the robot finds the right button, it needs to click it. But what if the app is stuck on a loading screen, or the button is hidden?
The Analogy: Imagine you are trying to teach a robot to play a game. You tell it, "Click the button." But the button is covered by a pop-up ad. The robot needs to figure out, "Oh, I need to close the ad first before I can click the button."
How ViBR does it:
- If the screen looks exactly like the video, the robot just clicks the button.
- If the screen looks different (e.g., a pop-up appeared), the robot pauses and asks the AI: "Hey, the screen doesn't match the video. What should I do to get it back to the right state?"
- The AI acts like a detective, looking at the current screen and the video, and says, "Okay, close the pop-up, then scroll down, then click the button."
- It repeats this process until the bug is reproduced.
Why is this a big deal?
- No "Touch Indicators" Needed: Previous tools required users to turn on a special "show my finger" setting when recording. Most people don't do that. ViBR works with any normal screen recording.
- No "Map" Needed: Other tools tried to build a giant map of every possible screen in the app beforehand. That takes forever and breaks easily. ViBR just looks at the video and the current screen in real-time.
- It's Cheap and Fast: The whole process costs only a few cents in computer power and takes a few minutes.
The Result
The researchers tested ViBR on real-world bug reports. It successfully recreated 72% of the bugs automatically. That's a huge jump compared to older methods, which struggled with anything that wasn't a perfect match.
In short: ViBR is like a translator that turns a user's messy video into a perfect set of instructions for a developer's phone, bridging the gap between "It broke on my screen" and "I can see it break on mine."
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.