💻 computer science

Using Gameplay Videos for Detecting Issues in Video Games

This paper proposes and empirically evaluates GELID, an approach for automatically detecting and categorizing game issues from gameplay videos, which achieves satisfactory results in segmenting and clustering specific issues but shows limitations in categorizing issue types and clustering by game context.

Original authors: Emanuela Guglielmi, Simone Scalabrino, Gabriele Bavota, Rocco Oliveto

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emanuela Guglielmi, Simone Scalabrino, Gabriele Bavota, Rocco Oliveto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine video games as massive, intricate theme parks. Developers spend years building rides and attractions, but no matter how hard they try, some rides might have a loose bolt, a broken track, or a glitchy animatronic. Usually, the only way to find these problems is to wait for park visitors (players) to complain.

But today, there's a new group of visitors: Streamers. These are people who play games live on the internet, talking to their audience as they go. They act like tour guides, but instead of just showing the scenery, they often scream, "Whoa, the floor just disappeared!" or "My character froze!" while playing.

The problem is that there are so many of these tour guides, and they are recording thousands of hours of footage. It's impossible for the theme park managers (game developers) to watch every single video to find the broken rides.

Enter GELID.

The authors of this paper created a robot detective named GELID (GamEpLay Issue Detector). Think of GELID as a super-efficient editor and librarian for these thousands of hours of video. Its job is to watch the streamer's footage, find the exact moments where something went wrong, and organize those clips into a neat filing system so developers can fix the problems quickly.

Here is how GELID works, broken down into four simple steps:

1. The Scissors (Video Segmentation)

Imagine a streamer recording a 2-hour video. GELID doesn't want to watch the whole thing. It needs to cut the video into small, bite-sized pieces.

How it works: It listens to the streamer's voice (the subtitles). When the streamer starts talking about a problem, GELID cuts the video a few seconds before they start and a few seconds after they stop.
The Analogy: It's like a film editor who only keeps the scenes where the actor says something important, throwing away the boring parts where they are just walking around silently.

2. The Sorter (Segment Categorization)

Now GELID has hundreds of small clips. It needs to figure out what kind of problem is in each clip.

How it works: It uses a computer brain (Machine Learning) to read the streamer's words and look at the video frames. It tries to sort the clips into four folders:
- Logic: The game rules are broken (e.g., a door opens when it shouldn't).
- Presentation: The graphics or sound are weird (e.g., a character's face is stretched).
- Performance: The game is lagging or crashing (e.g., "My frame rate dropped!").
- Balance: The game is too hard or too easy.
The Result: GELID tries to throw away the "boring" clips where the streamer is just chatting about lunch.
The Catch: This step was the hardest for the robot. It was often confused. It sometimes thought a comment about a "funny looking character" was a "broken game rule." It struggled to tell the difference between a visual glitch and a logic error.

3. The Map Maker (Context Clustering)

Once GELID finds the bad clips, it needs to know where in the game they happened.

How it works: It looks at the pictures in the video. If two clips look like they are in the same forest or the same castle room, it puts them in the same pile.
The Analogy: Imagine you have 50 photos of broken rides. GELID tries to group them by saying, "These 10 photos are all from the 'Haunted House' area, and these 5 are from the 'Space Station'."
The Catch: This was also tricky. Some games have areas that look almost identical (like two different forests that look the same). GELID got confused and mixed up the locations, grouping a broken ride in Forest A with a broken ride in Forest B.

4. The Grouping (Issue Clustering)

Finally, GELID looks at the clips it has grouped by location and asks: "Are these all the same specific problem?"

How it works: If 20 different streamers all showed a clip where a sword disappeared, GELID groups them together.
The Result: This step worked very well! The robot was excellent at realizing, "Hey, all these people are showing the exact same bug." It successfully organized the chaos into specific, actionable groups.

The Verdict: Did the Robot Succeed?

The paper reports a mixed bag of results, like a student who aced the math test but failed the art class:

The Good News: The robot is great at cutting the videos into useful pieces and grouping identical bugs together. It successfully organized the "what" and the "where" of specific bugs.
The Bad News: The robot struggles to understand the specific type of problem. It has a hard time telling if a streamer is talking about a visual glitch, a game rule error, or a performance lag. It also gets confused when trying to figure out exactly which part of the game the video was filmed in, especially if the game has many similar-looking areas.

In summary: GELID is a powerful tool that can sift through a mountain of video footage to find the "needles" (bugs) and stack them up. However, it still needs a human to double-check its work because it sometimes mislabels the type of needle or gets lost in the haystack. The authors plan to teach the robot better ways to listen and look in the future, but for now, it's a helpful assistant that isn't quite ready to work alone.

Technical Summary: Using Gameplay Videos for Detecting Issues in Video Games

Problem Statement
The video game industry faces significant challenges in quality assurance due to the complexity of game states, non-deterministic behaviors, and the difficulty of writing automated tests. Consequently, many bugs, glitches, and performance issues are only discovered after release when players encounter them. While streamers on platforms like Twitch and YouTube frequently encounter these issues during live gameplay and comment on them, they rarely report them formally to developers. Although large amounts of gameplay videos exist as a potential "goldmine" for developers, existing automated approaches (e.g., Lin et al., 2019) operate at the video level, relying on metadata to identify bug-reporting videos. These methods fail to pinpoint specific segments within long videos where the issue occurs, nor do they categorize the type of issue or the context in which it appeared, making them impractical for direct developer use.

Methodology: The GELID Approach
The authors propose GELID (GamEpLay Issue Detector), an automated approach designed to extract, categorize, and cluster relevant information from gameplay videos. GELID processes input videos through four sequential steps:

Video Segmentation: The system partitions videos into meaningful segments based on spoken content rather than visual shot transitions. It utilizes subtitle entries (text shown on screen) as anchors. To account for streamer reaction time, the system extracts video segments starting $t$ seconds before a subtitle begins and ending $t$ seconds after it ends. A keyword-matching filter (using a dictionary of 257 issue-related terms derived from game change notes and existing datasets) is applied to discard segments unlikely to contain useful information.
Segment Categorization: This step classifies segments into five categories: Logic, Presentation, Balance, Performance, and Non-informative. The authors employ a machine learning pipeline combining:
- Textual Features: Bag of Words (BoW), word2vec, and doc2vec derived from subtitles.
- Video-based Features: Structural Similarity (SSIM) and HSV histogram correlations between consecutive frames to detect visual anomalies (e.g., crashes or glitches).
- Algorithms: The study evaluates Random Forest, Logistic Regression, SMO, Multilayer Perceptron, and IBk.
Context-based Segment Grouping: Informative segments are grouped by the game area or level in which the issue occurred. This utilizes key frame extraction and pixel-by-pixel averaging to create summary frames. Clustering is performed using non-parametric algorithms (DBSCAN, OPTICS) with distance metrics based on SSIM and HSV histogram correlation.
Issue-based Segment Clustering: Segments reporting the same specific issue (e.g., a specific crash trigger) are clustered together to reduce developer analysis effort. This step combines textual and image-based features and employs DBSCAN, OPTICS, and Mean Shift algorithms.

Empirical Study and Results
The authors conducted an empirical study using 170 manually annotated videos for training (yielding 1,255 segments) and 80 gameplay videos from three specific games (Conan Exiles, DayZ, New World) for validation (yielding 604 segments).

Segmentation (RQ1): The approach achieved satisfactory results. A reaction time parameter ( $t$ ) of 0 seconds favored atomicity (segments could not be split further), while $t=5$ seconds favored interpretability. The best balance was found at $t=5$ .
Categorization (RQ2): The results were unsatisfactory. While the binary classification (informative vs. non-informative) achieved an AUC of 0.61 on the test set, the multi-class classification performed poorly, particularly for Performance and Balance issues, which the model failed to detect (0% recall). The authors attribute this to the noise in streamer commentary (incomplete sentences, generic exclamations) and the lack of sync between spoken words and visual events.
Context Clustering (RQ3): Results were mixed. The MoJoFM score (measuring similarity to manual clustering) ranged from 21.9% to 46.0% depending on the game. Performance was heavily dependent on the visual distinctiveness of game areas; games with visually similar levels (e.g., Conan Exiles) proved difficult to cluster accurately.
Issue-based Clustering (RQ4): Results were satisfactory. DBSCAN achieved an average MoJoFM of 72.7%, indicating that the system could effectively group segments reporting the same specific issue, likely because the task involved fewer, more distinct instances compared to context clustering.

Key Contributions and Significance
The paper's primary contribution is the definition and evaluation of GELID, a pipeline that moves beyond video-level analysis to segment-level detection and hierarchical organization of game issues. The authors publicly release a dataset of 604 manually segmented, labeled, and clustered video segments, along with the implementation of the approach.

The significance of the work lies in its identification of the specific bottlenecks in automated gameplay analysis. The authors modestly conclude that while extracting and clustering specific issues is feasible, automated categorization and context-based clustering currently lack the precision required for practical deployment. They highlight that streamer commentary is often too noisy and unsynchronized with visual events for current NLP and computer vision techniques to reliably distinguish issue types or game contexts without game-specific adaptations. The paper serves as a baseline for future research, suggesting that improving these steps may require larger contextual windows for text analysis, game-specific visual features, or specialized approaches for detecting visual anomalies independent of speech.

1. The Scissors (Video Segmentation)

2. The Sorter (Segment Categorization)

3. The Map Maker (Context Clustering)

4. The Grouping (Issue Clustering)

The Verdict: Did the Robot Succeed?

More like this