💻 computer science

EgoCS-400K: An Egocentric Gameplay Dataset for World Models

The paper introduces EgoCS-400K, a large-scale dataset comprising over 400,000 first-person Counter-Strike gameplay videos with temporally aligned actions, states, and events, designed to bridge the gap between passive web videos and controllable simulations for training interactive world models.

Original authors: Rongjin Guo, Dong Liang, Yuhao Liu, Fang Liu, Tianyu Huang, Gerhard P. Hancke, Rynson W. H. Lau

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rongjin Guo, Dong Liang, Yuhao Liu, Fang Liu, Tianyu Huang, Gerhard P. Hancke, Rynson W. H. Lau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to play a video game, or even better, how to understand how our own actions change the world around us. To do this, you need a massive library of training data. But here's the problem: most videos on the internet are like watching a movie. You see what happens, but you don't know why it happened or what the person was doing with their hands to make it happen.

The paper introduces EgoCS-400K, a massive new dataset designed to fix this. Think of it as a "super-replay" system for the video game Counter-Strike.

Here is the breakdown of what they built, using simple analogies:

1. The Problem: The "Blind Watcher"

Most video datasets are like a blind watcher. They can see a car crash on a video, but they don't have the driver's logbook. They don't know if the driver hit the brakes, turned the wheel, or if the engine failed.

Web videos: Great for seeing what things look like, but they lack the "control log" (the specific buttons pressed).
Robot data: Has the control log, but it's expensive to record and usually only shows small, simple tasks (like picking up a cup).
Simulators: Often lack real human behavior.

2. The Solution: The "Magic Replay"

The researchers found a perfect source of data: Counter-Strike game demos.
Think of a standard video recording as a photograph of a moment. A game "demo" file is different; it's like a recipe or a blueprint. It doesn't just show the picture; it records every single instruction the player gave the computer (e.g., "Move forward," "Turn left," "Shoot," "Throw grenade") at every split second.

Because they have the recipe, they can:

Replay the game to generate a clean, high-quality video from the player's eyes (First-Person view).
Read the recipe to know exactly what buttons were pressed, where the player was looking, and what the game state was at that exact moment.

3. What's Inside the Dataset?

The dataset is huge. It contains over 400,000 video clips (totaling 10,000 hours) from over 1,000 professional matches.

The "Ego" part: Every video is from the perspective of one specific player (like wearing a GoPro on their head).
The "CS" part: It covers 13 different maps in the game Counter-Strike.
The "400K" part: They rendered 10 different player viewpoints for every single round of the game, creating a massive library of perspectives.

4. The Secret Sauce: "Protected Chains" and "Smart Cutting"

One of the hardest parts of making this dataset was cutting the long game footage into useful chunks.

The Problem: If you just cut a video every 5 seconds, you might cut a grenade throw in half. The video would show the player holding the grenade, then suddenly the grenade is gone. That confuses the AI.
The Fix: The team built a smart "scissors" system. They identified "Protected Chains"—moments where an action is happening (like reloading a gun or throwing a grenade). The system is programmed never to cut the video in the middle of these chains.
The Result: The dataset is sliced into perfect, logical chunks where an action starts and finishes naturally, ensuring the AI learns the full story of an event.

5. The "Smart Narrator" (AI Captioning)

Usually, people write descriptions for videos. Here, they used an AI (a Vision-Language Model) to write the descriptions, but with a twist.

The Twist: The AI wasn't allowed to just guess. It was given the "recipe" (the button presses and game state) as a strict checklist.
How it works: Imagine a detective who has a transcript of a conversation. The AI looks at the video and the transcript. If the transcript says "Player pressed 'Reload'," the AI must mention reloading in the description. If the transcript says "Player turned left," the AI must mention the camera turning.
Why this matters: This prevents the AI from making things up (hallucinating). The descriptions are perfectly aligned with the actual actions, creating a tight link between what you see, what you do, and what you say.

Summary

EgoCS-400K is a giant library of video game replays where every video clip is perfectly synchronized with a detailed log of the player's actions, camera movements, and game events.

It bridges the gap between:

Passive videos (just watching).
Real-world robots (hard to collect data).

It offers a "middle ground" where researchers can train AI to understand how actions cause changes in the world, using the game as a safe, scalable, and perfectly recorded playground.

Technical Summary: EgoCS-400K

1. Problem Statement

The transition from passive video generation to interactive world modeling necessitates a fundamental shift in training data. While current large-scale video models rely on captioned web videos, world models require temporally aligned video-action-language trajectories. These trajectories must be grounded in the specific actions, camera motions, internal states, and events that drive future scene changes.

Existing data sources fail to meet these requirements at scale:

Web Video Datasets: Offer broad visual coverage but lack executable actions and reliable state supervision; language is weakly aligned with the causal actions.
Robotics Datasets: Provide precise action and state supervision but are costly to collect, limited in scene diversity, and often lack complex interaction trajectories.
Existing Simulators: Often lack large-scale, human-driven interaction trajectories.

The core challenge is obtaining a dataset that is human-driven, first-person, visually rich, temporally precise, and replay-grounded to bridge the gap between passive observation and controllable world simulation.

2. Methodology

The authors introduce EgoCS-400K, a large-scale egocentric dataset derived from professional Counter-Strike (CS:GO and CS2) match demos. The construction pipeline (Fig. 2) consists of three primary phases:

Phase 1: Data Collection and Rendering

Source: Public professional match demos from HLTV serve as the authoritative source of truth, containing round boundaries, player identities, positions, view angles, inputs, and game events.
Rendering: Using CS Demo Manager and the game client, the pipeline renders clean first-person videos from specific player viewpoints. Videos are rendered at 32 FPS to align with the game's 64-tick-per-second logic, ensuring deterministic alignment between frames and game ticks.
Filtering: Invalid recordings (e.g., spectator views, screen recording artifacts, or mismatched metadata) are discarded to ensure high-fidelity visual observations.

Phase 2: Parsing and Segmentation

Tick-Level Parsing: A custom parser (DemoParser2) extracts per-tick data (ticks.csv), including player position, pitch/yaw, velocity, movement buttons, weapon states, and ammo.
Action Reconstruction:
- Keyboard/Mouse: Discrete key states are decoded from button bitmasks. Mouse motion is reconstructed as view-angle deltas ( $\Delta\theta, \Delta\psi$ ) rather than raw hardware counts.
- Atomic Actions: Rule-based detectors map raw signals to temporally bounded atomic actions (e.g., firing, reloading, inspecting, grenade throwing) stored in events.csv.
Protected Chains & Segmentation:
- Protected Intervals: Actions that would cause semantic discontinuity if cut (e.g., weapon draws, grenade flights) are marked as "protected."
- Dynamic Programming (DP): The pipeline uses a DP algorithm to segment player-view sequences into training clips. The algorithm selects boundaries that minimize a cost function (penalizing gaps, context shortfalls, and deviation from target duration) while strictly avoiding cuts within protected intervals. This ensures segments preserve action integrity.

Phase 3: Prior-Guided VLM Captioning

To generate structured language supervision, the authors employ a Prior-Guided Vision-Language Model (VLM) approach:

Prior Construction: Global annotations are clipped to the target segment window.
Prior Filtering: To prevent hallucination and spurious grounding, priors are filtered:
- Action Priors: Retain only events with visible temporal changes (e.g., removing "fire" inputs during weapon-switch animations).
- Movement/Camera Priors: Aggregate displacement and view-angle changes, discarding low-evidence jitter.
Generation: The VLM receives the video, filtered priors, and a deterministic temporal skeleton. It outputs a structured JSON containing a scene_draft (visual details, environment progression) and a long_prompt (coherent caption), ensuring the language is temporally grounded in the action traces.

3. Key Contributions

EgoCS-400K Dataset: A large-scale dataset containing >400,000 first-person videos (>10,000 hours) from >1,000 professional matches, covering 13 maps and 40,000+ rounds. It provides 10 synchronized player viewpoints per round.
Multi-Grained Annotation Pipeline: A hierarchical organization ranging from per-tick state traces to protected atomic actions, action chains, and DP-planned training segments. All annotations are auditable back to the original replay file.
Replay-Grounded Supervision: Unlike manual labeling, the dataset derives actions, states, and camera motions directly from game engine logs, ensuring precise temporal alignment between visual observations and control signals.
Structured Captioning: A novel method for generating captions constrained by replay-derived facts, preventing the VLM from hallucinating unsupported events while capturing fine-grained visual details.

4. Results and Dataset Statistics

The paper presents the dataset as a completed resource rather than a benchmark with specific model performance metrics. Key statistics include:

Scale: >1,000 matches, >40,000 rounds, >400,000 video clips.
Duration: >10,000 hours of gameplay; average clip length ~90 seconds.
Granularity: 6 types of annotations per segment, including per-tick states, keyboard/mouse signals, action timelines, and multi-grained captions.
Coverage: 13 distinct maps and 10 player viewpoints per round.

The paper provides qualitative examples (Fig. 3) demonstrating the alignment of visual frames with synchronized input traces, action timelines, and generated prompts, showing how the dataset captures complex sequences like grenade preparation, flight, and weapon switching.

5. Significance and Claims

The authors position EgoCS-400K as a practical bridge between three existing data paradigms:

Passive Web Videos: Which lack action/state grounding.
Controllable Game Simulators: Which often lack human-driven diversity.
Costly Real-World Embodied Data: Which is difficult to scale.

Significance Claims:

Intermediate Testbed: The dataset serves as a scalable testbed for learning action-conditioned dynamics, embodied navigation, and active visual perception.
Transferable Priors: While the environment is simulated, it captures transferable egocentric priors such as navigation, visual search, camera control, partial observability, and multi-agent interaction.
Auditability: The "replay-grounded" nature allows for rigorous auditing of the relationship between an action and the resulting visual change, a feature missing in standard video-text datasets.

Limitations Acknowledged:
The authors explicitly state that EgoCS-400K is an intermediate testbed, not a direct model of real-world embodiment.

Domain Gaps: It lacks continuous physical interaction, tactile feedback, deformable objects, and non-combat everyday behaviors.
Action Space: Controls are discrete (keyboard/mouse) and governed by game physics rather than real-world dynamics.
Annotation Quality: Captions are VLM-generated with confidence flags, not manual gold labels, and may contain visual-detail errors.

In conclusion, EgoCS-400K provides the necessary infrastructure to train and evaluate world models that understand how human actions drive scene evolution, addressing a critical data gap in the shift toward interactive visual modeling.