WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation
This paper introduces WorldCam, a novel framework that utilizes camera pose as a unifying geometric representation to enhance interactive 3D gaming world models by enabling precise action control through Lie algebra-based camera embedding and ensuring long-horizon 3D consistency via global pose-guided retrieval, all supported by a new large-scale annotated gameplay dataset.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are playing a video game where you can type commands like "move forward" or "turn left," and the game instantly generates a brand-new, photorealistic world in response. That's the dream of an Interactive Gaming World Model.
However, current AI models are a bit like a clumsy tour guide. If you tell them to "turn left," they might spin the camera wildly, lose track of where they've been, or suddenly change the texture of the walls from brick to grass. They struggle to keep the world consistent over time or to move exactly how you want.
Enter WorldCam, a new AI system that acts like a master architect and a GPS navigator combined. Here is how it works, broken down into simple concepts:
1. The Problem: The "Abstract Signal" Mistake
Previous AI models treated your keyboard and mouse clicks like abstract magic words. They'd say, "Okay, the user pressed 'W', so I'll just make the video look like it's moving forward."
- The Flaw: This is like telling a driver, "Go north," without giving them a map. The driver might drift, turn the wrong way, or forget where they started. Because the AI didn't understand the geometry (the math of movement), the 3D world would eventually fall apart. If you walked in a circle, the AI might not realize you were back at the start, so the building you saw earlier would look completely different when you returned.
2. The Solution: The "GPS Anchor" (Camera Pose)
WorldCam changes the game by treating your actions not as magic words, but as precise GPS coordinates.
- The Lie Algebra Analogy: Imagine you are walking through a maze. If you walk forward and turn right at the same time, your path is a curve (a screw motion). Old AI models tried to calculate this by adding "forward" and "turn" separately, which leads to a jagged, wrong path.
- WorldCam uses a mathematical concept called Lie Algebra. Think of this as a perfectly calibrated compass and odometer. It calculates your exact position in 3D space (6 degrees of freedom) instantly. It knows exactly where you are relative to where you started, down to the millimeter.
3. The Magic Trick: The "Memory Book"
This is the most creative part. Because WorldCam knows your exact GPS position at every second, it can use that position as an index in a giant Memory Book.
- How it works:
- You walk forward, turn left, and explore a new room. The AI writes down your GPS coordinates and saves a "snapshot" of that room in its memory book.
- You decide to walk back. As you approach the spot where you saw that room earlier, the AI checks your current GPS coordinates against its Memory Book.
- The "Aha!" Moment: The AI realizes, "Oh! You are back at the exact same spot where we were 5 minutes ago!" It instantly pulls up the old snapshot from the Memory Book and pastes it into the new video.
- The Result: The room looks exactly the same as before. The walls, the lighting, and the objects are perfectly consistent. No more "glitchy" worlds that change when you look away and look back.
4. The "Attention Sink": The Anchor
When generating a long video, AI tends to get "drunk" on its own mistakes, causing the image to blur or warp over time (like a story that gets worse the longer you tell it).
- The Fix: WorldCam keeps a tiny "anchor" of the very first few frames of the video in its mind at all times. Think of this as a lighthouse. Even as the video gets long and complex, the AI constantly glances at the lighthouse to make sure it hasn't drifted off course, keeping the visual quality sharp and stable.
5. The Dataset: The "Training Gym"
To teach this AI, the researchers didn't just use simple blocky games (like Minecraft). They recorded 50 hours of real human gameplay from complex, realistic games (like Counter-Strike and Xonotic).
- They captured real humans running, jumping, spinning 360 degrees, and revisiting old spots.
- They labeled every single frame with the exact camera position and a text description of the scene. This gave the AI a massive library of "real" movement to learn from.
Summary: Why is this a big deal?
Think of previous AI video generators as improvisational actors. They make up the scene as they go, which is fun but often inconsistent.
WorldCam is like a scripted movie director with a perfect memory.
- Precise Control: If you say "turn right," it turns right exactly, not a little bit left.
- Long-Term Consistency: If you walk in a circle, the world closes the loop perfectly. The building you saw at the start is the same building when you return.
- Realism: It creates a world that feels like a real, physical place, not just a dream that fades away.
In short, WorldCam bridges the gap between "watching a cool video" and "playing in a real, consistent 3D world" that you can explore forever without the world falling apart.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.