AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond
The paper proposes AnyScene, a unified occupancy-centric framework that leverages a Spatial-Temporal Occupancy Diffusion Transformer and a Geometry-Grounded View Expansion module to generate highly controllable, high-fidelity, and temporally consistent multi-view driving videos from arbitrary BEV layouts without relying on reference frames.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you want to build a perfect, realistic movie of a city street, but you don't have a camera crew, no actors, and no actual cars. Instead, you just have a simple, top-down map (like a video game mini-map) where you can draw where the roads are, where the cars should be, and even change the weather with a text note.
That is essentially what AnyScene does. It is a new computer program that turns these simple, 2D top-down sketches into full, 3D, high-definition driving videos that look and feel real.
Here is how it works, broken down into three simple steps using everyday analogies:
1. The "Architect's Blueprint" (Turning Maps into 3D Space)
Most old methods tried to draw the video directly from the map, which often led to weird glitches—like a car floating in the air or a building disappearing when the camera moved.
AnyScene takes a different approach. First, it acts like a 3D Architect. It takes your 2D top-down map (the "BEV layout") and builds a "digital clay model" of the entire scene. In the paper, they call this Semantic Occupancy.
- The Analogy: Think of this like a giant, invisible grid of tiny Lego blocks filling the air. Some blocks are "road," some are "car," some are "tree," and some are "empty air."
- The Magic: The system builds this Lego model frame-by-frame. Because it builds the 3D structure first, it ensures that if a car is on the left side of the road in the map, it stays on the left side in the 3D world, no matter how the camera moves. This solves the problem of things looking "jittery" or inconsistent.
2. The "Director's Camera" (Turning the Model into a Movie)
Once the 3D Lego model is built, the system needs to turn it into a video. Old methods were like a director who was tied to a specific camera rig; they could only film from fixed angles or needed a "reference" video to copy.
AnyScene uses a new module called Geometry-Grounded View Expansion (GGVE).
- The Analogy: Imagine you have a perfect 3D sculpture of a city. AnyScene is like a director who can walk around that sculpture with a camera and film it from any angle they want, even angles that don't exist in the real world.
- The Magic: It doesn't need a reference video to copy. It looks at the 3D Lego model, calculates exactly what the light and shadows should look like from a new angle, and generates the video pixels. This means you can ask for a video from a drone, a car, or even a camera mounted on a bicycle, and it will generate it instantly without needing to be retrained.
3. The "Infinite Playground" (Why This Matters)
The paper highlights that this system is "controllable" and works "anywhere."
- The Analogy: Think of it as a Lego set that never runs out of pieces. You can draw a traffic jam in New York, then instantly change the map to a rainy street in Singapore, or even draw a completely made-up intersection that never existed. The system will generate a realistic video for all of them.
- The Result: It can create videos with 12 different camera views at once, or even more, all staying perfectly consistent with each other. If you edit the map to remove a car, the video instantly updates to show an empty street, with the shadows and reflections adjusting automatically.
In Summary
The paper claims that AnyScene is a two-step machine:
- Step 1: It reads a simple top-down drawing and builds a precise 3D "Lego world" (Occupancy).
- Step 2: It uses that 3D world to film a movie from any camera angle you want, with any weather or traffic pattern you draw.
The authors tested this on a new, high-speed dataset they created (nuCraftv2) and showed that their method creates cleaner, more consistent, and more controllable driving videos than previous systems, which often struggled to keep geometry consistent or required fixed camera setups.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.