Remote-Sensing City Layout Extraction with MLLM
This paper introduces "Code-as-City," a framework that leverages multimodal large language models to translate single top-down remote-sensing images into executable, editable city code, enabling the synchronized generation of 3D layouts and semantic projections with demonstrated performance on the CityLayout-100 dataset.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine looking down at a city from a satellite. You see a complex tapestry of streets, rooftops, parks, and rivers. For decades, computer systems have been trained to look at these images and draw simple boxes or colored shapes around what they see. They can tell you, "There is a road here" or "That is a building there." This works well for counting things or measuring how much land is covered by concrete versus grass. However, these systems produce a flat, static picture. They do not understand that a road connects two buildings, or that a park sits between them. They cannot tell you how to rebuild that city in a computer if the original image were lost, because they have not captured the rules of how the city is put together. They see the pixels, but they miss the logic.
This is the challenge researchers at the City University of Hong Kong and the Chinese Academy of Sciences set out to solve. They wanted to move beyond simple picture recognition to something more useful: turning a single satellite photo into a set of instructions that a computer can actually read and use to reconstruct the city. Their goal was to create a digital version of the city that keeps the identity of every object, remembers how they are connected, and can be edited or rebuilt later. Instead of just drawing a map, they wanted to write the code that builds the map.
To achieve this, the team developed a new approach they call "Code-as-City." They used a type of advanced artificial intelligence known as a multimodal large language model. Think of this model as a very smart assistant that can look at an image and understand it, but also write computer code. The researchers did not ask the AI to simply guess what is in the picture. Instead, they gave it a specific job: to look at the satellite photo and write a program that describes the city's layout. This program would not just be a list of names; it would be a set of steps that, when run, would draw the roads, place the buildings, and define the open spaces exactly as they appear in the photo.
The process works in a careful, step-by-step sequence. First, the system creates a rough sketch of the city's colors and shapes to serve as a guide. Then, the AI makes three separate passes to build the city. In the first pass, it identifies the roads, figuring out where they go and how wide they are. In the second pass, it looks at the land between the roads to identify parks, water, and sports fields, and it figures out how these areas relate to one another. In the final pass, it identifies the individual buildings and groups them together. At every stage, the AI refers back to the initial sketch and the original photo to make sure it is getting the details right.
Once the AI finishes writing the code, the system runs it. This execution turns the text instructions into a digital city graph, which is a structured map of all the objects and their connections. From this single source of truth, the system produces two things at once. The first is a 3D model of the city that you can rotate and view from different angles. The second is a flat, top-down view that looks just like the original satellite photo. Because both views come from the same set of instructions, they are perfectly synchronized. If you were to edit the code to move a building, both the 3D model and the flat map would update instantly to reflect that change.
The researchers tested this method on one hundred different scenes from a dataset called CityLayout-100. These scenes varied in complexity, from simple neighborhoods with few buildings to dense urban areas with many overlapping features. The results showed that the system could successfully translate the visual information into a working, editable city layout. When they compared the flat, generated maps against the original ground-truth data, the system achieved a high level of accuracy, correctly identifying the shapes and positions of buildings, roads, and other features in the majority of cases. The study found that providing the AI with that initial rough sketch was crucial; without it, the system struggled to keep the different parts of the city aligned correctly.
The significance of this work lies in what it makes possible. By turning a satellite image into a set of instructions, the researchers have created a bridge between a static photograph and a dynamic, editable digital asset. The system does not just describe what is there; it captures the logic of how the city is built. This means that the output is not just a picture, but a living record that can be inspected, corrected, and regenerated. It suggests a future where remote sensing does more than just monitor the world; it could provide the foundational data needed to maintain and update digital twins of our cities, keeping them accurate and usable for planning and design. The study demonstrates that visual observations can indeed be translated into a form that is both measurable and editable, offering a new way to interact with the complex geometry of our urban environments.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.