💻 computer science

CityRiSE: Reasoning Urban Socio-Economic Status in Large Vision-Language Models via Reinforcement Learning

The paper introduces CityRiSE, a novel framework that leverages reinforcement learning with carefully curated multi-modal data and verifiable rewards to enhance Large Vision-Language Models' ability to perform accurate, interpretable, and generalizable reasoning for urban socio-economic status prediction across diverse and unseen contexts.

Original authors: Tianhui Liu, Hetian Pang, Xin Zhang, Jie Feng, Pan Hui, Yong Li

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianhui Liu, Hetian Pang, Xin Zhang, Jie Feng, Pan Hui, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for fingerprints or footprints, you are looking at pictures of cities to guess how rich, healthy, or educated the people living there are. This field of science is called "urban socio-economic sensing." For a long time, computers have been good at spotting simple things in photos, like "that's a car" or "that's a tree." But guessing complex human stories—like the average income or life expectancy of a neighborhood—has been like asking a calculator to write a poem. It's hard because these numbers are abstract, and the data is messy. Recently, we've built "Large Vision-Language Models" (LVLMs), which are super-smart AI brains that can see pictures and read text at the same time. They are like students who have read every book in the library and seen every photo on the internet. The big question researchers are asking is: Can we teach these AI students to not just guess numbers, but to actually think through the clues in a city photo to figure out the story of the people living there?

This is where a new project called CityRiSE comes in. The researchers behind it realized that while these AI models are powerful, they often struggle to make accurate guesses about cities they haven't seen before, or about specific social issues they weren't explicitly taught. They also tend to just spit out a number without explaining why, which makes it hard to trust them. To fix this, the team didn't just feed the AI more pictures; they taught it how to reason using a method called Reinforcement Learning. Think of this like training a dog, but instead of treats, the AI gets a "reward" when it solves a puzzle correctly. The team built a special training game where the AI had to look at street views and satellite images, spot important clues (like the type of buildings, the presence of greenery, or the condition of the roads), and then explain its thinking step-by-step before giving a final answer.

The secret sauce of CityRiSE is how it rewards the AI. The researchers designed two types of rewards. The first is a "Regression Reward," which acts like a scorecard. If the AI guesses the economic status is a "7" and the real answer is "8," it gets a decent reward. If it guesses "1," it gets a big penalty. This teaches the AI that being close is better than being wildly wrong. The second is a "Keyword Reward," which is like a checklist. The AI gets points if it mentions specific, useful things in its explanation, like "vehicles," "greenery," or "street furniture." This forces the AI to look at the right parts of the picture and explain its logic, rather than just hallucinating an answer. They also gave the AI some extra practice puzzles, like figuring out which city a photo came from or counting objects, to sharpen its general reasoning skills.

The results are quite impressive. The team trained CityRiSE using only about 5,109 examples, which is a tiny amount compared to the millions of images other models usually need. Despite this small dataset, CityRiSE performed better than many existing models, including some very expensive, commercial AI systems. Even more exciting, it showed a rare ability called "generalization." This means it could look at a city it had never seen before (like a city in a different country) or try to predict a new type of statistic it had never been asked about (like mental health access instead of just income), and still make a good guess. The paper suggests that by learning to reason through the visual clues, the AI didn't just memorize the training data; it learned a flexible way of thinking that works in new situations.

In short, CityRiSE shows that if you teach a large AI model to "show its work" and reward it for finding the right visual clues, it can become a much better detective for understanding our cities. It moves beyond just being a black box that spits out numbers and becomes a tool that can explain why a neighborhood might be struggling or thriving, based on what it actually sees in the images. This approach suggests a future where AI can help us understand complex social issues in new places without needing massive amounts of pre-labeled data for every single new problem.

Technical Summary: CityRiSE

Problem Statement

Urban socio-economic sensing is critical for sustainable development but remains challenging due to the abstract nature of indicators (e.g., GDP, education levels) and the heterogeneity of visual data sources (satellite and street view imagery). While Large Vision-Language Models (LVLMs) offer new opportunities for multi-modal perception, existing approaches face three primary limitations:

Limited Generalization: Models often fail to transfer effectively to unseen cities or unseen socio-economic indicators.
High Data Costs: Supervised Fine-Tuning (SFT) methods typically require tens of thousands of labeled samples, limiting scalability in low-resource settings.
Lack of Interpretability: Most models output numerical predictions without providing reasoning steps, making their decision-making processes opaque and difficult to trust.

Current methods, including those based on knowledge graphs or contrastive learning, struggle to balance data efficiency with the ability to generalize across diverse urban contexts and prediction targets.

Methodology

The paper proposes CityRiSE, a framework that leverages Reinforcement Learning (RL) to enable emergent, interpretable reasoning in LVLMs for urban socio-economic status prediction. The methodology consists of three core components:

1. Transferable Dataset Construction

To facilitate generalization across cities and indicators, the authors constructed a dataset suite comprising:

Socio-Economic Indicator Data: Based on the CityLens dataset (17 cities, 6 domains), indicators are discretized into 10 bins (1–10) to normalize heterogeneous value scales. The evaluation splits include spatial transferability (unseen cities) and indicator generalization (unseen targets).
Perceptual Urban Reasoning Data: Three auxiliary tasks designed to cultivate transferable cognitive patterns:
- Spatial Reasoning: Determining the relative spatial distance of street view images.
- Geolocation: Predicting the city of origin from a single image.
- Socio-Economic Ranking: Comparing two images to determine which has a higher value for a specific indicator.
General Visual Reasoning Data: Tasks including object counting and pattern completion to build fundamental perceptual skills (e.g., symmetry, analogy) transferable to urban reasoning.

2. Guided Reward Design

The framework employs Group Relative Policy Optimization (GRPO) with a dual-component reward mechanism to guide the LVLM toward interpretable and accurate reasoning:

Keyword Reward: Encourages the model to attend to specific, task-relevant urban visual features (e.g., "person," "greenery," "building," "location"). This acts as a lightweight perceptual prior, grounding the model's reasoning in semantic evidence rather than arbitrary text generation.
Regression Reward: Based on Huber loss, this reward penalizes prediction errors non-linearly. It distinguishes between small and large deviations from ground truth, providing a more nuanced learning signal than binary correctness metrics.

3. Training Pipeline

CityRiSE is trained using GRPO on the combined datasets. The algorithm generates multiple candidate responses for each input, calculates rewards based on the dual-component design, and updates the policy to maximize the group-normalized advantage. This process allows the model to learn transferable reasoning patterns without relying on massive amounts of supervised labeled data.

Key Contributions

First RL-Driven Visual Reasoning for Urban Sensing: To the authors' knowledge, this is the first work to apply reinforcement learning with LVLMs to induce emergent visual reasoning for high-level urban socio-economic prediction.
Cross-Indicator and Cross-City Generalization: By learning to predict normalized indicator statuses, CityRiSE acquires transferable reasoning patterns. It demonstrates the first successful cross-indicator generalization in this domain, performing well on indicators and cities not seen during training.
Data Efficiency: The framework achieves strong performance using only 5,109 training samples, significantly outperforming proprietary commercial LVLMs and SFT-based models that rely on much larger datasets.
Interpretable Reasoning: Through verifiable reward design, CityRiSE induces coherent, goal-directed reasoning chains. This provides transparency and explanatory insights, moving beyond "black-box" numerical outputs.

Experimental Results

Extensive experiments were conducted on 10 urban socio-economic prediction tasks across three settings: In-domain, Unseen Cities, and Unseen Indicators.

Performance vs. Baselines: CityRiSE significantly outperformed existing baselines, including classic methods (SVF, UrbanCLIP), general LVLMs (Qwen, GPT-4, Gemini), and SFT-based domain models (UrbanMLLM, UI-CoT).
- In Unseen Indicators, CityRiSE achieved positive $R^2$ scores on all tasks, including challenging ones like Life Expectancy ($0.200$) and House Price ($0.218$), where other models often failed or produced negative scores.
- In Unseen Cities, it achieved the highest performance on Bachelor Ratio ( $R^2 = 0.286$ ), surpassing SFT-based models.
Ablation Studies:
- Reward Design: Removing either the Keyword or Regression reward consistently degraded performance. Removing both led to severe failure, confirming the necessity of both semantic grounding and numerical precision.
- Data Composition: Removing the auxiliary reasoning datasets (Perceptual and General Visual) caused significant performance drops, particularly in out-of-domain settings, validating their role in building transferable skills.
- Reasoning vs. Non-Reasoning: A variant of CityRiSE trained without intermediate reasoning steps performed worse than the full model, demonstrating that the reasoning process itself contributes to predictive accuracy.
Input Modality: Street-view imagery was found to be the primary source of fine-grained evidence, while satellite imagery provided complementary regional context.

Significance and Claims

The paper claims that CityRiSE represents a paradigm shift in urban socio-economic sensing by demonstrating that Reinforcement Learning can induce emergent, interpretable reasoning capabilities in LVLMs that surpass the generalization limits of supervised fine-tuning.

The authors emphasize that their approach:

Solves the "memorization vs. generalization" trade-off, where SFT excels at in-domain memorization but RL enables out-of-domain generalization.
Provides a scalable, data-efficient solution for global urban analysis, particularly valuable for regions with limited labeled data.
Offers a transparent framework where the model's reasoning can be inspected, fostering trust in AI-driven policy and planning decisions.

The work concludes that combining RL with LVLMs holds significant promise for creating generalist, interpretable systems capable of understanding complex urban socio-economic dynamics across diverse and unseen contexts.