Decoding Tourist Perception in Historic Urban Quarters with Multimodal Social Media Data: An AI-Based Framework and Evidence from Shanghai

This study introduces an AI-based multimodal framework that analyzes geotagged photos and reviews from Shanghai's historic quarters to decode tourist perceptions, revealing significant gaps between social media representations and on-site realities to guide heritage management and urban design.

The Gist

Imagine you are a city planner trying to fix up an old, historic neighborhood. You want to make it beautiful for tourists, but you also need to keep it livable for locals. The problem is, you don't really know what tourists actually notice, what they like, or what they hate. You could ask them to fill out surveys, but that's slow, expensive, and people often forget what they felt once they leave.

This paper proposes a clever solution: Let the tourists' own photos and reviews tell the story.

The researchers built an "AI Detective" that acts like a super-powered translator, turning millions of social media posts into a clear map of tourist feelings. Here is how they did it, broken down into three simple steps:

1. The "Eye-Tracker" (What do they look at?)

The Analogy: Imagine tourists are wearing special glasses that highlight exactly what they are staring at when they take a photo.
The Science: The team used an AI model (a type of computer vision) to scan thousands of photos taken in 12 different historic areas of Shanghai. Instead of just seeing a "photo," the AI broke the image down into pieces: That's a building, that's a tree, that's a street sign, that's a person.
The Discovery: They found that tourists don't just take random pictures. They consistently focus on specific things. In some areas, they zoom in on green trees and old buildings (making the place feel peaceful). In busy food streets, they focus on crowds and food. In artistic districts, they hunt for graffiti and art. This tells planners exactly what visual "ingredients" make a neighborhood feel special.

2. The "Filter Detective" (What do they show vs. what is real?)

The Analogy: Think of Instagram filters. Sometimes, a sunny day looks golden, but on your phone, it looks like a cool, blue-tinted dream. The researchers asked: Is the neighborhood actually blue, or are tourists just making it look that way?
The Science: They compared the colors in the tourists' social media photos against "Street View" photos (which are like a raw, unedited Google Maps snapshot of the actual street).
The Discovery: There was a gap! The real streets were often warmer (more oranges, yellows, and brick tones). But the tourists' photos were cooler (more blues and greens). It's like the tourists are putting a "cool, artistic filter" on reality. This suggests that tourists expect these places to feel a certain way, and they curate their photos to match that dream, even if the real street looks a bit different.

3. The "Emotion Translator" (What do they say?)

The Analogy: Imagine a robot that reads a thousand reviews and instantly sorts them into four different bins: Fun Stuff, The Look of the Place, The Amenities (like bathrooms), and The Shops.
The Science: They used a smart language AI to read the text reviews. Instead of just saying "This was good" or "This was bad," the AI figured out why. Did they hate the crowds? Did they love the history? Were they angry about expensive snacks?
The Discovery: They found that even if a place looks beautiful (the "Built Environment" gets high scores), people might still be unhappy if the bathrooms are dirty or the shops are too expensive. The AI helped pinpoint exactly which "bin" of the experience was broken.

The Big Picture: Why does this matter?

Think of this framework as a diagnostic tool for a city's health.

• Before: Planners were guessing. "Maybe we need more trees?" or "Maybe we need to fix the lights?"
• Now: They have a report card.
  • If the "Eye-Tracker" says people love the trees but the "Emotion Translator" says they hate the trash cans, the city knows to clean up the trash, not plant more trees.
  • If the "Filter Detective" says people expect a cool, blue vibe but the buildings are actually rusty orange, the city might need to paint the buildings or add lighting to match that expectation.

In short: This paper teaches us how to listen to the "digital whispers" of millions of tourists. By combining what they see, how they color it, and what they complain about, cities can design historic neighborhoods that aren't just preserved in glass cases, but are vibrant, happy places that actually work for the people visiting them.

Technical Summary

1. Problem Statement

Historic urban quarters face a critical planning challenge: balancing heritage preservation with the experiential quality required for tourism and daily life. Planners often lack scalable, evidence-based insights into what tourists notice, how they aesthetically represent these spaces, and what they criticize.

Existing research suffers from three main gaps:

• Data Silos: Visual content (photos) and textual content (reviews) are often analyzed in isolation, failing to link visual attention with narrative appraisal.
• Lack of Contextual Interpretation: Computational studies often focus on formal metrics (e.g., façade harmony) without connecting them to evaluative judgments in heritage contexts.
• Limited Scope: Historic quarters are underexamined compared to general destinations, with a lack of frameworks that connect attention, representation, and satisfaction mechanisms.

The study aims to bridge the gap between on-site reality and tourist perception/representation using a multimodal AI framework.

2. Methodology

The study proposes an AI-powered, multimodal framework applied to 12 historic urban quarters in Shanghai. The workflow consists of three integrated modules:

A. Data Collection & Preparation

• Sources: Geotagged photos and review texts were crawled from Dianping (a major Chinese lifestyle platform) between 2018–2024. Street view images were retrieved from Baidu Maps as a real-world baseline.
• Dataset: 32,949 unique reviews and 168,137 photos were processed. A balanced subset of 20,968 photos was created for image analysis.
• Semantic Taxonomy: A custom taxonomy of 22 visual categories (e.g., buildings, trees, artworks, vendors) was defined.
• Annotation Strategy: A hybrid approach was used:
  1. SAM (Segment Anything Model) assisted manual annotation of 1,000 initial photos.
  2. YOLOv8-Seg was fine-tuned to create an automatic labeling model.
  3. The final dataset was expanded using this automated model.

B. Module 1: Visual Focus Detection (Semantic Segmentation)

• Model Selection: After evaluating SegFormer, Mask2Former, and DeepLabV3, the study selected DeepLabV3+ with a MobileNetV2 backbone for its balance of accuracy and efficiency.
• Training Strategy: A two-stage training approach (freezing lower layers, then fine-tuning all layers) was used to prevent overfitting. Class weights were applied to the loss function to address data imbalance (e.g., rare categories like fountains).
• Output: Quantification of pixel proportions for each semantic class to determine what elements are "foregrounded" in tourist photos.
• Validation: Results were cross-verified with the CLIP model for high-level semantic understanding.

C. Module 2: Aesthetic Preference Analysis (Color Composition)

• Technique: Images were converted from RGB to HSV color space and corrected using the Gray World algorithm for white balance.
• Clustering: K-means clustering identified dominant color palettes (top 5 colors).
• Classification: Colors were mapped to the Chinese Color System (1,000 predefined hues).
• Comparison: The study compared color distributions between tourist photos and street-view baselines using the Kolmogorov-Smirnov (K-S) divergence metric to identify "perception–reality gaps."

D. Module 3: Multidimensional Satisfaction Assessment

• Dimensions: Reviews were analyzed across four planning-relevant dimensions:
  1. Tourist Activities
  2. Built Environment
  3. Service Facilities
  4. Business Formats
• Model: A Multi-task BERT model (BERT-base-Chinese) was trained with four independent classification heads to predict sentiment (-1: negative, 0: neutral, 1: positive) for each dimension.
• Hybrid Approach: Results were validated against a rule-based lexicon method and manual interpretation of sample reviews.

3. Key Results

Visual Attention Patterns

• Dominant Elements: Buildings (19.12% of pixels) and Trees (10.80%) are the most foregrounded elements.
• Quarter Specificity:
  • Leisure/Commercial Areas: Dominated by greenery and historic architecture.
  • High-Traffic Zones (e.g., The Bund): Humans and buildings dominate; greenery is scarce.
  • Artistic Hubs (e.g., Tian'ai Road): Significantly higher proportions of artworks (up to 12.81% in specific spots).
  • Food Streets: Higher frequency of food-related imagery.

Perception–Reality Gap (Color Analysis)

• Systematic Bias: Social media photos consistently exhibit cooler color tones (blues, greens) compared to the warmer tones (oranges, yellows, beiges) found in actual street views.
• Causes: This gap is attributed to platform processing, device computational photography (HDR, auto-white balance), and user curation (filters) rather than physical reality.
• Implication: Tourists curate images to match platform aesthetics, creating a "perception–representation bias" that planners must account for.

Sentiment Analysis

• The Multi-task BERT model successfully identified nuanced sentiments across the four dimensions.
• Key Findings:
  • Built Environment often receives positive scores even when Service Facilities (e.g., restrooms, crowding) or Business Formats (over-commercialization, high prices) are criticized.
  • Dissatisfaction is often clustered around overcrowding and commercialization, even in visually appealing locations.

4. Key Contributions

1. Integrated Framework: Developed a novel pipeline that explicitly links Visual Attention (what is seen), Representation (how it is portrayed via color), and Appraisal (sentiment in text) into a single analytical chain.
2. Specialized Semantic Taxonomy: Created a 22-category taxonomy and a fine-tuned segmentation pipeline specifically tailored for historic street environments, enabling granular measurement of tourist visual focus.
3. Perception–Reality Diagnostics: Introduced a method to quantify the chromatic gap between social media and street views, offering an interpretable tool to diagnose aesthetic mismatches and their relation to satisfaction.

5. Significance and Implications

• For Urban Planning: The framework moves beyond simple "destination image" studies to provide actionable, dimension-specific diagnostics. Planners can use these indicators to:
  • Optimize greening strategies based on visual focus data.
  • Address service bottlenecks (e.g., restrooms) identified in sentiment analysis.
  • Manage commercial balance to prevent perceived over-commercialization.
• For Heritage Management: The study highlights that conservation cannot focus solely on architectural control; it must also manage the experiential qualities (walkability, atmosphere) that tourists curate and value.
• Methodological Advancement: It demonstrates how AI (Deep Learning, NLP) can be embedded in theory-informed urban research to turn unstructured digital traces into scalable, planning-relevant evidence.

6. Limitations & Future Work

• Data Bias: User-generated content may not represent all visitor demographics.
• Technical Variance: Differences in camera devices and capture conditions (lighting, weather) introduce noise in color analysis.
• Temporal Factors: Seasonal and time-of-day effects were not fully disentangled.
• Future Directions: Integrating mobility traces, expanding to other cultural contexts, and refining color calibration strategies to distinguish between aesthetic preference and technical distortion.