FujiView: Multimodal Late-Fusion for Predicting Scenic Visibility

FujiView introduces a multimodal late-fusion framework and a large-scale dataset that combines webcam imagery with meteorological data to accurately predict scenic visibility around Mount Fuji, achieving high accuracy across short-term and next-day horizons.

The Gist

Imagine you are planning a trip to see Mount Fuji. You wake up, grab your coffee, and check the weather app. It says "sunny." Great! But when you look out the window, the mountain is completely hidden behind a thick wall of gray clouds. You've been fooled by the forecast.

This is the problem FujiView solves. It's a new smart system that acts like a super-powered travel assistant, combining two different ways of seeing the world to tell you: "Can I actually see Mount Fuji today, and will I be able to see it tomorrow?"

Here is how the paper explains this in simple terms, using some fun analogies.

1. The Problem: The "Weather Lie"

Mount Fuji is famous, but it's also shy. It hides behind clouds often, especially in the rainy season.

• The Old Way: Travelers used to rely on two things:
  1. Weather Forecasts: Like a weatherman guessing the future. Sometimes he's right, but he can't see the specific clouds blocking your view right now.
  2. Webcams: Like looking out a window. You can see what's happening right now, but you have no idea what will happen in an hour or tomorrow.
• The Gap: Neither method alone is perfect. The weatherman can't see the specific cloud blocking the mountain, and the webcam can't predict the future.

2. The Solution: The "Two-Headed Detective"

The researchers built a system called FujiView that acts like a detective with two heads. One head is a Visual Expert (a computer vision AI), and the other is a Weather Expert (a data analyst).

Instead of forcing them to share a brain immediately, they use a strategy called "Late Fusion." Think of it like a team meeting at the end of the day:

• The Visual Expert looks at a photo from a webcam and says, "Right now, the mountain looks 90% clear."
• The Weather Expert looks at the forecast and says, "In 24 hours, a storm front is moving in."
• The Team Meeting (Late Fusion): They combine their notes. "Okay, it's clear now, but the storm is coming. Let's tell the traveler: 'Go now, but don't wait until tomorrow.'"

3. The Ingredients: A Massive Library

To teach this system, the researchers built a giant library of data:

• The Eyes: They connected to 42 different webcams around Mount Fuji. Every 30 minutes, they took a picture.
• The Brain: They paired every single photo with weather data (temperature, wind, humidity, cloud cover) from the exact same moment.
• The Training: Humans manually looked at over 26,000 of these photos and labeled them:
  • 🌟 Perfect: The mountain is crystal clear.
  • ☁️ Clear: You can see it, but a few clouds are touching it.
  • 🌫️ Cloudy: It's there, but mostly hidden.
  • 🚫 Obscured: You can't see it at all.
  • ❌ Bad: The camera is broken or it's too dark.

They are building this library to be huge (over 300,000 photos eventually) so the AI can learn from every possible weather condition.

4. How It Works: The "Magic Trick" of Timing

The paper discovered something really interesting about how the two experts work together, depending on when you ask the question:

• For "Right Now" (Nowcasting):
  The Visual Expert is the boss. If you ask, "Can I see it right now?", the AI looks at the webcam photo. It's like looking out the window; you don't need a weather report to know if it's raining right this second. The photo is the most powerful clue.
• For "Tomorrow" (Forecasting):
  The Weather Expert takes over. If you ask, "Will it be clear tomorrow?", the webcam photo from today doesn't help much. The AI needs to look at the forecast data (wind patterns, pressure systems) to guess what the sky will look like 24 hours from now.
• The Sweet Spot (Late Fusion):
  The magic happens when they combine both.
  • Today: The photo confirms the weather report.
  • Tomorrow: The weather report predicts the future, but the photo from today acts as a "reality check." If the forecast says "sunny tomorrow" but the mountain is already looking foggy today, the system knows the forecast might be wrong and adjusts the prediction.

5. The Results: Why It Matters

The system is surprisingly good at its job:

• Same-Day Prediction: It gets it right about 89% of the time.
• Next-Day Prediction: It gets it right about 84% of the time.

Why does this matter?
Imagine you are a tourist with only one day in Japan. You don't want to waste your day driving to a spot where the mountain is hidden. FujiView can tell you: "Don't go to Tokyo today; the mountain is hidden. But drive to Shizuoka instead; the clouds are clearing there!"

This helps:

1. Travelers: Save time and money by going where the view is good.
2. Local Towns: Spread tourism to smaller towns that usually get overlooked.
3. Scientists: It creates a new way to study how we can predict what humans see, not just what the weather is.

The Bottom Line

FujiView is like a smart travel guide that doesn't just read the weather report; it actually looks at the sky, remembers what the sky looked like an hour ago, and uses that to guess what the sky will look like tomorrow. It's a new kind of "Scenic Visibility Forecasting" that helps us see the world more clearly, one mountain at a time.

Technical Summary

1. Problem Definition

The paper addresses the challenge of Scenic Visibility Forecasting (SVF): predicting whether a natural landmark (specifically Mount Fuji) will be visible to the human eye at a specific location and time.

• Context: Mount Fuji's visibility is highly unpredictable due to rapid atmospheric changes, cloud cover, and seasonal variations (e.g., the rainy season). Existing tools provide either raw webcam feeds or generic weather forecasts but fail to integrate these sources to answer the specific question: "Will I be able to see Mount Fuji today or tomorrow?"
• Complexity: Visibility is not a binary state but a continuum ranging from "Perfect" to "Obscured." The task requires fusing heterogeneous data modalities (visual imagery and numerical weather data) across varying time horizons (from immediate "nowcasting" to multi-day forecasting).
• Gap: There was a lack of large-scale, labeled datasets and standardized benchmarks for this specific multimodal task.

2. Methodology

The authors propose FujiView, a multimodal learning framework utilizing a Late-Fusion architecture.

A. Dataset Construction

• Scale: A continuously expanding dataset comprising over 113,000 webcam images (projected to exceed 320,000) from 42 locations around Mount Fuji.
• Data Sources:
  • Visual: 30-minute interval captures from public webcams.
  • Meteorological: Concurrent and forecasted weather data (temperature, humidity, cloud cover, wind, pressure, etc.) retrieved via the Open-Meteo API.
• Annotation:
  • Images are manually labeled into five classes: Perfect, Clear, Cloudy, Obscured, and Bad (unusable).
  • Quality Control: Automated filters remove dark/gray frames (>40% grayness) and duplicates.
  • Target Generation: Frame-level labels are aggregated into day-level binary targets (Visible vs. Not Visible) based on a threshold ($\theta=50\%$ of frames being visible). Targets are shifted to create multi-horizon datasets (+0d to +3d).

B. Model Architecture

The system employs a Late-Fusion strategy rather than early fusion (pixel-level concatenation) to leverage the complementary strengths of each modality without over-parameterization.

1. Vision Branch (YOLOv8):
   • A pre-trained YOLOv8n-cls model is fine-tuned on the labeled webcam images.
   • Instead of hard class predictions, the model outputs calibrated softmax probabilities ($P_{perfect}, P_{clear}, P_{cloudy}, P_{obscured}$).
   • These probabilities serve as continuous visual features, capturing uncertainty in ambiguous cases.
2. Meteorological Branch:
   • Structured tabular data including current conditions and forecasts up to 3 days ahead.
3. Fusion Learner (LightGBM):
   • A Gradient-Boosted Decision Tree (LightGBM) model acts as the fusion engine.
   • Inputs: Concatenated YOLO probability vectors + Weather variables + Metadata (camera ID, time, location).
   • Output: Binary classification (Visible/Not Visible) for specific time horizons (+0d, +1d, +2d, +3d).
   • Rationale: LightGBM is chosen for its efficiency with heterogeneous data, ability to handle mixed categorical/continuous/probabilistic inputs, and interpretability (SHAP values).

C. Training Strategy

• Horizons: The model is trained to predict visibility for:
  • +0d (Same-day/Samedaycasting): Using a first-frame snapshot or a 3-hour morning window.
  • +1d to +3d (Tomorrowcasting/Multi-day): Using forecasted weather data.
• Validation: 5-fold GroupKFold cross-validation (grouped by camera and date) to prevent data leakage.

3. Key Contributions

1. Large-Scale Multimodal Dataset: The creation of the largest known dataset for scenic visibility, pairing 100k+ images with aligned weather forecasts, released to support future research.
2. Late-Fusion Framework: Demonstration that fusing image-derived probabilities with weather data via LightGBM outperforms single-modality baselines.
3. Definition of SVF: Formalizing "Scenic Visibility Forecasting" as a distinct benchmark task that bridges computer vision and environmental forecasting, distinguishing between nowcasting (image-only) and forecasting (weather-driven).
4. Tools & Deployment: Release of an automated data collection pipeline, a web-based labeling platform, and a public-facing application.

4. Experimental Results

The study evaluated performance using Accuracy (ACC) and ROC-AUC across different time horizons.

• Short-Term (+0d / Same-day):
  • Dominant Modality: Vision (YOLO).
  • Performance: YOLO-only achieved 89.2% ACC and 0.94 AUC.
  • Insight: Visual features are the strongest predictor for immediate conditions. Adding weather data provided marginal gains or slight AUC reduction, suggesting weather data adds noise for immediate nowcasting.
• Medium-Term (+1d / Tomorrow):
  • Dominant Modality: Late Fusion.
  • Performance: Fusion achieved 74.1% ACC, outperforming YOLO-only (64.4%) and Weather-only (71.1%).
  • Insight: Combining current visual state with forecasted trends provides the most robust signal.
• Long-Term (+2d to +3d):
  • Dominant Modality: Weather/Fusion.
  • Performance: Fusion maintained 77.3% ACC at +2d and 71.5% at +3d.
  • Insight: As the horizon extends, weather forecasts become the primary predictive signal. However, YOLO probabilities remain non-negligible contributors, acting as a "reality check" against forecast errors.
• Feature Importance:
  • At +0d, $P_{perfect}$ and $P_{obscured}$ from YOLO are the top features.
  • At +1d and beyond, forecasted cloud cover becomes the single most important feature.
• Windowing Strategy: Using a 3-hour morning window (averaging frames) improved AUC for pure vision models but slightly reduced accuracy for fusion models at longer horizons, indicating a trade-off between noise reduction and preserving informative outliers.

5. Significance and Impact

• Practical Application: The system directly aids tourism planning, helping travelers decide when and where to visit Mount Fuji, thereby distributing tourism to lesser-known towns and supporting regional economic revitalization.
• Scientific Contribution:
  • Establishes SVF as a new benchmark for multimodal learning, highlighting the unique challenges of fusing static images with dynamic time-series weather data.
  • Demonstrates that Late Fusion is a superior strategy for this domain compared to early fusion, offering a balance of interpretability, efficiency, and accuracy.
  • Provides a template for applying similar multimodal approaches to other environmental forecasting tasks (e.g., air quality, wildfire smoke visibility, or other landmarks like the Matterhorn).
• Limitations & Future Work: The current dataset is biased toward the rainy season (high "Obscured" class). Future work aims to expand data collection year-round, incorporate temporal models (e.g., ConvLSTMs), and develop methods to utilize nighttime webcam data.

In conclusion, FujiView successfully bridges the gap between raw weather data and human-perceived visibility, proving that a principled late-fusion approach can deliver high-accuracy scenic forecasts essential for both travelers and environmental research.