💻 computer science

A Hybrid AI-IoT Framework for Predictive Conservation of Tropical Heritage Façades: Empirical Validation from a Multi-Site Study in Indonesia

This study empirically validates a hybrid AI-IoT framework that integrates continuous environmental sensing with machine learning to predict and provide early warnings for chromatic deterioration on tropical heritage façades in Indonesia, achieving high accuracy and generalizability across multiple sites.

Original authors: HASSAN GBRAN, Siti Rukayah, Atik Suprapti, Erni Setyowati, Iwan Setiawan

Published 2026-07-29

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: HASSAN GBRAN, Siti Rukayah, Atik Suprapti, Erni Setyowati, Iwan Setiawan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for fingerprints, you are looking for the invisible fingerprints of weather on old buildings. This field of science sits at the crossroads of heritage conservation (saving old, important buildings) and predictive technology (using data to guess what will happen before it happens). For a long time, people have known that old buildings in hot, wet places get sick. The "sickness" often starts as a change in color—like a wall turning a dark, damp shade or growing fuzzy green spots. These color changes are the building's way of coughing; they happen long before the walls actually crumble.

Traditionally, experts have to walk around with their eyes and cameras, guessing where the trouble is. It's like trying to find a leak in a roof by looking at the ceiling only after the water has already dripped. But what if we could give the building a "smartwatch"? That's where IoT (Internet of Things) comes in—tiny sensors that constantly whisper to us about the temperature and humidity. And what if we could teach a computer to listen to those whispers and look at the building's colors at the same time? That's the power of AI (Artificial Intelligence). The big question isn't just "Is the building sick?" but "Can we predict the sickness before it starts, giving us enough time to fix it?" This paper dives into exactly that, asking if we can build a digital crystal ball for old buildings in the tropics.

The Digital Doctor for Old Buildings

In the humid, rainy city of Semarang, Indonesia, three historic buildings are facing a silent enemy: the weather. These aren't just any buildings; they are the Lawang Sewu (a colonial administrative office), Pasar Johar (a traditional market), and the Tawang Railway Station. They are made of brick, plaster, and wood, and they are slowly losing their color and strength because of the constant heat and moisture.

The researchers, led by Hassan Gbran, decided to build a "Hybrid AI-IoT Framework." Think of this as giving the buildings a team of two detectives: one is a sensing team (the IoT) and the other is a brain (the AI).

The Sensing Team (The IoT)
First, they deployed 16 tiny sensor nodes across the three buildings. These sensors acted like a nervous system, constantly checking the air every 15 minutes. They measured the temperature and, most importantly, the Relative Humidity (RH). In these tropical places, the air is often so wet that it feels like a sponge. The sensors tracked how long the humidity stayed above 85%—a level that is like a "danger zone" for these materials. They even created a special score called the Humidity Stress Factor (HSF), which is like a "stress meter" that adds up how much wetness the building has endured over the last day, rather than just looking at the current moment.

The Brain (The AI)
While the sensors were busy watching the weather, the researchers took thousands of high-resolution photos of the buildings' walls. They used a technique called photogrammetry to turn these flat photos into 3D digital twins. Then, they broke these 3D models into millions of tiny tiles and analyzed the colors.

They used two types of AI detectives:

The Unsupervised Detective (Hierarchical Clustering): This AI looked at the colors and grouped them together without being told what they were. It's like sorting a bag of mixed marbles by color without knowing the names of the colors. It found groups of "damp spots," "moldy areas," and "normal bricks."
The Supervised Detective (Random Forest): This is the star of the show. This AI was trained by human experts who labeled thousands of tiles as "mold," "salt stains," or "healthy." The AI learned to recognize these patterns. Crucially, this AI didn't just look at the colors; it also looked at the Humidity Stress Factor from the sensors. It learned that if the "stress meter" is high, a certain color change is likely to happen soon.

The Big Discovery: A Head Start on Disaster

The results were impressive. The AI didn't just identify the damage; it predicted it.

Accuracy: The AI was a very good detective. It correctly identified the type of damage on the walls about 87% of the time (a score called F1). When they compared the AI's map of the damage to the actual damage, the AI was 17% to 51% more accurate than the unsupervised detective working alone. This proves that knowing the weather history helps the AI understand the colors better.
The Humidity Stress Factor: The researchers ran a test to see what happened if they removed the weather data. The AI's performance dropped by 9.2%. This confirmed that the "stress meter" (HSF) was the most important clue. It's not just about how wet it is right now, but how wet it has been for the last 24 hours.
The Crystal Ball (Lead Time): This is the coolest part. The system could identify a "high-risk event" (like a sudden burst of mold growth) 4.8 to 16.7 hours before the condition fully materialized in the test data. However, the paper is careful to note that these are technical lead times. They show the system can detect the warning signs early, but the study did not test whether building managers could actually act on these warnings within that window. Factors like staff availability, budget cycles, and decision protocols were not assessed, so the ability to "fix it before damage starts" remains a theoretical possibility rather than a proven operational reality.

How They Decided What to Fix First

The researchers didn't just want to find problems; they wanted to know which problems mattered most. They created a Priority Matrix. They combined the Risk Score (how likely is the wall to rot?) with a Heritage Value Index (how important is this wall to history?).

High Risk + High Value: This is the "Red Zone." For example, the main entrance of Lawang Sewu had a high heritage value and moderate risk, making it a top priority.
High Risk + Low Value: This is the "Technical Zone." The basement of Lawang Sewu had very high humidity and risk, but lower historical value. It still needed fixing, but maybe not with the same urgency as the main hall.
Low Risk + High Value: This is the "Watch Zone." Just keep an eye on it.

The Catch and The Future

The paper is very honest about its limits. The AI is a "blueprint," not a magic wand that works everywhere instantly.

It needs local tuning: The specific humidity threshold (85%) might work in Semarang, but a different tropical city might need a different number. The model needs to be "recalibrated" for new places.
It's a technical warning, not a magic fix: The AI can say, "Danger in 12 hours!" but it doesn't know if the building managers have the budget or the staff to actually fix it in that time. That part is still up to humans, and the study did not assess operational feasibility.
Subjectivity: While the AI is great at spotting colors, deciding if a color change is "mold" or just "dirt" still requires human experts. The researchers used three experts to agree on the labels, but there is still a little bit of human guesswork involved.

The Bottom Line

This study proves that we can stop treating old buildings like patients who only get help when they are already dying. By combining tiny sensors that watch the weather with smart AI that watches the colors, we can get a 4 to 16-hour technical head start on detecting high-risk conditions in tropical heritage sites. It's a shift from "reactive" (fixing the broken wall) to "predictive" (identifying the risk early). While the system needs more testing in other cities, better sensors for the future, and real-world operational trials to see if managers can act on the warnings, it offers a hopeful, data-driven way to keep our history standing tall in a changing climate.

Technical Summary: A Hybrid AI-IoT Framework for Predictive Conservation of Tropical Heritage Façades

Problem Statement
Tropical heritage façades in coastal urban environments suffer from accelerated chromatic deterioration driven by persistent high humidity, temperature fluctuations, and salt-laden air. Current conservation practices are predominantly reactive, relying on manual visual surveys that lack radiometric sensitivity and spatial completeness. Furthermore, existing environmental monitoring systems often operate in isolation from computational diagnostic pipelines. A critical methodological gap exists: environmental data (e.g., relative humidity) is frequently treated as a post-hoc explanatory variable rather than a predictive input, and humidity thresholds are often imported from temperate climates without local calibration. Additionally, translating purely chromatic clusters into semantically meaningful decay categories often relies on unstandardized expert interpretation, introducing subjectivity.

Methodology
The study proposes and validates a hybrid AI-IoT framework designed for predictive monitoring of chromatic deterioration. The methodology integrates continuous environmental sensing with a multi-stage machine learning pipeline across three heritage sites in Semarang, Indonesia: Lawang Sewu (colonial administrative), Pasar Johar (traditional market), and Tawang Railway Station (transport infrastructure).

Data Acquisition and Preprocessing:
- Visual Data: High-resolution photogrammetric models were generated using calibrated DSLR cameras (24 MP) and processed through a six-step pipeline including radiometric correction (ColorChecker), denoising, dense reconstruction, and UV texture baking. Ground Sampling Distance (GSD) ranged from 0.8 to 2.4 mm/pixel.
- Environmental Data: A network of 16 IoT sensor nodes (DHT22 sensors) deployed according to architectural logic (ventilation pathways, thermal gradients) recorded temperature and relative humidity at 15-minute intervals over 14 months. A systematic drift correction protocol using laboratory-grade reference sensors ensured data integrity (98.4% completeness).
Feature Engineering:
- Chromatic Features: UV textures were converted to HSV color space. Tiles (32x32 to 128x128 pixels) were analyzed for radiometric statistics, Gray Level Co-occurrence Matrix (GLCM) features, and Local Binary Patterns (LBP).
- Environmental Features: A novel Humidity Stress Factor (HSF) was derived using Principal Component Analysis (PCA). This composite feature integrates current relative humidity (RH), cumulative exposure to RH >85% over 24 hours, and lagged RH values (6, 12, and 24 hours), capturing the microclimate's memory effect.
Machine Learning Pipeline:
- Unsupervised Stage: Agglomerative hierarchical clustering (Ward linkage) in HSV space generated diagnostic priors (chromatic group maps) to identify latent groupings without annotated datasets.
- Supervised Stage: A Random Forest (RF) classifier was trained on expert-labeled tiles (48,830 pixels, 3,750 tiles) across five decay categories (biofilm, moisture darkening, efflorescence, pigment attenuation, neutral). The RF model incorporated radiometric, textural, and environmental features, including the HSF and the unsupervised clustering labels as categorical priors.
- Calibration and Validation: RF probability outputs were calibrated using Platt scaling. The framework underwent rigorous validation, including ablation studies, comparison with baseline models (Logistic Regression, SVM, XGBoost), and leave-one-site-out cross-validation to test spatial generalizability.
Decision Support:
- A Heritage Values Index (HVI) was developed via a Delphi process to quantify cultural significance.
- A Final Decision Score (FDS) was calculated as a weighted sum of the calibrated risk score (60%) and normalized HVI (40%), generating a priority intervention matrix to guide resource allocation.

Key Results

Classification Performance: The Random Forest classifier achieved a mean macro-F1 score of 0.87 (range 0.84–0.90) with overall accuracy exceeding 90%. The model outperformed baseline algorithms (XGBoost: 0.84, SVM: 0.81).
Feature Importance: The Humidity Stress Factor (HSF) was identified as the single most important predictor (importance score 0.264). An ablation study confirmed that removing the HSF reduced the macro-F1 score by 9.2% (from 0.87 to 0.79), validating the hypothesis that cumulative humidity stress, rather than instantaneous readings, drives deterioration.
Spatial Accuracy: Integrating environmental features and clustering priors improved the spatial Intersection-over-Union (IoU) by 17–51% compared to unsupervised clustering alone.
Predictive Capability: The framework demonstrated a technical lead time of 4.8 to 16.7 hours for high-risk events (conditional on correct detection), with a recall of 0.79. Expanding window cross-validation showed the model maintains predictive performance over 1–3 month horizons, though performance decays for 6-month forecasts.
Generalizability: Leave-one-site-out cross-validation yielded F1 scores between 0.79 and 0.82, indicating acceptable spatial transferability to unseen sites, provided local recalibration is applied.
Environmental Correlation: Mixed-effects modeling confirmed a significant positive association between cumulative hours of RH >85% and material deterioration ( $\beta = 0.18$ , $p < 0.001$ ).

Significance and Claims
The paper claims to provide a methodologically reproducible blueprint for predictive conservation in resource-constrained tropical contexts. Its primary contributions are:

Integration of Sensing and AI: Moving beyond post-hoc analysis, the study integrates environmental data directly as predictive features (via the HSF) to enable genuine early warning capabilities.
Hybrid Workflow: It demonstrates the efficacy of combining unsupervised clustering (to handle limited labeled data) with supervised Random Forest classification (for interpretability and robustness).
Contextual Calibration: The study empirically validates a humidity threshold of 85% for tropical settings, challenging the lower thresholds (75–80%) often cited in temperate-climate literature.
Decision Support: It offers a structured, auditable mechanism (FDS and Priority Matrix) that balances objective environmental risk with culturally defined heritage significance, addressing the challenge of resource allocation under uncertainty.

The authors explicitly state that the framework is not a universally pre-calibrated model but requires local recalibration of thresholds and parameters for transfer to other contexts. They acknowledge limitations regarding the subjectivity of semantic translation (mitigated by structured expert protocols), the technical nature of lead times (which do not guarantee operational response), and the need for future validation across broader geographic and climatic zones. The study concludes that such frameworks can support a transition from reactive repair to anticipatory management in heritage conservation.

The Digital Doctor for Old Buildings

The Big Discovery: A Head Start on Disaster

How They Decided What to Fix First

The Catch and The Future

The Bottom Line

More like this