💻 computer science

Ground-Truth-Aware Stress Testing of a Closed-Loop Smart-Building Digital Twin Under Sensor Drift and Missing Data

This paper introduces a ground-truth-aware simulation framework to demonstrate that while sensor drift and missing data can increase decision mismatches between practical controllers and an oracle in a closed-loop smart-building digital twin, these sensing errors do not necessarily lead to material degradation in aggregate energy, comfort, or CO2 outcomes.

Original authors: Lyes Saad Saoud

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lyes Saad Saoud

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern built environment, a new kind of intelligence is taking root. It is not a single machine, but a living digital mirror of a physical building, known as a digital twin. This virtual model does more than just display data; it simulates how a building breathes, heats, and cools in real time, allowing managers to test decisions before they are made in the real world. For these systems to work, they rely on a constant stream of information from sensors—devices that measure temperature, air quality, and occupancy. However, just like any physical instrument, these sensors are imperfect. They can drift out of calibration, lose power, or simply stop sending data. The critical question for engineers is not just whether a sensor is broken, but whether that brokenness actually changes how the building behaves. If a sensor lies about the air quality, does the building's computer make a mistake that wastes energy or makes people uncomfortable, or does the system absorb the error without consequence?

A recent study by Lyes Saad Saoud tackles this question by building a fully synthetic, ground-truth-aware simulation of a smart building. The researchers created a virtual environment containing twenty distinct zones, or rooms, and ran a complex simulation over thirty days. In this digital world, they could see the "true" state of the air at every moment, a secret known only to the researchers. They then introduced a layer of corruption to mimic real-world problems: they made the sensors drift, added random noise, and forced them to go silent for periods of time. The goal was to watch how different control strategies reacted to these lies. One strategy, called the "oracle," had direct access to the true state of the building and made perfect decisions. The other strategies had to rely on the corrupted sensor data, just as a real building would. The researchers wanted to see if the gap between the sensor's lie and the truth would cause the building to make a different choice, and if that different choice would actually change the physical outcome.

The results revealed a surprising and counterintuitive truth: a sensor can tell a lie, and the computer can make a different decision, without the building suffering any real harm. Under normal conditions, a controller using raw, unfiltered sensor data disagreed with the perfect oracle on about 2.59 percent of the decision steps. In other words, the computer chose to turn the ventilation on or off at a slightly different time than it would have if it knew the truth. Yet, when the researchers measured the actual physical results—the amount of carbon dioxide in the air, the energy used, and the comfort level of the occupants—they found almost no difference. The gap in carbon dioxide levels was so small it was nearly invisible, and the energy and comfort scores remained effectively unchanged. Even when the researchers increased the sensor drift to eight times its normal severity, causing the decision disagreement to jump to 7 percent, the physical outcomes remained stable. The building's physical dynamics acted as a buffer, smoothing out the small, momentary errors in decision-making.

This finding challenges a common assumption in the field: that more complex data processing is always better. The study tested a more sophisticated controller that tried to filter out bad data using a rolling average, a method that looks at the last few readings to smooth out the noise. The researchers expected this "fault-aware" approach to be more robust. Instead, it performed worse in terms of agreement with the perfect oracle. The added filtering introduced a delay, causing the controller to hesitate at the exact moments it needed to act, which led to more frequent disagreements with the ideal strategy. While the physical outcomes for this complex controller were still close to the oracle, the extra processing did not improve the result; it only made the system more likely to make a different choice than the perfect one. The study also showed that the ranking of which controller was "best" depended entirely on what the building manager valued most. If the priority was air quality, one controller looked best; if the priority was saving energy, a different one looked best. There was no single winner, only a trade-off that changed based on the goal.

The researchers were careful to note that these results come from a synthetic model, not a real building. The numbers and behaviors were generated by a computer program designed to test the logic of the system, not to predict the exact performance of a specific office tower. Because the model was not calibrated to real-world data, the study cannot claim that real buildings will behave exactly this way. However, the experiment successfully demonstrated a crucial distinction: decision error and outcome error are not the same thing. A sensor can be corrupted, and a controller can make a different choice, without the physical world noticing. This suggests that the industry may be over-engineering its solutions, spending effort to fix sensor data that does not actually need fixing to achieve a good result. The study concludes that to truly understand the health of a smart building, engineers must look at three layers at once: the quality of the sensor data, the decisions the computer makes, and the final physical outcome, rather than assuming that a perfect signal is required for a perfect building.

Technical Summary: Ground-Truth-Aware Stress Testing of a Closed-Loop Smart-Building Digital Twin

1. Problem Statement

While digital twins are increasingly deployed for smart building monitoring and control, existing evaluations often focus on state-estimation accuracy or fault detection rates. These metrics fail to address the critical downstream question for closed-loop operations: to what extent do sensing errors (drift, noise, missing data) materially alter control decisions and physical building outcomes?

A corrupted sensor may cause a controller to deviate from an ideal "oracle" strategy, yet the aggregate physical outcome (e.g., CO2 levels, energy consumption, comfort) may remain nearly unchanged. Conversely, small errors near control thresholds can trigger disproportionate consequences. Current literature lacks a framework that explicitly separates sensor error, decision disagreement, and physical outcome degradation to determine if sensor corruption is operationally consequential or benign.

2. Methodology

The paper introduces a ground-truth-aware, fully synthetic closed-loop simulation framework designed to isolate the propagation of sensing errors. The architecture consists of three distinct layers:

Latent Physical State: A physics-based simulator generates the true state of a 20-zone building (5 zones × 4 buildings) over 30 days at 15-minute intervals. It models occupancy-driven CO2 dynamics using a first-order mass-balance relaxation with a 45-minute time constant, alongside energy and comfort models.
Sensor Layer: This layer corrupts the latent state to produce observations. It introduces additive random-walk drift, measurement noise, and missing data (NaN). Crucially, missing values are not imputed as zero but handled explicitly.
Control Layer: Policies receive the corrupted observations with a one-step delay (decisions at time $t$ use data up to $t-1$ ). An "Oracle" controller has direct access to the latent ground-truth state, serving as the reference baseline.

Evaluation Framework:

Paired Monte Carlo Simulation: Using Common Random Numbers (CRN), all policies are evaluated under identical exogenous conditions (occupancy, temperature, etc.) to isolate the effect of control decisions.
Stress Scenarios: Experiments include nominal sensing, drift sweeps (0× to 8× nominal), missingness sweeps (0% to 20%), a combined drift-missingness grid, and an unseen operating distribution shift.
Metrics:
- Decision Level: Decision Mismatch Rate (DMR), False Intervention Rate (FIR), and Missed Intervention Rate (MIR) compared against the Oracle.
- Outcome Level: Gaps in CO2 exceedance, energy consumption, and comfort scores.
- Regret: Ground-Truth Regret (GTR) calculated under varying utility weights (air quality, energy, comfort).

Policies Evaluated:

P0 (No Control): Baseline.
P1 (Oracle): Ideal reference.
P2 (Raw Sensor): Uses the latest valid observation; holds previous action on missing data.
P3 (Filtered): Uses a 3-sample rolling median.
P4 (Fault-Aware): Combines rolling median, jump/range rejection, and occupancy-based fallback.

3. Key Contributions

Architecture: A ground-truth-aware testbed that enforces a strict separation between latent state, sensor corruption, and control decisions, including a one-step information delay.
Evaluation Framework: A paired Monte Carlo methodology using CRN to compare oracle-relative performance across multiple stress conditions.
Multi-Layer Metrics: The joint reporting of decision-level disagreement (DMR, FIR, MIR) and physical outcome gaps, highlighting that these layers do not scale linearly.
Empirical Findings:
- Sensor errors can measurably change individual control actions without producing materially different aggregate physical outcomes.
- Added preprocessing complexity (e.g., rolling medians) can worsen decision agreement with the oracle without improving physical outcomes.
Methodological Distinction: The paper establishes that "decision disagreement" is not synonymous with "outcome degradation."

4. Key Results

Nominal Performance: Under nominal sensing, the raw-sensor controller disagreed with the Oracle on 2.59% of decision steps. However, the resulting CO2 exceedance gap was negligible ( $-1.25 \times 10^{-4}$ ) with a 95% Monte Carlo interval of $[-4.65 \times 10^{-3}, 4.69 \times 10^{-3}]$ . Energy and comfort gaps were similarly insignificant.
Drift Sensitivity: As drift severity increased to 8× nominal, the raw-sensor decision mismatch rose to 7.00%. Despite this increase in decision disagreement, the physical outcome gaps (CO2, energy, comfort) remained small and within predeclared material-divergence thresholds.
Filtering Impact: The "Filtered" and "Fault-Aware" policies exhibited significantly higher decision mismatch (DMR $\approx$ 6.31% nominally) compared to the raw sensor, primarily due to the lag introduced by the rolling median. This increased disagreement did not translate to better physical outcomes; in fact, the filtered policies showed slightly worse comfort scores than the raw sensor.
Combined Stress: Across a 4×4 grid of drift and missingness (48 combinations), none of the sensor-driven policies crossed the predeclared thresholds for material divergence (DMR $\ge$ 0.10, $|\Delta CO_2| \ge$ 0.02, or $\Delta Comfort \ge$ 1.0).
Objective Dependence: Ground-Truth Regret rankings were highly sensitive to utility weights. No single policy was universally "best"; rankings shifted based on whether air quality, energy, or comfort was prioritized.

5. Significance and Claims

The paper explicitly frames its contribution as methodological rather than a claim of real-world building performance. The simulation is synthetic and uncalibrated to any specific physical facility.

Core Insight: The primary finding is the decoupling of decision disagreement from outcome degradation. In the evaluated threshold-control dynamics, sensor corruption alters control actions frequently, yet the physical system's inertia and the nature of the control logic prevent these errors from cascading into significant aggregate performance losses.
Implication for Verification: Digital twin robustness evaluation must report observation error, decision disagreement, and outcome degradation as three distinct layers. Relying solely on measurement accuracy or aggregate outcomes can lead to misleading conclusions about system robustness.
Caution on Preprocessing: The study provides a "negative result" suggesting that more complex sensor processing (like rolling medians) is not automatically more robust. In threshold-based control, such filters can introduce phase delays that increase decision mismatch without benefit.

The author concludes that while the framework successfully verifies the behavior of controllers under stress within the declared model, real-world validation requires parameter identification and calibration against measured building data, which is beyond the scope of this synthetic study.