⚡ electrical engineering

Experimental study on surveillance video-based indoor occupancy measurement with occupant-centric control

This paper presents an experimental study demonstrating that an LLM-enhanced vision-based occupancy measurement pipeline (YOLOv8+DeepSeek) significantly improves tracking accuracy and enables a 17.94% reduction in HVAC energy consumption through occupant-centric control in smart buildings.

Original authors: Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your office building is a giant, living organism. For years, it has been "sleeping" or "waking up" based on a rigid, old-fashioned alarm clock (a schedule). It turns the heat on at 8:00 AM and off at 6:00 PM, regardless of whether anyone is actually there. This wastes a ton of energy and sometimes leaves people shivering or sweating because the building doesn't know what's actually happening inside.

This paper is about teaching that building a new sense: sight.

Here is the story of how the researchers taught a building to "see" its occupants and use that vision to save energy and keep people comfortable, using a mix of old-school cameras and a brand-new "super-brain."

1. The Problem: The Building is Blind

Currently, buildings rely on "indirect clues" to know if people are inside. They might check if the Wi-Fi is busy or if the air smells like carbon dioxide. But these clues are like trying to guess how many people are in a room by listening to the hum of a refrigerator; it's often wrong.

The Risk: If the building thinks the room is empty when it's actually full, the AC turns off, and people get uncomfortable. If it thinks the room is full when it's empty, it wastes energy cooling air for ghosts.

2. The Solution: Giving the Building Eyes (and a Brain)

The researchers set up a simple experiment in a lab at Tsinghua University. They installed a standard security camera (like the ones you see in malls) and tried three different ways to count the people in the video feed:

Method A: The "Snapshot" Detective (Detection Only)
Imagine a security guard who takes a photo every second and counts the heads.
- The Flaw: If someone walks behind a pillar, the guard misses them. If a poster of a person is on the wall, the guard gets confused. The count jumps up and down wildly every second. It's too jittery for a building to trust.
Method B: The "Tracker" (Multi-Object Tracking)
This is like a guard who doesn't just count heads, but puts a little invisible name tag on every person and follows them as they move.
- The Improvement: Even if someone is briefly hidden, the guard remembers they are there. This makes the count much smoother and more stable. It's like switching from a shaky hand-held camera to a steady tripod.
Method C: The "Super-Brain" Refinement (LLM + Vision)
This is the star of the show. The researchers took the data from the camera and fed it into a Large Language Model (LLM)—a type of AI that is really good at reasoning and understanding context (like the "DeepSeek" model used here).
- The Analogy: Imagine the camera is a junior intern who is good at spotting things but gets confused by shadows or reflections. The LLM is the senior manager. The intern says, "I see 3 people, but wait, one of them looks like a reflection in the window, and the count just jumped from 0 to 5 in a split second—that's impossible!" The manager (LLM) says, "No, you're wrong. It's actually 2 people. Ignore the reflection."
- The LLM acts as a "truth filter," correcting the camera's mistakes by using logic and context.

3. The Result: A Smarter, Greener Building

The researchers didn't just stop at counting people. They connected this "Super-Brain" system to the building's HVAC (Heating, Ventilation, and Air Conditioning) system.

The Old Way: The building follows a schedule.
The New Way: The building uses the "Super-Brain" count to decide when to turn the heat/AC on or off.

The Magic Numbers:

The "Super-Brain" system (YOLOv8 + DeepSeek) was the most accurate, getting the count right about 88% of the time and identifying "occupied" vs. "empty" rooms with 93% accuracy.
Because it was so good at knowing exactly when the room was empty, it didn't waste energy cooling an empty room.
The Payoff: The building saved 17.94% of its energy just by using this smarter way of counting people. That's a massive amount of money and carbon emissions saved!

4. Why This Matters (The "So What?")

Think of this like a thermostat that finally learns your habits.

Without this tech: The building is like a stubborn parent who says, "It's 6 PM, time to go to bed!" even if you're still awake working.
With this tech: The building is like a thoughtful roommate who says, "Oh, you're still here? I'll keep the lights on. But now that you've left, I'll turn everything off."

The Big Takeaway

This paper proves that we don't need expensive, new sensors to make buildings smarter. We can use the cameras we already have, but we need to give them a "brain" (AI) to interpret what they see correctly. By combining computer vision (the eyes) with reasoning AI (the brain), we can make buildings that are not only energy-efficient but also more comfortable for the humans inside.

It's a small step for a camera, but a giant leap for saving the planet's energy.

1. Problem Statement

Accurate and temporally stable occupancy information is critical for Occupant-Centric Control (OCC) in smart buildings, particularly for optimizing Heating, Ventilation, and Air Conditioning (HVAC) systems to achieve Net Zero emissions. However, existing solutions face significant challenges:

Non-Vision Sensors: Methods using PIR, CO2, or WiFi often rely on indirect indicators, suffer from latency, and struggle with dynamic occupancy patterns.
Vision-Based Limitations: Traditional computer vision pipelines (detection-only or standard tracking) often produce unstable estimates due to occlusion, lighting changes, and identity switches.
Control Impact Gap: While improving detection accuracy is a common research goal, there is insufficient understanding of how specific occupancy measurement errors (e.g., false negatives) propagate to downstream HVAC control performance and energy savings.
Untapped Potential: The potential of Large Language Models (LLMs) and Vision-Language Models (VLMs) to refine noisy perceptual outputs through reasoning has not been systematically explored in building energy contexts.

2. Methodology

The study proposes a fully experimental framework comparing four distinct occupancy measurement pipelines using real surveillance video data from a research laboratory at Tsinghua University.

A. Experimental Setup

Dataset: 1,188 video clips (20,722 frames) collected over five non-consecutive days.
Ground Truth: Exhaustive manual frame-level annotations of occupant counts, serving as the reference for all evaluations.
Base Detector: All pipelines utilize YOLOv8 as the unified person detection backbone to ensure fair comparison.

B. The Four Pipelines Compared

Detection-Only: Frame-level counting based solely on YOLOv8 detections.
Tracking-Based (DeepSORT): YOLOv8 + DeepSORT for identity-consistent tracking.
Tracking-Based (ByteTrack): YOLOv8 + ByteTrack (a robust baseline associating high and low-confidence detections).
LLM-Enhanced Refinement: YOLOv8 detections are post-processed by an LLM.
- Mechanism: A lightweight screening step selects frames with ambiguity (e.g., low confidence, boundary counts).
- Models Tested:
  - Vision-Language (LLaVA): Processes image batches.
  - Text-Only (DeepSeek): Processes structured detector outputs (counts, confidence scores, temporal context) without raw images.
- Fusion Rule: The LLM output replaces the detector output only if the LLM's confidence exceeds the detector's confidence by a margin ( $\Delta = 0.15$ ).

C. Downstream Control Integration

The best-performing occupancy signals were integrated into a Model Predictive Control (MPC) framework within OpenStudio-EnergyPlus.

Control Logic: A supervisory MPC controller uses the occupancy estimate as an exogenous input to optimize heating/cooling setpoints over a 1-hour horizon.
Simulation: Annual simulations were run to evaluate energy consumption and thermal comfort (Predicted Percentage Dissatisfied - PPD) against a fixed schedule baseline.

3. Key Contributions

Systematic Benchmarking: A controlled comparison of detection-only, tracking-based, and LLM-refined pipelines using a real-world dataset with frame-level ground truth.
LLM Refinement Strategy: Introduction of a reasoning-based post-processing module that corrects false negatives and stabilizes counts without retraining the underlying detector.
Control-Centric Evaluation: Moving beyond simple accuracy metrics to quantify the impact of sensing quality on HVAC energy savings and thermal comfort in an MPC environment.
Performance Validation: Demonstration that the YOLOv8 + DeepSeek pipeline achieves the optimal balance of accuracy and stability, leading to significant energy reductions.

4. Key Results

A. Occupancy Measurement Performance

Tracking vs. Detection: Tracking-based methods (ByteTrack) improved temporal stability over detection-only but still suffered from identity switches and fragmentation.
LLM Superiority: The YOLOv8 + DeepSeek (Text-only LLM) pipeline outperformed all others:
- Accuracy: 0.8824 (vs. 0.8193 for best tracking baseline).
- F1-Score: 0.9320.
- Error Reduction: Lowest Mean Absolute Error (MAE: 0.8923) and Root Mean Square Error (RMSE: 1.3124).
- Critical Improvement: Significantly reduced False Negatives (missed occupancy), dropping from 1,503 (ByteTrack) to 1,073. This is crucial for OCC to avoid conditioning empty spaces or failing to condition occupied ones.

B. HVAC Control & Energy Impact

Integrating the occupancy signals into the MPC framework yielded the following results compared to a fixed schedule baseline:

Energy Savings:
- YOLOv8 + DeepSeek: Achieved the highest HVAC energy saving of 17.94% (Total consumption reduced from ~15,603 kWh to ~12,803 kWh).
- Other pipelines achieved savings between 7.09% and 15.67%.
Thermal Comfort (PPD):
- The DeepSeek pipeline resulted in the lowest Average PPD (8.59%), indicating superior thermal comfort.
- Pipelines with higher false-negative rates (e.g., DeepSORT) caused the MPC to apply setbacks during actual occupancy, leading to higher PPD (up to 13.34%).

5. Significance and Conclusion

This study establishes that occupancy sensing is a control-critical component of smart building operations, not merely a perception task.

Reasoning over Raw Perception: The research demonstrates that reasoning-based refinement (using LLMs to interpret structured detection data) is more effective than raw visual reasoning (VLMs) or standard tracking for fixed-camera indoor environments. It effectively corrects systematic errors that tracking algorithms miss.
Energy-Comfort Trade-off: High-quality occupancy data allows MPC controllers to safely apply energy-saving setbacks without compromising comfort. Conversely, noisy data leads to "false unoccupied" decisions, causing discomfort and negating energy savings.
Pathway to Net Zero: The proposed YOLOv8 + DeepSeek framework offers a practical, scalable methodology for achieving substantial HVAC energy reductions (nearly 18%) while maintaining or improving indoor environmental quality, directly supporting the 2050 Net Zero target.

Limitations & Future Work: The study was conducted in a single open-plan lab. Future work will explore multimodal fusion (integrating non-visual sensors) and testing in more complex, partitioned spaces to validate generalizability.