💻 computer science

SafeBuild-Bench: A Temporal-Robust Construction Safety Benchmark with Graph-Enhanced Data Mining

This paper introduces SafeBuild-Bench, a temporal-robust construction safety benchmark mined from over 100,000 industrial records using the novel GEMS graph-enhanced selection pipeline, which reveals that current multimodal large language models still struggle to achieve reliable safety understanding in realistic, variable site conditions.

Original authors: Yi Cui, Zilin Wang, Yijie Xu, Qianyi Cai, Huizai Yao, Shuai Jiang, Bingzhuo Zhong, Hui Xiong

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Cui, Zilin Wang, Yijie Xu, Qianyi Cai, Huizai Yao, Shuai Jiang, Bingzhuo Zhong, Hui Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to spot danger in a busy construction site. You might think the best way is to show it millions of photos of scaffolding, cranes, and workers. But here's the catch: most of those photos are boring, repetitive, and safe. They show a perfectly built wall or a worker wearing a helmet in broad daylight. If you only train the robot on these "easy" pictures, it might become a master at recognizing safe scenes but a total failure when it sees something weird, like a loose ladder in the rain or a missing guardrail on a foggy morning. This is the problem of "long-tail" hazards—the rare, tricky, and dangerous situations that actually cause accidents, but are hidden in a sea of boring data.

To solve this, scientists are building special tests called "benchmarks." Think of a benchmark like a final exam for AI. Instead of just asking, "Is this a chair?", a safety benchmark asks, "Is this specific scaffold setup dangerous right now, and why?" The challenge is that construction sites change every day. The weather shifts, the building grows, and the lighting changes. If an AI only learns from a static set of photos, it might get confused when the real world doesn't look exactly like its textbook. This paper tackles the question of how to build a better, more realistic exam for AI that can handle these changes and find the rare, dangerous moments hidden in massive piles of data.

The "Needle in a Haystack" Problem

The researchers behind this paper, SafeBuild-Bench, realized that construction safety archives are like a giant, messy library where 99% of the books are identical copies of "Safe Day 1," and only a few pages contain the actual instructions on how to survive "Dangerous Day 42." If you try to read every single page to find the danger, you'll waste years. If you just pick random pages, you'll likely miss the danger entirely.

To fix this, they created a new, super-charged test called SafeBuild-Bench. It's not just a list of pictures; it's a curated collection of 3,314 specific "mission scenarios" pulled from over 100,000 raw inspection records. These records come from real construction sites in China, collected over five months, covering everything from scaffolding to tower cranes. The magic isn't just in the pictures, but in the metadata: every single image keeps its original date and location tag. This allows the researchers to see if an AI is actually smart, or if it's just memorizing that "July looks like July" and "Site A looks like Site A."

The "Smart Filter" (GEMS)

How do you find the dangerous needles in that haystack without reading every single piece of hay? The authors invented a tool called GEMS (Graph-Enhanced Multimodal Selection). Imagine you are a librarian trying to pick the most interesting books for a reading club. You don't just grab random books. Instead, you have a robot assistant that does two things:

The "Confused" Detector: It asks a smart AI, "Hey, what's in this picture?" If the AI stammers, hesitates, or gets confused, the robot flags it. Confusion often means the picture is tricky or rare—exactly what you want to study.
The "Similarity" Map: It draws a map connecting pictures that look alike. If it picks a picture of a "wobbly ladder," it won't pick ten other pictures of "wobbly ladders" that look exactly the same. It uses a graph (a web of connections) to ensure it picks a diverse set of different kinds of trouble.

This combination creates a "Goldilocks" subset: not too easy, not too repetitive, but just the right amount of hard and rare.

The Big Test: Are Robots Ready?

The authors took this new benchmark and tested it against a bunch of the world's most advanced AI models (both the famous, expensive ones and the open-source ones). The results were a bit of a reality check.

Even the best AI models scored around 60 out of 100 overall. That's a passing grade in high school, but for a robot responsible for human safety? It's a fail.

The Good News: Some models were surprisingly good at describing what they saw. For example, one model could spot a hazard and say, "There's a missing guardrail," with high accuracy.
The Bad News: They were terrible at identifying the specific type of danger when given multiple-choice options. They often confused a "safe" scene with a "dangerous" one, or missed subtle hazards like a loose electrical box.
The Size Matters: Bigger models generally did better, but even the giants struggled with the tricky, long-tail hazards.

The "Time Travel" Surprise

One of the most interesting findings was how the models performed over time. Because the benchmark kept track of dates, the researchers could see how the AI did in July versus November. The scores weren't stable; they jumped around. A model might do great in July but stumble in October. This proves that the AI isn't truly "learning" safety rules; it's often just guessing based on the background or the time of year. If you tested it only on July data, you'd think it was a safety genius. Test it in November, and it's a disaster.

What This Means for the Future

The paper doesn't claim to have solved construction safety. Instead, it provides a much better ruler to measure how far we have to go. It suggests that to make AI safe for real-world use, we can't just throw more data at it. We need to be smarter about which data we use. By using tools like GEMS, we can strip away the boring, repetitive stuff and focus on the rare, dangerous moments that actually matter.

The authors also released all their code, the dataset, and the testing scripts for free. This means other scientists can now use this same "exam" to test their own robots, ensuring that everyone is measuring safety in the same, realistic way. It's a step toward a future where AI doesn't just recognize a worker, but truly understands when that worker is in danger.

Technical Summary: SafeBuild-Bench

Problem Statement

Construction safety monitoring relies on Multimodal Large Language Models (MLLMs) to interpret inspection archives. However, current models face significant reliability challenges in real-world deployment due to three primary factors:

Data Redundancy and Long-Tail Distribution: Industrial inspection archives are vast but "information-poor," dominated by repetitive, low-risk images while rare, high-impact hazards form a sparse long tail.
Temporal and Site Distribution Shift: Construction sites evolve due to weather, progress, lighting, and camera maintenance, creating temporal distribution shifts that static evaluation protocols often ignore.
Evaluation Limitations: Existing benchmarks typically rely on aggregate scores under i.i.d. assumptions, failing to stress-test models on time-specific or site-specific robustness, leading to overestimated deployment performance.

The paper addresses the practical question of how to build a construction safety benchmark that enables temporal and site-stratified evaluation, focuses on long-tail hazards rather than redundant data, and remains feasible to construct from large-scale, redundant data streams.

Methodology

1. Data Acquisition and Preprocessing

The authors curated a raw dataset of over 100,000 image-text pairs collected from more than 50 construction locations across five months (July–November 2025). The data covers diverse scenes (scaffolding, foundation pits, tower cranes) and environmental conditions (night, rain, fog). A strict privacy pipeline was implemented to anonymize faces and license plates using automated detection and manual verification.

2. GEMS: Graph-Enhanced Multimodal Selection

To distill an "informational core" from the redundant raw stream without requiring immediate expert verification of every sample, the authors developed GEMS (Graph-Enhanced Multimodal Selection). This automated pipeline consists of three stages:

Multimodal Representation Learning: Image-text pairs are mapped into a unified dense vector space using the Qwen3-VL-Embedding encoder to obtain joint representations ( $z_i$ ).
Epistemic Uncertainty Quantification: A proxy model (Qwen2.5-VL-3B-Instruct) generates responses to hazard-analysis prompts. The uncertainty score ( $u_i$ ) is calculated as the length-normalized negative log-likelihood (NLL) of the generated response. High NLL indicates candidate samples where the model is unsure, potentially signaling informative or difficult cases.
Graph-Theoretic Submodular Selection: A sparse $k$ -Nearest Neighbor ( $k$ -NN) graph is constructed based on cosine similarity of embeddings. The selection objective maximizes the total influence of a subset $S$ , defined as the sum of minimum influence (bounded by a saturation threshold $\tau$ ) received by all nodes. This balances informativeness (high uncertainty) with diversity (reducing near-duplicates).
Optimization: The Cost-Effective Lazy Forward (CELF) algorithm is employed to efficiently solve the submodular maximization problem, reducing computational complexity compared to standard greedy selection.

3. SafeBuild-Bench Construction

Using GEMS, the authors selected over 3,000 images to form the final benchmark, resulting in 3,314 task instances.

Metadata: Every instance retains original collection timestamps and site identifiers, enabling stratified analysis by month and location.
Expert Verification: Selected samples were reviewed by construction safety experts to assign primary hazard categories and reference descriptions based on established standards. Ambiguous cases were resolved through expert consensus.
Task Design:
- Hazard Identification (MCQ): 2,200 instances where models select the correct hazard from three distractors sampled from expert-defined "confusion groups" (visually or procedurally similar hazards).
- Hazard Description (Free-form): 1,114 instances requiring precise descriptions of hazards or confirmation of safety. Evaluation uses a standardized LLM-as-a-judge scheme (GPT-4o) with a fixed rubric, validated against human audits.

Key Contributions

SafeBuild-Bench: An expert-verified benchmark mined from 100k+ raw records, containing 3,314 task instances with long-tail hazards, temporal/site metadata, and annotations for both hazard identification and description.
Metadata-Driven Evaluation Design: A protocol enabling stratified temporal and site-level robustness analysis, accompanied by executable evaluation scripts and a scalable LLM-as-a-judge scheme.
GEMS Pipeline: An automated selection framework that combines epistemic uncertainty with graph-based diversity to identify informative candidates for expert verification. The authors validate GEMS on public instruction-tuning datasets, showing it can preserve robustness-oriented performance with significantly reduced data budgets.

Experimental Results

GEMS Validation (Proxy Study)

The authors validated GEMS on public instruction-tuning datasets (LLaVA-Instruct-150K and LLaVA-Mixed-665K) using LLaVA-v1.5-7B as the base model.

Data Efficiency: Fine-tuning on only 1% of GEMS-selected data (approx. 1.5k–6.6k samples) matched or exceeded the performance of full-data training (158k–665k samples) on robustness-oriented benchmarks (VizWiz, MMMU, POPE).
Redundancy Reduction: GEMS-selected subsets demonstrated lower within-subset redundancy and better category balance compared to random sampling, confirming its ability to filter noise while retaining hard cases.

SafeBuild-Bench Evaluation

The benchmark was evaluated on a diverse set of proprietary and open-source MLLMs (e.g., Gemini-3, Qwen3-VL, Kimi-K2.5, GPT-4o) in a zero-shot setting.

Overall Performance: Current MLLMs remain far from reliable, with the best overall score (Kimi-K2.5) reaching only 60.7.
Task Disparity: Models generally performed better on Hazard Detection (identifying presence/absence) than on Description Quality. For example, Qwen3-VL-Plus achieved a 79.9% detection rate but lagged in identification accuracy.
Temporal Variance: Performance varied significantly across month slices (July–November). For instance, Claude-Sonnet-4.5 accuracy ranged from 46.4% to 54.6% across months, demonstrating that aggregate scores mask temporal instability.
Error Analysis:
- Identification: Models struggled with categories requiring fine-grained discrimination (e.g., specific scaffold types) or implicit safety norms. A bias toward over-predicting hazards was observed in "No Hazard" cases.
- Description: Major failures were primarily driven by missed hazard detection (HDR=0) rather than linguistic deficiencies. Smaller models exhibited higher failure rates (44.4% for Qwen3-VL-4B) compared to larger models (27.3% for Kimi-K2.5).

Significance and Claims

The paper claims that SafeBuild-Bench provides a necessary shift from static, aggregate evaluation to temporal-robust, stratified analysis that better reflects real-world deployment risks in construction safety.

Feasibility of Data-Centric Mining: The study demonstrates that graph-enhanced data mining (GEMS) can effectively distill high-value, non-redundant subsets from massive industrial streams, making the construction of expert-verified benchmarks feasible.
Realistic Baselines: The results indicate that even state-of-the-art MLLMs are not yet reliable for construction safety understanding, highlighting a gap between general visual capabilities and domain-specific safety reasoning.
Operational Relevance: By retaining temporal and site metadata, the benchmark allows stakeholders to diagnose specific deployment risks (e.g., performance drops in specific months or sites) rather than relying on a single leaderboard score.

The authors remain modest regarding limitations, noting that the benchmark is grounded in Chinese construction regulations, requires sustained expert effort for scaling, and relies on proxy models for initial selection. They position the work as a foundational step toward more robust, safety-critical multimodal AI evaluation.