Technical Summary: SafeSceneReason

Problem Statement

Industrial safety understanding extends beyond the visual detection of workers, equipment, and personal protective equipment (PPE). Effective safety management requires models to assess compliance, identify hazardous interactions, explain accident mechanisms, and recommend preventive actions. Current multimodal large language models (MLLMs) and existing safety datasets primarily focus on isolated visual perception or single-cue violation recognition (e.g., detecting a missing hard hat). These approaches fail to support evidence-grounded reasoning that connects full-scene observations with safety rules, accident knowledge, and causal analysis. Consequently, strong general visual understanding does not guarantee reliable industrial-safety reasoning, particularly for tasks requiring multi-evidence synthesis, comparative analysis, and intervention-oriented decision-making.

Methodology

The authors introduce SafeSceneReason, a multimodal benchmark and companion training corpus designed to bridge the gap between workplace scene observation and accident investigation knowledge. The framework integrates two complementary data-construction pipelines:

1. Scene-Centric Pipeline (Proactive Inspection)

• Source: Annotated workplace images from SH17 and DsLMF+ datasets.
• Process:
  • Annotations (workers, PPE, machinery, tools) are normalized into executable safety scene graphs. Nodes represent entities; edges encode spatial and functional relations (e.g., "wearing," "operating," "inside hazard zone").
  • Person-PPE associations are established via bounding-box containment and spatial distance to evaluate safety at the individual level.
  • Deterministic Answer Generation: Declarative safety rules and executable programs generate reference answers. A Vision-Language Model (VLM) may supplement restricted attributes, but the final answer is derived strictly from program execution over the scene graph, ensuring reproducibility.
  • Output: 110,581 verified question-answer (QA) pairs covering perception, spatial reasoning, compliance assessment, and causal/counterfactual reasoning.

2. Report-Centric Pipeline (Retrospective Learning)

• Source: Over 80,000 publicly available accident reports from OSHA, NIOSH FACE, and the U.S. Chemical Safety Board (CSB).
• Process:
  • Reports are filtered for informative figures and contextual evidence.
  • Evidence Extraction: Figures, captions, and text are parsed. Evidence items are explicitly labeled as either directly observable from the image, visually supported but requiring text clarification, or available only from the report text. This prevents narrative facts from being misidentified as visual observations.
  • Reasoning Path Composition: Validated evidence graphs are transformed into multi-step reasoning paths (comparison, quantitative interpretation, causal analysis).
  • Output: 13,114 refined QA pairs with explicit reasoning paths, provenance, and verification records.

Evaluation Protocol

The benchmark evaluates eight representative vision-language models (including GPT-5 variants, Gemini, Claude, Kimi, and Qwen) across eight reasoning categories: quantitative, spatial/structural, causal, comparative, evidence synthesis, hazard identification, compliance assessment, and mitigation/decision making. The study includes:

• Zero-shot evaluation of base and proprietary models.
• Safety-domain fine-tuning of Qwen3.5-9B using LoRA with both Answer-Only and Chain-of-Thought (CoT) supervision.
• Strict parsing: Responses are scored only if the extracted option matches the reference exactly.

Key Contributions

1. SafeSceneReason Benchmark: A multimodal dataset connecting real-world workplace scenes with official accident investigation materials, covering the progression from perception to preventive decision-making.
2. Dual-Source Data Synthesis: Two auditable pipelines that separate factual inference from language generation. The scene-centric pipeline uses executable graphs for deterministic answers, while the report-centric pipeline constructs evidence-grounded multi-step questions from accident figures and text.
3. Provenance and Verification: All samples retain evidence references, intermediate reasoning representations, and verification results, enabling traceable evaluation and model training.
4. Comprehensive Evaluation: Systematic assessment of proprietary and open-source models reveals significant performance gaps in compositional, causal, and intervention-oriented reasoning.

Experimental Results

• Performance Hierarchy: There is a pronounced gap between general visual understanding and safety reasoning.
  • Top Performers: CoT-SFT Qwen3.5-9B achieved the highest micro-accuracy (89.0%), slightly outperforming GPT-5.6-Sol (88.7%). GPT-5.5 achieved the best macro-accuracy (89.5%).
  • Base Model Limitations: Untuned open-source models (Qwen3.5-9B and Qwen3.6-27B) scored significantly lower (37.4% and 40.9% micro-accuracy, respectively), demonstrating that general multimodal capability alone is insufficient.
• Impact of Supervision: Safety-domain fine-tuning with CoT supervision yielded massive gains (+51.6 points micro-accuracy) over the base model, particularly in comparative, quantitative, and compliance tasks. However, even fine-tuned models lagged behind top proprietary models in evidence synthesis and hazard identification.
• Domain Variability: Performance varied significantly by domain. Models performed well in agriculture and construction but struggled with chemical toxicology, biological hazards, and cancer-related safety, indicating that domain-specific terminology and causal knowledge are critical bottlenecks.
• Failure Modes:
  • Local vs. Global: Models often succeed at local object recognition but fail to compose multiple observations into a coherent safety conclusion.
  • Calibration: Base models exhibited response-format shortcuts (e.g., bias toward specific answer positions), which CoT training mitigated but did not fully eliminate.

Significance and Claims

The paper claims that SafeSceneReason provides a focused, representative subset of industrial safety tasks rather than a complete lifecycle solution. Its primary significance lies in demonstrating that strong general visual understanding does not yet guarantee reliable industrial-safety reasoning.

The authors argue that current models lack the ability to:

1. Ground observations in specific safety rules and accident mechanisms.
2. Synthesize evidence across multiple figures or scene elements to form causal explanations.
3. Select appropriate preventive interventions based on complex, multi-evidence scenarios.

By providing a benchmark with traceable evidence and executable reasoning paths, SafeSceneReason aims to support the development of multimodal systems whose safety judgments are not only accurate but also grounded, auditable, and appropriately cautious. The authors emphasize that high aggregate accuracy is insufficient for safety-critical applications, highlighting the need for specialized domain knowledge and robust evidence integration.