Multi-modal Rail Crossing Safety Analysis

Technical Summary: Multi-modal Rail Crossing Safety Analysis

Problem Statement
Highway–rail grade crossings represent a critical safety challenge in the United States, accounting for over 2,000 collisions and 200 fatalities annually. Current risk assessment methods, such as the Federal Railroad Administration's (FRA) Accident Prediction and Severity (APS) model, rely heavily on structured statistical data (e.g., train frequency, traffic volume). However, these models often fail to capture visual and context-dependent factors observable by highway users, such as sightlines, visual obstructions, and the visibility of warning devices. Furthermore, manual inspection is difficult to scale due to the sheer number of crossings and limited expert resources. The authors posit that integrating street-level visual evidence with historical accident records could provide a more robust, scalable method for safety assessment.

Methodology
The authors propose a proof-of-concept multimodal pipeline that combines street-level crossing imagery with FRA Form 57 accident records. The system utilizes Vision-Language Models (VLMs), specifically Google's Gemma 4 (E4B variant), to perform two complementary tasks:

1. Score-Based Analysis (Risk Scoring):

   • Input: The model receives a sequence of street-level images (captured within 20 meters of the crossing) and associated Form 57 incident records.
   • Tasks:
     • Binary Classification: Determining if a crossing is HIGH-RISK (annual predicted accidents > 0.09) or LOW-RISK.
     • Continuous Regression: Estimating the annual average predicted accident score.
   • Training Strategy: To handle the complexity of multimodal inputs and constrained output formats, the authors employ Low-Rank Adaptation (LoRA) for fine-tuning the base VLM.
   • Routed Regression: To address the highly skewed distribution of risk scores (where most crossings are low-risk), the authors introduce a routed pipeline. A binary classifier first routes crossings to either a "LOW-RISK" or "HIGH-RISK" specialized regression model. This is compared against an "oracle" router (using ground-truth labels) to establish an upper bound.

2. Description-Based Analysis (Visual Risk Inspection):

   • Framework: This task is grounded in accident theory using a "bowtie diagram" framework, which categorizes safety elements into Threats, Barriers (Preventive/Mitigative), and Escalation Factors.
   • Process: The VLM is prompted to identify predefined structural threats (e.g., sharp angles, multiple lanes), safety infrastructure (e.g., gate arms, crossbucks), and escalation factors (e.g., sunlight blinding, vegetation obscuring signs).
   • Augmentation: To evaluate robustness, images are augmented to simulate varying weather and illumination conditions (rain, strong sunlight, nighttime) using generative AI, allowing the model to assess how environmental factors impact the visibility of safety features.

Key Contributions

• Multimodal Pipeline: The proposal of a novel pipeline that fuses unstructured street-level imagery with structured FRA Form 57 incident records for railway crossing safety assessment.
• VLM-Based Risk Estimation: A study demonstrating how VLMs can estimate FRA-style safety scores and classify risk levels from multimodal data, moving beyond traditional variable-only models.
• Qualitative and Quantitative Analysis: An evaluation showing that VLMs can identify visual safety factors from the highway user's perspective (e.g., occluded signals, poor sightlines) that are not captured by standard scoring variables.
• Routed Prediction Strategy: The implementation of a routed regression approach to better handle the long-tailed distribution of accident scores, separating the problem into specialized sub-tasks.

Results

• Binary Classification: The fine-tuned VLM achieved a macro F1 score of 0.757 and a balanced accuracy of 0.767 for distinguishing between HIGH-RISK and LOW-RISK crossings. This significantly outperformed prompting-only baselines, particularly in identifying HIGH-RISK cases (Recall improved from 0.175 to 0.635 with fine-tuning).
• Score Prediction: For continuous score estimation, the routed fine-tuned pipeline achieved a Root Mean Square Error (RMSE) of 0.078 and a Pearson correlation of 0.492. While the single fine-tuned model had a lower correlation (0.245), the routed approach improved performance, though the authors note the "oracle" router (0.756 correlation) suggests the routing mechanism itself remains a bottleneck.
• Visual Inspection: Qualitative results indicated the model could successfully identify predefined threats and warning signs with high accuracy (e.g., 100% detection for sharp-angled crossings and traffic bottlenecks). The model also demonstrated the ability to assess escalation factors, correctly identifying sunlight as a critical risk factor in augmented night/day scenarios.

Significance and Claims
The paper claims that VLMs offer a promising avenue for scalable, automated railway crossing safety analysis. The authors assert that their system aligns with expert opinion and FRA scoring standards while capturing visual evidence that traditional statistical models miss. They emphasize that fine-tuning is essential for adapting compact VLMs to this specific domain and that a multi-module (routed) approach is necessary to handle the skewed nature of accident data. The work is presented as a proof-of-concept that validates the potential of combining visual and structured data to support systematic safety assessments, though the authors acknowledge limitations regarding the routing bottleneck and the need for future work on dynamic scenarios like video analysis.