Extracting and Analyzing Rail Crossing Behavior Signatures from Videos using Tensor Methods

This paper proposes a multi-view tensor decomposition framework that analyzes TimeSformer embeddings of railway crossing videos to identify latent behavioral signatures, revealing that location is a stronger determinant of driver behavior patterns than time of day and enabling scalable, targeted safety interventions.

The Gist

Imagine a busy railway crossing as a stage where a dramatic play happens every time a train approaches. The actors are the drivers, and the script changes depending on where the stage is located and what time of day the show is running.

For a long time, safety experts watched these plays one by one, trying to figure out why drivers sometimes ignored the red lights and gates. They looked at each crossing in isolation, like watching a single movie without ever comparing it to the others. This made it hard to spot the big picture: Do drivers at Crossing A act differently than drivers at Crossing B? Does the time of day matter more than the location?

This paper introduces a clever new way to watch all these movies at once using a "mathematical microscope" called Tensor Decomposition. Here is how it works, broken down into simple steps:

1. Breaking the Movie into Three Acts

The researchers didn't just watch the whole video. They realized that a crossing event has three distinct "acts," just like a play:

• Act 1: The Approach (Warning): The lights start flashing, and the gates start coming down. This is when drivers first see the danger.
• Act 2: The Wait: The gates are down, and the train is rumbling through. Drivers are stuck waiting.
• Act 3: The Clearance: The train passes, the gates go up, and traffic flows again.

They took videos from 31 different crossing events and used a smart AI (called TimeSformer) to turn each "act" into a unique digital fingerprint (an embedding).

2. The "Similarity Sandwich" (The Tensor)

Now, imagine you have a stack of three giant sheets of paper.

• Sheet 1 shows how similar the "Approach" acts were across all 31 videos.
• Sheet 2 shows how similar the "Waiting" acts were.
• Sheet 3 shows how similar the "Clearance" acts were.

The researchers stacked these three sheets together to form a 3D block of data (a Tensor). Think of this like a multi-layered sandwich where every slice tells a different part of the story, but they are all connected.

3. Finding the Hidden Patterns (The Magic Trick)

This is where the "Tensor Decomposition" comes in. It's like taking that complex sandwich and asking a super-smart algorithm to break it down into its simplest, most fundamental ingredients.

The algorithm asked: "If I look at all these videos, what are the core 'behavioral recipes' that make them tick?"

It found four main behavioral recipes (called components):

• Recipe A: Driven mostly by how drivers react when the lights first flash (The Approach).
• Recipe B: Driven by what happens while they are waiting for the train.
• Recipe C & D: A mix of everything, but with different flavors.

4. The Big Discovery: Location vs. Time

When the researchers looked at which videos fit which "recipe," they found something surprising:

• The "Time of Day" Myth: They thought maybe rush hour drivers act differently than night drivers. But when they colored the data by time (morning, noon, night), the colors were all mixed up like a bowl of fruit salad. Time didn't seem to matter much.
• The "Location" Reality: When they colored the data by where the crossing was, the colors snapped into neat, separate piles. It was like sorting Legos by color.

The Analogy: Imagine a group of people eating lunch.

• If you sort them by what time they ate (12:00 PM vs. 1:00 PM), they look random.
• But if you sort them by which restaurant they are in, they all look different. The "restaurant" (the crossing location) dictates their behavior more than the clock does.

5. Why This Matters

The study found that how drivers react when the lights first flash (The Approach) is the most important part of the story. It's the moment that best predicts how a driver will behave.

The Takeaway for Safety:
Instead of trying to fix every single crossing individually or worrying about the time of day, safety officials can now group crossings together based on their "behavioral DNA."

• If Crossing A and Crossing B both have the same "Approach-heavy" behavior, they can get the same safety upgrade (like better flashing lights).
• If Crossing C is totally unique (like the "NW 12th Street" crossing in the study), it gets a special, custom investigation.

In a Nutshell

This paper is like giving safety experts a super-powered pair of glasses. Instead of squinting at one video at a time, they can now see the hidden patterns across dozens of locations at once. They discovered that where you are matters more than when you are, and that the first few seconds of a warning are the most critical moment to catch a driver's attention. This helps engineers build smarter, safer roads by treating similar crossings as a team.

Technical Summary

1. Problem Statement

Railway crossing crashes are a significant safety concern in the US, primarily caused by drivers failing to yield to trains. Current safety analysis methods suffer from two main limitations:

• Isolation: Traditional approaches analyze crossings individually, missing opportunities to identify shared behavioral patterns across different locations.
• Lack of Temporal Granularity: Existing studies often use aggregate statistics or focus on single events, failing to capture how driver behavior evolves through distinct temporal phases of a crossing event (warning activation, gate lowering, train passage, and gate raising).

The authors aim to develop an automated framework to discover latent behavioral patterns across multiple crossings and temporal phases to enable targeted safety interventions.

2. Methodology

The proposed framework utilizes a Multi-View Tensor Decomposition approach. The pipeline consists of four key stages:

A. Data Preprocessing and Phase Annotation

• Dataset: 31 video clips collected from 4 distinct railway crossing locations in Lincoln, Nebraska.
• Phase Segmentation: Videos are manually annotated into five phases, with analysis focused on three critical constrained phases:
  1. Approach (Phase A): From warning light activation to gate lowering.
  2. Waiting (Phase B): From gates down to train clearance.
  3. Clearance (Phase C): From train clearance to gate raising.
• Temporal Labeling: Videos are categorized by time-of-day (off-peak, morning rush, midday, afternoon/evening).

B. Video Embedding Extraction

• Model: The authors use TimeSformer (a transformer-based video model pre-trained on Kinetics-400) to extract 768-dimensional embeddings.
• Sampling Strategy: To capture temporal dynamics rather than single moments, multiple clips are sampled per phase based on duration (1 clip for <20s, 3 for 20–60s, 5 for >60s).
• Aggregation: The final embedding for each phase is the mean of all sampled clip embeddings. This results in 93 total embeddings (31 videos × 3 phases).

C. Multi-View Tensor Construction

• Similarity Matrices: For each phase $p \in \{A, B, C\}$, a $31 \times 31$ symmetric matrix is constructed using pairwise cosine similarity between video embeddings.
• Tensor Formation: These three matrices are stacked along a third dimension to form a third-order tensor $\mathcal{X} \in \mathbb{R}^{31 \times 31 \times 3}$, where the frontal slices represent behavioral similarities for each specific phase.

D. Tensor Decomposition and Rank Selection

• Algorithm: Non-Negative Symmetric CP Decomposition is applied to factorize the tensor:
  $$\mathcal{X} \approx \sum_{r=1}^{R} \lambda_r \mathbf{a}_r \circ \mathbf{u}_r \circ \mathbf{u}_r$$
  • $\lambda_r$: Component weight.
  • $\mathbf{a}_r$: Phase loadings (temporal signature).
  • $\mathbf{u}_r$: Video loadings (behavioral profile of specific crossings).
  • Non-negativity ensures components are additive and interpretable (no bipolar cancellation).
• Rank Selection ($R$): The optimal rank was determined using three metrics:
  1. CORCONDIA: Validated structural fit (Rank 3 showed issues; Rank 4 was acceptable).
  2. Reconstruction Error: Showed diminishing returns after Rank 3–4.
  3. Holdout Validation: Masked 10% of data; Rank 4 provided a balance of generalization and expressiveness.
  • Final Choice: Rank 4.

3. Key Contributions

1. Multi-View Behavioral Framework: Introduced a tensor-based model that explicitly captures behavioral similarities across three distinct temporal phases, allowing for the evolution of driver behavior to be modeled.
2. Interpretable Component Discovery: Demonstrated that symmetric CP decomposition successfully extracts latent behavioral components with distinct temporal signatures, validated by rigorous rank selection diagnostics.
3. Cross-Location Analysis: Provided empirical evidence that crossing location is a stronger determinant of behavior than time of day, and that the Approach phase offers the most discriminative behavioral signatures.

4. Key Results

• Location Dominance:
  • Clustering: t-SNE visualization and component loadings revealed clear clustering based on location. For instance, "NW 12th Street" videos formed a distinct cluster (high loadings in Component 1), while "35th Street" videos were distributed across Components 2–4.
  • Time-of-Day: Temporal categories (rush hour vs. off-peak) showed substantial overlap in the component space, indicating time is a secondary factor compared to location.
• Phase Signatures:
  • Component 4: Dominated by the Approach phase (loading 1.52), suggesting initial driver response to warnings is highly discriminative.
  • Component 2: Dominated by Waiting and Clearance phases, capturing post-gate-lowering behavior.
• Within-Location Variability: Even within the same location (e.g., 35th Street), significant heterogeneity was observed (Component 3 loadings ranged from 0.0 to 1.2), implying that specific traffic conditions or situational variables also influence behavior beyond just the physical location.

5. Significance and Implications

• Scalable Safety Planning: This automated framework allows agencies to group crossings by behavioral similarity rather than just geography or traffic volume.
• Targeted Interventions:
  • Crossings with similar "Approach-dominant" signatures can be grouped for specific interventions (e.g., enhanced early warning systems).
  • Crossings forming distinct clusters (like NW 12th Street) can be flagged for expert review to understand unique infrastructure factors.
• Infrastructure vs. Temporal: The finding that location outweighs time-of-day suggests that infrastructure modifications may be more effective than temporal traffic management strategies for improving safety.

6. Limitations

• Lack of Metadata: The study could not correlate behavioral signatures with specific crossing characteristics (geometry, signage, speed limits) due to missing metadata.
• General-Purpose Model: TimeSformer was pre-trained on general actions (Kinetics-400). Fine-tuning on railway-specific data might improve sensitivity to domain-specific violations.
• Sample Size: The analysis was limited to 4 locations with unbalanced sampling, requiring broader validation for generalization.