Hazard-Aware Traffic Scene Graph Generation

This paper introduces a novel Traffic Scene Graph Generation framework that leverages accident data and depth cues to model safety-relevant relations between hazards and the ego vehicle, thereby enhancing situational awareness in complex driving scenarios.

The Gist

Imagine you are driving down a busy highway. Your eyes are darting everywhere: checking the rearview mirror, scanning the dashboard, looking at the traffic lights, and glancing at the pedestrians on the sidewalk. Your brain is trying to process everything at once. But here's the problem: you can't pay attention to everything. If you try to focus on a bird flying in the sky and a parked car three blocks away with the same intensity as the red light turning on right in front of you, you might miss the danger that actually matters.

This paper introduces a new AI system called HATS (Hazard-Aware Traffic Scene Graph Generation) that acts like a super-smart co-pilot designed specifically to solve this "attention overload" problem.

Here is how it works, broken down into simple concepts:

1. The Problem: Too Much Noise, Not Enough Signal

Existing AI systems are like a camera that takes a photo of the whole world and labels everything equally. It sees a "car," a "tree," a "cloud," and a "pedestrian." It doesn't know that the cloud is irrelevant, but the pedestrian stepping off the curb is a life-or-death emergency.

Current systems also use generic descriptions like "the car is on the road." But for a driver, what matters is: "The car is about to hit me from the left."

2. The Solution: The HATS Co-Pilot

The HATS model doesn't just look at the scene; it thinks like a driver. It has three main "brain parts" working together:

Part A: The "Path Filter" (ERES Module)

Imagine you are walking through a crowded party. You don't need to know who is wearing a red hat in the corner; you only care about the people walking toward your path.

• What HATS does: Before it even tries to understand the details, it looks at where your car is going (the "ego path"). It instantly ignores everything that isn't on that path (like distant mountains or parked cars far away). It filters out the noise so the system only focuses on the "candidates" that could actually touch your car.

Part B: The "Accident Encyclopedia" (The Knowledge Graph)

This is the paper's biggest innovation. Most AI learns only by looking at pictures. HATS also reads a massive library of past traffic accidents.

• The Analogy: Imagine a driving instructor who has studied thousands of police reports. They know that "a car turning left at an intersection" often leads to a "head-on collision," while "a car changing lanes on a highway" often leads to a "side-swipe."
• What HATS does: It connects the visual scene to this library of real-world crash data. It doesn't just see a car; it sees a car and asks, "Based on history, what kind of accident does this situation usually cause?" This helps it predict severity (Is this a minor annoyance or a deadly crash?).

Part C: The "Scene Graph" (The Final Report)

Once HATS filters the noise and checks the accident history, it creates a simple, color-coded map for the driver.

• The Output: Instead of a confusing list of 100 objects, it gives you a short list of the top 3 dangers.
  • Red Tag: "Sideswipe risk from the right, Caution."
  • Yellow Tag: "Pedestrian on the left, Info."
  • Green Tag: "Parked car, Ignore."

3. How It "Thinks" (The Secret Sauce)

The paper describes a few clever tricks the AI uses to get this right:

• Depth Perception: It doesn't just use a flat photo; it uses "depth cues" (like 3D glasses) to know exactly how far away things are. A car 10 feet away is dangerous; a car 100 feet away is not.
• The "Gating" Mechanism: Think of this as a bouncer at a club. The system has many different types of information (what the object looks like, how far it is, what the accident history says). The "bouncer" decides which piece of information is most important for the current situation. If the history says "high crash risk," the bouncer turns up the volume on that signal.
• Learning from Mistakes: The system was trained on a dataset where they manually labeled 820 images with specific danger levels. They found that the more data they fed it, the smarter it got, proving that this "accident encyclopedia" approach really works.

4. Why This Matters

In the real world, distracted driving is a huge killer. If an autonomous car (or a driver-assist system) screams "ALERT!" every time it sees a squirrel or a cloud, the driver will eventually ignore it (this is called "alarm fatigue").

HATS is different because it is selective. It only alerts you when something is actually relevant to your safety. It tells you what the danger is, where it is, and how bad it could be.

Summary

Think of HATS as a smart traffic filter.

1. It ignores the background noise (sky, distant trees).
2. It focuses only on things that could hit you.
3. It consults a database of past accidents to guess how dangerous the situation is.
4. It gives you a simple, color-coded warning list so you can react instantly without getting overwhelmed.

It's not just about "seeing" the road; it's about understanding the danger before it happens.

Technical Summary

1. Problem Statement

Existing Scene Graph Generation (SGG) and Panoptic Scene Graph (PSG) methods are ill-suited for autonomous driving because:

• Lack of Safety Relevance: They treat all entities equally, failing to prioritize hazards that require immediate driver attention.
• Generic Predicates: They rely on generic spatial relations (e.g., "on," "in") rather than traffic-specific interactions (e.g., "sideswipe," "head-on").
• Missing Context: They often ignore historical accident data and depth cues, which are critical for assessing severity and interaction mechanisms.
• Noise: Processing all scene entities (including irrelevant background) creates computational noise and obscures critical hazard relationships.

The authors propose a new task: Traffic Scene Graph Generation (TSGG), which aims to generate ego-centric graphs that explicitly model the relations between prominent hazards and the ego vehicle, including their effect mechanism, relative side, and severity level.

2. Methodology: The HATS Framework

The proposed Hazard-Aware Traffic Scene Graph Generation (HATS) model consists of two main branches: a Main Scene Graph Branch and an Auxiliary Knowledge Branch.

A. Main Scene Graph Branch

This branch processes visual input to generate the scene graph.

1. Panoptic Segmentation (PS) Module: Uses a ResNet50-based Mask2Former to holistically segment and classify all entities in the scene into non-overlapping instances.
2. Ego-path Related Entities Selection (ERES) Module:
   • Goal: Filter out irrelevant entities (e.g., sky, distant parked cars) to focus only on those impacting the ego-vehicle's path.
   • Mechanism: Extracts a "path token" from the binary path mask and uses a multi-head cross-attention mechanism to compute relevance scores between the ego-path and all segmented entities.
   • Output: Selects a subset of candidate entities for further processing, reducing noise and computational load.
3. Traffic Scene Graph Generation (TSGG) Module:
   • Input Fusion: Combines four representations for each candidate entity:
     • Visual: RGB and Disparity (depth) features.
     • Semantic: Decoder embeddings and path-relevance logits.
     • Geometric: Overlap with the ego-path and proximity.
     • Knowledge: Embeddings from the Knowledge Graph (KG).
   • Gated Fusion: A learned gating mechanism adaptively weights these cues to create a robust "ego-entity pair descriptor."
   • Prediction Heads: Three dedicated heads predict:
     • Mechanism: The interaction type (e.g., sideswipe, head-on, rear-end).
     • Side: Relative position (Left, Front, Right).
     • Severity: A 4-level scale (Info, Caution, Imminent, Relevant-but-not-critical).

B. Auxiliary Knowledge Branch (Knowledge Graph)

This branch provides prior knowledge derived from real-world traffic accident data to guide reasoning.

1. Data Source: Structured crash investigation files from the US NHTSA.
2. Construction Pipeline:
   • Node Creation: Extracts 16,039 nodes (26 types) including vehicles, occupants, crash conditions, and consequences.
   • Structural Wiring: Creates 122,263 deterministic edges based on schema (e.g., Vehicle → Involved In → Crash).
   • Causality Enrichment: Adds 31,104 causal edges (e.g., Factor → Leads To → Crash) to capture accident logic.
   • Bridging: Aligns KG nodes with Cityscapes semantic categories and TSGG predicates.
3. Knowledge Graph Embedding (KGE):
   • Literal-Aware Initialization: Encodes both categorical (e.g., vehicle class) and numerical (e.g., injury count) node properties.
   • Qualifier-Aware Message Passing: Uses FiLM (Feature-wise Linear Modulation) to adaptively scale and shift message passing based on edge qualifiers (e.g., source: actor, role: victim).
   • Transformer Scorer: Uses a Transformer decoder to score triplets, enabling attention-driven interaction between head entities and relations.

3. Key Contributions

1. New Task Formulation (TSGG): Introduced an ego-centric problem formulation focusing on prominent hazards and traffic-specific relations, supported by a new relational annotation dataset on Cityscapes.
2. Structured Accident Knowledge: First work to leverage structured traffic accident data (NHTSA) as a prior knowledge modality. They developed a 4-stage pipeline to convert flat tables into a unified, multi-relational Knowledge Graph.
3. Advanced KGE Method: Proposed a novel embedding method handling hyper-relational edges and multi-attributed nodes using literal-aware initialization and FiLM-based message passing, outperforming baselines like StarE.
4. Comprehensive Framework: Integrated visual, geometric, semantic, and historical accident data into a unified framework for hazard-aware reasoning.

4. Experimental Results

The model was evaluated on 820 annotated Cityscapes images across 10 tasks.

• Knowledge Graph Embedding (KGE):
  • Achieved ~99.8% H@10 in triplet prediction, significantly outperforming the baseline StarE (73.00%).
  • Validated that literal-aware initialization and qualifier-aware message passing are critical for capturing complex accident patterns.
• Hazard Prioritization (HP):
  • Outperformed the state-of-the-art CFHP model in mAP@K (54.87% vs 31.66% at K=3) and NDCG@K.
  • Demonstrated superior ability to rank critical hazards at the top of the list.
• Scene Graph Detection (SGDet):
  • Achieved 62.79% R@50 and 76.13% mR@50, significantly beating baselines (MOTIFS, VCTREE, PSGTR).
  • The high mR@K indicates the model successfully predicts rare but safety-critical relations, not just frequent ones.
• Relevance & Prominence Classification:
  • Achieved 95.03% F1 for entity relevance and 90.89% F1 for prominence, vastly outperforming visual-only baselines.
• Ablation Studies:
  • Removing the ERES module caused a >40% drop in performance, proving its necessity for filtering noise.
  • Removing KGE or Depth cues significantly degraded severity and mechanism prediction, confirming that visual features alone are insufficient for safety reasoning.

5. Significance

• Safety-Centric AI: Moves beyond generic object detection to situational awareness, explicitly identifying why an object is dangerous and how it might affect the ego vehicle.
• Interpretability: The output is an intuitive graph where hazards are color-coded by severity and annotated with interaction mechanisms (e.g., "sideswipe, right, caution"), providing actionable guidance for drivers or autonomous systems.
• Data-Driven Reasoning: Demonstrates the power of integrating structured historical accident data (KG) with real-time visual perception, bridging the gap between "what is seen" and "what is dangerous."
• Foundation for Future Work: Establishes a new benchmark and methodology for hazard-aware driving, paving the way for video-based reasoning and more robust autonomous driving stacks.