Technical Summary: Global Interaction Modeling with Human Knowledge for Important Traffic Objects Identification

1. Problem Statement

The identification of important traffic objects is a critical cognitive task for autonomous vehicles, requiring the system to reason about the risk and influence of surrounding entities on the ego vehicle's decision-making process. Existing approaches face two primary limitations:

1. Insufficient Spatio-Temporal Information: Most methods rely on ego-centric 2D visual data, which provides incomplete spatial and temporal context, making it difficult to model complex dynamic interactions among traffic objects.
2. Lack of Human Knowledge Integration: Current models often fail to incorporate human insights or textual descriptions, limiting their ability to reason about dynamic patterns and interpret complex driving scenarios effectively.
3. Data Granularity: While some studies utilize textual data, they often focus on coarse-grained scene captions or specific objects, lacking fine-grained textual features for all perceptible objects within the scene.

2. Methodology

The authors propose a novel framework that shifts from ego-centric 2D perception to a Bird's-Eye View (BEV) perspective, integrating multimodal contrastive learning to incorporate human knowledge. The framework consists of three main components:

2.1 Data Preprocessing and Input

The framework operates on top of a mature BEV perception system (e.g., LiDAR/Camera fusion) to generate 3D bounding boxes, trajectories, and object attributes. Unlike methods relying solely on raw images, this approach utilizes structured motion and scene attributes derived from the BEV map.

2.2 Contrastive Language-Motion Pre-training (CLMP)

To bridge the gap between dynamic motion patterns and human insights, the authors develop a CLMP model.

• Objective: Align object motion attributes with human-annotated behavioral descriptions (text) using multimodal contrastive learning.
• Mechanism: A Text Supervised Motion Encoder (TSM-Encoder) is pre-trained alongside a BERT text encoder. The model maximizes the cosine similarity between correct motion-text pairs and minimizes it for incorrect pairs.
• Outcome: The trained TSM-Encoder can generate pseudo-text features from raw motion attributes. These features encode human knowledge (e.g., "moving away," "making a left turn") without requiring explicit text input during the inference phase of the main task.

2.3 Interaction-Aware Multimodal Network (IAM-Network)

This network performs the core task of identifying and ranking object importance (Non-Important, Low, Medium, High).

• Latent Query Motion Encoder (LQM-Encoder): This module hierarchically processes motion attributes and scene attributes (e.g., traffic lights). It employs a learnable latent query and self-attention mechanisms to capture dynamic characteristics and model interactions between objects. Cross-attention is used to integrate structured scene information.
• Multimodal Refinement Module (MR-Module): This module adaptively refines the spatio-temporal representations generated by the LQM-Encoder. It integrates the pseudo-text features (human knowledge) derived from the CLMP model with the motion features.
  • Fusion Strategy: The module utilizes pairwise multiplication (a squeeze-and-excitation operation) to modulate the spatio-temporal features based on the textual-motion context. This allows the model to dynamically adjust its reasoning based on human priors.
• Classification: The final features are passed through a fully connected layer to output probabilities for four importance classes.

3. Key Contributions

1. Novel Framework: A new important traffic object identification framework that leverages a global BEV perspective for interaction modeling and multimodal contrastive learning to integrate human knowledge, overcoming the limitations of ego-centric 2D methods.
2. CLMP Model: A pre-training model that aligns motion attributes with textual descriptions, enabling the extraction of human knowledge from object motion attributes to generate pseudo-text features.
3. IAM-Network: An architecture that captures and adaptively refines spatio-temporal representations by integrating motion attributes, scene attributes, and pseudo-textual features, significantly enhancing reasoning capabilities for dynamic traffic characteristics.

4. Experimental Results

The framework was evaluated on the Rank2Tell dataset, which contains 116 interactive traffic scenarios with 3D object locations, importance levels, and behavioral captions.

• Performance: The proposed method outperformed existing state-of-the-art works (including Gao et al., Zhang et al., Li et al., Yu et al., and Sachdeva et al.) across all metrics (Accuracy and F1 scores for all classes).
  • Accuracy: Achieved 84.71%, compared to the next best of 80.93%.
  • Critical Classes: Showed significant improvements in identifying Medium (38.88% F1) and High (69.75% F1) importance objects, which are critical for safety.
• Ablation Studies:
  • Scene Attributes: Incorporating scene features (e.g., traffic signals) improved the F1 score for High importance objects by 2.39%.
  • Human Knowledge: The integration of CLMP and the MR-Module yielded the most significant gains, particularly for Low (28.07%) and Medium (38.88%) importance objects, demonstrating the value of human priors in resolving ambiguity.
  • Fusion Strategy: Late fusion with pairwise multiplication and a GRU-based TSM-Encoder was found to be the optimal configuration.
• Efficiency: The model maintains real-time performance with an inference time of 7.02ms and a parameter count of 6.8M.
• Visualization: t-SNE embeddings showed that the IAM-Network produces clearer separation between importance classes compared to baselines, particularly distinguishing between low and medium importance objects.

5. Significance and Claims

The paper claims that its primary significance lies in demonstrating that global interaction modeling under a BEV perspective, combined with multimodal contrastive learning, is superior to ego-centric visual methods for important object identification.

• Reasoning Capability: By bridging dynamic patterns with human insights, the framework enhances the model's ability to reason about complex, dynamic traffic scenarios that are often ambiguous in 2D views.
• Trustworthiness: The method addresses a critical challenge in developing trustworthy autonomous driving systems by enabling robust identification of risk-inducing objects.
• Practicality: The framework is designed to be deployed in intelligent vehicles with real-time BEV perception capabilities, effectively assisting in safer driving decisions.

The authors acknowledge limitations, noting that the current CLMP relies on hand-crafted rules for generating pseudo-text and that distinguishing between "Low" and "Medium" importance remains challenging due to annotation subjectivity. Future work aims to leverage Large Vision-Language Models (VLMs) for automatic text generation and explore continuous importance modeling.