Efficient Sequential Neural Network with Spatial-Temporal Attention and Linear LSTM for Robust Lane Detection Using Multi-Frame Images

Technical Summary: Efficient Sequential Neural Network with Spatial-Temporal Attention and Linear LSTM for Robust Lane Detection

Problem Statement
Lane detection is a fundamental perception task for automated vehicles (AVs) and Advanced Driver Assistance Systems (ADAS), particularly in mixed-traffic environments involving human-driven vehicles. While vision-based methods are the most feasible approach, current state-of-the-art techniques often lack versatility in delivering accurate, robust, and real-time compatible results. Traditional methods relying on hand-crafted features struggle with complex scenarios, while existing deep learning approaches frequently neglect critical image regions and their spatial-temporal (ST) salience. Consequently, performance degrades significantly in challenging conditions such as serious occlusion, dazzle lighting, and shadowed areas. Furthermore, many sequential models that attempt to utilize continuous frames are computationally heavy, failing to balance accuracy with the efficiency required for real-time deployment.

Methodology
The authors propose a novel sequential neural network architecture designed as a sequence-to-one segmentation task. The model accepts a sequence of multiple continuous image frames as input and outputs the lane detection result for the final frame in the sequence.

• Architecture: The backbone utilizes a standard UNet encoder-decoder structure. The encoder extracts features from the input frames, while the decoder upsamples these features to generate the segmentation mask. Skip connections are employed to retain spatial information from the encoder.
• Spatial-Temporal (ST) Attention Mechanism: Embedded between the encoder and decoder is a novel attention module designed to mimic human visual cognition by focusing on salient regions and frames. This module integrates features extracted by the encoder with hidden states from a temporal feature extractor (specifically LSTM or GRU).
  • The mechanism assigns attention weights to specific frames and regions within frames, dynamically prioritizing salient ST correlations.
  • The authors developed and tested three variants of this attention module:
    1. Temporal Attention (Tem_Att): Focuses on the significance of each frame in the sequence using trainable vectors.
    2. Spatial-Temporal Attention (ST_Att): Applies attention to specific regions within frames while considering temporal dependencies, using one-to-one connections.
    3. Spatial-Temporal Attention with Fully Connected Layers (STFC_Att): Enhances ST_Att by employing many-to-many connections via fully connected layers, allowing the model to capture spatial dependencies between pixels across the entire frame and temporal correlations across the sequence.
• Training Strategy: The model is trained using a supervised, end-to-end learning strategy with a weighted binary cross-entropy loss function to address class imbalance between lane pixels and background.

Key Contributions

1. Novel ST Attention Mechanisms: The introduction of three attention model variants (Tem_Att, ST_Att, STFC_Att) that effectively extract key features and exploit salient ST correlations among continuous image frames.
2. Efficient Sequential Architecture: The proposed model achieves high performance with a compact architecture. Compared to baseline sequential models (e.g., UNet_ConvLSTM, SegNet_ConvLSTM), the proposed models exhibit fewer parameters and reduced Multiply-Accumulate Operations (MACs), highlighting computational efficiency.
3. Robustness in Challenging Scenarios: The model demonstrates superior performance in difficult driving conditions, including occlusion, shadows, and blur, by leveraging ST information to infer lane positions even when they are partially or fully obscured in the current frame.
4. Generalizability: The approach is modular and compatible with various backbones (e.g., SCNN, SegNet) and can be integrated with self-supervised pre-training methods.

Experimental Results
The proposed models were evaluated on three large-scale open-source datasets: TuSimple, tvtLANE, and LLAMAS.

• Quantitative Performance: On the tvtLANE testset #1 (normal scenes), the proposed STFC_Att-SCNN_UNet_LSTM variant achieved the highest F1-measure (0.921), accuracy (98.20%), and precision (0.906), outperforming baselines like UNet_ConvLSTM and SegNet_ConvLSTM. Crucially, these results were achieved with significantly lower computational costs (e.g., ~44.9 G MACs vs. 217.0 G for SegNet_ConvLSTM).
• Challenging Scenarios: On tvtLANE testset #2 (12 challenging scenes), the proposed models maintained robustness, particularly in occluded and shadowed environments, often outperforming baselines in precision.
• Cross-Dataset Validation: Qualitative tests on the unlabeled Netherlands lane dataset (using a model trained only on LLAMAS) demonstrated strong transfer capabilities, successfully detecting lane numbers and locations without supervised training on the target domain.
• Ablation Studies: Visualizations of feature maps confirmed that the STFC_Att mechanism effectively "memorizes" lane positions from previous frames, maintaining coherent lane structures in areas where current frames are obscured by shadows or vehicles.

Significance and Claims
The paper claims that the proposed spatial-temporal attention mechanism successfully addresses the limitations of current vision-based lane detection methods by effectively utilizing the inherent ST information in driving scenes. The primary significance lies in the model's ability to outperform state-of-the-art methods in both normal and challenging scenarios while maintaining a lightweight architecture suitable for real-time applications. The authors emphasize that their approach offers a better balance between computational efficiency and detection robustness.

However, the authors modestly acknowledge limitations: the models struggle in specific underrepresented scenarios (e.g., mislabeled ground truth or extreme edge cases) and face challenges in cross-dataset transferability due to differences in lane structures and labeling formats between datasets (e.g., tvtLANE vs. LLAMAS). They conclude that future work should focus on integrated datasets and domain adaptation to further improve generalizability across diverse global driving conditions.