PoseDriver: A Unified Approach to Multi-Category Skeleton Detection for Autonomous Driving

1. Problem Statement

Autonomous driving systems require precise understanding of both dynamic objects (pedestrians, vehicles, animals) and static infrastructure (lanes) to ensure safety and effective decision-making. While bounding boxes and segmentation masks are common, they lack the structural granularity needed to predict intent (e.g., pedestrian head orientation) or complex trajectories.

• The Gap: Existing skeleton detection methods are typically single-category (e.g., only humans or only cars) or category-agnostic but rely on heavy auxiliary inputs (text prompts, image queries) and multiple inference passes.
• The Challenge: There is no unified architecture that can simultaneously detect skeletons for diverse categories (cars, pedestrians, animals, bicycles, and lanes) using only a single input image. Furthermore, traditional methods struggle with domain shifts when training on multiple datasets simultaneously and often fail to represent static elements like lanes, which lack natural semantic keypoints.

2. Methodology

The authors propose PoseDriver, a unified bottom-up framework that jointly detects skeletons across five key categories: cars, pedestrians, bicycles, animals, and lanes.

A. Core Architecture

Inspired by OpenPifPaf, PoseDriver utilizes:

• Composite Intensity Fields (CIF): To detect keypoints.
• Composite Association Fields (CAF): To associate keypoints into specific object instances.
• Unified Backbone: Instead of using separate models for each category, a single backbone processes the image to generate feature maps for all tasks.

B. Key Technical Innovations

1. Skeleton-Based Lane Detection:

   • Lanes are traditionally detected via segmentation or anchors. PoseDriver reframes lane detection as a skeleton problem.
   • Lanes are represented as a fixed set of ordered 2D keypoints (e.g., 24 points per lane).
   • To handle ambiguity, the authors interpolate ground truth annotations to ensure uniform pixel-distance distribution between points, preserving topological continuity without relying on semantic meaning (unlike human joints).

2. Bicycle Skeleton Annotation:

   • The authors created a new dataset by manually annotating 1,557 images from MS COCO with 6 keypoints per bicycle (rear wheel back/center, front wheel front/center, seat, handlebar center).

3. Backbone Optimization for Multi-Task Learning:

   • Normalization Strategy: The authors identified that Batch Normalization (BN) causes performance degradation in multi-domain training due to domain shift. They replaced BN with Layer Normalization (LN) or selected backbones designed without BN (e.g., ConvNeXt, Swin Transformer, ClipConvNeXt).
   • Feature Fusion: They integrated Feature Pyramid Networks (FPN) to handle objects at varying scales (e.g., distant cars vs. close pedestrians).
   • Task-Specific Heads: A Transformer block (Swin block) is inserted before each task head. This allows the network to share a backbone while refining features specifically for each category (e.g., distinguishing between rigid car structures and deformable animal poses).

4. Data Augmentation:

   • Mosaic Augmentation: Used to increase the likelihood of including large-scale objects in the training batch, addressing scale variation issues.

3. Key Contributions

• Unified Bottom-Up Framework: The first architecture to jointly detect skeletons for cars, pedestrians, bicycles, animals, and lanes in a single pass without auxiliary prompts.
• Skeleton-Based Lane Detection: A novel method representing lanes as fixed keypoints, achieving state-of-the-art (SOTA) performance on the OpenLane dataset.
• New Dataset: Introduction of a manually annotated Bicycle Skeleton Dataset (subset of MS COCO) to fill a gap in multi-category perception.
• Transfer Learning Validation: Demonstration that multi-task pre-training significantly improves performance on novel categories (bicycles) compared to single-task training.
• Normalization Analysis: Empirical evidence showing that removing Batch Normalization in favor of Layer Normalization or BN-free architectures is critical for stable multi-domain training.

4. Experimental Results

Lane Detection Performance

• OpenLane Dataset: PoseDriver achieved SOTA performance.
  • The Swin-L + FPN variant achieved an F1 score of 71.5%, outperforming the previous SOTA (CondLSTR-L) by 8.1 points.
  • The ConvNeXt variant achieved 67.5%, beating CondLSTR-L by 4.1 points.
  • The model showed superior robustness in extreme weather and night scenarios (up to 11.9 points improvement over baselines).
• CULane Dataset: The method achieved an F1 score of 80.01%, outperforming keypoint-based baselines like GANet and RCLane, particularly in crowded and shadowed scenes.

Multi-Task Skeleton Detection

• Backbone Comparison: Replacing ShuffleNet (with BN) with ConvNeXt and Swin (without BN) resulted in massive performance gains.
  • ConvNeXtv2 with FPN and Transformer heads achieved a mean score of 66.4% across tasks.
  • Swin-L achieved 65.4%.
  • In contrast, the original ShuffleNet with BN suffered a ~10-point drop in multi-task settings compared to single-task settings.
• Ablation Studies:
  • FPN: Consistently improved performance (e.g., +4.8% mean score for ConvNeXt).
  • Transformer Heads: Significantly boosted animal detection (a data-scarce category), improving AP by ~2-3% across backbones.
  • Downsampling: "Even Downsampling with Fixed Endpoints" for lanes yielded the best results (78.5% F1) compared to random or fixed-interval downsampling.

Generalization to New Categories (Bicycles)

• Models pre-trained on the four-task setting (pedestrians, animals, cars, lanes) were fine-tuned on the new bicycle dataset.
• Results: Pre-trained models significantly outperformed baselines trained from scratch on bicycles.
  • Swin-L: Improved from 71.6% (baseline) to 80.0% (pre-trained + fine-tuned).
  • ConvNeXtv2: Improved from 73.4% to 79.3%.
• This confirms that shared structural representations across diverse categories facilitate effective transfer learning.

5. Significance

PoseDriver bridges the gap between high-level decision-making requirements and real-time perception capabilities in autonomous driving. By unifying the detection of dynamic agents and static infrastructure into a single, efficient skeleton-based framework, it offers:

1. Efficiency: Eliminates the need for multiple detectors or heavy prompt encoders.
2. Richness: Provides structural and orientation data crucial for trajectory prediction and intent recognition.
3. Scalability: Demonstrates that a unified model can generalize to new categories (like bicycles) with minimal retraining, making it highly adaptable for evolving autonomous driving scenarios.
4. Robustness: Proves effective in challenging conditions (night, rain, occlusion), addressing critical safety gaps in current perception systems.