Driving with A Thousand Faces: A Benchmark for Closed-Loop Personalized End-to-End Autonomous Driving

This paper introduces Person2Drive, a comprehensive benchmark and platform for personalized end-to-end autonomous driving that addresses key challenges in data scarcity, evaluation metrics, and algorithmic adaptation by providing a scalable data collection system, style-based quantitative metrics, and a reward-driven framework to replicate diverse individual driving behaviors.

The Gist

Imagine you are learning to drive a car. You have a friend who is a very cautious driver, always stopping early and merging slowly. You have another friend who is a bit more aggressive, speeding up quickly and cutting in tight.

Now, imagine building a self-driving car. Most current self-driving cars are like a generic "average" driver. They try to learn one single style that fits everyone. They drive safely, but they feel robotic and don't quite match your personal comfort zone. If you are a cautious person, the car might feel too jerky. If you like a smooth ride, the car might feel too hesitant.

This paper, titled "Driving with A Thousand Faces," introduces a new system called Person2Drive. Its goal is simple: Make the self-driving car drive exactly like you.

Here is how they did it, explained with some everyday analogies:

1. The Problem: The "One-Size-Fits-All" Suit

Currently, self-driving cars are trained on massive datasets, but they treat all human drivers as the same. It's like buying a suit that is sized for the "average" person. It might fit okay, but it will never feel perfectly tailored to your specific shoulders or height. The researchers realized that human driving is full of personality—some people are smooth, some are snappy, some are conservative. The old systems ignored this "personality."

2. The Solution: A New Driving School (The Dataset)

To teach a car to drive like you, you first need to see you driving.

• The Setup: The researchers built a low-cost, open-source driving simulator (using a game engine called CARLA) that feels very real. They hooked up a real steering wheel and pedals to a computer screen.
• The Students: They invited 30 real people to drive the same routes over and over again.
• The Result: They created a massive library of data called Person2Drive. Unlike other datasets that just look at the road, this one looks at the driver. It captures how specifically you turn the wheel, how hard you brake, and how you handle a merge. It's like having a video library of 30 different people driving the same street, showing their unique "fingerprints" on the road.

3. The "Style Translator" (The Evaluation)

How do you measure if a car is driving like you? You can't just say, "It feels right." You need math.

• The Style Vector: The researchers created a "Style Vector." Think of this as a DNA test for driving. Instead of just looking at speed, they measured 10 specific habits (like how much you jerk the steering wheel, how far you leave between cars, how fast you accelerate).
• The Scorecard: They developed a new scoring system (called MMDSS) that acts like a compatibility meter. If the self-driving car's behavior matches your "DNA," the score is high (close to 100%). If it drives like a stranger, the score drops. This allows them to prove, with numbers, that the car is actually learning your style.

4. The "Personal Trainer" (The Adaptation)

This is the magic part. They didn't want to retrain the whole car from scratch (which would be slow and dangerous). Instead, they used a two-step training process:

1. The Basic Driver: First, they train a standard, safe self-driving model (like a student driver who knows the rules but has no personality).
2. The Style Coach: Then, they introduce a "Style Reward Model." Imagine a coach standing next to the car. Every time the car makes a move, the coach checks: "Does this move look like the human driver?"
   • If the car turns smoothly like you, the coach gives a "thumbs up" (reward).
   • If the car jerks like a robot, the coach gives a "thumbs down."
   • The car learns to adjust its "brain" just enough to please the coach, without forgetting how to drive safely.

5. The Results: A Car That Feels Like Home

When they tested this new system:

• Safety First: The car didn't become reckless. It stayed safe and followed traffic rules.
• Personality Restored: The car started driving with the same "flavor" as the human. If the human was a smooth driver, the car became smooth. If the human was a bit more assertive, the car adjusted to that too.
• The "Thousand Faces": The system proved it could adapt to many different people, creating a unique driving experience for each user.

Why This Matters

Think of it like Spotify vs. a Radio Station.

• Old Self-Driving Cars are like a radio station playing the same playlist for everyone. It's fine, but it's not yours.
• Person2Drive is like Spotify's algorithm. It listens to your specific taste and curates a playlist (a driving style) that feels perfectly tailored to you.

This research is a huge step toward human-centered AI. It moves us away from machines that just "work" and toward machines that understand us, making the future of self-driving cars not just safer, but more comfortable and trustworthy.

Technical Summary

1. Problem Statement

Current End-to-End Autonomous Driving (E2E-AD) systems predominantly learn a single "average" driving style, failing to capture the inherent diversity of human drivers shaped by cognitive patterns, habits, and personality. This limitation creates three critical gaps:

• Data: Existing datasets lack individual-level annotations, often rely on simulated agents rather than human drivers, or do not support closed-loop evaluation.
• Evaluation: There are no standardized, quantitative metrics to measure and compare individual driving styles.
• Algorithms: Existing methods struggle to learn continuous, fine-grained style representations directly from trajectory data within a unified E2E framework.

2. Methodology: The Person2Drive Framework

The authors propose Person2Drive, a comprehensive platform addressing the data, evaluation, and algorithmic challenges through a three-stage pipeline:

A. Personal Driving Data Collection

• System: A low-cost, open-source, human-in-the-loop data collection system built on CARLA. It utilizes a Logitech G29 steering wheel/pedal set and a fixed 2D navigation map to simulate realistic driving dynamics.
• Dataset (Person2Drive):
  • Scale: Collected from 30 human drivers.
  • Structure: Drivers repeatedly traverse identical 1km routes across diverse scenarios (intersections, merges, lane changes) in different CARLA towns.
  • Modalities: Synchronized multi-modal data including LiDAR, 6 cameras, 5 radars, IMU, GNSS, and HD-Map, sampled at 10 Hz.
  • Key Feature: Unlike prior datasets, it captures multiple trajectories per driver on the same route, enabling the analysis of intra-driver consistency and inter-driver variability.

B. Driving Style Evaluation (Metrics)

To quantify driving styles, the authors introduce a three-step evaluation framework:

1. Style Vector Indicators: They extract 10 representative style features (e.g., lateral jerk, braking frequency, TTC) based on vehicle control parameters. These are normalized using median normalization and selected based on variance to maximize inter-driver discriminability.
2. Quantification Metrics:
   • MMDSS (Maximum Mean Discrepancy Similarity Score): A normalized metric derived from MMD to measure distributional similarity between style vectors. It ranges from [0, 1], where higher values indicate greater style consistency.
   • KL Divergence: Used as a complementary metric to measure the directed distance between style distributions.
3. Scenario-Specific Normalization: Features are normalized per scenario to mitigate bias caused by environmental context (e.g., highway vs. urban), ensuring fair comparison across diverse conditions.

C. Personal Driving Style Adaptation (PDSA)

A three-stage training framework is proposed to adapt a base E2E model to a specific driver:

1. Stage 1: E2E Basic Model Training: A base model (based on DiffusionDrive) is trained on general data (Bench2Drive) to learn robust driving capabilities.
2. Stage 2: Driving Style Modeling: A lightweight Reward Model (a 3-layer MLP) is trained to map raw trajectories to the 10-dimensional style vectors. This replaces hand-crafted rules with a learnable, differentiable mapping.
3. Stage 3: Personal Style-Guided Finetuning:
   • The base model is fine-tuned using the target driver's historical trajectories.
   • Loss Function: The model minimizes the Mean Squared Error (MSE) between the predicted trajectory's style vector (via the Reward Model) and the ground-truth style vector.
   • Constraint: Only the final trajectory planner head is updated, preserving the backbone's general driving competence while injecting personalized style.

3. Key Contributions

1. Person2Drive Dataset: The first large-scale, closed-loop, human-in-the-loop dataset with individual driver identity labels and multiple repetitions per route, specifically designed for personalized E2E-AD.
2. Quantitative Evaluation Standard: Introduction of MMDSS and KL divergence based on style vectors, providing the first reproducible, quantitative method to assess individual driving style similarity and separation.
3. PDSA Framework: A novel, efficient adaptation strategy that uses a style reward model to guide fine-tuning, enabling E2E models to replicate specific driver styles without compromising safety or baseline performance.

4. Experimental Results

• Style Evaluation Validity:
  • On the Person2Drive dataset, Intra-person MMDSS (0.9267) was significantly higher than Inter-person MMDSS (0.6545), proving the metrics can distinguish between different drivers while maintaining consistency for the same driver.
  • The Reward Model successfully learned to predict style vectors with high fidelity (MMDSS deviation < 0.02 compared to ground truth).
• Adaptation Performance:
  • Style Alignment: The PDSA framework achieved the highest MMDSS scores (0.6840 average) compared to baselines and ablation studies (Direct Finetuning, PDSA without BEV), demonstrating superior style alignment.
  • Safety & Competence: Crucially, fine-tuning improved the overall driving score by +14.56 points while maintaining a success rate of 98.2% (negligible drop from 98.9%). This confirms that personalization does not degrade safety.
  • Visualization: Qualitative results showed PDSA successfully reproduced human-specific behaviors (e.g., lane-change timing, post-stop following gaps) that baseline models failed to capture.

5. Significance

• Human-Centered AI: This work shifts E2E-AD from a "one-size-fits-all" approach to a human-centric paradigm, enhancing user trust, comfort, and acceptance.
• Benchmarking: Person2Drive establishes a unified foundation for reproducible research in personalized autonomous driving, addressing the lack of standardized datasets and metrics.
• Practicality: The proposed PDSA method offers a computationally efficient solution (fine-tuning only the head) that can be deployed to adapt autonomous vehicles to individual user preferences without retraining the entire perception backbone.

In conclusion, Person2Drive bridges the gap between generic autonomous driving and personalized human behavior, providing the data, metrics, and algorithms necessary to build trustworthy, individualized E2E systems.