🤖 AI

FleetAgent: Teleoperation Assistant for Autonomous Fleets via Vectorized V2N Messages

FleetAgent is a cloud-hosted multimodal assistant that leverages a novel vector-to-embedding interface called VecFormer to process compact, structured vehicle-to-network messages, enabling efficient, explainable, and prioritized teleoperation monitoring for large-scale autonomous fleets while significantly reducing data transmission and memory overhead compared to raw sensor or text-based approaches.

Original authors: Juntong Peng, Qi Chen, Deyuan Qu, Takayuki Shimizu, Yaobin Chen, Ziran Wang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juntong Peng, Qi Chen, Deyuan Qu, Takayuki Shimizu, Yaobin Chen, Ziran Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive fleet of self-driving cars, like a school bus system for the whole city. Usually, these cars drive themselves perfectly. But sometimes, they get stuck, confused, or face a weird situation they can't solve alone. That's when they call for help from a human "remote pilot" sitting in a control center.

The problem? If every car tried to stream its live camera footage and sensor data to the control center, it would be like trying to download the entire internet on a single dial-up connection. It would be too slow, too expensive, and the human pilots would get overwhelmed trying to watch hundreds of video feeds at once.

Enter "FleetAgent." Think of FleetAgent as a super-smart, cloud-based assistant that acts as a translator and a filter for these self-driving cars. Here is how it works, broken down into simple concepts:

1. The "Sketch" Instead of the "Photo"

Instead of sending a high-definition video stream (which is huge and heavy), the cars send a tiny, compact "sketch" of the road.

The Old Way: Sending a 4K video of the street.
FleetAgent's Way: Sending a text message that says, "I am at this GPS spot, there is a red car 10 meters ahead moving left, and my plan is to turn right."
The Result: This "sketch" is 625 times smaller than a video stream. It's like sending a quick text message instead of a full movie file.

2. The "Smart Summarizer" (VecFormer)

The paper introduces a special tool called VecFormer. Imagine you have a giant library of books (the data), but you only have time to read the most important chapters.

Usually, AI models read everything, which fills up their "working memory" (called the KV-cache) and slows them down.
VecFormer is like a librarian who looks at the car's "sketch," instantly picks out the top K most important details (like the car right in front of you, not the one three blocks away), and feeds only those to the main AI brain.
This keeps the AI's memory usage 16 times smaller than if it tried to read a long text description of the scene.

3. The "Traffic Cop" for Human Pilots

The human pilots in the control center can't watch 1,000 cars at once. They need to know: Which car is in trouble right now?

FleetAgent doesn't just say, "Here is what the car sees."
It acts like a traffic cop, giving the pilot a natural language summary (e.g., "The car is hesitating at a crosswalk because a pedestrian is stepping out") and a Urgency Score (0 to 10).
If the score is a 9, the pilot knows to drop everything and look at that specific car immediately. If it's a 1, the car is fine, and the pilot can ignore it for now.

4. The "Training Ground" (VecEval)

To teach this system, the researchers couldn't just use real-world data because they needed examples of cars making mistakes to see if the AI could catch them.

They built a special dataset called VecEval.
They took real driving data and artificially created "imperfect" versions of the car's plans (like telling the car to speed up when it should stop).
They then had humans write down what should have happened and how urgent the situation was. This trained FleetAgent to spot bad plans and explain why they are bad.

The Bottom Line

The paper claims that by using this system:

Bandwidth is saved: You send 625 times less data than sending raw video.
Memory is saved: The AI needs 16 times less computer memory to process the info.
Speed is improved: The system responds faster (about 4.4 seconds) and is better at spotting dangerous situations than previous methods.
Safety is enhanced: It helps human pilots prioritize which cars need help, reducing the chance that a dangerous situation is missed.

In short, FleetAgent turns a chaotic flood of data into a neat, prioritized to-do list for human operators, making it possible to manage huge fleets of self-driving cars without crashing the network or the human's brain.

Technical Summary: FleetAgent

Problem Statement

Large-scale autonomous fleets increasingly rely on teleoperation to resolve rare, safety-critical failures that onboard systems cannot handle. However, current teleoperation paradigms face two primary bottlenecks:

Communication Constraints: Streaming raw sensor data (e.g., camera and LiDAR feeds) from vehicles to remote centers is costly in terms of bandwidth, latency, and stability, particularly under network congestion.
Operator Cognitive Load: Remote operators can only monitor a limited number of vehicles simultaneously. When switching between vehicles, they incur a high "context-switch cost" to reconstruct situational awareness from raw data or unstructured text.

Existing Multi-modal Large Language Models (MLLMs) show promise for reasoning about driving scenarios but typically require raw sensor inputs or long text descriptions, which are either too large to transmit or generate excessive context lengths that strain GPU memory (KV-cache) and reduce inference throughput.

Methodology: FleetAgent

The authors propose FleetAgent, a cloud-hosted teleoperation assistant designed to evaluate vehicle planning and generate structured explanations using compact, vectorized Vehicle-to-Network (V2N) messages.

1. Input Representation: Vectorized V2N Messages

Instead of raw images or verbose text, FleetAgent ingests a compact message containing:

Map Elements: Polylines representing lane dividers, boundaries, and crosswalks.
Dynamic Objects: Current position, heading, and predicted future trajectories of surrounding agents.
Ego Plan: Future waypoints of the autonomous vehicle's planned trajectory.

2. Core Architecture: VecFormer

To bridge the gap between structured vector data and token-based MLLMs, the authors introduce VecFormer, a vector-to-embedding interface with two key components:

Vector Encoders: Learnable modules that map input vectors (polylines/trajectories) into continuous embeddings compatible with the base MLLM.
Differentiable Top-K Context Selection: A mechanism that uses the ego plan as a query to compute attention scores over map and object embeddings. It selects the top- $K$ most relevant context vectors using sequential Gumbel-Softmax sampling. This ensures a bounded context length, preventing uncontrolled growth of the KV-cache and enabling efficient batch processing.

3. Training Strategy

The system is trained in three stages to adapt a pretrained MLLM (Qwen2.5-VL) with limited human-verified labels:

Masked Vector Reconstruction: Self-supervised pretraining of vector encoders to capture spatial and temporal structures.
Fixed-format Instruction Tuning: Aligning the encoded feature space with the MLLM's embedding space using short prompts, without requiring natural language annotations.
Supervised Fine-tuning (SFT): Using human-verified labels to train a feature projector and the context prioritization module, while applying prefix tuning and LoRA to the base LLM for reasoning and scoring.

4. Output

FleetAgent produces:

Structured Natural Language: Narration, explanation, and evaluation of the plan and scene to aid operator situational awareness.
Intervention Urgency Score: A score ($0-10$) to prioritize which vehicles require immediate operator attention.

Dataset: VecEval

To facilitate training and evaluation, the authors constructed VecEval, a dataset derived from nuScenes.

Imperfect Plan Generation: Counterfactual trajectories are generated by sampling from a clustered vocabulary of human driving behaviors, creating physically plausible but behaviorally imperfect plans (e.g., unsafe accelerations or lane violations).
Annotation: Each sample includes structured language responses (narration, explanation, evaluation) and an intervention urgency score, verified by human annotators to ensure factual correctness and calibrated urgency.

Key Contributions

FleetAgent Pipeline: A formulation for fleet-scale teleoperation that evaluates ego behavior from compact V2N messages, providing structured explanations and urgency scores for operator prioritization.
VecFormer Interface: A novel vector-to-embedding module with differentiable top-K context selection that bounds context length and significantly reduces KV-cache memory growth compared to tokenized text descriptions.
VecEval Dataset: A new dataset pairing structured driving contexts with imperfect plan variants and human-verified evaluation labels, addressing the lack of resources for teleoperation-specific plan evaluation.
System-Level Efficiency: Demonstration of significant reductions in bandwidth and memory footprint while maintaining competitive reasoning performance.

Empirical Results

The system was evaluated on the VecEval validation set and compared against baselines including raw images, BEV images, and tokenized language descriptions fed into models like Qwen2.5-VL-7B and GPT-4o.

Bandwidth Reduction: FleetAgent reduces uplink payload by up to 625 $\times$ compared to raw images (40.5 kB vs. ~25 MB).
Memory Efficiency: Compared to using tokenized text descriptions, FleetAgent reduces KV-cache memory usage by 16.54 $\times$ (1,241 MB vs. 20,522 MB) in local deployments, enabling larger batch sizes.
Performance: On VecEval, FleetAgent improves the Lingo-Judge score by 16.8% and reduces the Intervention Failure Rate (IFR) by 19.9% compared to Qwen2.5-VL-7B using language descriptions.
Latency: The system achieves a response time of 4.41 seconds, outperforming other local and API-based methods, with significantly lower end-to-end communication latency due to smaller packet sizes.
Cross-Dataset Validation: On the Nu-X dataset, FleetAgent outperforms previous state-of-the-art methods (e.g., ALN-P3), verifying the feasibility of using compact vectorized inputs on image-and-text pretrained VLMs.

Significance and Claims

The paper claims that FleetAgent demonstrates that compact structured V2N messaging can effectively enable efficient, explainable teleoperation monitoring for autonomous fleets. By decoupling the need for raw sensor streaming from the reasoning capabilities of MLLMs, the system addresses the scalability bottlenecks of bandwidth and operator cognitive load. The authors emphasize that their approach maintains competitive plan evaluation performance while drastically reducing the computational and communication resources required for cloud-hosted fleet management.

The authors note that their evaluation is currently offline due to the inaccessibility of online teleoperation systems. Future work is proposed to include closed-loop validation and human-subject studies to assess the practical benefits for operator decision-making.