Technical Summary: Jarvis – An Open-Source Edge-Deployable Voice Assistant for Autonomous Racecars

Problem Statement

In autonomous racing, decision-making occurs in fractions of a second, demanding software systems that prioritize speed and reliability. While autonomous vehicles operate with high autonomy, human operators still require a mechanism to issue high-level behavioral commands (e.g., "start," "stop," "pit lane") or react to unexpected track conditions. Traditional interfaces, such as Graphical User Interfaces (GUIs) or command lines, necessitate visual attention, increasing reaction times and diverting focus from live telemetry.

Voice assistants offer a hands-free alternative, but their deployment in time-critical racing scenarios faces significant hurdles:

1. Latency and Network Dependency: Online-hosted Large Language Models (LLMs) introduce variable inference latency and require stable network connections, which are unsuitable for split-second racing decisions.
2. Semantic Mismatch: Natural language is unbounded, whereas vehicle control requires a limited set of executable commands. Operators may phrase a "stop" command in numerous ways ("halt," "bring it to a stop"), requiring robust intent recognition to map these variations to specific actions.
3. Resource Constraints: Edge hardware in mobile racing environments has limited computational power, necessitating lightweight models that do not compromise on inference speed.

Existing solutions like LLM4AD utilize cloud-based APIs for general passenger comfort but suffer from high latency and network reliance. Conversely, lightweight Small Language Models (SLMs) have shown promise in specialized tasks but require domain-specific adaptation to achieve high accuracy in command classification.

Methodology

The authors developed Jarvis, an offline, edge-deployable voice assistant architecture designed specifically for high-level control of autonomous racecars. The system integrates three primary components:

1. System Architecture

Jarvis operates as a standalone module within a base station that interfaces with the vehicle's onboard software (ROS2-based). The workflow is as follows:

• Wake-Word Detection: Uses the openWakeWord framework to continuously monitor audio for the trigger phrase "Hey Jarvis!" using keyword spotting.
• Speech-to-Text (STT): Upon activation, the system performs local transcription using OpenAI's lightweight, English-only Whisper ("base.en") model. This ensures network-independent processing.
• Intent Classification: The transcribed text is processed by a fine-tuned LLM that maps natural language to predefined behavioral commands (e.g., speed targets, pit lane requests).
• Confirmation & Execution: The system provides auditory feedback via a Text-to-Speech (TTS) module (Coqui TTS with VITS). Crucially, the operator must confirm the command before it is transmitted via ROS2 to the base station, where safety checks are performed before forwarding to the vehicle.

2. Model Development and Fine-Tuning

The core innovation lies in the text-to-command classifier. The authors evaluated various online and local models before selecting Mistral 7B for domain-specific fine-tuning.

• Dataset Preparation: An initial dataset of 85 samples was insufficient for training. The authors expanded this to 1,645 samples (17 command classes + an Out-of-Scope class) using GPT-4 for augmentation techniques (synonym replacement, paraphrasing, reordering).
• Training Pipeline: The team employed QLoRA (Quantized Low-Rank Adaptation) with 4-bit quantization using the Unsloth framework. This allowed efficient fine-tuning on resource-constrained hardware (NVIDIA GeForce RTX 3060 Laptop GPU).
• Optimization: A grid search was conducted over learning rates, epochs, gradient accumulation steps, and LoRA ranks. The process involved two stages: an initial fine-tuning on a subset of commands to identify the best model architecture, followed by full fine-tuning on the complete dataset.

Key Contributions

The paper outlines three primary contributions:

1. Comparative Analysis: A benchmark comparing online-hosted models (e.g., GPT-4, GPT-3.5) against locally deployed open-weight models, evaluating the trade-off between inference latency and intent recognition accuracy.
2. Fine-Tuning Pipeline: A demonstrated pipeline for adapting a lightweight local model (Mistral 7B) to achieve superior intent recognition accuracy while maintaining low latency, outperforming cloud-based alternatives in this specific domain.
3. Open-Source Framework: The release of an open-source, offline voice assistant architecture that integrates wake-word detection, local STT, and command classification, validated on mobile hardware.

Experimental Results

The evaluation focused on Intent Classification Accuracy and Inference Time.

• Baseline Performance: Initial benchmarks showed that while online models (e.g., GPT-4o) achieved moderate accuracy (74–85%), they suffered from high latency (3.38s to 21.08s). Local models (e.g., Llama 3.2 3B) had low latency (<0.5s) but poor initial accuracy (<25%).
• Fine-Tuned Performance: After two stages of fine-tuning, the Mistral 7B model achieved:
  • Accuracy: 97.63% classification accuracy on the test set, surpassing all evaluated cloud-based models. This result was obtained by the best-performing fine-tuned variant configured as specified in Table III, following full fine-tuning on the complete training dataset.
  • Latency: An average processing time of 1.39 seconds (end-to-end from tokenized input to final token generation).
• Efficiency Gains: The fine-tuning process improved the model's accuracy from an initial 27.06% to 97.63%, demonstrating that domain-specific adaptation of 7B-scale models on 4-bit quantized hardware is highly effective.

Significance and Claims

The authors claim that for narrowly scoped command classification tasks in time-critical environments, domain-specific fine-tuning of lightweight local language models can outperform larger, online-hosted systems.

Key implications highlighted in the paper include:

• Deterministic Performance: By operating entirely on-device, Jarvis eliminates network dependency, ensuring consistent behavior regardless of connectivity conditions.
• Edge Viability: The successful deployment of a fine-tuned 7B model on a laptop-grade GPU (RTX 3060) proves that high-performance intent recognition is feasible on resource-constrained edge hardware.
• Safety and Control: The system prioritizes safety through a confirmation loop and strict command validation, making it suitable for high-speed racing where unintended actions are critical.

The paper concludes that while the current system is limited to predefined command spaces and synthetic data augmentation, it provides a robust, open-source foundation for interactive voice control in autonomous systems where low latency and network independence are essential. Future work is proposed to include multi-turn dialogue and bidirectional telemetry querying.