Technical Summary: LoCar – Localization-Aware Evaluation of In-Vehicle Assistants

Problem Statement

The integration of Large Language Models (LLMs) into Software-Defined Vehicles (SDVs) presents a critical challenge: existing benchmarks prioritize general reasoning and knowledge over the specific sociolinguistic and safety requirements of automotive deployment. In markets like South Korea, where cultural context and linguistic nuance are paramount, standard translation is insufficient. Current evaluation frameworks fail to address:

1. Fine-grained Sociolinguistic Control: The inability of LLMs to consistently maintain specific speech levels (honorifics) required for premium brand identity and user trust.
2. Strategic Dialogue Management: The lack of standards for evaluating proactive, safety-critical, and context-aware interaction behaviors in real-world driving scenarios.
3. Deployment-Oriented Metrics: The absence of evaluation criteria that balance efficiency, clarity, and safety under the cognitive load constraints of driving.

Methodology

The authors propose LoCar (Localization-aware evaluation for Car assistants), a framework specifically designed for Korean in-vehicle AI systems. The methodology consists of three core components:

1. Dataset Taxonomy and Construction

The framework defines a comprehensive taxonomy centered on two primary use cases: Car Expert (vehicle operation, diagnostics, and manual-based knowledge) and Navigation (route planning and situational awareness).

• Data Generation: A synthetic dataset was constructed using official owner's manuals and navigation resources. It includes 13 Key Performance Indicators (KPIs) organized into two layers:
  • Linguistic Style Layer: Evaluates Conciseness and three distinct honorific registers (Hae, Haeyo, Hapsyo).
  • Dialogue Competence Layer: Evaluates Single-turn capabilities (Implicit Understanding, Contextual Comprehension, Harmful Question Response) and Multi-turn capabilities (Clarification, Retention, Refinement, Reflection, Proactiveness, Troubleshooting).
• Augmentation: All instances were augmented with three Korean speech levels to test register consistency, resulting in a dataset of over 12,000 instances.

2. Hybrid Evaluation Pipeline

To ensure rigorous and reliable assessment, the authors developed an automated pipeline using an LLM-as-a-Judge paradigm, calibrated against human annotations.

• Human Calibration: A "golden set" of 803 human-annotated instances was created, with three annotators per sample. Inter-annotator agreement (IAA) was moderate to substantial for most metrics, though lower for subjective tasks like Clarification.
• Hybrid Honorific Verification: Recognizing that LLMs struggle to distinguish morphologically similar Korean honorifics (e.g., Haeyo vs. Hapsyo), the pipeline integrates a rule-based morphological suffix checker alongside contextual LLM reasoning. This hybrid approach improved human-judge agreement for honorific classification from 0.69 to 0.94.
• Ensemble Judging: Three heterogeneous LLMs (Gemini-2.5-Flash, GPT-5-mini, DeepSeek-v3.1) were selected based on their consistency with human annotations. Their decisions are aggregated via majority voting to mitigate model-specific bias.
• Gating Mechanism: A "Meaningful Information Inclusion" metric acts as a gate; if a response lacks substantive content, all dialogue competence metrics (except safety responses) are excluded from scoring.

3. Experimental Setup

The framework was applied to evaluate 11 state-of-the-art LLMs (including Korean-developed models and global APIs) across single-turn and multi-turn scenarios. Models were instructed to generate responses in randomly assigned speech levels, and compliance was measured against the 13 KPIs.

Key Results

The evaluation of 11 models revealed distinct performance patterns:

• Instability in Fine-Grained Honorifics: Despite generally high compliance with polite speech, models frequently conflated Haeyo and Hapsyo. This indicates that maintaining fixed, specific speech levels remains a significant linguistic challenge, even when general politeness is preserved.
• Single-Turn vs. Multi-Turn Disparity: Single-turn understanding metrics (Implicit Understanding, Contextual Comprehension) approached saturation across models. In contrast, multi-turn strategic metrics (Clarification, Proactiveness) exhibited significantly higher variance and lower average scores.
• Conservative Evaluation Bias: The framework demonstrated a conservative stance in ambiguous scenarios. For strategic tasks like Clarification and Proactiveness, the system favored stricter acceptance criteria, validating only interventions that met high safety and contextual certainty standards. This resulted in lower recall for these metrics compared to consistency-oriented tasks like Retention or Refinement.
• Latency vs. Performance: Analysis showed that inference latency did not consistently correlate with superior multi-turn strategic performance, suggesting that model size alone does not guarantee better conversational steering.

Significance and Contributions

The paper claims three primary contributions:

1. The LoCar Framework: The introduction of the first car-domain evaluation framework and dataset targeting localized sociolinguistic requirements, defining 13 KPIs to systematically assess linguistic realization and dialogue competence.
2. Empirical Evidence of Instability: The demonstration that fine-grained honorific control is unstable in current LLMs, highlighting that precise speech-level realization must be explicitly evaluated rather than assumed.
3. Deployment-Oriented Evaluation Strategy: The identification of a performance gap in strategic dialogue abilities, suggesting that deployment-level evaluation requires a conservative bias to prioritize reliability and safety over mere conversational fluency.

The authors conclude that automotive AI must evolve beyond general competence toward precise linguistic tailoring and reliable, safety-oriented interaction management. They emphasize that deployment-level evaluation must account for locally encoded sociolinguistic norms (such as Korean honorifics) and domain-specific constraints (cognitive load, safety) to ensure culturally resonant and safe global deployment.

Limitations: The authors note that the framework is currently exclusive to the Korean language and market, relying on morphological suffix detection that does not directly transfer to languages with different politeness encoding. Furthermore, the evaluation is conducted in an offline, text-based setting, excluding real-world factors like speech recognition errors and TTS rendering.