From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service

1. Problem Statement

Multilingual intent classification is critical for global logistics customer service, where systems must route user queries (e.g., tracking, refunds, claims) across diverse languages and hierarchical label spaces. However, existing benchmarks suffer from a significant synthetic-to-native gap:

• Data Source Bias: Most multilingual datasets (e.g., MTOP, MASSIVE) rely on machine-translated text or controlled data collection. These are typically cleaner, more standardized, and grammatically correct compared to real user queries.
• Evaluation Mismatch: Real-world customer queries are often short, noisy, contain shorthand, misspellings, code-mixing, and colloquialisms. Machine-translated test sets fail to capture this "noise," leading to overly optimistic estimates of model robustness.
• Domain Specificity: Existing datasets often focus on multi-turn dialogue or sentiment analysis rather than stand-alone, single-turn intent routing with complex hierarchical taxonomies required in logistics.

2. Methodology

A. Dataset Construction (The Benchmark)

The authors constructed a new public benchmark derived from 600,000 historical logistics customer-service logs, resulting in a curated dataset of approximately 30,000 de-identified, stand-alone user queries.

• Curation Pipeline:
  1. Filtering: Removal of personally identifiable information (PII), bot templates, duplicates, and malformed queries using rule-based filters and embedding-based semantic similarity.
  2. Quality Control: A semi-automatic pipeline using LLMs for pre-screening and consistency checks, followed by human verification.
  3. Annotation: Two annotators per language with a third expert for adjudication, achieving 95% agreement on a test subset.
• Taxonomy: The dataset is organized into a two-level hierarchy:
  • 13 Parent Intents (Coarse-grained, e.g., "Shipment Tracking").
  • 17 Leaf Intents (Fine-grained, e.g., "Delivery Failure / Status Dispute").
• Language Coverage:
  • Seen Languages: English, Spanish, Arabic (used for training).
  • Unseen (Zero-Shot) Languages: Indonesian, Chinese, and others (held out for zero-shot evaluation).
• Synthetic-to-Native Pairing: A unique feature is the provision of paired test sets:
  • Native: Real, noisy user queries.
  • Synthetic: Machine-translated versions of a filtered source subset (using TranslateGemma-27B), manually verified for intent preservation but lacking native noise.

B. Experimental Protocol

The authors benchmarked various model families under three task settings:

1. 13-way Parent Classification: Coarse-grained routing.
2. 17-way Flat Classification: Fine-grained routing without hierarchy.
3. 17-way Hierarchical Classification: Predicting leaf intents while respecting the parent-child structure.

Models Evaluated:

• Multilingual Encoders: mBERT, XLM-R, and an internal optimized variant (MMBERT).
• Embedding-based Classifiers: EmbeddingGemma and Gemma 3 270M (embedding + lightweight classifier).
• Instruction-Tuned Small Language Models (SLMs): Gemma 3 (270M, 1B), Qwen 3 (0.6B).
• Closed-Source LLMs: Seed-1.8, DeepSeek V3.2, Qwen-Max-Preview (evaluated via few-shot prompting).

Evaluation Metrics:

• Primary: Accuracy / Micro-F1 (reflecting real-world routing needs where overall accuracy is paramount).
• Secondary: Macro-F1 (to analyze long-tail performance).
• Conditions: Zero-shot transfer to unseen languages and direct comparison of Native vs. Translated test sets.

3. Key Contributions

1. Real-World Benchmark: Release of the first public, hierarchical multilingual intent classification benchmark derived from actual logistics traffic, preserving natural noise and distributional properties.
2. Synthetic-to-Native Gap Analysis: Introduction of a controlled evaluation framework using paired native and translated test sets to explicitly measure how much machine translation inflates performance metrics.
3. Comprehensive Benchmarking: Evaluation of encoders, embedding models, and SLMs across flat and hierarchical protocols, specifically focusing on long-tail intents and cross-lingual transfer.
4. Empirical Evidence of Overestimation: Demonstration that translated benchmarks systematically overstate model robustness compared to native queries.

4. Key Results

A. The Synthetic-to-Native Gap

• Performance Inflation: Machine-translated test sets consistently yield higher accuracy than native test sets across all model families.
• Impact on Long-Tail: The gap is most pronounced for long-tail intents and cross-lingual transfer. Translated text smooths out the noise (slang, typos) that causes real-world failures, leading to misleadingly high robustness scores.
• Example: A translated query might be grammatically perfect, while the native equivalent uses colloquial phrasing that confuses the model, resulting in a significant performance drop in the native setting.

B. Model Performance Trends

• SLMs vs. Encoders: Modern Small Language Models (SLMs) (specifically Gemma 3 270M and 1B) outperformed traditional classification-specialized encoders (like mBERT and XLM-R) in several settings, including zero-shot transfer to Chinese and Thai.
  • Notable Finding: The 270M Gemma model outperformed the 307M parameter MMBERT in Chinese evaluation conditions.
• Scaling Laws: Performance generally scaled with model size (Gemma 3 1B > Gemma 3 270M > Qwen 3 0.6B).
• Hierarchical Benefits: Hierarchical decoding provided modest gains for SLMs, whereas BERT-style classifiers required complex constrained decoding to achieve similar results.
• Zero-Shot Transfer: SLMs demonstrated superior ability to generalize to unseen languages (Indonesian, Chinese) without target-language supervision compared to traditional encoders.

C. Granularity Challenges

• Transitioning from 13-way parent tasks to 17-way fine-grained tasks caused larger performance drops for BERT-style models compared to SLMs, highlighting the difficulty of disambiguating operationally similar intents in noisy, real-world data.

5. Significance and Implications

• Re-evaluating Robustness: The paper argues that relying solely on translated benchmarks leads to a false sense of security regarding model deployment. Real-world robustness must be measured against native, noisy data.
• Shift in Architecture: The results suggest a paradigm shift in production NLU systems. Lightweight, instruction-tuned Generative LLMs (SLMs) are becoming competitive with or superior to specialized discriminative classifiers for intent routing, offering better zero-shot capabilities and easier taxonomy updates.
• Industry Impact: The authors have already begun shifting their online logistics system from legacy BERT pipelines to Gemma-based models based on these findings.
• Future Directions: The benchmark encourages the community to move beyond synthetic data, focusing on long-tail distributions, noise robustness, and genuine cross-lingual transfer in low-resource scenarios.

In conclusion, this work provides a critical reality check for multilingual NLU, offering a high-fidelity dataset and demonstrating that native evaluation is essential for accurately assessing the readiness of AI models for global logistics customer service.