Technical Summary: Workload-Driven Optimization for On-Device Real-Time Subtitle Translation

Problem Statement

This work addresses the specific constraints of real-time, on-device subtitle translation from English to Traditional Chinese (Taiwan variant). Unlike general-purpose language model serving or long-context machine translation, this workload is characterized by:

• Short inputs and outputs: Requests consist of a single current subtitle cue with limited preceding context (up to three cues).
• Batch-size-one inference: Interactive playback prevents the use of large-batch optimizations.
• Strict latency and privacy: Translations must be generated within the subtitle display window on consumer hardware (e.g., Apple M-series) without transmitting data to the cloud.

Under these conditions, traditional optimizations for long attention sequences or high throughput offer limited benefit. Instead, fixed per-request overheads, decode-time computation, and vocabulary projection costs become the primary bottlenecks. Preliminary profiling indicates that after applying GGUF quantization (reducing Transformer block costs), the vocabulary projection step becomes a dominant factor in decode latency.

Methodology

The proposed solution, LocalSubs, is built upon the LMT-60-0.6B multilingual translation model. The optimization strategy is workload-driven, focusing on reducing the output projection dimension and improving token density for the specific domain.

1. Vocabulary and Tokenizer Redesign

The original 151k-token vocabulary was replaced with a custom 64k-token ByteLevel BPE tokenizer trained specifically on English and Taiwan Traditional Chinese subtitle text.

• Impact: This reduces the output projection dimension by approximately 57.9% (from 151,936 to 64,024).
• Efficiency: The new tokenizer increases Chinese character density (1.40 characters per token vs. 1.05), reducing the number of tokens required to generate the same text.

2. Embedding Migration and Two-Stage Adaptation

Replacing the tokenizer invalidates the original embedding matrix. The authors propose a specific adaptation pipeline:

1. Embedding Migration: New token embeddings are initialized via string-level correspondence. Existing tokens are copied directly; new tokens are initialized by averaging the embeddings of their sub-token decompositions from the original tokenizer.
2. Embedding Calibration: A two-stage adaptation follows. First, Transformer layers are frozen while embeddings, the output head, and RMSNorm parameters are updated to align the new embedding space with the pretrained semantic space.
3. Full Supervised Fine-Tuning (SFT): The entire model is then fine-tuned on a quality-filtered, length-rebalanced subtitle dataset (233,088 examples).

3. Evaluation Protocol

• Quality: Evaluated on a fixed 500-example subset of the OpenSubtitles2024 test set using pairwise preference judgments against a fixed Google Translate anchor, judged by GPT-4o.
• Performance: Preliminary latency measurements were conducted on an Apple M2 using llama.cpp with GGUF Q5_K_M quantization and the Metal backend.

Key Results

Translation Quality

• Win Rate: LocalSubs achieved a 59.2% tie-excluded win rate against Google Translate.
• Comparison: This is comparable to the 64.6% win rate achieved by GPT-4o mini under the same evaluation protocol, despite LocalSubs operating under strict on-device, batch-size-one constraints without remote API reliance.
• Context Utilization: The model benefits significantly from context on short cues. On a subset of 260 short responses, the win rate with context (73.1%) was notably higher than without context (68.6%).
• Cue Length Dependency: Performance is strongest on short cues (82.0% win rate for 1–3 words) and declines as cue length increases, dropping to 45.7% for 8–15 words and 12.5% for 16+ words.

Latency and Throughput

• Speedup: A preliminary profiling model with the 64k vocabulary showed a 1.63× speedup in average end-to-end latency (56.8 ms vs. 92.7 ms) compared to the 151k-vocabulary baseline on Apple M2 Metal.
• Throughput: Decode throughput increased from 63.3 to 96.0 tokens/s (1.52×).
• Token Reduction: The 64k model generated 40.0% fewer output tokens (4.00 vs. 6.67) and used 7.7% fewer prompt tokens for English input.

Significance and Claims

The paper posits that for short-input, low-latency on-device workloads, tokenizer redesign is a critical serving architecture decision, not merely a preprocessing choice. By reducing the vocabulary size, the system directly lowers the computational cost of the output projection layer, which becomes the bottleneck after quantization.

The work demonstrates that a compact 0.6B model, when adapted with a domain-specific tokenizer and a structured embedding migration strategy, can achieve translation quality competitive with commercial cloud models (GPT-4o mini) while satisfying strict privacy and latency constraints.

Limitations and Future Work

The authors maintain a modest stance regarding their claims:

• Reproducibility: The latency results are preliminary; the raw logs, specific llama.cpp commits, and model checksums were not fully retained for the final evaluated model.
• Evaluation: The pairwise judgments rely on GPT-4o without a recorded snapshot, and human auditing is still required. The comparison between LocalSubs and GPT-4o mini was indirect (both vs. Google Translate), not a direct head-to-head comparison.
• Performance Gaps: The model struggles with longer cues, often omitting details or failing to preserve full meaning.
• Future Directions: The authors prioritize controlled comparisons across different vocabulary sizes (32k/64k/151k) under identical settings, full scenario-stratified evaluation, and the construction of a manually verified error taxonomy.