Less Is More: Engineering Challenges of On-Device Small Language Model Integration in a Mobile Application

1. Problem Statement

The paper addresses the significant gap between the theoretical readiness of on-device Small Language Models (SLMs) and the practical engineering reality of integrating them into production mobile applications. While frameworks like LiteRT-LM and MLC LLM enable running models (0.6B–4B parameters) locally on smartphones, offering zero-latency privacy and offline capability, developers lack empirical guidance on handling the specific failure modes of SLMs.

Unlike cloud-based Large Language Models (LLMs), SLMs exhibit distinct qualitative failure modes:

• Unreliable Format Compliance: They frequently wrap JSON in markdown code fences, translate keys, or produce malformed UTF-8.
• Weak Constraint Adherence: They struggle with numerical reasoning (e.g., counting word letters) and strict language constraints.
• Context Degradation: Performance degrades rapidly within a single session due to limited KV cache.
• Latency Issues: Generating complex structured outputs (e.g., full puzzles) can block the UI for minutes on mobile hardware.

The core question is: Can on-device SLMs deliver a viable, private, offline user experience in a production app, and what engineering strategies are required to make them reliable?

2. Methodology

The authors conducted a longitudinal practitioner case study over a 5-day development sprint (April 21–26, 2026).

• Subject System: Palabrita, a Wordle-style word-guessing game for Android (Kotlin/Jetpack Compose).
• Data Source: The complete Git history of the project, comprising 204 commits (approx. 90 AI-related), serving as a tamper-evident, contemporaneous record of every design decision, failure, and course correction.
• Models Evaluated: Nine models were tested (Gemma 4 E2B, Qwen3 0.6B, Phi-4 Mini, etc.), with the final production build shipping two: Gemma 4 E2B (2.6B params, premium tier) and Qwen3 0.6B (600M params, compact tier).
• Framework: LiteRT-LM (Google DeepMind) was selected for its full prompt control, model flexibility, and minimal integration friction compared to MLC LLM or AICore.
• Process: The team followed a "Spec Driven Development" workflow, iterating from an ambitious initial design to a pragmatic final architecture based on observed failures.

3. Key Contributions

The paper provides five major contributions to the field of mobile AI engineering:

1. A Taxonomy of Five Failure Categories:

   • F1: Output Format Violations: Markdown wrapping, key translation, and UTF-8 corruption.
   • F2: Constraint Violations: Incorrect word lengths, wrong languages, and repetition.
   • F3: Context Quality Degradation: Output quality drops after 3–5 generations due to KV cache saturation.
   • F4: Latency Incompatibility: UI blocking during synchronous generation (1–3 minutes).
   • F5: Model Selection Instability: Drastic quality variations between models, requiring unique prompt tuning for each.

2. The "Progressive Simplification Arc":
   A documented trajectory where the LLM's responsibility was systematically reduced from generating a full puzzle (word + category + difficulty + 5 hints) to only generating 3 hints for a pre-selected word. This arc demonstrates that reliability is achieved by minimizing the LLM's scope.

3. Prompt Engineering Iteration Process:
   A detailed account of four complete prompt rewrites in five days, highlighting specific techniques for SLMs:

   • Using "CRITICAL" priority markers.
   • Providing concrete examples (e.g., "estufa has 6 letters") rather than abstract rules.
   • Using full language names ("Brazilian Portuguese") instead of ISO codes ("pt").
   • Repeating constraints in both system and user prompts.

4. Architectural Anti-Pattern Resolution:
   A case study on the failure of using Android WorkManager for foreground LLM operations. The authors identified a mismatch between WorkManager's persistent state model and the real-time, in-memory nature of LLM inference, leading to 7 bugs in a single day. This was resolved by replacing WorkManager with Kotlin Coroutines for a cleaner, stateless generation flow.

5. Eight Actionable Design Heuristics:
   Practical guidelines for practitioners, including:

   • Minimize output schema complexity.
   • Stack multiple parsing fallbacks (defensive parsing).
   • Use contextual feedback in retries (include rejection reasons).
   • Rotate inference sessions to prevent degradation.
   • Always provide a deterministic fallback (the LLM should never gate the user experience).

4. Results and Findings

• Viability: On-device SLMs are viable for production, but only if the task is scoped to what the model can do reliably. The "sweet spot" identified was bounded creative text generation (writing hints) with deterministic fallbacks, rather than constrained factual generation (selecting valid words).
• Success Metrics:
  • Latency: Reduced from 15–30 seconds per puzzle (full generation) to 1–2 seconds (hint-only generation).
  • Reliability: By moving word selection to curated JSON assets, word-length violations dropped from ~30–50% to 0%.
  • Architecture: The final system uses a "Hint-Only" architecture where the LLM enhances the experience but does not gate it. A deterministic fallback provider ensures the game is fully playable even if the LLM fails 100% of the time.
• Parsing Evolution: The parsing pipeline evolved from a single deserializer to a 5-layer defensive pipeline (UTF-8 sanitization, code fence stripping, direct decode, regex extraction, and structural parsing) to handle SLM output variability.

5. Significance

• Engineering Reality Check: The paper challenges the assumption that SLMs are simply "smaller cloud models." It proves they require fundamentally different engineering approaches: defensive parsing instead of schema validation, contextual retries instead of blind repetition, and responsibility reduction instead of prompt elaboration.
• Privacy and Economics: The authors argue strongly for Edge-First AI. On-device inference eliminates the privacy risks of sending user data to the cloud and removes the recurring costs of API tokens, which are unsustainable at scale for consumer apps.
• Practical Guidance: By documenting the "5-day discovery process" and the specific heuristics derived from it, the paper allows other developers to skip the trial-and-error phase and implement robust on-device AI architectures immediately.
• Framework Implications: The findings suggest that inference frameworks need built-in output format enforcement (JSON mode), session quality metrics, and better model compatibility metadata to support mobile developers effectively.

Conclusion: The paper concludes that "Less is More." The most reliable on-device LLM feature is one where the LLM does the least. Success requires accepting that SLMs are imperfect tools that must be wrapped in robust, deterministic, and fallback-heavy architectures.