Technical Summary: TMO – Device-Cloud Collaborative LLM Inference with Three-M Offloading

1. Problem Statement

The deployment of Large Language Models (LLMs) faces a dichotomy: on-device deployment is constrained by computational, memory, and energy resources, often limiting response quality, while cloud deployment introduces latency, communication costs, and usage fees. These challenges are exacerbated in modern applications characterized by Three-M scenarios:

1. Multi-Modal: Tasks require diverse data sources (e.g., text, images from various camera angles).
2. Multi-Task: Applications range from simple message editing to complex activity recognition.
3. Multi-Turn: Conversations involve sequential dependencies where earlier decisions impact future resource availability and context.

Existing frameworks often fail to address the intricate trade-offs between response quality, latency, and cost across these dimensions. Specifically, they struggle with:

• Optimal Inference Location: Deciding whether to process a task on-device or offload to the cloud.
• Modality Selection: Determining which specific data modalities (e.g., which camera view) to upload, balancing information gain against transmission costs.
• Resource Constraints: Adapting to arbitrary, user-specified resource budgets (latency and cost limits) without retraining.
• Inference Uncertainty: Handling the stochastic nature of LLM outputs and the lack of ground-truth metrics for generative tasks during offline training.

2. Methodology: The TMO System

The authors propose TMO, a device-cloud collaborative inference system that orchestrates "Three-M" offloading. The core of TMO is a Resource-Constrained Reinforcement Learning (RCRL) decision engine.

A. System Architecture

• On-Device LLM: A lightweight model (e.g., Phi-3-mini) optimized for high-speed text processing of simple tasks.
• Cloud LLM: A large-scale, multi-modal model (e.g., GPT-4o) capable of handling complex tasks and diverse data inputs.
• Decision Engine: At each conversation turn $t$, the system selects an action $a_t = (\ell_t, M_t)$, where $\ell_t$ is the chosen LLM (device or cloud) and $M_t$ is the subset of modalities to upload.

B. RCRL Formulation

The problem is modeled as a Markov Decision Process (MDP):

• State ($s_t$): Includes a sliding window of the last $\tau$ turns, capturing previous LLM selections, uploaded modalities, and task categories. Crucially, the state is augmented with remaining resource budgets ($\xi_t$) to enable resource-aware decision-making.
• Action ($a_t$): Selection of the LLM type and the specific modalities to upload.
• Reward ($r_t$): A composite metric balancing:
  • Response Score ($S_{at}$): Estimated quality of the LLM response.
  • Association Metric ($\Lambda_{at}$): Relevance between the prompt and selected modalities (computed via CLIP).
  • Latency ($\psi_{at}$): Computation time (device) or communication + inference time (cloud).
  • Usage Cost ($\phi_{at}$): Energy consumption (device) or service fees (cloud).
  • Constraint Penalty: The reward is set to zero if cumulative resource consumption exceeds the user-specified budget $\xi_j$.

C. Key Technical Innovations

1. Uncertainty Handling via Nearest Neighbor Estimation: To address the non-deterministic nature of LLM judges and Out-of-Distribution (OOD) state-action pairs in offline training, TMO employs a k-nearest neighbor (k-NN) strategy. It estimates the response score for unseen state-action pairs by averaging the scores of the $k$ most similar historical pairs, weighted by Euclidean distance.
2. Resource-Aware Generalization: To ensure the policy generalizes to arbitrary user budgets without retraining, the authors:
   • Include remaining budget vectors directly in the state space.
   • Train using randomized resource budgets sampled from distributions (Uniform, Normal, Hybrid) rather than fixed values. This prevents overfitting to specific budget constraints.
3. Task-Modality Association: The system uses pre-trained CLIP models to compute similarity scores between text prompts and image modalities, guiding the selection of the most informative data sources.

3. The M4A1 Dataset

Recognizing the lack of suitable datasets for this specific problem, the authors curated M4A1, the first dataset designed for multi-modal, multi-task, multi-turn, and multi-LLM evaluation.

• Composition: 21,014 conversation turns derived from the ActionSense dataset (kitchen activities with first-person, side, and overhead views) and an instruction dataset.
• Structure: Each episode includes 2–5 turns, four task categories (Assistant, Recommendation, Query, Message Editing), and interactions with four LLM roles (On-device, Cloud, Reference, Judge).
• Generation Strategy: Unlike expert datasets, M4A1 uses randomized actions for LLM and modality selection. This ensures uniform coverage of the state-action space, making the k-NN estimator robust and low-variance for any policy decision.

4. Experimental Results

The authors evaluated TMO (using RC-A2C, RC-PPO, and RC-DQN) against several baselines:

• Naive Baselines: On-device only, Cloud only, and Random selection.
• LLM-as-Router: Using various LLMs (Phi-3, LLaMA, Mistral, GPT-4, etc.) as zero-shot decision engines.
• SOTA Exploration-Decision: AIwRG (Active Inference) and PerLLM (Bandit-based).

Key Findings:

• Performance: TMO (specifically RC-A2C) achieved the highest overall reward (0.86) compared to all baselines, significantly outperforming LLM-as-Router approaches which struggled with constraint violations and suboptimal modality selection.
• Constraint Adherence: TMO maintained near-zero constraint violations across diverse budget configurations, whereas baselines frequently exceeded latency or cost limits.
• Robustness:
  • Device Variations: TMO performed consistently across diverse hardware (from Raspberry Pi to iPhone 15 Pro), adapting its strategy based on device capabilities.
  • Budget Generalization: Training with randomized budgets allowed TMO to generalize to unseen budget constraints (Uniform, Normal, Hybrid) without retraining.
  • Latency Sensitivity: When cloud latency was artificially increased (up to 10x), TMO dynamically shifted traffic to the on-device LLM to maintain constraints.
• Ablation Studies: Removing any key component (LLM selection logic, modality selection, resource awareness, or uncertainty estimation) resulted in significant performance degradation, validating the necessity of the full framework.

5. Significance and Claims

The paper claims that TMO represents a fundamental shift in LLM deployment strategies by moving beyond static offloading or single-query routing. Its significance lies in:

1. Holistic Optimization: It is the first system to jointly optimize LLM selection, multi-modal data offloading, and multi-turn context under strict resource constraints.
2. Practical Adaptability: The resource-aware state representation and randomized training enable the system to adapt to arbitrary user-defined resource budgets in real-world scenarios without retraining.
3. Handling Uncertainty: The proposed nearest-neighbor estimation strategy provides a robust method for training RL policies on LLM inference data, addressing the inherent stochasticity and evaluation challenges of generative models.
4. Dataset Contribution: The release of M4A1 fills a critical gap in the literature, providing a standardized benchmark for evaluating collaborative inference in complex, multi-turn, multi-modal environments.

The authors conclude that while TMO significantly improves the trade-off between response quality, latency, and cost, future work will focus on incorporating dynamic runtime conditions (e.g., real-time network bandwidth, battery levels) to enable proactive adaptation.