Technical Summary: Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction

Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in long-horizon decision-making tasks, such as those found in agentic settings requiring tool use and environmental interaction. However, deploying full-precision LLMs for these complex, multi-step tasks incurs prohibitive inference costs. While quantization offers a promising avenue for efficiency, recent studies indicate that naively applying low-precision quantization throughout an agent's execution leads to significant performance degradation in real-world agentic tasks. This degradation is not uniform; quantized models often perform adequately on standard reasoning steps but fail at specific "critical steps" (e.g., precise placement or complex tool selection), leading to irreversible trajectory-level failures. Existing routing methods are typically designed for static, single-turn queries or operate at a coarse question level, making them unsuitable for the fine-grained, step-level decision-making required in long-horizon agent interactions.

Methodology: Dynamic Mixed-Precision Routing (DMR)

The authors propose Dynamic Mixed-Precision Routing (DMR), a lightweight framework that adaptively selects between high-precision (e.g., BF16) and low-precision (e.g., Int4) LLMs at each decision step of an agent's trajectory. The core insight is that interaction steps exhibit diverse sensitivities to quantization; a router can identify critical steps requiring high precision while allowing non-critical steps to utilize cheaper, low-precision inference.

1. Router Architecture

DMR employs a lightweight Transformer router (decoupled from action generation) that takes trajectory-level inputs (task description, history of actions/observations) and outputs a precision choice ($r_t \in \{1, \dots, K\}$) for the current step. The router is parameterized by $\theta$ and is significantly smaller than the routed LLMs (approximately 2–3% of their parameters).

2. Two-Stage Training Pipeline

The router is trained via a two-stage process:

• Stage 1: KL-Divergence–Based Supervised Learning (KL-ST)

  • Motivation: Low-precision models generally match high-precision behavior but exhibit large deviations at critical steps.
  • Process: The authors collect trajectories using high-precision rollouts. At each step, they compute the Kullback-Leibler (KL) divergence between the action distributions of the low-precision model and the high-precision reference.
  • Labeling: The KL distribution is observed to be bimodal (dense low-divergence region vs. sparse high-divergence tail). A threshold $\tau$ is applied to convert continuous KL values into binary labels ($y_t=1$ for high-precision needed, $0$ otherwise).
  • Training: The router is fine-tuned using class-weighted cross-entropy to predict these labels, serving as a warm start.
  • Calibration: To address distribution shifts between training (high-precision history) and deployment (mixed-precision history), a quantile-based threshold is computed on a held-out calibration pool to meet specific high-precision budget constraints ($\rho$).

• Stage 2: Group-Relative Policy Optimization (GRPO)

  • Motivation: KL-ST optimizes for local step sensitivity, but the ultimate goal is trajectory-level task success under a latency budget.
  • Process: Starting from the KL-ST router, the policy is refined using GRPO. This method samples groups of trajectories and optimizes a reward function that balances task success with execution cost.
  • Reward Function: $R(\tau) = \mathbb{I}[\text{success}] - \lambda \cdot C(\tau)$, where $C(\tau)$ is the total execution time calculated based on the number of steps, tokens generated by high/low precision models, and router overhead. GRPO does not require learning a separate value function, making it suitable for sparse, long-horizon rewards.

Key Contributions

1. Dynamic Mixed-Precision Framework: A novel approach that substantially reduces inference costs for long-horizon agentic tasks while maintaining task success rates, addressing the mismatch between static quantization and dynamic agent requirements.
2. Lightweight Router: Introduction of a router with only ~2–3% of the routed LLM's parameters, sufficient to identify precision-critical steps and learn an effective routing policy.
3. RL-Based Optimization: The use of GRPO to jointly optimize task performance and inference cost, moving beyond static supervision to learn a policy that balances the Pareto frontier of accuracy and latency.

Experimental Results

The framework was evaluated on two text-based agentic benchmarks: WebShop (online shopping) and ALFWorld (embodied household tasks), using Qwen3 models (1.7B and 4B) with AWQ and GPTQ quantization.

• Accuracy-Cost Trade-off: DMR consistently outperforms "always-quantized" baselines. In many configurations (e.g., 4B AWQ on ALFWorld), DMR matches or surpasses the full-precision (BF16) success rates while requiring significantly less time.
• Recovery from Degradation: In scenarios where the always-quantized baseline performs poorly (e.g., 1.7B GPTQ on WebShop), DMR recovers up to 93% of the performance gap between the quantized and full-precision models.
• Budget Control: The high-precision budget parameter ($K$) effectively controls the trade-off. Lower $K$ values prioritize latency reduction, while higher $K$ values recover more accuracy.
• Ablation Studies:
  • RL Refinement: GRPO refinement generally yields the best operating points compared to the supervised KL-ST router alone, particularly in complex settings.
  • Router Depth: Router capacity is not a bottleneck; an 8-layer router was found to be near-optimal, with deeper or shallower routers showing negligible or negative impacts on performance.
  • Threshold Sensitivity: The choice of the KL-threshold $\tau$ is robust within a "sparse region" of the KL distribution, but fails if the threshold cuts through a dense region of similar values.

Significance and Claims

The paper claims that Dynamic Mixed-Precision Routing (DMR) effectively revisits precision selection from a sequential decision-making perspective. It demonstrates that dynamic, step-level precision control is a viable strategy for agentic settings, allowing agents to leverage the complementary strengths of full-precision and quantized models. By identifying and selectively applying high precision only to critical steps, DMR achieves a superior accuracy-cost trade-off compared to single-precision baselines. The authors position this work as a step toward making long-horizon LLM agents more computationally feasible without sacrificing the robustness required for complex, real-world interactions. Future work is suggested to extend DMR to richer precision sets and deployment environments with native low-bit kernels.