Technical Summary: INAR-VL

Problem Statement

The deployment of Vision-Language Models (VLMs) in edge environments (e.g., mobile devices, robots, embedded cameras) faces a fundamental trade-off between latency/energy efficiency and prediction accuracy. Cloud execution offers high accuracy but incurs significant communication delays and energy costs, while edge-only execution is faster and more energy-efficient but suffers from limited model capacity and reduced accuracy, particularly on visually degraded inputs or complex reasoning tasks.

Existing routing approaches often rely on static placement decisions or text-only difficulty estimation, failing to account for the heterogeneity of multimodal inputs. Specifically, they do not adequately capture visual degradation effects (e.g., blur, lighting, compression artifacts) which significantly impact model performance. Furthermore, different VLM architectures exhibit complementary strengths (e.g., some excel at OCR/fine-grained recognition, others at reasoning), yet static systems cannot leverage this diversity dynamically. The challenge is to design a routing system that makes pre-inference decisions based on lightweight input signals to optimize model selection, input resolution, and execution placement (edge vs. cloud) without incurring the overhead of running expensive VLMs for characterization.

Methodology: INAR-VL

INAR-VL is a lightweight, input-aware routing system designed for a two-tier edge-cloud deployment. It operates by characterizing incoming requests before invoking any heavy VLM inference.

1. System Architecture

The system maintains a heterogeneous pool of four VLMs: two compact models (Qwen3-VL-2B, SmolVLM-2B) deployed on the edge (8 GB VRAM, INT8 precision) and two larger models (LLaVA-OV-8B, Qwen3-VL-8B) on the cloud (FP16 precision).

The routing pipeline consists of:

• Feature Extraction: Upon receiving an (image, question) pair, the system extracts a 10-dimensional feature vector prior to inference:
  • Visual Features (4): Blur, exposure, JPEG artifacts, and detail signals.
  • Text Features (6): Length, entity/numeric density, question type, vocabulary complexity, reasoning markers, and context-dependency markers.
• Feature Fusion: These signals are normalized and merged into a request descriptor.
• Pareto-Based Routing: The router evaluates valid configurations (model, precision, resolution, placement) to construct a per-request Pareto frontier. It predicts answer quality and estimated cost (latency, energy, network overhead) for each candidate.
• Decision Logic: The system prioritizes edge execution if the predicted quality is sufficient. Offloading to the cloud occurs only if the expected quality gain exceeds a calibrated threshold ($\delta_{min} = 0.03$) and justifies the additional cost.

2. Quality and Cost Prediction

The system uses a calibrated quality predictor ($\hat{q}$) and cost estimator ($\hat{c}$).

• Quality Prediction: The predicted quality accounts for the base accuracy of the model, precision adjustments, resolution scaling, and penalties for mismatches between input needs (e.g., high blur, high detail) and model strengths (blur robustness, detail/OCR capability, reasoning ability).
• Cost Estimation: Combines latency (proportional to model size and resolution), energy (based on GPU power and execution tier), and network terms (image transfer overhead).
• Calibration: Parameters are calibrated offline using a held-out dataset (1,000 samples per benchmark) to fit coefficients for resolution scaling, mismatch penalties, and cost weights. This calibration is hardware-aware but performed only once per deployment configuration.

Key Contributions

1. Input-Aware Multimodal Routing: INAR-VL introduces a routing mechanism that utilizes lightweight signals from both image quality and text complexity to guide decisions, addressing the heterogeneity of multimodal workloads that text-only or static approaches miss.
2. Joint Optimization: Unlike prior work that optimizes a single dimension, INAR-VL jointly optimizes model architecture, input resolution, and execution placement (edge vs. cloud) under practical constraints (memory, energy, network variability).
3. Deployment-Aware Policy: The system employs an edge-preference rule that explicitly balances accuracy recovery against latency and energy costs, ensuring that offloading only occurs when beneficial.
4. Practical Implementation: The system is implemented and evaluated on real hardware (NVIDIA RTX 4060 edge, RTX PRO 6000 cloud) with a routing overhead of only 10–30 ms.

Experimental Results

The system was evaluated on three Visual Question Answering (VQA) benchmarks: VQAv2, TextVQA, and GQA, using 2,000 samples per dataset.

• Accuracy vs. Efficiency: INAR-VL achieved a mean accuracy of 72.1%, recovering 71% of the accuracy gap between edge-only (66.5%) and cloud-only (74.4%) execution.
• Routing Distribution: The system executed 36% of requests on the edge, effectively filtering out low-complexity queries locally.
• Performance Gains: Compared to cloud-only execution, INAR-VL reduced average latency by 24% (from 2408 ms to 1826 ms) and energy consumption by 26% (from 26.0 J to 19.2 J).
• Ablation Studies:
  • Signal Importance: Text features were found to be the dominant signal; a text-only router achieved 71.4% accuracy, while an image-only router dropped to 68.6%.
  • Model Diversity: Using a diverse pool of models (Qwen and LLaVA architectures) improved mean accuracy by 0.9 percentage points compared to a static routing strategy using only Qwen models.
• Robustness: The system demonstrated stability across varying network bandwidths (10–1000 Mbps). A bandwidth guard (15 Mbps) ensures edge-only execution when connectivity is poor, maintaining consistent accuracy while latency scales with bandwidth.

Significance and Claims

The paper claims that INAR-VL provides a practical solution for deploying VLMs in resource-constrained edge environments. By explicitly modeling the trade-offs between input heterogeneity, model capabilities, and deployment constraints, the system achieves a superior latency-accuracy-energy trade-off compared to static baselines.

The authors emphasize that while the system does not reach the "Oracle" performance (90.6% accuracy, which selects the best model post-hindsight), it recovers the majority of the cloud's accuracy advantage while significantly reducing operational costs. The approach is presented as hardware-agnostic and generalizable; new hardware tiers or models can be integrated by recalibrating routing thresholds without altering the core methodology. The work highlights that input-aware routing is essential for multimodal workloads where static placement fails to adapt to the variability of visual and textual complexity.