Technical Summary: VOILA – Value-of-Information Guided Fidelity Selection for Cost-Aware Multimodal Question Answering

1. Problem Formulation

Modern multimodal vision-language systems (VLMs) achieve high accuracy by scaling model parameters and training data, but this incurs substantial storage, bandwidth, and computational overheads. While recent work has focused on adaptive model routing (dynamically selecting between lightweight and heavyweight models), these approaches implicitly assume that the input context is fixed and freely available.

In practice, multimodal inputs often reside across heterogeneous storage tiers with varying retrieval latencies and costs. For instance, mobile services store compressed thumbnails and captions on-device while full-resolution images reside in the cloud; enterprise systems use hot storage for frequent data and cold storage for archives. Crucially, these retrieval costs are incurred before any model processes the input. Consequently, post-hoc model routing is insufficient for end-to-end efficiency.

The paper formalizes VOILA (Value-Of-Information guided Latency-Aware fidelity selection) as a cost-sensitive information acquisition problem. The core challenge is to determine which representation of the available context should be retrieved before model execution, rather than just which model should answer the query. The objective is to select a fidelity $f$ from a discrete, ordered set $F$ (ranging from low-cost captions/thumbnails to high-cost full-resolution images) that maximizes expected utility:
$$f^* = \arg \max_{f \in F} \left( \text{Pr}(\hat{a}_f \text{ is correct} | q) - \lambda c(f) \right)$$
where the probability of correctness must be estimated without retrieving $f$ or running the VLM, and $c(f)$ represents the acquisition cost (latency, bandwidth, storage).

2. Methodology

VOILA employs a three-stage pipeline that operates entirely on question features prior to visual retrieval, making it model-agnostic and computationally lightweight.

2.1 Question-Conditioned Success Scoring

The system first estimates the likelihood that a specific fidelity will yield a correct answer based solely on the question text $q$.

• Feature Extraction: The system extracts TF-IDF features, lexical statistics (length, numeric indicators), and coarse question types (e.g., counting, color, spatial).
• Regression Model: A Gradient Boosted Regressor (GBR) is trained for each fidelity level $f$ to predict a raw success score $r_f(q)$. This model learns statistical patterns between question characteristics and cross-fidelity success rates during offline training.
• Efficiency: At inference, this step requires only a forward pass through a lightweight tree ensemble, avoiding visual retrieval or VLM execution.

2.2 Fidelity-Specific Probability Calibration

Raw regression scores are not calibrated probabilities and may suffer from systematic over- or under-confidence.

• Isotonic Regression: To produce reliable probabilities $\hat{p}_f(q) \in [0, 1]$, the system applies isotonic regression to map raw scores to empirical success frequencies on a held-out calibration set.
• Properties: This non-parametric approach preserves the rank ordering of the regressor, corrects arbitrary systematic biases, and provides proven convergence guarantees. This step is critical for distinguishing queries that can be answered at low fidelity from those requiring high fidelity.

2.3 Value-of-Information (VOI) Selection

Using the calibrated probabilities, VOILA makes a greedy decision to select the minimum-cost fidelity.

• Decision Rule: Starting from the lowest-cost fidelity, the system evaluates the Value of Information (VOI) for escalating to the next higher fidelity $f_j$:
  $$\text{VOI}(f_j | f_i) = [\hat{p}_{f_j}(q) - \hat{p}_{f_i}(q)] - \lambda c(f_j)$$
• Escalation: The system escalates to a higher fidelity only if the expected accuracy gain outweighs the acquisition cost ($\text{VOI} > 0$).
• Execution: Once the optimal fidelity is selected, the visual representation is retrieved, and the VLM is executed exactly once.

3. Key Contributions

The paper makes four primary technical contributions:

1. Empirical Characterization: The authors demonstrate that information requirements vary systematically across query types (e.g., OCR and counting tasks require high fidelity, while semantic yes/no questions do not) and that existing heuristics (confidence-based escalation, model scaling) fail to address these patterns. Specifically, model confidence is shown to be an unreliable signal for context sufficiency when visual information is missing.
2. VOILA Framework: A lightweight, two-stage system combining gradient-boosted prediction with isotonic calibration to enable query-adaptive fidelity selection with bounded utility regret.
3. Problem Formalization: The paper formalizes pre-retrieval context selection as a cost-sensitive information acquisition problem and provides a theoretical analysis showing that VOILA implements a near Bayes-optimal policy when probability estimates are calibrated.
4. Comprehensive Evaluation: Extensive testing across three deployment scenarios (Edge-Cloud, Agentic Memory, Mission-Critical Cyber-Physical) using five datasets and six VLMs (7B–235B parameters).

4. Results

The evaluation covers diverse datasets (VQA-v2, GQA, TextVQA, LoCoMo, FloodNet) and model scales (7B to 235B parameters).

• Cost-Accuracy Trade-off: VOILA consistently achieves 50–60% cost reductions while retaining 90–95% of full-resolution accuracy. The system lies on or near the Pareto frontier across all datasets and models, whereas fixed-fidelity baselines are consistently Pareto-dominated.
• Robustness to Scale: Gains persist across model sizes. Larger models improve absolute accuracy but do not compensate for insufficient input fidelity; VOILA remains effective regardless of model capacity.
• Out-of-Distribution Generalization: When trained on one dataset (e.g., VQA-v2) and tested on another (e.g., GQA), VOILA maintains stable accuracy-cost behavior, indicating that the routing policy generalizes based on query characteristics rather than dataset-specific statistics.
• Agentic Memory & Edge Systems: In the LoCoMo benchmark (long-horizon memory) and FloodNet (bandwidth-constrained disaster response), VOILA significantly outperforms fixed-fidelity baselines, recovering most of the full-resolution accuracy at a fraction of the retrieval cost.
• Ablation Studies:
  • Calibration: Isotonic calibration is the central enabler, reducing acquisition costs by 15–30% compared to uncalibrated routing while maintaining accuracy.
  • Predictor Choice: Once calibrated, the specific regressor architecture (GBR, Logistic Regression, Ridge, MLP) has minimal impact on performance, confirming that routing quality is driven by calibration rather than the predictor class.
  • Overhead: The fidelity selection process adds negligible overhead (~0.45ms per query) compared to VLM inference (2.4–62.8 seconds).

5. Significance and Claims

The paper claims that pre-retrieval fidelity selection is a distinct and fundamental learning problem, orthogonal to model routing. By optimizing what information to acquire rather than how to compute, VOILA addresses the bottleneck of retrieval costs in multimodal systems where inputs are distributed across heterogeneous storage tiers.

The authors argue that adaptive inference strategies that rely on post-hoc confidence scores are insufficient because model confidence often remains high even when the retrieved context lacks critical visual information. VOILA demonstrates that query-adaptive fidelity selection is vital for optimizing multimodal inference under resource constraints, offering a principled, cost-aware alternative to indiscriminate high-fidelity retrieval. The results establish that pre-retrieval selection is a robust learning primitive that complements existing model routing techniques.