💻 computer science

SAFE-Cascade: Cost-Adaptive Vision-Language Routing for Chart Question Answering

SAFE-Cascade is a cost-adaptive, transparent system for chart question answering that uses a learned router to selectively escalate queries from a lightweight text-only model to a vision-language model only when necessary, thereby reducing computational costs while maintaining comparable accuracy.

Original authors: Ayush Dwivedi, Qixin Wang, Ashvi Soni, Ruoteng Wang, Han Li, Animesh Mahapatra, Neeraj Agrawal, Xintao Wu

Published 2026-06-19

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ayush Dwivedi, Qixin Wang, Ashvi Soni, Ruoteng Wang, Han Li, Animesh Mahapatra, Neeraj Agrawal, Xintao Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy customer service desk for a company that deals with thousands of charts, graphs, and financial reports every day. People walk up and ask questions like, "What was the sales number for March?" or "Which bar is the tallest?"

To answer these, you have two types of employees:

The "Text Reader" (Cheap & Fast): This employee is great at reading printed text. If the answer is written clearly on the chart, they can find it instantly. They cost very little to hire.
The "Visual Expert" (Expensive & Powerful): This employee can look at the whole picture, understand complex shapes, compare colors, and do deep math. They are incredibly smart but cost a fortune to hire and take longer to work.

The Problem:
In the past, companies would hire the Visual Expert for every single question, even the simple ones. It was like calling a brain surgeon to fix a paper cut. It worked, but it was a waste of money and time.

The Solution: SAFE-Cascade
The paper introduces SAFE-Cascade, a smart "traffic controller" for your customer service desk. Instead of blindly calling the expensive expert for everyone, SAFE-Cascade uses a three-step process:

The Quick Scan (OCR): First, the system uses a tool to read all the text on the chart and passes it to the Text Reader. The Text Reader tries to answer the question using only the words it found.
The Smart Gatekeeper (The Router): Here is the magic part. A small, trained AI (the "Gatekeeper") looks at the question and the Text Reader's answer. It asks: "Is this answer good enough, or do we really need the Visual Expert?"
- If the question is simple (e.g., "What is the title?"), the Gatekeeper says, "Stop here! The Text Reader got it."
- If the question is tricky (e.g., "Which trend is steeper?"), the Gatekeeper says, "Escalate! We need the Visual Expert."
The Final Answer: The system gives you the answer, but it also shows you the "receipt": what text it read, what the cheap answer was, why it decided to call the expert, and how much it cost.

Why is this cool?
The paper shows that by using this "Gatekeeper," the system can save about 27% of the expensive calls to the Visual Expert.

The Result: It answers questions just as accurately as if you had used the expensive expert for everything, but it saves you money (about 9% less cost) and time.
The "Knob": The system has a slider (a threshold). You can turn it to be more cautious (call the expert more often to be safe) or more frugal (try to save money and risk a few mistakes). This lets users see the trade-off between saving money and getting the perfect answer.

In a Nutshell:
SAFE-Cascade is like a smart filter that stops you from paying for a luxury car ride when a bicycle would get you there just as fast. It makes AI systems cheaper, faster, and more transparent by only using the "heavy lifting" tools when they are truly necessary.

Technical Summary: SAFE-Cascade

Problem Statement
Vision-language models (VLMs) have demonstrated strong capabilities in chart question answering (ChartQA), yet invoking a full VLM for every query is often computationally expensive and unnecessary. Many chart questions can be resolved using Optical Character Recognition (OCR) text extraction combined with lightweight language reasoning, without requiring deep visual grounding. The core problem addressed is how to dynamically route queries between a low-cost text-only pathway and a high-cost VLM pathway to optimize inference costs while maintaining answer accuracy.

Methodology
SAFE-Cascade is an interactive, cost-adaptive system designed to route chart QA queries based on the estimated necessity of visual inference. The system operates through a four-stage pipeline:

OCR Extraction: Upon receiving a chart image and a natural language question, the system first extracts text, layout snippets, and tokens using Azure Document Intelligence.
Text-Only Answering: A text-only Large Language Model (specifically gpt-5-mini) processes the question and the extracted OCR text to generate a provisional answer.
Learned Routing: A Random Forest classifier acts as a router. It analyzes inference-time features—including OCR length/density, question length, numeric/comparison flags, question-OCR overlap, text-answer length, and indicators of unknown or empty answers—to estimate an escalation probability ( $p_{esc}$ ).
Conditional Escalation: If $p_{esc}$ exceeds a configurable threshold ( $\theta$ ), the system escalates the query to a VLM (specifically gemini-2.5-flash-image) which processes the original image and question. Otherwise, the text-only answer is accepted as final.

The system is implemented as a modular service with a transparent Streamlit interface. It exposes the entire decision process to the user, displaying OCR evidence, the provisional text answer, the routing probability, the final decision, and estimated cost/latency.

Key Contributions
The paper outlines four primary contributions:

Cost-Adaptive Chart QA System: A novel architecture that selectively routes between OCR-plus-text reasoning and full VLM reasoning rather than defaulting to the latter.
Transparent Routing Interface: Unlike "black box" routing, SAFE-Cascade visualizes the evidence (OCR snippets), the provisional answer, the router's confidence, and the final outcome, allowing users to inspect why a visual pathway was or was not invoked.
Interactive Operating Frontier: The system includes a threshold slider that allows users to interactively explore the trade-off between accuracy and cost (VLM invocation rate) in real-time.
Held-Out Validation: The system is validated on a held-out split of the ChartQA benchmark, demonstrating that learned routing can match full-VLM performance while significantly reducing VLM calls.

Results
Evaluated on a held-out test split of 375 examples from the ChartQA-2500 dataset:

Accuracy: SAFE-Cascade (with threshold $\theta=0.3$ ) achieved 69.1% unified accuracy. This is statistically comparable to the full-VLM baseline, which achieved 67.7% accuracy (confidence intervals overlap).
Efficiency: The system reduced VLM invocation by 26.9% (from 100% to 73.1%) and reduced estimated inference costs by 9.3% compared to the full-VLM baseline.
Routing Performance: The router achieved an escalation F1 score of 84.2%, with 87.4% recall and 81.3% precision. The primary failure mode identified was "missed escalation" (5.9%), where the system accepted an incorrect text answer that the VLM would have corrected.
Threshold Sensitivity: A threshold sweep revealed that lowering the threshold increases VLM usage and preserves accuracy, while raising it reduces costs but significantly increases the rate of missed escalations (e.g., at $\theta=0.7$ , missed escalations rise to 27.2%).

Significance and Claims
The authors position SAFE-Cascade not merely as a performance improvement over existing models, but as a transparent, tunable framework for multimodal inference. The paper claims that the system successfully demonstrates how selective modality routing can make knowledge systems more cost-aware and interpretable.

The authors modestly frame their results: they do not claim a statistically significant improvement in accuracy over the full-VLM baseline. Instead, they interpret the system as matching full-VLM performance while achieving substantial cost reductions. The significance lies in the ability to expose the "cost–accuracy frontier" to users, allowing practitioners to adjust the system's behavior based on specific budget or latency constraints. The work highlights that the primary gain comes from learning when the text pathway is insufficient, rather than simply adding OCR or using heuristic rules.

Limitations
The paper acknowledges several limitations: validation is currently restricted to the ChartQA dataset; latency values are estimates rather than measured production timings; the router is sequential (post-text) rather than speculative; and cost estimates are based on configured pricing rather than actual billing. Future work is suggested for cross-domain validation (e.g., DocVQA) and exploring pre-routing or parallel execution strategies.

Technical Summary: SAFE-Cascade

More like this