QuickGrasp: Responsive Video-Language Querying Service via Accelerated Tokenization and Edge-Augmented Inference

QuickGrasp is a responsive, QoS-aware system that bridges the accuracy-latency trade-off in video-language querying by employing a local-first architecture with on-demand edge augmentation, shared vision representations, and adaptive tokenization to match large model performance while significantly reducing response delays.

The Gist

Imagine you have a super-smart assistant who can watch any video and answer your questions about it. You ask, "What is the dog doing in the third minute?" or "Why did the character look sad?"

This assistant is powered by Video-Language Models (VLMs). They are incredibly smart, but they are also like giant, hungry elephants: they need massive amounts of computer power and memory to work.

The Problem: The "Too Slow" vs. "Too Dumb" Dilemma

Currently, you have two bad options when using these assistants:

1. The Cloud Option (The Long-Distance Call): You send the video and your question over the internet to a giant supercomputer in a data center.
   • Pros: The supercomputer is huge and very smart. It gets the answer right.
   • Cons: Sending a video file over the internet takes time. It's like mailing a heavy box across the ocean. By the time the answer comes back, you've forgotten what you asked, or the conversation feels awkward and broken.
2. The Local Option (The Pocket Calculator): You run a smaller version of the assistant directly on your phone or laptop.
   • Pros: It's instant! No waiting for the internet.
   • Cons: Your device isn't strong enough. The assistant is like a smart child who gets confused by complex stories. It often gives wrong answers or misses details.

The Goal: We want the instant speed of the local device with the super-smart brain of the cloud, without the long wait.

---

The Solution: QuickGrasp

The paper introduces QuickGrasp, a new system that acts like a smart traffic controller for your video questions. It uses a "Local-First" strategy with a clever trick called Edge-Augmentation.

Here is how it works, using a simple analogy:

1. The "Fast-Forward" Scanner (Accelerated Tokenization)

Before the assistant can even think, it has to "read" the video. Usually, this is like watching every single frame of a movie to find the important parts, which takes forever.

• The Old Way: Watching every second of a 1-hour movie to find a specific scene.
• QuickGrasp's Way: It looks at the video's "table of contents" (keyframes). It knows that big changes in a scene usually happen at specific points. It skips the boring, repetitive parts and only grabs the "highlight reels." It also processes the video in a continuous assembly line (pipelining) so it doesn't wait for one step to finish before starting the next.
• Result: It turns a 10-second video processing task into a 1-second task.

2. The "Confidence Check" (Query-Adaptive Routing)

When you ask a question, QuickGrasp first asks the small, local assistant to answer it. But instead of just taking the answer, it asks: "Are you sure?"

• The Analogy: Imagine a student taking a test.
  • If the student is 99% confident in their answer, the teacher (the system) says, "Great, submit it!" and you get an instant result.
  • If the student is unsure (low confidence), the system says, "Okay, don't guess. Let's call the expert."
• The Magic: The system doesn't just look at the question text; it looks at how the local model "feels" about the answer. This prevents sending easy questions to the cloud, saving time.

3. The "Shared Brain" (Shared Vision Representations)

This is the most clever part. Usually, if the local assistant fails, you have to send the entire raw video to the cloud expert. That's like mailing a whole movie reel.

• QuickGrasp's Trick: Because the local assistant and the cloud expert use the same "eyes" (vision encoder), the local assistant has already done the hard work of turning the video into "thoughts" (tokens).
• The Analogy: Instead of mailing the whole movie reel to the expert, the local assistant just writes a summary note of what it saw and sends that note. The expert reads the note and gives the final answer.
• Result: You send a tiny text message instead of a giant video file. The network delay is almost zero.

4. The "Goldilocks" Adjuster (Adaptive Token Density)

When the system does need to call the cloud expert, it has to decide how much detail to send.

• Too little detail: The expert can't see the clues and gets the answer wrong.
• Too much detail: The message is too big, and it takes too long to send.
• QuickGrasp's Solution: It uses a smart learning algorithm (like a gambler learning which slot machine pays out best) to figure out the perfect amount of detail for each specific question.
  • For a simple question ("Is the dog brown?"), it sends a low-detail summary.
  • For a complex question ("How many people are in the background?"), it sends a high-detail summary.

The Results: Why It Matters

The researchers built a prototype and tested it on thousands of videos. Here is what they found:

• Speed: QuickGrasp is up to 12.8 times faster than sending the whole video to the cloud.
• Accuracy: It is just as smart as the giant cloud models. It doesn't sacrifice intelligence for speed.
• User Experience: It makes talking to a video assistant feel like talking to a human friend—no awkward pauses, just instant, smart replies.

In a Nutshell

QuickGrasp is like having a smart intern (your phone) who handles 80% of the work instantly. If the intern gets stuck, they quickly write a short, smart memo (compressed data) to the boss (the cloud) for help. They don't mail the whole file; they just send the memo. The boss reads it, gives the final answer, and the intern passes it back to you. The whole process happens so fast you barely notice the boss was involved at all.

Technical Summary

1. Problem Statement

Video-Language Models (VLMs) are transforming video querying services by enabling interactive, human-like reasoning. However, deploying these models faces a critical trade-off between accuracy and response delay:

• Remote Cloud Deployment: Large VLMs (e.g., 7B+ parameters) offer high accuracy but suffer from unacceptable response delays due to network transmission of large video files and remote inference latency.
• Local Edge Deployment: Small VLMs (e.g., 2B parameters) run locally with low latency but lack the reasoning capabilities for complex queries, leading to significant accuracy drops.
• Existing Collaborative Approaches: Naive local-first strategies often fail because they either trigger edge offloading too frequently (negating latency benefits) or offload raw video data (incurring high transmission costs). Furthermore, the video tokenization stage (decoding and sampling) is often the dominant bottleneck in VLM inference, yet it is frequently overlooked in optimization research.

2. Methodology: QuickGrasp System Design

QuickGrasp is a local-first, QoS-aware system that bridges the gap between small local models and large edge models through three core technical innovations:

A. Accelerated Video Tokenization

To address the decoding bottleneck, QuickGrasp introduces a two-pronged approach:

1. Keyframe-Aligned Sampling: Instead of uniform fixed-rate sampling (which forces decoders to process many reference frames), QuickGrasp aligns sampling indices with video codec keyframes (I-frames). This leverages the video's compression structure to minimize I/O and decoding overhead, as keyframes are random-access points.
2. Pipelined Processing: The system overlaps three stages—decoding, preprocessing, and vision encoding—into a continuous pipeline. This hides the latency of individual stages, significantly reducing the total makespan, especially for long videos.

B. Query-Adaptive Collaborative Inference

QuickGrasp employs a local-first architecture where the small model (Local Node) handles the query first.

• Shared Vision Representations: To avoid redundant computation when offloading is necessary, the system shares the vision encoder pipeline. The edge model (Large VLM) reuses the raw vision tokens generated by the local node, eliminating the need to re-transmit raw video or re-tokenize.
• Confidence-Based Routing: Instead of static text-based routing, QuickGrasp uses a calibrated confidence score from the local VLM.
  • It applies Temperature Scaling to calibrate the local model's output probabilities, correcting overconfidence.
  • If the calibrated confidence exceeds a threshold, the local answer is accepted.
  • If confidence is low, the query is escalated to the edge.

C. Delay-Aware, Accuracy-Preserving Token Density Configuration

When offloading is triggered, the system must decide how many vision tokens to send to the edge. Sending too few hurts accuracy; sending too many increases network delay.

• Contextual Multi-Armed Bandit (CMAB): The system frames token density selection as a CMAB problem.
• Context Vector: The decision agent uses a context vector comprising:
  • Local model uncertainty (confidence, margin, entropy).
  • Semantic features (textual and visual embeddings compressed via PCA).
  • Cross-modal relevance scores and spectral complexity metrics.
• Neural Linear Bandit: A neural network extracts latent features from the context, while linear bandits (Bayesian linear regressors) on top of these features perform Thompson sampling to select the optimal token density. This allows the system to learn online which density yields the best accuracy-delay trade-off for specific query types.

3. Key Contributions

1. Accelerated Tokenization: A novel decoding-efficient design combining keyframe-aligned sampling and pipelined processing, reducing preprocessing delays by up to 67%.
2. Shared Architecture: A collaborative framework that shares vision representations between local and edge models, allowing the edge to process compact tokens rather than raw video, drastically reducing offloading bandwidth.
3. Adaptive Density Control: A QoS-aware mechanism using Contextual Multi-Armed Bandits to dynamically adjust vision token density, balancing network cost against reasoning quality.
4. Robust Calibration: Integration of temperature scaling to ensure reliable confidence-based routing, preventing the local model from prematurely accepting incorrect answers.

4. Experimental Results

The authors implemented a prototype using VideoChat-Flash (2B local, 7B edge) and evaluated it on three benchmarks: MVBench (short clips), Video-MME (medium/long), and MLVU-Test (very long videos).

• Response Delay: QuickGrasp achieved up to a 12.8x reduction in response delay compared to remote-hosted large VLMs (EdgeHosted).
  • On long videos (MLVU-Test), it was 4.4x faster than a standard collaborative baseline and 12.8x faster than full cloud offloading.
  • It also outperformed local-only execution (DeviceNative) on longer videos due to the accelerated tokenization.
• Accuracy: QuickGrasp matched the accuracy of the large remote VLM (7B) while significantly outperforming the local-only small VLM (2B).
  • On MVBench, it achieved accuracy comparable to the 7B model with a mean delay of ~819ms (vs. ~900ms for local-only).
  • The system successfully identified that ~60-76% of queries could be handled locally without accuracy loss.
• Ablation Studies:
  • Removing vision sharing (NoSharing) increased delay significantly due to redundant tokenization and raw video transmission.
  • Removing confidence calibration (Uncalibrated) led to accuracy drops because the local model accepted incorrect answers too often.
  • Adaptive Token Density outperformed fixed-density strategies, proving the necessity of dynamic configuration.

5. Significance

QuickGrasp represents a significant advancement in edge-cloud collaborative AI. It demonstrates that high-quality, open-world video understanding is feasible on resource-constrained devices without sacrificing responsiveness. By treating the edge server as an "accuracy enhancement facility" rather than a primary compute node, and by optimizing the often-neglected video tokenization stage, QuickGrasp provides a scalable, practical architecture for next-generation interactive video services (e.g., smart assistants, robotics, and forensic retrieval). It effectively solves the "accuracy-latency" dilemma inherent in deploying large multimodal models.