🤖 machine learning

Ask the Expert: Collaborative Inference for Vision Transformers with Near-Edge Accelerators

This paper proposes a collaborative inference framework that orchestrates a lightweight edge Vision Transformer with multiple near-edge expert models using a dynamic routing mechanism and progressive specialist training, achieving significant improvements in accuracy, latency, and energy efficiency compared to standalone edge or cloud execution.

Original authors: Hao Liu, Suhaib A. Fahmy

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Liu, Suhaib A. Fahmy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy, high-tech detective agency. Your goal is to solve thousands of visual mysteries (like identifying what's in a photo) as fast as possible, but you have a major problem: your main office is small and cramped (the Edge Device, like a Raspberry Pi), while the world's best experts work in a massive, expensive skyscraper far away (the Cloud).

Here is the dilemma:

The Small Office: It's fast and right there, but the detectives inside are junior agents. They can spot obvious things easily, but when they see something tricky, they get confused and make mistakes.
The Big Skyscraper: It has the world's greatest detectives who can solve anything, but calling them takes a long time (latency) and costs a fortune in energy.

The authors of this paper, Hao Liu and Suhaib Fahmy, came up with a brilliant middle-ground solution called "Ask the Expert." They built a system that acts like a smart manager, creating a partnership between the small office and a new, medium-sized "Near-Edge" branch office located just down the street.

Here is how their system works, broken down into simple steps:

1. The "Confidence Check" (The Junior Detective's First Look)

When a photo comes in, the junior detective in the small office takes a quick look.

If they are 100% sure: They solve the case immediately. No need to bother anyone else. This saves time and energy.
If they are unsure: Instead of guessing and getting it wrong, they pause. They don't just say, "I don't know." Instead, they say, "I'm not sure, but I think it's either a Cat, a Dog, or a Fox."

2. The "Top-K" Shortcut (The Magic List)

This is the paper's first big innovation. Usually, when a junior detective is unsure, they might send the whole messy photo to the big skyscraper. But that's slow.

Instead, this system uses the junior detective's "Top-K" list (their top 3 guesses).

The system looks at that short list (Cat, Dog, Fox).
It realizes, "Ah! The answer is definitely one of these three."
It then sends a tiny, specific request to the Near-Edge Branch Office (the medium-sized accelerator), saying: "Hey, we have a mystery. It's definitely a Cat, Dog, or Fox. Which one is it?"

The Analogy: Imagine you are at a party and you see someone who looks like a celebrity. You aren't sure which celebrity it is, but you know it's either Taylor Swift, Beyoncé, or Rihanna. Instead of calling the whole world to ask, you just text your friend who is a huge music fan and say, "Is it Taylor, Bey, or Rhi?" Your friend can answer instantly because they only have to choose between three options, not the whole world.

3. The "Specialist" Training (The Expert's Secret Weapon)

This is the second big innovation. Usually, if you have a team of experts, they are all "Generalists"—they know a little bit about everything. But the authors realized that if you only need to choose between a Cat, Dog, and Fox, you don't need a generalist; you need a Specialist.

They trained their Near-Edge experts specifically on these small groups of categories.

One expert is a master at distinguishing between animals.
Another is a master at distinguishing between vehicles.
Another is a master at distinguishing between food.

When the junior detective sends the "Cat, Dog, Fox" list, the system instantly routes the request to the Animal Specialist. Because this expert only focuses on animals, they are incredibly fast and accurate at solving that specific puzzle, much faster than a generalist who has to think about cars and food too.

4. The "Progressive" Training (Learning by Doing)

How do you teach these specialists to be so good? The authors used a clever training method called Progressive Specialist Training.

Phase 1: The experts start by learning everything (the whole world).
Phase 2: Slowly, they are forced to focus more and more on their specific group (e.g., just animals).
Result: They don't forget how to be smart, but they become super smart at their specific job. It's like a student who studies all subjects in school, but then spends their summer internship only studying biology, becoming a biology genius without forgetting math.

Why is this a game-changer?

The paper tested this on real hardware (like a Raspberry Pi and a powerful Nvidia Jetson) and found amazing results:

Speed: It was up to 45% faster than trying to run the complex model on the small device alone.
Energy: It used up to 46% less energy than sending everything to the cloud.
Accuracy: It was actually more accurate than just using a medium-sized model everywhere, because the "Junior Detective" filtered out the easy cases, letting the "Specialist" focus only on the hard ones.

The Bottom Line

This paper solves the "Goldilocks" problem of AI.

Too small: The device is too weak to solve hard problems.
Too big: The cloud is too slow and expensive to call for every problem.
Just right: A smart team where a fast, local worker handles the easy stuff and quickly passes the tricky stuff to a nearby, specialized expert who only looks at the specific options needed.

It's like having a local librarian who can quickly find the right section of the library for you, so you don't have to walk all the way to the main archive building for every single book you need.

1. Problem Statement

Deploying Vision Transformers (ViTs) on resource-constrained edge devices (e.g., Raspberry Pi) is hindered by their high computational complexity and latency. Conversely, offloading all inference to the cloud introduces significant round-trip time (RTT) latency and bandwidth costs. While "near-edge" accelerators (e.g., Nvidia Jetson Orin) offer a middle ground with better compute power, they are expensive to deploy individually on every device.

Existing collaborative inference paradigms face specific limitations:

Split Inference/Early Exits: These work well for CNNs but fail for ViTs because ViTs lack natural bottlenecks (due to constant columnar structure) and global self-attention creates complex dependencies that make compression difficult.
Mixture-of-Experts (MoE): Standard MoE models suffer from training instability (router collapse, expert underspecialization) due to coupled optimization. Furthermore, centralized MoE architectures are ill-suited for distributed edge-near-edge environments.
Binary Offloading: Simple "edge vs. cloud" decisions often ignore the potential of intermediate hardware or fail to utilize the specific strengths of lightweight models.

2. Methodology

The authors propose a Collaborative Inference Framework that orchestrates a lightweight "generalist" ViT on the edge device and multiple medium-sized "specialist" ViTs on a shared near-edge accelerator.

A. System Architecture

Edge Device: Runs a lightweight generalist ViT (e.g., DeiT-3H). It processes all inputs locally.
Near-Edge Accelerator: Hosts a library of specialist experts (e.g., DeiT-4H or DeiT-6H), each trained on specific subsets of the data classes.
Workflow:
1. The edge model performs inference.
2. If the prediction confidence (max softmax probability) exceeds a threshold $\tau$ , the result is accepted locally.
3. If confidence is low, the sample is offloaded to the near-edge accelerator.

B. Key Innovations

1. Top-k Routing Mechanism (Zero-Overhead Router)
Instead of training a separate router network (which adds overhead), the framework repurposes the edge model's Top-k predictions as the routing signal.

When a sample is uncertain, the system extracts the Top- $k$ class indices predicted by the edge model.
These indices are mapped to specific data partitions (subsets of classes).
The system selects the specific expert model trained on the union of these partitions to refine the prediction.
Benefit: This eliminates the need for a separate router model, ensuring zero additional inference overhead while leveraging the edge model's high recall in Top- $k$ predictions.

2. Progressive Specialist Training Strategy
To train the experts effectively, the authors introduce a curriculum learning approach that balances specialization (accuracy on a subset) with generalization (accuracy on the whole dataset).

Data Partitioning: The dataset is divided into $S$ non-overlapping subsets based on superclasses.
Dynamic Weighted Distillation: Experts are trained using a pre-trained large ViT as a teacher. The training loss is dynamically weighted:
- Early Epochs: The model focuses on the entire dataset to learn global features.
- Later Epochs: The weight shifts progressively toward the specific target subset ( $D_I$ ) to enforce specialization.
Formula: The weight $w_i^{(t)}$ for a sample increases linearly over epochs if the sample belongs to the expert's target domain, ensuring the expert becomes highly proficient on its specific classes without losing generalization capabilities.

3. Key Contributions

Novel Framework: A collaborative inference system decoupling the router and experts, utilizing edge-near-edge topology to balance latency and accuracy.
Zero-Overhead Routing: A mechanism using Top- $k$ predictions for dynamic expert selection, avoiding the cost of auxiliary router networks.
Progressive Training: A training strategy that explicitly enforces domain specialization while maintaining global accuracy, solving the trade-off often seen in static expert training.
Empirical Validation: Extensive experiments on a real-world testbed (RPi5/Orin Nano as edge, AGX Orin as near-edge) using the CIFAR-100 dataset.

4. Experimental Results

The framework was evaluated on CIFAR-100 using DeiT models (3H, 4H, 6H) on a Raspberry Pi 5 (Edge) and Nvidia Jetson Orin (Near-Edge).

Accuracy Improvements:
- The progressive training strategy improved expert specialization accuracy by 4.12% on target subsets compared to static experts.
- Overall system accuracy increased by 2.76% over static experts and up to 1.70% over "Generalist Co-inference" (using a large generalist model on the near-edge).
Latency Reduction:
- Compared to running the model entirely on the edge, the framework reduced latency by up to 45%.
- Compared to offloading everything to the near-edge, it reduced latency by 40–45% (by keeping easy samples local).
Energy Efficiency:
- Compared to pure near-edge offloading, the framework reduced energy consumption by up to 46% by minimizing data transmission and local computation for easy samples.
Trade-off Analysis: The method achieves a superior Pareto frontier, offering better accuracy for the same latency/energy cost compared to monolithic models or standard co-inference.

5. Significance and Impact

Scalability: This approach allows expensive near-edge accelerators to be shared among multiple edge devices, amortizing hardware costs while maintaining high performance.
ViT Optimization: It specifically addresses the architectural challenges of ViTs (lack of bottlenecks) where traditional split inference fails.
Sustainability: By reducing reliance on centralized cloud datacenters and optimizing energy usage on edge devices, it supports the development of energy-efficient, sustainable AI for IoT and cognitive cities.
Future Potential: The authors note that the "recall gap" (the difference between Top-1 and Top- $k$ accuracy) widens on larger datasets (e.g., ImageNet), suggesting this framework will become even more effective as model complexity and dataset sizes increase.

In summary, the paper presents a robust, efficient solution for deploying complex Vision Transformers in distributed edge environments by intelligently combining lightweight local processing with specialized remote refinement.