🤖 machine learning

User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling

This paper proposes a user-assisted collaborative distributed inference system that combines dedicated infrastructure with volunteered user resources to enable QoS-aware autoscaling, demonstrating through a high-dimensional Markov model and simulations that this approach significantly improves latency and request completion while reducing dedicated resource consumption as the user population grows.

Original authors: Alfreds Lapkovskis, Ali Beikmohammadi, Sindri Magnússon, Praveen Kumar Donta

Published 2026-08-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alfreds Lapkovskis, Ali Beikmohammadi, Sindri Magnússon, Praveen Kumar Donta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city where everyone is constantly asking for help. Sometimes they need a quick answer, but often they are asking for something incredibly complex, like asking a super-intelligent robot to write a story, solve a math problem, or recognize a face in a photo. This is called "AI inference." Right now, almost all of this heavy lifting is done in massive, centralized data centers—think of them as huge, fortress-like factories where thousands of powerful computers work together. While these factories are amazing, they are getting incredibly expensive to build and run, and they use so much energy that they are leaving a giant carbon footprint on our planet. It's like trying to power a whole city's traffic lights with a single, roaring generator; it works, but it's loud, costly, and hard to keep up with as more people join the road.

To solve this, scientists have been looking at "edge computing," which is like moving some of the work closer to the people who need it, perhaps to local neighborhood hubs. But there's another, more radical idea: what if we could borrow a tiny bit of computing power from the very people asking for help? This is the concept of "volunteer computing," where your phone, laptop, or tablet lends its spare brainpower to the group when it's not busy. The big question researchers are asking is: Can we mix these two worlds? Can we keep a few powerful servers to handle the basics while letting millions of regular users chip in to handle the rest, creating a system that scales up automatically without needing to build more giant factories?

This is exactly what the researchers in this paper set out to explore. They propose a new way of running AI services called "User-Assisted Collaborative Distributed Inference." Instead of relying solely on expensive, centralized servers, their system splits the work. A central server acts as the "captain," handling a baseline amount of work to ensure things run smoothly, but it also breaks down big tasks into smaller "subtasks" and sends them out to the devices of the users who are currently online. If a user's phone is sitting idle, it can grab a subtask, crunch the numbers, and send the result back. The system is designed to be smart: if the crowd of users is small, the server does most of the work; if the crowd grows huge, the system automatically leans more heavily on the volunteered power from all those devices.

To figure out if this idea actually works without crashing the whole system, the authors didn't just build a real-world prototype and hope for the best. Instead, they built a highly detailed "digital twin" of the system—a complex mathematical model that simulates how users, devices, and tasks interact over time. They treated the system like a living organism that changes every second, with users logging in and out, devices having different speeds, and tasks taking different amounts of time. They used this model to run thousands of simulations, testing different scenarios: What happens if we have 100 users? What if we have 10,000? What if the server is small, and what if it's huge? They compared their new "collaborative" approach against the old "centralized only" way.

The results from these simulations were quite promising. The researchers found that as the number of users grew, the collaborative system became significantly better than the traditional one. When the user population was small, the centralized server handled things just fine. But as the crowd swelled to 10,000 users, the centralized system started to struggle, with many requests getting canceled or taking a very long time to finish. In contrast, the collaborative system thrived. By offloading work to the users' devices, it managed to complete more requests and kept the "tail latency" (the time it takes for the slowest requests to finish) much lower. Perhaps most importantly, the collaborative system needed far less dedicated server power to achieve these results. The simulations suggested that with a moderate amount of server capacity—roughly 30 to 50 times the base unit—the system could handle massive demand by leaning on the users, whereas the centralized system would have needed to keep scaling up its hardware indefinitely, which is expensive and inefficient.

However, the paper is careful to note that this is a simulation, not a finished product ready for your phone today. The researchers tested different "strategies" for how the server should hand out tasks. One strategy was to just hand tasks out randomly to any available user, while another tried to be smarter by grouping similar tasks together on the same device to save time. The "random" approach worked surprisingly well, but the "smart grouping" approach sometimes caused issues if a user's device suddenly went offline, interrupting a whole batch of work. The study concludes that while the idea is feasible and the math looks good, there is still work to be done to perfect the scheduling rules and ensure the system is robust enough for the real world. Ultimately, the paper suggests that borrowing a little power from the crowd could be the key to making AI services cheaper, greener, and able to handle the massive demand of the future without building endless new data centers.

Technical Summary: User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling

Problem Statement
The paper addresses the escalating operational costs and environmental burdens associated with centralized AI inference services. As demand for AI inference grows, the costs of centralized cloud infrastructure scale proportionally, leading to prohibitive expenses and significant energy consumption (e.g., CO2 emissions and water usage). While edge and fog computing offer partial solutions by distributing computation, they often rely on dedicated infrastructure that still faces economic and architectural barriers to widespread adoption. Conversely, volunteer computing leverages idle resources from user devices but typically lacks the stability and Quality of Service (QoS) guarantees required for consumer-oriented AI applications due to resource heterogeneity and intermittent availability. The core challenge is to design a system that effectively partitions inference between dedicated servers and volunteered user resources to achieve scalable, cost-efficient autoscaling while maintaining strict QoS constraints.

Methodology
The authors propose a collaborative distributed inference system where a central server coordinates inference tasks across a mix of dedicated infrastructure and volunteered user devices. The system operates in three phases: users request inference, the server schedules subtasks across available resources based on a policy, and the server aggregates results to return responses.

To analyze and optimize this dynamic system, the authors develop a high-dimensional generative Markov model with structured temporal factorization. This model captures the stochastic interactions among users, resources, tasks, and policies. Key components of the methodology include:

System State Representation: The state $s_t$ is defined as a tuple of high-dimensional variables including user availability ( $o_t$ ), inference requests ( $q_t$ ), allocated resources ( $a_t$ ), subtask readiness ( $x_t$ ), subtask execution states ( $y_t$ ), and resource consumption ( $c_t$ ).
Generative Modeling:
- Availability: Modeled using parametric survival distributions (Weibull for online, log-normal for offline) to capture state durations and transition probabilities.
- Requests: Modeled with Pareto survival functions to reflect heavy-tailed inter-request times and user patience.
- Resources and Consumption: Utilizes ARMA (AutoRegressive Moving Average) models fitted to empirical measurements of bandwidth, CPU, memory, and storage to simulate resource trajectories and consumption patterns.
- Execution Dynamics: Subtask execution phases (download, execution, upload) are modeled using discrete-time hazard models fitted to computer-vision subtask traces under varying batch sizes and bandwidths.
Decision-Making Policies: The system employs two policies: a scheduler ( $\pi$ $π$ ) on the server side and an executor ( $\varsigma$ $ς$ ) on the node side. The study evaluates three scheduling strategies:
1. Centralized: All tasks run on the dedicated server.
2. Uniform: Tasks are assigned randomly to online user nodes that have sufficient capacity.
3. Affinity-based: Tasks are assigned to nodes that already host active executions of the same subtask type to maximize batching efficiency.
Optimization Framework: The model defines two optimization problems: maximizing QoS (minimizing latency) for a fixed scheduler, and minimizing dedicated resource allocation while satisfying QoS constraints (e.g., P99 latency thresholds).

Key Contributions

System Architecture: A novel collaborative distributed inference system that partitions workloads between dedicated servers and volunteered user devices, enabling autoscaling without proportional growth in centralized infrastructure.
Generative Model: A tractable, high-dimensional Markov model with temporal factorization that captures sparse dependencies among system variables, enabling efficient simulation, inference, and policy learning.
Empirical Evaluation: A comprehensive simulation study comparing centralized and distributed scheduling policies across varying user populations, resource capacities, and server configurations.

Results
The system was evaluated via simulations running for approximately 28 hours (100,000 time steps) with user populations ranging from 100 to 10,000 and varying server capacities. Key findings include:

Scalability and Completion Rates: As the user population grows, distributed scheduling (Uniform and Affinity-based) increasingly outperforms centralized scheduling. Distributed policies maintain higher request completion rates and lower cancellation rates, particularly when user device capacity is doubled (2x).
Latency (QoS): Distributed policies achieve significantly better P99 latency (tail latency) than centralized policies at large user scales (e.g., 10,000 users). Centralized latency degrades as demand saturates server capacity, whereas distributed policies offload work effectively.
Resource Efficiency: Distributed scheduling substantially reduces dedicated resource consumption. The simulations indicate that a server capacity of approximately 30x to 50x the base profile is sufficient to complement volunteered resources and maintain QoS, whereas centralized approaches require much higher capacity to achieve similar performance.
Policy Comparison: The Uniform policy generally outperforms the Affinity-based policy in terms of the balance between completion rates and cancellation risks. While Affinity-based scheduling improves batching, it increases the risk of simultaneous subtask failures due to node departures or resource constraints.

Significance and Claims
The paper claims to demonstrate the feasibility of user-assisted collaborative inference as a viable strategy for infrastructure-efficient autoscaling. It establishes that combining dedicated infrastructure with volunteered resources can absorb increasing demand without the proportional growth in centralized costs and environmental impact associated with traditional cloud scaling.

The authors emphasize that their primary contribution is the definition of the system architecture and the development of a dynamic model to analyze it, rather than the deployment of a production-ready scheduler. They assert that the proposed model provides a necessary foundation for future work in optimizing scheduling algorithms and QoS-aware resource allocation. The study concludes that distributed scheduling offers a practical path to reducing dedicated resource consumption while improving request completion and tail latency, provided that the system is supported by an appropriate generative model for analysis and optimization.

Technical Summary: User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling

More like this