QoS-aware autoscaling for AI inference becomes more complex when the load increases faster than the central infrastructure. This paper demonstrates how to distribute inference between the server and user resources without losing control over QoS.
The issue here is not with the inference itself, but with how it scales. A centralized service model becomes more expensive as demand grows, and in AI inference, operational costs quickly begin to dominate over the one-time cost of training. The authors also emphasize another constraint: cloud infrastructure scales reliably, but the cost of such scaling increases almost linearly with the load. Therefore, the focus is not on simply increasing server capacity, but on an architecture that can utilize additional resources where they already exist, namely on user devices.
The proposal revolves around collaborative distributed inference. The core idea is pragmatic: a dedicated server maintains a minimally necessary reliability and QoS bandwidth, while volunteered resources from users take on part of the computations as the online pool grows. This is a compromise between two extremes. A purely centralized scheme offers predictability but poorly handles increased demand. Pure volunteer computing reduces costs but suffers from heterogeneity and variable availability. The authors choose an intermediate model: the server remains the anchor of the system, while user resources operate as an elastic external buffer.
To analyze this buffer and avoid turning the architecture into a set of heuristics, the system is described through a high-dimensional generative Markov model. This is an important engineering move. The model considers users, resources, subtasks, state durations, and scheduling policies as interrelated but sparse variables. This approach allows not only simulating system behavior but also dissecting where exactly degradation occurs: at the download, execution, upload stages, or due to memory, bandwidth, and CPU shortages. For autoscaling, this is more useful than a static capacity assessment because QoS depends not on a single metric but on a chain of states.
The implementation divides the system into several state levels. There is user availability, their inference requests, allocated resources, readiness of subtasks, execution states, and total resource consumption. Each class of state uses its own probabilistic models. Availability and request timing are described through survival distributions. Resource trajectories are modeled using ARMA models. Generative models based on measurements are also used for completion phases and resource consumption. This is not just an academic detail. It is necessary for the scheduler to see not an abstract “free resource,” but a dynamic system with inertia, lags, and memory constraints.
The executor policy is noteworthy. It does not schedule work but checks local feasibility and strictly halts overconsumption. If a non-storage overflow occurs, active downloads on the node are paused. If storage is insufficient, downloads with the largest footprint are aborted. If resources are still inadequate, the system begins to sequentially offload from execution stages. This is a strict but understandable trade-off: it is better to stop part of the work than to allow uncontrolled degradation of the entire node. This separation into scheduler and executor brings the model closer to real-world operation, where policy for placement and policy for enforcement address different tasks.
On top of this, the authors compare three strategies: centralized, uniform, and affinity-based scheduling. The centralized policy sends everything to the server. The uniform policy distributes subtasks evenly among suitable online nodes. The affinity-based policy attempts to increase batching, that is, co-locating identical subtasks for denser execution. Here, the classic trade-off between throughput and tail latency is already evident. Larger batches can enhance efficiency, but they also tie several requests into one failure loop. If a node fails or the executor aborts a batch, the cost of error increases.
The results demonstrate exactly this. As the number of users grows, distributed scheduling becomes increasingly advantageous. The uniform policy most often provides the best balance between completed and canceled requests in experiments, especially with large user populations. It also ensures lower P99 latency under high loads. The affinity-based policy sometimes benefits from batching but more often loses in request completion and tail latency because large batches hold resources longer and suffer more from aborts. The centralized policy remains competitive at small scales but hits saturation in CPU and memory. The authors note that a range of about 30×–50× server capacity is already sufficient to complement volunteered resources well, and further scaling of the center yields diminishing returns.
The main conclusion is practical. This scheme does not eliminate dedicated infrastructure but reduces its necessary volume. The system remains QoS-aware because baseline capacity prevents service failure, while user devices take on the increased load. At the same time, the authors honestly acknowledge limitations: users are considered homogeneous, fairness is not modeled, and production-level security and privacy are left for future work. But as an architectural analysis, this is a strong piece of work. It shows that autoscaling for AI inference can be built not only around a center but also around a dynamic pool of external resources if the system has a model capable of withstanding such heterogeneity.
Information source
arXiv is the largest open preprint repository (since 1991, under the auspices of Cornell), where researchers quickly post working versions of papers; the materials are publicly accessible but do not undergo full peer review, so results should be considered preliminary and, where possible, checked against updated versions or peer‑reviewed journals. arxiv.org