LLM serving workload turns out to be significantly more complex than is typically modeled. FineServe demonstrates how real workloads break simplified assumptions and impact architecture.
Modern LLM platforms operate as always-on services with strict requirements for latency and throughput. The main issue is the unstable and heterogeneous LLM serving workload. Most studies have relied on synthetic or proxy traces where the workload is modeled as a stationary process. FineServe shows that this assumption is systematically incorrect. In real multi-model systems, the behavior of requests depends on the model architecture, its size, and the user scenario. As a result, degradation begins not due to peaks per se, but due to the mismatch between the workload model and the actual dynamics: burstiness, non-stationarity, and token variability compound each other.
The solution proposed in FineServe is to shift to a fine-grained analysis of the workload. Instead of aggregated metrics, the system considers three axes: architecture (Dense vs MoE), scale tier, and task intent. This is a pragmatic choice: it does not artificially complicate the system but provides enough signals for routing, scheduling, and capacity planning. The key trade-off is the increased complexity of modeling and generating workloads. However, without this, it is impossible to adequately assess multi-model LLM serving. In particular, it turns out that small Dense models create the most unstable traffic, while large MoE models are more predictable but prone to sharp spikes.
At the implementation level, FineServe relies on a production dataset: 1.48B requests, dozens of models, global infrastructure. An important detail is that only metadata (timestamps, token counts, model id) is used, without user content. This allows for the analysis of arrival patterns and token geometry without violating privacy. A two-tiered approach is used for workload modeling:
- Gamma distribution for inter-arrival time (IAT)
- Negative Binomial for millisecond aggregation of requests
This closes an important gap: classical models account for average intensity but ignore burstiness over short intervals. As a result, the system underestimates queue buildup and tail latency. FineServe shows that for Dense models <10B, this is critical: they exhibit high MSSD (sharp fluctuations), even with a relatively smooth average flow. A separate layer is token geometry. Here, a less obvious problem emerges. Input tokens have heavy tails (long-context), while output tokens are more constrained but still significant. Moreover, the input-output dependency is not linear. Dense models exhibit an inverted-bowl dependency: output increases to a certain input threshold, then falls. This is related to the transition from generation to summarization.
MoE models present a different picture: growth with saturation. This difference directly affects KV-cache, batching, and memory pressure. Unified evaluation models here introduce distortions. Task intent adds another layer of heterogeneity.
For example:
- Technical tasks behave like inverted-bowl
- Conversational tasks are almost independent of input length
This means that even with identical arrival rates, the system may experience different loads during the decode and prefill stages. Ignoring this factor leads to incorrect buffer allocation and latency degradation. The outcome is not only a dataset but also a workload generator. The FineServe workload generator can:
– Replay real traces (trace replay)
– Generate synthetic but realistic workloads (parametric synthesis)
The key idea is to mix per-model streams considering their statistical properties. This is important for benchmarking: instead of “average” load, the system can be tested under conditions close to production. At the same time, the authors explicitly note that precise metrics for system improvement are not provided. The focus is on the correctness of modeling rather than optimizing a specific implementation. The main takeaway for architects: LLM serving workload cannot be viewed as a single process. It is fundamentally heterogeneous and depends on:
- Model type
- Its size
- User scenario
A practical consequence is the need for architecture-aware strategies:
- For Dense models — fast schedulers sensitive to jitter
- For MoE — resilience to large bursts and sound capacity planning
FineServe does not offer a universal solution but establishes a more accurate model of reality. This is an evolutionary improvement that makes benchmarking and system design less naive.
Information source
arXiv is the largest open preprint repository (since 1991, under the auspices of Cornell), where researchers quickly post working versions of papers; the materials are publicly accessible but do not undergo full peer review, so results should be considered preliminary and, where possible, checked against updated versions or peer‑reviewed journals. arxiv.org