Client-side load balancing has become the key to stable latency under extreme fan-out. Let’s explore how Zalando moved routing inside the process and what compromises this brought.
The system faced challenges with predictable latency at high fan-out. A single batch request branched out to 100 parallel calls to product pods. Each call passed through Skipper — the edge load balancer of the cluster. As a result, the overall delay depended on the slowest of the 100 transitions. The team could not separate their own issues from latency spikes within Skipper, which they did not control at the specific request level. Under a load of about 1 million requests per second, this led to systemic opacity and increased tail latency.
The solution seemed counterintuitive: rather than strengthening the external balancer, they removed it from the critical path for internal fan-out. The team moved routing inside the process by implementing client-side load balancing. Skipper remained for edge traffic and simple requests. This was an important compromise: they did not replace the existing infrastructure but isolated the hot path with high call frequency. This approach reduces network hops and provides control over routing decisions, but shifts complexity into the application.
A key detail was the consistency of routing. They replicated Skipper’s algorithm exactly: xxHash64 and 100 virtual nodes on the endpoint. This allowed them to maintain identical hash rings. This is important because otherwise, a cache split would occur during migration. When adding or removing a node, only about 1/N of the keys are redistributed, limiting churn. The behavior was solidified with unit tests to eliminate discrepancies between implementations. This is an example of engineering priority: data stability is more important than speed of deployment.
The implementation required solving several non-trivial tasks. Polling, which overloaded the control plane, was replaced with a Kubernetes informer using a watch model. This reduced the load and sped up the response to changes. For rollout, they used gradual enabling through toggles from 1% to 100%. Special attention was paid to cache-aware scaling: new pods should not immediately receive the full stream. To achieve this, they added an N-ring fade-in with a curve of ^2.5 over 30 seconds. New instances warm up only relevant keys, which reduces latency spikes during scaling.
An attempt to optimize costs through availability-zone routing revealed a downside. The cache became fragmented, causing a spike in reads in DynamoDB. This is a classic trade-off between locality and cache consistency. Ultimately, they abandoned this optimization. Instead, they enhanced the system through jittered retry, FIFO shedding under overload, and more detailed observability. Improved logging allowed them to identify brief freezes of individual nodes, which are now handled automatically.
The results appear pragmatic. Latency has become more predictable. Infrastructure costs have decreased: the Skipper fleet shrank from over 50 pods to 8, and daily expenses dropped from $450 to $110. Observability improved: the team gained control over the decision-making point. While metrics on exact latency gains are not disclosed, indirect signs indicate a reduction in tail latency.
Importantly, the team clearly limits the applicability of this approach. Client-side load balancing is justified only under extreme loads and high fan-out. In most cases, mature solutions like Skipper or Envoy provide sufficient functionality without shifting complexity into application code. This is not a universal pattern but a targeted optimization for a specific load profile.
This approach has been discussed in the industry for a long time, but the case illustrates where the boundary of its feasibility lies. When the network and external balancer become part of the latency budget, moving logic inside the process may be justified. However, the cost is increased complexity, the need to maintain routing algorithms, and tighter coupling with infrastructure.