DNS routing for gRPC became a bottleneck during extreme connection growth. A single policy choice in Route 53 determined whether the system could handle the load.
When 121 million mobile devices simultaneously open persistent gRPC connections, DNS behavior ceases to be “background.” In this architecture, CloudFront acted as the edge, with a pool of Network Load Balancers (NLB) behind it hosting EKS workloads. The problem manifested during a sudden spike: traffic surged to over 110K RPS per second, and the system began to degrade. Errors between CloudFront and the origin reached a critical level, resulting in mass HTTP 500 errors at the peak of 14 million connections.
The root of the problem was not in capacity or compute. Route 53 with a Weighted routing policy returned only one IP for the DNS query. At the scale of CloudFront, this meant that all edge nodes resolved the same origin for the duration of the TTL. This created a classic thundering herd: one NLB received nearly all the traffic while the others remained idle. Adding new NLBs did not help. The reason was simple: distribution occurred not at the connection level, but at the DNS response level, which remained singular.
The solution appeared minimalist but changed the system’s behavior. Instead of Weighted routing, Multi-Value Answer routing was chosen. This mode returns up to 8 IP addresses for a single DNS response and supports health checks at the record level. As a result, CloudFront edge nodes began distributing connections among multiple origins immediately. This eliminated the concentration point at the start of connections. The trade-off here is evident: Multi-Value requires working with IP addresses (Elastic IP) and does not support Alias records, thus complicating configuration. However, the gain in load distribution proved critical.
The implementation did not require changes to the application code or architecture. It was a pure infrastructure shift. Static IPs were assigned to each NLB, and then Multi-Value records with health checks were created. An important limitation: Route 53 does not allow mixing policies for a single record, so the old Weighted records had to be removed. After updating the DNS, CloudFront began receiving multiple IPs and distributing gRPC connections at the first resolution, rather than after the TTL expired.
A key nuance relates to the nature of gRPC. Unlike short HTTP requests, long-lived connections are used here. An error in distribution at the start does not “spread out” over time but accumulates. This amplifies even a slight asymmetry in DNS routing. What appeared to be an acceptable deviation for stateless traffic turned into an overload of a single node.
After switching to Multi-Value Answer routing, the system changed its behavior immediately. During the next peak, no connection errors were recorded on the origin side. The final test—121 million devices and stable 110K+ RPS—passed without server-side errors. Metrics for latency or load distribution are not disclosed, but the fact that errors were eliminated indicates the removal of the bottleneck at the DNS level.
This case clearly demonstrates that DNS routing is part of the data plane, not just the control plane. When working with CloudFront and persistent protocols like gRPC or WebSocket, the choice of Route 53 policy directly impacts load distribution. Weighted routing provides predictable distribution at the request level but is unsuitable for long-lived connections. Multi-Value Answer is a more pragmatic choice in such scenarios, as it distributes the load at the first resolution stage.
In the industry, such limitations are known but often only manifest under extreme load. Here, the architecture was initially correct: edge, balancing, scalable origins. However, DNS behavior became a hidden bottleneck. The fix did not require scaling—only a change in the decision-making point.