The central collector on Alloy is no longer a sidecar, but a critical layer of infrastructure. If it receives metrics, logs, and traces for the entire platform, an error in sizing or monitoring can quickly turn into a systemic issue.
When Alloy operates as a standalone sidecar, the task is simple. When it becomes a centralized gateway and accepts the full telemetry stream of the enterprise platform, everything changes: volumes increase, the cost of errors rises, and degradation begins to affect all teams simultaneously. The initial case involved tens of millions of active series, terabytes of logs per day, and tens of thousands of trace spans per second. For such a scheme, it is no longer sufficient to “just launch and observe.” Capacity planning, honest load testing, and a separate monitoring path that does not depend on the collector itself are required.
The problem here is architectural. A central gateway is more convenient than per-team sidecar deployments because it centralizes data reception from application teams via OTLP, Prometheus Remote Write, and Loki write protocols. Alloy buffers, batches, processes, and sends data onward to Grafana Cloud. This simplifies operations but creates a bottleneck: if the gateway starts to struggle, everyone can see it. Therefore, the approach must be as rigorous as for any production service.
In the described implementation, the team first established the expected traffic profile. For metrics, around 17M active series were planned; for logs, 1 TB/day with a peak of 17.5 MB/s; for traces, about 1 TB/day with a peak of approximately 23 MB/s. Importantly, this was not only the current baseline but also headroom for several onboarding waves from new teams. This is a key engineering point: Kubernetes autoscaling cannot compensate for a sudden increase of this scale if the load arrives all at once and without a buffer.
Next came sizing. Grafana recommends relying on official documentation, and in this case, such a baseline was used as a starting point. The final resource budget amounted to approximately 195 GB memory and 28 CPU cores. Instead of several large pods, many smaller pods were chosen. This is a pragmatic compromise: it makes it easier to scale the fleet horizontally and respond more quickly to increased load. A memory request of 6 GiB was set for one pod, but without a CPU limit. This was done intentionally because CPU throttling in Kubernetes can introduce hidden latency in high-throughput workloads.
The architecture was then assembled in a production-like form. The central collector was placed behind an ingress controller in Kubernetes. All incoming streams—OTLP over HTTP/Protobuf, Prometheus Remote Write, and Loki HTTP push—terminate at the ingress and are distributed across the Alloy pod fleet. This reduces complexity on the client side and provides a single entry point. However, there is an important caveat: Alloy’s own monitoring was placed on a separate path. The Kubernetes Monitoring Helm chart scrapes the /metrics endpoint and sends data directly to Grafana Cloud, bypassing the central collector. This separation of streams is not a detail but a safeguard against blind spots. If the gateway itself degrades, its health metrics remain visible.
Before go-live, the team conducted load tests in a pre-production environment that mirrored the production configuration. Traffic was generated using k6 and xk6, and the tests were distributed across 10 Amazon EC2 m5.2xlarge instances. Access to the cluster went through the same ingress controller as in production. There were many tests, including loads at the expected production level and above it. This is important: testing only against the baseline does not show where real degradation begins and what the failure mode will be.
Monitoring during the tests is also worth noting. It followed an independent path through the Kubernetes Monitoring Helm chart. This allowed the team to see the state of the collector in real-time even when test traffic created significant pressure. For a gateway scenario, this is critical. If health metrics pass through the same system you are monitoring, at the moment of failure, you lose visibility precisely where it is needed most.
After launching in production, the scheme withstood the onboarding of new deployments and the growth of ingestion waves. The fleet scaled from a minimum of 30 pods, and the system at the time of writing maintained nearly 17M active series, 20-30 MB/s logs ingestion, and up to 150 MB/s traces ingestion. Metrics showed smooth operation of HPA: a baseline level, gradual growth, without sharp drops. There are no metrics indicating improvements in latency or throughput at the source, so it is correct to speak not of “acceleration” but of stable adaptation to increased load.
However, there were lessons that typically emerge only under stress. The first is WAL. Write-ahead log helps avoid data loss during a restart, but under sustained high throughput and backpressure from Grafana Cloud, it can grow faster than it can be cleared. In such a mode, it becomes a source of OOM kills. The second is GOMEMLIMIT. The Go runtime does not respect Kubernetes memory limits as many teams expect, so a soft ceiling around 80% of the memory limit is needed as an early signal for the garbage collector. If this limit is frequently triggered, CPU usage increases, which is beneficial: HPA based on CPU can scale out before memory becomes critical.
The outcome of this approach can be described without marketing. The central gateway on Alloy works if treated as a full-fledged production system. Planning should not be based on the current baseline but on the nearest growth. The monitoring path must be kept separate from the main data path. Testing should be conducted above the expected ceiling. And it should be understood that retry_on_failure, min-replica, and node capacity buffer are not embellishments of the configuration but mechanisms that separate managed degradation from data loss.