controller-runtime cache defines the behavior of Kubernetes controllers. Understanding this model directly impacts latency, memory, and read consistency.
The first failure in the mental model usually manifests under load. An engineer expects that r.Get() and r.List() go to the kube-apiserver and return the current state. In practice, the controller begins to make decisions based on stale data, memory consumption increases, and behavior becomes difficult to explain. Degradation intensifies as the number of reconcile cycles and objects grows. The root cause is singular: the controller-runtime cache is perceived as an optimization, although it is the foundation of the entire architecture.
The real model is contrary to expectations. The controller-runtime reads from a local in-memory cache, which is populated via list + watch. This reduces the load on the control plane and makes reads almost free in terms of latency. But the trade-off is evident: data may be stale, and the cache can consume gigabytes of memory. This is a pragmatic choice by Kubernetes: to reduce pressure on the API server and etcd at the cost of eventual consistency and increased memory footprint.
The architecture is built around a standard pipeline from client-go. The Reflector performs an initial list and then maintains a watch using resourceVersion. This ensures that events are not lost between the snapshot and the stream of changes. Next, DeltaFIFO buffers events and preserves their order for each object. Importantly, it does not collapse intermediate states, so the handler can receive a chain of changes. The Indexer stores objects in memory as a map with indices, which explains the microsecond latency for r.Get(). The SharedIndexInformer ties everything together and distributes events to subscribers, including controllers.
On top of this layer operates the workqueue. Here, deduplication occurs by the key namespace/name. Even if an object receives several updates in succession, there will be one item in the queue. This is critical for throughput: the controller is not overwhelmed by events, and reconciliation is performed on the already current state from the cache. This design separates responsibilities: the informer handles the stream of events, while the workqueue controls the frequency of processing.
Particular attention is given to consistency. When the controller reads an object, it receives a version with a specific resourceVersion. When writing via r.Update, the API server checks that the version matches the current one. If not, a 409 Conflict is returned. This is an optimistic concurrency control mechanism. It makes the system safe without locks but requires explicit conflict handling. Attempting to ignore this leads to unstable controller behavior.
Initialization is also important. The manager warms up the cache before starting reconciliation. This means that the first r.Get() already works with a fully populated snapshot. There is no state of an “empty cache.” If it is necessary to read directly from the API before startup, a separate APIReader is used. This is a rare but conscious bypass of the standard model.
The result of such a design is a predictable load on the API server and high throughput for controllers. But the price is the complexity of reasoning. It is essential to consider that:
- reads come from the cache, not from the API
- data may be stale
- events arrive as a stream, not as a final state
- reconciliation works with a deduplicated queue
Metrics in the original material are not provided, but the behavior of the system is well explained by the architecture. Keeping in mind the model “read from cache, write to API, sync via watch,” most unexpected effects become understandable.
This has long been an established approach in the industry. Kubernetes was originally built around watch, not polling. The controller-runtime merely packages this model into a convenient API. Therefore, the primary task of the engineer is not to memorize functions but to accept the system’s limitations and design controllers with consideration for eventual consistency and the characteristics of the cache.