A long restart of a stateful service rarely appears to be a security configuration issue. However, this is how the safe default in Kubernetes turned into 30 minutes of downtime for each restart.
The problem manifested at scale. Atlantis, which manages Terraform through GitLab MR, operates as a singleton StatefulSet and stores state in a PersistentVolume. Any restart—for credential updates or onboarding—blocks all plan/apply operations. With around 100 restarts per month, this resulted in over 50 hours of downtime. The symptom looked strange: the pod was created quickly but hung before starting the init container. Kubernetes events provided no explanation. The degradation only became noticeable after the PV grew to millions of files and exhausted inodes.
Analysis showed that the bottleneck was not in the network or in pulling images. Kubelet mounted the PV quickly, but then a hidden operation occurred: a recursive change of permissions (chgrp -R) across the entire file tree. This is a consequence of the securityContext configuration with fsGroup. By default, Kubernetes (fsGroupChangePolicy: Always) guarantees access by changing the ownership of each file upon every mount. On large volumes, this turns into an O(n) operation based on the number of files. Without this setting, Atlantis would not be able to operate as non-root, so completely abandoning fsGroup is not an option—it’s a compromise between security and startup time.
The solution turned out to be precise. An alternative is available in Kubernetes—fsGroupChangePolicy: OnRootMismatch. In this mode, kubelet only checks the root of the volume and does not initiate a recursive chgrp if the permissions are already correct. Before the change, we verified that processes were not altering group ownership within the PV. This is important: with inconsistent permissions, runtime access errors can occur. After that, we added one line to the spec.securityContext.
The result was a reduction in restart time from about 30 minutes to around 30 seconds. CPU or I/O metrics did not change—the acceleration was solely related to the elimination of the unnecessary file operation. In total, this returned about 50 hours of engineering time per month.
This case illustrates a typical pattern: Kubernetes’ safe defaults work well at small volumes, but scaling makes their linear operations noticeable. If you have large PersistentVolumes and long restarts, it’s worth checking:
- whether fsGroup is used
- what the value of fsGroupChangePolicy is
- whether there are recursive operations during mount
If there is no data on filesystem behavior, switching to OnRootMismatch requires caution. However, with a controlled write model, this is a pragmatic improvement that removes the hidden O(n) cost from the critical startup path.