The Kairos upgrade pipeline demonstrated how to create a predictable control plane upgrade process without manual SSH and overnight shifts. The importance lies not in the fact of automation itself, but in how the system handles errors, quorum, and incorrect changes in CR.
In this scheme, the system encountered an old problem of Kubernetes maintenance. When the control plane is updated manually, the operator must keep in mind the sequence of actions, the state of etcd, and the risk of simultaneous node reboots. In the original scenario, this was a management cluster on K3s HA with three control plane nodes, where the root of the platform must be as resilient as possible. Against the backdrop of increasing CVE risks, the priority shifted from new functionality to a simple and reliable upgrade process for Kubernetes.
A pragmatic approach was chosen: immutable OS, GitOps, and a chain of CNCF tools around Kairos Hadron. Kairos does not patch the system in place. It writes a new image to an inactive A/B partition and boots into it after a reboot. This reduces the risk of a failed upgrade, and rollback is simply a matter of reverting to the old partition. The compromise here is clear: the model requires strict discipline around images, CR, and signatures, but it provides predictable behavior in the event of failures.
The architecture turned out to be narrow in roles, and this is its strength. Gitea stores manifests, policies, and upgrade specs. Renovate tracks new tags from quay.io/kairos/hadron and opens a PR where the image tag and metadata.name in the upgrade CR are changed. Kyverno filters out incorrect upgrade CRs at the admission stage if the image does not match the template. Cosign verifies the image signature through GitHub Actions OIDC identity, meaning that not only the tag is checked, but also the provenance of the artifact. ArgoCD then applies the merged PR as drift, and the kairos-operator executes the upgrade: cordon, pull image, write the new A/B slot, reboot, wait for rejoin, and move to the next node.
The main engineering detail turned out not to be in the components themselves, but in their linkage. Initially, the upgrade spec had concurrency: 0, which was a misinterpretation. Instead of “one node at a time,” the parameter meant simultaneous processing of all nodes. In the homelab test, this led to the reboot of three control plane nodes at once. The etcd quorum survived, but that was a matter of luck, not resilient design. After correcting it to concurrency: 1, the pipeline began to behave as a production-minded upgrade process should.
Next, another typical problem of GitOps automation emerged: the system may appear correct but lack a trigger for actual action. NodeOpUpgrade is a one-time custom resource. The kairos-operator marks it as completed and does not reprocess it. Therefore, simply changing spec.image on the existing CR does nothing. Renovate needed to change both metadata.name so that ArgoCD would delete the old CR and create a new one. It also turned out that a non-existent extractVersionTemplate was used in the custom regex manager. It did nothing. Renovate updated the image but left the name unchanged. The operator did not see the new CR, and thus did not initiate the upgrade.
The fix was targeted but important. Switching to currentValueTemplate allowed for the correct transformation of the version format from v0-3-0 to v0.3.0 for comparison, and then assembling the dash format back. This is a good example of why human review is not completely eliminated in such a scheme. The human no longer performs the upgrade manually. But they are still needed to notice where automation silently does not do what it seems at first glance.
The result has already been recorded. The upgrade to Hadron v0.4.0 took a total wall-clock time of 11 minutes. After the merge, no manual intervention was required. The etcd quorum was not violated, and there was no workload disruption at all. The gateway cluster on a separate single-node Kairos deployment with Netbird is now connected to the same ArgoCD GitOps loop. The next logical step has already been identified: a CI dry-run stage with kairos-agent upgrade –recovery before the merge, to catch errors before they reach the live node.