K3s on-prem Kubernetes becomes manageable through the declarative approach of k0rdent and BYOT. We will explore how to build a reproducible cluster without scripts and manual assembly.
The problem does not manifest immediately — until the moment when on-prem Kubernetes begins to scale. Manually assembled clusters lose reproducibility. Any change requires scripts or manual configuration. Under such conditions, the cost of maintenance increases and the predictability of the system decreases. This is especially noticeable in environments where there are no ready-made managed solutions and one has to rely on Proxmox or similar infrastructure.
In this case, we chose the combination of K3s on-prem Kubernetes + k0rdent + Proxmox. The main idea is to transition from imperative management to declarative management. Instead of describing steps, the system receives a description of the desired state. k0rdent takes on the reconciliation loop and monitors compliance. This is a compromise: more initial configuration, but significantly less manual work in the long term. K3s here acts as a lightweight Kubernetes distribution optimized for on-prem and edge.
The architecture is built layer by layer. The flow looks like this: the user specifies the state through k0rdent, then virtual machines are created in Proxmox via BYOT (Bring Your Own Template), and then the Control Plane Provider and Bootstrap Provider for K3s are connected. Each layer addresses a single task. This reduces coupling and simplifies debugging.
A key point is the absence of a native Proxmox provider in k0rdent. Instead, a custom Infrastructure Provider was implemented via a Helm chart. It is responsible only for creating VMs. Kubernetes logic is not added to it. This separation of responsibilities makes the system transparent: infrastructure and orchestration do not mix.
Pre-prepared virtual machine templates in Proxmox were used in the implementation. They already include cloud-init, SSH access, and basic packages. This eliminates the need to build images during each provisioning. As a result, startup is accelerated and the number of failure points is reduced. However, there is a trade-off: templates need to be manually kept up to date.
After the VM creation, management transitions to the Control Plane Provider. It assigns node roles and forms the control plane. Here it is important that roles are assigned declaratively. The cluster becomes predictable: each node has a clearly defined function. This reduces the risk of configuration errors that often occur during manual assembly.
The Bootstrap Provider is responsible for installing K3s. It manages the lifecycle of the cluster: installation, updates, configuration. K3s was chosen due to its minimal dependencies and fast installation. This is especially important for on-prem, where resources and network may be limited.
After all stages are completed, the system enters a state of continuous reconciliation. If something deviates from the declaration, k0rdent automatically corrects it. This is a key difference from the traditional approach, where drift goes unnoticed until an incident occurs.
The result is a fully declarative K3s on-prem Kubernetes cluster. Scaling no longer requires manual preparation. Recreating the cluster becomes a repeatable operation. However, the initial data does not include quantitative metrics on deployment time or error reduction, so the assessment remains qualitative.
It is important to note that the BYOT approach makes the system flexible. If the infrastructure is not supported out of the box, it can be integrated through templates and Helm. This expands applicability but requires engineering discipline: errors in templates directly affect the entire cluster.
In the industry, such a transition to declarative management has long been discussed. However, in on-prem environments, it is more challenging due to the lack of standard providers. In this sense, the combination of k0rdent, Proxmox, and K3s appears to be a pragmatic path to unifying infrastructure management.