The everything is a object in Kubernetes is a very nice abstraction. The ability to influence it with just kubectl makes for only one control plane (to learn).

The networking, ingress, discovery, installation, etc. are all not that great. So I’m pondering marrying the things I love: have a plain Debian system, “normal” networking (no overlay/underlay/whatever), Debian packages, etc. And controlling it all via kubectl.

This should include a sane installation and allow for rollbacks. Metrics and observability should be supported, but will probably be more of the same (install the binaries and push configuration = k8s objects).

Technically this will boil down to:

  1. using K3S for a kubernetes control plane (single Go binary!)
  2. figuring out how to marry Debian to the kubernetes control plane (virtual kubelet?)
  3. connecting multiple machines to the “cluster” (wireguard/tailscale?)

As a proof a concept I will try to make virtual kubelet call out to systemd and APT to install and start a “container”. Note in this context containers don’t exist you just run processes. This will also get you over the hilarious limit of 110 pods per machine that k8s has.

A side effect is that it is easier for everything to be kept in a Git repo, yes, I’m using etckeeper now, but that’s not the same. (i.e. jumping on the GitOps bandwagon)

The ultimate question will be if the overhead will be worth it?

Update (24 Nov 2020)⌗

This is happening see https://github.com/miekg/vks.