Date: 2023-12-12
The Intro
I've moved over most of my lab to Kubernetes. It's been a lot of fun, and
I can see why k8s is loved by so many. I understand the value and simplicity
that containers bring, and having something assist with the orchestration is
pretty useful. I am by *no* means a k8s expert, but I've got all of my
back-end web stuff running on it, as well as some other independent services.
I'm looking to move my AI servers onto it next.
The Review
It's not all roses and candy-bars though, k8s has two major issues that I see:
One is that I've yet to find a good way to dump *everything* into a
config-file. I'd love to be able to run "kubectl export > config.json" and
have it output all of my configs. Pods, Services, Roles, Bindings, etc. It's
sometimes easier to make a few changes using kubectl, but for restoration
purposes, I'd like to have my config readily available to apply again.
The second big issue I've found is the networking plugins. I had some problems
with Calico and Flannel, and I also had a few friends who had issues with them
as well. The TL;DR is that they don't work exactly as documented, and need some
adjustments from the defaults. I was able to get flannel working, but their
bugginess let me a little irritated.
The Stack
I wrote a status script to help me get a quick
overview of how things are going when I hop on:
NAME READY STATUS NODE POD-IP SOURCE
apache-5b9bb5877-99bp6 1/1 Running kube3 10.11.2.123 ReplicaSet
baikal-6bf64f46f7-28drc 1/1 Running kube6 10.11.1.63 ReplicaSet
grafana-745b846855-d4phj 1/1 Running kube3 10.11.2.123 ReplicaSet
irc-7b959bd69-gtrw9 1/1 Running kube6 10.11.1.54 ReplicaSet
motion-0 1/1 Running kube3 10.11.2.121 StatefulSet
motion-1 1/1 Running kube6 10.11.1.140 StatefulSet
motion-2 1/1 Running kube3 10.11.2.122 StatefulSet
motion-3 1/1 Running kube6 10.11.1.131 StatefulSet
mpd 1/1 Running kube4 10.11.3.35 <none>
ra 1/1 Running kube4 10.11.3.91 <none>
repo-7d5594dc6d-pvjv5 1/1 Running kube6 10.11.1.85 ReplicaSet
syslog-6b858b7fd5-bblcq 1/1 Running kube6 10.11.1.139 ReplicaSet
victoriametrics-f55c4644-qg4rr 1/1 Running kube3 10.11.2.75 ReplicaSet
NAME TAINTS STATUS IP ARCH
kube-bullseye NoSchedule Ready 192.168.2.130 amd64
kube3 <none> Ready 192.168.2.133 amd64
kube4 <none> Ready 192.168.2.134 arm64
kube5 <none> Ready 192.168.2.135 arm64
kube6 <none> Ready 192.168.2.136 amd64
problematic non-default pods (none if output is empty):
NAMESPACE NAME READY STATUS RESTARTS AGE
I need to expand it a bit still. I'd like to add crons, volumes, leases,
events, etc. It'd be nice to have a "dashboard" of sorts. I know there are
pre-made tools for that, but I'd like to keep it cli-based (most of the
existing ones are web-based), and it's fun and helpful to build this out myself.
The Configs
I've built everything out by writing the configs first, and then applying
them. I've got them stored in a git repo, and I built a publishing script. I
haven't made the script public yet, but to summarize the two parts:
It checks to see if I need to rebuild any containers, and builds them with Docker.
Then it goes to my kubernetes master, and either runs a "kubectl rollout
restart", or a "kubectly apply -f /mnt/kube/$filethatneedsapplying.yml".
This is nice as I've had to rebuild the cluster from scratch a couple of times,
and to do that, I can basically run: kubectl apply -f /mnt/kube/*/*
This will apply all my configs, and kubernetes gets started on building it out.
The Future
I've got some more services to push into to kubernetes: I want to move my SQL
instances, and my WoW server into it. I've got to build out some sql seeding
infrastructure first, but I think it'll be nice to be able to spin up an
auto-slaving "sql-dump" pod, and have it save my data every hour or so.
The Leftovers
There are some things that I'd rather not move into kubernetes, namely my
apache front-end, and my email. I'd like those to be independent from the
cluster, so I can more freely mess around with rebuilds and whatnot.
The Conclusion
It's been nice playing with k8s. It takes some infrastructure building, but
once there, I can really move around freely. Failing over is easy, and it's
really nice to watch pods rebuild automagically if a node goes down. I wish
I could get my Virtual Machine infrastructure that smooth, but that's another
battle for another time |:P