A place to share, discuss, discover, assist with, gain assistance for, and critique self-hosted alternatives to our favorite web apps, web services, and online tools.
Selfhosted kubernetes cluster
I was thinking about migrating my services to self management k8s cluster, mostly for education purposes. I have some low traffic services running on one dedicated server on Scaleway now - there's some nextcloud, portainer, hastebin (sort of pastebin) mysql database, personal wiki, couple of half-dead blogs, stuff like that. Nothing resources heavy, nothing that needs tight SLA. I plan to add a little of my own software anytime soon, but I don't expect it to be very needy as well.
I'm planning on going with 3 nodes with min. 2 cores and 4 GB of RAM, and an additional service for storage. Does anyone here had experience with self hostes kubernetes? Do you have any advices? Do you think nodes like these are enough or I should upgrade (or downgrade)? Can you reccomend some providers that could offer this setup at low cost?
I used k3s and use the same node as master and runner. I started off with 3 nodes like you are planning, but soon discovered they for home use it just adds overhead in keeping up to date and backing up, and I had to find some way of doing persistent storage without standing up yet another VM as a NFS host, so downsized to a single node.
Check out kube-hetzner. Great project for self hosting with lots of options. Even supports Hetzners new Ampere servers for lower cost.
I deploy three management nodes and 3 agent nodes with longhorn storage.
kube-hetzner
If your host OS is Ubuntu can use snap and microk8s it’s what I use
sudo snap install microk8s —classic
There’s lots of different ways to get your cluster up look at the different ones available and make your decision on that
Kind Rancher K3s Microk8s
Are the ones that come to mind atm
Try Ubuntu & k0s with k0sctl.
I just run a single master node with a couple of slave nodes. K3S makes it easy to add new nodes to a cluster. Make sure you have a good data backup strategy though. If you're hosting databases, or other internal application state, and a drive dies, you are going to wish you had backups. For databases, just take a periodic dump of the DB and store it somewhere separate (ie. minio on a different node). For other applications, you'll want to take periodic snapshots of the entire filesystem, where the app stores its data.
Did this over the last month.
First I created a 3 node proxmox cluster from my existing homelab, and created 6 vms (2 per node) + 1 for a control plane. I used Debian for the OS and installing kubernetes, docker were fine aside from having to build the docker-cri, if you've never done it before.
k8s itself is fine, however if you plan to run things like minio with multiple disks, it may be easier to do it on bare metal.
Im self hosting k8s at home with a single custom build i7 8 core and 32 GB of ram. I just got tired of running after docker compose files everywhere and wanted it all under one roof. Plus work was getting heavy into it for our product so it seemed like a good idea. I'm really happy with my choice, and I haven't really stressed it much at all. Currently running MySQL, postgresql, rook-ceph, and they are probably the heaviest hitters right now. I also built it by hand using just kubeadm (well actually wrote/stole customized a bunch of ansible to manage a bare meta single node kubeadm cluster).
Forgot to say, Debian 12 is the host os, and I do have one or two kvm virts as well.
I use nixos for my cluster. If you aren't familiar with nixos I wouldn't recommend hosting k8s as the way to learn it. If you are it's an amazing way to run k3s.
Outside of that I would focus on whatever you can do to get the simplest k8s deployment (k3s et al) and then start focusing your learning on managing the configuration of kubernetes not the deployment.
If you want a simple storage solution I would make one node an nfs device and go from there.
If you want a magic (but complex) storage solution I would go with ceph. It's bulletproof to everything but you.
Cloud providers for k8s are going to be expensive because there's a lot of overhead with running k8s from a hosters perspective.
Those machines look fine to get started, so I wouldn't upgrade until you start running into problems, but even then I would focus on one big machine first instead of many little ones. Having many little machines gives you redundancy, and when you are just starting out that doesn't matter.
Thank you! :) I've heard about NixOS, but that's it - never used it and not even sure what the features are - will read about it for sure. So, if I understand you correctly - your advice is to stay on my dedicated server, setup k3s on NixOS, be prepared for some difficulties along the way and learn from there.
Just to clarify - I'm not looking for managed kubernetes from a cloud provider, sorry if I didn't make that clear. I'm looking for VMs or dedicated servers (just discovered they can be cheaper than VMs these days!) to set up and manage clusters myself. I probably wouldn't do much learning on the managed. At the moment I have a server with 16GB RAM and 8 cores - 1TB HDD might be the biggest performance bottleneck here, but I hope it will be sufficient for single node cluster.