I am working on setting up a home server but I want it to be reproducible if I need to make large changes, switch out hardware, or restore from a failure. What do you use to handle this?
Currently I’m migrating from compose.y(a)ml to terraform. I’m using proxmox -> 2x VM -> docker swarm. I will soon try to engineer a solution to quickly scale up and down any service I want using the same terraform codebase with rundeck. I have my configs as terraform templates and it gets deployed as a swarm config (or secret), then mapped to the container the same way.
How do you manage your home server configuration
Poorly, which is to say that I just let borgmatic back up all my compose files and hope for the best
Yep.
“I manage my server in yaml. Sometimes yml.”
Git controlled docker-compose files and backed up docker data volumes.pretty easy to go back to a point in time.
That’s actually a really good idea. From now on I will do the same. Thanks!
reproducible
You tried writing bash scripts that set things up for you, haven’t you? It’s NixOS for you.
NixOS
Out of curiosity: Are you running nix-ops with nix-secrets or how did you cover orchestration & credentials?
Not op, but using colmena for that
I use flakes and all hosts are configured from a single flake, where each host has its own configuration. I have some custom modules and even custom package in the same flake. I also use home manager. I have 4 hosts managed in total: home server, laptop, gaming PC, and a cloud server. All hosts were provisioned using nixos-anywhere + disko, except for the first one which was installed manually. For secrets I use sops-nix, encrypted secrets are stored in the same flake/repo.
With NixOS, you get a reproducible environment. When you need to change your hardware, you simply back up your data, write your NixOS configuration, and you can reproduce your previous environment.
I use it to manage all my services.
NixOS for configuration and restic for data
Incus and ansible
Proxmox on the metal, then every service as a docker container inside an LXC or VM. Proxmox does nice snapshots (to my NAS) making it a breeze to move them from machine to machine or blow away the Proxmox install and reimport them. All the docker compose files are in git, and the things I apply to every LXC/VM (my monitoring endpoint, apt cache setup etc) are all applied with ansible playbooks also in git. All the LXC’s are cloned from a golden image that has my keys, tailscale setup etc.
This is pretty much my setup as well. Proxmox on bare metal, then everything I do are in Ubuntu LXC containers, which have docker installed inside each of them running whatever docker stack.
I just installed Portainer and got the standalone agents installed on each LXC container, so it’s helped massively with managing each docker setup.
Of course you can do whatever base image you want for the LXC container, I just prefer Ubuntu for my homelab.
I do need to setup a golden image though to make stand-ups easier…one thing at a time though!
So you make in proxmox container (LXC) the docker container?
Yes, essentially I have:
Proxmox Baremetal ↪LXC1 ↪Docker Container1 ↪LXC2 ↪Docker Container2 ↪LXC3 ↪Docker Container 3Or using real services:
Proxmox Baremetal ↪Ubuntu LXC1 192.168.1.11 ↪Docker Stack ("Profana") ↪cadvisor grafana node_exporter prometheus ↪Ubuntu LXC2 192.168.1.12 ↪Docker Stack ("paperless-ngx") ↪paperless-ngx-webserver-1 apache/tika gotenberg postgresdb redis ↪Ubuntu LXC3 192.168.1.13 ↪Docker Stack ("teamspeak") ↪teamspeak mariadbI do have a AMP game server, which AMP is installed in the Ubuntu container directly, but AMP uses docker to create the game servers.
Doing it this way(individual Ubuntu containers with docker installed on each) allows me to stop and start individual services, take backups via proxmox, restore from backups, and also manage things a bit more directly with IP assignment.
I also have pfSense installed as a full VM on my Proxmox and pfSense handles all of my firewall rules and SSL cert management/renewals. So none of my ubuntu/docker containers need to configure SSL services, pfSense just does SSL offloading and injects my SSL certs as requests come in.
deleted by creator
systemd unit files, because its all podman containers.
Carefully
Terraform and ansible. Script service configuration and use source control. Containerize services where possible to make them system agnostic.
How do you decide what’s for Terraform and what’s for Ansible?
They’re good at different things.
Terraform is better at “here is a configuration file - make my infrastructure look like it” and Ansible is better at “do these things on these servers”.
In my case I use Terraform to create proxmox VMs and then Ansible provisions and configures software on those VMs.
I used to have a fille with every cli command and notes on how each thing was set up. When I had to reinstall it from scratch it took all day going through lots of manual steps and remembering how it should all go.
Recently I converted the whole thing to Ansible. Now I could rebuild my entire system on a brand new OS install with one command that completes in minutes. It’s all modular and I can add new services easily whether they are docker containers or scripts or whatever. If I ever break anything, it will reset everything to its intended state and leave it alone otherwise. And it’s free and pretty easy to learn and start using.
Plus I use git along with it for version control, so I can always revert to any previous configuration instantly.
I got a bunch of docker compose files and the envs documented so its easy to spin things up again or rollback changes. It works well enough if I’m good about keeping everything all up to date and not making changes without noting it down for myself later.
Snapshots largely, most everything is VMs and docker containers. I have one VM set aside for dev work to test configs before updating the prod boxes as well.
