Sergei Poljanski
71047d179d
Some checks failed
nixos / asxpio-services (push) Failing after 54s
Renovate now runs from its own repository for every repo, with the shared rules in its preset. Extend that preset here and keep only what is specific to this repo (the Ansible var manager, the Paper JRE, the runner and nix notes); drop the workflow that ran it for iac alone. Assisted-by: Claude:opus-5-5 |
||
|---|---|---|
| .. | ||
| hosts | ||
| migration | ||
| modules | ||
| secrets | ||
| .gitignore | ||
| .sops.yaml | ||
| flake.lock | ||
| flake.nix | ||
| README.md | ||
NixOS
The whole system of a server, as one flake. asxpio-services runs
NixOS; asxpio-ops is still Debian and managed by ../ansible.
flake.nix inputs (nixpkgs 26.05, disko, sops-nix), hosts
modules/base.nix users, SSH, fail2ban, journal cap
modules/hetzner.nix disk layout and boot for a Hetzner Cloud VM
hosts/asxpio-services/ network, WireGuard, firewall, containers, CI runner
hosts/rehearsal.nix overrides for a throwaway test server
secrets/*.yaml sops-encrypted secrets
migration/ one-off scripts for the move from Debian
Containers run under Podman, one systemd unit each:
systemctl status podman-minecraft, journalctl -u podman-sillytavern.
docker is an alias for podman, and /run/docker.sock is Podman's
Docker-compatible socket, which the CI runner uses to start jobs.
Changing something
Edit, push a branch (CI evaluates the system), merge to main (CI
builds it on the server and switches to it). By hand, over WireGuard:
nixos-rebuild switch --flake .#asxpio-services --target-host root@10.20.0.20
A deploy restarts only the units whose definition changed. Changing a
Minecraft container restarts that server; it gets 120 s to save first.
A bad deploy is undone with nixos-rebuild switch --rollback on the
host, or by picking the previous generation in the boot menu.
To update all packages: nix flake update, commit flake.lock.
Secrets
secrets/asxpio-services.yaml is encrypted with
sops for two keys listed in
.sops.yaml: the admin's age key and the server's SSH host key. The
server decrypts at boot into /run/secrets.
sops secrets/asxpio-services.yaml # edit; needs ~/.config/sops/age/keys.txt
The server's SSH host key is therefore part of its identity. Keep a copy
outside the server (the system archive of a backup contains
/etc/ssh), and hand it to a reinstall with --extra-files.
Installing or reinstalling a server
Never delete and recreate a production server: it is billed at an old price that a new server would not get. Reinstall in place.
-
Put the host key where the installer will copy it from:
mkdir -p extra/etc/ssh # then place ssh_host_ed25519_key{,.pub}, mode 600 -
Install. The running system is replaced; the disk is wiped.
nix run github:nix-community/nixos-anywhere -- \ --flake .#asxpio-services --extra-files extra \ --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20The installer has no WireGuard, so it is reached over the private network through asxpio-ops.
-
Restore data with
migration/restore-services.sh. Containers that own data do not start until/opt/.restoredexists, so nothing generates a fresh world where the restore is about to land.
If a server is ever lost entirely, terraform apply creates a bare one
and the steps above install onto it.
Rehearsal
.#rehearsal is the same system for a throwaway server: plain DHCP, SSH
from anywhere, no WireGuard, dummy secrets. Use it to try an install or
a risky change before production.
Its secrets are encrypted for a host key that was thrown away with the
last test server. For a new one, generate a key, put its age recipient
in .sops.yaml and re-encrypt:
ssh-keygen -t ed25519 -N '' -f extra/etc/ssh/ssh_host_ed25519_key
nix run nixpkgs#ssh-to-age < extra/etc/ssh/ssh_host_ed25519_key.pub # -> &rehearsal
sops updatekeys secrets/rehearsal.yaml