Sergei Poljanski
131124e0de
Write down the cutover for asxpio-services step by step, with the times measured in the rehearsal: backup 6 minutes, restore of 29 GB 3 minutes. The backup script now tolerates files changing under tar in live mode, which a running Minecraft world always does; in final mode, with the servers stopped, that remains an error. The restore script can leave the containers stopped for a restore test. Also checked on the test server before it was deleted: the production network layout (interfaces matched by MAC, static IPv6, the private network on enp7s0) comes up after a reboot with the same routes the Debian host has. Assisted-by: Claude:fable-5-2
95 lines
3.5 KiB
Markdown
95 lines
3.5 KiB
Markdown
# NixOS
|
|
|
|
The whole system of a server, as one flake. `asxpio-services` runs
|
|
NixOS; `asxpio-ops` is still Debian and managed by `../ansible`.
|
|
|
|
```
|
|
flake.nix inputs (nixpkgs 26.05, disko, sops-nix), hosts
|
|
modules/base.nix users, SSH, fail2ban, journal cap
|
|
modules/hetzner.nix disk layout and boot for a Hetzner Cloud VM
|
|
hosts/asxpio-services/ network, WireGuard, firewall, containers, CI runner
|
|
hosts/rehearsal.nix overrides for a throwaway test server
|
|
secrets/*.yaml sops-encrypted secrets
|
|
migration/ one-off scripts for the move from Debian
|
|
```
|
|
|
|
Containers run under Podman, one systemd unit each:
|
|
`systemctl status podman-minecraft`, `journalctl -u podman-sillytavern`.
|
|
`docker` is an alias for `podman`, and `/run/docker.sock` is Podman's
|
|
Docker-compatible socket, which the CI runner uses to start jobs.
|
|
|
|
## Changing something
|
|
|
|
Edit, push a branch (CI evaluates the system), merge to `main` (CI
|
|
builds it on the server and switches to it). By hand, over WireGuard:
|
|
|
|
```sh
|
|
nixos-rebuild switch --flake .#asxpio-services --target-host root@10.20.0.20
|
|
```
|
|
|
|
A deploy restarts only the units whose definition changed. Changing a
|
|
Minecraft container restarts that server; it gets 120 s to save first.
|
|
A bad deploy is undone with `nixos-rebuild switch --rollback` on the
|
|
host, or by picking the previous generation in the boot menu.
|
|
|
|
To update all packages: `nix flake update`, commit `flake.lock`.
|
|
|
|
## Secrets
|
|
|
|
`secrets/asxpio-services.yaml` is encrypted with
|
|
[sops](https://github.com/getsops/sops) for two keys listed in
|
|
`.sops.yaml`: the admin's age key and the server's SSH host key. The
|
|
server decrypts at boot into `/run/secrets`.
|
|
|
|
```sh
|
|
sops secrets/asxpio-services.yaml # edit; needs ~/.config/sops/age/keys.txt
|
|
```
|
|
|
|
The server's SSH host key is therefore part of its identity. Keep a copy
|
|
outside the server (the `system` archive of a backup contains
|
|
`/etc/ssh`), and hand it to a reinstall with `--extra-files`.
|
|
|
|
## Installing or reinstalling a server
|
|
|
|
Never delete and recreate a production server: it is billed at an old
|
|
price that a new server would not get. Reinstall in place.
|
|
|
|
1. Put the host key where the installer will copy it from:
|
|
|
|
```sh
|
|
mkdir -p extra/etc/ssh # then place ssh_host_ed25519_key{,.pub}, mode 600
|
|
```
|
|
|
|
2. Install. The running system is replaced; **the disk is wiped**.
|
|
|
|
```sh
|
|
nix run github:nix-community/nixos-anywhere -- \
|
|
--flake .#asxpio-services --extra-files extra \
|
|
--ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20
|
|
```
|
|
|
|
The installer has no WireGuard, so it is reached over the private
|
|
network through asxpio-ops.
|
|
|
|
3. Restore data with `migration/restore-services.sh`. Containers that
|
|
own data do not start until `/opt/.restored` exists, so nothing
|
|
generates a fresh world where the restore is about to land.
|
|
|
|
If a server is ever lost entirely, `terraform apply` creates a bare one
|
|
and the steps above install onto it.
|
|
|
|
## Rehearsal
|
|
|
|
`.#rehearsal` is the same system for a throwaway server: plain DHCP, SSH
|
|
from anywhere, no WireGuard, dummy secrets. Use it to try an install or
|
|
a risky change before production.
|
|
|
|
Its secrets are encrypted for a host key that was thrown away with the
|
|
last test server. For a new one, generate a key, put its age recipient
|
|
in `.sops.yaml` and re-encrypt:
|
|
|
|
```sh
|
|
ssh-keygen -t ed25519 -N '' -f extra/etc/ssh/ssh_host_ed25519_key
|
|
nix run nixpkgs#ssh-to-age < extra/etc/ssh/ssh_host_ed25519_key.pub # -> &rehearsal
|
|
sops updatekeys secrets/rehearsal.yaml
|
|
```
|