ansible rework

2025-11-04 11:06:35 +02:00 · 2025-11-04 11:06:35 +02:00 · eecff4345a
commit eecff4345a
parent 67414814c0
47 changed files with 1995 additions and 538 deletions
--- a/ansible/.gitignore
+++ b/ansible/.gitignore
@ -0,0 +1,64 @@
+# Ansible-specific
+*.retry
+*.log
+.ansible/
+
+# Vault password files (NEVER commit these!)
+.vault_pass
+.vault_password
+*vault_password*
+vault-password.txt
+
+# Decrypted vault files (NEVER commit these!)
+*_vault.yaml.decrypted
+vault.yaml.decrypted
+*.vault.dec
+*.decrypted
+
+# Backup files
+*.backup
+*.bak
+*~
+*.swp
+*.swo
+
+# WireGuard client configs (may contain private keys)
+playbooks/wireguard-configs/**/*.conf
+playbooks/wireguard-configs/**/private_key
+playbooks/wireguard-configs/**/public_key
+
+# But keep the directory structure
+!playbooks/wireguard-configs/.gitkeep
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+
+# OS files
+.DS_Store
+Thumbs.db
+*.lnk
+
+# Editor files
+.vscode/
+.idea/
+*.sublime-*
+
+# Local testing
+test/
+tmp/
+.tmp/
+
+# SSH control sockets
+/tmp/
+
+# Ansible facts cache
+*.fact
+facts.d/
+
+# Molecule (if using for testing)
+.molecule/
+.cache/
--- a/ansible/README.md
+++ b/ansible/README.md
@ -1,215 +1,60 @@
-# ASXP.IO Ansible
+# ASXP.IO Ansible Infrastructure

-Post-cloud-init configuration for asxpio servers.
+Automated deployment for asxpio servers: WireGuard VPN, SSH hardening, Traefik, Forgejo, and S3 storage.

-## Playbooks
-
-### setup-servers.yaml
-Post-deployment verification (cloud-init handles base setup).
+## Quick Start

 ```bash
-ansible-playbook playbooks/setup-servers.yaml
+# 1. Setup vault
+echo "your-password" > .vault_pass && chmod 600 .vault_pass
+./scripts/setup-vault.sh encrypt
+
+# 2. Stage 1: WireGuard VPN
+ansible-playbook playbooks/01-wireguard-setup.yaml
+# Connect to VPN, update inventory/hosts.yaml to use WireGuard IPs
+
+# 3. Stage 2: Full deployment (SSH hardening, Traefik, Forgejo, Runner)
+ansible-playbook playbooks/02-full-deployment.yaml
+
+# 4. Stage 3: S3 storage mounts
+ansible-playbook playbooks/03-setup-s3fs.yaml
 ```

-### wireguard.yaml
-Deploy WireGuard VPN server on asxpio-ops (10.20.0.10).
+## Infrastructure

-```bash
-# Initial deployment
-ansible-playbook playbooks/wireguard.yaml
+**Servers:**
+- **asxpio-ops** (CX22): Forgejo, Traefik, WireGuard VPN hub
+- **asxpio-services** (CX43): Forgejo Runner, Immich, game servers
+
+**Networks:**
+- VPC: 10.10.0.0/20
+- WireGuard: 10.20.0.0/24
+
+**S3 Buckets (mounted via s3fs):**
+- asxpio-terraform-main → /mnt/s3/terraform
+- asxpio-backups-main → /mnt/s3/backups
+- asxpio-immich-main → /mnt/s3/immich
+
+## Playbook Workflow
+
+1. **WireGuard Setup** - Deploy VPN, generate client configs
+2. **Full Deployment** - SSH hardening (no lockout), Traefik, Forgejo, Runner
+3. **S3 Storage** - Mount S3 buckets with automatic cache management
+
+All playbooks are idempotent - safe to rerun without breaking existing setup.
+
+## Configuration Files

-# Add new clients to inventory/host_vars/asxpio-ops.yaml, then re-run
-ansible-playbook playbooks/wireguard.yaml
 ```
-
-Client configs fetched to `./wireguard-configs/asxpio-ops/`
-
-### wireguard-client.yaml
-Deploy WireGuard client configs to servers.
-
-```bash
-ansible-playbook playbooks/wireguard-client.yaml
+inventory/
+├── hosts.yaml                      # Server inventory, IPs, groups
+├── group_vars/all/                 # Variables applied to all hosts
+│   ├── main.yaml                   # Forgejo version, ports, runner config
+│   ├── network.yaml                # VPC (10.10.0.0/20), WireGuard (10.20.0.0/24) IPs
+│   ├── services.yaml               # Service paths (/opt/*), ports, Docker networks
+│   ├── s3.yaml                     # S3 buckets, mount points, credentials reference
+│   └── vault.yaml                  # ENCRYPTED: S3 keys, Forgejo secrets, runner token
+└── host_vars/                      # Host-specific overrides
+    ├── asxpio-ops.yaml             # WireGuard server, clients list, S3 cache: 10GB
+    └── asxpio-services.yaml        # S3 cache: 80GB
 ```
-
-Deploys asxpio-services.conf (10.20.0.20) to asxpio-services server.
-
-### ssh-harden.yaml
-Harden SSH configuration with network restrictions and strong cryptography.
-
-```bash
-ansible-playbook playbooks/ssh-harden.yaml
-```
-
-**Security measures:**
- Shell access (asxpi/deploy/root): Only from VPC (10.10.0.0/20) or WireGuard (10.20.0.0/24)
- Forgejo git user: Public access on asxpio-ops for git operations (ssh://forgejo@git.asxp.io/...)
- Strong crypto: ChaCha20-Poly1305, AES-GCM, RSA-SHA2, Ed25519
- iptables: SSH blocked on asxpio-services except from trusted networks
- Enhanced sshd_config: MaxAuthTries 3, MaxSessions 5, no X11 forwarding
-
-**Access after hardening:**
- Admin access: Connect via WireGuard VPN first, then SSH
- Git operations: Public access to asxpio-ops (Forgejo handles authentication)
- Internal VPC: Servers can SSH to each other via 10.10.0.0/20
-
-### deploy-traefik.yaml
-Deploy Traefik reverse proxy with Let's Encrypt on asxpio-ops.
-
-```bash
-ansible-playbook playbooks/deploy-traefik.yaml
-```
-
-**Features:**
- Automatic HTTPS with Let's Encrypt (HTTP challenge)
- Dashboard accessible only via WireGuard (http://10.20.0.10:8080)
- Shared Docker network for service discovery
- Certificates stored in `/opt/traefik/letsencrypt/`
- Logs in `/opt/traefik/logs/`
-
-### deploy-forgejo.yaml
-Deploy fresh Forgejo Git server on asxpio-ops (no data migration).
-
-```bash
-ansible-playbook playbooks/deploy-forgejo.yaml
-```
-
-**Configuration:**
- Forgejo 12.x with SQLite database
- HTTP: port 3000 (internal), exposed via Traefik at git.asxp.io
- SSH: port 2222 (external) → 22 (container)
- Actions enabled with 365-day artifact retention
- LFS enabled, OAuth2 enabled
- Data stored in `/opt/forgejo/data/`
-
-**Important:** Configure secrets in `inventory/host_vars/asxpio-ops.yaml`:
-```yaml
-forgejo_secret_key: "YOUR-SECRET-KEY"
-forgejo_internal_token: "YOUR-INTERNAL-TOKEN"
-forgejo_jwt_secret: "YOUR-JWT-SECRET"
-forgejo_lfs_jwt_secret: "YOUR-LFS-JWT-SECRET"
-letsencrypt_email: "admin@asxp.io"
-```
-
-### deploy-forgejo-runner.yaml
-Deploy Forgejo Runner on asxpio-services for CI/CD.
-
-```bash
-ansible-playbook playbooks/deploy-forgejo-runner.yaml
-```
-
-**Configuration:**
- Forgejo Runner 11.x with Docker execution
- 14 pre-configured labels (ubuntu, debian, arch, node, python, nginx, alpine)
- Connects to Forgejo via VPC (http://10.10.0.10:3000)
- Requires manual registration after deployment
-
-**Post-deployment registration:**
-```bash
-# 1. Get registration token from Forgejo:
-#    Site Administration > Actions > Runners > Generate Token
-
-# 2. Register the runner:
-ssh asxpio-services
-docker exec -it forgejo-runner forgejo-runner register --no-interactive \
-  --token YOUR_REGISTRATION_TOKEN \
-  --name runner-asxpio-services \
-  --instance http://10.10.0.10:3000
-
-# 3. Restart runner:
-cd /opt/forgejo-runner && docker compose restart
-```
-
-### backup-forgejo.yaml
-Backup Forgejo from old server (10.0.0.1).
-
-```bash
-ansible-playbook playbooks/backup-forgejo.yaml
-```
-
-**What it does:**
-1. Stops Forgejo service temporarily
-2. Creates dump: `forgejo dump -c /etc/forgejo/app.ini`
-3. Creates additional archive of:
-   - Database: `/srv/gitea/gitea.db`
-   - Repositories: `/srv/gitea/data/`
-   - SSH keys: `/var/lib/forgejo/.ssh/`
-   - Runner registration: `/var/lib/forgejo-runner/.runner`
-4. Fetches backups to local machine: `/tmp/forgejo-backups/`
-5. Restarts Forgejo service
-
-**Backup includes:** Database, repositories (~513MB), configuration, SSH keys, runner registration.
-
-### restore-forgejo.yaml
-Restore Forgejo backup to new server (10.20.0.10 via WireGuard).
-
-```bash
-ansible-playbook playbooks/restore-forgejo.yaml
-```
-
-**Prerequisites:**
- Traefik deployed and running
- Forgejo deployed (fresh instance)
- Backup files in `/tmp/forgejo-backups/`
-
-**What it does:**
-1. Stops Forgejo container
-2. Extracts dump backup to `/opt/forgejo/`
-3. Restores database, repositories, SSH keys
-4. Sets correct ownership (UID 1000)
-5. Starts Forgejo container
-6. Waits for health check
-
-**Post-restore verification:**
-```bash
-# Test web interface
-curl -I https://git.asxp.io
-
-# Test git clone
-git clone https://git.asxp.io/asxpi/infrastructure.git
-
-# Test SSH access
-git clone git@git.asxp.io:asxpi/infrastructure.git
-```
-
-## Deployment Order
-
-### Phase 1: Infrastructure (Prerequisites)
-```bash
-# 1. Deploy VPN
-ansible-playbook playbooks/wireguard.yaml
-ansible-playbook playbooks/wireguard-client.yaml
-
-# 2. Harden SSH (Switch to WireGuard IP's in hosts.yaml before applying)
-ansible-playbook playbooks/ssh-harden.yaml
-
-# 3. Deploy reverse proxy
-ansible-playbook playbooks/deploy-traefik.yaml
-```
-
-### Phase 2: Forgejo Fresh Deployment
-```bash
-# 4. Deploy Forgejo (empty instance)
-ansible-playbook playbooks/deploy-forgejo.yaml
-
-# 5. Deploy runner
-ansible-playbook playbooks/deploy-forgejo-runner.yaml
-
-# 6. Register runner (manual step - see instructions above)
-```
-
-### Phase 3: Data Migration (When Ready)
-```bash
-# 7. Backup from old server
-ansible-playbook playbooks/backup-forgejo.yaml
-
-# 8. Restore to new server
-ansible-playbook playbooks/restore-forgejo.yaml
-
-# 9. Update DNS: git.asxp.io → 65.109.14.216
-# 10. Re-register runner with migrated instance
-```
-
-## Inventory
-
- **asxpio-ops** (65.109.14.216) - Traefik, Forgejo, monitoring, WireGuard server
- **asxpio-services** (95.217.19.240) - Forgejo Runner, Immich, game servers, XMPP
--- a/ansible/ansible.cfg
+++ b/ansible/ansible.cfg
@ -8,8 +8,9 @@ roles_path = roles
 host_key_checking = False
 module_name = shell
 display_skipped_hosts = no
-timeout=30
+timeout = 30
 remote_user = root
+vault_password_file = .vault_pass

 [connection]
 pipelining=True
--- a/ansible/inventory/group_vars/all/main.yaml
+++ b/ansible/inventory/group_vars/all/main.yaml
@ -0,0 +1,21 @@
+---
+# Shared configuration for all servers
+
+# Docker network for Traefik
+traefik_network: traefik
+
+# Forgejo configuration
+forgejo_version: "12"
+forgejo_domain: git.asxp.io
+forgejo_http_port: 3000
+forgejo_ssh_port: 2222  # External port (container uses 22)
+
+# Forgejo Runner configuration
+forgejo_runner_version: "11"
+forgejo_runner_capacity: 1
+forgejo_runner_timeout: 3h
+forgejo_runner_token: "{{ vault_forgejo_runner_token }}"
+
+# Backup configuration
+backup_dest_dir: /opt/forgejo-backup
+backup_old_server: asxpio-old
--- a/ansible/inventory/group_vars/all/network.yaml
+++ b/ansible/inventory/group_vars/all/network.yaml
@ -0,0 +1,24 @@
+---
+# Network configuration for all hosts
+# Centralized network definitions to avoid hardcoding IPs across the codebase
+
+# VPC Network (Hetzner private network)
+vpc_network: "10.10.0.0/20"
+vpc_network_cidr: "10.10.0.0/20"
+
+# WireGuard VPN Network
+wireguard_network: "10.20.0.0/24"
+wireguard_network_cidr: "10.20.0.0/24"
+
+# Host IP Addresses
+# These are used for service communication and should match inventory
+asxpio_ops_vpc_ip: "10.10.0.10"
+asxpio_ops_wireguard_ip: "10.20.0.10"
+
+asxpio_services_vpc_ip: "10.10.0.20"
+asxpio_services_wireguard_ip: "10.20.0.20"
+
+# Trusted Networks for SSH Access
+ssh_trusted_networks:
+  - "{{ wireguard_network }}"
+  - "{{ vpc_network }}"
--- a/ansible/inventory/group_vars/all/s3.yaml
+++ b/ansible/inventory/group_vars/all/s3.yaml
@ -0,0 +1,39 @@
+---
+# S3 Object Storage Configuration
+# These should be overridden in vault or environment-specific vars
+
+# S3 Endpoint Configuration
+s3_region: hel1
+s3_endpoint: "https://{{ s3_region }}.your-objectstorage.com"
+
+# S3 Credentials
+# WARNING: Store these in ansible-vault encrypted file!
+# Override these in group_vars/all/vault.yaml (encrypted)
+s3_access_key: "{{ vault_s3_access_key | default('') }}"
+s3_secret_key: "{{ vault_s3_secret_key | default('') }}"
+
+# S3 Bucket names
+s3_bucket_terraform_main: asxpio-terraform-main
+s3_bucket_terraform_repl: asxpio-terraform-repl
+s3_bucket_backups_main: asxpio-backups-main
+s3_bucket_backups_repl: asxpio-backups-repl
+s3_bucket_immich_main: asxpio-immich-main
+s3_bucket_immich_repl: asxpio-immich-repl
+
+# S3 Buckets to mount on servers (non-replication only)
+s3_buckets:
+  - name: "{{ s3_bucket_terraform_main }}"
+    mount_point: /mnt/s3/terraform
+    owner: root
+    group: root
+    mode: "0755"
+  - name: "{{ s3_bucket_backups_main }}"
+    mount_point: /mnt/s3/backups
+    owner: root
+    group: root
+    mode: "0755"
+  - name: "{{ s3_bucket_immich_main }}"
+    mount_point: /mnt/s3/immich
+    owner: root
+    group: docker
+    mode: "0775"
--- a/ansible/inventory/group_vars/all/services.yaml
+++ b/ansible/inventory/group_vars/all/services.yaml
@ -0,0 +1,30 @@
+---
+# Service configuration for all hosts
+# Centralized service definitions to avoid hardcoding paths and ports
+
+# Base paths for services
+services_base_path: "/opt"
+
+# Service-specific paths (derived from base path)
+traefik_base_path: "{{ services_base_path }}/traefik"
+forgejo_base_path: "{{ services_base_path }}/forgejo"
+forgejo_runner_base_path: "{{ services_base_path }}/forgejo-runner"
+
+# Service ports
+traefik_http_port: 80
+traefik_https_port: 443
+traefik_dashboard_port: 8080
+
+forgejo_http_port: 3000
+forgejo_ssh_port: 2222  # External SSH port for Git operations
+forgejo_internal_ssh_port: 22  # Internal container SSH port
+
+# WireGuard configuration
+wireguard_port: 60101
+wireguard_interface: "wg0"
+
+# SSH configuration
+ssh_port: 22
+
+# Docker configuration
+docker_network_name: "asxpio-net"
--- a/ansible/inventory/group_vars/all/vault.yaml
+++ b/ansible/inventory/group_vars/all/vault.yaml
@ -0,0 +1,87 @@
+$ANSIBLE_VAULT;1.1;AES256
+62313564376664646635373735353930353765633834323532646230633033393738376335313666
+3537393466636230353539313938656537333665636136640a653338306635333331383932663965
+39643137366165623666636135646638623830633563313133326231623030643665633062616336
+3035386162363566660a373666353366393034306135373830356166373861633035346637333339
+32346466363162353663663966323635346338333265303565386166373837626437356632383162
+64666536316438346464666165616437376230343763343738363738373130363833393730303138
+62393736326562313230333331633232363362613839393330353133316536343235383464303634
+34396163626530383564396637336365633033373461623233633630626237336231333466346631
+37323439636239663861653065663532343730313130663663626361353835356662613232663062
+62386130313533613430303035653430313866623335303134383738316430346262633639346334
+63326333653761613239663932306135366236376335643733366535666430633938613966386164
+61633137383866396533323831666462346438633830376561623431333666313531623133656230
+30343862383534623062343537326463623731323435363837383038316539376131363534366164
+30323666663338613466626534376562613539383563633765626238633763396235356639393831
+35636138356435376132336365666363633965623965343165303261323831363366666164373061
+31656239356664623038376231363765386664336135373637383165653966373035353836363264
+39366161646334376430623261353230653930623966616535303565383962633339323563363666
+30343834323430333562666230313334366339313838643662376437323236633761386333373336
+36653132393564653431396662613936636639653231646365353836633830616430323430363561
+33313130313633356539303363613731383930623937643461613665626437623365303766353862
+34386566396530343638313334306530633962653235623561326234363639313233666332616130
+62663235663863666230653363663863663031323563383235356465626463303834623464383835
+61626463343638626530376564343333633738313937326662643133633364353232643135666462
+66363965663436303638363535363332363033313766313335393235363230666234643764653732
+62653261656332326461363961353863346563323638326161656331323538643934383665393536
+37323265396662646163363066643237656235616462373337633338623661363037316131323762
+62653562626237316139363833633039636365643438656230656134363634393139386334376431
+36373534326439323130336431303531366565653263323364323139383632643436613839326434
+35643136363664626630346436656562626363633864356132393035343037633632366436393866
+65616363343062656336303463313431363333336365613632393163356531323838333062613664
+65393430313732346364323736333361363363396433313666636433626139353766346165363637
+34616634313833393162393738376466613338616433393235663930663630356561643237663063
+35363832363365653137313830356666323631653535363063366134303164303364643366613466
+66613030383131633735313065383736316635613437393439393437613363386234356535353534
+32613436346439643835636665656537336534643665323961666632613462633839616637393033
+62383966616666613565653732616164623136323861313365333532393931353338666565336430
+30626638363234623662333638633962313633306536323964343731653362313935363932626331
+38613935643861353830373839393465356334356361656335303732336363306263356463396665
+62633432613838363431653338356639383138343431303632666361313863663938643134626361
+66363666633431343831396637353463366366306135646163633563376437663663353033333865
+38323638333664623037386531323063376533623962366530636137363065616361646464663431
+32326363343163303535643865356539396664393233633639633639346137663635366133636437
+34373839623932626130336265623565313233646562653063343739333539306631353863393337
+61353531303339343963306463613339386530366432646336333039336639326165386562333835
+38373665396138616264646434316333363838333765353436393433656330373738333662373033
+37633234343265373133336331393538303965326366663134353936613439656561356438306231
+62313338383439653835316566663039303330306564626634346165386366393135646137393835
+63316431396439653738656239643565366632616133653039313736333034353032313533363662
+36383661663331633961353839386464656134623337353561613135303865363463346638346263
+32333634343162393266353235326635336332623566353838353566323030663732643531663234
+65616234643931346533333365643263323135366236316363636566613032643636623130306230
+33666266373432303132303438656533653331333636396235623164353730343961376361383838
+35386330616439363662396165643631616638656466313933396339303632323232363638326263
+62326337373534616463633062613261636631656464336663313239636432373230383434363937
+39353764623636343534613639353032643632373334623536373665383538623135316238393937
+33633666636531306666306432666462306137653036643866653365373265646437663362313835
+65613431643030313833333839666663366435613065323033666234653530663837363462383138
+37363739386231646234386265343531303163613563303833623534353832396131336235633362
+32666631653933376130616632333031306331323931336566663764373262376637313465383535
+35356332663931313862376365346361613632306166643433386436666435333233623361323934
+35373031663863343239636435646165323763313137626161623238383365323662623934383162
+34356236386339316533373139663562393531333831666136626162616366633461333863613063
+34376534663135343939373439303966633133353632616530626433303463656634306539653139
+63376561363331306266343033336162663334663638323231323532373239393562383238383831
+32323635356534636637303534393832326566306562633762313363363563643764663063656131
+38323061363438366330613239643937616430313865356366343435343736346633393736333131
+66656635336234663938373433646465383061366139313230663539303365616261336338363565
+36316130303836363366303437373762353061363733343861613530666663363738363032323631
+37613634656532626436343639383431353532316435376232663962636238383862353830313634
+39376262353332663438313062316430383366393265323064613038363763333465373561663430
+62306638623165386636316531396535313134396232343332336663333834636532393530376334
+34333037323830646330303562353439383031356165376564303338323630383337616131636133
+64663665306466633538323330333731386233306130366634646637663839323130633065313462
+66633337346264313561663135323034633636646131303234363132633335313966646534646461
+62353131346666323862323461623437363930333064373439333166656430646237393832373462
+63393430646531323835386462326635313861646335323564643630376131626137616231303335
+39623533383939353966356636656337343336336130643333323236303739393263633562633534
+38666632316432663332626437626337376533393139353731383835376131643035313866386265
+33323333613766333932333637353563366462303763613562333963373537343837653535653631
+33383331396363396439646534323832613165393062623332393537393731373234396634363237
+36343839363163356232333936663230336266376432373433393964313436616637633865636435
+33633835353337613233616438396537393166346464313563643536623835613636373436633562
+33366633326130646138373539656336363034346438326361333539626563613734636138376533
+62313863613237316237346330326330623462346539356330333265383163353733613464326238
+36363131613461643235373937326130383063353534393133636339373439656238333232653263
+6137343166303139343338663639306434656262343139653437
--- a/ansible/inventory/group_vars/all/vault.yaml.template
+++ b/ansible/inventory/group_vars/all/vault.yaml.template
@ -0,0 +1,56 @@
+---
+# Ansible Vault Template
+# Copy this file to vault.yaml and encrypt it with:
+#   ansible-vault encrypt vault.yaml
+#
+# Or create encrypted vault directly:
+#   ansible-vault create vault.yaml
+#
+# This file contains all sensitive secrets used across the infrastructure
+
+# ============================================================================
+# S3 Object Storage Credentials
+# ============================================================================
+vault_s3_access_key: "YOUR_S3_ACCESS_KEY_HERE"
+vault_s3_secret_key: "YOUR_S3_SECRET_KEY_HERE"
+
+# Optional: Alternative S3 endpoint if not using default
+# vault_s3_endpoint: "https://hel1.your-objectstorage.com"
+
+# ============================================================================
+# Forgejo Secrets
+# ============================================================================
+# Generate with: openssl rand -hex 32
+vault_forgejo_secret_key: "GENERATE_64_CHAR_HEX_STRING_HERE"
+
+# Generate with Forgejo or use existing JWT token
+vault_forgejo_internal_token: "eyJhbGc..."
+
+# Generate with: openssl rand -base64 32
+vault_forgejo_jwt_secret: "GENERATE_BASE64_STRING_HERE"
+vault_forgejo_lfs_jwt_secret: "GENERATE_BASE64_STRING_HERE"
+
+# ============================================================================
+# Forgejo Runner
+# ============================================================================
+# Generate from Forgejo UI: Site Admin → Actions → Runners → Create Runner
+vault_forgejo_runner_token: "GENERATE_FROM_FORGEJO_UI"
+
+# ============================================================================
+# SSL/TLS Certificates
+# ============================================================================
+# Email for Let's Encrypt certificate notifications
+vault_letsencrypt_email: "admin@example.com"
+
+# ============================================================================
+# Database Credentials (if needed for future services)
+# ============================================================================
+# vault_postgres_password: "GENERATE_SECURE_PASSWORD"
+# vault_mysql_root_password: "GENERATE_SECURE_PASSWORD"
+# vault_redis_password: "GENERATE_SECURE_PASSWORD"
+
+# ============================================================================
+# API Keys and Tokens (if needed for future services)
+# ============================================================================
+# vault_grafana_admin_password: "GENERATE_SECURE_PASSWORD"
+# vault_immich_db_password: "GENERATE_SECURE_PASSWORD"
--- a/ansible/inventory/host_vars/asxpio-ops.yaml
+++ b/ansible/inventory/host_vars/asxpio-ops.yaml
@ -1,5 +1,10 @@
 ---
-# WireGuard VPN Configuration for asxpio-ops
+# Host-specific configuration for asxpio-ops
+# Operations server: Forgejo, Traefik, WireGuard VPN hub
+
+# ============================================================================
+# WireGuard VPN Configuration
+# ============================================================================
 wireguard_port: 60101
 wireguard_interface: wg0
 wireguard_ipv4: 10.20.0.10
@ -22,16 +27,6 @@ wireguard_clients:
    ipv6: fd69:69:69::2
    preshared_key: true
    persistent_keepalive: 25
-  - name: phone
-    ipv4: 10.20.0.3
-    ipv6: fd69:69:69::3
-    preshared_key: true
-    persistent_keepalive: 25
-  - name: pixel9
-    ipv4: 10.20.0.4
-    ipv6: fd69:69:69::4
-    preshared_key: true
-    persistent_keepalive: 25

 # Optional: DNS servers for clients
 wireguard_dns_servers:
@ -42,14 +37,31 @@ wireguard_dns_servers:
 # wireguard_additional_routes:
 #   - 192.168.1.0/24

+# ============================================================================
 # SSH Hardening Configuration
+# ============================================================================
 # Allow public SSH access for Forgejo git operations
 ssh_allow_public_git: true
 ssh_git_user: git

-# Forgejo secrets (from old server)
-forgejo_secret_key: "***REDACTED***"
-forgejo_internal_token: "***REDACTED***"
-forgejo_jwt_secret: "***REDACTED***"
-forgejo_lfs_jwt_secret: "***REDACTED***"
-letsencrypt_email: "me@asxp.io"
+# ============================================================================
+# Forgejo Configuration
+# ============================================================================
+# Secrets are stored in encrypted vault (group_vars/all/vault.yaml)
+forgejo_secret_key: "{{ vault_forgejo_secret_key }}"
+forgejo_internal_token: "{{ vault_forgejo_internal_token }}"
+forgejo_jwt_secret: "{{ vault_forgejo_jwt_secret }}"
+forgejo_lfs_jwt_secret: "{{ vault_forgejo_lfs_jwt_secret }}"
+
+# ============================================================================
+# SSL/TLS Configuration
+# ============================================================================
+letsencrypt_email: "{{ vault_letsencrypt_email }}"
+
+# ============================================================================
+# S3FS Configuration
+# ============================================================================
+# ops server: CX22 with 40GB disk, limit cache to 10GB
+# Formula: Total_Disk - Desired_Cache_Size = ensure_diskfree
+# 40GB - 10GB = 30GB free
+s3fs_ensure_diskfree: 30720  # 30GB free = max 10GB cache
--- a/ansible/inventory/host_vars/asxpio-services.yaml
+++ b/ansible/inventory/host_vars/asxpio-services.yaml
@ -1,4 +1,17 @@
 ---
+# Host-specific configuration for asxpio-services
+# Services server: Forgejo Runner, Immich, game servers, XMPP
+
+# ============================================================================
 # SSH Hardening Configuration
+# ============================================================================
 # Restrict SSH to VPC and WireGuard networks only
 ssh_allow_public_git: false
+
+# ============================================================================
+# S3FS Configuration
+# ============================================================================
+# services server: CX43 with 160GB disk, limit cache to 80GB
+# Formula: Total_Disk - Desired_Cache_Size = ensure_diskfree
+# 160GB - 80GB = 80GB free
+s3fs_ensure_diskfree: 81920  # 80GB free = max 80GB cache
--- a/ansible/inventory/hosts.yaml
+++ b/ansible/inventory/hosts.yaml
@ -18,8 +18,13 @@ all:
        ops_servers:
          hosts:
            asxpio-ops:
-              ansible_host: 10.20.0.10
-              #ansible_host: 65.109.14.216
+              # Stage 1: Use public IP for WireGuard deployment
+              # ansible_host: 135.181.27.77  # Uncomment for Stage 1 if needed
+              # Stage 2: Use WireGuard IP (uncomment below after 01-wireguard-setup.yaml)
+              ansible_host: "{{ asxpio_ops_wireguard_ip }}"  # 10.20.0.10 via WireGuard
+              public_ip: 135.181.27.77
+              vpc_ip: "{{ asxpio_ops_vpc_ip }}"
+              wireguard_ip: "{{ asxpio_ops_wireguard_ip }}"
              server_role: ops
              server_type: cx22
              services:
@ -33,8 +38,13 @@ all:
        services_servers:
          hosts:
            asxpio-services:
-              ansible_host: 10.20.0.20
-              #ansible_host: 95.217.15.42
+              # Stage 1: Use public IP for WireGuard deployment
+              # ansible_host: 46.62.228.216  # Uncomment for Stage 1 if needed
+              # Stage 2: Use WireGuard IP (uncomment below after 01-wireguard-setup.yaml)
+              ansible_host: "{{ asxpio_services_wireguard_ip }}"  # 10.20.0.20 via WireGuard
+              public_ip: 46.62.228.216
+              vpc_ip: "{{ asxpio_services_vpc_ip }}"
+              wireguard_ip: "{{ asxpio_services_wireguard_ip }}"
              server_role: services
              server_type: cx43
              services:
--- a/ansible/playbooks/01-wireguard-setup.yaml
+++ b/ansible/playbooks/01-wireguard-setup.yaml
@ -0,0 +1,191 @@
+---
+# Stage 1: WireGuard VPN Setup
+#
+# This playbook must be run FIRST to establish the WireGuard VPN network.
+# After this playbook completes:
+# 1. WireGuard will be running on all hosts
+# 2. Client configs will be generated in playbooks/wireguard-configs/
+# 3. You MUST manually update inventory/hosts.yaml to use WireGuard IPs (10.20.0.x)
+# 4. Then you can run playbook 02-full-deployment.yaml
+#
+# Usage:
+#   ansible-playbook playbooks/01-wireguard-setup.yaml
+#
+
+- name: Deploy WireGuard VPN Server
+  hosts: ops_servers
+  become: true
+  gather_facts: true
+
+  pre_tasks:
+    - name: Validate required variables
+      ansible.builtin.assert:
+        that:
+          - wireguard_network is defined
+          - vpc_network is defined
+          - ansible_host is defined
+        fail_msg: "Required network variables are missing. Check group_vars/all/"
+        success_msg: "Network variables validated successfully"
+
+    - name: Display pre-deployment information
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          WireGuard VPN Server Deployment
+          ========================================
+          Target: {{ inventory_hostname }} (VPN Hub)
+          WireGuard network: {{ wireguard_network }}
+          VPC network: {{ vpc_network }}
+
+          This will deploy WireGuard VPN server and generate client configs.
+          ========================================
+
+  roles:
+    - role: wireguard
+
+- name: Deploy WireGuard VPN Client
+  hosts: services_servers
+  become: true
+  gather_facts: true
+
+  vars:
+    # Map each server to its client config name
+    wireguard_client_configs:
+      asxpio-services: asxpio-services
+
+  tasks:
+    - name: Display client deployment information
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          WireGuard VPN Client Deployment
+          ========================================
+          Target: {{ inventory_hostname }} (VPN Client)
+          Will connect to: {{ asxpio_ops_wireguard_ip }} (asxpio-ops)
+          ========================================
+
+    - name: Get client config name for this host
+      ansible.builtin.set_fact:
+        client_config_name: "{{ wireguard_client_configs[inventory_hostname] }}"
+      when: inventory_hostname in wireguard_client_configs
+
+    - name: Install WireGuard
+      ansible.builtin.apt:
+        name:
+          - wireguard
+          - wireguard-tools
+        state: present
+        update_cache: true
+
+    - name: Create WireGuard config directory
+      ansible.builtin.file:
+        path: /etc/wireguard
+        state: directory
+        mode: '0700'
+        owner: root
+        group: root
+
+    - name: Copy WireGuard client configuration
+      ansible.builtin.copy:
+        src: "{{ playbook_dir }}/wireguard-configs/asxpio-ops/{{ client_config_name }}.conf"
+        dest: "/etc/wireguard/{{ client_config_name }}.conf"
+        mode: '0600'
+        owner: root
+        group: root
+      when: client_config_name is defined
+
+    - name: Enable and start WireGuard interface
+      ansible.builtin.systemd:
+        name: "wg-quick@{{ client_config_name }}"
+        enabled: true
+        state: started
+      when: client_config_name is defined
+      register: wg_client_start
+
+    - name: Display client status
+      ansible.builtin.debug:
+        msg: |
+          ✅ WireGuard client configured on {{ inventory_hostname }}
+          Interface: wg-quick@{{ client_config_name }}
+          Status: {{ 'Started' if wg_client_start.changed else 'Already running' }}
+      when: client_config_name is defined
+
+- name: Post-deployment verification
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Display post-deployment instructions
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          WireGuard VPN Deployment Complete!
+          ========================================
+
+          Infrastructure deployed:
+          ✅ asxpio-ops: WireGuard VPN Server (hub)
+          ✅ asxpio-services: WireGuard VPN Client (connected to ops)
+
+          Server-to-server connectivity:
+          - asxpio-ops ↔ asxpio-services via WireGuard (10.20.0.0/24)
+
+          IMPORTANT NEXT STEPS FOR YOUR WORKSTATION:
+
+          1. Client configurations for your machine are in:
+             playbooks/wireguard-configs/asxpio-ops/
+
+          2. Import a client config to connect your workstation:
+             - Linux/Mac: sudo wg-quick up ./playbooks/wireguard-configs/asxpio-ops/<client>.conf
+             - Windows: Import the .conf file in WireGuard GUI
+
+          3. Test connectivity from your workstation:
+             ping {{ asxpio_ops_wireguard_ip }}
+             ping {{ asxpio_services_wireguard_ip }}
+
+          4. After connecting to WireGuard, run Stage 2:
+             ansible-playbook playbooks/02-full-deployment.yaml
+
+          NOTE: Stage 2 automatically uses WireGuard IPs.
+                No manual configuration changes needed!
+
+          ========================================
+
+    - name: Test WireGuard connectivity from controller
+      ansible.builtin.wait_for:
+        host: "{{ asxpio_ops_wireguard_ip }}"
+        port: 22
+        timeout: 5
+        state: started
+      register: wg_connectivity_test
+      failed_when: false
+
+    - name: Display connectivity test results
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          WireGuard Connectivity Test
+          ========================================
+
+          {% if wg_connectivity_test.failed %}
+          ❌ Cannot reach servers via WireGuard from your workstation yet
+
+          This is NORMAL if you haven't connected your workstation to WireGuard.
+
+          To connect:
+          1. Import client config: playbooks/wireguard-configs/asxpio-ops/<client>.conf
+          2. Start WireGuard connection on your workstation
+          3. Verify with: ping {{ asxpio_ops_wireguard_ip }}
+          4. Run Stage 2: ansible-playbook playbooks/02-full-deployment.yaml
+
+          {% else %}
+          ✅ WireGuard is reachable from your workstation!
+
+          Server-to-server VPN: Connected
+          Workstation-to-server VPN: Connected
+
+          You can now run Stage 2:
+            ansible-playbook playbooks/02-full-deployment.yaml
+
+          Stage 2 automatically uses WireGuard IPs - no manual configuration needed!
+          {% endif %}
+
+          ========================================
--- a/ansible/playbooks/02-full-deployment.yaml
+++ b/ansible/playbooks/02-full-deployment.yaml
@ -0,0 +1,272 @@
+---
+# Stage 2: Full Infrastructure Deployment
+#
+# PREREQUISITES:
+# 1. Stage 1 (01-wireguard-setup.yaml) must be completed
+# 2. inventory/hosts.yaml must be updated to use WireGuard IPs (10.20.0.x)
+# 3. You must be connected via WireGuard VPN
+#
+# This playbook deploys the complete infrastructure in one go:
+# - SSH hardening (with connection recovery)
+# - Traefik reverse proxy
+# - Forgejo Git server
+# - Forgejo CI/CD runner
+#
+# Usage:
+#   ansible-playbook playbooks/02-full-deployment.yaml
+#
+
+- name: Verify prerequisites
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Test WireGuard connectivity
+      ansible.builtin.wait_for:
+        host: "{{ asxpio_ops_wireguard_ip }}"
+        port: 22
+        timeout: 10
+        state: started
+      register: wg_test
+      failed_when: false
+
+    - name: Verify WireGuard is reachable
+      ansible.builtin.assert:
+        that:
+          - not wg_test.failed
+        fail_msg: |
+          ❌ ERROR: Cannot reach servers via WireGuard!
+
+          Tested: {{ asxpio_ops_wireguard_ip }}:22
+          Result: Connection failed
+
+          Stage 2 automatically uses WireGuard IPs and REQUIRES an active VPN connection.
+
+          To fix:
+          1. Ensure Stage 1 is complete:
+             ansible-playbook playbooks/01-wireguard-setup.yaml
+
+          2. Import WireGuard config:
+             playbooks/wireguard-configs/<hostname>/<client>.conf
+
+          3. Start WireGuard:
+             - Linux/Mac: sudo wg-quick up <config>.conf
+             - Windows: Import in WireGuard GUI and activate
+
+          4. Test connectivity:
+             ping {{ asxpio_ops_wireguard_ip }}
+             ping {{ asxpio_services_wireguard_ip }}
+
+          5. Re-run this playbook
+
+          Aborting deployment for safety...
+        success_msg: "✅ WireGuard connectivity verified"
+
+    - name: Display pre-deployment checklist
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          Pre-Deployment Checklist
+          ========================================
+
+          ✅ WireGuard connectivity verified
+          ✅ Ready for deployment
+
+          This playbook will:
+          1. Verify common setup
+          2. Harden SSH (will restrict access to WireGuard/VPC only)
+          3. Deploy Traefik reverse proxy
+          4. Deploy Forgejo Git server
+          5. Deploy Forgejo CI/CD runner (requires manual token setup)
+
+          ⚠️  IMPORTANT: This playbook uses WireGuard IPs automatically.
+                         After SSH hardening, you will ONLY be able to
+                         access servers via WireGuard VPN!
+
+          ========================================
+
+    - name: Pause for confirmation
+      ansible.builtin.pause:
+        prompt: "Press ENTER to continue or Ctrl+C to abort"
+
+- name: Common setup verification
+  hosts: asxpio:!old_servers
+  become: true
+  gather_facts: true
+  vars:
+    # Force WireGuard IPs for Stage 2 (SSH hardening requires it)
+    ansible_host: "{{ wireguard_ip }}"
+
+  roles:
+    - role: common
+
+  post_tasks:
+    - name: Display setup status
+      ansible.builtin.debug:
+        msg: "Common setup verified on {{ inventory_hostname }}"
+
+- name: Harden SSH configuration
+  hosts: asxpio:!old_servers
+  become: true
+  gather_facts: true
+  vars:
+    # Force WireGuard IPs for Stage 2 (SSH hardening requires it)
+    ansible_host: "{{ wireguard_ip }}"
+
+  pre_tasks:
+    - name: Display SSH hardening notice
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          SSH Hardening - {{ inventory_hostname }}
+          ========================================
+
+          SSH will be restarted asynchronously.
+          Connection may drop briefly but will automatically recover.
+
+          Trusted networks:
+          - WireGuard: {{ wireguard_network }}
+          - VPC: {{ vpc_network }}
+          ========================================
+
+  roles:
+    - role: ssh_hardening
+
+  post_tasks:
+    - name: Verify SSH connection after hardening
+      ansible.builtin.debug:
+        msg: "SSH hardening complete and connection verified on {{ inventory_hostname }}"
+
+- name: Deploy Traefik reverse proxy
+  hosts: asxpio-ops
+  become: true
+  gather_facts: true
+  vars:
+    # Force WireGuard IPs for Stage 2
+    ansible_host: "{{ wireguard_ip }}"
+
+  roles:
+    - role: traefik
+
+  post_tasks:
+    - name: Display Traefik status
+      ansible.builtin.debug:
+        msg: |
+          Traefik deployed successfully on {{ inventory_hostname }}
+          Dashboard: http://{{ asxpio_ops_wireguard_ip }}:{{ traefik_dashboard_port }}
+
+- name: Deploy Forgejo Git server
+  hosts: asxpio-ops
+  become: true
+  gather_facts: true
+  vars:
+    # Force WireGuard IPs for Stage 2
+    ansible_host: "{{ wireguard_ip }}"
+
+  roles:
+    - role: forgejo
+
+  post_tasks:
+    - name: Display Forgejo status
+      ansible.builtin.debug:
+        msg: |
+          Forgejo deployed successfully on {{ inventory_hostname }}
+          Web UI: http://{{ asxpio_ops_wireguard_ip }}:{{ forgejo_http_port }}
+          SSH: ssh://git@{{ asxpio_ops_wireguard_ip }}:{{ forgejo_ssh_port }}
+
+          ⚠️  IMPORTANT: Before deploying the runner, you must:
+          1. Access Forgejo web UI
+          2. Complete initial setup (create admin user)
+          3. Go to: Site Administration → Actions → Runners
+          4. Generate a runner registration token
+          5. Update forgejo_runner_token in inventory/group_vars/all.yaml (or vault)
+
+    - name: Pause for Forgejo runner token generation
+      ansible.builtin.pause:
+        prompt: |
+
+          ========================================
+          Forgejo Runner Registration Token Required
+          ========================================
+
+          The runner cannot be deployed without a registration token from Forgejo.
+
+          Steps to generate the token:
+          1. Open Forgejo: http://{{ asxpio_ops_wireguard_ip }}:{{ forgejo_http_port }}
+          2. Complete initial setup if not done (create admin account)
+          3. Login as admin
+          4. Navigate to: Site Administration → Actions → Runners
+          5. Click "Create new Runner" or "Generate Token"
+          6. Copy the registration token
+          7. Update forgejo_runner_token in inventory/group_vars/all.yaml
+             OR better: ansible-vault edit inventory/group_vars/vault.yaml
+
+          Press ENTER when ready to continue with runner deployment, or Ctrl+C to abort
+          ========================================
+
+- name: Deploy Forgejo CI/CD Runner
+  hosts: asxpio-services
+  become: true
+  gather_facts: true
+  vars:
+    # Force WireGuard IPs for Stage 2
+    ansible_host: "{{ wireguard_ip }}"
+
+  pre_tasks:
+    - name: Verify runner token is configured
+      ansible.builtin.assert:
+        that:
+          - forgejo_runner_token is defined
+          - forgejo_runner_token | length > 0
+          - forgejo_runner_token != "REPLACE_WITH_ACTUAL_TOKEN"
+        fail_msg: |
+          ERROR: forgejo_runner_token is not properly configured!
+
+          You must set a valid runner registration token from Forgejo:
+          1. Get token from: Site Administration → Actions → Runners
+          2. Set in inventory/group_vars/all.yaml or vault.yaml
+          3. Re-run this playbook
+        success_msg: "Runner token validated"
+
+  roles:
+    - role: forgejo_runner
+
+  post_tasks:
+    - name: Display runner status
+      ansible.builtin.debug:
+        msg: |
+          Forgejo Runner deployed successfully on {{ inventory_hostname }}
+          Check status: docker logs forgejo-runner
+
+- name: Display deployment summary
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Show final status
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          Full Infrastructure Deployment Complete!
+          ========================================
+
+          Services deployed:
+
+          Traefik (asxpio-services):
+            - Dashboard: http://{{ asxpio_services_wireguard_ip }}:8080
+
+          Forgejo (asxpio-ops):
+            - Web UI: http://{{ asxpio_ops_wireguard_ip }}:3000
+            - Git SSH: ssh://git@{{ asxpio_ops_wireguard_ip }}:2222
+
+          Forgejo Runner (asxpio-services):
+            - Status: docker logs forgejo-runner
+
+          SSH Access:
+            - Restricted to WireGuard (10.20.0.0/24) and VPC (10.10.0.0/20)
+            - Connect via WireGuard VPN to access servers
+
+          Next steps:
+            - Configure Forgejo admin account
+            - Set up repositories
+            - Test CI/CD pipeline
+
+          ========================================
--- a/ansible/playbooks/03-setup-s3fs.yaml
+++ b/ansible/playbooks/03-setup-s3fs.yaml
@ -0,0 +1,134 @@
+---
+# Stage 3: Setup S3FS on all servers
+#
+# This playbook sets up s3fs to mount S3 buckets as local filesystems.
+# Must be run after Stage 2 (full deployment) to ensure Docker is installed.
+#
+# Mounts the following buckets:
+# - asxpio-terraform-main -> /mnt/s3/terraform
+# - asxpio-backups-main -> /mnt/s3/backups
+# - asxpio-immich-main -> /mnt/s3/immich
+#
+# Cache sizes (per-host):
+# - asxpio-ops (40GB disk): 10GB cache max
+# - asxpio-services (160GB disk): 80GB cache max
+#
+# Usage:
+#   ansible-playbook playbooks/03-setup-s3fs.yaml
+#
+# IMPORTANT: Ensure S3 credentials are set in group_vars/all/vault.yaml:
+#   vault_s3_access_key: "your-access-key"
+#   vault_s3_secret_key: "your-secret-key"
+#
+
+- name: Setup S3FS on all servers
+  hosts: ops_servers,services_servers
+  become: true
+  gather_facts: true
+
+  pre_tasks:
+    - name: Validate S3 credentials are set
+      ansible.builtin.assert:
+        that:
+          - s3_access_key is defined
+          - s3_access_key != ""
+          - s3_secret_key is defined
+          - s3_secret_key != ""
+        fail_msg: |
+          S3 credentials not set!
+          Please set vault_s3_access_key and vault_s3_secret_key in:
+          - group_vars/all/vault.yaml (encrypted with ansible-vault)
+          OR
+          - Pass via command line: -e "s3_access_key=xxx s3_secret_key=yyy"
+        success_msg: "S3 credentials validated"
+
+    - name: Display S3 configuration
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          S3FS Setup on {{ inventory_hostname }}
+          ========================================
+          Target: {{ inventory_hostname }}
+          S3 Region: {{ s3_region }}
+          S3 Endpoint: {{ s3_endpoint }}
+
+          Buckets to mount:
+          {% for bucket in s3_buckets %}
+          - {{ bucket.name }} -> {{ bucket.mount_point }}
+          {% endfor %}
+          ========================================
+
+  roles:
+    - role: s3fs
+
+  post_tasks:
+    - name: Display mounted filesystems
+      ansible.builtin.command:
+        cmd: df -h
+      register: df_output
+      changed_when: false
+
+    - name: Show S3 mounts
+      ansible.builtin.debug:
+        msg: "{{ df_output.stdout_lines | select('search', 's3fs') | list }}"
+
+    - name: Create immich library directory structure
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        mode: '0775'
+        owner: root
+        group: docker
+      loop:
+        - /mnt/s3/immich/library
+        - /mnt/s3/immich/library/backups
+        - /mnt/s3/immich/library/encoded-video
+        - /mnt/s3/immich/library/library
+        - /mnt/s3/immich/library/profile
+        - /mnt/s3/immich/library/thumbs
+        - /mnt/s3/immich/library/upload
+      when: inventory_hostname in groups['services_servers']
+
+    - name: Display post-setup information
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          S3FS Setup Complete on {{ inventory_hostname }}!
+          ========================================
+
+          ✅ All S3 buckets mounted successfully
+
+          Mount points:
+          {% for bucket in s3_buckets %}
+          - {{ bucket.mount_point }} ({{ bucket.name }})
+          {% endfor %}
+
+          Cache configuration:
+          - Location: /var/cache/s3fs
+          - Max size: ~{{ (40 - (s3fs_ensure_diskfree / 1024)) | int }}GB
+          - Free space reserved: {{ (s3fs_ensure_diskfree / 1024) | int }}GB
+
+          {% if inventory_hostname in groups['services_servers'] %}
+          Immich library structure created at:
+          /mnt/s3/immich/library/
+          ├── backups/
+          ├── encoded-video/
+          ├── library/
+          ├── profile/
+          ├── thumbs/
+          └── upload/
+
+          Next steps for Immich migration:
+          1. Manually sync data from legacy server:
+             rsync -avz --info=progress2 \
+               asxpio-old:/srv/immich/library/ \
+               /mnt/s3/immich/library/
+
+          2. Backup Immich PostgreSQL database on legacy server:
+             docker exec immich_postgres pg_dumpall -U postgres | gzip > /tmp/immich-db.sql.gz
+
+          3. Deploy Immich service configured to use S3 storage:
+             IMMICH_MEDIA_LOCATION=/mnt/s3/immich/library
+          {% endif %}
+
+          ========================================
--- a/ansible/playbooks/deploy-forgejo-runner-standalone.yaml
+++ b/ansible/playbooks/deploy-forgejo-runner-standalone.yaml
@ -0,0 +1,226 @@
+---
+# Standalone Forgejo Runner Deployment
+#
+# PREREQUISITES:
+# 1. Forgejo must be deployed and running
+# 2. You must have created an admin account in Forgejo
+# 3. You must generate a runner registration token from Forgejo UI
+#
+# To get the registration token:
+#   1. Login to Forgejo as admin
+#   2. Go to: Site Administration → Actions → Runners
+#   3. Click "Create new Runner" or "Generate Token"
+#   4. Copy the token and update forgejo_runner_token variable
+#
+# Usage:
+#   ansible-playbook playbooks/deploy-forgejo-runner-standalone.yaml
+#
+
+- name: Verify Forgejo is accessible
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Check Forgejo is running
+      ansible.builtin.uri:
+        url: "http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}/"
+        status_code: 200
+        timeout: 10
+      register: forgejo_check
+      failed_when: false
+      delegate_to: localhost
+
+    - name: Display Forgejo status
+      ansible.builtin.debug:
+        msg: |
+          {% if forgejo_check.status == 200 %}
+          ✅ Forgejo is accessible at http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
+          {% else %}
+          ❌ ERROR: Cannot reach Forgejo!
+
+          Please ensure:
+          1. Forgejo is deployed and running
+          2. You can access: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
+          3. You are connected to the VPC network or WireGuard VPN
+
+          Deploy Forgejo first:
+            ansible-playbook playbooks/02-full-deployment.yaml
+
+          Or deploy just Forgejo:
+            ansible-playbook playbooks/legacy/deploy-forgejo.yaml
+          {% endif %}
+
+    - name: Fail if Forgejo is not accessible
+      ansible.builtin.fail:
+        msg: "Forgejo is not accessible. Please deploy Forgejo first."
+      when: forgejo_check.status != 200
+
+- name: Get runner registration token
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Prompt for runner token configuration
+      ansible.builtin.pause:
+        prompt: |
+
+          ========================================
+          Forgejo Runner Registration Token Required
+          ========================================
+
+          Before deploying the runner, you MUST generate a registration token from Forgejo.
+
+          Steps to generate the token:
+
+          1. Open Forgejo: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
+
+          2. Login as administrator
+
+          3. Navigate to: Site Administration → Actions → Runners
+             (Click your profile icon → Site Administration → Actions → Runners)
+
+          4. Click "Create new Runner" or "Generate registration token"
+
+          5. Copy the generated token (it looks like: "eyJhbGc...")
+
+          6. Update the token in one of these locations:
+
+             Option A - Plaintext (NOT RECOMMENDED):
+               Edit: inventory/group_vars/all.yaml
+               Set: forgejo_runner_token: "YOUR-TOKEN-HERE"
+
+             Option B - Ansible Vault (RECOMMENDED):
+               ansible-vault edit inventory/group_vars/vault.yaml
+               Set: vault_forgejo_runner_token: "YOUR-TOKEN-HERE"
+               Then in all.yaml: forgejo_runner_token: "{{ vault_forgejo_runner_token }}"
+
+          7. Save the file
+
+          8. Press ENTER to continue with runner deployment
+
+          Or press Ctrl+C to abort and configure the token first
+          ========================================
+
+- name: Deploy Forgejo CI/CD Runner
+  hosts: asxpio-services
+  become: true
+  gather_facts: true
+
+  pre_tasks:
+    - name: Verify runner token is configured
+      ansible.builtin.assert:
+        that:
+          - forgejo_runner_token is defined
+          - forgejo_runner_token | length > 0
+          - forgejo_runner_token != "REPLACE_WITH_ACTUAL_TOKEN"
+          - forgejo_runner_token != "***REDACTED***"  # Old token from git
+        fail_msg: |
+          ❌ ERROR: forgejo_runner_token is not properly configured!
+
+          Current value: {{ forgejo_runner_token | default('NOT SET') }}
+
+          You must set a valid runner registration token from Forgejo:
+
+          1. Get token from Forgejo UI:
+             Site Administration → Actions → Runners → Generate Token
+
+          2. Update in inventory/group_vars/all.yaml:
+             forgejo_runner_token: "YOUR-TOKEN-HERE"
+
+             OR better, use Ansible Vault (recommended):
+             ansible-vault edit inventory/group_vars/vault.yaml
+             vault_forgejo_runner_token: "YOUR-TOKEN-HERE"
+
+          3. Re-run this playbook
+
+          Aborting deployment...
+        success_msg: "✅ Runner token validated (length: {{ forgejo_runner_token | length }} chars)"
+
+    - name: Display deployment information
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          Deploying Forgejo Runner
+          ========================================
+
+          Target: {{ inventory_hostname }}
+          Forgejo instance: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
+          Runner name: runner-{{ inventory_hostname }}
+
+          This will:
+          1. Deploy runner container
+          2. Register with Forgejo instance
+          3. Start accepting CI/CD jobs
+          ========================================
+
+  roles:
+    - role: forgejo_runner
+
+  post_tasks:
+    - name: Wait for runner registration
+      ansible.builtin.pause:
+        seconds: 5
+        prompt: "Waiting for runner to register with Forgejo..."
+
+    - name: Display runner status
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          Forgejo Runner Deployment Complete!
+          ========================================
+
+          Runner deployed successfully on {{ inventory_hostname }}
+
+          Check status:
+            docker logs forgejo-runner
+
+          Verify in Forgejo UI:
+            1. Login to Forgejo
+            2. Go to: Site Administration → Actions → Runners
+            3. You should see: runner-{{ inventory_hostname }} (idle/active)
+
+          Test the runner:
+            1. Create a repository with .forgejo/workflows/test.yml
+            2. Push a commit
+            3. Check Actions tab for workflow run
+
+          Runner supports these labels:
+            - ubuntu-latest, ubuntu-22.04, ubuntu-24.04
+            - debian-latest, debian-12, debian-11
+            - node-20, node-22
+            - python-3.11, python-3.12
+            - alpine-latest, arch-latest
+            - nginx-latest, nginx-alpine
+
+          Example workflow:
+            .forgejo/workflows/test.yml:
+            ---
+            name: Test
+            on: [push]
+            jobs:
+              test:
+                runs-on: ubuntu-latest
+                steps:
+                  - uses: actions/checkout@v4
+                  - run: echo "Hello from Forgejo Actions!"
+
+          ========================================
+
+- name: Verify runner in Forgejo
+  hosts: localhost
+  gather_facts: false
+  tasks:
+    - name: Reminder to check Forgejo UI
+      ansible.builtin.debug:
+        msg: |
+
+          ⚠️  FINAL STEP: Verify runner in Forgejo UI
+
+          1. Open: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
+          2. Login as admin
+          3. Go to: Site Administration → Actions → Runners
+          4. Verify runner-asxpio-services appears in the list
+          5. Status should be "idle" (green) or "active"
+
+          If the runner doesn't appear:
+            - Check logs: docker logs forgejo-runner
+            - Verify token is correct
+            - Ensure runner can reach Forgejo at {{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
--- a/ansible/playbooks/deploy-forgejo-runner.yaml
+++ b/ansible/playbooks/deploy-forgejo-runner.yaml
@ -1,6 +0,0 @@
---
- name: Deploy Forgejo Runner
-  hosts: asxpio-services
-  become: true
-  roles:
-    - forgejo_runner
--- a/ansible/playbooks/deploy-forgejo.yaml
+++ b/ansible/playbooks/deploy-forgejo.yaml
@ -1,6 +0,0 @@
---
- name: Deploy Forgejo Git server
-  hosts: asxpio-ops
-  become: true
-  roles:
-    - forgejo
--- a/ansible/playbooks/deploy-traefik.yaml
+++ b/ansible/playbooks/deploy-traefik.yaml
@ -1,6 +0,0 @@
---
- name: Deploy Traefik reverse proxy
-  hosts: asxpio-ops
-  become: true
-  roles:
-    - traefik
--- a/ansible/playbooks/legacy/backup-forgejo.yaml
+++ b/ansible/playbooks/legacy/backup-forgejo.yaml
@ -0,0 +1,6 @@
+---
+- name: Backup Forgejo from old server
+  hosts: asxpio-old
+  become: true
+  roles:
+    - forgejo_backup
--- a/Show more
+++ b/Show more