ansible rework

2025-11-04 11:06:35 +02:00 · 2025-11-04 11:06:35 +02:00 · eecff4345a
commit eecff4345a
parent 67414814c0
47 changed files with 1995 additions and 538 deletions
--- a/ansible/.gitignore
+++ b/ansible/.gitignore
@ -0,0 +1,64 @@
 # Ansible-specific
 *.retry
 *.log
 .ansible/
 # Vault password files (NEVER commit these!)
 .vault_pass
 .vault_password
 *vault_password*
 vault-password.txt
 # Decrypted vault files (NEVER commit these!)
 *_vault.yaml.decrypted
 vault.yaml.decrypted
 *.vault.dec
 *.decrypted
 # Backup files
 *.backup
 *.bak
 *~
 *.swp
 *.swo
 # WireGuard client configs (may contain private keys)
 playbooks/wireguard-configs/**/*.conf
 playbooks/wireguard-configs/**/private_key
 playbooks/wireguard-configs/**/public_key
 # But keep the directory structure
 !playbooks/wireguard-configs/.gitkeep
 # Python
 __pycache__/
 *.py[cod]
 *$py.class
 *.so
 .Python
 # OS files
 .DS_Store
 Thumbs.db
 *.lnk
 # Editor files
 .vscode/
 .idea/
 *.sublime-*
 # Local testing
 test/
 tmp/
 .tmp/
 # SSH control sockets
 /tmp/
 # Ansible facts cache
 *.fact
 facts.d/
 # Molecule (if using for testing)
 .molecule/
 .cache/
--- a/ansible/README.md
+++ b/ansible/README.md
@ -1,215 +1,60 @@
-# ASXP.IO Ansible
+# ASXP.IO Ansible Infrastructure
-Post-cloud-init configuration for asxpio servers.
+Automated deployment for asxpio servers: WireGuard VPN, SSH hardening, Traefik, Forgejo, and S3 storage.
-## Playbooks
+## Quick Start
 ### setup-servers.yaml
 Post-deployment verification (cloud-init handles base setup).
 ```bash
-ansible-playbook playbooks/setup-servers.yaml
+# 1. Setup vault
 echo "your-password" > .vault_pass && chmod 600 .vault_pass
 ./scripts/setup-vault.sh encrypt
 # 2. Stage 1: WireGuard VPN
 ansible-playbook playbooks/01-wireguard-setup.yaml
 # Connect to VPN, update inventory/hosts.yaml to use WireGuard IPs
 # 3. Stage 2: Full deployment (SSH hardening, Traefik, Forgejo, Runner)
 ansible-playbook playbooks/02-full-deployment.yaml
 # 4. Stage 3: S3 storage mounts
 ansible-playbook playbooks/03-setup-s3fs.yaml
 ```
-### wireguard.yaml
+## Infrastructure
 Deploy WireGuard VPN server on asxpio-ops (10.20.0.10).
-```bash
+**Servers:**
-# Initial deployment
+- **asxpio-ops** (CX22): Forgejo, Traefik, WireGuard VPN hub
-ansible-playbook playbooks/wireguard.yaml
+- **asxpio-services** (CX43): Forgejo Runner, Immich, game servers
 **Networks:**
 - VPC: 10.10.0.0/20
 - WireGuard: 10.20.0.0/24
 **S3 Buckets (mounted via s3fs):**
 - asxpio-terraform-main → /mnt/s3/terraform
 - asxpio-backups-main → /mnt/s3/backups
 - asxpio-immich-main → /mnt/s3/immich
 ## Playbook Workflow
 1. **WireGuard Setup** - Deploy VPN, generate client configs
 2. **Full Deployment** - SSH hardening (no lockout), Traefik, Forgejo, Runner
 3. **S3 Storage** - Mount S3 buckets with automatic cache management
 All playbooks are idempotent - safe to rerun without breaking existing setup.
 ## Configuration Files
 # Add new clients to inventory/host_vars/asxpio-ops.yaml, then re-run
 ansible-playbook playbooks/wireguard.yaml
 ```
-
+inventory/
-Client configs fetched to `./wireguard-configs/asxpio-ops/`
+├── hosts.yaml                      # Server inventory, IPs, groups
-
+├── group_vars/all/                 # Variables applied to all hosts
-### wireguard-client.yaml
+│   ├── main.yaml                   # Forgejo version, ports, runner config
-Deploy WireGuard client configs to servers.
+│   ├── network.yaml                # VPC (10.10.0.0/20), WireGuard (10.20.0.0/24) IPs
-
+│   ├── services.yaml               # Service paths (/opt/*), ports, Docker networks
-```bash
+│   ├── s3.yaml                     # S3 buckets, mount points, credentials reference
-ansible-playbook playbooks/wireguard-client.yaml
+│   └── vault.yaml                  # ENCRYPTED: S3 keys, Forgejo secrets, runner token
 └── host_vars/                      # Host-specific overrides
    ├── asxpio-ops.yaml             # WireGuard server, clients list, S3 cache: 10GB
    └── asxpio-services.yaml        # S3 cache: 80GB
 ```
 Deploys asxpio-services.conf (10.20.0.20) to asxpio-services server.
 ### ssh-harden.yaml
 Harden SSH configuration with network restrictions and strong cryptography.
 ```bash
 ansible-playbook playbooks/ssh-harden.yaml
 ```
 **Security measures:**
 - Shell access (asxpi/deploy/root): Only from VPC (10.10.0.0/20) or WireGuard (10.20.0.0/24)
 - Forgejo git user: Public access on asxpio-ops for git operations (ssh://forgejo@git.asxp.io/...)
 - Strong crypto: ChaCha20-Poly1305, AES-GCM, RSA-SHA2, Ed25519
 - iptables: SSH blocked on asxpio-services except from trusted networks
 - Enhanced sshd_config: MaxAuthTries 3, MaxSessions 5, no X11 forwarding
 **Access after hardening:**
 - Admin access: Connect via WireGuard VPN first, then SSH
 - Git operations: Public access to asxpio-ops (Forgejo handles authentication)
 - Internal VPC: Servers can SSH to each other via 10.10.0.0/20
 ### deploy-traefik.yaml
 Deploy Traefik reverse proxy with Let's Encrypt on asxpio-ops.
 ```bash
 ansible-playbook playbooks/deploy-traefik.yaml
 ```
 **Features:**
 - Automatic HTTPS with Let's Encrypt (HTTP challenge)
 - Dashboard accessible only via WireGuard (http://10.20.0.10:8080)
 - Shared Docker network for service discovery
 - Certificates stored in `/opt/traefik/letsencrypt/`
 - Logs in `/opt/traefik/logs/`
 ### deploy-forgejo.yaml
 Deploy fresh Forgejo Git server on asxpio-ops (no data migration).
 ```bash
 ansible-playbook playbooks/deploy-forgejo.yaml
 ```
 **Configuration:**
 - Forgejo 12.x with SQLite database
 - HTTP: port 3000 (internal), exposed via Traefik at git.asxp.io
 - SSH: port 2222 (external) → 22 (container)
 - Actions enabled with 365-day artifact retention
 - LFS enabled, OAuth2 enabled
 - Data stored in `/opt/forgejo/data/`
 **Important:** Configure secrets in `inventory/host_vars/asxpio-ops.yaml`:
 ```yaml
 forgejo_secret_key: "YOUR-SECRET-KEY"
 forgejo_internal_token: "YOUR-INTERNAL-TOKEN"
 forgejo_jwt_secret: "YOUR-JWT-SECRET"
 forgejo_lfs_jwt_secret: "YOUR-LFS-JWT-SECRET"
 letsencrypt_email: "admin@asxp.io"
 ```
 ### deploy-forgejo-runner.yaml
 Deploy Forgejo Runner on asxpio-services for CI/CD.
 ```bash
 ansible-playbook playbooks/deploy-forgejo-runner.yaml
 ```
 **Configuration:**
 - Forgejo Runner 11.x with Docker execution
 - 14 pre-configured labels (ubuntu, debian, arch, node, python, nginx, alpine)
 - Connects to Forgejo via VPC (http://10.10.0.10:3000)
 - Requires manual registration after deployment
 **Post-deployment registration:**
 ```bash
 # 1. Get registration token from Forgejo:
 #    Site Administration > Actions > Runners > Generate Token
 # 2. Register the runner:
 ssh asxpio-services
 docker exec -it forgejo-runner forgejo-runner register --no-interactive \
  --token YOUR_REGISTRATION_TOKEN \
  --name runner-asxpio-services \
  --instance http://10.10.0.10:3000
 # 3. Restart runner:
 cd /opt/forgejo-runner && docker compose restart
 ```
 ### backup-forgejo.yaml
 Backup Forgejo from old server (10.0.0.1).
 ```bash
 ansible-playbook playbooks/backup-forgejo.yaml
 ```
 **What it does:**
 1. Stops Forgejo service temporarily
 2. Creates dump: `forgejo dump -c /etc/forgejo/app.ini`
 3. Creates additional archive of:
   - Database: `/srv/gitea/gitea.db`
   - Repositories: `/srv/gitea/data/`
   - SSH keys: `/var/lib/forgejo/.ssh/`
   - Runner registration: `/var/lib/forgejo-runner/.runner`
 4. Fetches backups to local machine: `/tmp/forgejo-backups/`
 5. Restarts Forgejo service
 **Backup includes:** Database, repositories (~513MB), configuration, SSH keys, runner registration.
 ### restore-forgejo.yaml
 Restore Forgejo backup to new server (10.20.0.10 via WireGuard).
 ```bash
 ansible-playbook playbooks/restore-forgejo.yaml
 ```
 **Prerequisites:**
 - Traefik deployed and running
 - Forgejo deployed (fresh instance)
 - Backup files in `/tmp/forgejo-backups/`
 **What it does:**
 1. Stops Forgejo container
 2. Extracts dump backup to `/opt/forgejo/`
 3. Restores database, repositories, SSH keys
 4. Sets correct ownership (UID 1000)
 5. Starts Forgejo container
 6. Waits for health check
 **Post-restore verification:**
 ```bash
 # Test web interface
 curl -I https://git.asxp.io
 # Test git clone
 git clone https://git.asxp.io/asxpi/infrastructure.git
 # Test SSH access
 git clone git@git.asxp.io:asxpi/infrastructure.git
 ```
 ## Deployment Order
 ### Phase 1: Infrastructure (Prerequisites)
 ```bash
 # 1. Deploy VPN
 ansible-playbook playbooks/wireguard.yaml
 ansible-playbook playbooks/wireguard-client.yaml
 # 2. Harden SSH (Switch to WireGuard IP's in hosts.yaml before applying)
 ansible-playbook playbooks/ssh-harden.yaml
 # 3. Deploy reverse proxy
 ansible-playbook playbooks/deploy-traefik.yaml
 ```
 ### Phase 2: Forgejo Fresh Deployment
 ```bash
 # 4. Deploy Forgejo (empty instance)
 ansible-playbook playbooks/deploy-forgejo.yaml
 # 5. Deploy runner
 ansible-playbook playbooks/deploy-forgejo-runner.yaml
 # 6. Register runner (manual step - see instructions above)
 ```
 ### Phase 3: Data Migration (When Ready)
 ```bash
 # 7. Backup from old server
 ansible-playbook playbooks/backup-forgejo.yaml
 # 8. Restore to new server
 ansible-playbook playbooks/restore-forgejo.yaml
 # 9. Update DNS: git.asxp.io → 65.109.14.216
 # 10. Re-register runner with migrated instance
 ```
 ## Inventory
 - **asxpio-ops** (65.109.14.216) - Traefik, Forgejo, monitoring, WireGuard server
 - **asxpio-services** (95.217.19.240) - Forgejo Runner, Immich, game servers, XMPP
--- a/ansible/ansible.cfg
+++ b/ansible/ansible.cfg
@ -8,8 +8,9 @@ roles_path = roles
 host_key_checking = False
 module_name = shell
 display_skipped_hosts = no
-timeout=30
+timeout = 30
 remote_user = root
 vault_password_file = .vault_pass
 [connection]
 pipelining=True
--- a/ansible/inventory/group_vars/all/main.yaml
+++ b/ansible/inventory/group_vars/all/main.yaml
@ -14,7 +14,7 @@ forgejo_ssh_port: 2222  # External port (container uses 22)
 forgejo_runner_version: "11"
 forgejo_runner_capacity: 1
 forgejo_runner_timeout: 3h
-forgejo_runner_token: "***REDACTED***"
+forgejo_runner_token: "{{ vault_forgejo_runner_token }}"
 # Backup configuration
 backup_dest_dir: /opt/forgejo-backup
--- a/ansible/inventory/group_vars/all/network.yaml
+++ b/ansible/inventory/group_vars/all/network.yaml
@ -0,0 +1,24 @@
 ---
 # Network configuration for all hosts
 # Centralized network definitions to avoid hardcoding IPs across the codebase
 # VPC Network (Hetzner private network)
 vpc_network: "10.10.0.0/20"
 vpc_network_cidr: "10.10.0.0/20"
 # WireGuard VPN Network
 wireguard_network: "10.20.0.0/24"
 wireguard_network_cidr: "10.20.0.0/24"
 # Host IP Addresses
 # These are used for service communication and should match inventory
 asxpio_ops_vpc_ip: "10.10.0.10"
 asxpio_ops_wireguard_ip: "10.20.0.10"
 asxpio_services_vpc_ip: "10.10.0.20"
 asxpio_services_wireguard_ip: "10.20.0.20"
 # Trusted Networks for SSH Access
 ssh_trusted_networks:
  - "{{ wireguard_network }}"
  - "{{ vpc_network }}"
--- a/ansible/inventory/group_vars/all/s3.yaml
+++ b/ansible/inventory/group_vars/all/s3.yaml
@ -0,0 +1,39 @@
 ---
 # S3 Object Storage Configuration
 # These should be overridden in vault or environment-specific vars
 # S3 Endpoint Configuration
 s3_region: hel1
 s3_endpoint: "https://{{ s3_region }}.your-objectstorage.com"
 # S3 Credentials
 # WARNING: Store these in ansible-vault encrypted file!
 # Override these in group_vars/all/vault.yaml (encrypted)
 s3_access_key: "{{ vault_s3_access_key | default('') }}"
 s3_secret_key: "{{ vault_s3_secret_key | default('') }}"
 # S3 Bucket names
 s3_bucket_terraform_main: asxpio-terraform-main
 s3_bucket_terraform_repl: asxpio-terraform-repl
 s3_bucket_backups_main: asxpio-backups-main
 s3_bucket_backups_repl: asxpio-backups-repl
 s3_bucket_immich_main: asxpio-immich-main
 s3_bucket_immich_repl: asxpio-immich-repl
 # S3 Buckets to mount on servers (non-replication only)
 s3_buckets:
  - name: "{{ s3_bucket_terraform_main }}"
    mount_point: /mnt/s3/terraform
    owner: root
    group: root
    mode: "0755"
  - name: "{{ s3_bucket_backups_main }}"
    mount_point: /mnt/s3/backups
    owner: root
    group: root
    mode: "0755"
  - name: "{{ s3_bucket_immich_main }}"
    mount_point: /mnt/s3/immich
    owner: root
    group: docker
    mode: "0775"
--- a/ansible/inventory/group_vars/all/services.yaml
+++ b/ansible/inventory/group_vars/all/services.yaml
@ -0,0 +1,30 @@
 ---
 # Service configuration for all hosts
 # Centralized service definitions to avoid hardcoding paths and ports
 # Base paths for services
 services_base_path: "/opt"
 # Service-specific paths (derived from base path)
 traefik_base_path: "{{ services_base_path }}/traefik"
 forgejo_base_path: "{{ services_base_path }}/forgejo"
 forgejo_runner_base_path: "{{ services_base_path }}/forgejo-runner"
 # Service ports
 traefik_http_port: 80
 traefik_https_port: 443
 traefik_dashboard_port: 8080
 forgejo_http_port: 3000
 forgejo_ssh_port: 2222  # External SSH port for Git operations
 forgejo_internal_ssh_port: 22  # Internal container SSH port
 # WireGuard configuration
 wireguard_port: 60101
 wireguard_interface: "wg0"
 # SSH configuration
 ssh_port: 22
 # Docker configuration
 docker_network_name: "asxpio-net"
--- a/ansible/inventory/group_vars/all/vault.yaml
+++ b/ansible/inventory/group_vars/all/vault.yaml
@ -0,0 +1,87 @@
 $ANSIBLE_VAULT;1.1;AES256
 62313564376664646635373735353930353765633834323532646230633033393738376335313666
 3537393466636230353539313938656537333665636136640a653338306635333331383932663965
 39643137366165623666636135646638623830633563313133326231623030643665633062616336
 3035386162363566660a373666353366393034306135373830356166373861633035346637333339
 32346466363162353663663966323635346338333265303565386166373837626437356632383162
 64666536316438346464666165616437376230343763343738363738373130363833393730303138
 62393736326562313230333331633232363362613839393330353133316536343235383464303634
 34396163626530383564396637336365633033373461623233633630626237336231333466346631
 37323439636239663861653065663532343730313130663663626361353835356662613232663062
 62386130313533613430303035653430313866623335303134383738316430346262633639346334
 63326333653761613239663932306135366236376335643733366535666430633938613966386164
 61633137383866396533323831666462346438633830376561623431333666313531623133656230
 30343862383534623062343537326463623731323435363837383038316539376131363534366164
 30323666663338613466626534376562613539383563633765626238633763396235356639393831
 35636138356435376132336365666363633965623965343165303261323831363366666164373061
 31656239356664623038376231363765386664336135373637383165653966373035353836363264
 39366161646334376430623261353230653930623966616535303565383962633339323563363666
 30343834323430333562666230313334366339313838643662376437323236633761386333373336
 36653132393564653431396662613936636639653231646365353836633830616430323430363561
 33313130313633356539303363613731383930623937643461613665626437623365303766353862
 34386566396530343638313334306530633962653235623561326234363639313233666332616130
 62663235663863666230653363663863663031323563383235356465626463303834623464383835
 61626463343638626530376564343333633738313937326662643133633364353232643135666462
 66363965663436303638363535363332363033313766313335393235363230666234643764653732
 62653261656332326461363961353863346563323638326161656331323538643934383665393536
 37323265396662646163363066643237656235616462373337633338623661363037316131323762
 62653562626237316139363833633039636365643438656230656134363634393139386334376431
 36373534326439323130336431303531366565653263323364323139383632643436613839326434
 35643136363664626630346436656562626363633864356132393035343037633632366436393866
 65616363343062656336303463313431363333336365613632393163356531323838333062613664
 65393430313732346364323736333361363363396433313666636433626139353766346165363637
 34616634313833393162393738376466613338616433393235663930663630356561643237663063
 35363832363365653137313830356666323631653535363063366134303164303364643366613466
 66613030383131633735313065383736316635613437393439393437613363386234356535353534
 32613436346439643835636665656537336534643665323961666632613462633839616637393033
 62383966616666613565653732616164623136323861313365333532393931353338666565336430
 30626638363234623662333638633962313633306536323964343731653362313935363932626331
 38613935643861353830373839393465356334356361656335303732336363306263356463396665
 62633432613838363431653338356639383138343431303632666361313863663938643134626361
 66363666633431343831396637353463366366306135646163633563376437663663353033333865
 38323638333664623037386531323063376533623962366530636137363065616361646464663431
 32326363343163303535643865356539396664393233633639633639346137663635366133636437
 34373839623932626130336265623565313233646562653063343739333539306631353863393337
 61353531303339343963306463613339386530366432646336333039336639326165386562333835
 38373665396138616264646434316333363838333765353436393433656330373738333662373033
 37633234343265373133336331393538303965326366663134353936613439656561356438306231
 62313338383439653835316566663039303330306564626634346165386366393135646137393835
 63316431396439653738656239643565366632616133653039313736333034353032313533363662
 36383661663331633961353839386464656134623337353561613135303865363463346638346263
 32333634343162393266353235326635336332623566353838353566323030663732643531663234
 65616234643931346533333365643263323135366236316363636566613032643636623130306230
 33666266373432303132303438656533653331333636396235623164353730343961376361383838
 35386330616439363662396165643631616638656466313933396339303632323232363638326263
 62326337373534616463633062613261636631656464336663313239636432373230383434363937
 39353764623636343534613639353032643632373334623536373665383538623135316238393937
 33633666636531306666306432666462306137653036643866653365373265646437663362313835
 65613431643030313833333839666663366435613065323033666234653530663837363462383138
 37363739386231646234386265343531303163613563303833623534353832396131336235633362
 32666631653933376130616632333031306331323931336566663764373262376637313465383535
 35356332663931313862376365346361613632306166643433386436666435333233623361323934
 35373031663863343239636435646165323763313137626161623238383365323662623934383162
 34356236386339316533373139663562393531333831666136626162616366633461333863613063
 34376534663135343939373439303966633133353632616530626433303463656634306539653139
 63376561363331306266343033336162663334663638323231323532373239393562383238383831
 32323635356534636637303534393832326566306562633762313363363563643764663063656131
 38323061363438366330613239643937616430313865356366343435343736346633393736333131
 66656635336234663938373433646465383061366139313230663539303365616261336338363565
 36316130303836363366303437373762353061363733343861613530666663363738363032323631
 37613634656532626436343639383431353532316435376232663962636238383862353830313634
 39376262353332663438313062316430383366393265323064613038363763333465373561663430
 62306638623165386636316531396535313134396232343332336663333834636532393530376334
 34333037323830646330303562353439383031356165376564303338323630383337616131636133
 64663665306466633538323330333731386233306130366634646637663839323130633065313462
 66633337346264313561663135323034633636646131303234363132633335313966646534646461
 62353131346666323862323461623437363930333064373439333166656430646237393832373462
 63393430646531323835386462326635313861646335323564643630376131626137616231303335
 39623533383939353966356636656337343336336130643333323236303739393263633562633534
 38666632316432663332626437626337376533393139353731383835376131643035313866386265
 33323333613766333932333637353563366462303763613562333963373537343837653535653631
 33383331396363396439646534323832613165393062623332393537393731373234396634363237
 36343839363163356232333936663230336266376432373433393964313436616637633865636435
 33633835353337613233616438396537393166346464313563643536623835613636373436633562
 33366633326130646138373539656336363034346438326361333539626563613734636138376533
 62313863613237316237346330326330623462346539356330333265383163353733613464326238
 36363131613461643235373937326130383063353534393133636339373439656238333232653263
 6137343166303139343338663639306434656262343139653437
--- a/ansible/inventory/group_vars/all/vault.yaml.template
+++ b/ansible/inventory/group_vars/all/vault.yaml.template
@ -0,0 +1,56 @@
 ---
 # Ansible Vault Template
 # Copy this file to vault.yaml and encrypt it with:
 #   ansible-vault encrypt vault.yaml
 #
 # Or create encrypted vault directly:
 #   ansible-vault create vault.yaml
 #
 # This file contains all sensitive secrets used across the infrastructure
 # ============================================================================
 # S3 Object Storage Credentials
 # ============================================================================
 vault_s3_access_key: "YOUR_S3_ACCESS_KEY_HERE"
 vault_s3_secret_key: "YOUR_S3_SECRET_KEY_HERE"
 # Optional: Alternative S3 endpoint if not using default
 # vault_s3_endpoint: "https://hel1.your-objectstorage.com"
 # ============================================================================
 # Forgejo Secrets
 # ============================================================================
 # Generate with: openssl rand -hex 32
 vault_forgejo_secret_key: "GENERATE_64_CHAR_HEX_STRING_HERE"
 # Generate with Forgejo or use existing JWT token
 vault_forgejo_internal_token: "eyJhbGc..."
 # Generate with: openssl rand -base64 32
 vault_forgejo_jwt_secret: "GENERATE_BASE64_STRING_HERE"
 vault_forgejo_lfs_jwt_secret: "GENERATE_BASE64_STRING_HERE"
 # ============================================================================
 # Forgejo Runner
 # ============================================================================
 # Generate from Forgejo UI: Site Admin → Actions → Runners → Create Runner
 vault_forgejo_runner_token: "GENERATE_FROM_FORGEJO_UI"
 # ============================================================================
 # SSL/TLS Certificates
 # ============================================================================
 # Email for Let's Encrypt certificate notifications
 vault_letsencrypt_email: "admin@example.com"
 # ============================================================================
 # Database Credentials (if needed for future services)
 # ============================================================================
 # vault_postgres_password: "GENERATE_SECURE_PASSWORD"
 # vault_mysql_root_password: "GENERATE_SECURE_PASSWORD"
 # vault_redis_password: "GENERATE_SECURE_PASSWORD"
 # ============================================================================
 # API Keys and Tokens (if needed for future services)
 # ============================================================================
 # vault_grafana_admin_password: "GENERATE_SECURE_PASSWORD"
 # vault_immich_db_password: "GENERATE_SECURE_PASSWORD"
--- a/ansible/inventory/host_vars/asxpio-ops.yaml
+++ b/ansible/inventory/host_vars/asxpio-ops.yaml
@ -1,5 +1,10 @@
 ---
-# WireGuard VPN Configuration for asxpio-ops
+# Host-specific configuration for asxpio-ops
 # Operations server: Forgejo, Traefik, WireGuard VPN hub
 # ============================================================================
 # WireGuard VPN Configuration
 # ============================================================================
 wireguard_port: 60101
 wireguard_interface: wg0
 wireguard_ipv4: 10.20.0.10
@ -22,34 +27,41 @@ wireguard_clients:
    ipv6: fd69:69:69::2
    preshared_key: true
    persistent_keepalive: 25
  - name: phone
    ipv4: 10.20.0.3
    ipv6: fd69:69:69::3
    preshared_key: true
    persistent_keepalive: 25
  - name: pixel9
    ipv4: 10.20.0.4
    ipv6: fd69:69:69::4
    preshared_key: true
    persistent_keepalive: 25
 # Optional: DNS servers for clients
 wireguard_dns_servers:
-   - 1.1.1.1
+  - 1.1.1.1
-   - 1.0.0.1
+  - 1.0.0.1
 # Optional: Additional routes for split tunneling
 # wireguard_additional_routes:
 #   - 192.168.1.0/24
 # ============================================================================
 # SSH Hardening Configuration
 # ============================================================================
 # Allow public SSH access for Forgejo git operations
 ssh_allow_public_git: true
 ssh_git_user: git
-# Forgejo secrets (from old server)
+# ============================================================================
-forgejo_secret_key: "***REDACTED***"
+# Forgejo Configuration
-forgejo_internal_token: "***REDACTED***"
+# ============================================================================
-forgejo_jwt_secret: "***REDACTED***"
+# Secrets are stored in encrypted vault (group_vars/all/vault.yaml)
-forgejo_lfs_jwt_secret: "***REDACTED***"
+forgejo_secret_key: "{{ vault_forgejo_secret_key }}"
-letsencrypt_email: "me@asxp.io"
+forgejo_internal_token: "{{ vault_forgejo_internal_token }}"
 forgejo_jwt_secret: "{{ vault_forgejo_jwt_secret }}"
 forgejo_lfs_jwt_secret: "{{ vault_forgejo_lfs_jwt_secret }}"
 # ============================================================================
 # SSL/TLS Configuration
 # ============================================================================
 letsencrypt_email: "{{ vault_letsencrypt_email }}"
 # ============================================================================
 # S3FS Configuration
 # ============================================================================
 # ops server: CX22 with 40GB disk, limit cache to 10GB
 # Formula: Total_Disk - Desired_Cache_Size = ensure_diskfree
 # 40GB - 10GB = 30GB free
 s3fs_ensure_diskfree: 30720  # 30GB free = max 10GB cache
--- a/ansible/inventory/host_vars/asxpio-services.yaml
+++ b/ansible/inventory/host_vars/asxpio-services.yaml
@ -1,4 +1,17 @@
 ---
 # Host-specific configuration for asxpio-services
 # Services server: Forgejo Runner, Immich, game servers, XMPP
 # ============================================================================
 # SSH Hardening Configuration
 # ============================================================================
 # Restrict SSH to VPC and WireGuard networks only
 ssh_allow_public_git: false
 # ============================================================================
 # S3FS Configuration
 # ============================================================================
 # services server: CX43 with 160GB disk, limit cache to 80GB
 # Formula: Total_Disk - Desired_Cache_Size = ensure_diskfree
 # 160GB - 80GB = 80GB free
 s3fs_ensure_diskfree: 81920  # 80GB free = max 80GB cache
--- a/ansible/inventory/hosts.yaml
+++ b/ansible/inventory/hosts.yaml
@ -18,8 +18,13 @@ all:
        ops_servers:
          hosts:
            asxpio-ops:
-              ansible_host: 10.20.0.10
+              # Stage 1: Use public IP for WireGuard deployment
-              #ansible_host: 65.109.14.216
+              # ansible_host: 135.181.27.77  # Uncomment for Stage 1 if needed
              # Stage 2: Use WireGuard IP (uncomment below after 01-wireguard-setup.yaml)
              ansible_host: "{{ asxpio_ops_wireguard_ip }}"  # 10.20.0.10 via WireGuard
              public_ip: 135.181.27.77
              vpc_ip: "{{ asxpio_ops_vpc_ip }}"
              wireguard_ip: "{{ asxpio_ops_wireguard_ip }}"
              server_role: ops
              server_type: cx22
              services:
@ -33,8 +38,13 @@ all:
        services_servers:
          hosts:
            asxpio-services:
-              ansible_host: 10.20.0.20
+              # Stage 1: Use public IP for WireGuard deployment
-              #ansible_host: 95.217.15.42
+              # ansible_host: 46.62.228.216  # Uncomment for Stage 1 if needed
              # Stage 2: Use WireGuard IP (uncomment below after 01-wireguard-setup.yaml)
              ansible_host: "{{ asxpio_services_wireguard_ip }}"  # 10.20.0.20 via WireGuard
              public_ip: 46.62.228.216
              vpc_ip: "{{ asxpio_services_vpc_ip }}"
              wireguard_ip: "{{ asxpio_services_wireguard_ip }}"
              server_role: services
              server_type: cx43
              services:
--- a/ansible/playbooks/01-wireguard-setup.yaml
+++ b/ansible/playbooks/01-wireguard-setup.yaml
@ -0,0 +1,191 @@
 ---
 # Stage 1: WireGuard VPN Setup
 #
 # This playbook must be run FIRST to establish the WireGuard VPN network.
 # After this playbook completes:
 # 1. WireGuard will be running on all hosts
 # 2. Client configs will be generated in playbooks/wireguard-configs/
 # 3. You MUST manually update inventory/hosts.yaml to use WireGuard IPs (10.20.0.x)
 # 4. Then you can run playbook 02-full-deployment.yaml
 #
 # Usage:
 #   ansible-playbook playbooks/01-wireguard-setup.yaml
 #
 - name: Deploy WireGuard VPN Server
  hosts: ops_servers
  become: true
  gather_facts: true
  pre_tasks:
    - name: Validate required variables
      ansible.builtin.assert:
        that:
          - wireguard_network is defined
          - vpc_network is defined
          - ansible_host is defined
        fail_msg: "Required network variables are missing. Check group_vars/all/"
        success_msg: "Network variables validated successfully"
    - name: Display pre-deployment information
      ansible.builtin.debug:
        msg: |
          ========================================
          WireGuard VPN Server Deployment
          ========================================
          Target: {{ inventory_hostname }} (VPN Hub)
          WireGuard network: {{ wireguard_network }}
          VPC network: {{ vpc_network }}
          This will deploy WireGuard VPN server and generate client configs.
          ========================================
  roles:
    - role: wireguard
 - name: Deploy WireGuard VPN Client
  hosts: services_servers
  become: true
  gather_facts: true
  vars:
    # Map each server to its client config name
    wireguard_client_configs:
      asxpio-services: asxpio-services
  tasks:
    - name: Display client deployment information
      ansible.builtin.debug:
        msg: |
          ========================================
          WireGuard VPN Client Deployment
          ========================================
          Target: {{ inventory_hostname }} (VPN Client)
          Will connect to: {{ asxpio_ops_wireguard_ip }} (asxpio-ops)
          ========================================
    - name: Get client config name for this host
      ansible.builtin.set_fact:
        client_config_name: "{{ wireguard_client_configs[inventory_hostname] }}"
      when: inventory_hostname in wireguard_client_configs
    - name: Install WireGuard
      ansible.builtin.apt:
        name:
          - wireguard
          - wireguard-tools
        state: present
        update_cache: true
    - name: Create WireGuard config directory
      ansible.builtin.file:
        path: /etc/wireguard
        state: directory
        mode: '0700'
        owner: root
        group: root
    - name: Copy WireGuard client configuration
      ansible.builtin.copy:
        src: "{{ playbook_dir }}/wireguard-configs/asxpio-ops/{{ client_config_name }}.conf"
        dest: "/etc/wireguard/{{ client_config_name }}.conf"
        mode: '0600'
        owner: root
        group: root
      when: client_config_name is defined
    - name: Enable and start WireGuard interface
      ansible.builtin.systemd:
        name: "wg-quick@{{ client_config_name }}"
        enabled: true
        state: started
      when: client_config_name is defined
      register: wg_client_start
    - name: Display client status
      ansible.builtin.debug:
        msg: |
          ✅ WireGuard client configured on {{ inventory_hostname }}
          Interface: wg-quick@{{ client_config_name }}
          Status: {{ 'Started' if wg_client_start.changed else 'Already running' }}
      when: client_config_name is defined
 - name: Post-deployment verification
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Display post-deployment instructions
      ansible.builtin.debug:
        msg: |
          ========================================
          WireGuard VPN Deployment Complete!
          ========================================
          Infrastructure deployed:
          ✅ asxpio-ops: WireGuard VPN Server (hub)
          ✅ asxpio-services: WireGuard VPN Client (connected to ops)
          Server-to-server connectivity:
          - asxpio-ops ↔ asxpio-services via WireGuard (10.20.0.0/24)
          IMPORTANT NEXT STEPS FOR YOUR WORKSTATION:
          1. Client configurations for your machine are in:
             playbooks/wireguard-configs/asxpio-ops/
          2. Import a client config to connect your workstation:
             - Linux/Mac: sudo wg-quick up ./playbooks/wireguard-configs/asxpio-ops/<client>.conf
             - Windows: Import the .conf file in WireGuard GUI
          3. Test connectivity from your workstation:
             ping {{ asxpio_ops_wireguard_ip }}
             ping {{ asxpio_services_wireguard_ip }}
          4. After connecting to WireGuard, run Stage 2:
             ansible-playbook playbooks/02-full-deployment.yaml
          NOTE: Stage 2 automatically uses WireGuard IPs.
                No manual configuration changes needed!
          ========================================
    - name: Test WireGuard connectivity from controller
      ansible.builtin.wait_for:
        host: "{{ asxpio_ops_wireguard_ip }}"
        port: 22
        timeout: 5
        state: started
      register: wg_connectivity_test
      failed_when: false
    - name: Display connectivity test results
      ansible.builtin.debug:
        msg: |
          ========================================
          WireGuard Connectivity Test
          ========================================
          {% if wg_connectivity_test.failed %}
          ❌ Cannot reach servers via WireGuard from your workstation yet
          This is NORMAL if you haven't connected your workstation to WireGuard.
          To connect:
          1. Import client config: playbooks/wireguard-configs/asxpio-ops/<client>.conf
          2. Start WireGuard connection on your workstation
          3. Verify with: ping {{ asxpio_ops_wireguard_ip }}
          4. Run Stage 2: ansible-playbook playbooks/02-full-deployment.yaml
          {% else %}
          ✅ WireGuard is reachable from your workstation!
          Server-to-server VPN: Connected
          Workstation-to-server VPN: Connected
          You can now run Stage 2:
            ansible-playbook playbooks/02-full-deployment.yaml
          Stage 2 automatically uses WireGuard IPs - no manual configuration needed!
          {% endif %}
          ========================================
--- a/ansible/playbooks/02-full-deployment.yaml
+++ b/ansible/playbooks/02-full-deployment.yaml
@ -0,0 +1,272 @@
 ---
 # Stage 2: Full Infrastructure Deployment
 #
 # PREREQUISITES:
 # 1. Stage 1 (01-wireguard-setup.yaml) must be completed
 # 2. inventory/hosts.yaml must be updated to use WireGuard IPs (10.20.0.x)
 # 3. You must be connected via WireGuard VPN
 #
 # This playbook deploys the complete infrastructure in one go:
 # - SSH hardening (with connection recovery)
 # - Traefik reverse proxy
 # - Forgejo Git server
 # - Forgejo CI/CD runner
 #
 # Usage:
 #   ansible-playbook playbooks/02-full-deployment.yaml
 #
 - name: Verify prerequisites
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Test WireGuard connectivity
      ansible.builtin.wait_for:
        host: "{{ asxpio_ops_wireguard_ip }}"
        port: 22
        timeout: 10
        state: started
      register: wg_test
      failed_when: false
    - name: Verify WireGuard is reachable
      ansible.builtin.assert:
        that:
          - not wg_test.failed
        fail_msg: |
          ❌ ERROR: Cannot reach servers via WireGuard!
          Tested: {{ asxpio_ops_wireguard_ip }}:22
          Result: Connection failed
          Stage 2 automatically uses WireGuard IPs and REQUIRES an active VPN connection.
          To fix:
          1. Ensure Stage 1 is complete:
             ansible-playbook playbooks/01-wireguard-setup.yaml
          2. Import WireGuard config:
             playbooks/wireguard-configs/<hostname>/<client>.conf
          3. Start WireGuard:
             - Linux/Mac: sudo wg-quick up <config>.conf
             - Windows: Import in WireGuard GUI and activate
          4. Test connectivity:
             ping {{ asxpio_ops_wireguard_ip }}
             ping {{ asxpio_services_wireguard_ip }}
          5. Re-run this playbook
          Aborting deployment for safety...
        success_msg: "✅ WireGuard connectivity verified"
    - name: Display pre-deployment checklist
      ansible.builtin.debug:
        msg: |
          ========================================
          Pre-Deployment Checklist
          ========================================
          ✅ WireGuard connectivity verified
          ✅ Ready for deployment
          This playbook will:
          1. Verify common setup
          2. Harden SSH (will restrict access to WireGuard/VPC only)
          3. Deploy Traefik reverse proxy
          4. Deploy Forgejo Git server
          5. Deploy Forgejo CI/CD runner (requires manual token setup)
          ⚠️  IMPORTANT: This playbook uses WireGuard IPs automatically.
                         After SSH hardening, you will ONLY be able to
                         access servers via WireGuard VPN!
          ========================================
    - name: Pause for confirmation
      ansible.builtin.pause:
        prompt: "Press ENTER to continue or Ctrl+C to abort"
 - name: Common setup verification
  hosts: asxpio:!old_servers
  become: true
  gather_facts: true
  vars:
    # Force WireGuard IPs for Stage 2 (SSH hardening requires it)
    ansible_host: "{{ wireguard_ip }}"
  roles:
    - role: common
  post_tasks:
    - name: Display setup status
      ansible.builtin.debug:
        msg: "Common setup verified on {{ inventory_hostname }}"
 - name: Harden SSH configuration
  hosts: asxpio:!old_servers
  become: true
  gather_facts: true
  vars:
    # Force WireGuard IPs for Stage 2 (SSH hardening requires it)
    ansible_host: "{{ wireguard_ip }}"
  pre_tasks:
    - name: Display SSH hardening notice
      ansible.builtin.debug:
        msg: |
          ========================================
          SSH Hardening - {{ inventory_hostname }}
          ========================================
          SSH will be restarted asynchronously.
          Connection may drop briefly but will automatically recover.
          Trusted networks:
          - WireGuard: {{ wireguard_network }}
          - VPC: {{ vpc_network }}
          ========================================
  roles:
    - role: ssh_hardening
  post_tasks:
    - name: Verify SSH connection after hardening
      ansible.builtin.debug:
        msg: "SSH hardening complete and connection verified on {{ inventory_hostname }}"
 - name: Deploy Traefik reverse proxy
  hosts: asxpio-ops
  become: true
  gather_facts: true
  vars:
    # Force WireGuard IPs for Stage 2
    ansible_host: "{{ wireguard_ip }}"
  roles:
    - role: traefik
  post_tasks:
    - name: Display Traefik status
      ansible.builtin.debug:
        msg: |
          Traefik deployed successfully on {{ inventory_hostname }}
          Dashboard: http://{{ asxpio_ops_wireguard_ip }}:{{ traefik_dashboard_port }}
 - name: Deploy Forgejo Git server
  hosts: asxpio-ops
  become: true
  gather_facts: true
  vars:
    # Force WireGuard IPs for Stage 2
    ansible_host: "{{ wireguard_ip }}"
  roles:
    - role: forgejo
  post_tasks:
    - name: Display Forgejo status
      ansible.builtin.debug:
        msg: |
          Forgejo deployed successfully on {{ inventory_hostname }}
          Web UI: http://{{ asxpio_ops_wireguard_ip }}:{{ forgejo_http_port }}
          SSH: ssh://git@{{ asxpio_ops_wireguard_ip }}:{{ forgejo_ssh_port }}
          ⚠️  IMPORTANT: Before deploying the runner, you must:
          1. Access Forgejo web UI
          2. Complete initial setup (create admin user)
          3. Go to: Site Administration → Actions → Runners
          4. Generate a runner registration token
          5. Update forgejo_runner_token in inventory/group_vars/all.yaml (or vault)
    - name: Pause for Forgejo runner token generation
      ansible.builtin.pause:
        prompt: |
          ========================================
          Forgejo Runner Registration Token Required
          ========================================
          The runner cannot be deployed without a registration token from Forgejo.
          Steps to generate the token:
          1. Open Forgejo: http://{{ asxpio_ops_wireguard_ip }}:{{ forgejo_http_port }}
          2. Complete initial setup if not done (create admin account)
          3. Login as admin
          4. Navigate to: Site Administration → Actions → Runners
          5. Click "Create new Runner" or "Generate Token"
          6. Copy the registration token
          7. Update forgejo_runner_token in inventory/group_vars/all.yaml
             OR better: ansible-vault edit inventory/group_vars/vault.yaml
          Press ENTER when ready to continue with runner deployment, or Ctrl+C to abort
          ========================================
 - name: Deploy Forgejo CI/CD Runner
  hosts: asxpio-services
  become: true
  gather_facts: true
  vars:
    # Force WireGuard IPs for Stage 2
    ansible_host: "{{ wireguard_ip }}"
  pre_tasks:
    - name: Verify runner token is configured
      ansible.builtin.assert:
        that:
          - forgejo_runner_token is defined
          - forgejo_runner_token | length > 0
          - forgejo_runner_token != "REPLACE_WITH_ACTUAL_TOKEN"
        fail_msg: |
          ERROR: forgejo_runner_token is not properly configured!
          You must set a valid runner registration token from Forgejo:
          1. Get token from: Site Administration → Actions → Runners
          2. Set in inventory/group_vars/all.yaml or vault.yaml
          3. Re-run this playbook
        success_msg: "Runner token validated"
  roles:
    - role: forgejo_runner
  post_tasks:
    - name: Display runner status
      ansible.builtin.debug:
        msg: |
          Forgejo Runner deployed successfully on {{ inventory_hostname }}
          Check status: docker logs forgejo-runner
 - name: Display deployment summary
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Show final status
      ansible.builtin.debug:
        msg: |
          ========================================
          Full Infrastructure Deployment Complete!
          ========================================
          Services deployed:
          Traefik (asxpio-services):
            - Dashboard: http://{{ asxpio_services_wireguard_ip }}:8080
          Forgejo (asxpio-ops):
            - Web UI: http://{{ asxpio_ops_wireguard_ip }}:3000
            - Git SSH: ssh://git@{{ asxpio_ops_wireguard_ip }}:2222
          Forgejo Runner (asxpio-services):
            - Status: docker logs forgejo-runner
          SSH Access:
            - Restricted to WireGuard (10.20.0.0/24) and VPC (10.10.0.0/20)
            - Connect via WireGuard VPN to access servers
          Next steps:
            - Configure Forgejo admin account
            - Set up repositories
            - Test CI/CD pipeline
          ========================================
--- a/ansible/playbooks/03-setup-s3fs.yaml
+++ b/ansible/playbooks/03-setup-s3fs.yaml
@ -0,0 +1,134 @@
 ---
 # Stage 3: Setup S3FS on all servers
 #
 # This playbook sets up s3fs to mount S3 buckets as local filesystems.
 # Must be run after Stage 2 (full deployment) to ensure Docker is installed.
 #
 # Mounts the following buckets:
 # - asxpio-terraform-main -> /mnt/s3/terraform
 # - asxpio-backups-main -> /mnt/s3/backups
 # - asxpio-immich-main -> /mnt/s3/immich
 #
 # Cache sizes (per-host):
 # - asxpio-ops (40GB disk): 10GB cache max
 # - asxpio-services (160GB disk): 80GB cache max
 #
 # Usage:
 #   ansible-playbook playbooks/03-setup-s3fs.yaml
 #
 # IMPORTANT: Ensure S3 credentials are set in group_vars/all/vault.yaml:
 #   vault_s3_access_key: "your-access-key"
 #   vault_s3_secret_key: "your-secret-key"
 #
 - name: Setup S3FS on all servers
  hosts: ops_servers,services_servers
  become: true
  gather_facts: true
  pre_tasks:
    - name: Validate S3 credentials are set
      ansible.builtin.assert:
        that:
          - s3_access_key is defined
          - s3_access_key != ""
          - s3_secret_key is defined
          - s3_secret_key != ""
        fail_msg: |
          S3 credentials not set!
          Please set vault_s3_access_key and vault_s3_secret_key in:
          - group_vars/all/vault.yaml (encrypted with ansible-vault)
          OR
          - Pass via command line: -e "s3_access_key=xxx s3_secret_key=yyy"
        success_msg: "S3 credentials validated"
    - name: Display S3 configuration
      ansible.builtin.debug:
        msg: |
          ========================================
          S3FS Setup on {{ inventory_hostname }}
          ========================================
          Target: {{ inventory_hostname }}
          S3 Region: {{ s3_region }}
          S3 Endpoint: {{ s3_endpoint }}
          Buckets to mount:
          {% for bucket in s3_buckets %}
          - {{ bucket.name }} -> {{ bucket.mount_point }}
          {% endfor %}
          ========================================
  roles:
    - role: s3fs
  post_tasks:
    - name: Display mounted filesystems
      ansible.builtin.command:
        cmd: df -h
      register: df_output
      changed_when: false
    - name: Show S3 mounts
      ansible.builtin.debug:
        msg: "{{ df_output.stdout_lines | select('search', 's3fs') | list }}"
    - name: Create immich library directory structure
      ansible.builtin.file:
        path: "{{ item }}"
        state: directory
        mode: '0775'
        owner: root
        group: docker
      loop:
        - /mnt/s3/immich/library
        - /mnt/s3/immich/library/backups
        - /mnt/s3/immich/library/encoded-video
        - /mnt/s3/immich/library/library
        - /mnt/s3/immich/library/profile
        - /mnt/s3/immich/library/thumbs
        - /mnt/s3/immich/library/upload
      when: inventory_hostname in groups['services_servers']
    - name: Display post-setup information
      ansible.builtin.debug:
        msg: |
          ========================================
          S3FS Setup Complete on {{ inventory_hostname }}!
          ========================================
          ✅ All S3 buckets mounted successfully
          Mount points:
          {% for bucket in s3_buckets %}
          - {{ bucket.mount_point }} ({{ bucket.name }})
          {% endfor %}
          Cache configuration:
          - Location: /var/cache/s3fs
          - Max size: ~{{ (40 - (s3fs_ensure_diskfree / 1024)) | int }}GB
          - Free space reserved: {{ (s3fs_ensure_diskfree / 1024) | int }}GB
          {% if inventory_hostname in groups['services_servers'] %}
          Immich library structure created at:
          /mnt/s3/immich/library/
          ├── backups/
          ├── encoded-video/
          ├── library/
          ├── profile/
          ├── thumbs/
          └── upload/
          Next steps for Immich migration:
          1. Manually sync data from legacy server:
             rsync -avz --info=progress2 \
               asxpio-old:/srv/immich/library/ \
               /mnt/s3/immich/library/
          2. Backup Immich PostgreSQL database on legacy server:
             docker exec immich_postgres pg_dumpall -U postgres | gzip > /tmp/immich-db.sql.gz
          3. Deploy Immich service configured to use S3 storage:
             IMMICH_MEDIA_LOCATION=/mnt/s3/immich/library
          {% endif %}
          ========================================
--- a/ansible/playbooks/deploy-forgejo-runner-standalone.yaml
+++ b/ansible/playbooks/deploy-forgejo-runner-standalone.yaml
@ -0,0 +1,226 @@
 ---
 # Standalone Forgejo Runner Deployment
 #
 # PREREQUISITES:
 # 1. Forgejo must be deployed and running
 # 2. You must have created an admin account in Forgejo
 # 3. You must generate a runner registration token from Forgejo UI
 #
 # To get the registration token:
 #   1. Login to Forgejo as admin
 #   2. Go to: Site Administration → Actions → Runners
 #   3. Click "Create new Runner" or "Generate Token"
 #   4. Copy the token and update forgejo_runner_token variable
 #
 # Usage:
 #   ansible-playbook playbooks/deploy-forgejo-runner-standalone.yaml
 #
 - name: Verify Forgejo is accessible
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Check Forgejo is running
      ansible.builtin.uri:
        url: "http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}/"
        status_code: 200
        timeout: 10
      register: forgejo_check
      failed_when: false
      delegate_to: localhost
    - name: Display Forgejo status
      ansible.builtin.debug:
        msg: |
          {% if forgejo_check.status == 200 %}
          ✅ Forgejo is accessible at http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
          {% else %}
          ❌ ERROR: Cannot reach Forgejo!
          Please ensure:
          1. Forgejo is deployed and running
          2. You can access: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
          3. You are connected to the VPC network or WireGuard VPN
          Deploy Forgejo first:
            ansible-playbook playbooks/02-full-deployment.yaml
          Or deploy just Forgejo:
            ansible-playbook playbooks/legacy/deploy-forgejo.yaml
          {% endif %}
    - name: Fail if Forgejo is not accessible
      ansible.builtin.fail:
        msg: "Forgejo is not accessible. Please deploy Forgejo first."
      when: forgejo_check.status != 200
 - name: Get runner registration token
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Prompt for runner token configuration
      ansible.builtin.pause:
        prompt: |
          ========================================
          Forgejo Runner Registration Token Required
          ========================================
          Before deploying the runner, you MUST generate a registration token from Forgejo.
          Steps to generate the token:
          1. Open Forgejo: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
          2. Login as administrator
          3. Navigate to: Site Administration → Actions → Runners
             (Click your profile icon → Site Administration → Actions → Runners)
          4. Click "Create new Runner" or "Generate registration token"
          5. Copy the generated token (it looks like: "eyJhbGc...")
          6. Update the token in one of these locations:
             Option A - Plaintext (NOT RECOMMENDED):
               Edit: inventory/group_vars/all.yaml
               Set: forgejo_runner_token: "YOUR-TOKEN-HERE"
             Option B - Ansible Vault (RECOMMENDED):
               ansible-vault edit inventory/group_vars/vault.yaml
               Set: vault_forgejo_runner_token: "YOUR-TOKEN-HERE"
               Then in all.yaml: forgejo_runner_token: "{{ vault_forgejo_runner_token }}"
          7. Save the file
          8. Press ENTER to continue with runner deployment
          Or press Ctrl+C to abort and configure the token first
          ========================================
 - name: Deploy Forgejo CI/CD Runner
  hosts: asxpio-services
  become: true
  gather_facts: true
  pre_tasks:
    - name: Verify runner token is configured
      ansible.builtin.assert:
        that:
          - forgejo_runner_token is defined
          - forgejo_runner_token | length > 0
          - forgejo_runner_token != "REPLACE_WITH_ACTUAL_TOKEN"
          - forgejo_runner_token != "***REDACTED***"  # Old token from git
        fail_msg: |
          ❌ ERROR: forgejo_runner_token is not properly configured!
          Current value: {{ forgejo_runner_token | default('NOT SET') }}
          You must set a valid runner registration token from Forgejo:
          1. Get token from Forgejo UI:
             Site Administration → Actions → Runners → Generate Token
          2. Update in inventory/group_vars/all.yaml:
             forgejo_runner_token: "YOUR-TOKEN-HERE"
             OR better, use Ansible Vault (recommended):
             ansible-vault edit inventory/group_vars/vault.yaml
             vault_forgejo_runner_token: "YOUR-TOKEN-HERE"
          3. Re-run this playbook
          Aborting deployment...
        success_msg: "✅ Runner token validated (length: {{ forgejo_runner_token | length }} chars)"
    - name: Display deployment information
      ansible.builtin.debug:
        msg: |
          ========================================
          Deploying Forgejo Runner
          ========================================
          Target: {{ inventory_hostname }}
          Forgejo instance: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
          Runner name: runner-{{ inventory_hostname }}
          This will:
          1. Deploy runner container
          2. Register with Forgejo instance
          3. Start accepting CI/CD jobs
          ========================================
  roles:
    - role: forgejo_runner
  post_tasks:
    - name: Wait for runner registration
      ansible.builtin.pause:
        seconds: 5
        prompt: "Waiting for runner to register with Forgejo..."
    - name: Display runner status
      ansible.builtin.debug:
        msg: |
          ========================================
          Forgejo Runner Deployment Complete!
          ========================================
          Runner deployed successfully on {{ inventory_hostname }}
          Check status:
            docker logs forgejo-runner
          Verify in Forgejo UI:
            1. Login to Forgejo
            2. Go to: Site Administration → Actions → Runners
            3. You should see: runner-{{ inventory_hostname }} (idle/active)
          Test the runner:
            1. Create a repository with .forgejo/workflows/test.yml
            2. Push a commit
            3. Check Actions tab for workflow run
          Runner supports these labels:
            - ubuntu-latest, ubuntu-22.04, ubuntu-24.04
            - debian-latest, debian-12, debian-11
            - node-20, node-22
            - python-3.11, python-3.12
            - alpine-latest, arch-latest
            - nginx-latest, nginx-alpine
          Example workflow:
            .forgejo/workflows/test.yml:
            ---
            name: Test
            on: [push]
            jobs:
              test:
                runs-on: ubuntu-latest
                steps:
                  - uses: actions/checkout@v4
                  - run: echo "Hello from Forgejo Actions!"
          ========================================
 - name: Verify runner in Forgejo
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Reminder to check Forgejo UI
      ansible.builtin.debug:
        msg: |
          ⚠️  FINAL STEP: Verify runner in Forgejo UI
          1. Open: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
          2. Login as admin
          3. Go to: Site Administration → Actions → Runners
          4. Verify runner-asxpio-services appears in the list
          5. Status should be "idle" (green) or "active"
          If the runner doesn't appear:
            - Check logs: docker logs forgejo-runner
            - Verify token is correct
            - Ensure runner can reach Forgejo at {{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
--- a/ansible/playbooks/deploy-forgejo-runner.yaml
+++ b/ansible/playbooks/deploy-forgejo-runner.yaml
@ -1,6 +0,0 @@
 ---
 - name: Deploy Forgejo Runner
  hosts: asxpio-services
  become: true
  roles:
    - forgejo_runner
--- a/ansible/playbooks/deploy-forgejo.yaml
+++ b/ansible/playbooks/deploy-forgejo.yaml
@ -1,6 +0,0 @@
 ---
 - name: Deploy Forgejo Git server
  hosts: asxpio-ops
  become: true
  roles:
    - forgejo
--- a/ansible/playbooks/deploy-traefik.yaml
+++ b/ansible/playbooks/deploy-traefik.yaml
@ -1,6 +0,0 @@
 ---
 - name: Deploy Traefik reverse proxy
  hosts: asxpio-ops
  become: true
  roles:
    - traefik
--- a/ansible/playbooks/legacy/backup-forgejo.yaml
+++ b/ansible/playbooks/legacy/backup-forgejo.yaml
--- a/ansible/playbooks/legacy/restore-forgejo.yaml
+++ b/ansible/playbooks/legacy/restore-forgejo.yaml
--- a/ansible/playbooks/setup-servers.yaml
+++ b/ansible/playbooks/setup-servers.yaml
@ -1,32 +0,0 @@
 ---
 # Main playbook for setting up asxpio servers
 # Usage: ansible-playbook playbooks/setup-servers.yml
 #
 # NOTE: Cloud-init handles base system setup:
 # - System packages (fail2ban, curl, htop, ncdu, tree, etc.)
 # - User management (asxpi + deploy users with sudo and docker access)
 # - SSH hardening (key-only auth, prohibit root password)
 # - Docker installation (docker-ce, docker-ce-cli, containerd.io, docker-compose-plugin)
 # - Bash prompt styling for asxpi user
 # - Docker image pruning cron job
 # - Directory structure creation
 #
 # This playbook handles post-cloud-init verification and additional configuration.
 # Application deployment (Traefik, Forgejo, monitoring, etc.)
 # will be handled via Forgejo Actions pipelines
 - name: Setup asxpio infrastructure servers
  hosts: asxpio
  become: yes
  gather_facts: yes
  roles:
    - common
  post_tasks:
    - name: Display server info
      debug:
        msg: |
          Server {{ inventory_hostname }} post-cloud-init setup complete!
          Role: {{ server_role }}
          Next: Deploy services via Forgejo Actions pipelines
--- a/ansible/playbooks/ssh-harden.yaml
+++ b/ansible/playbooks/ssh-harden.yaml
@ -1,6 +0,0 @@
 ---
 - name: Harden SSH configuration
  hosts: asxpio:!old_servers
  become: true
  roles:
    - ssh_hardening
--- a/ansible/playbooks/wireguard-client.yaml
+++ b/ansible/playbooks/wireguard-client.yaml
@ -1,59 +0,0 @@
 ---
 - name: Deploy WireGuard client configuration
  hosts: services_servers
  become: true
  vars:
    # Map each server to its client config name
    wireguard_client_configs:
      asxpio-services: asxpio-services
  tasks:
    - name: Get client config name for this host
      ansible.builtin.set_fact:
        client_config_name: "{{ wireguard_client_configs[inventory_hostname] }}"
      when: inventory_hostname in wireguard_client_configs
    - name: Install WireGuard
      ansible.builtin.apt:
        name:
          - wireguard
          - wireguard-tools
        state: present
        update_cache: true
    - name: Create WireGuard config directory
      ansible.builtin.file:
        path: /etc/wireguard
        state: directory
        mode: '0700'
        owner: root
        group: root
    - name: Copy WireGuard client configuration
      ansible.builtin.copy:
        src: "../playbooks/wireguard-configs/asxpio-ops/{{ client_config_name }}.conf"
        dest: "/etc/wireguard/{{ client_config_name }}.conf"
        mode: '0600'
        owner: root
        group: root
      when: client_config_name is defined
      notify: restart wireguard client
    - name: Enable WireGuard interface
      ansible.builtin.systemd:
        name: "wg-quick@{{ client_config_name }}"
        enabled: true
      when: client_config_name is defined
    - name: Start WireGuard interface
      ansible.builtin.systemd:
        name: "wg-quick@{{ client_config_name }}"
        state: started
      when: client_config_name is defined
  handlers:
    - name: restart wireguard client
      ansible.builtin.shell: |
        wg-quick down {{ client_config_name }} || true
        wg-quick up {{ client_config_name }}
      args:
        executable: /bin/bash
--- a/ansible/playbooks/wireguard.yaml
+++ b/ansible/playbooks/wireguard.yaml
@ -1,6 +0,0 @@
 ---
 - name: Setup WireGuard VPN
  hosts: ops_servers
  become: true
  roles:
    - wireguard
--- a/ansible/roles/common/tasks/main.yaml
+++ b/ansible/roles/common/tasks/main.yaml
@ -1,16 +0,0 @@
 # Cloud-init already handles:
 # - Package installation (fail2ban, curl, git, htop, ncdu, tree, tmux, etc.)
 # - fail2ban service enabled and started
 # - User creation (asxpi with sudo + docker, deploy with sudo + docker)
 # - SSH hardening (key-only auth, prohibit root password)
 # - Docker installation and configuration
 # - Docker image pruning cron job (weekly at 3am Monday)
 # - Directory structure creation
 #
 # This role is now for post-cloud-init configuration only
 - name: Verify cloud-init completed successfully
  command: cloud-init status --wait
  register: cloud_init_status
  changed_when: false
  failed_when: "'done' not in cloud_init_status.stdout"
--- a/ansible/roles/docker_service/defaults/main.yaml
+++ b/ansible/roles/docker_service/defaults/main.yaml
@ -0,0 +1,34 @@
 ---
 # Default variables for docker_service role
 # These can be overridden when including this role
 # Service configuration
 service_name: ""  # REQUIRED: Name of the service (e.g., "traefik", "forgejo")
 service_base_path: ""  # REQUIRED: Base installation path (e.g., "/opt/traefik")
 # Directories to create under service_base_path
 # Format: list of dicts with 'path' and 'mode' keys
 service_directories: []
 # Example:
 #   - { path: "data", mode: "0755" }
 #   - { path: "config", mode: "0644" }
 # Template files to deploy
 # Format: list of dicts with 'src', 'dest', and optional 'mode' keys
 service_templates: []
 # Example:
 #   - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
 #   - { src: "config.yaml.j2", dest: "config/config.yaml", mode: "0600" }
 # Docker compose configuration
 docker_compose_file: "docker-compose.yml"  # Name of the compose file
 docker_compose_project_name: "{{ service_name }}"  # Docker compose project name
 # Health check configuration
 health_check_enabled: true  # Whether to perform health check after deployment
 health_check_url: ""  # URL to check (empty = skip HTTP check)
 health_check_host: "localhost"  # Host to check port on (default: localhost)
 health_check_port: ""  # Port to check (empty = skip port check)
 health_check_delay: 10  # Seconds to wait before health check
 health_check_retries: 30  # Number of retries for health check
 health_check_retry_delay: 2  # Seconds between retries
--- a/ansible/roles/docker_service/tasks/main.yaml
+++ b/ansible/roles/docker_service/tasks/main.yaml
@ -0,0 +1,81 @@
 ---
 # Shared tasks for deploying Docker-based services
 # This role provides a standardized pattern for service deployment
 - name: Validate required variables
  ansible.builtin.assert:
    that:
      - service_name is defined
      - service_name | length > 0
      - service_base_path is defined
      - service_base_path | length > 0
    fail_msg: "service_name and service_base_path are required variables"
    success_msg: "Required variables validated for {{ service_name }}"
 - name: Create base service directory
  ansible.builtin.file:
    path: "{{ service_base_path }}"
    state: directory
    mode: '0755'
    owner: root
    group: root
 - name: Create service subdirectories
  ansible.builtin.file:
    path: "{{ service_base_path }}/{{ item.path }}"
    state: directory
    mode: "{{ item.mode | default('0755') }}"
    owner: root
    group: root
  loop: "{{ service_directories }}"
  when: service_directories | length > 0
 - name: Deploy service template files
  ansible.builtin.template:
    src: "{{ item.src }}"
    dest: "{{ service_base_path }}/{{ item.dest }}"
    mode: "{{ item.mode | default('0644') }}"
    owner: root
    group: root
  loop: "{{ service_templates }}"
  when: service_templates | length > 0
  register: template_result
 - name: Start service with Docker Compose
  community.docker.docker_compose_v2:
    project_src: "{{ service_base_path }}"
    project_name: "{{ docker_compose_project_name }}"
    files:
      - "{{ docker_compose_file }}"
    state: present
  register: compose_result
 - name: Wait before health check
  ansible.builtin.pause:
    seconds: "{{ health_check_delay }}"
  when: health_check_enabled and (health_check_url != "" or health_check_port != "")
 - name: Wait for service HTTP endpoint
  ansible.builtin.uri:
    url: "{{ health_check_url }}"
    status_code: [200, 301, 302, 401, 404]  # Accept various codes (service-dependent)
    timeout: 5
  register: health_result
  until: health_result is not failed
  retries: "{{ health_check_retries }}"
  delay: "{{ health_check_retry_delay }}"
  when: health_check_enabled and health_check_url != ""
  failed_when: false
 - name: Wait for service port
  ansible.builtin.wait_for:
    host: "{{ health_check_host }}"
    port: "{{ health_check_port }}"
    delay: 0
    timeout: "{{ health_check_retries * health_check_retry_delay }}"
    state: started
  when: health_check_enabled and health_check_port != ""
 - name: Display deployment status
  ansible.builtin.debug:
    msg: "{{ service_name }} deployment completed successfully"
--- a/ansible/roles/forgejo/tasks/main.yaml
+++ b/ansible/roles/forgejo/tasks/main.yaml
@ -1,42 +1,23 @@
 ---
 - name: Validate Forgejo variables
  ansible.builtin.import_tasks: validate.yaml
 - name: Configure SSH passthrough
-  import_tasks: ssh-passthrough.yaml
+  ansible.builtin.import_tasks: ssh-passthrough.yaml
- name: Create Forgejo directories
+- name: Deploy Forgejo using shared docker_service role
-  file:
+  ansible.builtin.include_role:
-    path: "{{ item }}"
+    name: docker_service
-    state: directory
+  vars:
-    mode: '0755'
+    service_name: "forgejo"
-  loop:
+    service_base_path: "{{ forgejo_base_path }}"
-    - /opt/forgejo
+    service_directories:
-    - /opt/forgejo/data
+      - { path: "data", mode: "0755" }
-    - /opt/forgejo/config
+      - { path: "config", mode: "0755" }
-
+    service_templates:
- name: Deploy Forgejo app.ini configuration
+      - { src: "app.ini.j2", dest: "config/app.ini", mode: "0644" }
-  template:
+      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
-    src: app.ini.j2
+    health_check_enabled: true
-    dest: /opt/forgejo/config/app.ini
+    health_check_url: "http://localhost:{{ forgejo_http_port }}/"
-    mode: '0644'
+    health_check_delay: 5
-  notify: Restart Forgejo
+    health_check_retries: 15
 - name: Deploy Forgejo docker-compose
  template:
    src: docker-compose.yml.j2
    dest: /opt/forgejo/docker-compose.yml
    mode: '0644'
  notify: Restart Forgejo
 - name: Start Forgejo
  community.docker.docker_compose_v2:
    project_src: /opt/forgejo
    state: present
  register: forgejo_output
 - name: Wait for Forgejo to be healthy
  uri:
    url: http://localhost:{{ forgejo_http_port }}/
    status_code: 200
  retries: 15
  delay: 5
  register: result
  until: result.status == 200
--- a/ansible/roles/forgejo/tasks/validate.yaml
+++ b/ansible/roles/forgejo/tasks/validate.yaml
@ -0,0 +1,45 @@
 ---
 # Variable validation for forgejo role
 - name: Validate Forgejo required variables
  ansible.builtin.assert:
    that:
      - forgejo_base_path is defined
      - forgejo_base_path | length > 0
      - forgejo_http_port is defined
      - forgejo_http_port | int > 0
      - forgejo_http_port | int < 65536
      - forgejo_ssh_port is defined
      - forgejo_ssh_port | int > 0
      - forgejo_ssh_port | int < 65536
      - forgejo_domain is defined
      - forgejo_domain | length > 0
    fail_msg: |
      Forgejo role requires:
      - forgejo_base_path (installation directory)
      - forgejo_http_port (1-65535)
      - forgejo_ssh_port (1-65535)
      - forgejo_domain (domain name)
    success_msg: "Forgejo basic variables validated successfully"
 - name: Validate Forgejo secrets (if not using vault)
  ansible.builtin.assert:
    that:
      - forgejo_secret_key is defined
      - forgejo_secret_key | length >= 32
      - forgejo_internal_token is defined
      - forgejo_internal_token | length > 0
      - forgejo_jwt_secret is defined
      - forgejo_jwt_secret | length >= 32
      - forgejo_lfs_jwt_secret is defined
      - forgejo_lfs_jwt_secret | length >= 32
    fail_msg: |
      Forgejo role requires secrets:
      - forgejo_secret_key (min 32 chars)
      - forgejo_internal_token
      - forgejo_jwt_secret (min 32 chars)
      - forgejo_lfs_jwt_secret (min 32 chars)
      WARNING: These should be stored in Ansible Vault!
    success_msg: "Forgejo secrets validated (WARNING: should use Ansible Vault)"
  when: not ansible_check_mode
--- a/ansible/roles/forgejo/templates/app.ini.j2
+++ b/ansible/roles/forgejo/templates/app.ini.j2
@ -40,7 +40,7 @@ INTERNAL_TOKEN = {{ forgejo_internal_token | default('CHANGE-ME-PLEASE-GENERATE-
 PASSWORD_HASH_ALGO = pbkdf2
 [service]
-DISABLE_REGISTRATION = false
+DISABLE_REGISTRATION = true
 REQUIRE_SIGNIN_VIEW = false
 REGISTER_EMAIL_CONFIRM = false
 ENABLE_NOTIFY_MAIL = false
--- a/ansible/roles/forgejo_backup/tasks/main.yaml
+++ b/ansible/roles/forgejo_backup/tasks/main.yaml
@ -16,19 +16,54 @@
    file_type: file
  register: existing_dumps
- name: Create backup archive from old server
+- name: Create temporary staging directory for backup
  file:
    path: /tmp/forgejo-backup-staging
    state: directory
    mode: '0755'
  become: true
 - name: Copy database to staging
  copy:
    src: /srv/gitea/gitea.db
    dest: /tmp/forgejo-backup-staging/gitea.db
    remote_src: true
  become: true
 - name: Create data directory in staging
  file:
    path: /tmp/forgejo-backup-staging/data
    state: directory
    mode: '0755'
  become: true
 - name: Copy data directories to staging
  copy:
    src: "/srv/gitea/data/{{ item }}/"
    dest: "/tmp/forgejo-backup-staging/data/{{ item }}/"
    remote_src: true
  loop:
    - avatars
    - repo-avatars
    - repo-archive
    - attachments
    - gitea-repositories
  become: true
  ignore_errors: true
 - name: Create backup archive from staging
  archive:
-    path:
+    path: /tmp/forgejo-backup-staging
      - /srv/gitea/gitea.db
      - /srv/gitea/data/avatars
      - /srv/gitea/data/repo-avatars
      - /srv/gitea/data/repo-archive
      - /srv/gitea/data/attachments
      - /srv/gitea/data/gitea-repositories
    dest: /tmp/forgejo-full-backup-{{ ansible_date_time.date }}.tar.gz
    format: gz
  become: true
 - name: Clean up staging directory
  file:
    path: /tmp/forgejo-backup-staging
    state: absent
  become: true
 - name: Display found backup files
  debug:
    msg: |
--- a/ansible/roles/forgejo_restore/tasks/main.yaml
+++ b/ansible/roles/forgejo_restore/tasks/main.yaml
@ -63,59 +63,45 @@
    dest: /tmp/forgejo-restore/
    remote_src: true
 - name: Set backup source path
  set_fact:
    backup_source: "/tmp/forgejo-restore/forgejo-backup-staging"
 - name: Copy database from backup
  copy:
-    src: /tmp/forgejo-restore/gitea.db
+    src: "{{ backup_source }}/gitea.db"
    dest: /opt/forgejo/data/gitea/gitea.db
    remote_src: true
    owner: 1000
    group: 1000
    mode: '0644'
- name: Copy avatars from backup
+- name: Copy data directories from backup
-  synchronize:
+  ansible.builtin.copy:
-    src: /tmp/forgejo-restore/data/avatars/
+    src: "{{ backup_source }}/data/{{ item.src }}/"
-    dest: /opt/forgejo/data/gitea/data/avatars/
+    dest: "{{ forgejo_base_path | default('/opt/forgejo') }}/data/{{ item.dest }}/"
-  delegate_to: "{{ inventory_hostname }}"
+    remote_src: true
-  ignore_errors: true
+    owner: 1000
-
+    group: 1000
- name: Copy repo-avatars from backup
+    mode: preserve
-  synchronize:
+  loop:
-    src: /tmp/forgejo-restore/data/repo-avatars/
+    - { src: "avatars", dest: "gitea/data/avatars" }
-    dest: /opt/forgejo/data/gitea/data/repo-avatars/
+    - { src: "repo-avatars", dest: "gitea/data/repo-avatars" }
-  delegate_to: "{{ inventory_hostname }}"
+    - { src: "repo-archive", dest: "gitea/data/repo-archive" }
-  ignore_errors: true
+    - { src: "attachments", dest: "gitea/data/attachments" }
-
+    - { src: "gitea-repositories", dest: "git/repositories" }
- name: Copy repo-archive from backup
+  ignore_errors: "{{ item.src != 'gitea-repositories' }}"  # Only fail on repositories copy failure
  synchronize:
    src: /tmp/forgejo-restore/data/repo-archive/
    dest: /opt/forgejo/data/gitea/data/repo-archive/
  delegate_to: "{{ inventory_hostname }}"
  ignore_errors: true
 - name: Copy attachments from backup
  synchronize:
    src: /tmp/forgejo-restore/data/attachments/
    dest: /opt/forgejo/data/gitea/data/attachments/
  delegate_to: "{{ inventory_hostname }}"
  ignore_errors: true
 - name: Copy repositories from backup
  synchronize:
    src: /tmp/forgejo-restore/data/gitea-repositories/
    dest: /opt/forgejo/data/git/repositories/
  delegate_to: "{{ inventory_hostname }}"
 - name: Set correct ownership for all Forgejo data
-  file:
+  ansible.builtin.file:
-    path: /opt/forgejo/data
+    path: "{{ forgejo_base_path | default('/opt/forgejo') }}/data"
    owner: 1000
    group: 1000
    recurse: true
 - name: Start Forgejo container
  community.docker.docker_compose_v2:
-    project_src: /opt/forgejo
+    project_src: "{{ forgejo_base_path | default('/opt/forgejo') }}"
    state: present
 - name: Wait for Forgejo to be healthy
@ -131,8 +117,9 @@
    state: absent
 - name: Verify users in database
-  shell: docker exec -u git forgejo forgejo admin user list
+  ansible.builtin.command: docker exec -u git forgejo forgejo admin user list
  register: user_list
  changed_when: false
  ignore_errors: true
 - name: Display restore information
@ -141,10 +128,10 @@
      Forgejo restore completed successfully!
      Users:
-      {{ user_list.stdout }}
+      {{ user_list.stdout_lines | join('\n      ') }}
      Forgejo is now running at:
-      - Internal: http://localhost:{{ forgejo_http_port }}
+      - Internal: http://{{ asxpio_ops_wireguard_ip }}:{{ forgejo_http_port }}
      - External: https://{{ forgejo_domain }}
      Next steps:
--- a/ansible/roles/forgejo_runner/tasks/main.yaml
+++ b/ansible/roles/forgejo_runner/tasks/main.yaml
@ -1,42 +1,29 @@
 ---
- name: Create Forgejo Runner directories
+- name: Validate Forgejo Runner variables
-  file:
+  ansible.builtin.import_tasks: validate.yaml
    path: "{{ item }}"
    state: directory
    mode: '0755'
  loop:
    - /opt/forgejo-runner
    - /opt/forgejo-runner/data
    - /opt/forgejo-runner/config
- name: Deploy Forgejo Runner configuration
+- name: Deploy Forgejo Runner using shared docker_service role
-  template:
+  ansible.builtin.include_role:
-    src: config.yaml.j2
+    name: docker_service
-    dest: /opt/forgejo-runner/config/config.yaml
+  vars:
-    mode: '0644'
+    service_name: "forgejo-runner"
-  notify: Restart Forgejo Runner
+    service_base_path: "{{ forgejo_runner_base_path }}"
-
+    service_directories:
- name: Deploy Forgejo Runner docker-compose
+      - { path: "data", mode: "0755" }
-  template:
+      - { path: "config", mode: "0755" }
-    src: docker-compose.yml.j2
+    service_templates:
-    dest: /opt/forgejo-runner/docker-compose.yml
+      - { src: "config.yaml.j2", dest: "config/config.yaml", mode: "0644" }
-    mode: '0644'
+      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
-  notify: Restart Forgejo Runner
+    health_check_enabled: false  # Runner doesn't expose HTTP port
 - name: Start Forgejo Runner
  community.docker.docker_compose_v2:
    project_src: /opt/forgejo-runner
    state: present
  register: runner_output
 - name: Check if runner is already registered
  ansible.builtin.stat:
-    path: /opt/forgejo-runner/data/.runner
+    path: "{{ forgejo_runner_base_path }}/data/.runner"
  register: runner_registered
 - name: Set permissions on data directory
  ansible.builtin.file:
-    path: /opt/forgejo-runner/data
+    path: "{{ forgejo_runner_base_path }}/data"
    state: directory
    mode: '0777'
    recurse: true
@ -44,7 +31,7 @@
 - name: Stop runner for registration
  community.docker.docker_compose_v2:
-    project_src: /opt/forgejo-runner
+    project_src: "{{ forgejo_runner_base_path }}"
    state: stopped
  when: not runner_registered.stat.exists
@ -52,31 +39,32 @@
  ansible.builtin.command:
    cmd: >
      docker run --rm
-      -v /opt/forgejo-runner/data:/data
+      -v {{ forgejo_runner_base_path }}/data:/data
-      -v /opt/forgejo-runner/config/config.yaml:/config.yaml
+      -v {{ forgejo_runner_base_path }}/config/config.yaml:/config.yaml
      code.forgejo.org/forgejo/runner:{{ forgejo_runner_version }}
      forgejo-runner register
      --no-interactive
      --token {{ forgejo_runner_token }}
      --name runner-{{ inventory_hostname }}
-      --instance http://10.10.0.10:{{ forgejo_http_port }}
+      --instance http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
      --labels ubuntu-latest:docker://node:20-bookworm,ubuntu-22.04:docker://node:20-bookworm,ubuntu-24.04:docker://node:22-bookworm,debian-latest:docker://debian:bookworm,debian-12:docker://debian:bookworm,debian-11:docker://debian:bullseye,arch-latest:docker://archlinux:latest,nginx-latest:docker://nginx:latest,nginx-alpine:docker://nginx:alpine,node-20:docker://node:20,node-22:docker://node:22,python-3.11:docker://python:3.11,python-3.12:docker://python:3.12,alpine-latest:docker://alpine:latest
  when: not runner_registered.stat.exists
  changed_when: true
  register: registration_result
 - name: Start runner after registration
  community.docker.docker_compose_v2:
-    project_src: /opt/forgejo-runner
+    project_src: "{{ forgejo_runner_base_path }}"
    state: present
  when: not runner_registered.stat.exists
 - name: Display runner status
-  debug:
+  ansible.builtin.debug:
    msg: |
      Forgejo Runner deployed successfully!
      Runner name: runner-{{ inventory_hostname }}
-      Forgejo instance: http://10.10.0.10:{{ forgejo_http_port }}
+      Forgejo instance: http://{{ asxpio_ops_vpc_ip }}:{{ forgejo_http_port }}
      {% if runner_registered.stat.exists %}
      Status: Already registered (using existing registration)
      {% else %}
--- a/ansible/roles/forgejo_runner/tasks/validate.yaml
+++ b/ansible/roles/forgejo_runner/tasks/validate.yaml
@ -0,0 +1,26 @@
 ---
 # Variable validation for forgejo_runner role
 - name: Validate Forgejo Runner required variables
  ansible.builtin.assert:
    that:
      - forgejo_runner_base_path is defined
      - forgejo_runner_base_path | length > 0
      - forgejo_runner_token is defined
      - forgejo_runner_token | length > 0
      - forgejo_runner_version is defined
      - forgejo_runner_version | length > 0
      - forgejo_http_port is defined
      - forgejo_http_port | int > 0
      - asxpio_ops_vpc_ip is defined
      - asxpio_ops_vpc_ip | length > 0
    fail_msg: |
      Forgejo Runner role requires:
      - forgejo_runner_base_path (installation directory)
      - forgejo_runner_token (runner registration token)
      - forgejo_runner_version (e.g., '3.5.1')
      - forgejo_http_port (Forgejo HTTP port)
      - asxpio_ops_vpc_ip (Forgejo server IP)
      WARNING: forgejo_runner_token should be stored in Ansible Vault!
    success_msg: "Forgejo Runner variables validated (WARNING: token should use Ansible Vault)"
--- a/ansible/roles/s3fs/defaults/main.yaml
+++ b/ansible/roles/s3fs/defaults/main.yaml
@ -0,0 +1,52 @@
 ---
 # S3FS role defaults
 # S3 Configuration
 s3_endpoint: "https://{{ s3_region }}.your-objectstorage.com"
 s3_region: "hel1"
 # S3 Credentials (should be overridden in group_vars or vault)
 s3_access_key: ""
 s3_secret_key: ""
 # S3 Buckets to mount (non-replication buckets only)
 s3_buckets:
  - name: asxpio-terraform-main
    mount_point: /mnt/s3/terraform
    owner: root
    group: root
    mode: "0755"
  - name: asxpio-backups-main
    mount_point: /mnt/s3/backups
    owner: root
    group: root
    mode: "0755"
  - name: asxpio-immich-main
    mount_point: /mnt/s3/immich
    owner: root
    group: docker
    mode: "0775"
 # S3FS mount options
 # Note: s3fs_ensure_diskfree is defined per-host in host_vars/
 s3fs_options:
  - "passwd_file=/etc/s3fs/.passwd-s3fs"
  - "use_path_request_style"
  - "url={{ s3_endpoint }}"
  - "allow_other"
  - "umask=0022"
  - "uid={{ s3fs_uid | default('0') }}"
  - "gid={{ s3fs_gid | default('0') }}"
  - "use_cache=/var/cache/s3fs"
  - "ensure_diskfree={{ s3fs_ensure_diskfree | default(81920) }}"
  - "max_stat_cache_size=10000"
  - "stat_cache_expire=900"
  - "parallel_count=15"
  - "multipart_size=52"
  - "nocopyapi"
  - "enable_noobj_cache"
 # Package dependencies
 s3fs_packages:
  - s3fs
  - fuse3
--- a/ansible/roles/s3fs/tasks/main.yaml
+++ b/ansible/roles/s3fs/tasks/main.yaml
@ -0,0 +1,96 @@
 ---
 # S3FS role tasks - Mount S3 buckets using s3fs
 - name: Install s3fs and dependencies
  ansible.builtin.apt:
    name: "{{ s3fs_packages }}"
    state: present
    update_cache: true
 - name: Create s3fs credentials directory
  ansible.builtin.file:
    path: /etc/s3fs
    state: directory
    mode: '0700'
    owner: root
    group: root
 - name: Create s3fs password file
  ansible.builtin.template:
    src: passwd-s3fs.j2
    dest: /etc/s3fs/.passwd-s3fs
    mode: '0600'
    owner: root
    group: root
  no_log: true
 - name: Create S3 cache directory
  ansible.builtin.file:
    path: /var/cache/s3fs
    state: directory
    mode: '0755'
    owner: root
    group: root
 - name: Create mount points for S3 buckets
  ansible.builtin.file:
    path: "{{ item.mount_point }}"
    state: directory
    mode: "{{ item.mode }}"
    owner: "{{ item.owner }}"
    group: "{{ item.group }}"
  loop: "{{ s3_buckets }}"
  loop_control:
    label: "{{ item.name }} -> {{ item.mount_point }}"
 - name: Add S3 buckets to /etc/fstab
  ansible.builtin.lineinfile:
    path: /etc/fstab
    line: "{{ item.name }} {{ item.mount_point }} fuse.s3fs _netdev,{{ s3fs_options | join(',') }} 0 0"
    regexp: "^{{ item.name }}\\s+{{ item.mount_point }}"
    state: present
  loop: "{{ s3_buckets }}"
  loop_control:
    label: "{{ item.name }} -> {{ item.mount_point }}"
 - name: Mount all S3 buckets
  ansible.builtin.command:
    cmd: "mount {{ item.mount_point }}"
  loop: "{{ s3_buckets }}"
  loop_control:
    label: "{{ item.name }} -> {{ item.mount_point }}"
  register: mount_result
  changed_when: mount_result.rc == 0
  failed_when: false
 - name: Verify S3 buckets are mounted
  ansible.builtin.command:
    cmd: "mountpoint -q {{ item.mount_point }}"
  loop: "{{ s3_buckets }}"
  loop_control:
    label: "{{ item.name }} -> {{ item.mount_point }}"
  register: verify_mount
  changed_when: false
  failed_when: verify_mount.rc != 0
 - name: Display mounted S3 buckets
  ansible.builtin.debug:
    msg: |
      ✅ S3 Buckets mounted successfully:
      {% for bucket in s3_buckets %}
      - {{ bucket.name }} -> {{ bucket.mount_point }}
      {% endfor %}
 - name: Test write access to Immich bucket
  ansible.builtin.copy:
    content: "S3FS test file created on {{ ansible_date_time.iso8601 }}"
    dest: "{{ s3_buckets | selectattr('name', 'equalto', 'asxpio-immich-main') | map(attribute='mount_point') | first }}/.s3fs-test"
    mode: '0644'
  when: "'asxpio-immich-main' in s3_buckets | map(attribute='name')"
  register: write_test
 - name: Remove test file
  ansible.builtin.file:
    path: "{{ s3_buckets | selectattr('name', 'equalto', 'asxpio-immich-main') | map(attribute='mount_point') | first }}/.s3fs-test"
    state: absent
  when: write_test is changed
--- a/ansible/roles/s3fs/templates/passwd-s3fs.j2
+++ b/ansible/roles/s3fs/templates/passwd-s3fs.j2
@ -0,0 +1 @@
 {{ s3_access_key }}:{{ s3_secret_key }}
--- a/ansible/roles/ssh_hardening/defaults/main.yaml
+++ b/ansible/roles/ssh_hardening/defaults/main.yaml
@ -2,9 +2,10 @@
 # SSH Hardening Role Default Variables
 # Networks allowed to access SSH (VPC and WireGuard)
 # These reference the centralized network variables from group_vars/network.yaml
 ssh_allowed_networks:
-  - 10.20.0.0/24  # WireGuard VPN
+  - "{{ wireguard_network }}"  # WireGuard VPN
-  - 10.10.0.0/20  # Internal VPC
+  - "{{ vpc_network }}"  # Internal VPC
 # Whether to allow public SSH access for git operations (Forgejo)
 # Set to true on Forgejo host, false on other servers
--- a/ansible/roles/ssh_hardening/tasks/main.yaml
+++ b/ansible/roles/ssh_hardening/tasks/main.yaml
@ -16,7 +16,30 @@
    owner: root
    group: root
    validate: 'sshd -t -f %s'
-  notify: restart sshd
+  register: ssh_config_changed
 - name: Restart SSH daemon asynchronously (to maintain connection)
  ansible.builtin.shell: "nohup systemctl restart sshd &"
  when: ssh_config_changed.changed
  changed_when: true
 - name: Wait for SSH port to become available
  ansible.builtin.wait_for:
    host: "{{ ansible_host | default(inventory_hostname) }}"
    port: 22
    delay: 2
    timeout: 30
    state: started
  when: ssh_config_changed.changed
  delegate_to: localhost
 - name: Reset SSH connection to pick up new configuration
  ansible.builtin.meta: reset_connection
  when: ssh_config_changed.changed
 - name: Verify SSH connection after restart
  ansible.builtin.ping:
  when: ssh_config_changed.changed
 - name: Configure iptables for SSH restrictions (non-Forgejo servers)
  block:
@ -25,7 +48,7 @@
        chain: INPUT
        protocol: tcp
        destination_port: '22'
-        source: 10.20.0.0/24
+        source: "{{ wireguard_network }}"
        jump: ACCEPT
        comment: Allow SSH from WireGuard VPN
@ -34,7 +57,7 @@
        chain: INPUT
        protocol: tcp
        destination_port: '22'
-        source: 10.10.0.0/20
+        source: "{{ vpc_network }}"
        jump: ACCEPT
        comment: Allow SSH from internal VPC
--- a/ansible/roles/ssh_hardening/templates/99-ssh-hardening.conf.j2
+++ b/ansible/roles/ssh_hardening/templates/99-ssh-hardening.conf.j2
@ -20,13 +20,10 @@ KexAlgorithms {{ ssh_kex_algorithms | join(',') }}
 PubkeyAcceptedAlgorithms {{ ssh_pubkey_accepted_algorithms | join(',') }}
 # Access control - restrict shell users to trusted networks only
-{% for user in ssh_shell_users %}
+{% if ssh_shell_users | length > 0 and ssh_allowed_networks | length > 0 %}
-{% for network in ssh_allowed_networks %}
+AllowUsers {% for user in ssh_shell_users %}{% for network in ssh_allowed_networks %}{{ user }}@{{ network }} {% endfor %}{% endfor %}{% if ssh_allow_public_git %}{{ ssh_git_user }}{% endif %}
 AllowUsers {{ user }}@{{ network }}
 {% endfor %}
 {% endfor %}
-{% if ssh_allow_public_git %}
+{% elif ssh_allow_public_git %}
 # Allow Forgejo git user from anywhere for git operations
 AllowUsers {{ ssh_git_user }}
 {% endif %}
--- a/ansible/roles/traefik/tasks/main.yaml
+++ b/ansible/roles/traefik/tasks/main.yaml
@ -1,14 +1,6 @@
 ---
- name: Create Traefik directories
+- name: Validate Traefik variables
-  file:
+  ansible.builtin.import_tasks: validate.yaml
    path: "{{ item }}"
    state: directory
    mode: '0755'
  loop:
    - /opt/traefik
    - /opt/traefik/config
    - /opt/traefik/letsencrypt
    - /opt/traefik/logs
 - name: Create Traefik Docker network
  community.docker.docker_network:
@ -16,36 +8,21 @@
    driver: bridge
    state: present
- name: Deploy Traefik static configuration
+- name: Deploy Traefik using shared docker_service role
-  template:
+  ansible.builtin.include_role:
-    src: traefik.yml.j2
+    name: docker_service
-    dest: /opt/traefik/traefik.yml
+  vars:
-    mode: '0644'
+    service_name: "traefik"
-  notify: Restart Traefik
+    service_base_path: "{{ traefik_base_path }}"
-
+    service_directories:
- name: Deploy Traefik dynamic configuration
+      - { path: "config", mode: "0755" }
-  template:
+      - { path: "letsencrypt", mode: "0755" }
-    src: dynamic.yml.j2
+      - { path: "logs", mode: "0755" }
-    dest: /opt/traefik/config/dynamic.yml
+    service_templates:
-    mode: '0644'
+      - { src: "traefik.yml.j2", dest: "traefik.yml", mode: "0644" }
-  notify: Restart Traefik
+      - { src: "dynamic.yml.j2", dest: "config/dynamic.yml", mode: "0644" }
-
+      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
- name: Deploy Traefik docker-compose
+    health_check_enabled: true
-  template:
+    health_check_host: "{{ wireguard_ip }}"  # Dashboard only accessible on WireGuard IP
-    src: docker-compose.yml.j2
+    health_check_port: "{{ traefik_dashboard_port }}"
-    dest: /opt/traefik/docker-compose.yml
+    health_check_delay: 3
    mode: '0644'
  notify: Restart Traefik
 - name: Start Traefik
  community.docker.docker_compose_v2:
    project_src: /opt/traefik
    state: present
  register: traefik_output
 - name: Wait for Traefik to be healthy
  wait_for:
    host: "{{ wireguard_ipv4 }}"
    port: 8080
    timeout: 30
    delay: 3
--- a/ansible/roles/traefik/tasks/validate.yaml
+++ b/ansible/roles/traefik/tasks/validate.yaml
@ -0,0 +1,23 @@
 ---
 # Variable validation for traefik role
 - name: Validate Traefik required variables
  ansible.builtin.assert:
    that:
      - traefik_base_path is defined
      - traefik_base_path | length > 0
      - traefik_network is defined
      - traefik_network | length > 0
      - traefik_dashboard_port is defined
      - traefik_dashboard_port | int > 0
      - traefik_dashboard_port | int < 65536
      - letsencrypt_email is defined
      - letsencrypt_email | length > 0
      - letsencrypt_email is match("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")
    fail_msg: |
      Traefik role requires:
      - traefik_base_path (installation directory)
      - traefik_network (Docker network name)
      - traefik_dashboard_port (1-65535)
      - letsencrypt_email (valid email address)
    success_msg: "Traefik variables validated successfully"
--- a/ansible/roles/wireguard/tasks/main.yaml
+++ b/ansible/roles/wireguard/tasks/main.yaml
@ -1,4 +1,7 @@
 ---
 - name: Validate WireGuard variables
  ansible.builtin.import_tasks: validate.yaml
 - name: Install WireGuard
  ansible.builtin.apt:
    name:
--- a/ansible/roles/wireguard/tasks/validate.yaml
+++ b/ansible/roles/wireguard/tasks/validate.yaml
@ -0,0 +1,19 @@
 ---
 # Variable validation for wireguard role
 - name: Validate WireGuard required variables
  ansible.builtin.assert:
    that:
      - wireguard_network is defined
      - wireguard_network | length > 0
      - wireguard_port is defined
      - wireguard_port | int > 0
      - wireguard_port | int < 65536
      - wireguard_interface is defined
      - wireguard_interface | length > 0
    fail_msg: |
      WireGuard role requires:
      - wireguard_network (CIDR notation, e.g., 10.20.0.0/24)
      - wireguard_port (1-65535)
      - wireguard_interface (e.g., wg0)
    success_msg: "WireGuard variables validated successfully"
--- a/ansible/roles/wireguard/templates/client.conf.j2
+++ b/ansible/roles/wireguard/templates/client.conf.j2
@ -14,7 +14,7 @@ PublicKey = {{ wireguard_server_public_key }}
 {% if item.preshared_key %}
 PresharedKey = {{ item.preshared_key }}
 {% endif %}
-Endpoint = {{ ansible_host }}:{{ wireguard_port }}
+Endpoint = {{ public_ip }}:{{ wireguard_port }}
 # Allow all VPN traffic (peer-to-peer connectivity)
 AllowedIPs = {{ wireguard_ipv4_network }},{{ wireguard_ipv6_network }}{% if wireguard_additional_routes | length > 0 %},{{ wireguard_additional_routes | join(',') }}{% endif %}
--- a/ansible/scripts/setup-vault.sh
+++ b/ansible/scripts/setup-vault.sh
@ -0,0 +1,221 @@
 #!/usr/bin/env bash
 set -euo pipefail
 # Ansible Vault Setup Script
 # This script helps you set up encrypted secrets for the Ansible infrastructure
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ANSIBLE_DIR="$(dirname "$SCRIPT_DIR")"
 VAULT_FILE="$ANSIBLE_DIR/inventory/group_vars/all/vault.yaml"
 VAULT_PASS_FILE="$ANSIBLE_DIR/.vault_pass"
 echo "========================================"
 echo "Ansible Vault Setup"
 echo "========================================"
 echo ""
 # Check if ansible-vault is available
 if ! command -v ansible-vault &> /dev/null; then
    echo "❌ Error: ansible-vault command not found"
    echo "   Please install Ansible first: pip install ansible"
    exit 1
 fi
 # Function to generate secrets
 generate_secrets() {
    echo "Generating new Forgejo secrets..."
    echo ""
    echo "SECRET_KEY (64 chars hex):"
    openssl rand -hex 32
    echo ""
    echo "JWT_SECRET (base64):"
    openssl rand -base64 32
    echo ""
    echo "LFS_JWT_SECRET (base64):"
    openssl rand -base64 32
    echo ""
    echo "Copy these values to your vault.yaml file"
    echo ""
 }
 # Function to create vault password file
 create_vault_password() {
    if [ -f "$VAULT_PASS_FILE" ]; then
        echo "⚠️  Vault password file already exists: $VAULT_PASS_FILE"
        read -p "Overwrite? (y/N): " -n 1 -r
        echo
        if [[ ! $REPLY =~ ^[Yy]$ ]]; then
            echo "Skipping vault password creation"
            return
        fi
    fi
    echo "Creating vault password file..."
    read -sp "Enter vault password: " password
    echo ""
    read -sp "Confirm password: " password2
    echo ""
    if [ "$password" != "$password2" ]; then
        echo "❌ Passwords don't match"
        exit 1
    fi
    echo "$password" > "$VAULT_PASS_FILE"
    chmod 600 "$VAULT_PASS_FILE"
    echo "✅ Vault password saved to: $VAULT_PASS_FILE"
    echo "   (This file is git-ignored)"
    echo ""
 }
 # Function to encrypt vault
 encrypt_vault() {
    if [ ! -f "$VAULT_FILE" ]; then
        echo "❌ Vault file not found: $VAULT_FILE"
        echo "   Please create it first from the template"
        exit 1
    fi
    # Check if already encrypted
    if head -n1 "$VAULT_FILE" | grep -q "ANSIBLE_VAULT"; then
        echo "⚠️  Vault file is already encrypted"
        return
    fi
    if [ ! -f "$VAULT_PASS_FILE" ]; then
        echo "❌ Vault password file not found: $VAULT_PASS_FILE"
        echo "   Run: $0 create-password"
        exit 1
    fi
    echo "Encrypting vault file..."
    ansible-vault encrypt "$VAULT_FILE" --vault-password-file="$VAULT_PASS_FILE" --encrypt-vault-id default
    echo "✅ Vault encrypted successfully"
    echo ""
 }
 # Function to decrypt vault
 decrypt_vault() {
    if [ ! -f "$VAULT_FILE" ]; then
        echo "❌ Vault file not found: $VAULT_FILE"
        exit 1
    fi
    # Check if encrypted
    if ! head -n1 "$VAULT_FILE" | grep -q "ANSIBLE_VAULT"; then
        echo "⚠️  Vault file is not encrypted"
        return
    fi
    if [ ! -f "$VAULT_PASS_FILE" ]; then
        echo "❌ Vault password file not found: $VAULT_PASS_FILE"
        exit 1
    fi
    echo "Decrypting vault file..."
    ansible-vault decrypt "$VAULT_FILE" --vault-password-file="$VAULT_PASS_FILE"
    echo "✅ Vault decrypted successfully"
    echo "⚠️  Remember to re-encrypt before committing!"
    echo ""
 }
 # Function to edit vault
 edit_vault() {
    if [ ! -f "$VAULT_FILE" ]; then
        echo "❌ Vault file not found: $VAULT_FILE"
        exit 1
    fi
    if [ ! -f "$VAULT_PASS_FILE" ]; then
        echo "❌ Vault password file not found: $VAULT_PASS_FILE"
        echo "   Run: $0 create-password"
        exit 1
    fi
    echo "Opening vault for editing..."
    ansible-vault edit "$VAULT_FILE" --vault-password-file="$VAULT_PASS_FILE"
 }
 # Function to view vault
 view_vault() {
    if [ ! -f "$VAULT_FILE" ]; then
        echo "❌ Vault file not found: $VAULT_FILE"
        exit 1
    fi
    if [ ! -f "$VAULT_PASS_FILE" ]; then
        echo "❌ Vault password file not found: $VAULT_PASS_FILE"
        exit 1
    fi
    ansible-vault view "$VAULT_FILE" --vault-password-file="$VAULT_PASS_FILE"
 }
 # Function to check vault status
 check_status() {
    echo "Vault Status:"
    echo "-------------"
    if [ -f "$VAULT_PASS_FILE" ]; then
        echo "✅ Vault password file exists: $VAULT_PASS_FILE"
    else
        echo "❌ Vault password file not found: $VAULT_PASS_FILE"
    fi
    if [ -f "$VAULT_FILE" ]; then
        if head -n1 "$VAULT_FILE" | grep -q "ANSIBLE_VAULT"; then
            echo "✅ Vault file is encrypted: $VAULT_FILE"
        else
            echo "⚠️  Vault file exists but is NOT encrypted: $VAULT_FILE"
            echo "   Run: $0 encrypt"
        fi
    else
        echo "❌ Vault file not found: $VAULT_FILE"
    fi
    echo ""
 }
 # Main menu
 case "${1:-}" in
    generate-secrets)
        generate_secrets
        ;;
    create-password)
        create_vault_password
        ;;
    encrypt)
        encrypt_vault
        ;;
    decrypt)
        decrypt_vault
        ;;
    edit)
        edit_vault
        ;;
    view)
        view_vault
        ;;
    status)
        check_status
        ;;
    *)
        echo "Usage: $0 {generate-secrets|create-password|encrypt|decrypt|edit|view|status}"
        echo ""
        echo "Commands:"
        echo "  generate-secrets  - Generate new Forgejo secrets"
        echo "  create-password   - Create vault password file (.vault_pass)"
        echo "  encrypt          - Encrypt vault.yaml"
        echo "  decrypt          - Decrypt vault.yaml (use carefully!)"
        echo "  edit             - Edit encrypted vault"
        echo "  view             - View encrypted vault contents"
        echo "  status           - Check vault status"
        echo ""
        echo "Quick setup:"
        echo "  1. $0 generate-secrets    # Generate new secrets"
        echo "  2. Edit vault.yaml with your secrets"
        echo "  3. $0 create-password     # Create password file"
        echo "  4. $0 encrypt            # Encrypt vault"
        echo ""
        exit 1
        ;;
 esac