Copyright (C) 2026 Doan Luu Duc Tai This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . STACK OPERATIONS Root: ~/stack. Workstation: local Linux. Production: ops@23.155.44.67. Public source: https://git.doan.luuduct.ai/stack. Production commands use bin/run, which selects packed Guix tools/config. Direct bin/stack uses host tools; never mix engines against the same storage. KEEP For new-host recovery keep the carry file described in GUIX.txt: encrypted application export plus matching checked release, Guix runtime and bootstrap. Private state is encrypted; public tools/source are plaintext. Keep its trusted checksum and an independent copy on another device/location. It preserves one .env, databases, TLS keys, bare repos, working Git history, binaries, browser assets, sources and exact service images. Engine storage, caches, host accounts/SSH/firewall, disks and OS configuration are excluded. Public Git alone cannot recover private state. Keep GPG secret key/passphrase separately; default backup recipient: 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 Keep an independent protected key export/revocation certificate. Chat owner and visitor private identity files/passwords also need separate backups. Never send private keys or passwords through chat. DEVELOP LOCAL cargo stack dev cargo stack status cargo stack down Open http://localhost:8080, /git/, /search/, http://chat.localhost:8080/. Alternate port: cargo stack dev --port 9080. Local mode binds loopback only; no public TLS or DNS changes. Missing .env is initialized with random secrets. infra/services.toml lists each service's environment, mounts and health check. Local development uses host Podman. Project storage is var/podman; bare podman ps uses another store. From the local checkout: podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a Keep both global options for logs/exec/images. Use bin/run status on production. Never transfer var/podman as backup; export saved images instead. CHECK AND BUILD LOCAL cargo install --locked --version 0.20.2 cargo-deny --root .tools cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools cargo stack check cargo run --release --locked -- release Check covers TypeScript compilers/typed ESLint/npm audit, Guix shell fixtures, formatting, strict workspace Clippy/tests/private Rustdoc, advisories, exact duplicate exceptions and licenses. cargo-about regenerates the license page. Dependency edits require deliberate lockfile review, full checks and notices. Never waive failures. Format with cargo fmt --all. Build the Guix pack first when adding/changing runtime tools: GUIX.txt. Release builds static Rust/chat WASM, corresponding source, cgit/chat images and complete image export. Output: dist/stack-release-TIMESTAMP.tar.gz and .sha256. Actual .env, live data, .git and prior backups are excluded. Build/test locally; no compiler or registry pull is needed on the matching x86_64 production host. Restore/inspect a disposable release before deployment. BACK UP PRODUCTION cd ~/stack ./bin/run backup --include-images Briefly pauses Caddy/cgit/SearXNG during file copy; resumes before compression and encryption. Chat stays live for SQLite online backup. Export includes chat/chat-local snapshots; excludes raw journals and server locks. Verifies recipient fingerprint, bare repos and exact image identities. Output has restrictive permissions: var/exports/stack-TIMESTAMP.tar.gz.gpg var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 Copy both files off the VPS; substitute actual filenames: umask 077 mkdir -p ~/stack/var/backups/production scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg ~/stack/var/backups/production/ scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 ~/stack/var/backups/production/ cd ~/stack/var/backups/production sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256 No scheduled backup. Choose retention/off-host copies; remove matching VPS exports only after verification. Never delete .env or var/data. Export without --include-images is not standalone production recovery. Pair the verified export with a checked release using infra/guix/bundle.sh; the resulting carry file restores with the matching manager/source/runtime. RECOVER ON LINUX Follow GUIX.txt for host prerequisites and GPG key access. On a fresh x86_64 Linux host, as the service user, with a new destination: tar -xzf recovery.tar.gz sudo ./stack-recovery/bootstrap ./stack-recovery/restore /home/ops/stack Capsule verifies/decrypts the export, activates its paired release and starts four services. A failed activation can resume with the same restore command. Keep hostname/origin and identity files unchanged; repoint DNS when IP changes. Recovery does not configure host SSH, firewall, account or boot startup. A raw format-2 export can be restored for inspection on a host that already has compatible tools and any required Guix closure: ./bin/run restore /path/export.tar.gz.gpg /path/stack-restored Raw restore validates all manifest paths/hashes, defaults local mode with a fresh project identity, starts nothing and refuses an existing destination. Public release archives restore similarly but contain no private data. An old raw export may lack the launcher/runtime; use a paired capsule for complete recovery. Test owner/visitor identities and history after recovery. HOST AND BOOT SETUP Host/provider firewall: retain SSH and allow HTTP 80/tcp, HTTPS 443/tcp+udp for applicable IPv4/IPv6. Never publish backend container ports. A points to the new IPv4; publish AAAA only with working IPv6. Keep www/WKD aliases. Rootless IDs, mapping helpers, runtime directory and low-port sysctl: GUIX.txt. Debian host-tools fallback: infra/host-setup.sh installs distro Podman and prerequisites. Guix supplies its own Podman/network tools. cd ~/stack ./bin/run up --production systemctl --user link ~/stack/infra/stack.service systemctl --user daemon-reload systemctl --user enable --now stack.service systemctl --user status stack.service Administrator enables linger for the service user. Unit assumes ~/stack; adapt paths for another location. Caddy stores credentials in var/data/caddy, issues/renews public certificates and redirects HTTP/www. ACME needs public DNS/connectivity. Containers use unless-stopped restart policy. Current VPS SSH Git compatibility path /srv/git/stack links to /home/ops/stack/var/data/repos/stack. Preserve it for existing clone URLs; public HTTPS Git needs no host symlink. DEPLOY A CHECKED RELEASE Review/stage public changes, sign and push the current branch: git diff --cached --stat git commit -S3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 -m 'Describe the change' git push origin HEAD Do not deploy with git pull. Take a verified independent export first; retain complete old worktree/binary/unit, var/state.json, image-lock.json, images.tar and prior engine storage for rollback. Guix upgrade/rollback rules: GUIX.txt. Copy the checked public release and checksum to ~/stack/var/incoming. Before activation install any new closure through its verified capsule bootstrap; an existing managed Guix store needs separate planning. Bash on VPS, after that bootstrap; substitute actual TIMESTAMP: ( set -eu cd ~/stack STACK_ROOT=$PWD STACK_RELEASE=$PWD/var/incoming/stack-release-TIMESTAMP.tar.gz (cd var/incoming && sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256) STACK_STAGE=$(mktemp -d "$PWD/var/incoming/.activate.XXXXXX") tar -xzf "$STACK_RELEASE" -C "$STACK_STAGE" stack/bin/stack stack/bin/guix-runtime.tar.gz stack/infra/guix/containers.conf mkdir "$STACK_STAGE/runtime" tar -xzf "$STACK_STAGE/stack/bin/guix-runtime.tar.gz" --no-same-owner -C "$STACK_STAGE/runtime" env PATH="$STACK_STAGE/runtime/bin:$PATH" CONTAINERS_CONF="$STACK_STAGE/stack/infra/guix/containers.conf" "$STACK_STAGE/stack/bin/stack" --root "$STACK_ROOT" activate "$STACK_RELEASE" systemctl --user daemon-reload systemctl --user restart stack.service ./bin/run status chmod -R u+w -- "$STACK_STAGE" rm -rf -- "$STACK_STAGE" ) Use the NEW manager and matching runtime. Activation verifies archive/platform, replaces public source/binaries/images, preserves .env/var/data/.git and replaces containers. Expect interruption. The installed unit may already link to infra/stack.service; reload its definition rather than copying over itself. If activation fails, retain staging/data and diagnose before retry or rollback. Verify public routes, TLS/WKD, owner and database before removing incoming files. RECOVERY LIMITS; KEY ROTATION Other CPU architectures need native release builds. Host DNS/PTR, firewall, accounts and SSH remain outside the carry file. New databases need consistent snapshot/recovery support before inclusion. Preserve both participants' chat files/passwords; server backups cannot replace them. CHAT-PRODUCTION.txt. For a new backup recipient, publish its public key, update root .env and test encryption/recovery. One export can select --recipient FULL_FINGERPRINT and --key /path/public-key.asc. Keep old secret keys while retained exports need them. LICENSE PAGE; REDIRECTS cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves it on the main-site origin; chat redirects its former local license URL there. Source exports retain offline license texts. cgit Source/Raw expose the generated file. Homepage /message and /message/ redirect to the configured chat origin.