diff options
| author | Doan Luu Duc Tai <doan@luuduct.ai> | 2026-11-01 08:25:58 -0600 |
|---|---|---|
| committer | Doan Luu Duc Tai <doan@luuduct.ai> | 2026-11-01 08:25:58 -0600 |
| commit | 607be5ee239a9b7969eab96145c8af4ee57053d1 (patch) | |
| tree | 582ada863d690ed8f113f266f2afc8252d421c39 /docs/OPERATIONS.txt | |
| parent | 9349a2de648e241da8e00ba7ae25cb2a877ff3e9 (diff) | |
Diffstat (limited to 'docs/OPERATIONS.txt')
| -rw-r--r-- | docs/OPERATIONS.txt | 312 |
1 files changed, 131 insertions, 181 deletions
diff --git a/docs/OPERATIONS.txt b/docs/OPERATIONS.txt index 91ab10b..bf675be 100644 --- a/docs/OPERATIONS.txt +++ b/docs/OPERATIONS.txt @@ -15,84 +15,82 @@ along with this program. If not, see <https://www.gnu.org/licenses/>. STACK OPERATIONS - Root: ~/stack. Develop on local Linux; production: ops@23.155.44.67. + Root: ~/stack. Workstation: local Linux. Production: ops@23.155.44.67. Public source: https://git.doan.luuduct.ai/stack. - GNU Affero General Public License, version 3 or later. - Experimental Guix capsule: GUIX.txt. Commands below use host Podman. + Production commands use bin/run, which selects packed Guix tools/config. + Direct bin/stack uses host tools; never mix engines against the same storage. KEEP - Full --include-images encrypted export: public source, compiled CLI, .env, - persistent data, certificates, bare repos, working Git history if present, - exact service images, checksum manifest. No old server filesystem needed. - Excludes engine storage, caches, old exports, host packages/SSH keys/firewall, - disks, OS configuration. Recreate host setup separately. - Public Git alone cannot recover .env, live data, TLS keys, or var/ repos. - - Decryption needs GPG private key + passphrase. Keep key export/revocation - certificate in KeePassXC; keep independent database copy. Server holds public - key only. Never send private key/password through chat. - Default recipient: 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6. + For new-host recovery keep the carry file described in GUIX.txt: encrypted + application export plus matching checked release, Guix runtime and bootstrap. + Private state is encrypted; public tools/source are plaintext. Keep its trusted + checksum and an independent copy on another device/location. + It preserves one .env, databases, TLS keys, bare repos, working Git history, + binaries, browser assets, sources and exact service images. Engine storage, + caches, host accounts/SSH/firewall, disks and OS configuration are excluded. + Public Git alone cannot recover private state. + + Keep GPG secret key/passphrase separately; default backup recipient: + 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 + Keep an independent protected key export/revocation certificate. Chat owner + and visitor private identity files/passwords also need separate backups. + Never send private keys or passwords through chat. DEVELOP LOCAL - From ~/stack on workstation: - cargo stack dev cargo stack status cargo stack down - Open http://localhost:8080, http://localhost:8080/git/, - http://localhost:8080/search/. Alternate port: - - cargo stack dev --port 9080 - - Local mode: loopback only, local links, unchanged public DNS, no public TLS. - Missing .env: create with random SearXNG secret. Edit secrets there; never commit. - infra/: service settings. web/: content. crates/: new backends; add workspace members. - infra/services.toml declares services. Each gets only listed environment keys. + Open http://localhost:8080, /git/, /search/, http://chat.localhost:8080/. + Alternate port: cargo stack dev --port 9080. Local mode binds loopback only; + no public TLS or DNS changes. Missing .env is initialized with random secrets. + infra/services.toml lists each service's environment, mounts and health check. - Project engine storage: var/podman/. Bare 'podman ps' uses another store. - Inspect project containers from ~/stack: + Local development uses host Podman. Project storage is var/podman; bare + podman ps uses another store. From the local checkout: podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a - Keep both global options for 'logs CONTAINER', 'exec CONTAINER', 'images'. - cargo stack status gives names. Never migrate var/podman/; export images. + Keep both global options for logs/exec/images. Use bin/run status on production. + Never transfer var/podman as backup; export saved images instead. -CHECK LOCAL - - Install development audit tools once: +CHECK AND BUILD LOCAL cargo install --locked --version 0.20.2 cargo-deny --root .tools cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools - cargo stack check + cargo run --release --locked -- release - Checks Guix shell/bootstrap fixtures (bubblewrap), formatting, workspace Clippy, tests, private - rustdoc, advisories, duplicate versions, sources, licenses. cargo-about - regenerates THIRD-PARTY-LICENSES.html. Format with cargo fmt --all. - Dependency changes: update Cargo.lock deliberately; run full check; commit - lockfile and license page. Never disable audit failures to ship. + Check covers TypeScript compilers/typed ESLint/npm audit, Guix shell fixtures, + formatting, strict workspace Clippy/tests/private Rustdoc, advisories, exact + duplicate exceptions and licenses. cargo-about regenerates the license page. + Dependency edits require deliberate lockfile review, full checks and notices. + Never waive failures. Format with cargo fmt --all. -BACK UP PRODUCTION + Build the Guix pack first when adding/changing runtime tools: GUIX.txt. + Release builds static Rust/chat WASM, corresponding source, cgit/chat images + and complete image export. Output: dist/stack-release-TIMESTAMP.tar.gz and + .sha256. Actual .env, live data, .git and prior backups are excluded. + Build/test locally; no compiler or registry pull is needed on the matching + x86_64 production host. Restore/inspect a disposable release before deployment. - On VPS: +BACK UP PRODUCTION cd ~/stack - ./bin/stack backup --include-images + ./bin/run backup --include-images Briefly pauses Caddy/cgit/SearXNG during file copy; resumes before compression - and encryption. Chat stays live for SQLite online backup. Snapshots include - chat and chat-local; exclude raw journals and server locks. - Verifies public recipient fingerprint, checks bare repos, saves exact images: + and encryption. Chat stays live for SQLite online backup. Export includes + chat/chat-local snapshots; excludes raw journals and server locks. + Verifies recipient fingerprint, bare repos and exact image identities. + Output has restrictive permissions: var/exports/stack-TIMESTAMP.tar.gz.gpg var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 - Restrictive permissions. Removes temporary plaintext on success/handled errors. - No scheduled backup. Choose retention and off-host copies yourself. - On workstation, copy actual filenames printed above: + Copy both files off the VPS; substitute actual filenames: umask 077 mkdir -p ~/stack/var/backups/production @@ -101,166 +99,118 @@ BACK UP PRODUCTION cd ~/stack/var/backups/production sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256 - Replace TIMESTAMP. Checksum detects corruption; keep trusted export + checksum. - Keep another copy on separate device/location. After verification, optionally - delete matching VPS export files only. Never delete var/data or .env. - ./bin/stack backup without images needs registry/build access for local startup; - not standalone production recovery. Prefer --include-images. - -RESTORE ON LINUX - - Host tools: Podman, Git, GnuPG, rootless networking prerequisites. - Target user needs >=65536 subordinate UIDs/GIDs. Use distro-supported Podman. - Prebuilt binary/images: x86_64 Linux (linux/amd64). - Copy export + checksum; verify checksum first. If needed, import key: - - gpg --import private-key-backup.asc - - GnuPG prompts locally. Never pass passphrase through arguments or .env. - Existing compatible CLI: - - ./bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack-restored - - Destination must not exist. Restore decrypts privately; rejects unsafe paths - and links; checks every manifest entry; assigns unique project identity; - defaults local; starts nothing; never overwrites existing installation. + No scheduled backup. Choose retention/off-host copies; remove matching VPS + exports only after verification. Never delete .env or var/data. + Export without --include-images is not standalone production recovery. + Pair the verified export with a checked release using infra/guix/bundle.sh; + the resulting carry file restores with the matching manager/source/runtime. - No CLI/toolchain yet: bootstrap trusted export with GnuPG/tar. Bash/fish: +RECOVER ON LINUX - umask 077 - mkdir -p ~/stack-bootstrap - gpg --output ~/stack-bootstrap/export.tar.gz --decrypt /path/to/stack-TIMESTAMP.tar.gz.gpg - tar -xzf ~/stack-bootstrap/export.tar.gz -C ~/stack-bootstrap stack/bin/stack - ~/stack-bootstrap/stack/bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack - - Bootstrap executes extracted code: use only your trusted export. - Rust restore validates full archive. After success, delete ~/stack-bootstrap: - it holds decrypted archive. Retain encrypted backup. - Test restored stack locally first: - - cd ~/stack - ./bin/stack up --local --port 8080 - ./bin/stack status + Follow GUIX.txt for host prerequisites and GPG key access. On a fresh x86_64 + Linux host, as the service user, with a new destination: - Saved images load automatically; image lock verifies identities. Matching - architecture needs no registry pull/build. Check homepage, Git, search, WKD. - ./bin/stack down before production setup. - Same restore accepts release archives: source/binary/images, no .env/live data. - Starting restored release creates fresh state; cannot recover lost data. + tar -xzf recovery.tar.gz + sudo ./stack-recovery/bootstrap + ./stack-recovery/restore /home/ops/stack -NEW PRODUCTION HOST + Capsule verifies/decrypts the export, activates its paired release and starts + four services. A failed activation can resume with the same restore command. + Keep hostname/origin and identity files unchanged; repoint DNS when IP changes. + Recovery does not configure host SSH, firewall, account or boot startup. - Restore into target user's ~/stack. Debian: review infra/host-setup.sh; - replace 'ops' if needed; run: + A raw format-2 export can be restored for inspection on a host that already + has compatible tools and any required Guix closure: - sudo sh ~/stack/infra/host-setup.sh + ./bin/run restore /path/export.tar.gz.gpg /path/stack-restored - Installs Podman/network tools, checks existing subordinate ranges, enables - user linger, sets: + Raw restore validates all manifest paths/hashes, defaults local mode with a + fresh project identity, starts nothing and refuses an existing destination. + Public release archives restore similarly but contain no private data. + An old raw export may lack the launcher/runtime; use a paired capsule for + complete recovery. Test owner/visitor identities and history after recovery. - net.ipv4.ip_unprivileged_port_start=80 +HOST AND BOOT SETUP - System-wide rootless ports >=80; ports <80 stay privileged. SSH/firewall unchanged. - Missing ranges: allocate unused ranges through distro account tools; rerun. - Other distros: equivalent packages, sysctl, linger. Supplied boot unit needs - systemd user manager. - - Host/provider firewall: allow 22/tcp, 80/tcp, 443/tcp, 443/udp for applicable - IPv4/IPv6. Never publish backend container ports. Preserve SSH; verify address - before changing DNS. A: new IPv4. AAAA: working new IPv6 only; remove stale AAAA. - Hosts live in root .env; edit and reapply rendered settings. infra/Caddyfile - is public template. Keys/render-only commands: CONFIGURATION.txt. - Preserve www aliases and openpgpkey names pointing to canonical hosts. + Host/provider firewall: retain SSH and allow HTTP 80/tcp, HTTPS 443/tcp+udp + for applicable IPv4/IPv6. Never publish backend container ports. A points to + the new IPv4; publish AAAA only with working IPv6. Keep www/WKD aliases. + Rootless IDs, mapping helpers, runtime directory and low-port sysctl: GUIX.txt. + Debian host-tools fallback: infra/host-setup.sh installs distro Podman and + prerequisites. Guix supplies its own Podman/network tools. cd ~/stack - ./bin/stack up --production - - Production binds 80/tcp, 443/tcp, 443/udp. Caddy issues/renews certificates, - redirects HTTP to HTTPS and www to canonical URLs. State: var/data/caddy/. - Public issuance needs working public DNS/connectivity. - Enable boot startup: - + ./bin/run up --production systemctl --user link ~/stack/infra/stack.service systemctl --user daemon-reload systemctl --user enable --now stack.service systemctl --user status stack.service - Unit assumes ~/stack; edit paths for other location. Current unit uses bin/run: - confirm launcher exists before enabling. Linger keeps user service alive - without login. Containers use unless-stopped restart policy. + Administrator enables linger for the service user. Unit assumes ~/stack; + adapt paths for another location. Caddy stores credentials in var/data/caddy, + issues/renews public certificates and redirects HTTP/www. ACME needs public + DNS/connectivity. Containers use unless-stopped restart policy. - Optional former VPS SSH Git path; administrator only: + Current VPS SSH Git compatibility path /srv/git/stack links to + /home/ops/stack/var/data/repos/stack. Preserve it for existing clone URLs; + public HTTPS Git needs no host symlink. - sudo install -d -m 755 /srv/git - sudo ln -s /home/ops/stack/var/data/repos/stack /srv/git/stack +DEPLOY A CHECKED RELEASE - Never overwrite existing repo. Current VPS already has this layout link. - Public HTTPS Git needs no host symlink. + Review/stage public changes, sign and push the current branch: -BUILD AND DEPLOY - - Develop/check on workstation: - - cd ~/stack - cargo stack release - - Release reruns full check; builds static Rust CLI/chat + browser WASM; - packages matching source; builds cgit/chat containers; saves configured images. - Output: dist/stack-release-TIMESTAMP.tar.gz + .sha256. - Excludes .env, live data, .git history, previous backups. - Restore into fresh local test directory; up --local on free port; inspect - four services; stop test. Commit/push reviewed public changes: - - git add -A git diff --cached --stat git commit -S3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 -m 'Describe the change' - git push origin main - - Review staged files. .gitignore excludes secrets/runtime/artifacts. - Use RSA-2048 signed commits; author/key: POLICY.txt. - Deploy archive, never git pull. Copy actual filenames: + git push origin HEAD - ssh ops@23.155.44.67 'mkdir -p ~/stack/var/incoming' - scp dist/stack-release-TIMESTAMP.tar.gz ops@23.155.44.67:stack/var/incoming/ - scp dist/stack-release-TIMESTAMP.tar.gz.sha256 ops@23.155.44.67:stack/var/incoming/ + Do not deploy with git pull. Take a verified independent export first; retain + complete old worktree/binary/unit, var/state.json, image-lock.json, images.tar + and prior engine storage for rollback. Guix upgrade/rollback rules: GUIX.txt. + Copy the checked public release and checksum to ~/stack/var/incoming. + Before activation install any new closure through its verified capsule + bootstrap; an existing managed Guix store needs separate planning. - On VPS: + Bash on VPS, after that bootstrap; substitute actual TIMESTAMP: - cd ~/stack/var/incoming - sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256 + ( + set -eu cd ~/stack - mkdir -p var/incoming/bootstrap - tar -xzf var/incoming/stack-release-TIMESTAMP.tar.gz -C var/incoming/bootstrap stack/bin/stack - var/incoming/bootstrap/stack/bin/stack --root "$PWD" activate var/incoming/stack-release-TIMESTAMP.tar.gz - ./bin/stack status - rm -rf -- var/incoming/bootstrap - - Use verified NEW release binary: old binary may reject new service schema. - Activation verifies archive/platform, replaces public source/binary, loads - exact images, preserves .env/var/data, replaces running containers. No compiler, - Rust build, image build, or private-config overwrite. Expect brief interruption. - Take verified export before changes needing recovery. Verify production; - optionally delete corresponding incoming archive/checksum. - -RECOVERY LIMITS; ROTATE KEYS - - Application export is not bootable VPS image. Accounts, SSH, firewall, - packages remain host duties. Other CPU architecture needs native release. - Registry needed only without saved images or when rebuilding another target. - State: files, Git repos, Caddy, encrypted SQLite chat. SQLite uses online backup. - Recovery proof: reopen BOTH participants' unchanged identities; decrypt history. - See CHAT-PRODUCTION.txt. Future databases need supported snapshot/recovery. - - New backup recipient: publish public key, update default fingerprint, test - encryption/restore. Single export: --recipient FULL_FINGERPRINT --key /path/public-key.asc. - Retain old secret keys while any retained export needs them. - Restore accepts verified format-2 application archives. + STACK_ROOT=$PWD + STACK_RELEASE=$PWD/var/incoming/stack-release-TIMESTAMP.tar.gz + (cd var/incoming && sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256) + STACK_STAGE=$(mktemp -d "$PWD/var/incoming/.activate.XXXXXX") + tar -xzf "$STACK_RELEASE" -C "$STACK_STAGE" stack/bin/stack stack/bin/guix-runtime.tar.gz stack/infra/guix/containers.conf + mkdir "$STACK_STAGE/runtime" + tar -xzf "$STACK_STAGE/stack/bin/guix-runtime.tar.gz" --no-same-owner -C "$STACK_STAGE/runtime" + env PATH="$STACK_STAGE/runtime/bin:$PATH" CONTAINERS_CONF="$STACK_STAGE/stack/infra/guix/containers.conf" "$STACK_STAGE/stack/bin/stack" --root "$STACK_ROOT" activate "$STACK_RELEASE" + systemctl --user daemon-reload + systemctl --user restart stack.service + ./bin/run status + chmod -R u+w -- "$STACK_STAGE" + rm -rf -- "$STACK_STAGE" + ) + + Use the NEW manager and matching runtime. Activation verifies archive/platform, + replaces public source/binaries/images, preserves .env/var/data/.git and + replaces containers. Expect interruption. The installed unit may already link + to infra/stack.service; reload its definition rather than copying over itself. + If activation fails, retain staging/data and diagnose before retry or rollback. + Verify public routes, TLS/WKD, owner and database before removing incoming files. + +RECOVERY LIMITS; KEY ROTATION + + Other CPU architectures need native release builds. Host DNS/PTR, firewall, + accounts and SSH remain outside the carry file. New databases need consistent + snapshot/recovery support before inclusion. Preserve both participants' chat + files/passwords; server backups cannot replace them. CHAT-PRODUCTION.txt. + + For a new backup recipient, publish its public key, update root .env and test + encryption/recovery. One export can select --recipient FULL_FINGERPRINT and + --key /path/public-key.asc. Keep old secret keys while retained exports need them. LICENSE PAGE; REDIRECTS - cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves canonical - https://doan.luuduct.ai/THIRD-PARTY-LICENSES.html; other deployments use SITE_HOST. - Website/chat links share page; local preview uses local main-site origin. - cgit Source/Raw expose generated file; chat /THIRD-PARTY-LICENSES.html redirects - to canonical page. Source exports retain offline license texts. - Homepage /message and /message/ redirect to current chat origin. + cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves it on the + main-site origin; chat redirects its former local license URL there. Source + exports retain offline license texts. cgit Source/Raw expose the generated file. + Homepage /message and /message/ redirect to the configured chat origin. |
