Copyright (C) 2026 Doan Luu Duc Tai This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see . STACK OPERATIONS Root: ~/stack. Develop on local Linux; production: ops@23.155.44.67. Public source: https://git.doan.luuduct.ai/stack. GNU Affero General Public License, version 3 or later. KEEP Full --include-images encrypted export: public source, compiled CLI, .env, persistent data, certificates, bare repos, working Git history if present, exact service images, checksum manifest. No old server filesystem needed. Excludes engine storage, caches, old exports, host packages/SSH keys/firewall, disks, OS configuration. Recreate host setup separately. Public Git alone cannot recover .env, live data, TLS keys, or var/ repos. Decryption needs GPG private key + passphrase. Keep key export/revocation certificate in KeePassXC; keep independent database copy. Server holds public key only. Never send private key/password through chat. Default recipient: 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6. DEVELOP LOCAL From ~/stack on workstation: cargo stack dev cargo stack status cargo stack down Open http://localhost:8080, http://localhost:8080/git/, http://localhost:8080/search/. Alternate port: cargo stack dev --port 9080 Local mode: loopback only, local links, unchanged public DNS, no public TLS. Missing .env: create with random SearXNG secret. Edit secrets there; never commit. infra/: service settings. web/: content. crates/: new backends; add workspace members. infra/services.toml declares services. Each gets only listed environment keys. Project engine storage: var/podman/. Bare 'podman ps' uses another store. Inspect project containers from ~/stack: podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a Keep both global options for 'logs CONTAINER', 'exec CONTAINER', 'images'. cargo stack status gives names. Never migrate var/podman/; export images. CHECK LOCAL Install development audit tools once: cargo install --locked --version 0.20.2 cargo-deny --root .tools cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools cargo stack check Checks formatting, all workspace Clippy targets/features, tests, private rustdoc, advisories, duplicate versions, sources, licenses. cargo-about regenerates THIRD-PARTY-LICENSES.html. Format with cargo fmt --all. Dependency changes: update Cargo.lock deliberately; run full check; commit lockfile and license page. Never disable audit failures to ship. BACK UP PRODUCTION On VPS: cd ~/stack ./bin/stack backup --include-images Briefly pauses Caddy/cgit/SearXNG during file copy; resumes before compression and encryption. Chat stays live for SQLite online backup. Snapshots include chat and chat-local; exclude raw journals and server locks. Verifies public recipient fingerprint, checks bare repos, saves exact images: var/exports/stack-TIMESTAMP.tar.gz.gpg var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 Restrictive permissions. Removes temporary plaintext on success/handled errors. No scheduled backup. Choose retention and off-host copies yourself. On workstation, copy actual filenames printed above: umask 077 mkdir -p ~/stack/var/backups/production scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg ~/stack/var/backups/production/ scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 ~/stack/var/backups/production/ cd ~/stack/var/backups/production sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256 Replace TIMESTAMP. Checksum detects corruption; keep trusted export + checksum. Keep another copy on separate device/location. After verification, optionally delete matching VPS export files only. Never delete var/data or .env. ./bin/stack backup without images needs registry/build access for local startup; not standalone production recovery. Prefer --include-images. RESTORE ON LINUX Host tools: Podman, Git, GnuPG, rootless networking prerequisites. Target user needs >=65536 subordinate UIDs/GIDs. Use distro-supported Podman. Prebuilt binary/images: x86_64 Linux (linux/amd64). Copy export + checksum; verify checksum first. If needed, import key: gpg --import private-key-backup.asc GnuPG prompts locally. Never pass passphrase through arguments or .env. Existing compatible CLI: ./bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack-restored Destination must not exist. Restore decrypts privately; rejects unsafe paths and links; checks every manifest entry; assigns unique project identity; defaults local; starts nothing; never overwrites existing installation. No CLI/toolchain yet: bootstrap trusted export with GnuPG/tar. Bash/fish: umask 077 mkdir -p ~/stack-bootstrap gpg --output ~/stack-bootstrap/export.tar.gz --decrypt /path/to/stack-TIMESTAMP.tar.gz.gpg tar -xzf ~/stack-bootstrap/export.tar.gz -C ~/stack-bootstrap stack/bin/stack ~/stack-bootstrap/stack/bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack Bootstrap executes extracted code: use only your trusted export. Rust restore validates full archive. After success, delete ~/stack-bootstrap: it holds decrypted archive. Retain encrypted backup. Test restored stack locally first: cd ~/stack ./bin/stack up --local --port 8080 ./bin/stack status Saved images load automatically; image lock verifies identities. Matching architecture needs no registry pull/build. Check homepage, Git, search, WKD. ./bin/stack down before production setup. Same restore accepts release archives: source/binary/images, no .env/live data. Starting restored release creates fresh state; cannot recover lost data. NEW PRODUCTION HOST Restore into target user's ~/stack. Debian: review infra/host-setup.sh; replace 'ops' if needed; run: sudo sh ~/stack/infra/host-setup.sh Installs Podman/network tools, checks existing subordinate ranges, enables user linger, sets: net.ipv4.ip_unprivileged_port_start=80 System-wide rootless ports >=80; ports <80 stay privileged. SSH/firewall unchanged. Missing ranges: allocate unused ranges through distro account tools; rerun. Other distros: equivalent packages, sysctl, linger. Supplied boot unit needs systemd user manager. Host/provider firewall: allow 22/tcp, 80/tcp, 443/tcp, 443/udp for applicable IPv4/IPv6. Never publish backend container ports. Preserve SSH; verify address before changing DNS. A: new IPv4. AAAA: working new IPv6 only; remove stale AAAA. Hosts live in root .env; edit and reapply rendered settings. infra/Caddyfile is public template. Keys/render-only commands: CONFIGURATION.txt. Preserve www aliases and openpgpkey names pointing to canonical hosts. cd ~/stack ./bin/stack up --production Production binds 80/tcp, 443/tcp, 443/udp. Caddy issues/renews certificates, redirects HTTP to HTTPS and www to canonical URLs. State: var/data/caddy/. Public issuance needs working public DNS/connectivity. Enable boot startup: systemctl --user link ~/stack/infra/stack.service systemctl --user daemon-reload systemctl --user enable --now stack.service systemctl --user status stack.service Unit assumes ~/stack; edit paths for other location. Uses bin/stack. Linger keeps user service alive without login. Containers use unless-stopped restart policy. Optional former VPS SSH Git path; administrator only: sudo install -d -m 755 /srv/git sudo ln -s /home/ops/stack/var/data/repos/stack /srv/git/stack Never overwrite existing repo. Current VPS already has this layout link. Public HTTPS Git needs no host symlink. BUILD AND DEPLOY Develop/check on workstation: cd ~/stack cargo stack release Release reruns full check; builds static Rust CLI/chat + browser WASM; packages matching source; builds cgit/chat containers; saves configured images. Output: dist/stack-release-TIMESTAMP.tar.gz + .sha256. Excludes .env, live data, .git history, previous backups. Restore into fresh local test directory; up --local on free port; inspect four services; stop test. Commit/push reviewed public changes: git add -A git diff --cached --stat git commit -S3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 -m 'Describe the change' git push origin main Review staged files. .gitignore excludes secrets/runtime/artifacts. Use RSA-2048 signed commits; author/key: POLICY.txt. Deploy archive, never git pull. Copy actual filenames: ssh ops@23.155.44.67 'mkdir -p ~/stack/var/incoming' scp dist/stack-release-TIMESTAMP.tar.gz ops@23.155.44.67:stack/var/incoming/ scp dist/stack-release-TIMESTAMP.tar.gz.sha256 ops@23.155.44.67:stack/var/incoming/ On VPS: cd ~/stack/var/incoming sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256 cd ~/stack mkdir -p var/incoming/bootstrap tar -xzf var/incoming/stack-release-TIMESTAMP.tar.gz -C var/incoming/bootstrap stack/bin/stack var/incoming/bootstrap/stack/bin/stack --root "$PWD" activate var/incoming/stack-release-TIMESTAMP.tar.gz ./bin/stack status rm -rf -- var/incoming/bootstrap Use verified NEW release binary: old binary may reject new service schema. Activation verifies archive/platform, replaces public source/binary, loads exact images, preserves .env/var/data, replaces running containers. No compiler, Rust build, image build, or private-config overwrite. Expect brief interruption. Take verified export before changes needing recovery. Verify production; optionally delete corresponding incoming archive/checksum. RECOVERY LIMITS; ROTATE KEYS Application export is not bootable VPS image. Accounts, SSH, firewall, packages remain host duties. Other CPU architecture needs native release. Registry needed only without saved images or when rebuilding another target. State: files, Git repos, Caddy, encrypted SQLite chat. SQLite uses online backup. Recovery proof: reopen BOTH participants' unchanged identities; decrypt history. See CHAT-PRODUCTION.txt. Future databases need supported snapshot/recovery. New backup recipient: publish public key, update default fingerprint, test encryption/restore. Single export: --recipient FULL_FINGERPRINT --key /path/public-key.asc. Retain old secret keys while any retained export needs them. Restore accepts verified format-2 application archives. LICENSE PAGE; REDIRECTS cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves canonical https://doan.luuduct.ai/THIRD-PARTY-LICENSES.html; other deployments use SITE_HOST. Website/chat links share page; local preview uses local main-site origin. cgit Source/Raw expose generated file; chat /THIRD-PARTY-LICENSES.html redirects to canonical page. Source exports retain offline license texts. Homepage /message and /message/ redirect to current chat origin.