summaryrefslogtreecommitdiff
path: root/docs/OPERATIONS.txt
diff options
context:
space:
mode:
authorDoan Luu Duc Tai <doan@luuduct.ai>2026-11-01 08:25:58 -0600
committerDoan Luu Duc Tai <doan@luuduct.ai>2026-11-01 08:25:58 -0600
commit607be5ee239a9b7969eab96145c8af4ee57053d1 (patch)
tree582ada863d690ed8f113f266f2afc8252d421c39 /docs/OPERATIONS.txt
parent9349a2de648e241da8e00ba7ae25cb2a877ff3e9 (diff)
chore: remove obsolete migration code and docsHEADmain
Diffstat (limited to 'docs/OPERATIONS.txt')
-rw-r--r--docs/OPERATIONS.txt312
1 files changed, 131 insertions, 181 deletions
diff --git a/docs/OPERATIONS.txt b/docs/OPERATIONS.txt
index 91ab10b..bf675be 100644
--- a/docs/OPERATIONS.txt
+++ b/docs/OPERATIONS.txt
@@ -15,84 +15,82 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
STACK OPERATIONS
- Root: ~/stack. Develop on local Linux; production: ops@23.155.44.67.
+ Root: ~/stack. Workstation: local Linux. Production: ops@23.155.44.67.
Public source: https://git.doan.luuduct.ai/stack.
- GNU Affero General Public License, version 3 or later.
- Experimental Guix capsule: GUIX.txt. Commands below use host Podman.
+ Production commands use bin/run, which selects packed Guix tools/config.
+ Direct bin/stack uses host tools; never mix engines against the same storage.
KEEP
- Full --include-images encrypted export: public source, compiled CLI, .env,
- persistent data, certificates, bare repos, working Git history if present,
- exact service images, checksum manifest. No old server filesystem needed.
- Excludes engine storage, caches, old exports, host packages/SSH keys/firewall,
- disks, OS configuration. Recreate host setup separately.
- Public Git alone cannot recover .env, live data, TLS keys, or var/ repos.
-
- Decryption needs GPG private key + passphrase. Keep key export/revocation
- certificate in KeePassXC; keep independent database copy. Server holds public
- key only. Never send private key/password through chat.
- Default recipient: 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6.
+ For new-host recovery keep the carry file described in GUIX.txt: encrypted
+ application export plus matching checked release, Guix runtime and bootstrap.
+ Private state is encrypted; public tools/source are plaintext. Keep its trusted
+ checksum and an independent copy on another device/location.
+ It preserves one .env, databases, TLS keys, bare repos, working Git history,
+ binaries, browser assets, sources and exact service images. Engine storage,
+ caches, host accounts/SSH/firewall, disks and OS configuration are excluded.
+ Public Git alone cannot recover private state.
+
+ Keep GPG secret key/passphrase separately; default backup recipient:
+ 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6
+ Keep an independent protected key export/revocation certificate. Chat owner
+ and visitor private identity files/passwords also need separate backups.
+ Never send private keys or passwords through chat.
DEVELOP LOCAL
- From ~/stack on workstation:
-
cargo stack dev
cargo stack status
cargo stack down
- Open http://localhost:8080, http://localhost:8080/git/,
- http://localhost:8080/search/. Alternate port:
-
- cargo stack dev --port 9080
-
- Local mode: loopback only, local links, unchanged public DNS, no public TLS.
- Missing .env: create with random SearXNG secret. Edit secrets there; never commit.
- infra/: service settings. web/: content. crates/: new backends; add workspace members.
- infra/services.toml declares services. Each gets only listed environment keys.
+ Open http://localhost:8080, /git/, /search/, http://chat.localhost:8080/.
+ Alternate port: cargo stack dev --port 9080. Local mode binds loopback only;
+ no public TLS or DNS changes. Missing .env is initialized with random secrets.
+ infra/services.toml lists each service's environment, mounts and health check.
- Project engine storage: var/podman/. Bare 'podman ps' uses another store.
- Inspect project containers from ~/stack:
+ Local development uses host Podman. Project storage is var/podman; bare
+ podman ps uses another store. From the local checkout:
podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a
- Keep both global options for 'logs CONTAINER', 'exec CONTAINER', 'images'.
- cargo stack status gives names. Never migrate var/podman/; export images.
+ Keep both global options for logs/exec/images. Use bin/run status on production.
+ Never transfer var/podman as backup; export saved images instead.
-CHECK LOCAL
-
- Install development audit tools once:
+CHECK AND BUILD LOCAL
cargo install --locked --version 0.20.2 cargo-deny --root .tools
cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools
-
cargo stack check
+ cargo run --release --locked -- release
- Checks Guix shell/bootstrap fixtures (bubblewrap), formatting, workspace Clippy, tests, private
- rustdoc, advisories, duplicate versions, sources, licenses. cargo-about
- regenerates THIRD-PARTY-LICENSES.html. Format with cargo fmt --all.
- Dependency changes: update Cargo.lock deliberately; run full check; commit
- lockfile and license page. Never disable audit failures to ship.
+ Check covers TypeScript compilers/typed ESLint/npm audit, Guix shell fixtures,
+ formatting, strict workspace Clippy/tests/private Rustdoc, advisories, exact
+ duplicate exceptions and licenses. cargo-about regenerates the license page.
+ Dependency edits require deliberate lockfile review, full checks and notices.
+ Never waive failures. Format with cargo fmt --all.
-BACK UP PRODUCTION
+ Build the Guix pack first when adding/changing runtime tools: GUIX.txt.
+ Release builds static Rust/chat WASM, corresponding source, cgit/chat images
+ and complete image export. Output: dist/stack-release-TIMESTAMP.tar.gz and
+ .sha256. Actual .env, live data, .git and prior backups are excluded.
+ Build/test locally; no compiler or registry pull is needed on the matching
+ x86_64 production host. Restore/inspect a disposable release before deployment.
- On VPS:
+BACK UP PRODUCTION
cd ~/stack
- ./bin/stack backup --include-images
+ ./bin/run backup --include-images
Briefly pauses Caddy/cgit/SearXNG during file copy; resumes before compression
- and encryption. Chat stays live for SQLite online backup. Snapshots include
- chat and chat-local; exclude raw journals and server locks.
- Verifies public recipient fingerprint, checks bare repos, saves exact images:
+ and encryption. Chat stays live for SQLite online backup. Export includes
+ chat/chat-local snapshots; excludes raw journals and server locks.
+ Verifies recipient fingerprint, bare repos and exact image identities.
+ Output has restrictive permissions:
var/exports/stack-TIMESTAMP.tar.gz.gpg
var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256
- Restrictive permissions. Removes temporary plaintext on success/handled errors.
- No scheduled backup. Choose retention and off-host copies yourself.
- On workstation, copy actual filenames printed above:
+ Copy both files off the VPS; substitute actual filenames:
umask 077
mkdir -p ~/stack/var/backups/production
@@ -101,166 +99,118 @@ BACK UP PRODUCTION
cd ~/stack/var/backups/production
sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256
- Replace TIMESTAMP. Checksum detects corruption; keep trusted export + checksum.
- Keep another copy on separate device/location. After verification, optionally
- delete matching VPS export files only. Never delete var/data or .env.
- ./bin/stack backup without images needs registry/build access for local startup;
- not standalone production recovery. Prefer --include-images.
-
-RESTORE ON LINUX
-
- Host tools: Podman, Git, GnuPG, rootless networking prerequisites.
- Target user needs >=65536 subordinate UIDs/GIDs. Use distro-supported Podman.
- Prebuilt binary/images: x86_64 Linux (linux/amd64).
- Copy export + checksum; verify checksum first. If needed, import key:
-
- gpg --import private-key-backup.asc
-
- GnuPG prompts locally. Never pass passphrase through arguments or .env.
- Existing compatible CLI:
-
- ./bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack-restored
-
- Destination must not exist. Restore decrypts privately; rejects unsafe paths
- and links; checks every manifest entry; assigns unique project identity;
- defaults local; starts nothing; never overwrites existing installation.
+ No scheduled backup. Choose retention/off-host copies; remove matching VPS
+ exports only after verification. Never delete .env or var/data.
+ Export without --include-images is not standalone production recovery.
+ Pair the verified export with a checked release using infra/guix/bundle.sh;
+ the resulting carry file restores with the matching manager/source/runtime.
- No CLI/toolchain yet: bootstrap trusted export with GnuPG/tar. Bash/fish:
+RECOVER ON LINUX
- umask 077
- mkdir -p ~/stack-bootstrap
- gpg --output ~/stack-bootstrap/export.tar.gz --decrypt /path/to/stack-TIMESTAMP.tar.gz.gpg
- tar -xzf ~/stack-bootstrap/export.tar.gz -C ~/stack-bootstrap stack/bin/stack
- ~/stack-bootstrap/stack/bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack
-
- Bootstrap executes extracted code: use only your trusted export.
- Rust restore validates full archive. After success, delete ~/stack-bootstrap:
- it holds decrypted archive. Retain encrypted backup.
- Test restored stack locally first:
-
- cd ~/stack
- ./bin/stack up --local --port 8080
- ./bin/stack status
+ Follow GUIX.txt for host prerequisites and GPG key access. On a fresh x86_64
+ Linux host, as the service user, with a new destination:
- Saved images load automatically; image lock verifies identities. Matching
- architecture needs no registry pull/build. Check homepage, Git, search, WKD.
- ./bin/stack down before production setup.
- Same restore accepts release archives: source/binary/images, no .env/live data.
- Starting restored release creates fresh state; cannot recover lost data.
+ tar -xzf recovery.tar.gz
+ sudo ./stack-recovery/bootstrap
+ ./stack-recovery/restore /home/ops/stack
-NEW PRODUCTION HOST
+ Capsule verifies/decrypts the export, activates its paired release and starts
+ four services. A failed activation can resume with the same restore command.
+ Keep hostname/origin and identity files unchanged; repoint DNS when IP changes.
+ Recovery does not configure host SSH, firewall, account or boot startup.
- Restore into target user's ~/stack. Debian: review infra/host-setup.sh;
- replace 'ops' if needed; run:
+ A raw format-2 export can be restored for inspection on a host that already
+ has compatible tools and any required Guix closure:
- sudo sh ~/stack/infra/host-setup.sh
+ ./bin/run restore /path/export.tar.gz.gpg /path/stack-restored
- Installs Podman/network tools, checks existing subordinate ranges, enables
- user linger, sets:
+ Raw restore validates all manifest paths/hashes, defaults local mode with a
+ fresh project identity, starts nothing and refuses an existing destination.
+ Public release archives restore similarly but contain no private data.
+ An old raw export may lack the launcher/runtime; use a paired capsule for
+ complete recovery. Test owner/visitor identities and history after recovery.
- net.ipv4.ip_unprivileged_port_start=80
+HOST AND BOOT SETUP
- System-wide rootless ports >=80; ports <80 stay privileged. SSH/firewall unchanged.
- Missing ranges: allocate unused ranges through distro account tools; rerun.
- Other distros: equivalent packages, sysctl, linger. Supplied boot unit needs
- systemd user manager.
-
- Host/provider firewall: allow 22/tcp, 80/tcp, 443/tcp, 443/udp for applicable
- IPv4/IPv6. Never publish backend container ports. Preserve SSH; verify address
- before changing DNS. A: new IPv4. AAAA: working new IPv6 only; remove stale AAAA.
- Hosts live in root .env; edit and reapply rendered settings. infra/Caddyfile
- is public template. Keys/render-only commands: CONFIGURATION.txt.
- Preserve www aliases and openpgpkey names pointing to canonical hosts.
+ Host/provider firewall: retain SSH and allow HTTP 80/tcp, HTTPS 443/tcp+udp
+ for applicable IPv4/IPv6. Never publish backend container ports. A points to
+ the new IPv4; publish AAAA only with working IPv6. Keep www/WKD aliases.
+ Rootless IDs, mapping helpers, runtime directory and low-port sysctl: GUIX.txt.
+ Debian host-tools fallback: infra/host-setup.sh installs distro Podman and
+ prerequisites. Guix supplies its own Podman/network tools.
cd ~/stack
- ./bin/stack up --production
-
- Production binds 80/tcp, 443/tcp, 443/udp. Caddy issues/renews certificates,
- redirects HTTP to HTTPS and www to canonical URLs. State: var/data/caddy/.
- Public issuance needs working public DNS/connectivity.
- Enable boot startup:
-
+ ./bin/run up --production
systemctl --user link ~/stack/infra/stack.service
systemctl --user daemon-reload
systemctl --user enable --now stack.service
systemctl --user status stack.service
- Unit assumes ~/stack; edit paths for other location. Current unit uses bin/run:
- confirm launcher exists before enabling. Linger keeps user service alive
- without login. Containers use unless-stopped restart policy.
+ Administrator enables linger for the service user. Unit assumes ~/stack;
+ adapt paths for another location. Caddy stores credentials in var/data/caddy,
+ issues/renews public certificates and redirects HTTP/www. ACME needs public
+ DNS/connectivity. Containers use unless-stopped restart policy.
- Optional former VPS SSH Git path; administrator only:
+ Current VPS SSH Git compatibility path /srv/git/stack links to
+ /home/ops/stack/var/data/repos/stack. Preserve it for existing clone URLs;
+ public HTTPS Git needs no host symlink.
- sudo install -d -m 755 /srv/git
- sudo ln -s /home/ops/stack/var/data/repos/stack /srv/git/stack
+DEPLOY A CHECKED RELEASE
- Never overwrite existing repo. Current VPS already has this layout link.
- Public HTTPS Git needs no host symlink.
+ Review/stage public changes, sign and push the current branch:
-BUILD AND DEPLOY
-
- Develop/check on workstation:
-
- cd ~/stack
- cargo stack release
-
- Release reruns full check; builds static Rust CLI/chat + browser WASM;
- packages matching source; builds cgit/chat containers; saves configured images.
- Output: dist/stack-release-TIMESTAMP.tar.gz + .sha256.
- Excludes .env, live data, .git history, previous backups.
- Restore into fresh local test directory; up --local on free port; inspect
- four services; stop test. Commit/push reviewed public changes:
-
- git add -A
git diff --cached --stat
git commit -S3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 -m 'Describe the change'
- git push origin main
-
- Review staged files. .gitignore excludes secrets/runtime/artifacts.
- Use RSA-2048 signed commits; author/key: POLICY.txt.
- Deploy archive, never git pull. Copy actual filenames:
+ git push origin HEAD
- ssh ops@23.155.44.67 'mkdir -p ~/stack/var/incoming'
- scp dist/stack-release-TIMESTAMP.tar.gz ops@23.155.44.67:stack/var/incoming/
- scp dist/stack-release-TIMESTAMP.tar.gz.sha256 ops@23.155.44.67:stack/var/incoming/
+ Do not deploy with git pull. Take a verified independent export first; retain
+ complete old worktree/binary/unit, var/state.json, image-lock.json, images.tar
+ and prior engine storage for rollback. Guix upgrade/rollback rules: GUIX.txt.
+ Copy the checked public release and checksum to ~/stack/var/incoming.
+ Before activation install any new closure through its verified capsule
+ bootstrap; an existing managed Guix store needs separate planning.
- On VPS:
+ Bash on VPS, after that bootstrap; substitute actual TIMESTAMP:
- cd ~/stack/var/incoming
- sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256
+ (
+ set -eu
cd ~/stack
- mkdir -p var/incoming/bootstrap
- tar -xzf var/incoming/stack-release-TIMESTAMP.tar.gz -C var/incoming/bootstrap stack/bin/stack
- var/incoming/bootstrap/stack/bin/stack --root "$PWD" activate var/incoming/stack-release-TIMESTAMP.tar.gz
- ./bin/stack status
- rm -rf -- var/incoming/bootstrap
-
- Use verified NEW release binary: old binary may reject new service schema.
- Activation verifies archive/platform, replaces public source/binary, loads
- exact images, preserves .env/var/data, replaces running containers. No compiler,
- Rust build, image build, or private-config overwrite. Expect brief interruption.
- Take verified export before changes needing recovery. Verify production;
- optionally delete corresponding incoming archive/checksum.
-
-RECOVERY LIMITS; ROTATE KEYS
-
- Application export is not bootable VPS image. Accounts, SSH, firewall,
- packages remain host duties. Other CPU architecture needs native release.
- Registry needed only without saved images or when rebuilding another target.
- State: files, Git repos, Caddy, encrypted SQLite chat. SQLite uses online backup.
- Recovery proof: reopen BOTH participants' unchanged identities; decrypt history.
- See CHAT-PRODUCTION.txt. Future databases need supported snapshot/recovery.
-
- New backup recipient: publish public key, update default fingerprint, test
- encryption/restore. Single export: --recipient FULL_FINGERPRINT --key /path/public-key.asc.
- Retain old secret keys while any retained export needs them.
- Restore accepts verified format-2 application archives.
+ STACK_ROOT=$PWD
+ STACK_RELEASE=$PWD/var/incoming/stack-release-TIMESTAMP.tar.gz
+ (cd var/incoming && sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256)
+ STACK_STAGE=$(mktemp -d "$PWD/var/incoming/.activate.XXXXXX")
+ tar -xzf "$STACK_RELEASE" -C "$STACK_STAGE" stack/bin/stack stack/bin/guix-runtime.tar.gz stack/infra/guix/containers.conf
+ mkdir "$STACK_STAGE/runtime"
+ tar -xzf "$STACK_STAGE/stack/bin/guix-runtime.tar.gz" --no-same-owner -C "$STACK_STAGE/runtime"
+ env PATH="$STACK_STAGE/runtime/bin:$PATH" CONTAINERS_CONF="$STACK_STAGE/stack/infra/guix/containers.conf" "$STACK_STAGE/stack/bin/stack" --root "$STACK_ROOT" activate "$STACK_RELEASE"
+ systemctl --user daemon-reload
+ systemctl --user restart stack.service
+ ./bin/run status
+ chmod -R u+w -- "$STACK_STAGE"
+ rm -rf -- "$STACK_STAGE"
+ )
+
+ Use the NEW manager and matching runtime. Activation verifies archive/platform,
+ replaces public source/binaries/images, preserves .env/var/data/.git and
+ replaces containers. Expect interruption. The installed unit may already link
+ to infra/stack.service; reload its definition rather than copying over itself.
+ If activation fails, retain staging/data and diagnose before retry or rollback.
+ Verify public routes, TLS/WKD, owner and database before removing incoming files.
+
+RECOVERY LIMITS; KEY ROTATION
+
+ Other CPU architectures need native release builds. Host DNS/PTR, firewall,
+ accounts and SSH remain outside the carry file. New databases need consistent
+ snapshot/recovery support before inclusion. Preserve both participants' chat
+ files/passwords; server backups cannot replace them. CHAT-PRODUCTION.txt.
+
+ For a new backup recipient, publish its public key, update root .env and test
+ encryption/recovery. One export can select --recipient FULL_FINGERPRINT and
+ --key /path/public-key.asc. Keep old secret keys while retained exports need them.
LICENSE PAGE; REDIRECTS
- cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves canonical
- https://doan.luuduct.ai/THIRD-PARTY-LICENSES.html; other deployments use SITE_HOST.
- Website/chat links share page; local preview uses local main-site origin.
- cgit Source/Raw expose generated file; chat /THIRD-PARTY-LICENSES.html redirects
- to canonical page. Source exports retain offline license texts.
- Homepage /message and /message/ redirect to current chat origin.
+ cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves it on the
+ main-site origin; chat redirects its former local license URL there. Source
+ exports retain offline license texts. cgit Source/Raw expose the generated file.
+ Homepage /message and /message/ redirect to the configured chat origin.