diff options
Diffstat (limited to 'docs/OPERATIONS.txt')
| -rw-r--r-- | docs/OPERATIONS.txt | 354 |
1 files changed, 135 insertions, 219 deletions
diff --git a/docs/OPERATIONS.txt b/docs/OPERATIONS.txt index 2c9fa53..55dc81c 100644 --- a/docs/OPERATIONS.txt +++ b/docs/OPERATIONS.txt @@ -13,113 +13,85 @@ GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with this program. If not, see <https://www.gnu.org/licenses/>. -STACK OPERATIONS AND RECOVERY +STACK OPERATIONS - Application root: ~/stack - Development machine: local Linux workstation - Production host: ops@23.155.44.67 - Public source: https://git.doan.luuduct.ai/stack - License: GNU Affero General Public License, version 3 or later + Root: ~/stack. Develop on local Linux; production: ops@23.155.44.67. + Public source: https://git.doan.luuduct.ai/stack. + GNU Affero General Public License, version 3 or later. -WHAT YOU NEED TO KEEP +KEEP - A full encrypted export made with --include-images contains public source, - the compiled management command, .env, persistent application data, - certificate state, bare repositories, working Git history if present, and - the exact service images. It includes a file checksum manifest. + Full --include-images encrypted export: public source, compiled CLI, .env, + persistent data, certificates, bare repos, working Git history if present, + exact service images, checksum manifest. No old server filesystem needed. + Excludes engine storage, caches, old exports, host packages/SSH keys/firewall, + disks, OS configuration. Recreate host setup separately. + Public Git alone cannot recover .env, live data, TLS keys, or var/ repos. - It excludes the container engine's storage, build caches, old exports, - host packages, host SSH keys, firewall rules, disks, and OS configuration. - These are recreated on a new host. No old server filesystem is required. + Decryption needs GPG private key + passphrase. Keep key export/revocation + certificate in KeePassXC; keep independent database copy. Server holds public + key only. Never send private key/password through chat. + Default recipient: 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6. - The public Git repository alone is not a backup: it excludes .env, live - data, private certificates, and application repositories under var/. +DEVELOP LOCAL - Your GPG private key and its passphrase are needed to decrypt backups. - Keep the private-key export and revocation certificate in KeePassXC and - keep an independent copy of the database. The server has only the public - key. This guide never needs your private key or password in chat. - - Default backup recipient fingerprint: - 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 - -DAILY DEVELOPMENT - - Run these from ~/stack on your workstation: + From ~/stack on workstation: cargo stack dev cargo stack status cargo stack down - Visit http://localhost:8080, http://localhost:8080/git/, and - http://localhost:8080/search/. To use different ports: + Open http://localhost:8080, http://localhost:8080/git/, + http://localhost:8080/search/. Alternate port: cargo stack dev --port 9080 - Local mode binds loopback only, rewrites public service links locally, - keeps public DNS untouched, and does not request public TLS certificates. - It automatically creates .env with a random SearXNG secret if absent. - Edit .env to change configured secrets. Do not commit it. - - Public service settings belong in infra/, and website content in web/. - New Rust backends belong in crates/, explicitly added to Cargo.toml. - Services are declared in infra/services.toml. A service receives only the - .env keys listed in its environment array, not every project secret. + Local mode: loopback only, local links, unchanged public DNS, no public TLS. + Missing .env: create with random SearXNG secret. Edit secrets there; never commit. + infra/: service settings. web/: content. crates/: new backends; add workspace members. + infra/services.toml declares services. Each gets only listed environment keys. - Rust commands use a project-specific Podman store under var/podman/. - A bare 'podman ps' lists your normal store, which may be empty. To inspect - this project's store manually, run from ~/stack: + Project engine storage: var/podman/. Bare 'podman ps' uses another store. + Inspect project containers from ~/stack: podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a - Reuse those two global options before 'logs CONTAINER', 'exec CONTAINER', - or 'images'. Container names are shown by cargo stack status. - Never copy var/podman/ to another machine; export images instead. + Keep both global options for 'logs CONTAINER', 'exec CONTAINER', 'images'. + cargo stack status gives names. Never migrate var/podman/; export images. -STRICT LOCAL CHECKS +CHECK LOCAL - Install the two development-only tools once: + Install development audit tools once: cargo install --locked --version 0.20.2 cargo-deny --root .tools cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools - Then run: - cargo stack check - This checks formatting, all workspace Clippy targets/features, tests, - private rustdoc, dependency advisories, duplicate versions, sources, and - licenses. It regenerates THIRD-PARTY-LICENSES.html through cargo-about. - Use cargo fmt --all when formatting needs correction. - - After changing dependency requirements, update Cargo.lock deliberately, - run the complete check, and commit the lockfile and generated license page. - Never disable audit failures merely to get a release out. + Checks formatting, all workspace Clippy targets/features, tests, private + rustdoc, advisories, duplicate versions, sources, licenses. cargo-about + regenerates THIRD-PARTY-LICENSES.html. Format with cargo fmt --all. + Dependency changes: update Cargo.lock deliberately; run full check; commit + lockfile and license page. Never disable audit failures to ship. -EXPORT AND BACK UP PRODUCTION +BACK UP PRODUCTION - On the VPS: + On VPS: cd ~/stack ./bin/stack backup --include-images - The command briefly pauses Caddy, cgit, and SearXNG while copying their - application files, then resumes them before compression and encryption. - Chat remains unpaused so SQLite online backups can complete during active - writes. Consistent snapshots cover both chat and chat-local databases; raw - SQLite journals and server lock files are excluded. - It uses the published public key, verifies the recipient fingerprint, - checks bare Git repositories, saves exact images, and produces: + Briefly pauses Caddy/cgit/SearXNG during file copy; resumes before compression + and encryption. Chat stays live for SQLite online backup. Snapshots include + chat and chat-local; exclude raw journals and server locks. + Verifies public recipient fingerprint, checks bare repos, saves exact images: var/exports/stack-TIMESTAMP.tar.gz.gpg var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 - Both files have restrictive permissions. Temporary plaintext staging is - removed on normal completion and handled errors. No scheduled backup job - is installed. You decide when and where to retain exports. - - On your workstation, create a destination under stack and copy the two - actual filenames printed by the command: + Restrictive permissions. Removes temporary plaintext on success/handled errors. + No scheduled backup. Choose retention and off-host copies yourself. + On workstation, copy actual filenames printed above: umask 077 mkdir -p ~/stack/var/backups/production @@ -128,43 +100,31 @@ EXPORT AND BACK UP PRODUCTION cd ~/stack/var/backups/production sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256 - Replace TIMESTAMP with the actual timestamp. A checksum detects accidental - corruption; keep the trusted encrypted file and checksum together. - Retain at least one copy on another device or location. - - Once copied and verified, delete only the corresponding VPS export files - if desired. Do not delete var/data or .env. + Replace TIMESTAMP. Checksum detects corruption; keep trusted export + checksum. + Keep another copy on separate device/location. After verification, optionally + delete matching VPS export files only. Never delete var/data or .env. + ./bin/stack backup without images needs registry/build access for local startup; + not standalone production recovery. Prefer --include-images. - A smaller export without images is available through ./bin/stack backup. - It needs registry/build access for local startup and is not a standalone - production deployment. Prefer --include-images for disaster recovery. +RESTORE ON LINUX -RESTORE ON ANOTHER LINUX MACHINE - - Install Podman, Git, GnuPG, and normal rootless networking prerequisites. - Rootless Podman needs at least 65536 subordinate UIDs and GIDs for the - target user. Use that distribution's supported Podman installation. - The current ready-made binary/images target x86_64 Linux (linux/amd64). - - Copy the encrypted export and its checksum to the new machine. Verify - the checksum first. Import your secret-key backup if not already present: + Host tools: Podman, Git, GnuPG, rootless networking prerequisites. + Target user needs >=65536 subordinate UIDs/GIDs. Use distro-supported Podman. + Prebuilt binary/images: x86_64 Linux (linux/amd64). + Copy export + checksum; verify checksum first. If needed, import key: gpg --import private-key-backup.asc - GnuPG asks for the key passphrase through its normal local interface. - Never pass it on the command line or put it in .env. - - If you already have a compatible stack command, restore with it: + GnuPG prompts locally. Never pass passphrase through arguments or .env. + Existing compatible CLI: ./bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack-restored - The destination must not exist. Restore decrypts privately, rejects unsafe - archive paths and links, checks every manifest entry, assigns a unique - project identity, and defaults to local mode. It does not start services. - It refuses to overwrite an existing installation. + Destination must not exist. Restore decrypts privately; rejects unsafe paths + and links; checks every manifest entry; assigns unique project identity; + defaults local; starts nothing; never overwrites existing installation. - For a new machine with no stack command or Rust toolchain, bootstrap with - standard GnuPG and tar. The shell commands below work in bash and fish: + No CLI/toolchain yet: bootstrap trusted export with GnuPG/tar. Bash/fish: umask 077 mkdir -p ~/stack-bootstrap @@ -172,125 +132,98 @@ RESTORE ON ANOTHER LINUX MACHINE tar -xzf ~/stack-bootstrap/export.tar.gz -C ~/stack-bootstrap stack/bin/stack ~/stack-bootstrap/stack/bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack - Only use this bootstrap with your own trusted export; the extracted binary - is executable code. The restore command validates the complete archive. - After successful restoration, delete ~/stack-bootstrap, which contains - the temporary decrypted archive. Keep only the encrypted export as backup. - - Start the recovered stack locally first: + Bootstrap executes extracted code: use only your trusted export. + Rust restore validates full archive. After success, delete ~/stack-bootstrap: + it holds decrypted archive. Retain encrypted backup. + Test restored stack locally first: cd ~/stack ./bin/stack up --local --port 8080 ./bin/stack status - Saved images load automatically and their identities are checked against - the image lock. No registry pull or compilation is needed on a matching - architecture. Check the homepage, Git repositories, search, and WKD. - Then stop it with ./bin/stack down before configuring production. + Saved images load automatically; image lock verifies identities. Matching + architecture needs no registry pull/build. Check homepage, Git, search, WKD. + ./bin/stack down before production setup. + Same restore accepts release archives: source/binary/images, no .env/live data. + Starting restored release creates fresh state; cannot recover lost data. - A release archive can be restored by the same command. Releases contain - source, a binary, and images, but no .env or application data. Starting a - restored release creates fresh state; it does not recover lost data. +NEW PRODUCTION HOST -SET UP A NEW PRODUCTION HOST - - Restore to the target user's ~/stack, using the steps above. On Debian, - adapt the username 'ops' in infra/host-setup.sh if necessary, review it, - and run: + Restore into target user's ~/stack. Debian: review infra/host-setup.sh; + replace 'ops' if needed; run: sudo sh ~/stack/infra/host-setup.sh - It installs Podman and networking tools, checks existing subordinate - UID/GID ranges, enables user linger for boot, and sets: + Installs Podman/network tools, checks existing subordinate ranges, enables + user linger, sets: net.ipv4.ip_unprivileged_port_start=80 - This allows rootless listeners on ports 80 and above system-wide; ports - below 80 remain privileged. It does not change SSH or your firewall. - If the user lacks subordinate ranges, allocate unused ranges through - the distribution's account administration before rerunning the script. - - On other Linux distributions, install equivalent prerequisites and set - that sysctl and user linger using the host's supported tools. Production - requires a systemd user manager for this project's supplied boot service. - - Allow inbound 22/tcp, 80/tcp, 443/tcp, and 443/udp through the host/provider - firewall, for IPv4 and IPv6 as applicable. Backend container ports must not - be published. Preserve your SSH access and verify the address before DNS - changes. - - Point public A records to the new IPv4 address, and AAAA records only to a - working new IPv6 address. Remove stale AAAA records. Public hostnames are - configured in the root .env. Edit those values and reapply the service - configuration; infra/Caddyfile is now a public rendering template. - See CONFIGURATION.txt for all keys and the render-only commands. - Keep the www aliases and openpgpkey names pointing to the canonical hosts. + System-wide rootless ports >=80; ports <80 stay privileged. SSH/firewall unchanged. + Missing ranges: allocate unused ranges through distro account tools; rerun. + Other distros: equivalent packages, sysctl, linger. Supplied boot unit needs + systemd user manager. - Start production: + Host/provider firewall: allow 22/tcp, 80/tcp, 443/tcp, 443/udp for applicable + IPv4/IPv6. Never publish backend container ports. Preserve SSH; verify address + before changing DNS. A: new IPv4. AAAA: working new IPv6 only; remove stale AAAA. + Hosts live in root .env; edit and reapply rendered settings. infra/Caddyfile + is public template. Keys/render-only commands: CONFIGURATION.txt. + Preserve www aliases and openpgpkey names pointing to canonical hosts. cd ~/stack ./bin/stack up --production - Production binds 80/tcp, 443/tcp, and 443/udp. Caddy manages certificates - and renewals automatically, redirects HTTP to HTTPS, and redirects www - aliases to the canonical URLs. Certificate state is in var/data/caddy/. - Public certificate issuance needs working public DNS and connectivity. - - Enable startup at boot: + Production binds 80/tcp, 443/tcp, 443/udp. Caddy issues/renews certificates, + redirects HTTP to HTTPS and www to canonical URLs. State: var/data/caddy/. + Public issuance needs working public DNS/connectivity. + Enable boot startup: systemctl --user link ~/stack/infra/stack.service systemctl --user daemon-reload systemctl --user enable --now stack.service systemctl --user status stack.service - This unit assumes ~/stack. Edit its paths if deliberately using another - location. User linger keeps it running without an interactive login. - Application containers use an unless-stopped restart policy. + Unit assumes ~/stack; edit paths for other location. Uses bin/stack. + Linger keeps user service alive + without login. Containers use unless-stopped restart policy. - If offering SSH Git access at the former VPS path, create this optional - host symlink as the administrator: + Optional former VPS SSH Git path; administrator only: sudo install -d -m 755 /srv/git sudo ln -s /home/ops/stack/var/data/repos/stack /srv/git/stack - Do not overwrite an existing repository there. The current VPS already - has this link, updated for the new layout. Public HTTPS Git works without - the optional host link. + Never overwrite existing repo. Current VPS already has this layout link. + Public HTTPS Git needs no host symlink. -MAKE AND DEPLOY A RELEASE +BUILD AND DEPLOY - Develop and run the checks on your workstation, then: + Develop/check on workstation: cd ~/stack cargo stack release - Release reruns the full check, builds static Rust management and chat - binaries plus the browser WASM module, packages corresponding sources, - builds cgit/chat containers locally, and saves the configured image set. - Outputs are dist/stack-release-TIMESTAMP.tar.gz and its .sha256 file. - No .env, live data, .git history, or previous backup is included. - - Test the release in a fresh local directory before deploying, using the - restore command followed by up --local on an unused port. Inspect the - same four services and then stop the test installation. - - Commit and push reviewed public changes with normal Git: + Release reruns full check; builds static Rust CLI/chat + browser WASM; + packages matching source; builds cgit/chat containers; saves configured images. + Output: dist/stack-release-TIMESTAMP.tar.gz + .sha256. + Excludes .env, live data, .git history, previous backups. + Restore into fresh local test directory; up --local on free port; inspect + four services; stop test. Commit/push reviewed public changes: git add -A git diff --cached --stat - git commit -m 'Describe the change' + git commit -S3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 -m 'Describe the change' git push origin main - .gitignore excludes configured secrets and all runtime/artifact trees. - Review the staged files. Deployment uses the release archive, not git pull. - - Copy the actual archive and checksum to production: + Review staged files. .gitignore excludes secrets/runtime/artifacts. + Use RSA-2048 signed commits; author/key: POLICY.txt. + Deploy archive, never git pull. Copy actual filenames: ssh ops@23.155.44.67 'mkdir -p ~/stack/var/incoming' scp dist/stack-release-TIMESTAMP.tar.gz ops@23.155.44.67:stack/var/incoming/ scp dist/stack-release-TIMESTAMP.tar.gz.sha256 ops@23.155.44.67:stack/var/incoming/ - On the VPS: + On VPS: cd ~/stack/var/incoming sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256 @@ -301,49 +234,32 @@ MAKE AND DEPLOY A RELEASE ./bin/stack status rm -rf -- var/incoming/bootstrap - Use the management binary from the verified new release for activation. - Older binaries can reject newer service schemas. The verified release - binary handles activation without installing a compiler or overwriting - private configuration. - - Activation verifies the release and platform, replaces public source and - the binary, loads exact images, preserves .env and var/data, and replaces - the running containers. It does not compile Rust or build container images. - Expect a brief interruption while containers are replaced. Take a verified - export first if a future change could require recovery. - - After verifying production, delete the corresponding incoming archive and - checksum if you do not want release copies left on the VPS. - -RECOVERY LIMITS AND SECRET ROTATION - - Exports capture this application, not an entire bootable VPS image. - Host account, SSH, firewall, and package setup remain host responsibilities. - Changing CPU architecture requires a native release build. Registry access - is needed only when images are omitted or rebuilding for another target. - - Current persistent state includes files, Git repositories, Caddy state, - and encrypted chat records in SQLite. Chat uses the SQLite online backup - API; recovery testing must reopen both participants' unchanged identity - files and decrypt history. See CHAT-PRODUCTION.txt for this release's tests. - Future databases need their own supported snapshot and recovery procedure. - - If changing the OpenPGP backup recipient, publish its public key, update - the command's default fingerprint, and test encryption and restoration. - For one export, use --recipient FULL_FINGERPRINT --key /path/public-key.asc. - Keep old secret keys while any retained export still depends on them. - - The restore command accepts verified format-2 application archives. - -CENTRAL THIRD-PARTY LICENSE PAGE - - cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves the rendered - document at https://doan.luuduct.ai/THIRD-PARTY-LICENSES.html, using the - configured SITE_HOST for other deployments. Website and chat license links - use that same canonical page. Local previews use the local main-site origin. - - The generated file is accessible through cgit's normal Source and Raw views. - Chat's /THIRD-PARTY-LICENSES.html URL redirects to the canonical site page. - Source exports retain license texts for offline use. - - Homepage /message and /message/ URLs redirect to the current chat origin. + Use verified NEW release binary: old binary may reject new service schema. + Activation verifies archive/platform, replaces public source/binary, loads + exact images, preserves .env/var/data, replaces running containers. No compiler, + Rust build, image build, or private-config overwrite. Expect brief interruption. + Take verified export before changes needing recovery. Verify production; + optionally delete corresponding incoming archive/checksum. + +RECOVERY LIMITS; ROTATE KEYS + + Application export is not bootable VPS image. Accounts, SSH, firewall, + packages remain host duties. Other CPU architecture needs native release. + Registry needed only without saved images or when rebuilding another target. + State: files, Git repos, Caddy, encrypted SQLite chat. SQLite uses online backup. + Recovery proof: reopen BOTH participants' unchanged identities; decrypt history. + See CHAT-PRODUCTION.txt. Future databases need supported snapshot/recovery. + + New backup recipient: publish public key, update default fingerprint, test + encryption/restore. Single export: --recipient FULL_FINGERPRINT --key /path/public-key.asc. + Retain old secret keys while any retained export needs them. + Restore accepts verified format-2 application archives. + +LICENSE PAGE; REDIRECTS + + cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves canonical + https://doan.luuduct.ai/THIRD-PARTY-LICENSES.html; other deployments use SITE_HOST. + Website/chat links share page; local preview uses local main-site origin. + cgit Source/Raw expose generated file; chat /THIRD-PARTY-LICENSES.html redirects + to canonical page. Source exports retain offline license texts. + Homepage /message and /message/ redirect to current chat origin. |