diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/OPERATIONS.txt | 322 | ||||
| -rw-r--r-- | docs/POLICY.txt | 74 |
2 files changed, 396 insertions, 0 deletions
diff --git a/docs/OPERATIONS.txt b/docs/OPERATIONS.txt new file mode 100644 index 0000000..6cd9a6b --- /dev/null +++ b/docs/OPERATIONS.txt @@ -0,0 +1,322 @@ +Copyright (C) 2026 Doan Luu Duc Tai <doan@luuduct.ai> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. + +STACK OPERATIONS AND RECOVERY + + Application root: ~/stack + Development machine: local Linux workstation + Production host: ops@23.155.44.67 + Public source: https://git.doan.luuduct.ai/stack + License: GNU Affero General Public License, version 3 or later + +WHAT YOU NEED TO KEEP + + A full encrypted export made with --include-images contains public source, + the compiled management command, .env, persistent application data, + certificate state, bare repositories, working Git history if present, and + the exact service images. It includes a file checksum manifest. + + It excludes the container engine's storage, build caches, old exports, + host packages, host SSH keys, firewall rules, disks, and OS configuration. + These are recreated on a new host. No old server filesystem is required. + + The public Git repository alone is not a backup: it excludes .env, live + data, private certificates, and application repositories under var/. + + Your GPG private key and its passphrase are needed to decrypt backups. + Keep the private-key export and revocation certificate in KeePassXC and + keep an independent copy of the database. The server has only the public + key. This guide never needs your private key or password in chat. + + Default backup recipient fingerprint: + 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 + +DAILY DEVELOPMENT + + Run these from ~/stack on your workstation: + + cargo stack dev + cargo stack status + cargo stack down + + Visit http://localhost:8080, http://localhost:8080/git/, and + http://localhost:8080/search/. To use different ports: + + cargo stack dev --port 9080 + + Local mode binds loopback only, rewrites public service links locally, + keeps public DNS untouched, and does not request public TLS certificates. + It automatically creates .env with a random SearXNG secret if absent. + Edit .env to change configured secrets. Do not commit it. + + Public service settings belong in infra/, and website content in web/. + New Rust backends belong in crates/, explicitly added to Cargo.toml. + Services are declared in infra/services.toml. A service receives only the + .env keys listed in its environment array, not every project secret. + + Rust commands use a project-specific Podman store under var/podman/. + A bare 'podman ps' lists your normal store, which may be empty. To inspect + this project's store manually, run from ~/stack: + + podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a + + Reuse those two global options before 'logs CONTAINER', 'exec CONTAINER', + or 'images'. Container names are shown by cargo stack status. + Never copy var/podman/ to another machine; export images instead. + +STRICT LOCAL CHECKS + + Install the two development-only tools once: + + cargo install --locked --version 0.20.2 cargo-deny --root .tools + cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools + + Then run: + + cargo stack check + + This checks formatting, all workspace Clippy targets/features, tests, + private rustdoc, dependency advisories, duplicate versions, sources, and + licenses. It regenerates THIRD-PARTY-LICENSES.html through cargo-about. + Use cargo fmt --all when formatting needs correction. + + After changing dependency requirements, update Cargo.lock deliberately, + run the complete check, and commit the lockfile and generated license page. + Never disable audit failures merely to get a release out. + +EXPORT AND BACK UP PRODUCTION + + On the VPS: + + cd ~/stack + ./bin/stack backup --include-images + + The command briefly pauses the three services while copying consistent + application files, then resumes them before compression and encryption. + It uses the published public key, verifies the recipient fingerprint, + checks bare Git repositories, saves exact images, and produces: + + var/exports/stack-TIMESTAMP.tar.gz.gpg + var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 + + Both files have restrictive permissions. Temporary plaintext staging is + removed on normal completion and handled errors. No scheduled backup job + is installed. You decide when and where to retain exports. + + On your workstation, create a destination under stack and copy the two + actual filenames printed by the command: + + mkdir -p ~/stack/var/backups/production + scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg ~/stack/var/backups/production/ + scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 ~/stack/var/backups/production/ + cd ~/stack/var/backups/production + sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256 + + Replace TIMESTAMP with the actual timestamp. A checksum detects accidental + corruption; keep the trusted encrypted file and checksum together. + Retain at least one copy on another device or location. + + Once copied and verified, delete only the corresponding VPS export files + if desired. Do not delete var/data or .env. + + A smaller export without images is available through ./bin/stack backup. + It needs registry/build access for local startup and is not a standalone + production deployment. Prefer --include-images for disaster recovery. + +RESTORE ON ANOTHER LINUX MACHINE + + Install Podman, Git, GnuPG, and normal rootless networking prerequisites. + Rootless Podman needs at least 65536 subordinate UIDs and GIDs for the + target user. Use that distribution's supported Podman installation. + The current ready-made binary/images target x86_64 Linux (linux/amd64). + + Copy the encrypted export and its checksum to the new machine. Verify + the checksum first. Import your secret-key backup if not already present: + + gpg --import private-key-backup.asc + + GnuPG asks for the key passphrase through its normal local interface. + Never pass it on the command line or put it in .env. + + If you already have a compatible stack command, restore with it: + + ./bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack-restored + + The destination must not exist. Restore decrypts privately, rejects unsafe + archive paths and links, checks every manifest entry, assigns a unique + project identity, and defaults to local mode. It does not start services. + It refuses to overwrite an existing installation. + + For a new machine with no stack command or Rust toolchain, bootstrap with + standard GnuPG and tar. The shell commands below work in bash and fish: + + umask 077 + mkdir -p ~/stack-bootstrap + gpg --output ~/stack-bootstrap/export.tar.gz --decrypt /path/to/stack-TIMESTAMP.tar.gz.gpg + tar -xzf ~/stack-bootstrap/export.tar.gz -C ~/stack-bootstrap stack/bin/stack + ~/stack-bootstrap/stack/bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack + + Only use this bootstrap with your own trusted export; the extracted binary + is executable code. The restore command validates the complete archive. + After successful restoration, delete ~/stack-bootstrap, which contains + the temporary decrypted archive. Keep only the encrypted export as backup. + + Start the recovered stack locally first: + + cd ~/stack + ./bin/stack up --local --port 8080 + ./bin/stack status + + Saved images load automatically and their identities are checked against + the image lock. No registry pull or compilation is needed on a matching + architecture. Check the homepage, Git repositories, search, and WKD. + Then stop it with ./bin/stack down before configuring production. + + A release archive can be restored by the same command. Releases contain + source, a binary, and images, but no .env or application data. Starting a + restored release creates fresh state; it does not recover lost data. + +SET UP A NEW PRODUCTION HOST + + Restore to the target user's ~/stack, using the steps above. On Debian, + adapt the username 'ops' in infra/host-setup.sh if necessary, review it, + and run: + + sudo sh ~/stack/infra/host-setup.sh + + It installs Podman and networking tools, checks existing subordinate + UID/GID ranges, enables user linger for boot, and sets: + + net.ipv4.ip_unprivileged_port_start=80 + + This allows rootless listeners on ports 80 and above system-wide; ports + below 80 remain privileged. It does not change SSH or your firewall. + If the user lacks subordinate ranges, allocate unused ranges through + the distribution's account administration before rerunning the script. + + On other Linux distributions, install equivalent prerequisites and set + that sysctl and user linger using the host's supported tools. Production + requires a systemd user manager for this project's supplied boot service. + + Allow inbound 22/tcp, 80/tcp, 443/tcp, and 443/udp through the host/provider + firewall, for IPv4 and IPv6 as applicable. Backend container ports must not + be published. Preserve your SSH access and verify the address before DNS + changes. There is no need to restore Docker's old generated firewall rules. + + Point public A records to the new IPv4 address, and AAAA records only to a + working new IPv6 address. Remove stale AAAA records. Public hostnames are + listed in infra/Caddyfile. Update canonical hosts there if changing domains. + Keep the www aliases and openpgpkey names pointing to the canonical hosts. + + Start production: + + cd ~/stack + ./bin/stack up --production + + Production binds 80/tcp, 443/tcp, and 443/udp. Caddy manages certificates + and renewals automatically, redirects HTTP to HTTPS, and redirects www + aliases to the canonical URLs. Certificate state is in var/data/caddy/. + Public certificate issuance needs working public DNS and connectivity. + + Enable startup at boot: + + systemctl --user link ~/stack/infra/stack.service + systemctl --user daemon-reload + systemctl --user enable --now stack.service + systemctl --user status stack.service + + This unit assumes ~/stack. Edit its paths if deliberately using another + location. User linger keeps it running without an interactive login. + Application containers use an unless-stopped restart policy. + + If offering SSH Git access at the former VPS path, create this optional + host symlink as the administrator: + + sudo install -d -m 755 /srv/git + sudo ln -s /home/ops/stack/var/data/repos/stack /srv/git/stack + + Do not overwrite an existing repository there. The current VPS already + has this link, updated for the new layout. Public HTTPS Git works without + the optional host link. + +MAKE AND DEPLOY A RELEASE + + Develop and run the checks on your workstation, then: + + cd ~/stack + cargo stack release + + Release reruns the full check, builds a static Rust binary locally, + builds the cgit container locally, and packages the configured image set. + Outputs are dist/stack-release-TIMESTAMP.tar.gz and its .sha256 file. + No .env, live data, .git history, or previous backup is included. + + Test the release in a fresh local directory before deploying, using the + restore command followed by up --local on an unused port. Inspect the + same four services and then stop the test installation. + + Commit and push reviewed public changes with normal Git: + + git add -A + git diff --cached --stat + git commit -m 'Describe the change' + git push origin main + + .gitignore excludes configured secrets and all runtime/artifact trees. + Review the staged files. Deployment uses the release archive, not git pull. + + Copy the actual archive and checksum to production: + + ssh ops@23.155.44.67 'mkdir -p ~/stack/var/incoming' + scp dist/stack-release-TIMESTAMP.tar.gz ops@23.155.44.67:stack/var/incoming/ + scp dist/stack-release-TIMESTAMP.tar.gz.sha256 ops@23.155.44.67:stack/var/incoming/ + + On the VPS: + + cd ~/stack/var/incoming + sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256 + cd ~/stack + ./bin/stack activate var/incoming/stack-release-TIMESTAMP.tar.gz + ./bin/stack status + + Activation verifies the release and platform, replaces public source and + the binary, loads exact images, preserves .env and var/data, and replaces + the running containers. It does not compile Rust or build container images. + Expect a brief interruption while containers are replaced. Take a verified + export first if a future change could require recovery. + + After verifying production, delete the corresponding incoming archive and + checksum if you do not want release copies left on the VPS. + +RECOVERY LIMITS AND SECRET ROTATION + + Exports capture this application, not an entire bootable VPS image. + Host account, SSH, firewall, and package setup remain host responsibilities. + Changing CPU architecture requires a native release build. Registry access + is needed only when images are omitted or rebuilding for another target. + + Current persistent state is files, Git repositories, and Caddy state. + When adding a database, add its supported flush/dump procedure to the + export workflow and test recovery before relying on that database's backup. + + If changing the OpenPGP backup recipient, publish its public key, update + the command's default fingerprint, and test encryption and restoration. + For one export, use --recipient FULL_FINGERPRINT --key /path/public-key.asc. + Keep old secret keys while any retained export still depends on them. + + Legacy format-1 exports made before the Rust migration use their embedded + historical Python recovery helper. Keep that helper with those exports. + The current command intentionally accepts only its verified format-2 + archives. Fresh format-2 production backups replace the normal workflow. diff --git a/docs/POLICY.txt b/docs/POLICY.txt new file mode 100644 index 0000000..f9e3614 --- /dev/null +++ b/docs/POLICY.txt @@ -0,0 +1,74 @@ +Copyright (C) 2026 Doan Luu Duc Tai <doan@luuduct.ai> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as +published by the Free Software Foundation, either version 3 of the +License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. + +PROJECT CONVENTIONS + + License: GNU Affero General Public License, version 3 or later. + SPDX identifier: AGPL-3.0-or-later. + Copyright holder: Doan Luu Duc Tai <doan@luuduct.ai>, 2026. + + Every nontrivial first-party source, configuration, template, and prose + file carries the AGPL copyright and permission header. LICENSE is the + unmodified license text. Binary public keys and empty WKD policy files + retain their protocol format. Generated Cargo.lock retains Cargo's own + metadata as well as the project notice. + + Do not add explanatory source comments. Express intent through structure + and names. Legal headers are the required exception. Use .txt for prose + documentation. The generated HTML license page is an explicit exception. + Rust documentation attributes provide crate metadata without comments. + + Keep configured secrets only in the root .env. Never commit .env or var/. + .env.example lists required keys with placeholders. Public hostnames, + public GPG keys, paths, configuration, and source belong in Git. + TLS account/certificate private keys, databases, and other generated + private state are persistent data in var/data, not .env configuration. + The GPG secret key and revocation certificate remain outside this project. + +STRICTNESS + + Preserve the requested Rust and Clippy deny groups, including pedantic + and nursery. The one additional exception is redundant_pub_crate: its + advice to make internal binary items public conflicts with unreachable_pub, + which remains denied. Internal interfaces remain explicitly crate-private. + No unsafe code, unwrap, expect, todo, debug output, arithmetic shortcuts, + or broad warning suppression is introduced. + + cargo-deny rejects duplicate crate versions, wildcard requirements, unknown + registries, unknown Git sources, and yanked versions. There are no duplicate + version skip exceptions. Its AGPL allowance is scoped to this package. + The dependency allowlist retains the requested permissive licenses and + Zlib; an unused allowance may produce an informational audit warning. + + Human license names follow https://www.gnu.org/licenses/license-list.html. + In particular, display Expat License for SPDX MIT. Do not alter original + copyright notices, legal texts, crate metadata, or machine SPDX identifiers. + cargo-about covers Rust dependencies. Supporting container applications + and their operating-system packages carry additional upstream licenses. + +PORTABILITY + + All first-party project files and state live below stack/. Standard host + integration consists of packages, subordinate UID/GID allocations, one + low-port sysctl setting, user linger, and a systemd user-service link. + Host SSH keys, firewall policy, accounts, and disks are host administration, + not application export contents. + + Releases currently contain a static x86_64 Linux command and native + linux/amd64 container images. They are portable across compatible x86_64 + Linux machines. An ARM machine needs a native release build; the command + rejects mismatched production images. Development mode may fetch/build + native images on an architecture mismatch. Container source and Rust + source are public so new targets can be added without restoring an OS. |