path: root/docs/OPERATIONS.txt

Copyright (C) 2026 Doan Luu Duc Tai <doan@luuduct.ai>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.

STACK OPERATIONS AND RECOVERY

    Application root: ~/stack
    Development machine: local Linux workstation
    Production host: ops@23.155.44.67
    Public source: https://git.doan.luuduct.ai/stack
    License: GNU Affero General Public License, version 3 or later

WHAT YOU NEED TO KEEP

    A full encrypted export made with --include-images contains public source,
    the compiled management command, .env, persistent application data,
    certificate state, bare repositories, working Git history if present, and
    the exact service images. It includes a file checksum manifest.

    It excludes the container engine's storage, build caches, old exports,
    host packages, host SSH keys, firewall rules, disks, and OS configuration.
    These are recreated on a new host. No old server filesystem is required.

    The public Git repository alone is not a backup: it excludes .env, live
    data, private certificates, and application repositories under var/.

    Your GPG private key and its passphrase are needed to decrypt backups.
    Keep the private-key export and revocation certificate in KeePassXC and
    keep an independent copy of the database. The server has only the public
    key. This guide never needs your private key or password in chat.

    Default backup recipient fingerprint:
        3346AA3150F8F16828BB0D14F8C66DF77AFECEA6

DAILY DEVELOPMENT

    Run these from ~/stack on your workstation:

        cargo stack dev
        cargo stack status
        cargo stack down

    Visit http://localhost:8080, http://localhost:8080/git/, and
    http://localhost:8080/search/. To use different ports:

        cargo stack dev --port 9080

    Local mode binds loopback only, rewrites public service links locally,
    keeps public DNS untouched, and does not request public TLS certificates.
    It automatically creates .env with a random SearXNG secret if absent.
    Edit .env to change configured secrets. Do not commit it.

    Public service settings belong in infra/, and website content in web/.
    New Rust backends belong in crates/, explicitly added to Cargo.toml.
    Services are declared in infra/services.toml. A service receives only the
    .env keys listed in its environment array, not every project secret.

    Rust commands use a project-specific Podman store under var/podman/.
    A bare 'podman ps' lists your normal store, which may be empty. To inspect
    this project's store manually, run from ~/stack:

        podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a

    Reuse those two global options before 'logs CONTAINER', 'exec CONTAINER',
    or 'images'. Container names are shown by cargo stack status.
    Never copy var/podman/ to another machine; export images instead.

STRICT LOCAL CHECKS

    Install the two development-only tools once:

        cargo install --locked --version 0.20.2 cargo-deny --root .tools
        cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools

    Then run:

        cargo stack check

    This checks formatting, all workspace Clippy targets/features, tests,
    private rustdoc, dependency advisories, duplicate versions, sources, and
    licenses. It regenerates THIRD-PARTY-LICENSES.html through cargo-about.
    Use cargo fmt --all when formatting needs correction.

    After changing dependency requirements, update Cargo.lock deliberately,
    run the complete check, and commit the lockfile and generated license page.
    Never disable audit failures merely to get a release out.

EXPORT AND BACK UP PRODUCTION

    On the VPS:

        cd ~/stack
        ./bin/stack backup --include-images

    The command briefly pauses the three services while copying consistent
    application files, then resumes them before compression and encryption.
    It uses the published public key, verifies the recipient fingerprint,
    checks bare Git repositories, saves exact images, and produces:

        var/exports/stack-TIMESTAMP.tar.gz.gpg
        var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256

    Both files have restrictive permissions. Temporary plaintext staging is
    removed on normal completion and handled errors. No scheduled backup job
    is installed. You decide when and where to retain exports.

    On your workstation, create a destination under stack and copy the two
    actual filenames printed by the command:

        mkdir -p ~/stack/var/backups/production
        scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg ~/stack/var/backups/production/
        scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 ~/stack/var/backups/production/
        cd ~/stack/var/backups/production
        sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256

    Replace TIMESTAMP with the actual timestamp. A checksum detects accidental
    corruption; keep the trusted encrypted file and checksum together.
    Retain at least one copy on another device or location.

    Once copied and verified, delete only the corresponding VPS export files
    if desired. Do not delete var/data or .env.

    A smaller export without images is available through ./bin/stack backup.
    It needs registry/build access for local startup and is not a standalone
    production deployment. Prefer --include-images for disaster recovery.

RESTORE ON ANOTHER LINUX MACHINE

    Install Podman, Git, GnuPG, and normal rootless networking prerequisites.
    Rootless Podman needs at least 65536 subordinate UIDs and GIDs for the
    target user. Use that distribution's supported Podman installation.
    The current ready-made binary/images target x86_64 Linux (linux/amd64).

    Copy the encrypted export and its checksum to the new machine. Verify
    the checksum first. Import your secret-key backup if not already present:

        gpg --import private-key-backup.asc

    GnuPG asks for the key passphrase through its normal local interface.
    Never pass it on the command line or put it in .env.

    If you already have a compatible stack command, restore with it:

        ./bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack-restored

    The destination must not exist. Restore decrypts privately, rejects unsafe
    archive paths and links, checks every manifest entry, assigns a unique
    project identity, and defaults to local mode. It does not start services.
    It refuses to overwrite an existing installation.

    For a new machine with no stack command or Rust toolchain, bootstrap with
    standard GnuPG and tar. The shell commands below work in bash and fish:

        umask 077
        mkdir -p ~/stack-bootstrap
        gpg --output ~/stack-bootstrap/export.tar.gz --decrypt /path/to/stack-TIMESTAMP.tar.gz.gpg
        tar -xzf ~/stack-bootstrap/export.tar.gz -C ~/stack-bootstrap stack/bin/stack
        ~/stack-bootstrap/stack/bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack

    Only use this bootstrap with your own trusted export; the extracted binary
    is executable code. The restore command validates the complete archive.
    After successful restoration, delete ~/stack-bootstrap, which contains
    the temporary decrypted archive. Keep only the encrypted export as backup.

    Start the recovered stack locally first:

        cd ~/stack
        ./bin/stack up --local --port 8080
        ./bin/stack status

    Saved images load automatically and their identities are checked against
    the image lock. No registry pull or compilation is needed on a matching
    architecture. Check the homepage, Git repositories, search, and WKD.
    Then stop it with ./bin/stack down before configuring production.

    A release archive can be restored by the same command. Releases contain
    source, a binary, and images, but no .env or application data. Starting a
    restored release creates fresh state; it does not recover lost data.

SET UP A NEW PRODUCTION HOST

    Restore to the target user's ~/stack, using the steps above. On Debian,
    adapt the username 'ops' in infra/host-setup.sh if necessary, review it,
    and run:

        sudo sh ~/stack/infra/host-setup.sh

    It installs Podman and networking tools, checks existing subordinate
    UID/GID ranges, enables user linger for boot, and sets:

        net.ipv4.ip_unprivileged_port_start=80

    This allows rootless listeners on ports 80 and above system-wide; ports
    below 80 remain privileged. It does not change SSH or your firewall.
    If the user lacks subordinate ranges, allocate unused ranges through
    the distribution's account administration before rerunning the script.

    On other Linux distributions, install equivalent prerequisites and set
    that sysctl and user linger using the host's supported tools. Production
    requires a systemd user manager for this project's supplied boot service.

    Allow inbound 22/tcp, 80/tcp, 443/tcp, and 443/udp through the host/provider
    firewall, for IPv4 and IPv6 as applicable. Backend container ports must not
    be published. Preserve your SSH access and verify the address before DNS
    changes. There is no need to restore Docker's old generated firewall rules.

    Point public A records to the new IPv4 address, and AAAA records only to a
    working new IPv6 address. Remove stale AAAA records. Public hostnames are
    listed in infra/Caddyfile. Update canonical hosts there if changing domains.
    Keep the www aliases and openpgpkey names pointing to the canonical hosts.

    Start production:

        cd ~/stack
        ./bin/stack up --production

    Production binds 80/tcp, 443/tcp, and 443/udp. Caddy manages certificates
    and renewals automatically, redirects HTTP to HTTPS, and redirects www
    aliases to the canonical URLs. Certificate state is in var/data/caddy/.
    Public certificate issuance needs working public DNS and connectivity.

    Enable startup at boot:

        systemctl --user link ~/stack/infra/stack.service
        systemctl --user daemon-reload
        systemctl --user enable --now stack.service
        systemctl --user status stack.service

    This unit assumes ~/stack. Edit its paths if deliberately using another
    location. User linger keeps it running without an interactive login.
    Application containers use an unless-stopped restart policy.

    If offering SSH Git access at the former VPS path, create this optional
    host symlink as the administrator:

        sudo install -d -m 755 /srv/git
        sudo ln -s /home/ops/stack/var/data/repos/stack /srv/git/stack

    Do not overwrite an existing repository there. The current VPS already
    has this link, updated for the new layout. Public HTTPS Git works without
    the optional host link.

MAKE AND DEPLOY A RELEASE

    Develop and run the checks on your workstation, then:

        cd ~/stack
        cargo stack release

    Release reruns the full check, builds a static Rust binary locally,
    builds the cgit container locally, and packages the configured image set.
    Outputs are dist/stack-release-TIMESTAMP.tar.gz and its .sha256 file.
    No .env, live data, .git history, or previous backup is included.

    Test the release in a fresh local directory before deploying, using the
    restore command followed by up --local on an unused port. Inspect the
    same four services and then stop the test installation.

    Commit and push reviewed public changes with normal Git:

        git add -A
        git diff --cached --stat
        git commit -m 'Describe the change'
        git push origin main

    .gitignore excludes configured secrets and all runtime/artifact trees.
    Review the staged files. Deployment uses the release archive, not git pull.

    Copy the actual archive and checksum to production:

        ssh ops@23.155.44.67 'mkdir -p ~/stack/var/incoming'
        scp dist/stack-release-TIMESTAMP.tar.gz ops@23.155.44.67:stack/var/incoming/
        scp dist/stack-release-TIMESTAMP.tar.gz.sha256 ops@23.155.44.67:stack/var/incoming/

    On the VPS:

        cd ~/stack/var/incoming
        sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256
        cd ~/stack
        ./bin/stack activate var/incoming/stack-release-TIMESTAMP.tar.gz
        ./bin/stack status

    Activation verifies the release and platform, replaces public source and
    the binary, loads exact images, preserves .env and var/data, and replaces
    the running containers. It does not compile Rust or build container images.
    Expect a brief interruption while containers are replaced. Take a verified
    export first if a future change could require recovery.

    After verifying production, delete the corresponding incoming archive and
    checksum if you do not want release copies left on the VPS.

RECOVERY LIMITS AND SECRET ROTATION

    Exports capture this application, not an entire bootable VPS image.
    Host account, SSH, firewall, and package setup remain host responsibilities.
    Changing CPU architecture requires a native release build. Registry access
    is needed only when images are omitted or rebuilding for another target.

    Current persistent state is files, Git repositories, and Caddy state.
    When adding a database, add its supported flush/dump procedure to the
    export workflow and test recovery before relying on that database's backup.

    If changing the OpenPGP backup recipient, publish its public key, update
    the command's default fingerprint, and test encryption and restoration.
    For one export, use --recipient FULL_FINGERPRINT --key /path/public-key.asc.
    Keep old secret keys while any retained export still depends on them.

    Legacy format-1 exports made before the Rust migration use their embedded
    historical Python recovery helper. Keep that helper with those exports.
    The current command intentionally accepts only its verified format-2
    archives. Fresh format-2 production backups replace the normal workflow.