1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
|
Copyright (C) 2026 Doan Luu Duc Tai <doan@luuduct.ai>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
STACK OPERATIONS
Root: ~/stack. Workstation: local Linux. Production: ops@23.155.44.67.
Public source: https://git.doan.luuduct.ai/stack.
Production commands use bin/run, which selects packed Guix tools/config.
Direct bin/stack uses host tools; never mix engines against the same storage.
KEEP
For new-host recovery keep the carry file described in GUIX.txt: encrypted
application export plus matching checked release, Guix runtime and bootstrap.
Private state is encrypted; public tools/source are plaintext. Keep its trusted
checksum and an independent copy on another device/location.
It preserves one .env, databases, TLS keys, bare repos, working Git history,
binaries, browser assets, sources and exact service images. Engine storage,
caches, host accounts/SSH/firewall, disks and OS configuration are excluded.
Public Git alone cannot recover private state.
Keep GPG secret key/passphrase separately; default backup recipient:
3346AA3150F8F16828BB0D14F8C66DF77AFECEA6
Keep an independent protected key export/revocation certificate. Chat owner
and visitor private identity files/passwords also need separate backups.
Never send private keys or passwords through chat.
DEVELOP LOCAL
cargo stack dev
cargo stack status
cargo stack down
Open http://localhost:8080, /git/, /search/, http://chat.localhost:8080/.
Alternate port: cargo stack dev --port 9080. Local mode binds loopback only;
no public TLS or DNS changes. Missing .env is initialized with random secrets.
infra/services.toml lists each service's environment, mounts and health check.
Local development uses host Podman. Project storage is var/podman; bare
podman ps uses another store. From the local checkout:
podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a
Keep both global options for logs/exec/images. Use bin/run status on production.
Never transfer var/podman as backup; export saved images instead.
CHECK AND BUILD LOCAL
cargo install --locked --version 0.20.2 cargo-deny --root .tools
cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools
cargo stack check
cargo run --release --locked -- release
Check covers TypeScript compilers/typed ESLint/npm audit, Guix shell fixtures,
formatting, strict workspace Clippy/tests/private Rustdoc, advisories, exact
duplicate exceptions and licenses. cargo-about regenerates the license page.
Dependency edits require deliberate lockfile review, full checks and notices.
Never waive failures. Format with cargo fmt --all.
Build the Guix pack first when adding/changing runtime tools: GUIX.txt.
Release builds static Rust/chat WASM, corresponding source, cgit/chat images
and complete image export. Output: dist/stack-release-TIMESTAMP.tar.gz and
.sha256. Actual .env, live data, .git and prior backups are excluded.
Build/test locally; no compiler or registry pull is needed on the matching
x86_64 production host. Restore/inspect a disposable release before deployment.
BACK UP PRODUCTION
cd ~/stack
./bin/run backup --include-images
Briefly pauses Caddy/cgit/SearXNG during file copy; resumes before compression
and encryption. Chat stays live for SQLite online backup. Export includes
chat/chat-local snapshots; excludes raw journals and server locks.
Verifies recipient fingerprint, bare repos and exact image identities.
Output has restrictive permissions:
var/exports/stack-TIMESTAMP.tar.gz.gpg
var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256
Copy both files off the VPS; substitute actual filenames:
umask 077
mkdir -p ~/stack/var/backups/production
scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg ~/stack/var/backups/production/
scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 ~/stack/var/backups/production/
cd ~/stack/var/backups/production
sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256
No scheduled backup. Choose retention/off-host copies; remove matching VPS
exports only after verification. Never delete .env or var/data.
Export without --include-images is not standalone production recovery.
Pair the verified export with a checked release using infra/guix/bundle.sh;
the resulting carry file restores with the matching manager/source/runtime.
RECOVER ON LINUX
Follow GUIX.txt for host prerequisites and GPG key access. On a fresh x86_64
Linux host, as the service user, with a new destination:
tar -xzf recovery.tar.gz
sudo ./stack-recovery/bootstrap
./stack-recovery/restore /home/ops/stack
Capsule verifies/decrypts the export, activates its paired release and starts
four services. A failed activation can resume with the same restore command.
Keep hostname/origin and identity files unchanged; repoint DNS when IP changes.
Recovery does not configure host SSH, firewall, account or boot startup.
A raw format-2 export can be restored for inspection on a host that already
has compatible tools and any required Guix closure:
./bin/run restore /path/export.tar.gz.gpg /path/stack-restored
Raw restore validates all manifest paths/hashes, defaults local mode with a
fresh project identity, starts nothing and refuses an existing destination.
Public release archives restore similarly but contain no private data.
An old raw export may lack the launcher/runtime; use a paired capsule for
complete recovery. Test owner/visitor identities and history after recovery.
HOST AND BOOT SETUP
Host/provider firewall: retain SSH and allow HTTP 80/tcp, HTTPS 443/tcp+udp
for applicable IPv4/IPv6. Never publish backend container ports. A points to
the new IPv4; publish AAAA only with working IPv6. Keep www/WKD aliases.
Rootless IDs, mapping helpers, runtime directory and low-port sysctl: GUIX.txt.
Debian host-tools fallback: infra/host-setup.sh installs distro Podman and
prerequisites. Guix supplies its own Podman/network tools.
cd ~/stack
./bin/run up --production
systemctl --user link ~/stack/infra/stack.service
systemctl --user daemon-reload
systemctl --user enable --now stack.service
systemctl --user status stack.service
Administrator enables linger for the service user. Unit assumes ~/stack;
adapt paths for another location. Caddy stores credentials in var/data/caddy,
issues/renews public certificates and redirects HTTP/www. ACME needs public
DNS/connectivity. Containers use unless-stopped restart policy.
Current VPS SSH Git compatibility path /srv/git/stack links to
/home/ops/stack/var/data/repos/stack. Preserve it for existing clone URLs;
public HTTPS Git needs no host symlink.
DEPLOY A CHECKED RELEASE
Review/stage public changes, sign and push the current branch:
git diff --cached --stat
git commit -S3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 -m 'Describe the change'
git push origin HEAD
Do not deploy with git pull. Take a verified independent export first; retain
complete old worktree/binary/unit, var/state.json, image-lock.json, images.tar
and prior engine storage for rollback. Guix upgrade/rollback rules: GUIX.txt.
Copy the checked public release and checksum to ~/stack/var/incoming.
Before activation install any new closure through its verified capsule
bootstrap; an existing managed Guix store needs separate planning.
Bash on VPS, after that bootstrap; substitute actual TIMESTAMP:
(
set -eu
cd ~/stack
STACK_ROOT=$PWD
STACK_RELEASE=$PWD/var/incoming/stack-release-TIMESTAMP.tar.gz
(cd var/incoming && sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256)
STACK_STAGE=$(mktemp -d "$PWD/var/incoming/.activate.XXXXXX")
tar -xzf "$STACK_RELEASE" -C "$STACK_STAGE" stack/bin/stack stack/bin/guix-runtime.tar.gz stack/infra/guix/containers.conf
mkdir "$STACK_STAGE/runtime"
tar -xzf "$STACK_STAGE/stack/bin/guix-runtime.tar.gz" --no-same-owner -C "$STACK_STAGE/runtime"
env PATH="$STACK_STAGE/runtime/bin:$PATH" CONTAINERS_CONF="$STACK_STAGE/stack/infra/guix/containers.conf" "$STACK_STAGE/stack/bin/stack" --root "$STACK_ROOT" activate "$STACK_RELEASE"
systemctl --user daemon-reload
systemctl --user restart stack.service
./bin/run status
chmod -R u+w -- "$STACK_STAGE"
rm -rf -- "$STACK_STAGE"
)
Use the NEW manager and matching runtime. Activation verifies archive/platform,
replaces public source/binaries/images, preserves .env/var/data/.git and
replaces containers. Expect interruption. The installed unit may already link
to infra/stack.service; reload its definition rather than copying over itself.
If activation fails, retain staging/data and diagnose before retry or rollback.
Verify public routes, TLS/WKD, owner and database before removing incoming files.
RECOVERY LIMITS; KEY ROTATION
Other CPU architectures need native release builds. Host DNS/PTR, firewall,
accounts and SSH remain outside the carry file. New databases need consistent
snapshot/recovery support before inclusion. Preserve both participants' chat
files/passwords; server backups cannot replace them. CHAT-PRODUCTION.txt.
For a new backup recipient, publish its public key, update root .env and test
encryption/recovery. One export can select --recipient FULL_FINGERPRINT and
--key /path/public-key.asc. Keep old secret keys while retained exports need them.
LICENSE PAGE; REDIRECTS
cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves it on the
main-site origin; chat redirects its former local license URL there. Source
exports retain offline license texts. cgit Source/Raw expose the generated file.
Homepage /message and /message/ redirect to the configured chat origin.
|
