1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
|
Copyright (C) 2026 Doan Luu Duc Tai <doan@luuduct.ai>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
STACK OPERATIONS
Root: ~/stack. Develop on local Linux; production: ops@23.155.44.67.
Public source: https://git.doan.luuduct.ai/stack.
GNU Affero General Public License, version 3 or later.
KEEP
Full --include-images encrypted export: public source, compiled CLI, .env,
persistent data, certificates, bare repos, working Git history if present,
exact service images, checksum manifest. No old server filesystem needed.
Excludes engine storage, caches, old exports, host packages/SSH keys/firewall,
disks, OS configuration. Recreate host setup separately.
Public Git alone cannot recover .env, live data, TLS keys, or var/ repos.
Decryption needs GPG private key + passphrase. Keep key export/revocation
certificate in KeePassXC; keep independent database copy. Server holds public
key only. Never send private key/password through chat.
Default recipient: 3346AA3150F8F16828BB0D14F8C66DF77AFECEA6.
DEVELOP LOCAL
From ~/stack on workstation:
cargo stack dev
cargo stack status
cargo stack down
Open http://localhost:8080, http://localhost:8080/git/,
http://localhost:8080/search/. Alternate port:
cargo stack dev --port 9080
Local mode: loopback only, local links, unchanged public DNS, no public TLS.
Missing .env: create with random SearXNG secret. Edit secrets there; never commit.
infra/: service settings. web/: content. crates/: new backends; add workspace members.
infra/services.toml declares services. Each gets only listed environment keys.
Project engine storage: var/podman/. Bare 'podman ps' uses another store.
Inspect project containers from ~/stack:
podman --root "$PWD/var/podman/storage" --runroot "$PWD/var/podman/run" ps -a
Keep both global options for 'logs CONTAINER', 'exec CONTAINER', 'images'.
cargo stack status gives names. Never migrate var/podman/; export images.
CHECK LOCAL
Install development audit tools once:
cargo install --locked --version 0.20.2 cargo-deny --root .tools
cargo install --locked --version 0.9.3 --features cli cargo-about --root .tools
cargo stack check
Checks formatting, all workspace Clippy targets/features, tests, private
rustdoc, advisories, duplicate versions, sources, licenses. cargo-about
regenerates THIRD-PARTY-LICENSES.html. Format with cargo fmt --all.
Dependency changes: update Cargo.lock deliberately; run full check; commit
lockfile and license page. Never disable audit failures to ship.
BACK UP PRODUCTION
On VPS:
cd ~/stack
./bin/stack backup --include-images
Briefly pauses Caddy/cgit/SearXNG during file copy; resumes before compression
and encryption. Chat stays live for SQLite online backup. Snapshots include
chat and chat-local; exclude raw journals and server locks.
Verifies public recipient fingerprint, checks bare repos, saves exact images:
var/exports/stack-TIMESTAMP.tar.gz.gpg
var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256
Restrictive permissions. Removes temporary plaintext on success/handled errors.
No scheduled backup. Choose retention and off-host copies yourself.
On workstation, copy actual filenames printed above:
umask 077
mkdir -p ~/stack/var/backups/production
scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg ~/stack/var/backups/production/
scp ops@23.155.44.67:stack/var/exports/stack-TIMESTAMP.tar.gz.gpg.sha256 ~/stack/var/backups/production/
cd ~/stack/var/backups/production
sha256sum -c stack-TIMESTAMP.tar.gz.gpg.sha256
Replace TIMESTAMP. Checksum detects corruption; keep trusted export + checksum.
Keep another copy on separate device/location. After verification, optionally
delete matching VPS export files only. Never delete var/data or .env.
./bin/stack backup without images needs registry/build access for local startup;
not standalone production recovery. Prefer --include-images.
RESTORE ON LINUX
Host tools: Podman, Git, GnuPG, rootless networking prerequisites.
Target user needs >=65536 subordinate UIDs/GIDs. Use distro-supported Podman.
Prebuilt binary/images: x86_64 Linux (linux/amd64).
Copy export + checksum; verify checksum first. If needed, import key:
gpg --import private-key-backup.asc
GnuPG prompts locally. Never pass passphrase through arguments or .env.
Existing compatible CLI:
./bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack-restored
Destination must not exist. Restore decrypts privately; rejects unsafe paths
and links; checks every manifest entry; assigns unique project identity;
defaults local; starts nothing; never overwrites existing installation.
No CLI/toolchain yet: bootstrap trusted export with GnuPG/tar. Bash/fish:
umask 077
mkdir -p ~/stack-bootstrap
gpg --output ~/stack-bootstrap/export.tar.gz --decrypt /path/to/stack-TIMESTAMP.tar.gz.gpg
tar -xzf ~/stack-bootstrap/export.tar.gz -C ~/stack-bootstrap stack/bin/stack
~/stack-bootstrap/stack/bin/stack restore /path/to/stack-TIMESTAMP.tar.gz.gpg ~/stack
Bootstrap executes extracted code: use only your trusted export.
Rust restore validates full archive. After success, delete ~/stack-bootstrap:
it holds decrypted archive. Retain encrypted backup.
Test restored stack locally first:
cd ~/stack
./bin/stack up --local --port 8080
./bin/stack status
Saved images load automatically; image lock verifies identities. Matching
architecture needs no registry pull/build. Check homepage, Git, search, WKD.
./bin/stack down before production setup.
Same restore accepts release archives: source/binary/images, no .env/live data.
Starting restored release creates fresh state; cannot recover lost data.
NEW PRODUCTION HOST
Restore into target user's ~/stack. Debian: review infra/host-setup.sh;
replace 'ops' if needed; run:
sudo sh ~/stack/infra/host-setup.sh
Installs Podman/network tools, checks existing subordinate ranges, enables
user linger, sets:
net.ipv4.ip_unprivileged_port_start=80
System-wide rootless ports >=80; ports <80 stay privileged. SSH/firewall unchanged.
Missing ranges: allocate unused ranges through distro account tools; rerun.
Other distros: equivalent packages, sysctl, linger. Supplied boot unit needs
systemd user manager.
Host/provider firewall: allow 22/tcp, 80/tcp, 443/tcp, 443/udp for applicable
IPv4/IPv6. Never publish backend container ports. Preserve SSH; verify address
before changing DNS. A: new IPv4. AAAA: working new IPv6 only; remove stale AAAA.
Hosts live in root .env; edit and reapply rendered settings. infra/Caddyfile
is public template. Keys/render-only commands: CONFIGURATION.txt.
Preserve www aliases and openpgpkey names pointing to canonical hosts.
cd ~/stack
./bin/stack up --production
Production binds 80/tcp, 443/tcp, 443/udp. Caddy issues/renews certificates,
redirects HTTP to HTTPS and www to canonical URLs. State: var/data/caddy/.
Public issuance needs working public DNS/connectivity.
Enable boot startup:
systemctl --user link ~/stack/infra/stack.service
systemctl --user daemon-reload
systemctl --user enable --now stack.service
systemctl --user status stack.service
Unit assumes ~/stack; edit paths for other location. Uses bin/stack.
Linger keeps user service alive
without login. Containers use unless-stopped restart policy.
Optional former VPS SSH Git path; administrator only:
sudo install -d -m 755 /srv/git
sudo ln -s /home/ops/stack/var/data/repos/stack /srv/git/stack
Never overwrite existing repo. Current VPS already has this layout link.
Public HTTPS Git needs no host symlink.
BUILD AND DEPLOY
Develop/check on workstation:
cd ~/stack
cargo stack release
Release reruns full check; builds static Rust CLI/chat + browser WASM;
packages matching source; builds cgit/chat containers; saves configured images.
Output: dist/stack-release-TIMESTAMP.tar.gz + .sha256.
Excludes .env, live data, .git history, previous backups.
Restore into fresh local test directory; up --local on free port; inspect
four services; stop test. Commit/push reviewed public changes:
git add -A
git diff --cached --stat
git commit -S3346AA3150F8F16828BB0D14F8C66DF77AFECEA6 -m 'Describe the change'
git push origin main
Review staged files. .gitignore excludes secrets/runtime/artifacts.
Use RSA-2048 signed commits; author/key: POLICY.txt.
Deploy archive, never git pull. Copy actual filenames:
ssh ops@23.155.44.67 'mkdir -p ~/stack/var/incoming'
scp dist/stack-release-TIMESTAMP.tar.gz ops@23.155.44.67:stack/var/incoming/
scp dist/stack-release-TIMESTAMP.tar.gz.sha256 ops@23.155.44.67:stack/var/incoming/
On VPS:
cd ~/stack/var/incoming
sha256sum -c stack-release-TIMESTAMP.tar.gz.sha256
cd ~/stack
mkdir -p var/incoming/bootstrap
tar -xzf var/incoming/stack-release-TIMESTAMP.tar.gz -C var/incoming/bootstrap stack/bin/stack
var/incoming/bootstrap/stack/bin/stack --root "$PWD" activate var/incoming/stack-release-TIMESTAMP.tar.gz
./bin/stack status
rm -rf -- var/incoming/bootstrap
Use verified NEW release binary: old binary may reject new service schema.
Activation verifies archive/platform, replaces public source/binary, loads
exact images, preserves .env/var/data, replaces running containers. No compiler,
Rust build, image build, or private-config overwrite. Expect brief interruption.
Take verified export before changes needing recovery. Verify production;
optionally delete corresponding incoming archive/checksum.
RECOVERY LIMITS; ROTATE KEYS
Application export is not bootable VPS image. Accounts, SSH, firewall,
packages remain host duties. Other CPU architecture needs native release.
Registry needed only without saved images or when rebuilding another target.
State: files, Git repos, Caddy, encrypted SQLite chat. SQLite uses online backup.
Recovery proof: reopen BOTH participants' unchanged identities; decrypt history.
See CHAT-PRODUCTION.txt. Future databases need supported snapshot/recovery.
New backup recipient: publish public key, update default fingerprint, test
encryption/restore. Single export: --recipient FULL_FINGERPRINT --key /path/public-key.asc.
Retain old secret keys while any retained export needs them.
Restore accepts verified format-2 application archives.
LICENSE PAGE; REDIRECTS
cargo-about generates web/THIRD-PARTY-LICENSES.html. Caddy serves canonical
https://doan.luuduct.ai/THIRD-PARTY-LICENSES.html; other deployments use SITE_HOST.
Website/chat links share page; local preview uses local main-site origin.
cgit Source/Raw expose generated file; chat /THIRD-PARTY-LICENSES.html redirects
to canonical page. Source exports retain offline license texts.
Homepage /message and /message/ redirect to current chat origin.
|
