1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
|
Copyright (C) 2026 Doan Luu Duc Tai <doan@luuduct.ai>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
GUIX RECOVERY
Goal: one .tar.gz, unpack/bootstrap, one command restores and starts services.
Private state is GPG-encrypted inside the tarball. Bootstrap, binaries and
public release/source are plaintext. Outer tarball is not encrypted or signed.
Trust its origin; SHA256SUMS detects corruption, not malicious replacement.
Keep decryption key/passphrase separately; never send them through chat.
Guix supplies Podman/tools. Existing Caddy/cgit/SearXNG/chat OCI applications,
static musl/OpenSSL builds, domains, HTTPS/WKD and crypto remain.
No new Rust/npm dependency, Guix System install, or native service rewrite.
One root .env and public .env.example; databases/TLS/owner/history: var/data.
BUILD LOCAL
sh infra/guix/build.sh
cargo run --release --locked -- release
Needs working Guix daemon, x86_64 Linux and rootless Podman prerequisites.
Channel commits: bin/guix-channels.scm. time-machine builds the manifest's
bin/guix-runtime.tar.gz. Release runs full checks and packages matching source,
binaries, browser assets and saved images. No compilation on recovery host.
Podman info smoke test is not clean-host/container lifecycle proof.
Ordinary pack uses canonical /gnu/store. Former -RR wrapper creates one-ID
user namespace; privileged helpers cannot map host subordinate IDs there.
Builder's canonical store could bypass relocation and hide this defect.
https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pack.html
https://guix.gnu.org/manual/devel/en/html_node/Replicating-Guix.html
https://raw.githubusercontent.com/containers/podman/main/docs/tutorials/rootless_tutorial.md
MAKE CARRY FILE
Production: create consistent encrypted export; copy and verify it locally:
./bin/stack backup --include-images
Bundle a trusted freshly checked release with that export:
sh infra/guix/bundle.sh /path/backup.tar.gz.gpg /path/release.tar.gz /path/recovery.tar.gz
Bundler executes the trusted release's CLI to validate it with restore
--release. Backup-kind archives, private .env/var/data, missing runtime/chat
artifacts and missing saved images are refused before publication.
CLI, scripts, config, runtime and provenance all come from the same release.
Original release archive is unchanged and includes corresponding source and
LICENSE. Original encrypted backup retains prior source/work and rollback.
Runtime is duplicated outside release for bootstrap simplicity; extra disk
space avoids another installer/format. One carry file remains sufficient.
Old backups remain supported; release must implement this migration's
--release validation and runtime export. Blindly reusing old management binary
loses Guix files on the next backup. Restore now activates matching new source
and binary, including canonical infra/guix/run.sh needed by later backups.
BOOTSTRAP HOST; RESTORE
Fresh x86_64 Linux: trusted shell, GNU coreutils/tar/gzip/findutils/diffutils,
awk, util-linux flock, sudo/root access. Bootstrap refuses /var/guix and foreign
/gnu trees. Installs only /gnu/store entries; never host /bin or /etc links.
tar -xzf recovery.tar.gz
sudo ./stack-recovery/bootstrap
Root-owned non-writable closure entries are staged on /gnu's filesystem,
published without overwriting, then recorded by pack digest. Retry reconciles
unfinished installs; later packs retain old entries. Root install records and
lock: /gnu/.stack-runtimes. Target needs no Guix daemon/downloads. Existing
managed Guix hosts need separate planning; do not merge unmanaged store items.
Administrator also supplies privileged newuidmap/newgidmap, >=65536 subordinate
UIDs/GIDs, user namespaces, writable login XDG_RUNTIME_DIR, low-port sysctl
net.ipv4.ip_unprivileged_port_start <=80. DNS/firewall/ports remain host duties.
OPERATIONS.txt Debian setup installs distro Podman too: existing fallback,
not evidence of a Guix-only target. Service user imports GPG key locally.
./stack-recovery/restore /home/ops/stack
New destination required. Capsule checks hashes/bootstrap/prerequisites,
decrypts through Rust manifest/path validation, requires saved backup images,
then activates its paired release under the capsule's Guix runtime/config.
Activation installs new public source/binaries/images, preserves .git, .env,
var/data, and starts four services. Source worktree becomes new release;
original source remains in encrypted backup/history. Only tested version
pairs establish application/data compatibility; format-2 integrity cannot.
Failed activation retains data plus private var/guix-recovery intent marker.
Fix cause and rerun the same restore command with the same capsule/destination.
Matching backup/release digests permit resume without decrypting/restoring
twice; unrelated destinations are refused. Recovery lock excludes concurrent
activation and is closed in child processes. Success removes pending marker.
Do not replace an interrupted activation with plain up; it cannot finish
copying a release. Completed recoveries use normal management commands:
/home/ops/stack/bin/run up --production
Launcher supplies --root and Guix-only CONTAINERS_CONF; do not pass --root.
Runtime expands privately into content-addressed var/guix; old packs remain.
Activation atomically replaces the archive/provenance files, including existing
read-only packs; regression test covers replacement followed by export/restore.
Direct bin/stack and releases without pack use host tools. Guix symlinks remain
inside opaque runtime pack; Rust application extractor still rejects links.
Boot setup remains explicit: systemd user service plus linger, OPERATIONS.txt.
Unit uses bin/run at ~/stack; adapt paths for another destination.
CHECKS; DEPLOYMENT GATE
sh infra/guix/test.sh
cargo stack check
Shell tests need bubblewrap. Synthetic packs test actual bootstrap/filesystem
behavior. Synthetic recovery tests actual orchestration, failed activation,
resume, intent/lock refusal, private sentinels, and child FD closure. Fake tools
do not prove real encryption, image import, services or kernel UID mapping.
Rust tests prove archive-kind refusal before publication and runtime roundtrip.
Actual static CLI also checked synthetic bundle validation/publication/privacy.
No strict Rust/Clippy/Rustdoc, typed ESLint, LibreJS or crypto gate weakened.
Existing documented lint/dependency exceptions remain unchanged.
Clean-host test exposed missing distro container signature policy. Image load
now supplies a temporary policy: reject by default; accept local Docker/OCI
archives only. Export supplies a separate temporary policy accepting only
containers-storage; registry transport remains rejected. Verified archive
manifests and architecture/layer/image-ID locks
still gate startup. Registry transport is not accepted by this load policy;
global host policy stays unchanged. Policies are generated by AGPL Rust code;
the foreign schema rejects comment/legal metadata, so no headerless source file
is added. The initial failed VM retained decrypted data and pending intent.
A second clean-host check found legacy backups had collapsed Caddy's /data
and /config into one caddy/ tree. After manifest verification, backup restore
moves credentials into data/caddy and autosave into config/caddy; bytes and
permissions are retained. Conflicting old/new layouts are refused before
publishing a destination. New exports copy /data/. and /config/. separately.
Before production, in order:
1. Full checked release with real Guix runtime and source/binary pair.
2. Fresh host without Guix/distro Podman/Git/GPG: offline bootstrap/restore;
four services, exact images, detached UID/network helpers.
3. .env count, domains, HTTPS/WKD, original owner fingerprint; unchanged chat
rows and history decryption with both original participant identities.
4. Backup recovered stack, second recovery, reboot startup.
5. First host-Podman activation and Guix runtime upgrades. Ordinary direct
activate retains its process PATH; deploy through matching release runtime.
6. Fresh verified independent production backup; deploy only prebuilt release;
verify public routes/state, retain prior release/runtime and encrypted export.
Rollback: stop new containers before old runtime binds production ports.
Keep .env/var/data; never delete closure entries used by running processes.
VERIFIED — 2026-11-01
Offline KVM guest: official Debian 12 x86_64 image, no initial /gnu, Guix,
Podman, Git or GPG. Host supplies only documented kernel/account prerequisites.
Network restricted before bootstrap; readonly carry file plus separately
forwarded GPG agent; no private key copied. Bootstrap and one restore command
loaded four saved images and started all services without source builds.
Strict HTTPS verified all four domains using original certificates; both WKD
routes matched the original key. All 651 immutable snapshot files matched
after legacy Caddy path normalization: 18 active TLS keys, 16 certificates.
All 126 release source/binary files matched. Original .env, owner fingerprint
and every canonical SQLite table matched. Production snapshot contained zero
conversations/messages; original real-message decryption therefore untestable.
Disposable Chromium/Firefox tests verified both participant identities,
54-message history, encrypted snapshots/recovery and GNU LibreJS 7.21.1.
Full local release gates passed live npm/Rust advisory checks, strict fmt,
Clippy, Rustdoc, typed ESLint and both TypeScript compilers; no suppressions.
Required OpenSSL 3.6.5 hybrid suites, protected OpenPGP v6 identity, wrong-pass
and tamper controls passed. Synthetic shell tests cover bootstrap collisions,
interrupted recovery/retry, lock exclusion and child descriptor closure.
Runtime: 500687877 bytes, SHA256:
c5c7b849da152ee7a98b01602a1e91285ce4e3a80021a9cecb5f1e594dfc7810
Guix channel 230aa373f315f247852ee07dff34146e9b480aec; 113 store entries.
Git 2.52.0, GnuPG 2.4.7, Podman 5.7.0, conmon 2.1.13, crun 1.25.1.
Empty helper lists force PATH lookup, selecting packed conmon/crun. Ordinary
host config stays unchanged. Rootless cleanup handles immutable user-owned
temporary trees without changing symlink targets or the published pack.
https://github.com/containers/podman/blob/v5.7.0/vendor/go.podman.io/common/pkg/config/config.go#L849-L875
https://github.com/containers/podman/blob/v5.7.0/cmd/podman/images/save.go#L91-L94
Deployment gate above still applies to each carried artifact. Retain complete
original worktree, binary/launcher, installed user unit and engine storage
independently before activation. Reload the installed unit after switching to
bin/run. For rollback stop Guix containers first, retain their engine tree,
restore prior engine path, binary/launcher and unit; invoke prior activation
with host PATH and Guix CONTAINERS_CONF overrides unset. Keep current .env and
var/data. An old release alone does not remove a newer Guix launcher/runtime.
Private evidence: var/guix-*.log and .tools/guix-production/. These are ignored
and include private state hashes; do not publish them with public source.
|
