Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

...
Author SHA1 Message Date
atlas
c6a778f666 lint: tighten atomic-write-secret.nix's header comment; fix vale contractions in persistence.md
The header comment grew to 39 lines across two audit-driven rounds,
over the 30-line comment-block-lint max. Trimmed to 30: merged the
value-as-argument rationale with its /proc/cmdline justification into
one paragraph, cut the usage example to one call instead of two, and
condensed the args paragraph — no content dropped, just restatement.

persistence.md's first-boot-migration marker paragraph used "it is"
and "there is not" — vale's Microsoft.Contractions rule (this repo's
config) wants the contracted forms, and "is not" also collides with
"is nothing" as a literal substring, which is what actually tripped
the error. Reworded to "it's" / "there's nothing", no meaning change.

Lint-only: no script logic changed, gates re-run below are all lint
checks (no module-eval, no cargo).

Refs #4723
2026-09-26 21:50:03 +02:00
atlas
770f68c272 host-modules: atomic_write_secret takes the value as an argument, not stdin
The pipe contract had a gap: if a producer piped into
atomic_write_secret exited non-zero after writing partial output,
cat still saw a clean EOF and wrote that partial content through to
the live target via mv — pipefail only reported the failure
afterward, once the bad write was already committed. The helper now
takes the value as its 4th argument and writes it itself with printf
(a shell builtin, so the value never touches an external process's
own argv/environ, same as a function argument never does), so there
is no pipe left to fail silently.

Callers that compute the value with a command now capture it into a
variable first (`value=$(cmd)`), which fails under `set -e` before
atomic_write_secret is ever called — swarm-bao.nix's pin.env site is
the one that needed this (`pin_env_value="BAO_HSM_PIN=$(cat ...)"`).
All seven call sites converted; output is byte-identical (same
printf '%s\n' framing, now applied inside the helper instead of by
each caller).

Refs #4723
2026-09-26 21:50:03 +02:00
atlas
7a9fadc21a host-modules: convert swarm-bao's HSM-PIN env writer to atomic_write_secret
swarm-bao-token's pin.env write (the BAO_HSM_PIN EnvironmentFile for
openbao's pkcs11 seal) had the same write-then-chmod-on-live-path
shape as the sites already converted: printf > path directly on the
live file, chmod after. Same fix, same helper. Content
("BAO_HSM_PIN=<user-pin>\n") and final mode (0400, root-owned — no
chown, same as before) are unchanged; pin.env stays at the same path,
so openbao's EnvironmentFile= reference needs no change.

Refs #4723
2026-09-26 21:50:03 +02:00
atlas
5466cec066 host-modules: fix atomic_write_secret's leftover-tmp bug; convert swarm-bao's forwarder-oidc writer too
atomic_write_secret's cleanup trap used RETURN, which never fires when
set -e aborts the function mid-body (a failing cat/chmod/chown), so a
secret-bearing temp file was left behind instead of being removed.
The write now runs in a subshell with its own EXIT trap, invoked via a
named handler (so `local rc=$?` is a normal, shellcheck-visible
assignment) that only removes the temp file when the subshell's exit
status is nonzero — the subshell's trap table is private, so a calling
unit's own EXIT trap is untouched. Reproduced the leftover-tmp bug
against the prior commit, confirmed it's gone, and confirmed both the
success path and a caller's own EXIT trap still work as before.

swarm-bao.nix's swarm-bao-forwarder-oidc unit had the identical
write-then-chmod-on-live-path defect as the four sites already fixed
here (fetches an OIDC client secret from swarm-bao, printfs it to the
live host path, chowns/chmods after) and was missed by the original
sweep. Converted it to atomic_write_secret; content and final
owner/mode (root:root, 0400) are unchanged.

Refs #4723
2026-09-26 21:50:03 +02:00
atlas
ed53e9abcc host-modules: write credential files atomically; agent-modules: retry a failed .claude migration
Four host glue units fetched a secret from swarm-bao and rendered it
with `> path; chmod`: a reader racing the write could see a truncated
file, and briefly one at the wrong mode before the chmod landed.
glue-matrix-bao-token.nix, glue-queue-agent-credential.nix (both
files), swarm-grafana.nix and swarm-otel.nix now write to a same-
directory temp file, set its final mode/owner, then `mv -f` it over
the target — a shared `atomic_write_secret` helper
(nix/host-modules/lib/atomic-write-secret.nix) so the five call sites
share one implementation.

The first-boot `/root/.claude` migration in nix/agent-modules/user.nix
wrote its done-marker unconditionally, so a failed `cp` (disk full,
permission error) left the marker behind and no boot ever retried the
copy. The marker is now written only when there was nothing to
migrate or the copy succeeded; `cp -an`'s no-clobber semantics already
make a retry after a partial copy safe.

Refs #4723
2026-09-26 21:50:03 +02:00
8 changed files with 120 additions and 31 deletions

View file

@ -535,9 +535,12 @@ container lifetime:
2. **Migrate any leftover `/root/.claude` content into
`${homeDir}/.claude`** — legacy `claude` wrote to root's
empty home; the bind mount didn't exist yet. Marker
(`/var/lib/hive-agent-user-migrated`) guards single-shot.
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
only written once there's nothing left to migrate, so a `cp`
failure leaves it absent and the next boot retries.
`cp -an` (no-clobber) so any pre-existing files at the new
location win — never blow over data already there.
location win — never blow over data already there, and so a
retry after a partial copy is as safe as the first attempt.
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
recursively so the agent user can read/write it. Wildcard
matches the single agent that container sees; `-h` skips

View file

@ -172,16 +172,29 @@ in
userName=${lib.escapeShellArg userName}
mkdir -p "$homeDir"
chown "$userName:$userName" "$homeDir"
# The marker is only written once nothing is left to migrate — either
# there was nothing under /root/.claude, or `cp` copied it all. A
# failed `cp` (disk full, permission error) leaves the marker absent,
# so the next boot's activation retries; `-an` never clobbers a file
# this attempt already placed, so a retry after a partial copy is
# exactly as safe as the first attempt.
marker=/var/lib/hive-agent-user-migrated
if [ ! -e "$marker" ] && [ -d /root/.claude ] && [ "$(ls -A /root/.claude 2>/dev/null)" ]; then
mkdir -p "$homeDir/.claude"
if cp -an /root/.claude/. "$homeDir/.claude/" 2>/dev/null; then
rm -rf /root/.claude
echo "hive-agent-user-migrate: moved /root/.claude → $homeDir/.claude"
if [ ! -e "$marker" ]; then
if [ -d /root/.claude ] && [ "$(ls -A /root/.claude 2>/dev/null)" ]; then
mkdir -p "$homeDir/.claude"
if cp -an /root/.claude/. "$homeDir/.claude/" 2>/dev/null; then
rm -rf /root/.claude
echo "hive-agent-user-migrate: moved /root/.claude → $homeDir/.claude"
mkdir -p "$(dirname "$marker")"
: > "$marker"
else
echo "hive-agent-user-migrate: copying /root/.claude to $homeDir/.claude failed; will retry next boot" >&2
fi
else
mkdir -p "$(dirname "$marker")"
: > "$marker"
fi
fi
mkdir -p "$(dirname "$marker")"
: > "$marker"
# Scope state + harness chowns to THIS container's own dirs only.
# The glob `/agents/*/state` also matches other agents' state dirs
# bind-mounted into a `ManageRootAgent` holder's container, which

View file

@ -80,6 +80,8 @@ let
# resolution, so this is the kind of mistake only reading the target module
# catches.
matrixMachine = "hive-matrix";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
in
{
config = lib.mkMerge [
@ -187,6 +189,8 @@ in
script = ''
set -euo pipefail
${atomicWriteSecret}
# A sealed or uninitialised store answers on the port and never
# answers the read, so "the store is up" is not the same as "the
# store can answer". `TimeoutStartSec` above is the bound; the
@ -237,9 +241,7 @@ in
exit 0
fi
umask 077
printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
atomic_write_secret 0600 "" ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} "$token"
# Re-stamp the registration file from the token just written. The
# token is half an agreement — the registration the homeserver loads

View file

@ -77,6 +77,8 @@ let
# its own use of it: it is asserted set wherever
# `deploy.hive-controller.enable` is.
credentialPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/queue/agent";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
in
{
options.services.hyperhive.deploy.hive-controller.queue = {
@ -217,6 +219,8 @@ in
script = ''
set -euo pipefail
${atomicWriteSecret}
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
@ -279,14 +283,11 @@ in
install -d -m 0755 ${lib.escapeShellArg credentialDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile}
chmod 0600 ${lib.escapeShellArg secretFile}
atomic_write_secret 0600 "" ${lib.escapeShellArg secretFile} "$secret"
# `0644` on purpose: an OIDC client id is presented to the token
# endpoint on every connection and is public by construction.
printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile}
chmod 0644 ${lib.escapeShellArg clientIdFile}
atomic_write_secret 0644 "" ${lib.escapeShellArg clientIdFile} "$client_id"
'';
};
})

View file

@ -0,0 +1,60 @@
# Shared shell step for a systemd oneshot that lands a freshly-fetched
# secret on disk: never `> path` the live file directly, because a reader
# racing the write can open it between the truncate and the write, or
# between the write and a `chmod` that follows it, and see an empty file
# or one at the wrong mode. `mktemp` always creates its file at 0600
# regardless of umask, so the temp file is private for its whole life; only
# the `chmod`/`chown`/`mv -f` sequence below ever makes the target mode and
# owner visible, and only once the content is already final.
#
# The value is a function ARGUMENT, not piped in on stdin: a producer that
# exits non-zero partway through a pipe still leaves `cat` a clean EOF, so
# a truncated read is indistinguishable from an intentionally short one,
# and the partial content would still land at the live path via `mv`. A
# caller that computes the value with a command captures it into a
# variable first (`value=$(cmd)`, which fails under `set -e` before this
# function is ever called) and passes the finished value in — never handed
# to an external program, so it never appears in a process's own
# `/proc/<pid>/cmdline`, and written with `printf`, a shell builtin.
#
# Pure function — NOT a NixOS module. Call it from a module's `let`:
#
# atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
# script = ''
# ${atomicWriteSecret}
# atomic_write_secret 0600 "" "$path" "$secret"
# '';
#
# Args: mode, an owner for `chown` ("" to keep mktemp's root:root), target
# path, value — written with a trailing newline like every call site's
# prior `printf '%s\n'`. Requires `coreutils` on the caller's `path`.
{ }:
''
atomic_write_secret() {
local mode="$1" owner="$2" target="$3" value="$4" tmp
tmp="$(mktemp "$(dirname -- "$target")/.$(basename -- "$target").XXXXXX")"
# The write happens in a subshell so its own EXIT trap cleans up `$tmp`
# on failure (a `RETURN` trap does not fire when `set -e` aborts the
# function mid-body) without touching an EXIT trap the calling script's
# own `script` may already have — subshell traps are local to the
# subshell. A named handler, not an inline trap string, so `local rc=$?`
# is a normal function-local assignment shellcheck can see: it only
# removes `$tmp` when `$?` is nonzero — on the ordinary path the
# subshell also exits successfully, and `$tmp` has to survive that to
# reach the `mv` below.
(
_atomic_write_secret_cleanup() {
local rc=$?
[ "$rc" -eq 0 ] || rm -f "$tmp"
exit "$rc"
}
trap _atomic_write_secret_cleanup EXIT
printf '%s\n' "$value" > "$tmp"
chmod "$mode" "$tmp"
if [ -n "$owner" ]; then
chown "$owner" "$tmp"
fi
)
mv -f "$tmp" "$target"
}
''

View file

@ -764,6 +764,8 @@ let
forwarderSecretInContainer = "/var/lib/swarm-bao-otel-oidc/${cfg.otel.clientId}.secret";
forwarderHostSecretPath = "/var/lib/nixos-containers/${cfg.machine}${forwarderSecretInContainer}";
forwarderHostSecretDir = builtins.dirOf forwarderHostSecretPath;
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
# The collector runs under `DynamicUser` and opens `client_secret_file`
# itself, so there is no uid to hand a 0400 file to. `LoadCredential` reads
# it as root before the sandbox exists and re-exposes it under a path that
@ -1797,6 +1799,8 @@ in
script = ''
set -euo pipefail
${atomicWriteSecret}
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable store, and this unit
# retries on all three — so it reports which one rather than
@ -1837,10 +1841,7 @@ in
# an argument in /proc the way `install <<<"$secret"` or an `echo`
# from `path` would.
install -d -m 0755 ${lib.escapeShellArg forwarderHostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg forwarderHostSecretPath}
chown root:root ${lib.escapeShellArg forwarderHostSecretPath}
chmod 0400 ${lib.escapeShellArg forwarderHostSecretPath}
atomic_write_secret 0400 root:root ${lib.escapeShellArg forwarderHostSecretPath} "$secret"
'';
};
})
@ -2594,6 +2595,9 @@ in
};
script = ''
set -euo pipefail
${atomicWriteSecret}
install -d -m 0770 -g ${tokenGroup} ${tokenStoreDir}
# Required whenever the store is not at its default location, or the
# library cannot find the token the seal asks for.
@ -2643,8 +2647,12 @@ in
chgrp ${tokenGroup} ${tokenStoreDir}/tpm2_pkcs11.sqlite3
chmod 0660 ${tokenStoreDir}/tpm2_pkcs11.sqlite3
( umask 077; printf 'BAO_HSM_PIN=%s\n' "$(cat ${tokenStoreDir}/user-pin)" > ${pinEnvFile} )
chmod 0400 ${pinEnvFile}
# Captured first, not piped: a `$(cmd)` that fails aborts here
# under `set -e`, before the helper — which cannot see a
# producer's exit code once its output is inside the pipe —
# is ever called.
pin_env_value="BAO_HSM_PIN=$(cat ${tokenStoreDir}/user-pin)"
atomic_write_secret 0400 "" ${pinEnvFile} "$pin_env_value"
'';
};

View file

@ -168,6 +168,8 @@ let
nginxGid = config.ids.gids.nginx;
grafanaUid = config.ids.uids.grafana;
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
in
{
# `enable` moved to `services.hyperhive.deploy.grafana.enable` — see
@ -633,6 +635,8 @@ in
script = ''
set -euo pipefail
${atomicWriteSecret}
# `bao`'s own message is the only thing separating a missing value from
# a refused identity from an unreachable host. This unit's degraded
# mode is correct for all three, so it reports which one rather than
@ -689,10 +693,7 @@ in
# grants the group nothing; if this mode ever widens, the gid has to be
# discovered at runtime rather than assumed.
install -d -m 0755 ${lib.escapeShellArg hostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg hostSecretPath}
chown ${toString config.ids.uids.grafana}:0 ${lib.escapeShellArg hostSecretPath}
chmod 0400 ${lib.escapeShellArg hostSecretPath}
atomic_write_secret 0400 ${lib.escapeShellArg "${toString config.ids.uids.grafana}:0"} ${lib.escapeShellArg hostSecretPath} "$secret"
'';
};

View file

@ -166,6 +166,8 @@ let
# collector cannot open, discovered at runtime and nowhere else.
collectorSecretPath = "/run/credentials/opentelemetry-collector.service/${collectorCredentialId}";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
# `attrNames` is sorted, so this is a function of the hive SET and not of
# the order anyone wrote it in.
#
@ -817,6 +819,8 @@ in
script = ''
set -euo pipefail
${atomicWriteSecret}
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one
@ -862,10 +866,7 @@ in
# no uid to give this to — `LoadCredential` reads it as root before
# the sandbox exists and re-exposes it to whichever uid the unit got.
install -d -m 0755 ${lib.escapeShellArg collectorHostSecretDir}
umask 077
printf '%s\n' "$secret" > ${lib.escapeShellArg collectorHostSecretPath}
chown root:root ${lib.escapeShellArg collectorHostSecretPath}
chmod 0400 ${lib.escapeShellArg collectorHostSecretPath}
atomic_write_secret 0400 root:root ${lib.escapeShellArg collectorHostSecretPath} "$secret"
'';
};