diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 8cdc24ee..6d10a8a5 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -129,7 +129,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives//` and drop the setting | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | -| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | +| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents//backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file | | an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents//proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | diff --git a/nix/host-modules/hive-forge/default.nix b/nix/host-modules/hive-forge/default.nix index 59548c80..f32e4f3e 100644 --- a/nix/host-modules/hive-forge/default.nix +++ b/nix/host-modules/hive-forge/default.nix @@ -71,6 +71,12 @@ let baoCfg = config.services.hyperhive.swarm.bao; storeRetry = import ../lib/store-retry.nix { }; + retryInJob = import ../lib/retry-in-job.nix { }; + + # How long the controller-account mint re-checks that the forge lists the + # account it just ensured. The mint is in the forge container's boot wait, + # whose `container@` limit is a minute. + accountRecheckSec = 15; storeRefreshTimer = import ../lib/store-refresh-timer.nix { }; atomicWriteSecret = import ../lib/atomic-write-secret.nix { }; refreshConsumer = import ../lib/refresh-consumer.nix { }; @@ -1052,14 +1058,28 @@ in # an already-provisioned non-admin account to work around # that (every hive from before this fix has one) — she # promotes those by hand instead. - if ! forgejo admin user list | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then + ${retryInJob} + + # The list is read whole before `grep` sees it: `grep -q` leaves + # at its first match, and under `pipefail` forgejo's write to the + # closed pipe would fail the pipeline for an account that is + # listed. + account_listed() { + local list + list="$(forgejo admin user list "$@")" || return 1 + grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]" <<<"$list" + } + + if ! account_listed; then forgejo admin user create \ --username ${lib.escapeShellArg swarmControllerForgeUser} \ --email ${lib.escapeShellArg "${swarmControllerForgeUser}@hyperhive.local"} \ --admin --random-password --must-change-password=false echo "created forge account ${swarmControllerForgeUser}" fi - if ! forgejo admin user list | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then + # This unit is part of the container's boot wait, so the re-check + # stays well under its start limit (`accountRecheckSec`). + if ! retry_in_job ${toString accountRecheckSec} account_listed; then echo "forge account ${swarmControllerForgeUser} absent after creation" >&2 exit 1 fi @@ -1099,7 +1119,7 @@ in # The revocation below goes through the forge's admin API, # which refuses a token of an account that is not a site # admin; each retry would then leave one more token live. - if ! forgejo admin user list --admin | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then + if ! account_listed --admin; then echo "forge account ${swarmControllerForgeUser} is not a site admin; promote it by hand. No token minted." >&2 exit 1 fi diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index facb846a..daa9ebf8 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -427,6 +427,12 @@ let privateNetwork = true; storeRetry = import ./lib/store-retry.nix { }; storeRefreshTimer = import ./lib/store-refresh-timer.nix { }; + retryInJob = import ./lib/retry-in-job.nix { }; + + # How long the controller-token unit waits for Grafana's socket before it + # fails into `storeRetry`'s restart. `container@` is ready when the guest's + # boot finishes, which can be before Grafana listens. + socketWaitSec = 60; in { # What the service IS to every hive is `swarm.grafana` in @@ -1127,9 +1133,9 @@ in SyslogIdentifier = "swarm-bao-grafana-controller-token"; # Grafana's answers, the new token among them, pass through a temp file. PrivateTmp = true; - # Each call below is bounded by curl's own timeouts; a run makes up to - # about ten of them. - TimeoutStartSec = "3min"; + # The socket wait (`socketWaitSec`) plus the calls below, each bounded + # by curl's own timeouts; a run makes up to about ten of them. + TimeoutStartSec = "4min"; }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; @@ -1173,6 +1179,18 @@ in exit 1 } + ${retryInJob} + + # Any HTTP answer counts; a refused connection is curl's exit status 7. + grafana_listening() { + curl --silent --output /dev/null --connect-timeout 5 --max-time 15 \ + --unix-socket ${lib.escapeShellArg socketPath} http://localhost/api/health + } + if ! retry_in_job ${toString socketWaitSec} grafana_listening; then + echo "Grafana's socket ${socketPath} did not answer within ${toString socketWaitSec} seconds." >&2 + exit 1 + fi + if [ ! -s ${lib.escapeShellArg adminPasswordPath} ]; then echo "no Grafana admin password at ${adminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2 exit 1 diff --git a/nix/module-eval/forge-placement.nix b/nix/module-eval/forge-placement.nix index 4c275940..81393dec 100644 --- a/nix/module-eval/forge-placement.nix +++ b/nix/module-eval/forge-placement.nix @@ -356,7 +356,7 @@ let order = l: lib.length l < 2 || (at (lib.elemAt l 0) < at (lib.elemAt l 1) && order (lib.tail l)); steps = [ "if [ -s \"$name_path\" ]; then" - "forgejo admin user list --admin" + "account_listed --admin" "bao login -method=cert" "forgejo admin user generate-access-token" "bao kv put" @@ -391,6 +391,31 @@ let )) && lib.elem "forgejo-swarm-controller-account.service" c.swarm-bridge-dns.before; } + { + # The mint is in the forge container's boot wait (a oneshot in + # `multi-user.target`), so what it waits for inside its start stays under + # `container@`'s start limit. The account check reads the list whole: a + # `grep -q` on a pipe under `pipefail` fails an account that is listed. + name = "the forge's mint re-checks the account for a bounded time, under the container's start limit, and reads the list without a pipe"; + ok = + let + u = mint allLocal; + s = u.script; + parts = lib.splitString "if ! retry_in_job " s; + budget = lib.toInt (lib.head (lib.splitString " " (lib.elemAt parts 1))); + in + lib.length parts == 2 + && lib.hasInfix "retry_in_job() {" s + && lib.hasInfix "${toString budget} account_listed; then" s + && budget > 0 + && budget <= 20 + && allLocal.containers.hive-forge.timeoutStartSec == "1min" + && lib.elem "multi-user.target" u.wantedBy + && lib.hasInfix "list=\"$(forgejo admin user list \"$@\")\"" s + && lib.hasInfix "account_listed --admin" s + && !(lib.hasInfix "forgejo admin user list |" s) + && !(lib.hasInfix "forgejo admin user list --admin |" s); + } { # The absence arm: a forge host without the store has no leaf unless one # is placed, and the mint says which options to set before it mints. diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index 60613d18..99a66366 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -310,6 +310,30 @@ let && lib.elem "swarm-bao-grafana-admin.service" unit.after && grafanaRemoteAuthelia.systemd.timers ? swarm-bao-grafana-controller-token; } + { + # The container is ready when the guest's boot finishes, which can be + # before Grafana listens, so the unit waits for the socket inside its own + # start job, ahead of every call that needs it. The wait is bounded and + # sits inside the unit's own timeout, or systemd kills the job first. + name = "the controller's Grafana token waits a bounded time for Grafana's socket before its first call"; + ok = + let + unit = grafanaRemoteAuthelia.systemd.services.swarm-bao-grafana-controller-token; + s = unit.script; + parts = lib.splitString "if ! retry_in_job " s; + budget = lib.toInt (lib.head (lib.splitString " " (lib.elemAt parts 1))); + at = needle: lib.stringLength (lib.head (lib.splitString needle s)); + in + lib.length parts == 2 + && lib.hasInfix "retry_in_job() {" s + && lib.hasInfix "${toString budget} grafana_listening; then" s + && lib.hasInfix "http://localhost/api/health" s + && budget > 0 + && budget <= 120 + && unit.serviceConfig.TimeoutStartSec == "4min" + && at "retry_in_job ${toString budget} grafana_listening" < at "no Grafana admin password at" + && at "retry_in_job ${toString budget} grafana_listening" < at "status=\"$(grafana \"$admin\" GET"; + } { # Same 403-not-a-miss reason as the matrix and queue arms below: the # reader's grant covers the `services` prefix, so a path outside it is