wait for grafana's socket and read the forge's user list whole in the mint
This commit is contained in:
atlas
parent
c0429bf617
commit
279428a66f
5 changed files with 95 additions and 8 deletions
|
|
@ -129,7 +129,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
|
|||
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
|
||||
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives/<hive>/` and drop the setting |
|
||||
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
|
||||
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
|
||||
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
|
||||
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
|
||||
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
|
||||
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
|
||||
|
|
|
|||
|
|
@ -71,6 +71,12 @@ let
|
|||
|
||||
baoCfg = config.services.hyperhive.swarm.bao;
|
||||
storeRetry = import ../lib/store-retry.nix { };
|
||||
retryInJob = import ../lib/retry-in-job.nix { };
|
||||
|
||||
# How long the controller-account mint re-checks that the forge lists the
|
||||
# account it just ensured. The mint is in the forge container's boot wait,
|
||||
# whose `container@` limit is a minute.
|
||||
accountRecheckSec = 15;
|
||||
storeRefreshTimer = import ../lib/store-refresh-timer.nix { };
|
||||
atomicWriteSecret = import ../lib/atomic-write-secret.nix { };
|
||||
refreshConsumer = import ../lib/refresh-consumer.nix { };
|
||||
|
|
@ -1052,14 +1058,28 @@ in
|
|||
# an already-provisioned non-admin account to work around
|
||||
# that (every hive from before this fix has one) — she
|
||||
# promotes those by hand instead.
|
||||
if ! forgejo admin user list | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then
|
||||
${retryInJob}
|
||||
|
||||
# The list is read whole before `grep` sees it: `grep -q` leaves
|
||||
# at its first match, and under `pipefail` forgejo's write to the
|
||||
# closed pipe would fail the pipeline for an account that is
|
||||
# listed.
|
||||
account_listed() {
|
||||
local list
|
||||
list="$(forgejo admin user list "$@")" || return 1
|
||||
grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]" <<<"$list"
|
||||
}
|
||||
|
||||
if ! account_listed; then
|
||||
forgejo admin user create \
|
||||
--username ${lib.escapeShellArg swarmControllerForgeUser} \
|
||||
--email ${lib.escapeShellArg "${swarmControllerForgeUser}@hyperhive.local"} \
|
||||
--admin --random-password --must-change-password=false
|
||||
echo "created forge account ${swarmControllerForgeUser}"
|
||||
fi
|
||||
if ! forgejo admin user list | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then
|
||||
# This unit is part of the container's boot wait, so the re-check
|
||||
# stays well under its start limit (`accountRecheckSec`).
|
||||
if ! retry_in_job ${toString accountRecheckSec} account_listed; then
|
||||
echo "forge account ${swarmControllerForgeUser} absent after creation" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
|
@ -1099,7 +1119,7 @@ in
|
|||
# The revocation below goes through the forge's admin API,
|
||||
# which refuses a token of an account that is not a site
|
||||
# admin; each retry would then leave one more token live.
|
||||
if ! forgejo admin user list --admin | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then
|
||||
if ! account_listed --admin; then
|
||||
echo "forge account ${swarmControllerForgeUser} is not a site admin; promote it by hand. No token minted." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -427,6 +427,12 @@ let
|
|||
privateNetwork = true;
|
||||
storeRetry = import ./lib/store-retry.nix { };
|
||||
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||
retryInJob = import ./lib/retry-in-job.nix { };
|
||||
|
||||
# How long the controller-token unit waits for Grafana's socket before it
|
||||
# fails into `storeRetry`'s restart. `container@` is ready when the guest's
|
||||
# boot finishes, which can be before Grafana listens.
|
||||
socketWaitSec = 60;
|
||||
in
|
||||
{
|
||||
# What the service IS to every hive is `swarm.grafana` in
|
||||
|
|
@ -1127,9 +1133,9 @@ in
|
|||
SyslogIdentifier = "swarm-bao-grafana-controller-token";
|
||||
# Grafana's answers, the new token among them, pass through a temp file.
|
||||
PrivateTmp = true;
|
||||
# Each call below is bounded by curl's own timeouts; a run makes up to
|
||||
# about ten of them.
|
||||
TimeoutStartSec = "3min";
|
||||
# The socket wait (`socketWaitSec`) plus the calls below, each bounded
|
||||
# by curl's own timeouts; a run makes up to about ten of them.
|
||||
TimeoutStartSec = "4min";
|
||||
};
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
|
|
@ -1173,6 +1179,18 @@ in
|
|||
exit 1
|
||||
}
|
||||
|
||||
${retryInJob}
|
||||
|
||||
# Any HTTP answer counts; a refused connection is curl's exit status 7.
|
||||
grafana_listening() {
|
||||
curl --silent --output /dev/null --connect-timeout 5 --max-time 15 \
|
||||
--unix-socket ${lib.escapeShellArg socketPath} http://localhost/api/health
|
||||
}
|
||||
if ! retry_in_job ${toString socketWaitSec} grafana_listening; then
|
||||
echo "Grafana's socket ${socketPath} did not answer within ${toString socketWaitSec} seconds." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -s ${lib.escapeShellArg adminPasswordPath} ]; then
|
||||
echo "no Grafana admin password at ${adminPasswordPath} yet; swarm-bao-grafana-admin writes it." >&2
|
||||
exit 1
|
||||
|
|
|
|||
|
|
@ -356,7 +356,7 @@ let
|
|||
order = l: lib.length l < 2 || (at (lib.elemAt l 0) < at (lib.elemAt l 1) && order (lib.tail l));
|
||||
steps = [
|
||||
"if [ -s \"$name_path\" ]; then"
|
||||
"forgejo admin user list --admin"
|
||||
"account_listed --admin"
|
||||
"bao login -method=cert"
|
||||
"forgejo admin user generate-access-token"
|
||||
"bao kv put"
|
||||
|
|
@ -391,6 +391,31 @@ let
|
|||
))
|
||||
&& lib.elem "forgejo-swarm-controller-account.service" c.swarm-bridge-dns.before;
|
||||
}
|
||||
{
|
||||
# The mint is in the forge container's boot wait (a oneshot in
|
||||
# `multi-user.target`), so what it waits for inside its start stays under
|
||||
# `container@`'s start limit. The account check reads the list whole: a
|
||||
# `grep -q` on a pipe under `pipefail` fails an account that is listed.
|
||||
name = "the forge's mint re-checks the account for a bounded time, under the container's start limit, and reads the list without a pipe";
|
||||
ok =
|
||||
let
|
||||
u = mint allLocal;
|
||||
s = u.script;
|
||||
parts = lib.splitString "if ! retry_in_job " s;
|
||||
budget = lib.toInt (lib.head (lib.splitString " " (lib.elemAt parts 1)));
|
||||
in
|
||||
lib.length parts == 2
|
||||
&& lib.hasInfix "retry_in_job() {" s
|
||||
&& lib.hasInfix "${toString budget} account_listed; then" s
|
||||
&& budget > 0
|
||||
&& budget <= 20
|
||||
&& allLocal.containers.hive-forge.timeoutStartSec == "1min"
|
||||
&& lib.elem "multi-user.target" u.wantedBy
|
||||
&& lib.hasInfix "list=\"$(forgejo admin user list \"$@\")\"" s
|
||||
&& lib.hasInfix "account_listed --admin" s
|
||||
&& !(lib.hasInfix "forgejo admin user list |" s)
|
||||
&& !(lib.hasInfix "forgejo admin user list --admin |" s);
|
||||
}
|
||||
{
|
||||
# The absence arm: a forge host without the store has no leaf unless one
|
||||
# is placed, and the mint says which options to set before it mints.
|
||||
|
|
|
|||
|
|
@ -310,6 +310,30 @@ let
|
|||
&& lib.elem "swarm-bao-grafana-admin.service" unit.after
|
||||
&& grafanaRemoteAuthelia.systemd.timers ? swarm-bao-grafana-controller-token;
|
||||
}
|
||||
{
|
||||
# The container is ready when the guest's boot finishes, which can be
|
||||
# before Grafana listens, so the unit waits for the socket inside its own
|
||||
# start job, ahead of every call that needs it. The wait is bounded and
|
||||
# sits inside the unit's own timeout, or systemd kills the job first.
|
||||
name = "the controller's Grafana token waits a bounded time for Grafana's socket before its first call";
|
||||
ok =
|
||||
let
|
||||
unit = grafanaRemoteAuthelia.systemd.services.swarm-bao-grafana-controller-token;
|
||||
s = unit.script;
|
||||
parts = lib.splitString "if ! retry_in_job " s;
|
||||
budget = lib.toInt (lib.head (lib.splitString " " (lib.elemAt parts 1)));
|
||||
at = needle: lib.stringLength (lib.head (lib.splitString needle s));
|
||||
in
|
||||
lib.length parts == 2
|
||||
&& lib.hasInfix "retry_in_job() {" s
|
||||
&& lib.hasInfix "${toString budget} grafana_listening; then" s
|
||||
&& lib.hasInfix "http://localhost/api/health" s
|
||||
&& budget > 0
|
||||
&& budget <= 120
|
||||
&& unit.serviceConfig.TimeoutStartSec == "4min"
|
||||
&& at "retry_in_job ${toString budget} grafana_listening" < at "no Grafana admin password at"
|
||||
&& at "retry_in_job ${toString budget} grafana_listening" < at "status=\"$(grafana \"$admin\" GET";
|
||||
}
|
||||
{
|
||||
# Same 403-not-a-miss reason as the matrix and queue arms below: the
|
||||
# reader's grant covers the `services` prefix, so a path outside it is
|
||||
|
|
|
|||
Loading…
Reference in a new issue