vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

start the container journal forwarder's collector from its secret fetch, with backoff

Browse source
This commit is contained in:
atlas 2026-11-01 18:52:06 +01:00 • committed by mara
commit a0deb29e59
3 changed files with 51 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials-reference.md
View file

@ -134,7 +134,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the fetch fails fast while the store is down and retries with a delay that grows to 5 minutes; the attempt that succeeds starts the collector, which isn't part of the container's boot, and it resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |

37
nix/container-modules/swarm-journal-forwarder.nix
View file

@ -275,14 +275,29 @@ in
# `RestartSec` is ours: the 100ms default burns the start-rate limit while
# the secret is still on its way, leaving the unit failed instead of
# retrying.
systemd.services.opentelemetry-collector.serviceConfig = {
LoadCredential = [ "${credentialId}:${credentialSource}" ];
RestartSec = 15;
};
systemd.services.opentelemetry-collector = lib.mkMerge [
{
serviceConfig = {
LoadCredential = [ "${credentialId}:${credentialSource}" ];
RestartSec = 15;
};
}
# With a fetched secret the collector is not part of the boot
# transaction: `LoadCredential` refuses a missing file, and a unit that
# failed on it before the secret landed is not started again by its
# `Restart=`. The fetch starts the collector once it has written the
# secret, so the guest reaches READY without waiting on the store.
(lib.mkIf fetchesHere {
wantedBy = lib.mkForce [ ];
after = [ "${fetchUnit}.service" ];
})
];
# Reads the client secret with the container's own store identity. A store
# or authelia that is not up yet is a failed run that retries
# (../host-modules/lib/store-retry.nix), and a collector that restarts until the file lands.
# that is not up yet fails the attempt in seconds, which keeps the guest's
# boot short, and ../host-modules/lib/store-retry.nix starts it again with
# a growing delay until the store answers. The attempt that succeeds
# starts the collector.
systemd.services.${fetchUnit} = lib.mkIf fetchesHere {
description = "fetch this container's journal forwarder OIDC client secret from the swarm secret store";
after = [ "network-online.target" ];
@ -300,7 +315,13 @@ in
# an active unit is a no-op.
Type = "oneshot";
SyslogIdentifier = fetchUnit;
# Bounds one attempt, and so the guest's boot when the store is down.
TimeoutStartSec = 30;
# Each attempt is a store login: the delay grows from `RestartSec` to
# `RestartMaxDelaySec` over `RestartSteps` restarts (systemd.service(5)).
RestartSec = 5;
RestartSteps = 6;
RestartMaxDelaySec = 300;
};
environment = {
BAO_ADDR = cfg.store.addr;
@ -349,6 +370,10 @@ in
# `LoadCredential` copies the file at start only: a rotated secret
# reaches the collector by restarting it.
refresh_unit opentelemetry-collector.service ${lib.escapeShellArg secretFile}
# Not in the boot transaction (`wantedBy` is forced empty above), so
# nothing else starts it. A running collector is untouched by this.
systemctl start --no-block opentelemetry-collector.service
'';
};

19
nix/module-eval/nats-authelia.nix
View file

@ -147,6 +147,25 @@ let
&& !(lib.elem "--link-journal=host" container.extraFlags)
&& lib.any (lib.hasPrefix "--load-credential=${svc.credential}-cert:") container.extraFlags;
}
{
# The guest is ready when its job queue empties, so a long fetch job
# at boot would hold the container's start past its limit.
name = "the ${svc.name} forwarder's fetch retries with backoff outside the boot wait and starts the collector";
ok =
let
collector = c.systemd.services.opentelemetry-collector;
sc = fetch.serviceConfig;
in
collector.wantedBy == [ ]
&& lib.elem "swarm-journal-forwarder-oidc.service" collector.after
&& lib.elem "multi-user.target" fetch.wantedBy
&& sc.Restart == "on-failure"
&& sc.TimeoutStartSec <= 30
&& sc.RestartSteps > 0
&& sc.RestartMaxDelaySec > sc.RestartSec
&& !(lib.hasInfix "retry_in_job" fetch.script)
&& lib.hasInfix "systemctl start --no-block opentelemetry-collector.service" fetch.script;
}
{
name = "the ${svc.name} forwarder's metrics port is its own and loopback";
ok =