The nix-store gauge comment claimed a semconv name it deliberately does
not use; it now says why the name is hyperhive-scoped. Two doc pages
still described the removed banner for the limits-read and
knowledge-pull failures. The warning alert's comment now says
pending_logins pages on purpose.
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
no behaviour change: a pure move of items and tests into new modules, plus
visibility edits (pub(crate)) and path updates in sibling modules that named
crate-root items. same routes, same OpenAPI, 561 tests before and after.
why: #5470, new features stop landing in one 6.4k-line file by default.
files outside swarm-controller/src: docs/agent-lifecycle/agent-roster.md,
swarm-ui JobsPage.tsx and AgentsPage.tsx, nix/host-modules/swarm-controller.nix
and swarm-grafana.nix hold comments that named main.rs for items that moved.
hh-swarm-queue-connect-failed matched "swarm queue connect failed; this
agent publishes nothing upward", which nothing has logged since the
harness started retrying its first connect and exiting (235bc481b). The
rule now matches the error the harness logs when every attempt has
failed, CONNECT_EXHAUSTED in swarm_queue.rs, and its comment and summary
describe the retry-then-exit behaviour.
A failed MintAgentIdentity node logged nothing: the error reached only
the job graph. mint_identity now warns with the agent and the error, and
hh-agent-identity-mint-failed fires on that line.
Refs #4651
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.
The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.
The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
A swarm-ci-metrics host unit on a 5-minute timer lists every forge
repo's Actions runs as the new read-only ci-metrics account and pushes
each finished run's duration and queue wait over OTLP, stamped at the
run's stop time. The first pass backfills from run 1. swarm-controller
mints the account's token into bao and keeps a ci-metrics team with
read on the Actions unit in every organisation.
Closes#5376
Ten service modules (forge, matrix chat, authelia, grafana, otel, nats,
swarm-ui, victorialogs, victoriametrics, bao UI) wrote
services.nginx.virtualHosts themselves, each applying the gateway's
listen set, per-name TLS attrs and security headers by hand. They now
declare services.hyperhive.gateway.vhosts.<name> with only their own
locations and options; ./declared-vhosts.nix renders it into the nginx
vhost with listen, TLS and headers added.
An eval assertion refuses a declared vhost that sets listen, addSSL,
onlySSL, enableACME, useACMEHost, sslCertificate or sslCertificateKey,
and one declared while the gateway is off. nats' own leaf is the new
`certificate` field. The rendered nginx vhosts are unchanged.
Each container now runs the journal forwarder the queue's and the builder's
containers run, pushing to a receiver of its own on the swarm collector. Each
authenticates as a new machine OIDC client of its own
(swarm.<service>.otel.clientId), registered by a glue file wherever authelia
runs, minted by authelia and published to the store by the secret publisher.
Grafana's browser login client is unchanged.
The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service (the runner's, Grafana's,
and the log store responder's), now also handed into the container as an
nspawn system credential. Each of those three policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.
The three containers drop --link-journal=host, and the host collector stops
reading them.
The gateway nginx serves stub_status on a loopback server of its own,
127.0.0.1:8085 (services.hyperhive.gateway.statusPort). The hive-tier
collector reads it with the contrib nginx receiver, which the pinned
otelcol-contrib 0.151.0 ships, so no exporter daemon is added. The
receiver dials http:// only (a unix: or http+unix: endpoint fails with
"unsupported protocol scheme"), so the listener is loopback TCP with
allow 127.0.0.1 / deny all rather than a unix socket.
The new gateway dashboard plots nginx.requests, nginx.connections_current
by state, and nginx.connections_accepted against handled, per hive.
Part of #5342.
Grafana's VictoriaLogs datasource now targets https://logs.<swarm> and sends
the same static bearer as the metrics datasource. The log store's vhost routes
a 64-hex Authorization bearer to a second swarm-metrics-auth instance
(swarm-logs-grafana-auth, cert role swarm-logs-auth, which gains a read on the
bearer's store path). swarm-agent. tokens and everything else keep their routes.
Part of #3742 (DoD 3, logs).
Three dashboards over metrics already in VictoriaMetrics: the swarm
controller's job graph and forge push/commit counters, the forge
process's runtime, and the OTEL collectors' self-telemetry.
The three delivery units made /run/swarm-grafana-secrets only after
`bao login` and `bao kv get`, so with the store sealed or unreachable the
bind source was never created by them and a switch could restart the
container without it. Each script now starts with the install, from one
shared `ensureSecretDir`, and the tmpfiles comment says what is true.
The module-eval case asserts the install precedes `bao login` in all three
scripts. docs/swarm/sso.md states that after a reboot Grafana stays down
until the secret store answers.
Part of #4998 (slice 2b: grafana).
The three delivery units still read from bao under Grafana's own leaf, but
now write to /run/swarm-grafana-secrets on the host (tmpfs), which the
container bind-mounts read-only at the same path. Before, they wrote to
/var/lib/nixos-containers/swarm-grafana/var/lib/{grafana-oidc,grafana-admin,
grafana-metrics}/ on persistent disk, where a superseded value stayed after
rotation and went into backups of the container root. Host tmpfiles now
removes the three old files.
Rotation is unchanged: refresh_consumer restarts Grafana (or starts it if it
failed) when a file changes, since Grafana reads each $__file{} once as it
starts. After a reboot the files are fetched again; until the store answers
Grafana stays down, where before it came up on the last copy from disk.
Part of #4998 (slice 2b).
swarm-bao-grafana-controller-token, on the host that runs Grafana, logs in
to Grafana's API over its socket as the built-in admin (the password
swarm-bao-grafana-admin minted, #5138) and keeps a `swarm-controller`
service account that is an enabled Viewer. It mints a token for that
account and writes it to bao at
swarm/controller/swarm-controller/grafana/token under Grafana's leaf. A
ten-minute timer runs it again. The unit keeps a stored token Grafana
accepts. If Grafana refuses the stored token with 401, or bao holds
none, the unit deletes every token of the account and mints a new one.
Grafana's grant gains create/read/update on that leaf, and the
controller's gains read.
swarm-controller serves GET /api/alerts from Grafana's
/api/prometheus/grafana/api/v1/alerts. It reads the token from bao,
keeps it in memory, and re-reads bao once on a 401. The HTTP client has
5s connect and 15s request timeouts. Any failure is a 503 that names its
cause. swarm-ui gets an /alerts page listing them, built the same way as
the hives page.
This is option (c) on #5129, the operator's choice.
The metrics vhost forwarded the store's read API (query, series, labels,
export, vmui, ...) with no credential, so anything that could reach
metrics.<swarm> could read every hive's and agent's metrics. mara ruled on
checked by an auth_request responder, and operators use their authelia
session.
- swarm-metrics-auth (new crate): auth_request responder on the metrics
store's host. Compares the presented bearer, in constant time, with the
one stored at swarm/services/<grafana client id>/metrics/bearer, read
under its own bao leaf and cached for 60s. Static user, the post-#5264
swarm-logs-auth unit shape.
- swarm-bao-grafana-metrics-bearer: on Grafana's host, creates that bearer
in bao if absent (create without update, Grafana's existing leaf) and
writes "Bearer <value>" into the container; the datasource sends it as
its Authorization header via $__file{}.
- The vhost: read locations take auth_request through a map (a bearer goes
to swarm-metrics-auth, anything else to authelia, no login redirect);
vmui takes an authelia session with the login redirect; forceSSL like
the logs vhost. The authelia subrequest location is renamed from
/__metrics_push_authz to /__metrics_authelia since it now serves reads.
- Authelia: the collector keeps every path; everyone else is refused the
collector's push and /metrics paths; group admins is admitted to the
rest; then deny.
- bao: a swarm-metrics-auth cert role and policy (read on the one leaf),
its leaf minted by glue-bao-tls, the CN reserved as a hive name, and
Grafana's grant gains create on the bearer leaf.
Fixes#3596
Grafana's own metrics were scraped over loopback through the
`grafana-metrics` nginx vhost, declared only where grafana and the
collector share a host, so a collector on any other host had no job for
it. Its target is now declared on every host as
`https://<swarm.grafana.domain>/metrics`, next to the other published
targets, and grafana's vhost serves `= /metrics` behind the collector's
auth subrequest (`/__grafana_metrics_authz`, the forge's shape). The
rules derived from `publishedScrapeTargets` admit the collector's client
there and deny every other subject.
The loopback scrape is gone with the vhost that served it and its
`swarm.grafana.metricsPort` option, so a single host still scrapes
grafana once, now by name. A config that sets that option now fails
evaluation with "option does not exist".
On the published vhost `/metrics` was answered with 403 for everyone.
It now answers the collector's token, and a plain session gets authelia's
`deny`. The design note that ruled the published vhost out for scraping
predates the collector's credential; it is replaced by the location's
own comment and a line on `swarm.grafana.domain`.
Part of #5205
Each store reader (forge's OIDC fetch and token publisher, matrix's OIDC
fetch, the hive collector's fetch, Grafana's OIDC fetch and admin password,
the queue's callout-xkey unit, the CI runner's token fetch, the swarm
collector's OIDC fetch) used to read `deploy.bao.enable` in its own module
to add `after`/`wants` on `container@<store machine>` and
`after`/`requires` on `swarm-bao-pki.service`. Per the #3742 ruling, that
cross-service read moves into glue-bao-readers-store-order.nix, which sets
the same edges when the store and the reader are both on this host and
restates each reader's own gate so no stub unit is rendered.
Rendered `after`/`wants`/`requires` are identical as sets to main on 16
fixtures, store here and elsewhere. New module-eval cases in existing
fixtures pin both arms.
Part of #3742.
Grafana runs with metrics.enabled and no basic_auth_*, so its Prometheus
exposition was served to any anonymous client of grafana.<swarm> through
the published vhost's `location "/"`. Add an exact-match `= /metrics`
location returning 403 there. The loopback `grafana-metrics` vhost and
the collector's scrape of it are unchanged.
Comments on the metricsPort option and the loopback vhost now state that
the published vhost refuses /metrics; the gateway doc says the same.
A module-eval case pins the refusing location and that `/` still proxies.
The metrics datasource defaulted to http://127.0.0.1:<port>, which only
reached the store when Grafana and VictoriaMetrics ran on the same host.
Swarm-tier services may not assume colocation (#4499), so the default is
now https://<swarm.victoriametrics.domain>, the gateway vhost the store's
host already serves. That vhost's query routes carry no auth_request, and
the Grafana container already trusts the hive CA for its authelia token
exchange, so no credential is added.
The logs datasource stays on loopback: the log store's query route needs
a bearer token Grafana does not hold yet.
Part of #3742 (item 4, metrics).
Seven host oneshots fetch a secret from the swarm secret store into a file
their consumer loads at start: swarm-bao-grafana-oidc, swarm-bao-otel-oidc,
swarm-bao-matrix-oidc, swarm-bao-ci-runner, swarm-bao-forge-oidc,
swarm-bao-hive-otel-oidc and swarm-bao-forwarder-oidc. Each was
RemainAfterExit with no timer, so it ran at boot or deploy and never again:
a secret re-minted in the store reached its consumer only through an
operator restarting the fetch.
lib/store-refresh-timer.nix gives each one a timer (OnUnitInactiveSec 10min,
RandomizedDelaySec 1min, the agent's own store re-read interval), and each
drops RemainAfterExit so a timer start is not a no-op. Each script already
writes the file only when secret_differs, and refresh_consumer/refresh_unit
restart the consumer only when the file is newer than its last start, so a
run reading an unchanged value restarts nothing.
Two docs said these units do not run on a timer; credentials.md's
ci-runner row said the runner token is not re-pulled.
Part of #5017 (slice T).
Grafana's built-in admin kept the nixpkgs default password `admin`, and
Grafana's API takes basic auth whatever `auth.disable_login_form` says, so
the public vhost gave server-admin to `admin:admin`.
`swarm-bao-grafana-admin` creates a random password in the store under
Grafana's existing leaf when none is stored (`create` without `update`, so
the first one is kept), then copies it into the container.
`security.admin_password` reads it for a new database, and a `grafana cli
admin reset-admin-password` in `grafana.service`'s `preStart` applies it to
an existing one on every start.
Closes#5138
Agents reach the operator over matrix now (#5115). A `send` to
`operator` is refused, by the broker and by the MCP allow-list check,
with an error that names the matrix MCP tools. The dashboard's Y3R C4LL
tab goes with it: the operator inbox, `GET /api/operator-inbox`,
`Broker::unread_for_recipient`, and the browser notifications that only
ever fired on operator-bound messages.
The harness no longer posts turn failures and plugin-install failures
to the operator inbox. Both are already logged at warn; a new Grafana
rule, `hh-agent-turn-failed`, fires per agent on the harness's
`turn failed` line.
Dashboard-to-agent messaging (op-send, the agent web UI's message box,
mark-all-read for agent inboxes) is unchanged.
Decisions per mara on #5115: q1 a, q2 a plus a grafana alert, q3 keep
dashboard-to-agent messaging and leave swarm-level code alone.
Six more comments and option descriptions across five modules said the hive's leaf or grant reads every secret in the store. It reads swarm/agents/* and swarm/hives/<hive>/* and nothing else, so they now say that, in the wording the clientCertFile option already used. Also the matrix-ctl check's label no longer calls the hive's leaf store-wide.
The hive's bao policy grants read on swarm/agents/* and swarm/hives/<hive>/* only. Five comments across four modules still listed swarm/services/* among its reads, or said a shared leaf would let a unit fetch another service's OIDC client secret, or that the hive's leaf reads every secret in the store. Comment and option-description text only.
The socket's directory is its access control, so it has to stay dedicated
to Grafana's one socket. Nothing in the tree sets the option; making it
read-only turns "point it at a shared directory" into an eval error
instead of a convention.
A rename shim contributes a definition to its target, which a read-only
option refuses, so the swarm.grafana.socketDir alias goes with it. The
module-eval fixture no longer sets the old path; a new case asserts that
defining the option fails evaluation and that the default reaches nginx.
A deploy request whose rev hive-c0re refuses (fetch failed, or the rev
does not descend from applied/main) no longer tries to comment on the PR
that merged it. Off the forge host that attempt always failed with "forge
core token absent". The refusal now logs at ERROR, message unchanged, and
the new `hh-config-rev-refused` Grafana rule fires on it.
Deleted with the posting: actions::post_rev_deploy_failure_to_pr, the
whole forge/pr_comment.rs (config_repo, merged_pr_for_commit,
post_pr_comment), and ReaderToken::value, whose only caller was
merged_pr_for_commit.
Refs #4851
Each of these wrote its file in place (or set the mode after the
write), so a write cut short by a kill or a full disk left a partial
file behind, and some secrets existed for a moment at umask mode:
- swarm-authelia/swarm-grafana/hive-forge/swarm-bao generators: a
partial file passed the `-s`/`-e` guard and was kept forever. They now
capture the value first and land it with `atomic_write_secret`.
- swarm-controller webhook secret: a truncated file failed the 64-hex
check and the secret was regenerated, out of step with the forge
hooks. It is now written 0600 to a sibling temp, fsynced, renamed.
- hive-c0re htpasswd: the temp is now fsynced and set 0644 before the
rename, like the sibling agents.conf writer, instead of relying on
c0re's umask.
- hivectl wg init: the private key is created 0400 in a temp and
renamed, instead of written at umask mode and chmod'ed after.
Closes#5068
The docs, option descriptions and comments that named the swarm root, the
swarm CA or the hive CA as an existing trust anchor now name the secret
store's pki root or the trust bundle that holds it:
- docs/swarm/README.md, secrets.md, bao.md
- deploy.bao.clientCaFile's and gateway.verifiedProxyTo's descriptions
- comments in hive-c0re/src/meta.rs, hive-agent/src/swarm_queue.rs,
swarm-logs/src/auth.rs, swarm-controller/src/vcs_metrics.rs,
hive-forge, swarm-authelia, swarm-ci-runner, glue-bao-tls, swarm-bao,
swarm-grafana, swarm-otel, otel, hive-c0re, lib/hive-ca-trust.nix,
hive-forge/Cargo.toml and the flake-update workflow
Nothing behaves differently: every edit is a nix, Rust or TOML comment, an
option description, or a test assertion message. The Rust edits change the
workspace source, which this PR already rebuilds for hivectl.
Refs #4588
Rewrite the remaining narration in files the previous commit touched:
"moved"/"sat"/"was REMOVED"/"old default" comments in swarm-bao,
swarm-otel, glue-bao-tls, nats-authelia, swarm-controller, swarm-ui,
swarm-otel-service, swarm-authelia-service, the deploy.nix rename table,
hive-ci, hive-forge/service, hive-tls, hyperhive, swarm-nats, swarm.nix
and a few module-eval fixtures, so each says where a thing is or why it
is so now.
Revert the comment edits inside shell script strings in hive-tls,
swarm-nats and the forge token mint to their previous text, so those
generated units are unchanged. The only unit still changing is
swarm-bao-pki (two comment lines in its script). Refs #3902
Rewrite comments and option descriptions under nix/ that narrate how the
code got here ("used to", "new name", "sat at the TOP", "no longer",
renamed/moved narration, attributions) to say what the code does now and
why. Comments and description strings only; no logic changes.
Files in open PR #4970's diff are left alone. Refs #3902
Every hive's homeserver loaded a per-hive `hyperhive` appservice
registration, and the swarm minted and delivered an as_token for it
through bao. No client presented that token: swarm-controller creates
every `@hive-<hive>` sender and every agent account with the swarm
appservice token (hive_sender.rs, agent_token.rs), and no Rust code read
the per-hive one.
Removed, with everything that existed only to serve it:
- hive-matrix.nix: the `hyperhive` registration, its activation-time
render, the host bind mount of /var/lib/hyperhive/matrix-appservice,
its LoadCredential entry, and the internal options
`deploy.matrix.appserviceTokenFile` / `appserviceRegistrationScript`.
tuwunel now loads only the swarm registration.
- glue-matrix-bao-token.nix and its `swarm-bao-matrix-token` unit.
- glue-bao-tls.nix: the matrix-token leaf and its defaults.
- swarm-bao.nix: the `swarm-matrix-token-<hive>` roles and policies
(`swarm-bao-matrix-token-policy`), and the options
`deploy.bao.matrixTokenClient{Cert,Key}File` and
`matrixTokenCommonNamePrefix`, now mkRemovedOptionModule.
- swarm-secret-publisher.nix: the per-hive token mint and its state dir.
- swarm.nix: the reserved `<matrixTokenCommonNamePrefix>-<hive>` CN.
- swarm-secret-client: `matrix::appservice_token_path`.
The swarm registration (swarm-matrix-ctl, swarm-controller) is untouched.
Comments and docs that pointed at the removed glue file now point at
glue-queue-agent-credential.nix, which has the same reader shape.
Refs #4428
The hive policy rendered by policy::render granted every hive cert read
on secret/data/swarm/services/*, which holds each swarm service's OIDC
client secret. No reader of that tree authenticates with a hive cert:
grafana, the otel collector and the store forwarder each read with
their own leaf, and the bao UI's secret is read by the store's granter.
render() now emits two stanzas: swarm/agents/* and the hive's own
swarm/hives/<hive>/*. Kind::Service had no other caller and is removed.
Docs and comments that described the grant as a decision are rewritten
to the current state.
Closes#4960
The swarm-controller builds the Forge quick link from
services.hyperhive.swarm.forge.domain, replacing hive-forge/default.nix's
per-host entry, so all seven swarm-service links come from swarm-level
options.
Also drops the remaining references to the removed matrix GUI switch:
the HiveUrls / Urls / hive_urls docs, the hivectl.md `open` note and
the gateway.md vhost-map rows, which name `gatewayHost` instead. The
grafana, victoriametrics and victorialogs modules' comments no longer
mention a quick-link they do not define.
Refs #4885
The swarm-controller module builds the Authelia, Grafana, Metrics and
Logs quick links from services.hyperhive.swarm.<service>.domain, on the
controller's host, instead of each service module adding its entry only
on the host that runs it. A controller whose swarm runs those services
on other hosts lists them in its /api/links popover.
Forge, matrix and bao links are not moved yet: forge waits on #4891,
matrix and bao on whether their GUI gate becomes swarm-level.
Refs #4885
`swarm.grafana` (what the metrics UI is to every hive: container name,
domain, metrics port, OIDC client) moves to
nix/host-modules/swarm-grafana-service.nix, together with `domainBase`,
the only helper it reads besides `cfg`. Everything else -- the
`deploy.grafana` options, the whole `config` block including
`containers.swarm-grafana`, and the helpers only they read -- stays in
nix/host-modules/swarm-grafana.nix, which default.nix now imports
alongside the new file.
Both halves read `cfg` (`swarm.grafana.oidc.redirectUri` defaults from
`cfg.domain`; the config block reads `cfg` throughout). It is an option
read, so each file binds it from `config.services.hyperhive.swarm.grafana`.
The service file has no `hyperhiveCfg`, so its `swarmDomain` reads
`config.services.hyperhive.swarm.domain` directly, as
swarm-nats-service.nix does.
A pure move: option paths, option definitions and config are unchanged
apart from the two comments on either side of the cut, which now name the
file the other half lives in.
Refs #3742
The swarm collector's journald receiver read only the units listed in
`services.hyperhive.swarm.otel.journaldUnits`. A unit nobody listed
never reached the store, and a misspelt entry shipped nothing without
an error. The list existed to keep an operator's desktop session out of
a store every swarm operator can read, but the receiver can only match
positively, so the only way to express "not user sessions" was to name
every service instead.
The receiver now reads the whole host journal, and a new
`filter/exclude-user-sessions` processor in the `logs/<swarm>` pipeline
drops records whose `_SYSTEMD_SLICE` is `user-<uid>.slice` (session
scopes and `user@<uid>.service`). The per-hive `logs/<hive>` pipelines
carry agent-container journals only and get no filter.
`journaldUnits` is removed with `mkRemovedOptionModule`, together with
its non-empty assertion and the entry each host module added. The four
module-eval membership checks go with it, replaced by one structural
case in swarm-otel-core.
Closes#3646
Adds rules to the `hyperhive` alerting folder next to the two from #4811:
- swarm queue connect failed (per hive/agent, 1h window, fires at once:
the line is logged once per agent process and the connection is never
retried, so the window is how long the rule stays firing)
- swarm agent state publish skipped (per hive/agent, 15m, for 5m, as the
swarm terminal rule)
- swarm queue credential missing (per hive/agent, 15m, for 5m)
- agent credential renewal failing (10m, for 15m, as the forge reconcile
rule: same 5m pass cadence, one line per failed pass)
- no log records from <hive>, one per `swarm.hives` entry: fires when
the hive's ungrouped count over 10m is below 1, and on no data. A new
`logAbsenceRule` helper wraps `logCountRule` with the inverted
threshold and noDataState = Alerting.
No contact point and no notification policy: the rules show under
Alerting -> Alert rules and are delivered nowhere.
Refs #4717
Refs #3900
The restart decision was a shell variable set by comparing the fetched
value with the file just before overwriting it. A run that wrote the
file and then failed before the restart (the matrix unit's registration
render, or `systemctl --machine` finding no bus yet) left a retry that
saw an unchanged file and never restarted the consumer.
The file is now written only when the value differs, so its mtime marks
the last real change, and `refresh_consumer <machine> <unit> <path>`
compares that mtime with the consumer's ActiveEnterTimestamp on every
run, the shape the openbao client-CA refresh in swarm-bao.nix already
uses. A consumer that started after the last change is left alone; a
running one is try-restarted, a failed one reset and started, all with
--no-block, and nothing happens while the container is down.
The helper's comment block also exceeded the 30-line limit
(`comment-block lint` failed on d871467d); its per-function notes now
sit beside the functions.
module-eval-bao-grants asserts the gated write, the path the refresh is
keyed on, and the mtime-vs-start comparison for each consumer.
Refs #4662
The previous commit's comments said a unit in auto-restart keeps its
start job, so anything ordered after it waits for the whole 24h retry
window. That is wrong under the default RestartMode=normal: each failed
attempt passes through `failed`, which ends that start job. `After=`
dependents proceed after one attempt, `Requires=` dependents fail with
`dependency`, and the retries continue as fresh start jobs. The
2026-09-24 journal shows it with the already-2880 swarm-services-cert:
nginx got "Dependency failed" 1ms after the first failure, and
switch-to-configuration exited before the first restart was scheduled.
The comments in lib/store-retry.nix, glue-matrix-bao-token.nix,
glue-queue-agent-credential.nix, swarm-otel.nix and swarm-grafana.nix
now say that, and so does docs/swarm/credentials.md.
Because dependents start after one attempt, a consumer that loads its
credential at start never sees a value a later attempt lands, or a
rotated one. nix/host-modules/lib/refresh-consumer.nix adds
`secret_differs` and `refresh_consumer`, and the four fetch units whose
consumers take a start-time copy call them after the write, only when
the value changed:
- swarm-bao-matrix-token -> tuwunel.service in hive-matrix
- swarm-bao-otel-oidc -> opentelemetry-collector.service in swarm-otel
- swarm-bao-grafana-oidc -> grafana.service in the grafana container
- swarm-bao-forwarder-oidc -> opentelemetry-collector.service in swarm-bao
A running consumer is try-restarted, a failed one is reset and started,
all with --no-block. Inline in the fetch script rather than a
PathChanged path unit because the fetch script is the only writer and
already knows whether the value changed, and it is the same shape as
this PR's nginx hook and swarm-bao-nats-tls's restart of nats.
module-eval-bao-grants gains one case per consumer.
Refs #4662
Six credential-fetch units retried 4 times at 15s, so an apply during
which the store or gateway was down for more than about a minute left
them in start-limit-hit, and nothing started them again once the store
came back. The swarm-services leaf could also land after nginx had
already given up on it, and the hook that propagates a new leaf only
reloaded a running nginx, so a stopped one stayed down until a second
apply.
- nix/host-modules/lib/store-retry.nix: the 2880 x 30s / 25h window
shape swarm-services-cert already had, as one attrset.
- swarm-services-cert, swarm-bao-otel-oidc, swarm-bao-forwarder-oidc,
swarm-bao-matrix-token, swarm-bao-queue-agent, swarm-bao-grafana-oidc,
hive-agent-bao-identity and hive-agent-forge-token use it.
queue-identity.nix no longer has a fetch unit (ccb5bd3b), and
forge-token.nix is a fetch unit with the same short budget that was
added after the census in #4662.
- The swarm-services-cert propagation hook now reset-fails and starts
(--no-block) a loaded nginx that is not active; an active nginx keeps
the re-import + reload.
- module-eval-bao-grants: one case pinning the shape on every host-side
fetch unit, swarm-services-cert included.
Refs #4662
Two Grafana-managed rules in a "hyperhive" folder, one group evaluated
every 1m, each a LogsQL stats count over the VictoriaLogs datasource:
- forge reconcile failing: `swarm forge objects: write failed; retrying
next pass` over 10m, for 15m. The pass runs every 5m, so one failed
pass stays under `for` and a pass failing on every tick fires.
- swarm terminal publish skipped: `swarm terminal: publish skipped` over
15m, for 5m, one instance per hive/agent.
No contact point or notification policy. Grafana 13 routes to its
built-in `empty` receiver when none is provisioned, so firing rules
are visible under Alerting and sent nowhere, with no send errors.
Refs #4717
hive-ci, hive-forge, hive-matrix, swarm-authelia, swarm-bao, swarm-grafana,
swarm-nats, swarm-otel and swarm-victorialogs now import
./swarm-container.nix and drop their own copies of the stateVersion,
firewall and resolvconf lines. Each binds privateNetwork once in its
top-level let and passes it to both the host attr and the in-container
option, as swarm-victoriametrics already does.
hive-forge (25.11) and swarm-otel (the host's value) keep their own
stateVersion over the module's mkDefault. hive-ci sets privateNetwork =
true and writesOwnResolvConf = false, which leaves its firewall and
resolvconf on, as before. hive-matrix keeps its useHostResolvConf
override and static resolv.conf; its resolvconf mkForce now comes from
the module default.
Every container's system.build.toplevel drvPath and host-side attrs
evaluate identical to the parent commit.
module-eval-swarm-services-switch gains a fixture with all ten service
containers and checks that each one's in-container privateNetwork equals
its host-side value, that the nine on the host netns run no firewall or
resolvconf, that hive-ci keeps both, and that hive-forge keeps its pinned
stateVersion.
Refs #3773
The pipe contract had a gap: if a producer piped into
atomic_write_secret exited non-zero after writing partial output,
cat still saw a clean EOF and wrote that partial content through to
the live target via mv — pipefail only reported the failure
afterward, once the bad write was already committed. The helper now
takes the value as its 4th argument and writes it itself with printf
(a shell builtin, so the value never touches an external process's
own argv/environ, same as a function argument never does), so there
is no pipe left to fail silently.
Callers that compute the value with a command now capture it into a
variable first (`value=$(cmd)`), which fails under `set -e` before
atomic_write_secret is ever called — swarm-bao.nix's pin.env site is
the one that needed this (`pin_env_value="BAO_HSM_PIN=$(cat ...)"`).
All seven call sites converted; output is byte-identical (same
printf '%s\n' framing, now applied inside the helper instead of by
each caller).
Refs #4723
Four host glue units fetched a secret from swarm-bao and rendered it
with `> path; chmod`: a reader racing the write could see a truncated
file, and briefly one at the wrong mode before the chmod landed.
glue-matrix-bao-token.nix, glue-queue-agent-credential.nix (both
files), swarm-grafana.nix and swarm-otel.nix now write to a same-
directory temp file, set its final mode/owner, then `mv -f` it over
the target — a shared `atomic_write_secret` helper
(nix/host-modules/lib/atomic-write-secret.nix) so the five call sites
share one implementation.
The first-boot `/root/.claude` migration in nix/agent-modules/user.nix
wrote its done-marker unconditionally, so a failed `cp` (disk full,
permission error) left the marker behind and no boot ever retried the
copy. The marker is now written only when there was nothing to
migrate or the copy succeeded; `cp -an`'s no-clobber semantics already
make a retry after a partial copy safe.
Refs #4723
`services.hyperhive.enable` and `services.hyperhive.c0re.enable` are gone.
One switch, `services.hyperhive.deploy.hive-controller.enable` (default
false, as the old toggle was), now gates hive-c0re and hive-priv. Both old
paths are `mkRenamedOptionModule` shims in deploy.nix, so a host config
that still sets either evaluates as before and gets a rename warning.
Every other read of the old toggle is resolved, including the 29 made
through the `hyperhiveCfg`/`hiveCfg` aliases:
- Dropped: each swarm service and its glue keeps only its own deploy
toggle (authelia, bao and its PKI glue, grafana, victorialogs,
victoriametrics, the secret publisher, swarm-ca, the OIDC client rows,
the controller/nats/matrix-ctl/publisher/services-issuer identities),
the forge, and the `domain` deprecation warning.
- To deploy.hive-controller.enable: the queue-agent credential reader and
its assertion, which feed hive-c0re and write under its state dir, plus
their policy-order entry; the network identity assertions; hive-tls's
two writes into hive-c0re's environment.
- hive-tls runs where the gateway runs self-signed
(`gateway.enable && useSelfSigned`), not on every host.
- The matrix appservice-token reader and its assertion stay on
`deploy.matrix.enable` plus their client-identity checks. They read
deploy.matrix's token file and registration script; their deploy.bao
inputs are the client-half options a hive sets to read a store it does
not run, so gating on deploy.bao.enable would drop the tested
remote-reader case.
- The `hiveName` assertion moves from hive-network.nix to hyperhive.nix
and fires wherever the hive, the store or the homeserver runs: each
turns the name into an identifier with no fallback.
On a host with `deploy.allSwarmServices` and no hive, the documented
services-host recipe, authelia, bao, grafana, victorialogs,
victoriametrics, the OIDC client rows and the hive CA now render; before,
the old toggle being off left them out.
Refs #4500
