#4756 moves agent client certificates onto a PKI mount of their own,
`pki-agents`, whose root bao generates internally. The unit that sets
that mount up runs as the bao granter, and the granter's policy is only
written while #4754's one-time bootstrap token is in place. Adding these
grants after an operator has done that step would cost a second token
placement, so they go into the granter's policy here, before it.
Six stanzas: enable and tune the mount, list its issuers, read its CA,
generate its root internally, and write `swarm-*` roles on it. No root
delete or sudo: agent cert-auth roles pin that root by value, so
replacing it must not be something a deploy can do.
Adds deploy.bao.agentPkiMountPath (default `pki-agents`), which the
stanzas are rendered from. The module-eval case pinning the granter's
policy now lists all seventeen stanzas, and the "grants nothing outside"
case also refuses the agent mount's root, issue, sign and a roles/*
glob.
Every unit that writes a bao policy or cert-auth role ran only while the
operator-placed bootstrap token existed, and skipped silently otherwise.
The token lives 24h, so on any real swarm a PR adding or changing a grant
deployed with its unit skipped, and each one needed a manual token refresh
(plus a root `bao policy write` when it added a path).
A `bao-granter` principal now writes them. Its leaf is minted by
swarm-bao-pki on the store host (0600 root, never copied off it), and its
policy covers `swarm-*` policies, `swarm-*` cert-auth roles and
`pki/roles/swarm-*` by glob, plus the mount and services-root paths the
controller's unit already used. All ten granting units
(controller, secret-publisher, matrix-ctl, matrix-token, queue-agent,
grafana-oidc, otel-oidc, forwarder-oidc, services-issuer, nats-tls) log in
with it instead of reading the token. They keep the 2880 x 30s retry, now
require swarm-bao-pki, and when the store refuses the granter they fail
and print the one-time step instead of skipping.
swarm-bao-granter-role is the one unit left on the token. It enables the
auth mounts (moved out of the controller's unit) and writes the granter's
own policy and role. The bootstrap policy is renamed `bao-bootstrap` and
shrinks to those five stanzas; it is shipped at
/etc/hyperhive/bao-bootstrap-policy.hcl. The old name `swarm-bootstrap`
matched the granter's own `swarm-*` glob.
The granter's CN joins certAuthCns, so no hive can be named into its role.
An assertion keeps both pki role names under `swarm-`. With no client CA
the granting units no longer render, and a warning says so.
module-eval pins the granter's policy stanza by stanza, what it cannot
reach, that every call a granting unit makes is granted, and that only
swarm-bao-granter-role reads the token.
Refs #4704
/etc/tmpfiles.d/hyperhive-agents.conf was a boot-time backstop (#2290)
that pre-created every agent's bind sources. The start preamble already
creates them for every c0re-driven start, and on this host only hive-c0re
starts agent containers. The file was also the reason the socket dir's
owner had to be declared there, which is how it spent its life at
`0777 root root` whenever the uid could not be resolved (#4742).
- hive-priv gains `EnsureAgentSocketDir { name }`, called from
`set_nspawn_flags` in every start path. It creates
`/run/hive-agent/<name>` `0751 root:root` with mkdirat relative to an
O_DIRECTORY|O_NOFOLLOW fd for the parent. An existing entry has to be a
directory (fstatat AT_SYMLINK_NOFOLLOW); anything else is refused, and a
directory is left alone. hive-c0re's own create_dir_all went: its /run
is read-only under ProtectSystem=strict.
- The container's `hive-agent-user-migrate` activation chowns that dir to
the agent user and sets 0751, the same way it already handles state/ and
harness/. It refuses a symlink or non-directory there, since `test -d`
and chmod follow links. No host-side passwd parse, and no window where
the dir is world-writable.
- `/run/hyperhive/agents/<name>` stays created by hive-c0re itself
(`ensure_agent_runtime_dir`). It holds the `mcp.sock` that hive-c0re
binds as hive-core, so it must not become root- or agent-owned.
- The `/run/hive-agent` parent is declared in hive-priv.nix, `0755
root:root`, instead of hive-gateway's hive-core rule. hive-priv is its
only writer now, and hive-priv's ReadWritePaths needs it to exist.
- The manager start in `ensure_root_agent` now goes through
`converge_start_preamble` + `start_with_fallback`. It was a bare start,
so after a reboot the manager's bind sources existed only because of the
tmpfiles file, and its limits drop-in did not exist at all.
- Removed: `sync_tmpfiles`, `agent_uid_gid` / `parse_passwd_uid_gid`,
`priv_client::sync_agent_tmpfiles`, `AgentTmpfilesEntry`, the tmpfiles
body builder and their tests, plus the three call sites.
- Legacy: hive-priv unlinks the file at every start, ignoring ENOENT.
`SyncAgentTmpfiles` stays one release as a payload-ignoring variant that
does the same unlink and returns Ok, for an older hive-c0re.
Salvaged from #4752: the boundary.md correction that nginx only dials,
because ProtectSystem=strict makes its /run read-only.
Behaviour change: a manual `nixos-container start h-<name>` right after a
reboot, before hive-c0re has started that agent, now fails on a missing
bind source instead of starting.
Closes#4742
`checks.module-eval-core-toggle` is red on main: "a gateway's services
leaf asks for only the swarm names that host fronts" read its leaf
request off `bare`, on the premise that `bare` fronts forge. That was
true where the property was written, before 978164dc (#4705) defaulted
`deploy.forgejo.enable` to false. The two merged in sequence, and on main
`bare` renders only the `_` and `h1.t.local` vhosts, so its
`localServiceDomains` is [] and the script carries `alt_names=''`,
never `alt_names=forge.t.local`.
The narrowing itself is right: a host that runs no forge fronts no
swarm name and asks for no services leaf. Only the fixture was stale.
The property now reads a hive with `deploy.forgejo.enable = true`,
whose request is `common_name=forge.t.local alt_names=forge.t.local`
while `auth.t.local` stays in the swarm-wide set.
The renewal timer from 0649673e is not involved: the check's derivation
is identical at 0649673e and at its parent, and ed2ec52f replayed onto
978164dc^ holds while replayed onto 978164dc it fails.
Refs #4587
`services.hyperhive.enable` and `services.hyperhive.c0re.enable` are gone.
One switch, `services.hyperhive.deploy.hive-controller.enable` (default
false, as the old toggle was), now gates hive-c0re and hive-priv. Both old
paths are `mkRenamedOptionModule` shims in deploy.nix, so a host config
that still sets either evaluates as before and gets a rename warning.
Every other read of the old toggle is resolved, including the 29 made
through the `hyperhiveCfg`/`hiveCfg` aliases:
- Dropped: each swarm service and its glue keeps only its own deploy
toggle (authelia, bao and its PKI glue, grafana, victorialogs,
victoriametrics, the secret publisher, swarm-ca, the OIDC client rows,
the controller/nats/matrix-ctl/publisher/services-issuer identities),
the forge, and the `domain` deprecation warning.
- To deploy.hive-controller.enable: the queue-agent credential reader and
its assertion, which feed hive-c0re and write under its state dir, plus
their policy-order entry; the network identity assertions; hive-tls's
two writes into hive-c0re's environment.
- hive-tls runs where the gateway runs self-signed
(`gateway.enable && useSelfSigned`), not on every host.
- The matrix appservice-token reader and its assertion stay on
`deploy.matrix.enable` plus their client-identity checks. They read
deploy.matrix's token file and registration script; their deploy.bao
inputs are the client-half options a hive sets to read a store it does
not run, so gating on deploy.bao.enable would drop the tested
remote-reader case.
- The `hiveName` assertion moves from hive-network.nix to hyperhive.nix
and fires wherever the hive, the store or the homeserver runs: each
turns the name into an identifier with no fallback.
On a host with `deploy.allSwarmServices` and no hive, the documented
services-host recipe, authelia, bao, grafana, victorialogs,
victoriametrics, the OIDC client rows and the hive CA now render; before,
the old toggle being off left them out.
Refs #4500
Every gateway asked the store's `pki/issue/swarm-services` for the whole
swarm's service set, so a private key on any gateway host could serve a
valid certificate for services that host does not front and never has.
`swarm.localServiceDomains` derives the per-host subset by filtering
`swarm.serviceDomains` against the vhosts this host actually renders —
the deploy flags those vhosts are already guarded on, read once rather
than copied into a second filter. The leaf request and the coverage
guard that decides whether to re-issue both read it, so they cannot
disagree about which names the leaf owes.
The sub-CA's name constraint and the role's `allowed_domains` stay the
swarm-wide set: every host's subset is inside it, and narrowing the
constraint per host would turn one signing into N.
The store's `swarm-services` role issues the services leaf for 720h, and
`swarm-services-cert` only ever ran at boot or rebuild: it is a
`RemainAfterExit` oneshot wanted by `multi-user.target` and no timer
targeted it. A hive not rebuilt within 30 days served an expired leaf.
`swarm-services-cert-renew` runs the same script from a daily timer. It
is a unit of its own because a timer starting the `RemainAfterExit` unit
is a no-op, and restarting that unit instead would propagate through
`hive-gateway-self-signed-cert`'s `Requires=` to nginx, so a sealed store
would take the gateway down over a still-valid leaf. Nothing requires or
orders against the new unit; it has no `Restart=`, so a failure stays in
`systemctl --failed` until the next tick, and the script only moves files
into place after the store has answered.
The re-issue threshold was `checkend 2592000`, the whole 30-day
lifetime, so every run re-issued. It is now half the role's lifetime,
read from a new internal option `deploy.bao.servicesPkiLeafTtlHours`
that the role's `ttl`/`max_ttl` also read. Boot and timer share the
script and so the threshold. The services-root re-check reads the
same option, at the store's own replacement threshold (hours × 3600),
so the hive asks for a new leaf when the store replaces its root. A
`flock` keeps the two runs from interleaving one issuance's key with another's leaf.
`checks.module-eval-hive-tls` pins the timer, that the unit it starts
re-runs the issuance without `RemainAfterExit`, that nothing depends on
it, and that both the leaf and root thresholds move with the option.
Closes#4587
#4708 defaulted deploy.forgejo.enable to false, and the hive-forge module's
whole config block (including the mirror-declared-without-a-controller
warning) now hangs off it. controllerWithMirrors and mirrorsNoController
both relied on the old always-on default to reach that warning path.
The orgs agent-configs/internal/agents (plus mirror owners), the
operators team in agents and agent-configs, the pull-mirrors,
internal/docs, internal/knowledge (public, README-seeded) and the
agent-configs org avatar are one set per forge. hive-c0re ensured them in
its boot sweep, as the core admin, and only on the hive co-located with
the forge container.
swarm-controller now reconciles them at start and every 5 minutes
(forge/objects.rs: observe -> pure plan -> apply). A failed object logs
a warn line plus a pass summary and is retried next tick. create_repo
ensures the agent-configs org and its operators team first, so a config
repo's merge gate never depends on the periodic pass having run.
hive-c0re drops ensure_org, SEEDED_ORGS, ensure_mirrors/ensure_mirror_repo,
ensure_operators_team, ensure_shared_docs_repo, ensure_knowledge_repo/
set_repo_public, seed_readme, ensure_config_org_avatar and the one-shot
knowledge::remove_webhook cleanup, with their now-unused helpers.
nix: the mirror list moves from the hive-c0re unit
(HYPERHIVE_FORGE_MIRRORS) to the swarm-controller unit
(SWARM_CONTROLLER_FORGE_MIRRORS), with an eval warning when mirrors are
declared on a host that runs no controller. c0re.orgAvatarPng is renamed
to deploy.swarm-controller.configOrgAvatarPng.
Refs #3782
The daemon reads each account's token from `swarm/agents/<agent>/matrix/`
as the agent itself, inside its own container, and falls back to the file
only when the store has none. This is #4519's read, without its `main`
carve-out: the swarm now mints `main` there and no hive writes the file.
The daemon unit gets the agent's store identity, spelled the way
forge-token.nix spells it. A timer re-starts it while it is down: a token
the swarm mints or replaces in the store changes no file, so the path
watcher never fires for it, and a daemon that exited on a replaced token
would otherwise stay down until the container restarts.
A `MintAgentMatrixAccount` node creates the agent's account on the swarm's
homeserver with the swarm appservice token, stores its token at
`swarm/agents/<agent>/matrix/main`, and reads it back with whoami before
reporting success. It is a root of agent creation, `after_any` into the
deploy, and a five-minute backfill over every agent with a store identity
queues the same node — the shape of the forge-token mint.
The decision reads the stored token back rather than only checking that one
is stored: the swarm and a hive both pin the device `hyperhive-<agent>`, so
each login replaces the other's token. A failed read plans nothing, so an
outage never rotates every agent's token.
`matrixHomeserverUrl` now defaults to the swarm's `chat.` vhost, since the
mint is what consults it.
The matrix container renders the swarm registration before tuwunel starts
(`requiredBy` it, no network), tuwunel loads it as a second `.yaml`
credential, and a publish unit hands its token to the store. tuwunel 1.9.1
refuses only a duplicate id or as_token, not overlapping non-exclusive
namespaces, so it sits beside the hive's `hyperhive` registration.
`admin_execute` promotes exactly `@swarm` at boot. The module-eval pin
narrows from "no account" to that one list, compared whole so a second
entry fails; `admin_execute_errors_ignore` is pinned too. matrix-ctl may
write the swarm token's leaf, and the controller may only read it.
Everything is gated on matrix-ctl's store identity: with nobody to publish
the token, the registration would be an admin credential nobody reads.
bao's own in-container collector labelled every log line and metric it
forwards `service.name=swarm-bao` (`processors.resource.attributes`,
keyed off `swarm.bao.machine`), while every panel in the shipped
Grafana bao dashboard queries the literal `service.name="bao"` — no
panel has matched since the scrape moved into that collector.
Rename it in the collector instead of templating the dashboard: a new
`collectorServiceName` binding in swarm-bao.nix, deliberately not
`cfg.machine` (that value names the container/receiver, not bao's
display identity), stamps `service.name="bao"` directly. bao.json is
back to its origin/main shape, unchanged.
Adds a module-eval case to checks.module-eval-grafana that reads the
collector's own evaluated config and asserts its service.name matches
every selector the shipped dashboard uses; verified invert-proof by
setting the value back to cfg.machine and confirming that specific
case (and only it) fails.
[oauth2_client] turns on auto-registration through the authelia login
source, with the account named after authelia's preferred_username.
DISABLE_REGISTRATION stays true: forgejo 16's auto-registration checks
only ALLOW_ONLY_INTERNAL_REGISTRATION, so local sign-up stays off.
ACCOUNT_LINKING is `login`, forgejo's default, set explicitly. With
`auto`, an SSO login whose name matches an existing local account would
be handed that account, and agents, `core` and `swarm-controller` all
have one. `login` asks for that account's own password instead.
Refs #3782
--link-journal=host bind-mounts a host directory journald never
writes into for this container (empty, root:nogroup, confirmed on
the live host — #4527). Dropping it falls back to nixpkgs' default
--link-journal=try-guest, the same shape every agent container
already uses, so the in-container collector's journald receiver
(directory = /var/log/journal) now reads a journal that is actually
written.
merge stays true and services.hyperhive.swarm.otel.journaldUnits is
untouched so this deploy changes exactly one thing; comments that
described the old host-linked shape are rewritten to match.
Adds a module-eval assertion (bao-otel-collector.nix) that the
container's extraFlags never re-add --link-journal=host.
Refs #4527, #4499.
`swarm-bao-forwarder-oidc` fetches the store container's collector secret,
one path, and was the last reader still logging in with
`deploy.bao.clientCertFile`: the hive's own leaf, whose policy reads every
agent's credentials, the hive's tree and every service's OIDC secret. The
four-way split gave grafana's and the swarm collector's readers leaves of
their own and left this one behind.
It now holds `forwarder-oidc.pem`, minted by `swarm-bao-pki`, and logs in
under the `swarm-forwarder-oidc` cert-auth role, whose policy reads
`secret/data/swarm/services/<store forwarder client id>/oidc/client` and
nothing else. The role is written by `swarm-bao-forwarder-oidc-policy`
from the bootstrap token, which gains the two grants that unit calls, and
the reader is ordered after it. The subject is reserved as a hive name. A
store host whose pair is null is refused at eval rather than falling back
to the hive's leaf.
The hive's own role and `client.pem` are untouched; nothing is revoked.
Every hive with hyperhive enabled ran its own hive-forge container, and
its gateway answered forge.<swarm> with its own bridge IP, so on a
multi-host swarm each hive talked to its own forge.
deploy.forgejo.enable defaults to false and allSwarmServices sets it with
mkDefault, like authelia and bao; singleHostSwarm gets it through that.
The forge's OIDC client moves to a glue module gated on authelia, so a
split authelia/forge swarm still registers it. CI now requires the forge
on the same host, and the controller's forgeTokenFile defaults to null
where the forge is not.
Closes#4705
Refs #3782
A remote hive dialled nothing until an operator copied the queue's URL
into it, though the URL is the same string everywhere. statusPublish.natsUrl,
queue.agentNatsUrl and controller.queue.natsUrl now default to
tls://<swarm.nats.domain>:<port> unconditionally.
The statusPublish assertion treated a URL without a secret as a half
config. With the URL a default on every hive, only the secret claims
publishing: the assertion now refuses a secret without a URL or token
endpoint, and hive-c0re's status environment is gated on the secret too,
so a hive without one publishes nothing instead of reading a missing
credential.
swarm-bao-nats-tls-policy acts with the bootstrap token, and main's
module-eval-bao-grants now fails any such unit whose calls the policy
file does not grant. Adds its three paths and counts it among the units
the check must see.
The queue listened in plaintext on 4222, reached by bridge IP or loopback,
and nothing in-tree opened it to another hive. It now has a name, serves a
certificate for that name alone, and refuses clients that do not speak TLS.
- `swarm.nats.domain`, default `nats.<swarm.domain>`, a sibling name like
`swarm.bao.domain`. The queue host answers it via `gateway.localNames`;
every other hive resolves it through the operator's DNS, as for bao.
- `pki/roles/swarm-nats` allows that one name (bare domain, no subdomains,
IPs or localhost, server flag). A `swarm-nats` cert-auth role and policy
may only `update` `pki/issue/swarm-nats`, written by
`swarm-bao-nats-tls-policy`. The login leaf is minted by glue-bao-tls and
paired by glue-nats-bao-identity. `deploy.bao.natsCommonName` is reserved
as a hive name.
- `swarm-bao-nats-tls` issues the leaf into a directory bound read-only into
the container, restarts nats when it rotates, and re-runs daily.
It joins glue-bao-readers-policy-order, so it is ordered after its policy
unit (`after` and `wants`, never `requires`) where the store is on the
same host. The policy unit joins the store's journald list.
- nats gets `tls {}`, with the key via `LoadCredential`, and no
`allow_non_tls`. `validateConfig` is now off in every mode, because the
build-time check loads a leaf that only exists at runtime.
- 4222 is also open on `wg-hive` when the host is on the mesh, never
host-wide.
- `statusPublish.natsUrl`, `queue.agentNatsUrl`, the controller's URL under
`singleHostSwarm`, and the auth responder all dial
`tls://<swarm.nats.domain>:<port>`. swarm-queue-client hands its CA file
to the NATS connection too, so hive-c0re and the controller trust the
leaf's root.
- docs/swarm/README.md: the queue URL and the one DNS record a multi-host
swarm needs.
module-eval-nats-tls pins the role, the policy, the served leaf, the
firewall, the ordering, and a scan of every `*_NATS_URL` and the
responder's URL across the host and its containers.
Closes#4626
setup.md's copy of the swarm-bootstrap policy still granted only the
controller's first six paths, while eight units now act with the
bootstrap token. The policy moves to
nix/host-modules/swarm-bao-bootstrap-policy.hcl, now covering every path
those units call. module-eval-bao-grants reads that file and fails
when a unit whose script uses the token calls a path the file does not
grant.
The four readers' ordering after their policy units only applies where
the store and that reader share a host, so it is colocation glue and
does not belong in the reader modules (two of which are main modules).
glue-bao-readers-policy-order.nix now sets the after+wants edges, gated
on deploy.bao.enable AND the reader's own gate, so a store host without
a reader gains no stub unit.
The units on the path a deploy takes to TLS, the store's grants and the
swarm collector itself were not on the host collector's journald
allowlist, so an ingest outage one of them caused showed in the store
only as every source going quiet at once.
Each module names its own units, per the option's rule:
- hive-tls.nix: hive-tls-ca, swarm-services-cert
- hive-gateway: hive-gateway-self-signed-cert (self-signed mode only)
- swarm-bao.nix: the seven grant units beside
swarm-bao-services-issuer-policy
- swarm-otel.nix: container@<machine>, and
nixos-rebuild-switch-to-configuration, the transient unit nixos-rebuild
runs the activation in and whose syslog lines carry its status
The module-eval arm pins each unit as both listed and defined, since a
listed name that matches nothing is silent.
The container's collector has never shipped a line. `journalctl --follow`
— which the journald receiver passes unconditionally — scopes itself to
the current boot unless `--merge` is given too, and `--link-journal=host`
makes /var/log/journal the HOST's journal tree, where this container's
current boot has no entry. journalctl exited 1 with "No journal boot
entry found for the specified boot (+0)" and the receiver respawned it
every ~2s, so nothing was ever read and nothing was ever exported.
`merge = true` is the receiver's key for `--merge` (buildArgs() in
pkg/stanza/operator/input/journald/config_linux.go at tag
receiver/journaldreceiver/v0.151.0, the deployed collector's version),
and --merge is what clears the implicit boot scope in journalctl.c
(systemd v260.4, the version on the host).
The module-eval arm pins both halves: the boot filter is gone AND the
directory is still the host-linked one — either alone is satisfiable by
the broken config.
mint_and_verify reads the queue credential back before writing it, so a
re-run keeps the value a live agent already authenticates with. that read
is read_optional, which maps only a 404 to absence — so with create/update
alone every mint aborted on a 403 at its first store read.
read on the same paths the stanza already grants create and update, and
nothing else: no list, no delete, no patch.
Every deploy on a hive host, the replacement opentelemetry-collector
reaches bind() while the outgoing process still holds
127.0.0.1:8888 (its self-scrape endpoint). nixpkgs sets
Restart = "always" with no RestartSec, so the unit spends its five
default attempts in under two seconds, hits start-limit-hit and stops
retrying — ~27s of telemetry blackout per deploy.
RestartSec = 5 with a 12-attempt burst over a 120s window rides the
race out instead: the blackout ends within one interval of the port
coming free, and 55s of it being held is survivable where 2s was not.
StartLimitBurst/StartLimitIntervalSec go at the systemd.services attr
level, which NixOS renders into [Unit]; under serviceConfig systemd
ignores them silently. module-eval-hive-otel asserts the placement.
The four-way client-cert split gives each store reader its own leaf, and
three of the four readers render only where their own leaf exists. On a
host that mints its own PKI glue-bao-tls.nix defaults all eight, so there
is nothing to do; on a hand-configured remote-store hive, omitting one
pair used to mean that unit silently did not render — a privilege-
narrowing unit absent from a green build, with the missing unit as the
only evidence.
Each of the three now asserts its own pair, shaped after
swarm-grafana.nix's haveClientIdentity assertion and named to the pair it
needs. What differs from Grafana's is the gate: these fire only where the
host demonstrably reads the store (it holds deploy.bao.clientCertFile and
clientKeyFile) and the consumer is on. A host with no store identity is
the supported no-store deployment and still evaluates; the collector's
no-secret degrade is untouched, because that host holds no clientCertFile
either.
Also rewords three passive-voice sentences in docs/swarm/secrets.md that
vale flagged, and documents what the refusal costs and where it stays
silent.
Four units read one path each out of the store, and all four logged in
holding `deploy.bao.clientCertFile` — the hive's own leaf. Bao identifies
a principal by the subject of the certificate it presents, so four
readers behind one certificate were ONE principal, and the only grant
expressible was the union of what the four need: read on
`swarm/agents/*`, `swarm/hives/<hive>/*` and `swarm/services/*`. The unit
fetching Grafana's OIDC client secret could fetch every agent credential
in the swarm; the one fetching this hive's matrix token could fetch
Grafana's. Least privilege was not misconfigured here, it was
unrepresentable.
Each now holds a leaf, a cert-auth role and a policy of its own, and each
policy is the single `secret/data/…` path that unit's own script names —
spelled to the leaf, not to a prefix, the way matrix-ctl's already is.
Following the four exemplars in-tree rather than building a mechanism:
`signLeaf` mints the leaves, `swarm-bao.nix` writes the roles from the
bootstrap token, the consumers name their own pair.
Two of the four are written PER HIVE and two are not, which is the shape
of the paths rather than a preference. A matrix appservice token and a
queue credential live under `swarm/hives/<name>/` and every hive runs a
reader for its own, so one role for all of them would have to be granted
`hives/*` — letting one hive read another's, a reach no hive has today.
An OIDC client secret lives under `swarm/services/<client-id>/` and a
swarm registers each exactly once, so one role each is enough. The
per-hive subjects are `<prefix>-<hive>` and swarm.nix reserves every
composed spelling as a hive name, so a hive cannot be named into another
hive's role.
The shared leaf stays: hive-c0re still passes it into its container, the
`bao` CLI wrapper still defaults to it, and the three
`glue-*-bao-identity.nix` files derive the PKI directory from it.
module-eval-bao-grants gains a negative arm per principal — each pins the
three stanzas the hive's leaf carried and the two wildcards a later
widening would reach for, so a policy that grows fails here rather than
in a store. Plus the consuming side: repointing a unit back at the hive's
leaf would evaluate, deploy and log in, and silently restore the union.
A hive that reads a store on another machine now places one leaf per
principal instead of one shared by four. That cost is the point, and
docs/swarm/secrets.md lists the pairs.
Every agent on a hive authenticates to the swarm queue with the same
hive-scoped OIDC client, so at the auth callout one agent is
indistinguishable from its co-hived neighbours. The commit before this
one mints a secret per agent at swarm level into
secret/swarm/agents/<agent>/queue; nothing read it.
Read it here, and read it from the container itself. A hive courier in
the path would be the hive vouching for which agent this is, which is
the property a per-agent credential exists to remove -- so the agent
logs in to the store with the certificate hive-agent-bao-identity
already proves it can log in with, and reads its own path. The store
certificate is for reaching the store and nothing else: what the new
unit writes to /run is the secret it read back, and nothing hands a
BAO_CLIENT_* path to anything queue-shaped.
The read needs no policy change. render_agent grants read on
secret/data/swarm/agents/<agent>/*, which covers this path and the
bao-mtls one beside it alike -- which is also why this unit degrades
where the identity check fails. A refusal this unit sees and that check
did not cannot be a policy that drifted; it is an object not yet minted,
the ordinary state of every agent created before its swarm knew to mint
one.
The harness resolves the path and reports which credential this agent
can present. It does not yet present it: the auth-callout responder
still verifies only the hive-scoped token, and an agent offering a
credential nothing on the other end reads back would simply be refused.
Teaching swarm-nats-auth to read the same path is the next slice.
Review response on #4620: not having SSO is not a supported
deployment, so the type should not permit it, and the docs paragraph
explaining why SSO is always present is redundant once the type says
so.
- swarm.authelia.url drops types.nullOr.
- Every consumer's null-arm is gone: two option defaults
(swarm-controller's and swarm's own statusPublish.tokenEndpoint)
that produced an empty/null placeholder when the URL was null now
unconditionally compute the real derived URL. Five now-dead
"assertion = ... != null" guards (swarm-authelia's bridge,
swarm-grafana, swarm-otel, swarm-nats, hive-forge, hive-matrix) are
removed as unreachable — in every case the same URL was already
interpolated unconditionally a few lines below the guard.
- grafanaNoSso, the module-eval fixture whose sole purpose was
exercising the now-unsupported no-IdP refusal, is removed along
with its dedicated test case; swarm.authelia.url = null is a type
error now, not a value that reaches that assertion.
- docs/swarm/services.md: cut the clause about setting the option to
null and the sentence explaining why the URL is co-location-
independent — both redundant now that the type enforces it.
`swarm.authelia.url` defaulted to `https://<domain>` only when this host
ran the container, and to `null` otherwise — so the address a client is
given was a statement about co-location rather than about the swarm. A
swarm has one SSO provider; every hive addresses the same name and
resolution decides which address that reaches, exactly as
`swarm.otel.domain` already works.
The option stays nullable: "this swarm has no IdP" is still expressible,
it is just now something an operator states rather than something not
running the container produces. The Grafana fixture that exercised the
no-IdP refusal says it explicitly.
Closes#4536
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.
What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.
Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.
The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.
Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.
The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
Both earlier versions asked the wrong host. `hyperhive.otel.enable` asked
whether this host runs a HIVE collector; `deploy.swarm-otel.enable` asked
whether this host runs the SWARM one. Neither answers the question the
forwarder actually has — "is there a collector to forward to" — and that
question cannot be false: a swarm always runs at least one instance of
every swarm-level service. So the forwarder renders under the condition
already enclosing it, that the store is deployed here, and nothing else.
`scrapeHere` deliberately keeps its `deploy.*` gate one line up. It is a
loopback metrics listener, which genuinely only works where the scraper
is — the two are different tiers, and the name says so.
The module-eval case that pins it is the split topology: the swarm
collector on another host, nothing local naming it, and the forwarder
still enabled and still addressed at `swarm.otel.domain`'s route. Both
removed gates render nothing in that fixture, which the co-located ones
they shipped with could not show.
The forwarder pointed at the hive bridge address and was gated on the
hive's `otel.enable`, so it existed only where a hive collector stood
beside it. It now exports to `swarm.otel.domain` — the gateway-served
name that resolves locally when co-located and over the network
otherwise — on the swarm tier's own producer route, and is gated on
`deploy.swarm-otel.enable` like its sibling `scrapeHere`.
Refs #4526
Every container is supposed to run a collector that passes its logs and
metrics to the next hop. swarm-bao did not: its journal reached the store
only because `--link-journal=host` puts it in the host tree, where the
swarm collector — a different container — reads it through a unit
allowlist. That is the topology being retired, and in this deployment it
delivers nothing: no `_SYSTEMD_UNIT` value in the seven-day store mentions
openbao at all.
So the store's container now runs its own journal forwarder, copied from
an agent container's (nix/agent-modules/otel.nix): the whole journal, no
unit allowlist, pushed to the same first hop every agent on the host
already exports to. A local collector reads the local journal, so there is
nothing for a list of unit names to disagree with.
The `swarm.otel.journaldUnits` entry and `--link-journal=host` both stay.
Every sibling swarm container still rides the shared collector, and they
come out once each of them has a forwarder of its own.
Closes#4526
A swarm runs one homeserver and every hive on it logged in as the same
`@hive:` localpart, holding the same access token out of one swarm-wide
store path. That is one matrix identity for N hives: the homeserver
cannot attribute an action to the hive that took it, and revoking one
hive's standing revokes every hive's.
Three changes, and the third is the one that makes the other two real:
- **The localpart carries the hive's name** (`hive-<hive>`), derived in
one place, `swarm_secret_client::matrix::hive_localpart`.
`hive-matrix.nix` renders the same string as the appservice
registration's `sender_localpart`, so the shared account stops being
created rather than merely stops being used.
- **The store path is templated by hive**, not a constant. The
"a swarm runs one homeserver, so this is a constant rather than a
parameter" rationale went with it; it stopped holding the moment two
hives shared the homeserver it describes.
- **The path moved out from under the grant every hive has.** It sat at
`swarm/services/matrix/sender-token`, inside the
`secret/data/swarm/services/*` read stanza `policy::render` gives every
hive. It now sits under that hive's own stanza,
`secret/data/swarm/hives/<hive>/*`, which interpolates the reader's
name — so a hive reads its own token and is refused another's. The
policy renderer itself is unchanged: narrowing the `services/*` grant
would break the OIDC-secret read it exists for, and moving the
credential is what this needed instead. A policy test walks the
rendered stanzas and asserts none of hive alpha's covers hive beta's
sender token, so a later stanza that widened it fails here.
`swarm-matrix-ctl` takes a new required `MATRIX_MINT_HIVE` and writes
that hive's path; its store grant in `swarm-bao.nix` follows, scoped to
one hive's leaf via the new `deploy.bao.matrixCtlHiveName` (defaulting to
this host's `hiveName`) rather than a `hives/*` wildcard, which would
hand the matrix container every hive's token back.
Migration: no outage at deploy. `ensure_hive_user` short-circuits on the
local token file, so a hive keeps running on what it has; with no such
file it reads the new per-hive path, finds nothing, and falls through to
the existing register-or-appservice-login ladder against its own
localpart — which needs only the per-hive `as_token` on local disk. The
old shared object is read by nothing afterwards. Rooms do not follow the
identity, and that is the one operator step; both ways out are written
into `docs/integrations/matrix.md`.
No admin standing is granted to the per-hive accounts: `admin_execute`
stays empty and the assertion pinning it is untouched.
Renames `swarm-matrix-minter` and reshapes it around subcommands. Minting
is now `swarm-matrix-ctl mint`.
Running rust inside `containers.hive-matrix` is not free: it needs its own
store identity, its own cert role and its own bind mounts, and every one of
those is per-*container*, not per-task. A second single-purpose crate would
have had to duplicate that plumbing to add one action, so the next thing
that has to run in there should be a verb here rather than a new crate.
The old name guaranteed the opposite.
`main.rs` is clap dispatch; the minting logic moves to `mint.rs` unchanged.
A bare invocation is refused: `mint` writes a credential, so "no verb"
defaulting to it would make a typo in the unit mint rather than fail.
The environment prefix moves with it, `MATRIX_MINTER_*` → `MATRIX_MINT_*`.
Scoped to the verb and not to the binary, because a binary-scoped prefix is
one the next verb has to share or widen, and a widened one never narrows
again. A test asserts every variable carries the verb's prefix.
The principal renames too. The cert role, bao policy, granting unit, leaf
filename and `certAuthCns` entry all have to spell one string the same way,
so leaving them as `swarm-matrix-minter` would have rebuilt the naming
split this branch exists to remove. Renaming the nix options alongside is
free here: every one of them is introduced by this PR and has never been
released, so no operator config names them yet.
`ExecStart` now names the verb, which is a contract between a nix string
and a clap enum that fails at deploy time with no local signal. Both ends
assert it: `mint_is_spelled_the_way_the_unit_invokes_it` in the crate, and
a new module-eval arm reading the rendered `ExecStart`.
docs/getting-started/setup.md drops the sender token from its "live on the
host" list: setup does not touch this credential, so a setup guide has no
reason to name it.
The store path and every identifier around it called this an admin
token. It is not one: of ~15 hive-c0re call sites only two need
homeserver admin, and the homeserver no longer promotes the account at
boot, so the name overstated both what the credential is and what it may
do.
Renaming it to the account was not enough either. "The `@hive:` token"
reads as the token of a hive user, and no such user is provisioned —
`@hive:<server_name>` is the appservice registration's own
`sender_localpart`, an account the homeserver creates for itself when it
loads the registration.
So it is the **sender token**: the matrix appservice sender account's
access token, at `swarm/services/matrix/sender-token`. The name says
what it authenticates as rather than what it may do, which is the part
that was wrong.
The path has one constructor, and the bao grant, the grant assertion and
three unit tests pin its literal independently — so a half-finished
rename fails a check rather than leaving the minter and its readers
disagreeing at runtime. `tracing` messages are renamed with the code, so
the journal reads the way the source does.
The host-side file keeps its name (`matrix/access-token`): it carried no
admin framing, and renaming it would orphan the file on every deployed
hive for nothing.
`docs/tools/hivectl-cli.md` is regenerated from the clap tree.
A swarm runs one homeserver and a homeserver has one appservice sender
account, so "mint it once" is a property of the thing being minted
rather than something a lock has to enforce. That is what makes this
account the one to move first: no trigger route, no controller change
and no agent list — a boot-time oneshot beside tuwunel is the whole
mechanism.
`swarm-matrix-minter` runs inside `containers.hive-matrix`, which
already holds the appservice token: the rendered registration is bound
in read-only because that is how tuwunel is handed it. What the
container lacked was an identity of its own, so this adds one — a leaf
from the store's CA with a grant of exactly one path, not the hive's
leaf, which reads every secret in the store.
Both ends of the credential ship here. The minter reads the path it
publishes to before it touches the homeserver, and returning on a
non-empty read IS the "only once"; `hive-c0re`'s `ensure_hive_user`
reads the same path, authenticating with the hive name already in
`HYPERHIVE_HIVE_NAME`. The existing mint-then-`M_USER_IN_USE`-login
ladder stays as the fallback for a store that is empty, unconfigured or
unreachable, which is every swarm deployed before this — so nothing
needs backfilling and nothing breaks if the rest of the sequence never
lands.
The credential is not an admin credential, and is not named like one.
It is the access token of the appservice registration's own
`sender_localpart` — `@hive:<server_name>`, an account the homeserver
creates for itself when it loads the registration. The store path is
`swarm/services/matrix/sender-token`, the host path is
`matrix/access-token`, and the homeserver no longer runs an
`admin_execute` promotion for that account at boot. Everything the hive
provisions with it — the Space, the chat room, their hierarchy and join
rules, the invites — rides on being the creator of those rooms at power
level 100, not on homeserver admin; there is no Synapse admin API here
to need, tuwunel has none.
Two operations do need an admin *sender* and therefore stop working:
`hivectl matrix promote-user` and `hivectl matrix reset-password`, both
`!admin …` messages into `#admins:<server>`, plus the password-reset
recovery path that an agent with a lost password file falls back to.
They are swarm-level operations and are left failing loudly rather than
served by an over-privileged token every other call site would also
carry. The sweep's own admin-rights check and self-repair go with them:
an account that is deliberately not an admin has nothing to check.
`ephemeral = false` stays, and hive root can still read the container's
filesystem. Accepted: what this buys is identity separation — no hive
*process* holds or reads the appservice token — not physical isolation.
Refs #4345
Grafana's log-level buttons filter on a field called `level`; no row in
the store has one. The store's name for it is `severity_text`, chosen by
VictoriaLogs' OTLP ingester rather than by us — v1.52.0's
`app/vlinsert/opentelemetry/pb.go` writes it unconditionally and the
ingest parameters have no `_level_field` to rename it with. So the
mapping is made on the reader: the VictoriaLogs datasource, which was
provisioned with no `jsonData` at all, now carries a `logLevelRules`
entry per severity the journald parser can emit.
`logLevelRules` is the datasource plugin's only level-related jsonData
key — there is no field-name setting and no OpenTelemetry preset to
switch on. It is read off `instanceSettings.jsonData` in the plugin's
`datasource.ts` and typed in its `configuration/LogLevelRules/types.ts`,
both recovered from the sourcemap shipped in the pinned artifact
(`grafanaPlugins.victoriametrics-logs-datasource` 0.26.3). Each enabled
rule appends an `OR severity_text:="INFO"`-shaped term to the query a
level button emits, next to the `level:…` term that matches nothing.
A wrong rule here fails silently: Grafana provisions unknown jsonData
without complaint and the buttons go on returning zero rows. The three
ways to get one wrong — a non-literal `enabled`, a non-canonical `level`
spelling, a value that is not the stored text — are recorded at the
binding, and a module-eval arm pins them along with the real failure
mode, a severity added to `nix/journald-severity.nix` and not here.
`Unspecified` is left unmapped on purpose: it is the store's own
rendering of an absent severity and the thing the logs dashboard's "no
severity" panel counts.
Refs #4560
Both journald receivers import one PRIORITY mapping, and each tier's
suite was asserting the whole table against its own receiver. That checks
one file twice: invert the table and two cases fail saying the same
thing, which tells you nothing about which of the two possible defects
you have.
Split by subject instead. The table's contents — the inverted direction,
overwrite_text, parse_from/on_error — belong to the file that holds them,
so they get a suite of their own reading that file directly, with no
fixture at all. Each tier keeps a case, reduced to the question only it
can answer: does MY receiver carry the shared mapping.
Both tier cases stay. They cover different receivers over different
journals — the agent container's own and the swarm collector's host
journal — and one tier quietly losing its parser while the other keeps
one is exactly the half-fixed state worth catching.
Membership rather than equality of the whole operator list, so a tier
that later grows an unrelated operator of its own still passes.
Checked against seven defect scenarios: each fails exactly one case, and
names the right one.
The direction is the part a reviewer cannot check by looking, so it is
asserted at both ends of the table and in both tiers' groups: an inverted
mapping still maps every value to something, and a case that only asks
whether a severity parser exists passes on the exact defect. The reader
that turns a rendered operator list back into a PRIORITY -> name function
lives in lib.nix, since both tiers need it.
The panel is asserted on its query rather than its title, because a panel
that keeps the title and loses the expression renders an empty graph that
looks exactly like zero prioless lines.
The single module-eval derivation forced ~62 full nixosSystem
fixtures live at once to compute its cases list: 10.6GB peak RSS /
5m25s to evaluate, by far the dominant cost in nix flake check.
Splits it into 21 independent checks.module-eval-* derivations
(1-7 fixtures each) sharing builders/helpers via module-eval/lib.nix,
so no single derivation needs more than a handful of fixtures live
at once. A few cases spanning two clusters carry a small duplicated
fixture rather than threading shared state through lib.nix.
