`swarm-bao-forwarder-oidc` fetches the store container's collector secret,
one path, and was the last reader still logging in with
`deploy.bao.clientCertFile`: the hive's own leaf, whose policy reads every
agent's credentials, the hive's tree and every service's OIDC secret. The
four-way split gave grafana's and the swarm collector's readers leaves of
their own and left this one behind.
It now holds `forwarder-oidc.pem`, minted by `swarm-bao-pki`, and logs in
under the `swarm-forwarder-oidc` cert-auth role, whose policy reads
`secret/data/swarm/services/<store forwarder client id>/oidc/client` and
nothing else. The role is written by `swarm-bao-forwarder-oidc-policy`
from the bootstrap token, which gains the two grants that unit calls, and
the reader is ordered after it. The subject is reserved as a hive name. A
store host whose pair is null is refused at eval rather than falling back
to the hive's leaf.
The hive's own role and `client.pem` are untouched; nothing is revoked.
hive-priv writes an agent's matrix-token 0600 and owned by the agent,
so hive-c0re, running as hive-core, cannot read it. The read-based
token-present guard in ensure_user_for therefore never fired, and every
matrix sweep (boot, every 30 minutes, every rebuild) re-minted each
agent's token through the appservice login and restarted its
hive-matrix-daemon.
Decide presence with a stat instead: a non-empty regular file counts as
present. hive-core can stat the file through the 0755 state dir.
Closes#4665
Every hive with hyperhive enabled ran its own hive-forge container, and
its gateway answered forge.<swarm> with its own bridge IP, so on a
multi-host swarm each hive talked to its own forge.
deploy.forgejo.enable defaults to false and allSwarmServices sets it with
mkDefault, like authelia and bao; singleHostSwarm gets it through that.
The forge's OIDC client moves to a glue module gated on authelia, so a
split authelia/forge swarm still registers it. CI now requires the forge
on the same host, and the controller's forgeTokenFile defaults to null
where the forge is not.
Closes#4705
Refs #3782
The stub-client helper in forge.rs built Client { api } after #4703
added a url field, breaking compilation of swarm-controller's test
target on main. Clone the URL the helper already builds for
Forgejo::new so it can also populate Client.url.
Closes#4709
crash_watch's 10s poll and auto_update's ensure_root_agent both read
lifecycle::list().await.unwrap_or_default(), which turned a failed read
into 'zero containers'. In crash_watch that made every previously-running
agent look like it crashed simultaneously (prev.difference(current) over
an empty current), and left prev empty for the next cycle too, so a
second wave of false 'agent logged in' / 'agent needs login' events fired
against the next successful read. In ensure_root_agent it read as
'manager container missing' and called lifecycle::spawn on a manager
that might already exist.
Both sites now treat a list error as its own outcome: log it at warn and
skip the cycle's decision entirely. crash_watch leaves prev exactly as
the last good read produced it. ensure_root_agent attempts no spawn.
Factors each site's decision into a pure helper (plan_cycle /
plan_root_agent) matching the check_not_live / confirm_gone_after_failed_destroy
pattern, with unit tests for the error case, a control for the readable
case, and (for crash_watch) an invert-proof run locally against the old
unwrap_or_default logic before reverting.
The rewritten ruth step skipped her store identity, without which
neither the backfill nor her container's fetch can reach her token. With
the backfill now creating a missing forge user, two commands cover her:
swarmctl agent mint-identity ruth --hive <hive>, then hivectl agent ruth
rebuild so hive-c0re hands the identity to her container.
Refs #3782
An agent with a hive-agent-* store identity but no forge user was
observed as NoForgeUser and dropped by plan(), so it never got a token.
plan() now keeps it, and queue_forge_token_mints inserts CreateForgeUser
ahead of MintAgentForgeToken with after_ok, the edge declare_agent_job
already uses. ensure_agent_user folds an existing user into success, so
the extra node is a no-op for agents that have one.
Refs #3782
credentials.md gains the forge-token row and drops the claim that the
forge token never passes through the store. setup.md says plainly that
an agent spawned on the hive alone, ruth's bootstrap included, now gets
no forge user from anything. CLI references regenerated.
Refs #3782
Delete ensure_user_for and mint_and_persist_agent_token, the user step
of sync_agent (the per-rebuild re-mint, #4644) and of
forge_after_first_spawn, and the hive-priv WriteAgentForgeToken request
that wrote the token into the agent's state dir. hivectl forge
create-user now refuses an agent and points at swarmctl agent
mint-forge-token. mint_token, ensure_user_exists and TOKEN_SCOPES stay:
provision_user_token and the core bootstrap still call them.
Refs #3782
forge-token.nix fetches swarm/agents/<agent>/forge-token under the
agent's own store identity into /run/hive-agent-forge-token/token, and
re-fetches on a timer so a rotation lands. hive-forge, the git
credential helper, hive-forge-notify, forge-avatar-sync and the web UI
read that file first and fall back to <state>/forge-token.
tea-login is deleted: it copied the token into ~/.config/tea, which
docs/swarm/credentials.md forbids for a store secret. hive-forge covers
the same verbs. swarmctl gains agent mint-forge-token.
Refs #3782
A MintAgentForgeToken node mints a fixed-name swarm-agent token with the
admin API, keeps it when the stored value's last eight and the normalised
scopes match the forge's list, and otherwise deletes and re-creates it.
The token is stored at swarm/agents/<agent>/forge-token. Agent creation
inserts the node, and a pass at start and every five minutes inserts it
for every agent holding a store identity whose token is missing or stale.
Refs #3782
disable_repo_creation's ownership guard now matches either the aligned
{agent}@hyperhive.local email or the legacy {agent}@hive.local one
hive-c0re::forge::users::ensure_user_exists used before its own
ensure_user_email alignment pass existed. That pass only runs on the
forge-host hive, and only once it has a core token and has ticked, so a
real pre-existing agent can still carry the old email when this node reads
it. Without this, the guard would bail "not this agent's" on a genuine
agent during that rollout window (argus, round 2).
Verified separately (not a code change): swarm-controller's forge account
is a site admin (created with --admin), and Forgejo's
convert.toUser/ToUser only hides an account's email when the caller isn't
the admin and isn't the account itself (services/convert/user.go), so
user_get already returns the real email regardless of hide_email on the
target account. No endpoint change needed for that half of the review.
disable_repo_creation now reads the account once before PATCHing
max_repo_creation/source_id, and fails the node if the email isn't the
{agent}@hyperhive.local marker create_agent_user itself sets. The 409/422
create-fold (#4681) only proves some account with that name exists, not
that this node created it, so a pre-existing non-agent account sharing an
agent's chosen name could otherwise get locked onto local auth with repo
creation disabled. Leaves the fold untouched (Option B, per atlas/argus on
#4693); the read moves into disable_repo_creation instead.
Also fixes the nix-sandboxed cargo-test check: forgejo_api::Forgejo::new
builds a reqwest client that eagerly resolves TLS roots via
rustls-native-certs even for the tests' plain-http loopback stub server,
which panics with "No CA certificates were loaded from the system" in the
CA-less build sandbox. Gives that check's nativeBuildInputs pkgs.cacert and
sets SSL_CERT_FILE, same pattern this repo's runtime deployment already
uses for the same reqwest/rustls resolution.
`ensure_agent_user` (the `CreateForgeUser` node) created the agent's
forge account but never set `max_repo_creation = 0`, relying on
hive-c0re's per-hive `ensure_repo_creation_disabled` pass to lock it
down later. That pass is being removed (#3507, #4669), and it is the
only guard against an agent token creating, owning and self-merging in
its own repo.
The node now PATCHes `max_repo_creation = 0` via `admin_edit_user`
after the create, on both the created and the already-exists path, and
a refused PATCH fails the node with Forgejo's message. The body mirrors
hive-c0re's `sparse_edit_user_option` (`login_name` + `source_id = 0`,
everything else unset).
The PATCH is unconditional: Forgejo's API never returns
`max_repo_creation` (it is in `EditUserOption` only, not `User`), so
there is no current value to verify against first.
Closes#4689
list_repos_with_open_issues only read one repo_search page (forgejo's
30-row default), so a repo past position 30 in the default alpha sort
silently dropped out of the issue-report/repo-dropdown data source, with
no truncation signal. The comment claiming RepoSearchQuery has no page
field was wrong -- Request::page()/page_size() are generic builder
methods independent of the query struct.
Adds page_search_results, a small paging loop over a fetch closure for
search-shaped (data: Option<Vec<T>>, no header) responses that can't use
the existing .all() helper (that needs a (Headers, Vec<T>) response
shape). Pages until a short page or a 40-page bound, erroring on the
bound rather than truncating again.
Closes#4675
A remote hive dialled nothing until an operator copied the queue's URL
into it, though the URL is the same string everywhere. statusPublish.natsUrl,
queue.agentNatsUrl and controller.queue.natsUrl now default to
tls://<swarm.nats.domain>:<port> unconditionally.
The statusPublish assertion treated a URL without a secret as a half
config. With the URL a default on every hive, only the secret claims
publishing: the assertion now refuses a secret without a URL or token
endpoint, and hive-c0re's status environment is gated on the secret too,
so a hive without one publishes nothing instead of reading a missing
credential.
swarm-bao-nats-tls-policy acts with the bootstrap token, and main's
module-eval-bao-grants now fails any such unit whose calls the policy
file does not grant. Adds its three paths and counts it among the units
the check must see.
The queue listened in plaintext on 4222, reached by bridge IP or loopback,
and nothing in-tree opened it to another hive. It now has a name, serves a
certificate for that name alone, and refuses clients that do not speak TLS.
- `swarm.nats.domain`, default `nats.<swarm.domain>`, a sibling name like
`swarm.bao.domain`. The queue host answers it via `gateway.localNames`;
every other hive resolves it through the operator's DNS, as for bao.
- `pki/roles/swarm-nats` allows that one name (bare domain, no subdomains,
IPs or localhost, server flag). A `swarm-nats` cert-auth role and policy
may only `update` `pki/issue/swarm-nats`, written by
`swarm-bao-nats-tls-policy`. The login leaf is minted by glue-bao-tls and
paired by glue-nats-bao-identity. `deploy.bao.natsCommonName` is reserved
as a hive name.
- `swarm-bao-nats-tls` issues the leaf into a directory bound read-only into
the container, restarts nats when it rotates, and re-runs daily.
It joins glue-bao-readers-policy-order, so it is ordered after its policy
unit (`after` and `wants`, never `requires`) where the store is on the
same host. The policy unit joins the store's journald list.
- nats gets `tls {}`, with the key via `LoadCredential`, and no
`allow_non_tls`. `validateConfig` is now off in every mode, because the
build-time check loads a leaf that only exists at runtime.
- 4222 is also open on `wg-hive` when the host is on the mesh, never
host-wide.
- `statusPublish.natsUrl`, `queue.agentNatsUrl`, the controller's URL under
`singleHostSwarm`, and the auth responder all dial
`tls://<swarm.nats.domain>:<port>`. swarm-queue-client hands its CA file
to the NATS connection too, so hive-c0re and the controller trust the
leaf's root.
- docs/swarm/README.md: the queue URL and the one DNS record a multi-host
swarm needs.
module-eval-nats-tls pins the role, the policy, the served leaf, the
firewall, the ordering, and a scan of every `*_NATS_URL` and the
responder's URL across the host and its containers.
Closes#4626
get_issue_report mapped every error from Client::issue_report to 503
(StatusUnavailable) unconditionally. issue_report calls
issue_list_issues(org, repo, ...) first, and a Forgejo 404 there — a
typo'd or deleted repo — was flattened into the same 503 a genuine
forge outage produces, which tells a client to retry a request that
will never succeed.
Downcast the anyhow error back to forgejo_api::ForgejoError (same
pattern main.rs's wanted_error_status uses for
swarm_queue_client::Error) and check it structurally against the
ApiErrorKind::NotFound / UnexpectedStatusCode(404) shapes forgejo's
generated client produces for a 404, rather than string-matching the
rendered message. Only that case answers 404, naming the org/repo;
every other forge failure still answers 503. Updates the route's
OpenAPI response list to document the 404.
Closes#4701
setup.md's copy of the swarm-bootstrap policy still granted only the
controller's first six paths, while eight units now act with the
bootstrap token. The policy moves to
nix/host-modules/swarm-bao-bootstrap-policy.hcl, now covering every path
those units call. module-eval-bao-grants reads that file and fails
when a unit whose script uses the token calls a path the file does not
grant.
Every agent becomes a Forgejo user of the same name, and nothing upstream
of `CreateForgeUser` knew what Forgejo refuses: `admin`, `api`, `foo-` or a
41-character name passed name validation and failed one node into
provisioning with Forgejo's 422.
`nix/reserved-names.nix` gains the 22 reserved usernames of Forgejo
v16.0.5 (`models/user/user.go:639-680`) that `[a-z0-9-]` can spell, and
the bare `-` (`models/repo/repo.go:67`). The dot and underscore entries are
left out, since our charset cannot produce them. The header's admission
rule grows a third class — a username the forge refuses — because that is
a failure behind the refusal.
The shape rules are not literals, so they live in
`hive_types::forge_username_violation`: no leading `-`, no `--`, no
trailing `-`, at most 40 characters. Beside `is_reserved_name`, not in
`Ident::parse`: an `Ident` is also a hive, label, account and subagent
name, and parsing runs on every read of an existing name.
`create_agent` used to WARN on a reserved name, deliberately: an operator
with agents already created under a colliding name would otherwise be
unable to re-run creation. That reason is kept, and narrowed to what it
protects. A name breaking either rule is now refused with a 400 naming the
rule when the name is NOT in the swarm roster, and still only warned
about when it is, so re-creating an existing agent keeps working. The
roster is read only for a rule-breaking name; when it cannot be read, a new
name and an existing one look alike, and this warns as before. The
hive-collision warning is unchanged.
put_matrix_account writes the credential to bao before checking that
the queue it must notify is actually connected — only that a queue is
configured, via state.status.as_ref(). While the queue is Pending or
Disconnected, client.flush().await hangs (async-nats does not process
commands during the initial connect retry), so the request hangs until
nginx times out and the credential is already stored. Add the
ensure_connected check every sibling queue route already makes
(wanted.rs, term_stream.rs, agent_state_stream.rs), placed immediately
before store::connect() so a request that cannot be delivered never
reaches the store.
Same shape, lower impact, in webhook::announce_knowledge_change: it
awaits publish/flush inline in the webhook handler, so a disconnected
queue can run past forgejo's short delivery timeout. Add the same
ensure_connected guard, warn and return.
set_agent_state and get_hive_wanted map every writer error to 500,
including swarm_queue_client::Error::NotConnected surfaced through
WantedWriter::view/set. Add wanted_error_status, which downcasts the
anyhow::Error back to the concrete type and maps NotConnected to 503
(retryable) while leaving every other failure at 500. Update both
routes' OpenAPI descriptions to say so.
Closes#4688
The four readers' ordering after their policy units only applies where
the store and that reader share a host, so it is colocation glue and
does not belong in the reader modules (two of which are main modules).
glue-bao-readers-policy-order.nix now sets the after+wants edges, gated
on deploy.bao.enable AND the reader's own gate, so a store host without
a reader gains no stub unit.
The four readers (matrix-token, queue-agent, grafana-oidc, otel-oidc) log
in against a cert-auth role that their own swarm-bao-*-policy unit writes.
They were ordered after swarm-bao-pki and the store's container but not
after that unit, so on an apply a reader could log in before its role
existed and be refused by `allowed_common_names` until a retry landed
after the role did.
After= plus Wants= on the policy unit, never Requires=: the policy unit
skips by ConditionPathExists once the bootstrap token is gone, and a
skipped unit counts as done for ordering.
The units on the path a deploy takes to TLS, the store's grants and the
swarm collector itself were not on the host collector's journald
allowlist, so an ingest outage one of them caused showed in the store
only as every source going quiet at once.
Each module names its own units, per the option's rule:
- hive-tls.nix: hive-tls-ca, swarm-services-cert
- hive-gateway: hive-gateway-self-signed-cert (self-signed mode only)
- swarm-bao.nix: the seven grant units beside
swarm-bao-services-issuer-policy
- swarm-otel.nix: container@<machine>, and
nixos-rebuild-switch-to-configuration, the transient unit nixos-rebuild
runs the activation in and whose syslog lines carry its status
The module-eval arm pins each unit as both listed and defined, since a
listed name that matches nothing is silent.
The comment above the bail! named the gap itself: stderr went through
redact_secret_line, but args.join(" ") did not. hive-c0re passes a
live --password value as an argument on user create and
change-password, so any non-zero forgejo admin exit put the plaintext
password in the error string, and from there into hive-c0re's warn!
log (journal + VictoriaLogs) and hivectl's returned error.
Add describe_forge_admin, a pure function that names the invocation by
its leading verb path and stops at the first flag, same approach as
hive-c0re's own describe_forge_admin (forge/mod.rs, from #2936) and
for the same reason: the verbs are a closed set this crate chooses,
argument values never are, so an allowlist over shape excludes any
future secret-bearing flag by construction instead of by someone
remembering to redact its value.
Unit tests cover: a --password value dropped, the --password=value
form also dropped (it starts with '-', so the take_while excludes the
whole argument), and a control that the verb path still appears.
Closes#4670.
print_url only printed when browser_download_url was Some, so a 2xx
upload response with no URL printed nothing and still exited 0 — a
caller doing url=$(hive-forge attach-issue …) got an empty string with
no signal anything went wrong.
require_url now errors, naming the attachment's id/name, when the
forge returns no download URL.
labels <n> remove used to silently skip an unknown label name (lookup_id
returning None just did nothing) and discard each remove call's Result,
so a typo or a forge-rejected removal (403, insufficient permission)
both exited 0 with the label untouched.
Resolve names through the same resolve_ids labels add already uses, so
an unknown name is refused before anything is removed. After removing,
re-fetch and diff against what was requested — following assign.rs's
same before/after check — and bail! naming any label still present.
send_agent_snapshot_to_file created `dest` with create_new(true) before
checking whether the parent snapshot exists, and left dest behind on any
spawn()/wait_with_output() failure too. Cleanup only ran on the one
remaining path: btrfs send spawning fine and exiting non-zero.
Every other failure listed in the wire doc left a zero-byte export, and
because dest already existed, a retry against the same --dest always hit
the no-overwrite guard — the only way out was an operator manually
removing the file.
Fixes#4687.
- check_send_snapshot_preconditions() validates the snapshot and its
optional parent before dest is touched at all.
- open_export_dest() combines that check with the create_new open, so the
ordering can't drift apart again, and is unit-testable without btrfs.
- PartialExportGuard removes dest on drop unless disarmed, covering every
failure path after dest is created (including the spawn/wait `?`s that
previously leaked it), not just the btrfs-exit-nonzero case.
Forgejo answers 422 for six different causes on admin user create
(ErrUserAlreadyExist, ErrEmailAlreadyUsed, ErrNameReserved,
ErrNameCharsNotAllowed, ErrEmailInvalid, ErrNamePatternNotAllowed) and
several on repo create, but is_already_exists() treated every one of
them as a conflict. A reserved or otherwise-refused name silently
folded to Done, so CreateForgeUser reported success with no user
created, and the graph's real failure only surfaced one node later as
a misleading AddRepoMember error.
ensure_agent_user and ensure_org_repo now trust a 409 unconditionally
(folds_into_success) but confirm a 422 with a follow-up user_get /
repo_get before folding it to success; an unconfirmed 422 fails with
forgejo's own message at error level. Webhook registration still uses
the old is_already_exists — it has no comparable follow-up read, so it
is out of scope here.
Closes#4678
destroy has no manager-name check (actions.rs:837 says the root
container is 'destroyable like any other'), and crash-watch has no
name check at all (polls every managed container uniformly). The
pointer to broker.rs/actions.rs/crash_watch.rs for 'owner-check logic'
is also stale — none of the three hold any.
The PR retiring loose-ends' manager-only visibility replaced the false
claim with prose narrating its retirement, which still adds lines for
a removal. Delete the dead claim outright instead of documenting that
it used to be true: drop the three added sentences in
agent-hierarchy.md, and shorten socket_server/mod.rs's authority
comment to state only what's true now (tool-group membership for the
orchestration verbs) rather than adding an explanatory clause for the
removed capability check.
Follow-up to 729c5b4f42, which dropped the
`agent` parameter from get_loose_ends and deleted Capability::QueryAgentState
with it. Three prose sites still describe the interface that commit removed:
- socket_server/mod.rs's module doc claimed authority on this socket derives
from "capabilities for the hive-wide queries and tool-group membership for
the orchestration verbs". There are no capability-gated queries left —
`git grep -n has_cap -- hive-c0re/src/socket_server/` returns nothing, and
require_group("scheduling") is the only gate in dispatch_orchestration.
- agent-hierarchy.md listed loose-ends visibility ("manager sees hive-wide,
sub-agents only their own") as one of the manager-only overrides that
"exist across hive-c0re today". It doesn't, and the sentence pointed a
reader at loose_ends.rs for owner-check logic that is no longer there.
- conventions.md's Loose-ends wire shape still spoke of "the agent-flavour
and manager-flavour requests" and "the agent-flavour list". There is one
request shape.
No behaviour change: prose only.
Refs #4480
post_purge_tombstone's live-container guard used lifecycle::list().await
.unwrap_or_default(), so a failed list() (hive-priv socket unreachable,
restarting, ...) read as 'no live containers' and let the purge proceed —
deleting agent_state_dir/applied_dir for an agent whose container may
still be running. Factor the decision into check_not_live(), a pure
helper that refuses on both 'still listed' and 'list unreadable', with
unit tests covering both plus the allowed case.
Closes#4671
The marker lives in the container's own /etc, which root inside the
container controls, and hive-priv writes it as host root. The write
followed a symlink at the leaf and at etc, so an absolute symlink
planted in the container redirected a host-root truncate-and-write to a
host path.
Open etc with O_DIRECTORY|O_NOFOLLOW and the leaf relative to that fd
with O_NOFOLLOW, and require a regular file (opened O_NONBLOCK so a
FIFO cannot hang the helper). The marker keeps mode 0644.
Closes#4667
lifecycle::destroy logged a failed `nixos-container destroy` and returned
Ok, so the DestroyContainer node went green, the agent was unregistered,
and with purge the after_ok PurgeState deleted its state while the
container config and root still existed.
Propagate the error unless the container list, read after the failure,
no longer names the container. An unreadable list fails too.
A failed `bao read` of the services root made the checkend pipeline
non-zero, so the unit deleted a working root and minted a new trust
anchor. A failed `bao list` of the issuers likewise looked like an empty
mount. Both now fail the unit, which retries on its own restart budget;
the root is replaced only when openssl parsed the returned certificate
and -checkend said it expires inside a leaf's window.
Closes#4663
The container's collector has never shipped a line. `journalctl --follow`
— which the journald receiver passes unconditionally — scopes itself to
the current boot unless `--merge` is given too, and `--link-journal=host`
makes /var/log/journal the HOST's journal tree, where this container's
current boot has no entry. journalctl exited 1 with "No journal boot
entry found for the specified boot (+0)" and the receiver respawned it
every ~2s, so nothing was ever read and nothing was ever exported.
`merge = true` is the receiver's key for `--merge` (buildArgs() in
pkg/stanza/operator/input/journald/config_linux.go at tag
receiver/journaldreceiver/v0.151.0, the deployed collector's version),
and --merge is what clears the implicit boot scope in journalctl.c
(systemd v260.4, the version on the host).
The module-eval arm pins both halves: the boot filter is gone AND the
directory is still the host-linked one — either alone is satisfiable by
the broken config.
`interrupt` killed the child and trusted that to end the run. It didn't:
the default signal is SIGINT, and `claude --print` handles SIGINT by
writing its terminal `result` event and exiting **zero**. A zero exit is
`TurnEnd::Complete`, so `spawn_and_track`'s killed-turn early return —
the code that was supposed to stop the run — never fired, and the loop
went straight on to spawn the next goal turn. An interrupted run carried
on to completion; the interrupt changed nothing about the outcome.
Measured, not inferred: `claude --print --verbose --output-format
stream-json`, SIGINT'd mid-turn, exits 0 on every run.
So record the reason instead of inferring it. `interrupt` writes
`StopReason::Cancelled` — the same path `goal_reached`/`need_help`
already use — while it still holds the `running` lock, so there is no
instant in which the name has lost its cancel handle but not yet gained
its stop reason. `plan_after_turn` checks stop reasons ahead of the goal,
so the run stops whichever way the child ends up exiting. `force`'s
SIGKILL still produces a `TurnEnd::Killed`, and `status` still prefers
the kill record it already had.
`continue`'s budget reset is untouched: it clears the stop reason and
hands back a fresh allowance, which is what makes a cancel a pause an
operator can undo.
The test spawns the real continuation loop against a fake claude that
exits 0 on SIGINT, interrupts it, and asserts turn two never starts —
it reports `left: 2` without the fix.
`cmp` lives in diffutils, not coreutils, so the root-changed test exited
127 with "command not found". Inside `if ! cmp -s`, a 127 reads as
"differs" and errexit never sees it, so `rootchanged` was 1 on every run
and the trust-bundle rebuild it guards bounced `hive-tls-ca` after each
issuance — the exact "only a changed one, or every boot would bounce a
unit with nothing to do" the comment there rules out.
Found in the journal of a hive that had just issued a leaf successfully:
the unit logged "the services root changed" on a run where the store had
left the issuer alone.
Three defects in the store-issued path, each of which alone kept nginx
from starting at all. The gateway's cert import `Requires=` this leaf, so
a leaf that is never issued is not a name mismatch — it is an empty
listener, and the swarm's own forge stopped answering on :443.
`swarm-services-cert` declared `Before=hive-tls-ca` for the trust
bundle's sake while also being `After=` the store's container, which is
itself `After=hive-tls-ca`. systemd resolved the cycle the only way it
can, by deleting the job, so the leaf went unissued on every activation.
The edge is gone; the bundle converges the other way round, through the
restart this unit already performed when the root it wrote was new.
The `pki` mount was enabled without `-max-lease-ttl`, so bao clamped the
30-year root to the 768h default and then refused every issue call,
because a leaf of the mount's own default length would outlive the CA
signing it. The mount is tuned on every run, the role pins a 720h leaf,
and a root that can no longer cover one is replaced rather than left to
refuse forever. A hive tests its own copy of that certificate against the
same threshold, so both ends reach a fresh leaf without signalling.
`swarm-bao-pki` mints the services-issuer leaf that opens the mount, but
only `swarm-bao-certs` required it. `RemainAfterExit` plus an
already-active unit means an activation that ADDS a leaf mints nothing —
which is how a host whose config named `services-issuer.pem` came to have
no such file. A target wants it now, like every sibling granting unit.
Active-voice / contraction rewrites only, no meaning changes; the
allowed_domains/SANs sentence and the published-cert table cell were
checked against the parked services-issuer/role-type/vhost-scope
questions on #4622 and don't touch any of them.
The `pki` mount had no issuer and no principal could log in to it, so the
swarm's service certificates were still minted by two openssl hops from a
root key on disk. Close both halves and retire the openssl path with them.
The mount now generates its own root, once. The granting unit asks bao
whether an issuer already exists (`bao list pki/issuers`) before calling
`pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role
and the grant without touching the anchor — a root that changed per boot
would invalidate every certificate issued under it and every browser
taught to trust it. The guard asks the store rather than looking for a
marker file on this host's disk: a file is a claim about a mount that may
have been restored from a snapshot or disabled and re-enabled underneath
it.
`swarm-services-issuer` stops being an inert policy. A fourth cert-auth
role attaches it, following the shape the controller, the publisher and
matrix-ctl already use, and glue-bao-tls.nix signs the leaf carrying its
CN — that credential is what opens the mount, so it cannot come out of it.
`swarm-services-cert.service` logs in with that leaf, calls
`pki/issue/swarm-services`, and writes the result to the path
hive-tls.nix already wrote and the gateway already copies from. The
sub-CA layer does not move; it stops existing. The role's
`allowed_domains`, read from the same `swarm.serviceDomains` the SANs
come from, enforces at issue time what the sub-CA encoded in x509
`nameConstraints`, and with the root inside the mount there is nothing
left for an intermediate to be an intermediate of.
Not a flag day: the issuing root is published beside the leaf as
`swarm-services-root.pem` (0644) and joins `trust-bundle.pem`, where the
swarm root still sits. A leaf chaining to the old sub-CA and one issued
by the store both verify against the same bundle, so hives can be
rebuilt in any order. The same file is what an operator hands a browser
— readable without a store login, which matters because every listener
demands a client certificate.
The eval-time warning about uncovered service names is gone rather than
reworded. It fired on "this host does not hold the swarm root key", which
was the reason a hive could end up serving its own leaf on a
swarm-service name. Every hive now asks the store with its own identity,
so that stopped being the thing that decides.
Closes#4586
mint_and_verify reads the queue credential back before writing it, so a
re-run keeps the value a live agent already authenticates with. that read
is read_optional, which maps only a 404 to absence — so with create/update
alone every mint aborted on a 403 at its first store read.
read on the same paths the stanza already grants create and update, and
nothing else: no list, no delete, no patch.
