Each of these wrote its file in place (or set the mode after the
write), so a write cut short by a kill or a full disk left a partial
file behind, and some secrets existed for a moment at umask mode:
- swarm-authelia/swarm-grafana/hive-forge/swarm-bao generators: a
partial file passed the `-s`/`-e` guard and was kept forever. They now
capture the value first and land it with `atomic_write_secret`.
- swarm-controller webhook secret: a truncated file failed the 64-hex
check and the secret was regenerated, out of step with the forge
hooks. It is now written 0600 to a sibling temp, fsynced, renamed.
- hive-c0re htpasswd: the temp is now fsynced and set 0644 before the
rename, like the sibling agents.conf writer, instead of relying on
c0re's umask.
- hivectl wg init: the private key is created 0400 in a temp and
renamed, instead of written at umask mode and chmod'ed after.
Closes#5068
- swarm-controller webhook: regenerate the secret only when the file is
absent. Any other read error is returned with the path; the caller
already logs "webhook secret unavailable" and the endpoint answers 503.
Overwriting meant every registered hook kept the old secret and every
delivery failed HMAC, with nothing re-registering them.
- swarm-queue-client wanted: open_read_only/watch return Result; only
STREAM_NOT_FOUND on the bucket maps to None. hive-c0re's wanted pull
fails the DAG node with the cause instead of logging "nothing has been
declared"; the watch warns with the error chain.
- swarm-nats-auth: warn when the OIDC secret re-read task dies.
- swarm-controller forge: listing a repo's config PRs failing is warn,
not debug.
Closes#5056
A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):
- `forge::enable` tries the store once at start. On failure it logs once and
re-reads in the background with backoff (30s doubling to a 5-minute cap,
no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
poll and the webhook registration start from that moment, through
`start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
"no forge" before and work after.
A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.
Refs #4998
An operator's merge on the forge deploys a config PR through
swarm-controller's DeployRequest{rev}. The hive-side path that queued a
MergeConfigPr approval and merged the PR as `core` goes:
- the `/webhook/config-pr` receiver, its HMAC secret, the WebhookRegister
boot node and the org-hook registration; the hive vhost's `/webhook/`
location
- the 5-minute config-PR poll
- ApprovalKind::MergeConfigPr, its dashboard card, and the deploy DAG it
drove (DeployWindow, MergeVerify, DeployApply, FinalizeDeploy,
DeployTail), with verify_commit, the two-phase meta deploy, rollback
refs, the PR-failure comment and forge/pr_merge.rs
- `fetched_sha`, `sha_short`/`pr_number` on approval events, and
`sha`/`tag` on HelperEvent::ApprovalResolved: only the merge path set
them
`config_repo`, `merged_pr_for_commit` and `post_pr_comment` move to
forge/pr_comment.rs for the merged-rev deploy's refusal comment.
Approvals v5 drops stored `merge_config_pr` rows; a test reopens a v4
database holding them.
Closes#4850
A config PR merged in the Forgejo UI changed nothing on the hive: the
hive's webhook ignores `closed`, its poll then cancels the dashboard
card, and `applied/main` stays where it was.
swarm-controller reads `merged`/`merge_commit_sha` off the
`pull_request` delivery it already receives for `agent-configs`, finds
the hive placing the agent by scanning every hive's wanted state (the
scan `declarations_elsewhere` already ran, factored out), and queues a
`TriggerDeploy` carrying the rev. Zero or several claimants deploy
nothing and log the claimants.
`DeployRequest` gains `rev: Option<String>` with `serde(default)`, so
rev-less payloads from either side keep decoding.
hive-c0re, given a rev for an agent it runs: a no-op when
`applied/main` already is the rev (a dashboard merge deploys its own
PR); otherwise it fetches the forge `main` with the core token,
requires the rev to descend from `applied/main` (the ancestry gate,
factored out of `run_deploy_merge_verify`), fast-forwards by CAS and
queues the usual relocking rebuild. No eval-verify on this path, per
mara (#4850 c90075). A refusal is commented on the PR that merged the
rev, found by commit.
swarm-controller's forge-objects pass converges every config repo's
`main` rule to merge whitelist `operators` + `core` and approval
whitelist `operators`. The hive's boot PATCH stops forcing
`enable_approvals_whitelist` off, so the two do not fight.
Refs #4850
The orgs agent-configs/internal/agents (plus mirror owners), the
operators team in agents and agent-configs, the pull-mirrors,
internal/docs, internal/knowledge (public, README-seeded) and the
agent-configs org avatar are one set per forge. hive-c0re ensured them in
its boot sweep, as the core admin, and only on the hive co-located with
the forge container.
swarm-controller now reconciles them at start and every 5 minutes
(forge/objects.rs: observe -> pure plan -> apply). A failed object logs
a warn line plus a pass summary and is retried next tick. create_repo
ensures the agent-configs org and its operators team first, so a config
repo's merge gate never depends on the periodic pass having run.
hive-c0re drops ensure_org, SEEDED_ORGS, ensure_mirrors/ensure_mirror_repo,
ensure_operators_team, ensure_shared_docs_repo, ensure_knowledge_repo/
set_repo_public, seed_readme, ensure_config_org_avatar and the one-shot
knowledge::remove_webhook cleanup, with their now-unused helpers.
nix: the mirror list moves from the hive-c0re unit
(HYPERHIVE_FORGE_MIRRORS) to the swarm-controller unit
(SWARM_CONTROLLER_FORGE_MIRRORS), with an eval warning when mirrors are
declared on a host that runs no controller. c0re.orgAvatarPng is renamed
to deploy.swarm-controller.configOrgAvatarPng.
Refs #3782
put_matrix_account writes the credential to bao before checking that
the queue it must notify is actually connected — only that a queue is
configured, via state.status.as_ref(). While the queue is Pending or
Disconnected, client.flush().await hangs (async-nats does not process
commands during the initial connect retry), so the request hangs until
nginx times out and the credential is already stored. Add the
ensure_connected check every sibling queue route already makes
(wanted.rs, term_stream.rs, agent_state_stream.rs), placed immediately
before store::connect() so a request that cannot be delivered never
reaches the store.
Same shape, lower impact, in webhook::announce_knowledge_change: it
awaits publish/flush inline in the webhook handler, so a disconnected
queue can run past forgejo's short delivery timeout. Add the same
ensure_connected guard, warn and return.
set_agent_state and get_hive_wanted map every writer error to 500,
including swarm_queue_client::Error::NotConnected surfaced through
WantedWriter::view/set. Add wanted_error_status, which downcasts the
anyhow::Error back to the concrete type and maps NotConnected to 503
(retryable) while leaving every other failure at 500. Update both
routes' OpenAPI descriptions to say so.
Closes#4688
The function's own doc comment still said one subject per hive, four
lines above an inline comment correctly saying one publish for all of
them. Left behind by the single-writer rework.
Swept the rest of the touched files for the old design's vocabulary
rather than fixing only the line argus quoted: every other per-hive
mention is about webhook registration or the callout policy's
extra-subject template, both of which are still per-hive and still
accurate.
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.
The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.
status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.
That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.
The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.
Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.
The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
The test's doc named knowledge::ensure_webhook as one of two hive-side
reapers. That function is gone; its replacement matches the full URL and
cannot touch another hive's hook.
The assertion arm stays. The hazard is not what this repository's source
says, it is what is deployed — a hive still running the previous version
reaps by suffix until it is upgraded, so the arm guards the transition
rather than a current code path. Recorded when to drop it.
The controller verified a knowledge delivery, logged it, and returned OK.
Nothing downstream ever heard about it, so a hive learned the repository
had changed only by registering its own webhook — which is the
last-writer-wins contention this issue is about.
The event carries no payload. The hive-side handler this replaces reads
two fields from Forgejo's push webhook and uses neither — both are
filters — then runs `git pull`, which re-derives everything from the
repository. What crosses the queue is an edge trigger, and fields would
invent a contract nobody reads.
One subject per hive, so the callout policy can express "this hive may
hear its own events" at all; a subject with no hive component is the same
subject for every hive.
`ConfigPr` deliveries are deliberately not forwarded. A hive does not
want to hear that a config PR opened — it wants to be told when to
rebuild from main, which the controller decides after a merge rather than
by relaying this delivery. That is deploy coordination's job, and the
empty arm is there so the omission reads as scoped rather than forgotten.
Fails soft: a missed announcement costs a hive stale knowledge until its
next boot pull, which is the same cost as a webhook delivery to a hive
that was down — what this replaces. A permission failure cannot be
observed at the call site (a NATS authorization violation is reported
asynchronously on the connection, reaching a client as a timeout or as
nothing), so the doc says the flush proves only that the bytes left this
process and points at the server log.
mara: "pls remove historical wording, only present pls".
The correction was written as a diff against what the docs used to claim
-- "this used to say X", "where this is going", "the intended state for
now". That is a changelog, and a reader arriving cold has to reconstruct
the current truth from it. The reasoning about why the old shape was
wrong belongs in the PR that changed it, not in the file.
Now says what is true: the controller interprets a delivery and emits a
semantic message; receipt is all that is wired today because the
swarm->hive channel does not exist yet.
mara asked for the docs and comments to be fixed in one PR before the
implementation, since a half-corrected description is what confuses a
reader coming back after a context compact.
Three more places said or implied "forward the payload":
- post_webhook_forge's doc ("relays the delivery to every hive ... the
payload is never parsed here")
- its OpenAPI request_body description ("the delivery can be relayed
unmodified") -- this one is published in the spec, so it was the most
externally visible of them
- DeliveryKind::as_str, which called the hook kind the event's routing
key; the message is semantic, and which hook it arrived on is an input
to deriving it rather than the thing sent
docs/swarm/README.md gains the direction next to the interim state, so
"two hooks, one of which only logs" reads as a step rather than a design.
mara's ruling: the controller should not forward received events, it
should create specific messages from them ("knowledge repo changed",
"deploy agent foo to rev abc123") and send those to whoever needs them.
The module doc said the opposite as settled design.
The argument it made was wrong in a specific way worth keeping: it
counted this daemon as a *second* place deciding what a delivery means,
while assuming the hives stay interpreters. They do not -- the parsing
moves rather than duplicating, so the end state has one interpreter, not
two.
Doc-only. The behaviour is unchanged and still receipt-only; what changes
is that the next reader is not told the wrong direction.
The endpoint landed inert: nothing pointed at it, so the only way to see
it work was to mint an HMAC by hand. Register the two swarm-wide hooks
at startup so a real forge event produces a journal line.
Registered ALONGSIDE the per-hive hooks, not instead of them. Every hive
keeps receiving and acting on its own deliveries; the controller gets a
copy and logs it. Moving the registration is a later step and has to be:
fan-out swarm->hive does not exist yet, so a hook moved now would point
at a receiver that forwards nowhere, silently on both sides.
Deliberately no stale-hook deletion arm, unlike the two per-hive
registrars this otherwise mirrors: theirs delete hooks matching their own
path with a foreign base, and the hives' hooks are not stale.
The route prefix is what keeps this safe. Both hive-side registrars
delete any hook ending in /webhook/knowledge or /webhook/config-pr with a
different base, so a swarm hook under those paths would be deleted by
every hive on every boot. Serving them under /webhook/forge/ avoids it,
and a test pins it -- there is nothing else that can.
SWARM_CONTROLLER_PUBLIC_URL is set only where the swarm vhost is served,
because a hook whose target_url nothing answers is worse than no hook.
A Forgejo webhook has one target_url, so every hive registering the same
swarm-wide hooks is last-writer-wins rather than idempotent. The controller
is the only swarm-wide thing in the deployment, so it becomes the receiver.
It verifies the HMAC and treats the payload as opaque bytes keyed by the hook
kind in the URL path; it deliberately does not parse the payload, because the
hives' existing handlers already decide what a delivery means.
Nothing is registered against the endpoint yet. The replacement path is built
and observable before anything takes the old one away, so the swarm's single
target_url never points at a receiver that forwards nowhere.
