The six hand-written granter policy units (secret-publisher, matrix-ctl,
services-issuer, nats-tls, hive-leaf, operator-viewer) repeated the same
skeleton readerPolicyUnit already carried: ordering after the controller's
unit, pki requirement, path, granter environment, unseal retry window and
oneshot serviceConfig. They differed only in description and the script
after the granter login.
granterPolicyUnit now owns that skeleton; the six units and readerPolicyUnit
pass a description and a script body. Every rendered unit is unchanged
byte-for-byte.
The publisher was wanted only by container@hive-forge.service. A switch
starts a new unit only through the targets that want it, and the container
is already running, so the deploy that introduced the publisher left it dead
until the next container restart or boot; the path unit cannot help, because
the mint's only-once guard never rewrites the file.
- The publisher is also wanted by multi-user.target, like swarm-secret-publish.
- A token file that is not there yet exits 0 instead of waiting 120s and
failing: the path unit runs the publisher when the mint writes it, so a
switch with the forge container down does not fail. TimeoutStartSec is 60.
- swarm-controller loses its After= on the publisher: a switch issues the two
jobs separately, so it ordered nothing there, and the background re-read
covers the gap.
- The controller's background re-read logs at info once the backoff is
capped, one line per five minutes, so a forge host that never got its
publisher leaf stays visible; earlier retries stay at debug.
Refs #4998
The publisher was a oneshot with RemainAfterExit=true, started only through
`wantedBy = container@hive-forge.service`. Once it had run it stayed
`active (exited)`, so a later container start made its start a no-op: a token
re-minted without a host reboot or a unit change never reached the store.
- RemainAfterExit is dropped, so each start runs the script again.
- systemd.paths.swarm-bao-forge-publisher watches the token file in the
container (PathChanged) and starts the publisher when the mint writes it,
the same shape as swarm-secret-publish's path unit.
- The code comment, secrets.md and credentials.md now name both triggers;
forge-placement pins the missing RemainAfterExit and the path unit.
Refs #4998
A controller that started before the forge's host had published its token
ran without forge access until someone restarted it: on a split-host first
deploy that is a manual step. Now the forge client lives in a slot that is
filled once (`Late<T>`, an `Arc<OnceLock<T>>`):
- `forge::enable` tries the store once at start. On failure it logs once and
re-reads in the background with backoff (30s doubling to a 5-minute cap,
no log per retry), fills the slot and logs once when the token is there.
- The forge passes (#5014's per-hive token mint among them), the config-PR
poll and the webhook registration start from that moment, through
`start_forge`'s callback.
- Job nodes and HTTP handlers read the slot when they run, so they answer
"no forge" before and work after.
A token replaced in the store after the slot is filled still reaches the
controller with its next restart. The round-1 change that shared
`queue_identity::fetch_with_retry` is reverted; nothing else uses it.
Refs #4998
The controller's forge token never entered bao: the forge minted it in its
container, hive-forge-swarm-controller-token copied it to a host file, and the
controller LoadCredential'ed that file. forgeTokenFile defaulted to that path
only where the forge ran, so a controller on any other host had no forge
access unless an operator copied the file by hand.
Now the forge's host publishes it and the controller reads it:
- swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token)
waits for the in-container token, logs in with its own forge-publisher leaf
and writes it to swarm/controller/swarm-controller/forge/token. It runs on
every forge container start, the old unit's trigger.
- New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix,
policy create/update on that one path, written by
swarm-bao-forge-publisher-policy, CN reserved as a hive name.
- The controller policy gains read on that path. swarm-controller reads it
at start under its own certificate with the OIDC secret's retry, and runs
without forge access if it still cannot.
- forgeTokenFile, its LoadCredential and the forgeEnv gate are gone;
SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is
removed, and a tmpfiles rule deletes the old host copy on the forge host.
The in-container file stays: the mint unit's only-once guard reads it, and
it is the publisher's source.
Refs #4998
Agent creation gains a GrantDocsRead node (after CreateForgeUser) that adds
the agent as a read collaborator on internal/docs, and forge::docs_grant
runs a pass over the store's hive-agent-* roster at start and every five
minutes that queues the same job for any agent that is not one. The pass only
adds; an unreadable roster or forge read changes nothing.
hive-c0re drops shared_docs_access, its call in sync_agent, and the
SHARED_ORG / SHARED_DOCS_REPO constants only it used.
`hive-<hive>` is each hive's read-only forge account, and an agent's forge
account takes the agent's name, so an agent called `hive-<x>` would claim
hive `<x>`'s account. `hive_types::reserved_agent_prefix_violation` names
the rule, and `broken_name_rules` applies it, so both creation paths that
go through it (`POST /api/agents` and placing an undeclared agent on a
hive) refuse such a name for a new agent. An agent already in the roster
keeps its name with a warning, as for every other naming rule.
The email check between hive and agent accounts stays as the second
guard, for agents created before this rule.
Refs #4851
swarm-controller now gives every hive in its directory a forge account,
`hive-<hive>`, in a new `hives` team of `agent-configs` (read on
`repo.code` + `repo.pulls`, every repo including later ones), and mints
one `read:repository` token named `swarm-hive` into the store at
`swarm/hives/<hive>/forge/token`. A five-minute pass queues a
`MintHiveForgeToken` node for a hive whose token is missing or no longer
matches the forge, whose account is missing, or whose team membership or
team settings drifted. A hive whose store or forge read failed is never
queued, so an outage cannot rotate every hive's token. The token decision
reuses `agent_token`'s classifier and replace step, now parameterised by
name and scopes.
An agent may be called `hive-<x>`. The hive account is created with
`hive-<hive>@hives.hyperhive.local` and the job refuses an account by that
name with any other email before it locks it down, adds it to the team or
mints for it; agent creation already refuses the hive's account by the
same email check.
The controller's bao policy gains the `swarm/hives/+/forge/token` leaf
(create/read/update), pinned by the bao-grants module-eval assertion. The
hive's own policy already reads `swarm/hives/<hive>/*`.
hive-c0re reads that token through `forge::reader_token()`, falling back
to the core token file when the store holds none, and uses it for:
fetching a merged config rev (swarm deploy, `hivectl forge
reconcile-config`), reseeding a purged `applied`, the meta-update
cascade's fast-forward, seeding `proposed` on first deploy, and finding
the PR that merged a refused rev. Those reads no longer require a local
`hive-forge` container. The `/knowledge` clone moves from the forge
host's boot sweep into the knowledge pull, so every hive clones it.
Posting the deploy-failure comment on that PR stays on the core token.
Refs #4851
The reseed bail text, the `ensure_all` doc, the forge module doc and a
rebuild comment still described hive-c0re mirroring the applied config to
`agent-configs/<name>`. State what the code does now; scope the
config-changes doc to swarm-created agents and say what a legacy agent
whose repo has no `main` gets.
Refs #4851
Per the operator's ruling on #4851 (option (a)): the hive holds no
write identity on the forge, so the `push_config` mirror of an applied
repo's tags and `main` to `agent-configs/<name>` is removed, along with
its `run_config_push` helper and its call in `sync_agent`.
Docs and comments that described the mirror now say that forge `main`
advances only by an operator's merge.
Refs #4851
systemd's default OOMPolicy=stop ends the whole unit when the kernel
kills any process in it; set continue, as hive-subagent-daemon does.
Module-eval case asserts it.
The reader's subscribe allow-list spelled `$SWARM.term.*`,
`$SWARM.term.*.sub.*` and `$SWARM.agent-state.*` as literals, copies of
private prefixes in hive-agent and swarm-controller. A rename on either side
left the policy behind and the stream stopped with only a connection-level
server error.
swarm-queue-client now owns `term_subject`, `agent_state_subject` and their
`*_wildcard` forms, and `subagent_term::subject_wildcard`; hive-agent's
publishers, swarm-controller's relays and the policy all build from them.
Subjects are byte-identical.
A hive's queue credential and the swarm controller's could subscribe to
every subject in the swarm account, so a hive could read every other
hive's deploy requests and every agent's terminal, state, input and
request replies.
Their grants now carry a subscribe allow-list:
- a hive `hive-<h>`: `$SWARM.knowledge`, `$SWARM.deploy.<h>` and its
reply inbox `_INBOX.client.hive-<h>.>`
- the reader (swarm-controller): `$SWARM.term.*`, `$SWARM.term.*.sub.*`,
`$SWARM.agent-state.*` and its reply inbox
`_INBOX.client.<client-id>.>`
`swarm_queue_client::connect`, which hive-c0re and swarm-controller both
use, now connects with `_INBOX.client.<client-id>` as its inbox prefix,
the same shape agents got for their own credential. In async-nats 0.50
every request reply and every ordered consumer (the hive's wanted-state
watch, the controller's KV key listings) delivers under that prefix, so
the one inbox grant covers them all.
A hive or controller still on a build without the prefix has every
JetStream reply refused once the responder runs this policy, so the
clients need to be rebuilt before, or with, swarm-nats-auth.
Closes#4945
`continue_` cleared the stop reason, reset the turn budget and remembered
the caller's `dir` before the session lookup ran, and nothing restored
them when the lookup refused. With the strict lookup a refusal is an
expected outcome (the caller omitted `dir`), so every such retry erased
why the session stopped from `status`, gave a capped goal session its
budget back, and made the wrong `dir` the remembered one for the next
bare `continue`.
The lookup now runs first, against the dir peeked rather than committed;
only a session that is found gets `State::begin_continue`, which does
all three. Same for the ACP path's recorded-session check.
resume_target no longer falls back to the bare name when the project dir
holds no session titled it: continue (and a goal run's next turn) refuses
before anything spawns, naming the directory searched. A store that cannot
be resolved refuses too, with the io error. The fallback notice is gone.
resume_target logs the fallback and returns a notice naming the project
dir searched; continue appends it to its reply and to a failed resume's
error, so a caller hitting claude's 'matches 2 sessions' knows to pass dir.
The test's final assertion now pins the whole attach list so the continue
site is covered on its own, and the fallback branch has a test.
`continue`, and every later turn of a goal run, passed `--resume <name>`.
claude 2.1.287 resolves a title across the project dirs of every git
worktree of the repo and refuses when two sessions match, while `start`
archives a prior session only in its own cwd's project dir. A name first
started under one worktree and then started again under a sibling stayed
live in both, and every resume of it failed with `--resume "<name>"
matches 2 sessions`. That is the #5000 incident: build-4921a/b were first
run under /home/atlas/wt/4901-build, then under their own worktrees.
Both resume sites now look up the live session titled `name` in the
config's own project dir (the same `SessionStore::find_by_title` lookup
`start` archives with) and pass its id. The id comes off disk at each
resume, so it survives a daemon restart. When that dir holds no session
under the name, the name is passed as before and claude's answer stands.
The test drives a fake claude that keeps sessions on disk and resolves
titles across project dirs the way claude does. Its script is written by
a child process, so no write fd on it exists in the test binary for a
concurrent fork to inherit (the ETXTBSY of #4898).
When the store re-read failed or timed out, introspect fell through to the
unchanged-secret branch and logged "the stored OIDC client secret is the one
the IdP refused" after the re-read's own failure line. The re-read now says
whether it read the store, and a failed one denies without the second line.
The verdict is unchanged.
The tmpfiles `r` rule for the stale `oidc-client.secret` in the
swarm-nats container sat in swarm-nats.nix under `deploy.nats.enable`,
so a host that turned the queue off kept the copy. It now sits in the
ungated removal shim next to the seed copies' rules, which is where the
S2 plan put it and the one file to delete when the deprecation window
closes. The module-eval case asserts the rule on a host with the queue
off as well.
The re-read of the responder's own OIDC client secret had no timeout:
neither swarm-secret-client nor vaultrs nor reqwest sets one by default,
and while a re-read runs every later refusal joins it rather than
starting another. A store connection whose peer stays alive but silent
therefore held off rotation pickup indefinitely. The spawned re-read
now gives up after 10 s, logs that it did (without the secret), keeps
the secret in use, and frees the slot for the next refusal's re-read.
The link from a non-2xx introspection (identify_caller's
`Answer::Refused`) to that re-read had no test: every existing test
handed `OidcSecret::introspect` an `Answer` directly. A new test runs
the real `identify_caller` against a loopback IdP that answers 401 to
the old secret and admits the new one, so mapping a non-2xx to
anything but `Refused` turns it red. axum is a dev-dependency for that
stub, as in swarm-controller's forge tests.
The responder reads its OIDC client secret at
`swarm/services/<client-id>/oidc/client` with the store login it already
opens for the callout keys, and holds it in memory. `--client-secret-file`
is removed.
When introspection answers non-2xx, the responder re-reads the secret from
the store once and, if it changed, retries the introspection once. Both fit
inside the request's 1.5s introspection budget, under the server's 2s
callout timeout: the budget is now one deadline for the whole request. A
re-read that does not finish by then keeps running in the background, the
request is denied, and the next request introspects with what it read. One
re-read runs at a time; a refusal while one runs waits on it.
In swarm-nats.nix the `swarm-bao-nats-auth-oidc` fetch unit, the flag and
the `oidc-client.secret` LoadCredential entry are gone, along with the
unit's policy ordering in glue-bao-readers-policy-order.nix. A tmpfiles `r`
rule removes the copy a container still holds at
/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret;
nothing reads it.
No policy change: the `swarm-nats-auth` role already reads this path
(`natsAuthReaders` in swarm-bao.nix), which is the role the removed fetch
unit logged in as.
Deploy order: none, and no operator step. The store's grant already exists,
so the queue host can be deployed before, after or without the store's host
and authelia's. Rolling back re-creates the fetch unit and the file; the
store leaf is untouched.
On a hive fronting no swarm name, swarm-services-cert's token-less
GET pki/ca/pem presents baoHiveIdentity{Cert,Key}File again. The fetch
never logs in and grants nothing; the client certificate only completes
the store listener's mTLS. A hive host without the store therefore needs
no copy of the services-issuer pair.
The issue path, which logs in to pki/issue/swarm-services, still presents
only deploy.bao.servicesIssuerClient{Cert,Key}File.
module-eval-hive-tls: the root-fetch cases pin the hive identity and the
absence of the services-issuer pair (a fixture holding both, and one
holding only the hive identity); the issue-path case pins the pair on a
hive host and on a host without a hive; a gateway without the pair
evaluates and its issue path names both options.
Root-fetch identity: option (a), mara on PR #5002 (comment 92591).
Refs #4921
Drop the eval assertion b78190e1 added. It refused every self-signed
gateway host that does not run the store, which is the default hive
shape, and contradicts docs/swarm/secrets.md's supported no-store
deployment. The approved #4921 plan has such a host evaluate and fail
at runtime; `requireServicesIssuer` already exits 1 naming both
options, on the root-fetch branch and before issuing.
The module-eval case that pinned `hiveWithController { }` as refused
now pins it, and a hive-less gateway with no pair, as evaluating with
the unit's refusal in the script. The fixture pairs the assertion
forced into forge-placement, nats-tls and swarm-otel-identity are
reverted.
Refs #4921
swarm-services-cert issues the swarm-services leaf a self-signed gateway
serves, and on a host fronting no swarm name fetches the store's pki root.
It is gated on the gateway, not on a hive, yet its leaf pair was named
under deploy.hive-controller.tls and defaulted from the hive reader's cert
directory, and the root fetch presented the hive's own identity. A gateway
host with no hive had no such identity, so that branch was red by
construction.
- The pair moves to deploy.bao.servicesIssuerClient{Cert,Key}File, beside
the store's other host-unit readers, and glue-bao-tls.nix defaults it on
the store's host from the leaf it already mints. The old
deploy.hive-controller.tls.baoClient{Cert,Key}File paths are
mkRenamedOptionModule aliases. glue-services-issuer-bao-identity.nix,
which derived the default from deploy.bao.clientCertFile, is gone.
- The no-swarm-name root fetch presents the services-issuer pair. The hive
identity is now only hive-tls-resign's.
- A self-signed gateway with no pair fails evaluation, naming both options.
A pair named but unreadable still fails the unit, naming the paths.
Refs #4921
`swarm-bao-nats-callout-xkey` runs `ensure-callout-keys`, importing
/var/lib/swarm-nats-callout/{issuer,callout-user}.seed when present so the
keys keep their public halves, then fetches all three public halves from
the store. The verb removes each seed file once a read of the store returns
a seed; when the store's seed differs, it is kept and a warning names the
file. No tmpfiles rule touches those files, since one could run before the
import with the store sealed and leave the next run to mint a new key.
The server renders the issuer and user public halves from the fetched files,
checked for shape as the xkey's is, and does not start without them. The
responder reads both seeds from the store, so the copies into the container
and their LoadCredential entries are gone, and a tmpfiles rule removes the
copies a container still holds. `swarm-nats-callout-keys` is removed.
On the store's host the fetch unit is also ordered after
`swarm-bao-secret-publisher-policy`, so the publisher's denial of the seed
paths is written before the unit creates or keeps a seed there.
`deploy.nats.autoGenerateCallout`, `calloutUserPublicKey`,
`calloutIssuerPublicKey`, `calloutUserSeedFile` and `calloutIssuerSeedFile`
are removed with `mkRemovedOptionModule`, under `swarm.nats` too, each
message saying the keys live in the store.
The `swarm-nats-auth` policy grants create and read, never update, on the
seed and public half of the callout issuer and user keys, beside the xkey's
two leaves: `ensure-callout-keys` creates them under that role, and the
responder reads both seeds as it starts.
The `swarm-secret-publisher` policy, which may write the rest of
`swarm/services/`, is denied both seed paths exactly, as it is the xkey's.
A planted issuer seed would be the authority to admit anyone to the queue.
Both policies are written by the granter on every deploy; nothing about the
granter's own policy changes.
The responder loads the issuer and callout-user seeds from the secret
store under its own role, beside the xkey, and no longer takes
--issuer-seed-file / --user-seed-file.
`ensure-xkey` becomes `ensure-callout-keys`: it ensures the xkey as before,
and for each nkey creates the seed create-only when the store holds none,
from --import-{issuer,user}-seed-file when that file is present, so a key
kept on disk keeps its public half. Once a read of the store returns a seed,
the import file is removed; when that seed differs from the file the
store's is kept and a warning names the file. The public halves are
published create-only at swarm/public/swarm-nats/callout-{issuer,user}.
The KeyStore trait and its test double move to `keystore` so both key
modules share them.
glue-nats-bao-identity.nix and glue-matrix-ctl-bao-identity.nix defaulted
deploy.nats.baoClient{Cert,Key}File and deploy.matrix.ctlBaoClient{Cert,Key}File
to siblings of deploy.bao.clientCertFile, the hive's own store reader. That
pointed a store host whose reader was moved at files nobody mints, and on a
queue or homeserver host without the store that held a placed hive reader
invented a nats.pem / matrix-ctl.pem path beside it.
Both glues are now gated on the store running on this host
(deploy.bao.enable, the condition glue-bao-tls.nix mints under) and name the
directory the minting unit writes to, through nix/host-modules/lib/bao-pki-dir.nix.
Host shapes:
- Store host, default reader: unchanged. The evaluated paths stay
/var/lib/swarm-bao-pki/nats{,-key}.pem and
/var/lib/swarm-bao-pki/matrix-ctl{,-key}.pem, and the rendered units are
the same.
- Store host whose hive reader was moved: now gets the leaf minted there.
- Queue or homeserver host without the store, holding a hive reader and not
naming these options: no default any more. It still evaluates (neither
option has an eval assertion). swarm-bao-nats-tls exits 1 naming
deploy.nats.baoClient{Cert,Key}File when the queue's certificate is missing
or due for renewal; swarm-bao-matrix-oidc exits 1 naming
ctlBaoClient{Cert,Key}File, and the swarm appservice render/publish units
and the PKI bind mount no longer render. If that host had placed nats.pem
or matrix-ctl.pem beside its hive reader, it worked before and now needs
both options named.
- Hosts without the store and without a hive reader, and hosts naming the
options: unchanged.
Both refusals and option descriptions now say the default comes from
deploy.bao.enable.
Refs #4921
glue-nats-auth-bao-identity.nix defaulted deploy.nats.authBaoClient{Cert,Key}File
to a sibling of deploy.bao.clientCertFile, which is the hive's own store
reader. That tied a swarm service to a hive identity, pointed a store host
whose reader was moved at a file nobody mints, and on a queue host without
the store that held only a placed hive reader invented a nats-auth.pem path
that passed the eval assertion and failed at runtime.
The glue is now gated on the store running on this host (deploy.bao.enable,
the condition glue-bao-tls.nix mints under) and names the directory the
minting unit writes to, shared through nix/host-modules/lib/bao-pki-dir.nix.
A queue host without the store gets no default and the eval assertion
refuses it until the responder's own leaf is named; the assertion message
now names deploy.bao.enable as where the default comes from.
On a default store host the evaluated paths are unchanged:
/var/lib/swarm-bao-pki/nats-auth{,-key}.pem.
Refs #4921
