swarm-metrics reads the swarm's VictoriaMetrics from inside an agent
container: query, query-range, series, labels and label-values, each one
GET against the store's Prometheus read API through the gateway, printing
the store's JSON. It authenticates the way swarm-logs does, with the
agent's own queue secret read from bao under the agent's leaf, sent as
a swarm-agent bearer.
On the metrics store's host the read API's auth map gains an agent arm
ahead of the Grafana one. It routes to swarm-metrics-agent-auth, a
swarm-logs-auth instance logged in under the existing swarm-metrics-auth
leaf, whose bao grant now includes swarm/agents/+/queue read. The check is
rate-limited per address like the log store's
(deploy.victoriametrics.agentAuth{Rate,Burst,VerdictTtl}).
nix/agent-modules/metrics.nix installs the wrapped binary when the hive
forwards HIVE_AGENT_METRICS_URL (from swarm.victoriametrics.domain) and a
store address, the same gate as swarm-logs. Adds the base:swarm-metrics
skill and the CLI reference with its freshness check.
Fixes#5395
A swarm-ci-metrics host unit on a 5-minute timer lists every forge
repo's Actions runs as the new read-only ci-metrics account and pushes
each finished run's duration and queue wait over OTLP, stamped at the
run's stop time. The first pass backfills from run 1. swarm-controller
mints the account's token into bao and keeps a ci-metrics team with
read on the Actions unit in every organisation.
Closes#5376
The dashboard, its API, the swagger UI and the agent routes move to their
own vhost, `swarm.dashboardDomain` = `<hive>.hives.<swarm.domain>`, under
the swarm domain so authelia's session cookie reaches it. `/`, `/api/` and
`/api/docs/` ask authelia through `auth_request`, and authelia's
`operatorsOnly` rule for each hive's dashboard name lets only the admins
group in. The hive domain keeps matrix discovery and redirects every other
path to the new name.
The name rides as a SAN on the hive's own gateway leaf: the per-hive bao
role allows it, and hive-tls asks for it and checks coverage.
HTTP Basic auth is gone: the htpasswd writer and its test, the
`hivectl gateway` verbs and host-socket requests, the `gateway.auth`
options (now a removed-option error), the unauthorized page and the bcrypt
dependency. tmpfiles deletes a leftover gateway.htpasswd, which held
bcrypt hashes at 0644.
c0re hands out dashboard and agent URLs on the new name
(HYPERHIVE_DASHBOARD_DOMAIN), and the swarm controller's hive roster
carries each hive's dashboard name for the swarm UI's hive list.
Closes#3469
A new swarm service, `swarm-claude-login`, in its own container on the host
that sets `deploy.claude-login.enable` (defaulted on by `allSwarmServices`
where the host has the minter's store leaf, like the review classifier).
- swarm-ui: an agent card showing "login needed" gets a "log in" button.
Its dialog starts the login, shows the URL claude printed, and takes the
pasted code.
- swarm-controller: `POST/DELETE /api/agents/{name}/claude-login` and
`POST /api/agents/{name}/claude-login/code`, behind the swarm UI's
authelia-gated `/api/`, relay the steps to the minter as requests on
`$SWARM.claude-login.request`.
- swarm-claude-login: runs `claude auth login` over piped stdio with HOME
and CLAUDE_CONFIG_DIR in a per-login scratch dir, scrapes the first
https URL, writes the code to stdin, and stores the `.credentials.json`
at `swarm/agents/<agent>/claude-credentials` with a KV v2 write, a new
version over any earlier one. The scratch dir goes on every exit path; a
login waits 15 minutes for its code.
- hive-agent: while parked in `wait_for_login`, reads that leaf every 30 s
under the agent's own cert and installs it only when its KV version is
newer than the one recorded in `<harness>/claude-credentials-version`.
An equal or older version is never put back, because claude rotates the
refresh token and the stored copy is stale after the first refresh.
- swarm-secret-client: `claude_login` paths and `SecretStore::read_versioned`.
- Credentials: a `claude-login` store leaf minted by glue-bao-tls, a
`swarm-claude-login` policy (create/update on
`swarm/agents/+/claude-credentials`, read on its own queue client
secret), an authelia machine client, and a `--claude-login-client`
grant in the queue's auth-callout responder. The unit runs as a static
system user.
The hive's own web-UI login is unchanged.
Refs #5268
The metrics vhost forwarded the store's read API (query, series, labels,
export, vmui, ...) with no credential, so anything that could reach
metrics.<swarm> could read every hive's and agent's metrics. mara ruled on
checked by an auth_request responder, and operators use their authelia
session.
- swarm-metrics-auth (new crate): auth_request responder on the metrics
store's host. Compares the presented bearer, in constant time, with the
one stored at swarm/services/<grafana client id>/metrics/bearer, read
under its own bao leaf and cached for 60s. Static user, the post-#5264
swarm-logs-auth unit shape.
- swarm-bao-grafana-metrics-bearer: on Grafana's host, creates that bearer
in bao if absent (create without update, Grafana's existing leaf) and
writes "Bearer <value>" into the container; the datasource sends it as
its Authorization header via $__file{}.
- The vhost: read locations take auth_request through a map (a bearer goes
to swarm-metrics-auth, anything else to authelia, no login redirect);
vmui takes an authelia session with the login redirect; forceSSL like
the logs vhost. The authelia subrequest location is renamed from
/__metrics_push_authz to /__metrics_authelia since it now serves reads.
- Authelia: the collector keeps every path; everyone else is refused the
collector's push and /metrics paths; group admins is admitted to the
rest; then deny.
- bao: a swarm-metrics-auth cert role and policy (read on the one leaf),
its leaf minted by glue-bao-tls, the CN reserved as a hive name, and
Grafana's grant gains create on the bearer leaf.
Fixes#3596
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
A new `auth_request` responder on the log store's host. nginx sends it
every `/select/logsql/` request whose `Authorization` is
`Bearer swarm-agent.<agent>.<secret>`; it answers 200 when the secret is
the one stored at `swarm/agents/<agent>/queue` (`swarm-agent-verify`),
401 otherwise. Every other bearer still goes to authelia, so agents
still presenting the hive's shared OIDC client keep working.
`auth_request` takes a fixed URI, so the split is a `map` on
`$http_authorization` plus an internal location that rewrites to the
map's pick. Checked against nginx 1.30.5 with stub upstreams: an agent
token only the stub validator accepts gets 200, a token only the stub
authelia accepts gets 200, every wrong one 401, and the internal
locations 404 from outside.
The responder serves a unix socket in its own traverse-only runtime
directory and reads the store under its own leaf
(`swarm-logs-auth`), whose policy is `swarm/agents/+/queue`, read-only.
glue-bao-tls mints the leaf where the store runs; elsewhere the
operator places it, and eval refuses a log store host without one.
Part of #4459.
`CredentialSource`, `Store` and `verify` move from swarm-nats-auth's
`agent_token` module into a small library crate, so a second verifier
can make the same check: the secret in a `swarm-agent.<agent>.<secret>`
token equals the one stored at `swarm/agents/<agent>/queue`, read under
the verifier's own certificate.
`verify` now takes its timeout as an argument; swarm-nats-auth passes
`INTROSPECTION_TIMEOUT` as before. No behaviour change. The store-denial
tests moved with the code; swarm-nats-auth keeps the ones that go
through its policy and its bound.
Part of #4459.
A new service crate. It reads its queue client secret from the store at
start, takes try-outs on the review classifier subject in its queue
group, and for each run reads the Anthropic API key from the store,
builds the shared context (commits with full messages, diff with
lockfiles collapsed), runs one claude --print per classification with no
session, no tools, no MCP server and the key in the child's environment
only, validates the JSON answer and aggregates the rows. A missing key or
a run where no classification got an answer fails the run; one failed
classification is an excluded row with its error. One journal line per
run names the operator, repo, PR, head SHA, rows and overall verdict.
swarm-secret-client gains the two paths the service reads.
hive-claude 0.1.4 (hive-claude #28) sends the idle watchdog's signal as
SIGINT to claude's process group instead of SIGKILL. It then waits for
claude to exit, with no SIGKILL fallback. claude stops the commands its
Bash tool started when it shuts down on SIGINT, and a SIGKILL left them
running. This is the last SIGKILL of a claude process from #4999.
hive-agent's `idle_timeout` -> `Error::IdleTimeout` -> `TurnError::ApiStall`
path is unchanged. The difference is that the error now arrives only
once claude has exited. A claude that ignores the SIGINT keeps the turn
open until it is killed by hand or the unit stops.
The Cargo.lock change is only hive-claude's version and checksum. A plain
`cargo update -p hive-claude --precise 0.1.4` also moved seven crates'
Windows-only `windows-sys` edges from 0.61.2 to 0.52.0, unrelated to this
bump. The minimal lock passes `--locked`.
The turn-loop and web-UI docs, and hive-agent's watchdog docs and stall
note, now say SIGINT instead of "killed". That includes the ACP spots
that the previous commit's `Drop` change made stale.
hive-claude 0.1.3 spawns claude as the leader of its own process group,
and its `Cancel::cancel` and idle watchdog signal that group instead of
claude's pid alone. Commands claude started (a `nix build`, a test run)
used to survive a subagent `interrupt` and keep running.
Bump hive-claude to 0.1.3. The subagent `interrupt` call site needs no
code change: `Cancel::cancel(force)` kept its signature.
The harness's `/cancel` and `/logout` did not go through hive-claude:
they ran `kill -INT <pid>` on the claude child they found in /proc. They
now SIGINT the group that child leads, through `libc::kill`. A `claude`
child that leads no group (the `claude plugin` calls) shares the
harness's group and is still signalled alone.
Tests: an interrupted subagent's grandchild dies, for both SIGINT and
`force`, and fails on hive-claude 0.1.2; a harness cancel stops a
turn's grandchild, and fails when only the pid is signalled.
A `Runtime` trait (run / compact / archive) with two backends:
- claude: a pass-through to hive_claude's InfiniteSession and
SessionStore, so a claude turn is the same spawn, session handling
and errors as before.
- acp: a generic Agent Client Protocol client. It spawns the command,
args and env from RuntimeSpec (HIVE_RUNTIME / HIVE_ACP_COMMAND /
HIVE_ACP_ARGS / HIVE_ACP_ENV), refuses an agent whose
mcpCapabilities.http is not true, passes the claude --mcp-config
servers as ACP mcpServers, keeps one session id in a file
(session/load after a restart, session/new otherwise), and maps
session/update into claude stream-json events plus usage_update into
Telemetry. Permission requests are answered by a caller-supplied
policy on the ACP tool kind. compact returns Unsupported for now.
The crate depends on no hyperhive binary crate, so the subagent
daemon can move onto it without pulling in hive-agent.
Refs #4391
A five-minute pass over every agent some hive's wanted state declares as
anything but destroyed queues, per agent:
- `MintAgentIdentity` (the node agent creation uses) when the stored
certificate at swarm/agents/<agent>/bao-mtls is past half its validity,
read from its own notBefore/notAfter: day 45 of the role's 90;
- the new `RenewAgentQueueCredential` node when the queue secret at
swarm/agents/<agent>/queue is 45 days old or has no mint time. The node
re-decides, writes a fresh value with `minted_at`, reads it back, and logs
the agent and the old age.
When both are due the secret node runs after_any the certificate node,
because mint_and_verify compares the queue secret it read with the one it
reads back. A credential that is not stored is never created here.
`queue::AgentCredential` gains an optional `minted_at` (unix seconds);
agent creation now sets it. Stored objects without it decode unchanged and
count as due, so every existing queue secret is re-minted on the first pass.
Both replacements reach the agent at its next start. The old certificate
stays valid until it expires; the old queue secret does not, so a queue
reconnect before that restart is denied.
Adds x509-cert 0.2 (with der_derive and flagset) to read the validity.
docs/swarm/credentials.md: the renewal column splits into automatic re-mint
and automatic re-pull, filled from the code as it stands.
An `auth_token` spelled `swarm-agent.<agent>.<secret>` is no longer sent
to introspection. The responder reads `swarm/agents/<agent>/queue` with
an identity of its own, checks that the stored object names the same
agent, compares the secret in constant time, and grants the subjects
`--agent-token-publish-subject` lists with `{agent}` expanded. Every
other outcome denies: a malformed token, no store identity, nothing
stored, a failed or slow lookup, a different secret. A token without
the prefix takes the OIDC path unchanged.
The journal's `auth request` line names such a caller `agent:<agent>`;
the hive-shared credential keeps `hive-<h>-agent`.
The new principal: a `swarm-nats-auth` cert-auth role and policy with
read on `secret/data/swarm/agents/+/queue` alone, a leaf signed by the
store's PKI glue, and `glue-nats-auth-bao-identity.nix` pairing the two.
The copy unit delivers the identity into the queue's container, and an
absent leaf is delivered empty so the responder still starts and only
agent tokens are refused.
The policy and role are written by `swarm-bao-nats-auth-policy`, logged in
as the bao granter: both names fall under its `swarm-*` globs, so the
deploy writes them with no operator step. module-eval counts it among the
granting units, so every generic granting-unit case covers it.
The secret compare uses `subtle`, already in the lock file through the
TLS stack; no workspace crate offered one directly.
An agent's store identity was signed in swarm-controller's memory by a CA
a controller-host unit generated on disk, and the listener never trusted
that CA. Agent leaves now come from the store itself: a `pki-agents` PKI
mount whose root openbao generates internally, so the agent CA's key
never exists outside the store.
- swarm-bao-agent-pki (new, store host, as the bao granter): enables and
tunes the mount, generates the root once (guarded on an empty issuer
list, no replace branch), upserts the `swarm-agent` role (client
certificates named `hive-agent-*` only, 90 days), caches the CA at
/var/lib/swarm-bao-tls/agent-ca.pem and composes the listener bundle.
- The listener's tls_client_ca_file is a new listener-client-ca.pem
(client-ca.pem, then the agent CA). Host cert-auth roles still pin
client-ca.pem, so an agent leaf satisfies no host role. swarm-bao-certs
composes the same bundle before openbao starts.
- openbao reads tls_client_ca_file only at start, so when the bundle
changed after openbao started, swarm-bao-agent-pki restarts
openbao.service in the container; under `seal = "shamir"` it prints
the step instead. Once swarm-bao-certs has a cached CA, later boots
start openbao with it and do not restart.
- The controller policy gains exactly `update` on
pki-agents/issue/swarm-agent. mint_and_verify now asks that role for
the leaf (the store generates the key), writes the agent's cert-auth
role pinning the issuing CA bao returned, and writes the agent's
policy as render_agent alone: the hive-shared queue credential stanza
is gone.
- deploy.bao.agentPkiRoleName (must start `swarm-`, asserted with the
other pki role names); swarm-controller gets
SWARM_CONTROLLER_AGENT_PKI_MOUNT/_ROLE from the deploy.bao options.
Deleted: swarm-controller-agent-ca and its options (agentCaFile,
agentCaKeyFile), env, LoadCredential entries and assertion;
agent_identity's Authority, rcgen signing and validity window; the
rcgen and time dependencies of swarm-controller (rcgen leaves the
workspace); policy::render_agent_with_queue and its tests. The CN-prefix
assertion policy.rs said was owed is not: agent and host roles pin
different CAs.
Migration is re-creating each agent after deploy; that overwrites the
stale role and policy.
Closes#4756
rustls is built with both `ring` (async-nats's `ring` feature) and
`aws-lc-rs` (reqwest's `rustls` feature), so it cannot pick a
process-level default by itself. Since the queue started requiring TLS
(1d261b3f), async-nats builds its config with `ClientConfig::builder()`,
which panics without an installed default. The panic kills the async-nats
connector task, and every queue client (swarm-controller, hive-c0re, all
hive-agents) has sat in `Pending` since the 2026-09-25 23:04Z deploy.
Add `swarm_queue_client::install_crypto_provider()`, which installs
aws-lc-rs and ignores the "already installed" error. It is called first in
`main` of every binary that links async-nats: hive-agent, hive-c0re,
swarm-controller, swarm-nats-auth. `connect()` also calls it, so a new
binary that dials through this crate is covered without remembering to.
aws-lc-rs because reqwest already falls back to it when no default is
installed, so HTTPS in these processes keeps its current provider. The
other rustls users in the tree reach it only through reqwest, which never
panics here.
Closes#4738
The swarm gets an appservice identity of its own, separate from each hive's
`hyperhive` registration. `swarm-matrix-ctl appservice render` mints its
tokens inside the matrix container when they are absent and renders the
registration tuwunel loads; `appservice publish` writes its as_token to
`swarm/controller/swarm-controller/matrix/appservice-token`, the one kind no
hive's policy grants.
The homeserver calls move out of swarm-matrix-ctl into swarm-matrix-client,
with a `whoami`, so swarm-controller can mint agents' accounts through the
same pinned device id instead of a copy of them.
Every agent on a hive authenticates to the swarm queue with the same
hive-scoped secret today, so at the auth callout one agent is
indistinguishable from its co-hived neighbours and no subject can be
scoped to one of them.
Mint a secret per agent instead, at swarm level, into
secret/swarm/agents/<agent>/queue -- inside the stanza every agent's ACL
document already grants, so no policy changes and no existing agent's
document is rewritten. It is written by the same node that already mints
the agent's certificate, and read back under the agent's own token
before that node reports success.
The secret is not derived from the agent's mTLS identity: the two
credentials answer different questions and coupling their lifetimes
would mean renewing either implied renewing the other. Nothing here
rotates a queue secret -- a re-run keeps the existing value and only
corrects the principal it names, because this function is re-run
deliberately against agents that are already connected. Revoking one
means deleting the path.
Nothing reads the new credential yet; this is the minting half.
A record written to stdout carries no priority, so journald files the
whole stream at one level and the swarm log store shows `info` whatever
level `tracing` gave it. Under a systemd unit the process's stdout
already *is* the journal, so the fix is to speak the journal protocol
directly and let each record carry its own severity.
New `hive-log` crate holds the one sink chooser, called by `hive-c0re`,
`hive-agent` and `swarm-controller`. It builds the same `EnvFilter`
those binaries always built, then installs exactly one layer — never
both, since a journald layer stacked on the `fmt` layer under a unit
stores every record twice.
The choice is an fstat compare, not a presence test: a child inherits
`$JOURNAL_STREAM` even when its own stdout was redirected elsewhere, so
the variable existing proves nothing. The crate parses `dev:inode` out
of it and compares both numbers against an fstat of stdout, the
descriptor the `fmt` layer writes to by default. No match, unset, or
unparseable takes the `fmt` branch. A journald layer that fails to
construct despite a match falls back to `fmt` and warns through it —
a process must never fail to start because of its logger.
Renames `swarm-matrix-minter` and reshapes it around subcommands. Minting
is now `swarm-matrix-ctl mint`.
Running rust inside `containers.hive-matrix` is not free: it needs its own
store identity, its own cert role and its own bind mounts, and every one of
those is per-*container*, not per-task. A second single-purpose crate would
have had to duplicate that plumbing to add one action, so the next thing
that has to run in there should be a verb here rather than a new crate.
The old name guaranteed the opposite.
`main.rs` is clap dispatch; the minting logic moves to `mint.rs` unchanged.
A bare invocation is refused: `mint` writes a credential, so "no verb"
defaulting to it would make a typo in the unit mint rather than fail.
The environment prefix moves with it, `MATRIX_MINTER_*` → `MATRIX_MINT_*`.
Scoped to the verb and not to the binary, because a binary-scoped prefix is
one the next verb has to share or widen, and a widened one never narrows
again. A test asserts every variable carries the verb's prefix.
The principal renames too. The cert role, bao policy, granting unit, leaf
filename and `certAuthCns` entry all have to spell one string the same way,
so leaving them as `swarm-matrix-minter` would have rebuilt the naming
split this branch exists to remove. Renaming the nix options alongside is
free here: every one of them is introduced by this PR and has never been
released, so no operator config names them yet.
`ExecStart` now names the verb, which is a contract between a nix string
and a clap enum that fails at deploy time with no local signal. Both ends
assert it: `mint_is_spelled_the_way_the_unit_invokes_it` in the crate, and
a new module-eval arm reading the rendered `ExecStart`.
docs/getting-started/setup.md drops the sender token from its "live on the
host" list: setup does not touch this credential, so a setup guide has no
reason to name it.
A swarm runs one homeserver and a homeserver has one appservice sender
account, so "mint it once" is a property of the thing being minted
rather than something a lock has to enforce. That is what makes this
account the one to move first: no trigger route, no controller change
and no agent list — a boot-time oneshot beside tuwunel is the whole
mechanism.
`swarm-matrix-minter` runs inside `containers.hive-matrix`, which
already holds the appservice token: the rendered registration is bound
in read-only because that is how tuwunel is handed it. What the
container lacked was an identity of its own, so this adds one — a leaf
from the store's CA with a grant of exactly one path, not the hive's
leaf, which reads every secret in the store.
Both ends of the credential ship here. The minter reads the path it
publishes to before it touches the homeserver, and returning on a
non-empty read IS the "only once"; `hive-c0re`'s `ensure_hive_user`
reads the same path, authenticating with the hive name already in
`HYPERHIVE_HIVE_NAME`. The existing mint-then-`M_USER_IN_USE`-login
ladder stays as the fallback for a store that is empty, unconfigured or
unreachable, which is every swarm deployed before this — so nothing
needs backfilling and nothing breaks if the rest of the sequence never
lands.
The credential is not an admin credential, and is not named like one.
It is the access token of the appservice registration's own
`sender_localpart` — `@hive:<server_name>`, an account the homeserver
creates for itself when it loads the registration. The store path is
`swarm/services/matrix/sender-token`, the host path is
`matrix/access-token`, and the homeserver no longer runs an
`admin_execute` promotion for that account at boot. Everything the hive
provisions with it — the Space, the chat room, their hierarchy and join
rules, the invites — rides on being the creator of those rooms at power
level 100, not on homeserver admin; there is no Synapse admin API here
to need, tuwunel has none.
Two operations do need an admin *sender* and therefore stop working:
`hivectl matrix promote-user` and `hivectl matrix reset-password`, both
`!admin …` messages into `#admins:<server>`, plus the password-reset
recovery path that an agent with a lost password file falls back to.
They are swarm-level operations and are left failing loudly rather than
served by an over-privileged token every other call site would also
carry. The sweep's own admin-rights check and self-repair go with them:
an account that is deliberately not an admin has nothing to check.
`ephemeral = false` stays, and hive root can still read the container's
filesystem. Accepted: what this buys is identity separation — no hive
*process* holds or reads the appservice token — not physical isolation.
Refs #4345
`swarm/agents/<agent>/bao-mtls` did not exist, and neither did any
per-agent identity at the secret store: `policy::agent_object_name`,
`render_agent` and `render_agent_with_queue` had been written and never
called outside their own tests. An agent's only "per-agent" secret today
is read under the HIVE's certificate, through a wide grant on
`swarm/agents/*` — so "per-agent" was presentational.
The swarm now mints the certificate, so no hive ever needs the capability
to mint one. `swarm-controller` is the service that does it: it already
logs in to the store, and its existing grant already covers exactly the
three objects written here (`create/update` on
`secret/data/swarm/agents/*`, `sys/policies/acl/hive-*` and
`auth/cert/certs/hive-*`). No new bao grant, and nothing co-located — a
cert-auth role pins its authority by value, per role, so the controller
issues from its own CA on its own host and pins that CA in the role it
writes. No existing role changes.
The mint node does not report success on a write. After publishing it
connects again, with the leaf it just issued and under the role it just
wrote, and reads the path back — so the policy, the role, the common name
and the leaf are exercised in production on every agent creation. A
certificate this code mints that the role this code writes will not accept
turns the job node red at creation time instead of surfacing later as an
agent container that cannot start.
`TriggerDeploy` gains an `after_any` edge on the mint, not `after_ok`: a
hive cannot pass down a certificate the swarm has not published, but a
host with no authority configured must still create agents exactly as it
does today.
The private key is generated in memory and never written to disk on the
controller — `SecretStore::connect_with_identity` takes the PEM the minter
is already holding, so nothing is written out purely to be logged in with.
Refs #4137
An agent can reach VictoriaLogs only through the gateway, and since the
machine query route landed the way to read it has been to hand-roll a
client_credentials token request and a curl, per query. This is the CLI
that closes that: `swarm-logs query '<LogsQL>'`, matched log lines on
stdout, so the answer pipes into grep like any other command's.
Built to the plan posted on the tracker thread: own crate, own
docs/tools reference generated off the clap tree, `query` as the one
verb, and the JSON error body surfaced on a non-200 rather than
swallowed. No `tail`: streaming is a different endpoint with a different
response shape, and folding it in here would be a fatter scope than the
ask.
Minting the token is NOT implemented here — swarm-queue-client already
owns the client_credentials request, its error type and its CA handling,
and a token-endpoint fix has to be findable in one place. What this crate
adds is the agent-shaped half: the client id arrives as a *file* beside
the secret, so nothing outside nix/agent-modules/queue.nix spells
`hive-<name>-agent` twice. That is the same problem hive-agent's
swarm_queue module solves, and swarm-logs/src/auth.rs is its `decide`
restated over this binary's inputs.
⚠️ The plan named one thing to verify empirically before calling the auth
settled: whether authelia's bearer policy for the logs vhost accepts the
agent client's audience. Measured from inside a container: it does not.
The client minted a token fine but with `aud: []` and `scp: []`, asking
for the logs URL as an audience answered `invalid_target`, and presenting
the audience-less token to the gateway answered a bare 401. So
swarm-authelia.nix's agentClients gains `authelia.bearer.authz` and the
query URL as a second audience — authelia authorises a bearer token by
the URL being requested, and that URL is now one binding read by three
places rather than three spellings of one address.
The URL reaches an agent the same way its queue coordinates do: computed
on the host (a container cannot derive a gateway address), forwarded by
hive_c0re::meta into the container's option set, and consumed by a new
agent module that installs the binary *wrapped* with its coordinates —
the shape swarm-controller.nix installs swarmctl in. Gated on the queue
credential as well as on the URL: a binary that can only answer 401 is
worse than no binary, because an agent reads a 401 as "no logs", which is
the exact confusion the store's machine route was added to end.
`write_policy` called `vaultrs::sys::policy::set`, which targets
`sys/policy/<name>` — the deprecated alias the store gates as a path of
its own. Every grant in this tree names `sys/policies/acl/hive-*`, so a
correctly-scoped controller was refused with a 403 and no hive read
policy has ever been written, on any deployment. The doc comment three
lines above the call already named the modern path; the code is what
moves to meet it.
vaultrs has no endpoint for that path (`grep policies/acl` over 0.8.0:
zero hits, against 8 for `sys/policy`), so this defines one over its own
endpoint machinery — which keeps the client's token header and `/v1`
prefix middleware rather than re-deriving them.
The alternative was to widen the grant to cover the legacy path. This
way needs no policy change at all: the deployed grant already permits
the write, so no store is re-bootstrapped and nothing is entrenched on
an alias upstream has deprecated.
Two tests pin the path and the body shape. The reason this survived
deployment is that nothing in the tree ever asserted either one.
Closes#4177.
mara ruled (a) on #3726: a thin workspace crate over `vaultrs` rather than
keeping bao access in nix and having each end trigger units. The HTTP is the
SDK's job; what this crate owns is the things the controller and a hive must
say *identically*, and which have no other home because neither end is senior
to the other.
Three such agreements:
`path::matrix_account` builds where a credential lives. It is fallible rather
than a `format!`, because both names reach it from elsewhere -- the agent name
from the topology, the account name from an agent's own config -- and a `/` or
`..` in either does not produce a malformed path, it produces a valid path to
a *different agent's* secret. The charset mirrors the KV bucket-name rule.
`Credential`'s `value` field is not a free choice: glue-matrix-bao-token.nix
reads the store with `bao kv get -field=value`, so the name is load-bearing
for a consumer no Rust test can reach. A test pins the serialised shape.
`client::Settings` reads BAO_ADDR / BAO_CLIENT_CERT / BAO_CLIENT_KEY /
BAO_CACERT explicitly instead of letting vaultrs fall through to its own
defaults, which look for VAULT_ADDR / VAULT_CLIENT_CERT / VAULT_CLIENT_KEY.
Every unit in this tree sets the BAO_ spellings, so the defaults would yield a
client with no identity at all -- surfacing as a TLS handshake failure, which
names neither the missing variable nor the reason.
The env read is split from the connect so every misconfiguration arm is
testable without a reachable store and without touching process-global env.
Dependency impact, measured against the lock at forge/main rather than assumed:
native-tls 0 -> 0, openssl-sys 0 -> 0, one reqwest (0.13.4) which vaultrs
shares, and 10 new crates that are all derive/proc-macro helpers.
Refs #3726
forgejo-api 0.11.0 links reqwest 0.12 while the workspace is on 0.13, and
cargo unifies features per (crate, VERSION) — not per crate. So forgejo-api's
internal client was a *second* reqwest, built with neither the workspace's TLS
features nor the system trust store, and every https call it made to a gateway
name failed `invalid peer certificate: UnknownIssuer` while a client built in
this workspace succeeded against the same host.
Pinning to 0.11.1 puts one reqwest in the tree, which is what makes the
existing "feature unification gives forgejo-api our TLS backend" comment true
rather than aspirational — the claim is a fact about the lockfile, so the pin
is load-bearing and says so.
The reqwest feature list moves with it: `rustls-native-certs` and
`webpki-roots` were 0.13.1-era names that no longer exist, and `rustls` now
carries the platform verifier (OS trust store, honours SSL_CERT_FILE). Naming
a feature that is gone is a hard resolution error rather than a silent no-op,
so the list had to change in the same commit as the pin.
Refs #3391
A hive publishing its own status needs the same connect the controller
already has - mint an authelia token, present it at CONNECT for the
callout responder, let async-nats re-run the callback per attempt. Only
the use differs: the controller reads, a hive writes.
Copying it would put credential handling in two places, and a
token-refresh fix would then have to be found twice. That is the same
reasoning that already put hive-sock-client in its own crate rather than
in each daemon that speaks to a unix socket.
`from_env` takes a prefix rather than hardcoding SWARM_CONTROLLER_*: the
variables belong to the consuming unit, since a NixOS module sets them
alongside its other options. What is shared is the RULE - all four
together or none at all - not the spelling. The half-set case gains a
test, because it is the case the rule exists for and it previously had
none.
No jetstream/kv feature on the crate: it ends at a connected client, and
what a consumer does with it should be visible in that consumer's own
Cargo.toml.
Behaviour-preserving, and proven that way rather than by inspection: the
full behavioural gate (real nats-server, credential rotation, mutation)
is 20/0 unchanged, and the controller's own tests still pass.
Per mara on the PR: all deps go into workspace level so versions stay
consistent. async-nats, data-encoding, nkeys and nats-jwt are new
[workspace.dependencies] entries; sha2 and futures-util now come from
there.
The crate had asked for sha2 0.10 while the workspace standard is 0.11,
and for the futures facade where the workspace carries futures-util. Both
resolved without adding a Cargo.lock entry - each was already in the tree
via something else - so the drift was invisible in the lock and would only
have surfaced as two versions of a hashing crate in one binary.
Slice 1 shipped the NATS container with an auth_callout block and no
responder, which is the fail-closed state: the server answers
auth_required and admits nobody. This crate is what lets it say yes.
Connects as the callout-exempt user by nkey (never by name - the server
refuses to start if that entry carries a username), subscribes to
$SYS.REQ.USER.AUTH, validates the presented bearer token against
authelia's introspection endpoint, and replies with a signed NATS user
JWT. A denial is a signed response carrying an error, never silence: a
server that hears nothing cannot tell a refusing responder from a dead
one, so staying quiet would turn every rejection into a timeout and hide
an outage inside what looks like ordinary denials.
Everything that is not an explicit active:true denies - network error,
timeout, non-2xx, unparseable body, no token at all. Those are exactly
the conditions under which an attacker would most like this to fall
open. The introspection budget is held under the server's own 2s
auth_callout timeout by a test, since the two numbers live in different
languages in different files.
nats-jwt mints the user JWT. It cannot mint the authorization_response
wrapper - its claim enum is closed and its claims carry no aud, which
the response needs so a reply cannot be replayed at another server in
the cluster - so that half is hand-written, and a test builds a user
token both ways and requires the bytes to match. That is the only
honest basis for trusting the hand-written path on the shape the crate
does not model.
async-nats is taken with default-features off: the default set carries
jetstream, kv, object-store, websockets and service, none of which a
callout responder speaks.
mara asked whether these can live in Cargo.toml. They can, and they
should: [workspace.lints.rustdoc] sits alongside the existing clippy
table, every crate already opts in via `[lints] workspace = true`, and
the toolchain is well past the 1.74 that introduced lint tables.
The reason it is better than RUSTDOCFLAGS on the check: a plain local
`cargo doc` now fails exactly the way CI does. Setting the lints only in
the nix derivation would have made CI the one place the gate exists,
which is the same "you meet it too late" problem the gate was written to
solve.
The check keeps --workspace --no-deps --document-private-items and drops
the RUSTDOCFLAGS block entirely.
Verified by mutation rather than assumption, with no RUSTDOCFLAGS set
anywhere: clean tree exits 0; reintroducing one broken link makes
`cargo doc -p hive-jobq` exit 101 with `error: public documentation for
Outcome links to private item Scheduler::complete`. An error rather than
a warning is the proof the deny came from the lint table.
The swarm-authelia module states that its users database is written by
swarm-controller, but nothing ever granted the means. This adds the tool
that does it.
swarmctl runs as root on the controller's host and acts directly. The
rootless alternative was examined and does not work: relocating the users
file into a directory the controller owns only turns a write problem into
a read problem, because authelia must then reach across the same boundary
in the other direction. Making that read work needs either a hand-pinned
gid or world-readable password hashes.
The user store is two files, one authoritative: users.json is canonical,
users.yml is a rendered artifact. That split is what lets the crate work
without a YAML parser -- the workspace has none, and adding one costs a
crates.io fetch, a lock update and a vendor hash for a schema we fully
control and only ever emit.
Passwords are generated by authelia rather than passed to it: argv is
world-readable, so a password on a command line is readable by any local
process for the lifetime of the call.
The three derived facts swarmctl needs about the authelia container --
machine, unit and the host-side users path -- become readOnly options on
the authelia module rather than literals repeated at the call site.
First half of the swarm-controller slice: the crate, its workspace entry
and its daemonBins entry, so the systemd unit that follows has a binary
to point at.
It serves one health endpoint and owns no state. That is the whole
intent -- this makes the unit real (service user, runtime and state
directories, socket, nginx reachability) so the swarm-level surfaces
that follow have somewhere to land. Inventing those surfaces now would
bake in a shape nobody has agreed to.
The socket gets its own runtime directory rather than sharing
hive-c0re's. nginx reaches a unix upstream by having the socket's
directory bind-mounted into the gateway container, so co-locating this
socket with the host admin socket would hand the gateway that socket
too.
The per-crate features = ["chrono"] override was a local deviation with a
comment defending it. The feature belongs in the workspace list, where every
crate sees the same utoipa.
The wire types were in hive-host-sock, which is the host *socket* crate — so
anything living there is core-shaped by construction, and the projection had
quietly grown two core dependencies to match: it selected roots by matching
NodeKind::Dag, and rendered payloads through free functions in hive-c0re that
nothing obliged a second host to write.
hive-jobq is the wrong home too. That crate is the scheduler — logic — and
folding presentation in means every consumer of it carries a JSON vocabulary
it may never serve.
So: a new hive-jobq-wire. A host implements WireNode for its payload N and
WireResource for its resource name R; GraphWire::wire_snapshot is
blanket-implemented for Graph<N, R> when both hold, and for nothing else. A
payload that has never said how it displays has no way onto the wire.
wire_snapshot takes the roots to serve rather than reading Graph::roots
itself. Nothing is ever removed from a Graph, so retention is a policy only
the host can hold; hive-c0re passes visible_roots(), which is the existing
MAX_HISTORY_DAGS bound selected structurally (a root is a node with no
parent) instead of by node kind.
Swagger UI itself is nginx-hosted now (iris's 86a39c4c), so c0re
carrying its own vendored copy via utoipa-swagger-ui was a straight
duplicate — dropped the dependency (root Cargo.toml + hive-c0re's),
swapped the SwaggerUi::new(...).url(...) mount for a plain
/api/openapi.json GET route serving the same OpenApi doc as JSON.
Verified: cargo build/clippy/test -p hive-c0re clean, Cargo.lock
dropped utoipa-swagger-ui + utoipa-swagger-ui-vendored with no other
changes, nix fmt clean.
Third time the operator asked for a guid and got a substitute: first an
i64 index, then a {random job id, per-builder counter} pair. The pair was
defensible in isolation -- a foreign handle misses rather than colliding,
with no new dependency -- but "an equivalent that avoids a dep" is a
counter-proposal, not an implementation.
It is also simpler as a guid, which was the question asked: NodeGuid(Uuid)
drops the `job` field, the `next_seq` counter, `fresh_job_id()` and the
`Cell` import, and halves the type's doc. One random draw per node rather
than one per builder -- noise next to what a node does when it runs.
uuid 1.24 was already in Cargo.lock as a transitive dependency, so this
adds an edge rather than a package.
