mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

Compare commits

base: mara:c79c41e0047dd184a42b89996ece0b5f58cbd333
Branches Tags
mara:main
...
compare: mara:5486fc03339a17c85f46e86ca6818a28b97f855f
Branches Tags
mara:main
Author SHA1 Message Date
atlas
5486fc0333 report per agent whether a hive can pull its latest closure from the cache 
for each agent the hive looks up its record in agent-closures, fetches the closure's narinfo from the cache with the pull token and checks that a signature by the cache's public key covers it. the result is a log line and the gauge hyperhive.agent.closure_pullable, with a reason label when it is 0. nothing is downloaded and deploys still build locally. part 3 of 3: the probe needs the cache facts and the closure read grant from parts 1 and 2.
 2026-10-28 01:19:56 +01:00
atlas
a55d55e494 read the cache facts on the hive and let hives read the closure bucket 
hive-c0re reads its cache facts from the store every five minutes and writes /run/hyperhive/attic/{netrc,substituter,public-key}, each 0600. it accepts only an https url, so the pull token never travels in clear. hives get a read grant on the agent-closures bucket, and swarm-queue-client a reader for it. part 2 of 3: nothing uses the files or the reader until the probe lands.
 2026-10-28 01:19:56 +01:00
atlas
1e5070796d copy the cache's pull facts to each hive and create them in the in-swarm cache 
the in-swarm cache stores its address, name and public key beside its tokens, create-only. swarm-controller copies the four cache facts to swarm/hives/<hive>/attic/* every five minutes, with read on the sources and write on the copies. part 1 of 3: nothing reads the copies until the hive reader and probe land.
 2026-10-28 01:19:56 +01:00
31 changed files with 1853 additions and 80 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Cargo.lock generated
View file

@ -1774,6 +1774,7 @@ dependencies = [
"clap",
"clap-markdown",
"clap_complete",
"ed25519-dalek",
"forgejo-api",
"futures-util",
"hive-core-agent-sock",

3
Cargo.toml
View file

@ -248,6 +248,9 @@ matrix-sdk = { version = "0.18", default-features = false, features = [
futures-util = "0.3"
hmac = "0.13"
sha2 = "0.11"
# Verifies the cache signature over a narinfo (hive-c0re). Already in the lock
# through nkeys.
ed25519-dalek = "2"
# Constant-time comparison of a presented secret against the stored one
# (`swarm-agent-verify`). Already in the tree through the TLS stack.
subtle = "2.6"

19
docs/scheduler/observability.md
View file

@ -334,15 +334,16 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
(= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source |
| --------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| Metric | Unit | Kind | Source |
| --------------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `rev-mismatch`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept

74
docs/swarm/builder.md
View file

@ -249,8 +249,9 @@ KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key.
The record's `cached` field says whether the builder pushed the closure to
[the binary cache](#binary-cache). Nothing reads the bucket yet. The closure stays
unrooted in the builder host's store.
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
[report whether it can pull them](#what-hives-do-with-the-cache); no deploy
uses a record yet. The closure stays unrooted in the builder host's store.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A
@ -311,12 +312,21 @@ principal and create what's missing.
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
the builder reads, at `secret/swarm/services/attic/push-token`, and a
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
It also stores the facts a hive pulls with: the cache's address at
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
`secret/swarm/services/attic/cache`, and the public key it signs with at
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
A unit creates each secret only while the store holds none, so a second start
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
write the new token to its path in the store: nothing mints over a value that
exists. A new signing key invalidates both tokens.
The unit creates the address, name and key once and never updates them. After
you change `swarm.attic.domain`, or after the server's database goes and the
cache gets a new key, delete the stale leaf and restart
`swarm-bao-attic-tokens`.
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
and `swarm` on every host, whether or not it runs the cache, so a builder on any
host pushes to it.
@ -345,6 +355,14 @@ the swarm at an existing tenant:
or `bao kv put` with `value=-` and the token on stdin, so the token is in no
command line. The token needs push rights on that cache. Until it
exists, every push fails and builds stay green with `cached` false.
Seed the hives' pull token the same way at
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
3. Seed the three facts hives pull with, each as field `value`:
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
slash; hives refuse `http://`), `secret/swarm/services/attic/cache` (the cache's name) and
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
cache's page or `attic cache info`). Until all four exist, hives report
`no-credentials`.
The builder checks the instance's certificate against the system roots and
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
@ -358,6 +376,58 @@ The swarm then renders no container, vhost or unit of its own cache, and
creates no secret in the store. The store's host still mints the cache's
leaf and writes its role, which nothing logs in to.
## What hives do with the cache
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
leaves (`pull-token`, `url`, `cache`, `public-key`) to
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
own prefix. It writes a copy only when the value differs, skips a leaf the
store doesn't hold, and leaves a copy whose source disappeared.
`hive-c0re` reads its four leaves every five minutes under the hive's own store
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the
token goes out in a header and in the netrc; a hive that finds another scheme
writes nothing. No deploy reads the files yet: deploys still build locally.
The same pass checks each of the hive's agents. It reads the agent's record in
`agent-closures`, then fetches the closure's narinfo from
`<url>/<cache>/<hash>.narinfo` with the pull token as the basic-auth password.
It checks that a signature by the public key covers the narinfo, in Rust, with
no help from the host's nix daemon. It downloads nothing. The result is a log
line (at info when it changes) and the gauge
`hyperhive.agent.closure_pullable{agent}` (the `hive` label comes from the
collector, as for every `hyperhive.*` metric): `1` when the hive can pull the
latest closure, else `0` with a `reason` label. The first check that fails
decides the reason:
| `reason` | Meaning |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `no-credentials` | One of the four leaves is missing or unusable (a URL that isn't `https` is unusable). The swarm has no cache, the controller is too old to copy them, or you haven't seeded them yet. |
| `no-record` | `agent-closures` holds no build for the agent. |
| `rev-mismatch` | The record's build came from another `internal/meta` commit than its head, or the hive can't read the head. |
| `not-cached` | The record says the builder didn't push the closure, or the cache answers 404 for the narinfo. |
| `fetch-failed` | The request failed, or the cache answered with a status other than 200 or 404 (a refused token is a 401 or 403). |
| `bad-signature` | The narinfo describes another path, or no signature by the public key covers it. |
A failed read of the store, the queue or the forge leaves the last result in
place and raises a warning after three failed passes. A hive reports nothing
before its first pass.
Mixed versions:
| controller | hive | what happens |
| ---------- | ---- | ----------------------------------------------------------------------------------- |
| new | new | the copies exist and the hive reports per agent |
| old | new | the controller copies nothing, so the hive reports `no-credentials` for every agent |
| new | old | the controller writes copies nobody reads |
| any | any | a swarm with no cache seeds no leaves, and every hive reports `no-credentials` |
The signature check follows Nix's fingerprint format but hasn't run against a
live attic. A `bad-signature` on every agent in the first pass after a deploy
means check attic's fingerprint format; it says nothing about an attacker. A
`fetch-failed` on every agent points at the URL, the token or the network.
## Credentials
| What | Where | Who reads it |

74
docs/swarm/credentials-reference.md
View file

@ -1,3 +1,5 @@
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Nothing uses the files yet. |
# Credentials: the per-secret reference
Every credential in the swarm's secret store, in one table: who mints it, who
@ -14,40 +16,44 @@ the rest of the cell says how.
<!-- vale write-good.Passive = NO -->
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice sync` in `hive-matrix`: seeded from the container's own tokens when the store holds none, a new pair only when the container has none either | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` write the new pair into the container and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
| store path | minter | reader | automatic re-mint | automatic re-pull |
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice sync` in `hive-matrix`: seeded from the container's own tokens when the store holds none, a new pair only when the container has none either | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` write the new pair into the container and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), at agent creation | `hive-c0re`, then the agent | ✅ re-issued every 5 minutes once past half its 7-day validity | ✅ the agent re-reads every 10 minutes; it renews only while its current leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | the agent; the queue, log and metrics auth services | ✅ re-minted at 45 days; skipped for `Destroyed` agents, whose path is deleted | ✅ read on every reconnect; an open connection keeps the old secret for up to an hour; the log and metrics stores honour a replaced secret for up to `deploy.victorialogs.authVerdictTtl` and `deploy.victoriametrics.agentAuthVerdictTtl` (60 s by default) |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or no longer matching the forge | ✅ every 10 minutes |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when you link an external forge account in the swarm UI | the agent | ❌ your token; delete it and link the label again | ✅ every 2 minutes |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when you link a GitHub account in the swarm UI | the agent | ❌ your token; delete it and link again | ✅ every 2 minutes |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, when you log the agent in from its card; one version per login | the agent, while it waits for a login | ❌ log in again | ✅ every 30 seconds while waiting |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or rejected by the homeserver | ✅ the hive re-reads on every sweep |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call |
| `swarm/hives/<hive>/oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute; ❌ Grafana restarts when its copy changes |
| `swarm/services/<grafana client id>/secret-key` | `swarm-bao-grafana-secret-key` on Grafana's host, from the key an earlier release generated; a new one only when there is none | Grafana | ❌ minted once; a different key leaves Grafana unable to read its stored datasource secrets | ❌ Grafana reads it at start; the unit restarts Grafana when the file changes |
| `swarm/services/swarm-authelia/keys/<name>`, for `jwt`, `session`, `storage-encryption`, `oidc-hmac` and `oidc-issuer` | authelia's `authelia-swarm-secrets` unit in its container, under the secret publisher's identity, from the key file an earlier release generated; a new one only when there is none | authelia | ❌ minted once; a different storage-encryption key leaves authelia's database unreadable and a different issuer key invalidates every issued token | ❌ read when the unit runs; the unit restarts authelia when the file changes |
| `swarm/services/review-classifier/anthropic-api-key` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ write a new token | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ❌ as the URL. A cache recreated with a new key leaves the stored one stale: every hive then reports `bad-signature` | ✅ copied to the hives every 5 minutes |
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->

2
hive-c0re/Cargo.toml
View file

@ -48,6 +48,7 @@ listenfd = "1"
petgraph.workspace = true
hmac.workspace = true
sha2.workspace = true
ed25519-dalek.workspace = true
rusqlite.workspace = true
serde.workspace = true
serde_json.workspace = true
@ -61,6 +62,7 @@ swarm-queue-client = { workspace = true, features = [
"agent-limits",
"agent-permissions",
"agent-archetype",
"agent-closures",
] }
swarm-matrix-client.workspace = true
swarm-secret-client = { workspace = true, features = ["otel"] }

7
hive-c0re/src/attic/mod.rs Normal file
View file

@ -0,0 +1,7 @@
//! The swarm's binary cache, as this hive sees it: the pull credentials it is
//! given ([`reader`]), the narinfo check ([`narinfo`]), and the probe that
//! reports per agent whether the latest closure could be pulled ([`probe`]).
pub mod narinfo;
pub mod probe;
pub mod reader;

303
hive-c0re/src/attic/narinfo.rs Normal file
View file

@ -0,0 +1,303 @@
//! A closure's narinfo in the binary cache: fetching it with the pull token,
//! and checking its signature against the cache's public key.
//!
//! The fingerprint is Nix's: `1;<store path>;<nar hash>;<nar size>;<references
//! as full paths, comma-separated>`, signed with ed25519. A key is
//! `<name>:<base64 of 32 bytes>`, a signature `<name>:<base64 of 64 bytes>`.
use std::time::Duration;
use anyhow::{Context as _, Result, anyhow, bail};
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD;
use ed25519_dalek::{Signature, Verifier as _, VerifyingKey};
use super::reader::Facts;
/// How long one narinfo request may take.
const FETCH_TIMEOUT: Duration = Duration::from_secs(20);
/// A client for the cache: no redirects, since the token rides in a header.
///
/// # Errors
/// The TLS roots cannot be loaded.
pub fn client() -> Result<reqwest::Client> {
Ok(reqwest::Client::builder()
.timeout(FETCH_TIMEOUT)
.redirect(reqwest::redirect::Policy::none())
.build()?)
}
/// The narinfo of `store_path` from the cache `facts` name.
///
/// `Ok(None)` is the cache saying it does not hold the path (a 404).
///
/// # Errors
/// A request that failed, or any status but 200 and 404. The message names the
/// status, never the token.
pub async fn fetch(
client: &reqwest::Client,
facts: &Facts,
store_path: &str,
) -> Result<Option<String>> {
let hash = path_hash(store_path)?;
let url = format!("{}/{hash}.narinfo", facts.substituter());
let response = client
.get(&url)
.basic_auth("hive", Some(facts.token()))
.send()
.await
.map_err(|e| anyhow!("GET {url}: {}", e.without_url()))?;
match response.status() {
reqwest::StatusCode::OK => {
Ok(Some(response.text().await.map_err(|e| {
anyhow!("reading {url}: {}", e.without_url())
})?))
}
reqwest::StatusCode::NOT_FOUND => Ok(None),
status => bail!("GET {url}: {status}"),
}
}
/// The 32-character hash a narinfo is named by: `/nix/store/<hash>-<name>`.
fn path_hash(store_path: &str) -> Result<&str> {
let base = store_path.rsplit('/').next().unwrap_or_default();
match base.split_once('-') {
Some((hash, name)) if hash.len() == 32 && !name.is_empty() => Ok(hash),
_ => bail!("{store_path:?} is not a store path"),
}
}
/// Whether `narinfo` describes `store_path` and carries a signature by
/// `public_key` over it.
///
/// # Errors
/// A narinfo that is malformed, describes another path, or has no valid
/// signature by the key.
pub fn verify(narinfo: &str, store_path: &str, public_key: &str) -> Result<()> {
let field = |name: &str| {
narinfo
.lines()
.find_map(|l| l.strip_prefix(name).and_then(|r| r.strip_prefix(": ")))
};
let path = field("StorePath").context("no StorePath")?;
if path != store_path {
bail!("the narinfo describes {path}, not {store_path}");
}
let hash = nar_hash_nix32(field("NarHash").context("no NarHash")?)?;
let size: u64 = field("NarSize")
.context("no NarSize")?
.parse()
.context("NarSize is not a number")?;
let dir = store_path.rsplit_once('/').map_or("", |(d, _)| d);
let mut refs: Vec<&str> = field("References")
.unwrap_or_default()
.split_whitespace()
.collect();
refs.sort_unstable();
let refs: Vec<String> = refs.iter().map(|r| format!("{dir}/{r}")).collect();
let fingerprint = format!("1;{store_path};{hash};{size};{}", refs.join(","));
let (key_name, key_b64) = public_key
.split_once(':')
.context("the public key has no name")?;
let key = VerifyingKey::from_bytes(
&STANDARD
.decode(key_b64)?
.try_into()
.map_err(|_| anyhow!("the public key is not 32 bytes"))?,
)?;
for sig in narinfo.lines().filter_map(|l| l.strip_prefix("Sig: ")) {
let Some((name, b64)) = sig.split_once(':') else {
continue;
};
if name != key_name {
continue;
}
let Ok(bytes) = STANDARD.decode(b64) else {
continue;
};
let Ok(signature) = Signature::from_slice(&bytes) else {
continue;
};
if key.verify(fingerprint.as_bytes(), &signature).is_ok() {
return Ok(());
}
}
bail!("no signature by {key_name} verifies over the fingerprint")
}
/// `sha256:<hash>` as Nix prints it: the hash in its base-32 form. A narinfo
/// carries it that way; hex is accepted too.
fn nar_hash_nix32(value: &str) -> Result<String> {
let digest = value
.strip_prefix("sha256:")
.context("NarHash is not sha256")?;
match digest.len() {
52 => Ok(value.to_owned()),
64 => {
let bytes = (0..32)
.map(|i| u8::from_str_radix(&digest[2 * i..2 * i + 2], 16))
.collect::<Result<Vec<u8>, _>>()
.context("NarHash is not hex")?;
Ok(format!("sha256:{}", nix32(&bytes)))
}
_ => bail!("NarHash is in a form this check does not read"),
}
}
/// Nix's base-32: its own alphabet, least significant bits first.
fn nix32(bytes: &[u8]) -> String {
const ALPHABET: &[u8; 32] = b"0123456789abcdfghijklmnpqrsvwxyz";
let len = (bytes.len() * 8 - 1) / 5 + 1;
(0..len)
.rev()
.map(|n| {
let (i, j) = (n * 5 / 8, n * 5 % 8);
let lo = u16::from(bytes[i]) >> j;
let hi = bytes.get(i + 1).map_or(0, |b| u16::from(*b) << (8 - j));
char::from(ALPHABET[usize::from((lo | hi) & 0x1f)])
})
.collect()
}
#[cfg(test)]
mod tests {
use ed25519_dalek::{Signer as _, SigningKey};
use super::*;
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
const DEP: &str = "ffffffffffffffffffffffffffffffff-glibc";
const HASH: &str = "sha256:0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73";
fn key(seed: u8) -> SigningKey {
SigningKey::from_bytes(&[seed; 32])
}
fn public(name: &str, key: &SigningKey) -> String {
format!("{name}:{}", STANDARD.encode(key.verifying_key().to_bytes()))
}
/// A narinfo signed by `key` under `name` over the fingerprint of `PATH`.
fn signed(name: &str, key: &SigningKey) -> String {
let fingerprint = format!("1;{PATH};{HASH};120;/nix/store/{DEP}");
let sig = STANDARD.encode(key.sign(fingerprint.as_bytes()).to_bytes());
format!(
"StorePath: {PATH}\nURL: nar/x.nar\nCompression: none\nNarHash: {HASH}\n\
NarSize: 120\nReferences: {DEP}\nSig: {name}:{sig}\n"
)
}
#[test]
fn nix32_matches_nixs_own_encoding() {
let empty_sha256 = [
0xe3, 0xb0, 0xc4, 0x42, 0x98, 0xfc, 0x1c, 0x14, 0x9a, 0xfb, 0xf4, 0xc8, 0x99, 0x6f,
0xb9, 0x24, 0x27, 0xae, 0x41, 0xe4, 0x64, 0x9b, 0x93, 0x4c, 0xa4, 0x95, 0x99, 0x1b,
0x78, 0x52, 0xb8, 0x55,
];
assert_eq!(
format!("sha256:{}", nix32(&empty_sha256)),
HASH,
"from `nix-hash --to-base32`"
);
let hex = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
assert_eq!(nar_hash_nix32(&format!("sha256:{hex}")).expect("hex"), HASH);
}
#[test]
fn a_signature_by_the_key_verifies() {
let k = key(7);
verify(&signed("swarm", &k), PATH, &public("swarm", &k)).expect("verifies");
}
#[test]
fn a_signature_by_another_key_or_over_other_content_does_not() {
let (ours, theirs) = (key(7), key(9));
let err = verify(&signed("swarm", &theirs), PATH, &public("swarm", &ours));
assert!(err.is_err(), "a stranger's signature under our key's name");
// The key's name must match a Sig line's, whatever the bytes.
let err = verify(&signed("other", &ours), PATH, &public("swarm", &ours));
assert!(err.is_err(), "a signature under another name");
let tampered = signed("swarm", &ours).replace("NarSize: 120", "NarSize: 121");
assert!(verify(&tampered, PATH, &public("swarm", &ours)).is_err());
let unsigned = signed("swarm", &ours).replace("Sig: ", "Sig-ignored: ");
assert!(verify(&unsigned, PATH, &public("swarm", &ours)).is_err());
}
#[test]
fn a_narinfo_for_another_path_is_refused_even_when_signed() {
let k = key(7);
let other = "/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-other";
assert!(verify(&signed("swarm", &k), other, &public("swarm", &k)).is_err());
}
#[test]
fn the_narinfo_is_named_by_the_paths_hash() {
assert_eq!(
path_hash(PATH).expect("a store path"),
"0123456789abcdfghijklmnpqrsvwxyz"
);
assert!(path_hash("/nix/store/short-name").is_err());
assert!(path_hash("nonsense").is_err());
}
/// A cache on loopback that answers a narinfo only to this basic-auth
/// password, 404 for another path, and 500 for a third.
async fn serve() -> (String, tokio::task::JoinHandle<()>) {
use axum::http::{HeaderMap, StatusCode};
let app = axum::Router::new().route(
"/swarm/{file}",
axum::routing::get(|axum::extract::Path(file): axum::extract::Path<String>, headers: HeaderMap| async move {
let expected = format!("Basic {}", STANDARD.encode("hive:pull-secret"));
let ok = headers.get("authorization").is_some_and(|v| *v == *expected);
match (ok, file.as_str()) {
(false, _) => (StatusCode::UNAUTHORIZED, String::new()),
(true, "0123456789abcdfghijklmnpqrsvwxyz.narinfo") => (StatusCode::OK, "narinfo-body".to_owned()),
(true, "ffffffffffffffffffffffffffffffff.narinfo") => (StatusCode::INTERNAL_SERVER_ERROR, String::new()),
(true, _) => (StatusCode::NOT_FOUND, String::new()),
}
}),
);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("bind");
let base = format!("http://{}", listener.local_addr().expect("addr"));
(
base,
tokio::spawn(async move { axum::serve(listener, app).await.expect("serve") }),
)
}
#[tokio::test]
async fn the_fetch_sends_the_token_and_tells_missing_from_failed() {
let (base, server) = serve().await;
// No system roots in the build sandbox; the requests are plain HTTP.
let client = reqwest::Client::builder()
.tls_certs_only([])
.build()
.expect("client");
let with = |token: &str| Facts::unchecked_for_test(token, &base, "swarm", "swarm:AAAA");
let found = fetch(&client, &with("pull-secret"), PATH).await;
let missing = fetch(
&client,
&with("pull-secret"),
"/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-x",
)
.await;
let broken = fetch(&client, &with("pull-secret"), &format!("/nix/store/{DEP}")).await;
let refused = fetch(&client, &with("wrong"), PATH).await;
server.abort();
assert_eq!(found.expect("200").as_deref(), Some("narinfo-body"));
assert!(missing.expect("404 is not an error").is_none());
assert!(broken.expect_err("500").to_string().contains("500"));
let refused = refused.expect_err("401");
assert!(refused.to_string().contains("401"));
assert!(
!refused.to_string().contains("wrong"),
"the token stays out of the message"
);
}
}

396
hive-c0re/src/attic/probe.rs Normal file
View file

@ -0,0 +1,396 @@
//! Whether this hive could pull the latest closure of each of its agents from
//! the swarm's binary cache, and the answer as a log line and a gauge.
//!
//! Every [`PASS_INTERVAL`] each agent's record in `agent-closures` is looked up
//! and its narinfo fetched with the pull token and checked against the cache's
//! public key. Nothing is downloaded and nothing is deployed from it.
//!
//! The verdict is the first of these that fails, in this order:
//! no credentials (`no-credentials`), no record (`no-record`), a record built
//! from another `internal/meta` commit than the head (`rev-mismatch`), a record
//! the builder did not push (`not-cached`), the cache not holding the narinfo
//! (`not-cached`), a request that failed or was refused (`fetch-failed`), a
//! narinfo no signature of the key covers (`bad-signature`).
use std::sync::Mutex;
use std::time::Duration;
use anyhow::{Context as _, Result};
use swarm_queue_client::agent_closures::{self, Read};
use swarm_wire::agent_closures::AgentClosure;
use super::narinfo;
use super::reader::{self, Facts};
use crate::stats::sweep_health::{self, SweepHealth};
/// How often every agent is probed.
pub const PASS_INTERVAL: Duration = Duration::from_mins(5);
/// At [`PASS_INTERVAL`] this is ~15 minutes of failing passes.
const FAILURES_BEFORE_WARNING: u32 = 3;
/// Why a closure cannot be pulled; the gauge's `reason` label.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Reason {
NoRecord,
RevMismatch,
NotCached,
NoCredentials,
FetchFailed,
BadSignature,
}
impl Reason {
/// The label value.
pub const fn label(self) -> &'static str {
match self {
Reason::NoRecord => "no-record",
Reason::RevMismatch => "rev-mismatch",
Reason::NotCached => "not-cached",
Reason::NoCredentials => "no-credentials",
Reason::FetchFailed => "fetch-failed",
Reason::BadSignature => "bad-signature",
}
}
}
/// A closure that cannot be pulled, and what the log line says about it.
#[derive(Debug)]
pub struct Unpullable {
pub reason: Reason,
pub detail: String,
}
fn no(reason: Reason, detail: impl Into<String>) -> Unpullable {
Unpullable {
reason,
detail: detail.into(),
}
}
/// The cache's narinfo for a store path, so a test can stand in for it.
pub(crate) trait Cache {
/// `None` when the cache does not hold the path.
async fn narinfo(&self, facts: &Facts, store_path: &str) -> Result<Option<String>>;
}
struct Http(reqwest::Client);
impl Cache for Http {
async fn narinfo(&self, facts: &Facts, store_path: &str) -> Result<Option<String>> {
narinfo::fetch(&self.0, facts, store_path).await
}
}
/// Whether the closure `record` names can be pulled with `facts`. `head` is the
/// `internal/meta` commit a deploy would build from.
pub(crate) async fn judge(
facts: Option<&Facts>,
record: Option<&AgentClosure>,
head: Option<&str>,
cache: &impl Cache,
) -> Result<(), Unpullable> {
let facts = facts.ok_or_else(|| {
no(
Reason::NoCredentials,
"the swarm has given this hive no cache facts",
)
})?;
let record =
record.ok_or_else(|| no(Reason::NoRecord, "no build is recorded for the agent"))?;
if head != Some(record.meta_rev.as_str()) {
return Err(no(
Reason::RevMismatch,
format!(
"built from {}, internal/meta is at {}",
record.meta_rev,
head.unwrap_or("nothing")
),
));
}
if !record.cached {
return Err(no(
Reason::NotCached,
"the builder did not push the closure",
));
}
let text = match cache.narinfo(facts, &record.store_path).await {
Ok(Some(text)) => text,
Ok(None) => return Err(no(Reason::NotCached, "the cache does not hold the narinfo")),
Err(e) => return Err(no(Reason::FetchFailed, format!("{e:#}"))),
};
narinfo::verify(&text, &record.store_path, facts.public_key())
.map_err(|e| no(Reason::BadSignature, format!("{e:#}")))
}
/// The last pass's verdict per agent: `None` when it can be pulled.
static VERDICTS: Mutex<Vec<(String, Option<Reason>)>> = Mutex::new(Vec::new());
/// The verdicts of the last pass, for the gauge.
pub fn verdicts() -> Vec<(String, Option<Reason>)> {
VERDICTS.lock().map(|g| g.clone()).unwrap_or_default()
}
/// Probe every agent now and every [`PASS_INTERVAL`] until shutdown.
pub async fn follow(client: async_nats::Client, mut shutdown: tokio::sync::watch::Receiver<bool>) {
let mut health = SweepHealth::new("attic_pull_probe", "warn", FAILURES_BEFORE_WARNING);
loop {
match pass(&client).await {
Ok(()) => health.record_ok(),
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "attic: the pull probe failed; the last verdicts stand");
let err = format!("{e:#}");
health.record_err(|ctx| {
let age = ctx.since_last_ok.map_or_else(
|| "no success this session".to_owned(),
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
);
format!(
"checking whether agents' closures can be pulled is failing ({} consecutive, {age}) \
— deploys are unaffected: {err}",
ctx.consecutive
)
});
}
}
tokio::select! {
() = tokio::time::sleep(PASS_INTERVAL) => {}
_ = shutdown.changed() => {
tracing::info!("attic: shutdown signal received");
return;
}
}
}
}
/// One pass: publish the facts, then judge every agent.
///
/// # Errors
/// A read that failed (the store, the queue, the forge), which leaves the last
/// verdicts as they are: the failure says nothing about any agent.
async fn pass(client: &async_nats::Client) -> Result<()> {
swarm_queue_client::ensure_connected(client)?;
let facts = reader::sync().await?;
let agents = crate::lifecycle::agents_for_meta_listing()
.await
.context("enumerating this hive's agents")?;
let head = match &facts {
Some(_) => meta_head().await?,
None => None,
};
let http = Http(narinfo::client()?);
let mut now = Vec::with_capacity(agents.len());
for spec in agents {
let record = match &facts {
Some(_) => read_record(client, &spec.name).await?,
None => None,
};
let verdict = judge(facts.as_ref(), record.as_ref(), head.as_deref(), &http).await;
now.push((spec.name, verdict.as_ref().err().map(|u| u.reason)));
report(&now[now.len() - 1], verdict.err().map(|u| u.detail));
}
if let Ok(mut g) = VERDICTS.lock() {
*g = now;
}
Ok(())
}
async fn read_record(client: &async_nats::Client, agent: &str) -> Result<Option<AgentClosure>> {
match agent_closures::read(client, agent)
.await
.map_err(|e| anyhow::anyhow!(e))
.with_context(|| format!("reading the closure record of {agent}"))?
{
Read::Current(record) => Ok(record),
Read::NoBucket => Ok(None),
}
}
/// The `internal/meta` head, or `None` when the swarm has not published it or
/// this hive has no forge token.
async fn meta_head() -> Result<Option<String>> {
use crate::meta_rev::MetaSource as _;
match crate::forge::internal_meta_source().await {
Some(source) => source
.meta_head()
.await
.context("reading internal/meta's head"),
None => Ok(None),
}
}
/// The log line: at info when the verdict differs from the last pass, else
/// debug.
fn report((agent, verdict): &(String, Option<Reason>), detail: Option<String>) {
let changed = verdicts()
.iter()
.find(|(a, _)| a == agent)
.is_none_or(|(_, v)| v != verdict);
let reason = verdict.map_or("", Reason::label);
let pullable = verdict.is_none();
let detail = detail.unwrap_or_default();
if changed {
tracing::info!(%agent, pullable, reason, %detail, "attic: can this hive pull the agent's latest closure");
} else {
tracing::debug!(%agent, pullable, reason, %detail, "attic: can this hive pull the agent's latest closure");
}
}
#[cfg(test)]
mod tests {
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD;
use ed25519_dalek::{Signer as _, SigningKey};
use super::*;
const PATH: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
const HASH: &str = "sha256:0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73";
const REV: &str = "a1b2c3";
fn key() -> SigningKey {
SigningKey::from_bytes(&[7; 32])
}
fn facts() -> Facts {
let public = format!(
"swarm:{}",
STANDARD.encode(key().verifying_key().to_bytes())
);
Facts::parse("tok", "https://cache.example", "swarm", &public).expect("facts")
}
fn record() -> AgentClosure {
AgentClosure {
meta_rev: REV.to_owned(),
store_path: PATH.to_owned(),
system: "x86_64-linux".to_owned(),
built_unix: 1,
cached: true,
}
}
fn narinfo_signed_by(key: &SigningKey) -> String {
let sig = key.sign(format!("1;{PATH};{HASH};120;").as_bytes());
format!(
"StorePath: {PATH}\nNarHash: {HASH}\nNarSize: 120\nReferences: \nSig: swarm:{}\n",
STANDARD.encode(sig.to_bytes())
)
}
/// What the cache answers for the narinfo.
struct Answers(Result<Option<String>, &'static str>);
impl Cache for Answers {
async fn narinfo(&self, _: &Facts, _: &str) -> Result<Option<String>> {
self.0.clone().map_err(|e| anyhow::anyhow!(e))
}
}
async fn reason(
facts: Option<&Facts>,
record: Option<&AgentClosure>,
head: Option<&str>,
cache: Answers,
) -> Option<Reason> {
judge(facts, record, head, &cache)
.await
.err()
.map(|u| u.reason)
}
fn good() -> Answers {
Answers(Ok(Some(narinfo_signed_by(&key()))))
}
#[tokio::test]
async fn a_signed_narinfo_for_the_head_build_is_pullable() {
assert_eq!(
reason(Some(&facts()), Some(&record()), Some(REV), good()).await,
None
);
}
#[tokio::test]
async fn no_facts_is_no_credentials_even_with_a_record() {
let r = reason(None, Some(&record()), Some(REV), good()).await;
assert_eq!(r, Some(Reason::NoCredentials));
}
#[tokio::test]
async fn no_record_is_no_record() {
let r = reason(Some(&facts()), None, Some(REV), good()).await;
assert_eq!(r, Some(Reason::NoRecord));
}
#[tokio::test]
async fn a_record_from_another_commit_is_a_rev_mismatch() {
let r = reason(Some(&facts()), Some(&record()), Some("other"), good()).await;
assert_eq!(r, Some(Reason::RevMismatch));
let r = reason(Some(&facts()), Some(&record()), None, good()).await;
assert_eq!(r, Some(Reason::RevMismatch), "no head to compare with");
}
#[tokio::test]
async fn an_unpushed_record_or_a_missing_narinfo_is_not_cached() {
let unpushed = AgentClosure {
cached: false,
..record()
};
let r = reason(Some(&facts()), Some(&unpushed), Some(REV), good()).await;
assert_eq!(r, Some(Reason::NotCached));
let r = reason(
Some(&facts()),
Some(&record()),
Some(REV),
Answers(Ok(None)),
)
.await;
assert_eq!(r, Some(Reason::NotCached));
}
#[tokio::test]
async fn a_failed_request_is_fetch_failed() {
let r = reason(
Some(&facts()),
Some(&record()),
Some(REV),
Answers(Err("401")),
)
.await;
assert_eq!(r, Some(Reason::FetchFailed));
}
#[tokio::test]
async fn a_narinfo_the_key_did_not_sign_is_a_bad_signature() {
let forged = Answers(Ok(Some(narinfo_signed_by(&SigningKey::from_bytes(
&[9; 32],
)))));
let r = reason(Some(&facts()), Some(&record()), Some(REV), forged).await;
assert_eq!(r, Some(Reason::BadSignature));
}
#[test]
fn the_labels_are_the_documented_ones() {
let labels = [
Reason::NoRecord,
Reason::RevMismatch,
Reason::NotCached,
Reason::NoCredentials,
Reason::FetchFailed,
Reason::BadSignature,
]
.map(Reason::label);
assert_eq!(
labels,
[
"no-record",
"rev-mismatch",
"not-cached",
"no-credentials",
"fetch-failed",
"bad-signature"
]
);
}
}

283
hive-c0re/src/attic/reader.rs Normal file
View file

@ -0,0 +1,283 @@
//! What this hive pulls from the swarm's binary cache with: four facts
//! `swarm-controller` copies to `swarm/hives/<hive>/attic/*`
//! ([`swarm_secret_client::attic`]), read here under the hive's own identity.
//!
//! [`Facts::publish`] puts them in `/run/hyperhive/attic/` as a netrc, the
//! substituter URL and the public key, where nix and the root helper read them.
use std::path::Path;
use anyhow::{Context as _, Result, bail};
use swarm_secret_client::attic::{Fact, Leaf, hive_path};
/// The pull token, the cache's address and its signing key.
///
/// No `Debug` derive: `token` reads from the cache.
pub struct Facts {
token: String,
url: String,
cache: String,
public_key: String,
}
impl Facts {
/// The facts as stored, checked: each is one token without whitespace,
/// since they land in a netrc line and a URL.
///
/// # Errors
/// A value that is empty, holds whitespace, or is not what its leaf is
/// (a URL with an `https` scheme and a host; a public key `<name>:<base64>`).
/// An `http` URL is refused because the token would travel in clear.
pub fn parse(token: &str, url: &str, cache: &str, public_key: &str) -> Result<Self> {
let [token, url, cache, public_key] = [token, url, cache, public_key].map(str::trim);
for (what, v) in [
("pull token", token),
("url", url),
("cache", cache),
("public key", public_key),
] {
if v.is_empty() || v.contains(char::is_whitespace) {
bail!("the stored {what} is empty or holds whitespace");
}
}
let parsed = url::Url::parse(url).context("the stored url does not parse")?;
if parsed.scheme() != "https" || parsed.host_str().is_none() {
bail!("the stored url is not an https address");
}
if public_key
.split_once(':')
.is_none_or(|(n, k)| n.is_empty() || k.is_empty())
{
bail!("the stored public key is not <name>:<key>");
}
Ok(Self {
token: token.to_owned(),
url: url.trim_end_matches('/').to_owned(),
cache: cache.to_owned(),
public_key: public_key.to_owned(),
})
}
/// The URL nix names the cache by: the server, then the cache.
pub fn substituter(&self) -> String {
format!("{}/{}", self.url, self.cache)
}
pub fn public_key(&self) -> &str {
&self.public_key
}
pub fn token(&self) -> &str {
&self.token
}
/// Facts for a test that reaches a server on loopback: no scheme check,
/// so the one place an `http` URL gets past [`Facts::parse`] is a test.
#[cfg(test)]
pub(crate) fn unchecked_for_test(
token: &str,
url: &str,
cache: &str,
public_key: &str,
) -> Self {
Self {
token: token.to_owned(),
url: url.to_owned(),
cache: cache.to_owned(),
public_key: public_key.to_owned(),
}
}
/// The netrc line the server's host authenticates with. The token is the
/// password; the server ignores the login.
fn netrc(&self) -> String {
let host = url::Url::parse(&self.url)
.ok()
.and_then(|u| u.host_str().map(str::to_owned))
.unwrap_or_default();
format!("machine {host} password {}\n", self.token)
}
/// Write the three files into `dir`: `netrc`, `substituter` and
/// `public-key`. Each is `0600`, and one that already holds its content is
/// left alone.
///
/// # Errors
/// A file that cannot be written. The ones before it are written.
pub fn publish(&self, dir: &Path) -> Result<()> {
for (name, content) in [
("netrc", self.netrc()),
("substituter", format!("{}\n", self.substituter())),
("public-key", format!("{}\n", self.public_key)),
] {
let path = dir.join(name);
if std::fs::read(&path).is_ok_and(|have| have == content.as_bytes()) {
continue;
}
crate::secret_file::write_secret_0600(&path, content.as_bytes())?;
}
Ok(())
}
}
/// Read the facts and put them in [`crate::paths::ATTIC_DIR`]. The facts, or
/// `None` when the swarm has none for this hive.
///
/// # Errors
/// As [`stored`]. A file that can't be written is logged, not an error.
pub async fn sync() -> Result<Option<Facts>> {
let facts = stored().await?;
if let Some(facts) = &facts
&& let Err(e) = facts.publish(Path::new(crate::paths::ATTIC_DIR))
{
tracing::warn!(error = %format!("{e:#}"), "attic: could not write the cache files");
}
Ok(facts)
}
/// This hive's facts from the store.
///
/// `Ok(None)` when the swarm has none for it: no cache configured, the
/// controller has not copied them yet (or is too old to), or one of the four is
/// missing or unusable. The last is logged; the others are the ordinary states.
///
/// # Errors
/// The store cannot be reached or refuses a read, which says nothing about
/// whether the facts exist.
pub async fn stored() -> Result<Option<Facts>> {
let Some(hive) = std::env::var("HYPERHIVE_HIVE_NAME")
.ok()
.filter(|h| !h.is_empty())
else {
return Ok(None);
};
let store = swarm_secret_client::SecretStore::from_env(&hive)
.await
.context("logging in to the swarm secret store")?;
let mut values = Vec::with_capacity(Leaf::ALL.len());
for leaf in Leaf::ALL {
let path = hive_path(&hive, leaf)?;
let fact = store
.read_optional::<Fact>(&path)
.await
.with_context(|| format!("reading {path}"))?;
let Some(fact) = fact else {
tracing::debug!(%path, "attic: no such fact in the store");
return Ok(None);
};
values.push(fact.value);
}
let [token, url, cache, key] = values.as_slice() else {
return Ok(None);
};
match Facts::parse(token, url, cache, key) {
Ok(facts) => Ok(Some(facts)),
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "attic: the stored cache facts are unusable");
Ok(None)
}
}
}
#[cfg(test)]
mod tests {
use std::os::unix::fs::PermissionsExt as _;
use super::*;
fn facts() -> Facts {
Facts::parse(
"tok-secret",
"https://cache.example.org/",
"swarm",
"swarm:AAAA",
)
.expect("valid")
}
fn mode(path: &Path) -> u32 {
std::fs::metadata(path).expect("stat").permissions().mode() & 0o777
}
#[test]
fn the_files_hold_the_facts_and_every_one_is_0600() {
let dir = tempfile::tempdir().expect("tempdir");
let out = dir.path().join("attic");
facts().publish(&out).expect("published");
let read = |n: &str| std::fs::read_to_string(out.join(n)).expect(n);
assert_eq!(
read("netrc"),
"machine cache.example.org password tok-secret\n"
);
assert_eq!(read("substituter"), "https://cache.example.org/swarm\n");
assert_eq!(read("public-key"), "swarm:AAAA\n");
for name in ["netrc", "substituter", "public-key"] {
assert_eq!(mode(&out.join(name)), 0o600, "{name}");
}
}
#[test]
fn a_rotated_token_replaces_the_netrc_and_a_current_file_is_left_alone() {
let dir = tempfile::tempdir().expect("tempdir");
facts().publish(dir.path()).expect("first");
let ino = |n: &str| {
std::os::unix::fs::MetadataExt::ino(
&std::fs::metadata(dir.path().join(n)).expect("stat"),
)
};
let (netrc, substituter) = (ino("netrc"), ino("substituter"));
let rotated = Facts::parse(
"tok-new",
"https://cache.example.org",
"swarm",
"swarm:AAAA",
)
.expect("valid");
rotated.publish(dir.path()).expect("second");
assert_ne!(ino("netrc"), netrc, "the changed file is rewritten");
assert_eq!(ino("substituter"), substituter, "the unchanged one is not");
assert_eq!(
std::fs::read_to_string(dir.path().join("netrc")).expect("read"),
"machine cache.example.org password tok-new\n"
);
}
/// The token is sent as a header and written to a netrc, so a URL that
/// isn't https is refused whatever the host, loopback included.
#[test]
fn an_http_url_is_refused_and_an_https_one_is_not() {
for url in [
"http://cache.example",
"http://127.0.0.1:8910",
"HTTP://cache.example",
] {
let err = Facts::parse("tok", url, "swarm", "swarm:AAAA")
.err()
.expect(url);
assert!(err.to_string().contains("https"), "{err:#}");
}
assert!(Facts::parse("tok", "https://cache.example", "swarm", "swarm:AAAA").is_ok());
}
#[test]
fn a_fact_that_would_break_a_netrc_or_a_url_is_refused() {
let ok = ("tok", "https://c.example", "swarm", "swarm:AAAA");
for (token, url, cache, key) in [
("to k", ok.1, ok.2, ok.3),
("tok\npassword x", ok.1, ok.2, ok.3),
("", ok.1, ok.2, ok.3),
(ok.0, "ftp://c.example", ok.2, ok.3),
(ok.0, "http://c.example", ok.2, ok.3),
(ok.0, "not a url", ok.2, ok.3),
(ok.0, ok.1, "sw arm", ok.3),
(ok.0, ok.1, ok.2, "nokeyname"),
] {
assert!(
Facts::parse(token, url, cache, key).is_err(),
"{token:?} {url:?}"
);
}
// The control: the same call accepts good values.
assert!(Facts::parse(ok.0, ok.1, ok.2, ok.3).is_ok());
}
}

1
hive-c0re/src/main.rs
View file

@ -16,6 +16,7 @@ use clap::{Parser, Subcommand};
mod actions;
mod agent_config;
mod agent_set;
mod attic;
mod container_view;
mod coordinator;
mod dashboard;

5
hive-c0re/src/paths.rs
View file

@ -231,6 +231,11 @@ pub fn gateway_agents_conf() -> PathBuf {
// `nix/host-modules/lib/hive-forge-git.nix` — must match.
pub const FORGE_HIVE_TOKEN: &str = "/run/hyperhive/forge-hive-token";
/// `attic/` — the swarm cache's pull credentials for nix: `netrc`, `substituter`
/// and `public-key`, each `0600`, copied from the swarm secret store. Tmpfs:
/// re-published on every boot.
pub const ATTIC_DIR: &str = "/run/hyperhive/attic";
/// `/run/hyperhive` — the runtime root (host admin socket + per-agent dirs).
#[must_use]
pub fn runtime_root() -> PathBuf {

20
hive-c0re/src/stats/otel_metrics.rs
View file

@ -256,9 +256,29 @@ fn register_instruments(provider: &SdkMeterProvider, snapshot: Arc<Mutex<Vec<Con
})
.build();
register_closure_pullable(&meter);
register_hive_instruments(&meter);
}
/// 1 when this hive can fetch the agent's latest closure from the swarm's
/// cache and verify its signature, else 0 with the `reason` it cannot. Nothing
/// is reported until the first probe pass has run.
fn register_closure_pullable(meter: &opentelemetry::metrics::Meter) {
meter
.u64_observable_gauge("hyperhive.agent.closure_pullable")
.with_unit("1")
.with_callback(|obs| {
for (agent, reason) in crate::attic::probe::verdicts() {
let mut attrs = vec![KeyValue::new("agent", agent)];
if let Some(reason) = reason {
attrs.push(KeyValue::new("reason", reason.label()));
}
obs.observe(u64::from(reason.is_none()), &attrs);
}
})
.build();
}
/// The gauge value for `agent`: 1 when unrestricted, 0 when not. An agent
/// missing from `stored` reads 1.
fn unrestricted_value(stored: &permissions::Stored, agent: &str) -> u64 {

7
hive-c0re/src/swarm_status.rs
View file

@ -109,6 +109,13 @@ pub fn spawn(
shutdown.clone(),
));
// Whether each agent's latest closure could be pulled from the swarm's
// cache, on the same connection.
tokio::spawn(crate::attic::probe::follow(
client.clone(),
shutdown.clone(),
));
// The permissions the swarm grants, and the import of the meta repo's.
tokio::spawn(crate::workers::swarm_permissions::follow(
client.clone(),

3
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -80,7 +80,8 @@ let
policies = [ "swarm-bao-grafana-oidc-policy" ];
};
# ./swarm-attic.nix, under the cache's leaf, which may create the signing
# key and the two tokens only once that policy is written.
# key, the two tokens and the cache's public facts only once that policy is
# written.
swarm-bao-attic-secret = {
here = deployCfg.attic.enable;
policies = [ "swarm-bao-attic-policy" ];

2
nix/host-modules/glue-bao-tls.nix
View file

@ -330,7 +330,7 @@ in
[ -s ${pkiDir}/notify-router.pem ] || ${signLeaf} ${pkiDir} notify-router \
${lib.escapeShellArg deployCfg.bao.notifyRouterCommonName} "" clientAuth
# The in-swarm cache's, which creates and reads its three secrets.
# The in-swarm cache's, which creates and reads its six secrets.
# Minted here for the matrix-ctl leaf's reason: the cache is a swarm
# service, so on a cache host without the store this is the file an
# operator copies.

29
nix/host-modules/lib/attic-init.nix
View file

@ -13,14 +13,22 @@
#
# `prelude` expects `./atomic-write-secret.nix` and `./refresh-consumer.nix`
# above it, and `bao`, `coreutils` and `systemd` on the caller's `path`.
# `ensureJwt` also wants `openssl` there; `provision` wants `atticadm`, `attic`
# and `curl`.
# `ensureJwt` also wants `openssl` there; `provision` wants `atticadm`, `attic`,
# `curl` and `sed`.
{ lib }:
{
# Store paths in the form `bao kv` takes, mount first.
jwtPath,
pullPath,
pushPath,
# The cache's public facts, created only while absent: the server's address
# (scheme included), the cache's name, and the key its narinfo are signed
# with. Hives pull with them.
urlPath,
cachePath,
publicKeyPath,
# The address `urlPath` holds, which clients reach the server at.
url,
# The cache's name on the server.
cache,
# Where the host reaches atticd, scheme included.
@ -98,9 +106,9 @@ in
refresh_consumer ${q machine} ${q unit} ${q envFile}
'';
# The cache and the two tokens, once atticd answers. The tokens are signed
# here with the key the store holds; the cache is created through the API by a
# token that lives for five minutes and is never stored.
# The cache, its two tokens and its public facts, once atticd answers. The
# tokens are signed here with the key the store holds; the cache is created
# through the API by a token that lives for five minutes and is never stored.
provision = ''
up=0
for _ in $(seq 1 30); do
@ -161,5 +169,16 @@ in
exit 1
fi
fi
# What hives pull with. The key is the one the server reports for the cache;
# `attic` prints it on stderr.
mint_url() { printf '%s' ${q url}; }
mint_cache() { printf '%s' ${q cache}; }
mint_public_key() {
attic cache info ${q "local:${cache}"} 2>&1 | sed -n 's/^ *Public Key: *//p'
}
ensure_secret ${q urlPath} mint_url >/dev/null || exit 1
ensure_secret ${q cachePath} mint_cache >/dev/null || exit 1
ensure_secret ${q publicKeyPath} mint_public_key >/dev/null || exit 1
'';
}

15
nix/host-modules/swarm-attic.nix
View file

@ -6,7 +6,8 @@
# Two host units run under the cache's own store leaf, the shape
# ./swarm-grafana.nix gives its admin password. One creates the key atticd signs
# tokens with, if the store holds none, and hands it to the container. The other
# creates the cache and the push and pull tokens, each only while absent
# creates the cache, the push and pull tokens, and the facts hives pull with
# (its address, name and public key), each only while absent
# (./lib/attic-init.nix). Neither needs a manual step.
#
# The container shares the host's network namespace, as ./swarm-victorialogs.nix
@ -73,6 +74,10 @@ let
jwtPath = "secret/swarm/services/attic/jwt-secret";
pullPath = "secret/swarm/services/attic/pull-token";
pushPath = "secret/swarm/services/attic/push-token";
urlPath = "secret/swarm/services/attic/url";
cachePath = "secret/swarm/services/attic/cache";
publicKeyPath = "secret/swarm/services/attic/public-key";
url = "https://${atticCfg.domain}";
cache = atticCfg.cache;
inherit
endpoint
@ -295,10 +300,11 @@ in
'';
};
# The cache and its two tokens, once atticd answers. The push token is the
# path the swarm builder reads; the pull token is for the hives.
# The cache, its two tokens and its public facts, once atticd answers. The
# push token is the path the swarm builder reads; the pull token and the
# facts are for the hives, which the controller copies them to.
systemd.services.swarm-bao-attic-tokens = {
description = "create the swarm cache, and its push and pull tokens in the secret store, if absent";
description = "create the swarm cache, and its tokens and public facts in the secret store, if absent";
after = [
"container@${machine}.service"
"swarm-bao-attic-secret.service"
@ -311,6 +317,7 @@ in
cfg.clientPackage
pkgs.coreutils
pkgs.curl
pkgs.gnused
];
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {

48
nix/host-modules/swarm-bao.nix
View file

@ -482,6 +482,14 @@ let
capabilities = ["create", "read", "update"]
}
${lib.concatMapStrings readStanza (
map (leaf: "${credentialMountPath}/data/${leaf}") atticFactLeaves
)}
${lib.concatMapStrings (name: ''
path "${credentialMountPath}/data/swarm/hives/+/attic/${name}" {
capabilities = ["create", "read", "update"]
}
'') atticHiveFactNames}
path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" {
capabilities = ["create", "update"]
}
@ -718,6 +726,28 @@ let
atticJwtLeaf = "swarm/services/attic/jwt-secret";
atticPullTokenLeaf = "swarm/services/attic/pull-token";
# The cache's three public facts, beside the pull token: its address, its
# name, and the key its narinfo are signed with. The in-swarm cache's host
# unit creates them; for an external cache the operator writes them.
# `swarm-controller` reads all four and copies each to
# `swarm/hives/<hive>/attic/<name>`
# (`swarm_secret_client::attic::Leaf`).
atticUrlLeaf = "swarm/services/attic/url";
atticCacheLeaf = "swarm/services/attic/cache";
atticPublicKeyLeaf = "swarm/services/attic/public-key";
atticFactLeaves = [
atticPullTokenLeaf
atticUrlLeaf
atticCacheLeaf
atticPublicKeyLeaf
];
atticHiveFactNames = [
"pull-token"
"url"
"cache"
"public-key"
];
# The CI run poller's forge token, the nix half of
# `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it
# and reads it back before deciding whether to re-mint; the poller reads it
@ -1199,7 +1229,7 @@ let
}
];
# The in-swarm cache's three secrets, create and read and nothing more: its
# The in-swarm cache's six secrets, create and read and nothing more: its
# host units create each one only while it is absent, so none needs `update`,
# and a rewrite of a signing key or a token an operator holds is refused. The
# cache container's journal forwarder reads its client secret under the same
@ -1215,11 +1245,13 @@ let
capabilities = ["create", "read"]
}
'')
[
atticJwtLeaf
atticPushTokenLeaf
atticPullTokenLeaf
]
(
[
atticJwtLeaf
atticPushTokenLeaf
]
++ atticFactLeaves
)
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.attic.otel.clientId}/oidc/client";
}
];
@ -2474,8 +2506,8 @@ in
description = ''
Subject the store's `swarm-attic` cert-auth role accepts: the identity
the in-swarm cache's host units present. Its grant is create and read on
three paths, `swarm/services/attic/jwt-secret`, `push-token` and
`pull-token`, and nothing else.
six paths under `swarm/services/attic/`, `jwt-secret`, `push-token`,
`pull-token`, `url`, `cache` and `public-key`, and nothing else.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
'';

61
nix/module-eval/bao-grants.nix
View file

@ -1722,6 +1722,38 @@ let
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
&& !(lib.hasInfix "secret/metadata/swarm/hives" s);
}
{
# `attic_copy` reads the cache's four facts and writes each into every
# hive's `attic/` prefix. Pinned as whole stanzas, per leaf: a read of
# the services prefix would reach every service's secrets, and `*` under
# `hives/` every hive's queue credential. The read side has no
# `create`/`update`, so the controller cannot replace the cache's token.
name = "the controller reads the cache's four facts and writes a copy of each into every hive";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
leaves = [
"pull-token"
"url"
"cache"
"public-key"
];
grants = grantsIn s;
in
lib.all (
l:
(grantFor grants "secret/data/swarm/services/attic/${l}").caps == [ "read" ]
&&
(grantFor grants "secret/data/swarm/hives/+/attic/${l}").caps == [
"create"
"read"
"update"
]
) leaves
&& lib.hasInfix "path \"secret/data/swarm/hives/+/attic/pull-token\" {\n capabilities = [\"create\", \"read\", \"update\"]\n}" s
&& (grantFor grants "secret/data/swarm/services/attic/jwt-secret") == null
&& (grantFor grants "secret/data/swarm/services/attic/push-token") == null;
}
{
# `forge::runner_token` writes the runner's registration token and keeps
# the last value in memory, so its stanza has no `read`. Pinned as the
@ -1988,11 +2020,11 @@ let
lib.hasInfix "/notify-router.pem ]" s && lib.hasInfix "swarm-notify-router \"\" clientAuth" s;
}
{
# The cache's host units create each of its three secrets and read them
# The cache's host units create each of its six secrets and read them
# back, and may not replace one: no `update`. Its container's journal
# forwarder reads its own client secret, read only. The notification
# router's grant is the control that `grantsIn` reads a one-path grant too.
name = "the cache's grant is create and read on its signing key and its two tokens, read on its forwarder client, and nothing else";
name = "the cache's grant is create and read on its signing key, its two tokens and its three public facts, read on its forwarder client, and nothing else";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-attic-policy.script;
@ -2005,24 +2037,33 @@ let
(grantFor grants "secret/data/swarm/services/attic/jwt-secret").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/push-token").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/pull-token").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/url").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/cache").caps == caps
&& (grantFor grants "secret/data/swarm/services/attic/public-key").caps == caps
&&
(grantFor grants "secret/data/swarm/services/swarm-attic-collector/oidc/client").caps == [ "read" ]
&& lib.length grants == 4
&& lib.length grants == 7
&& lib.length (grantsIn baoGrantHere.systemd.services.swarm-bao-notify-router-policy.script) == 1
&& lib.hasInfix "auth/cert/certs/swarm-attic" s
&& lib.hasInfix "allowed_common_names=swarm-attic" s
&& lib.hasInfix "token_policies=swarm-attic" s;
}
{
# The signing key and the pull token are the cache's alone; the push
# token is also the builder's, and is read there.
name = "no granting unit but the cache's names its signing key or its pull token";
# The signing key is the cache's alone; the push token is also the
# builder's, and is read there. The pull token is also the controller's,
# which only reads it (pinned above).
name = "no granting unit but the cache's names its signing key, and only it and the controller name its pull token";
ok =
lib.filter (
u:
lib.hasInfix "services/attic/jwt-secret" baoGrantHere.systemd.services.${u}.script
|| lib.hasInfix "services/attic/pull-token" baoGrantHere.systemd.services.${u}.script
) grantingUnitNames == [ "swarm-bao-attic-policy" ];
u: lib.hasInfix "services/attic/jwt-secret" baoGrantHere.systemd.services.${u}.script
) grantingUnitNames == [ "swarm-bao-attic-policy" ]
&&
lib.filter (
u: lib.hasInfix "services/attic/pull-token" baoGrantHere.systemd.services.${u}.script
) grantingUnitNames == [
"swarm-bao-controller-policy"
"swarm-bao-attic-policy"
];
}
{
name = "the PKI unit signs the cache's leaf under its own subject";

Some files were not shown because too many files have changed in this diff Show more