report per agent whether a hive can pull its latest closure from the cache
for each agent the hive looks up its record in agent-closures, fetches the closure's narinfo from the cache with the pull token and checks that a signature by the cache's public key covers it. the result is a log line and the gauge hyperhive.agent.closure_pullable, with a reason label when it is 0. nothing is downloaded and deploys still build locally. part 3 of 3: the probe needs the cache facts and the closure read grant from parts 1 and 2.
This commit is contained in:
atlas
mara
parent
a55d55e494
commit
5486fc0333
2 changed files with 812 additions and 34 deletions
|
|
@ -334,15 +334,16 @@ exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
|||
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
|
||||
(= the `h-<agent>` machine) and the hive `agent` label:
|
||||
|
||||
| Metric | Unit | Kind | Source |
|
||||
| --------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
|
||||
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
|
||||
| Metric | Unit | Kind | Source |
|
||||
| --------------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + writable rootfs, measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
|
||||
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
|
||||
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `rev-mismatch`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
|
|
|
|||
|
|
@ -249,8 +249,9 @@ KV bucket: the `internal/meta` commit, the closure's store path, the builder's
|
|||
platform and the time. Only the controller writes the bucket, and the builder
|
||||
holds no grant on it. A failed build writes nothing and leaves the previous key.
|
||||
The record's `cached` field says whether the builder pushed the closure to
|
||||
[the binary cache](#binary-cache). Nothing reads the bucket yet. The closure stays
|
||||
unrooted in the builder host's store.
|
||||
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
|
||||
[report whether it can pull them](#what-hives-do-with-the-cache); no deploy
|
||||
uses a record yet. The closure stays unrooted in the builder host's store.
|
||||
|
||||
The controller gives the `builder` forge account read on `internal/meta` when it
|
||||
updates the flake, and again before each build, since the repo is private. A
|
||||
|
|
@ -360,8 +361,8 @@ the swarm at an existing tenant:
|
|||
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
|
||||
slash; hives refuse `http://`), `secret/swarm/services/attic/cache` (the cache's name) and
|
||||
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
|
||||
cache's page or `attic cache info`). The controller copies the leaves that
|
||||
exist.
|
||||
cache's page or `attic cache info`). Until all four exist, hives report
|
||||
`no-credentials`.
|
||||
|
||||
The builder checks the instance's certificate against the system roots and
|
||||
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
|
||||
|
|
@ -389,6 +390,44 @@ identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
|
|||
token goes out in a header and in the netrc; a hive that finds another scheme
|
||||
writes nothing. No deploy reads the files yet: deploys still build locally.
|
||||
|
||||
The same pass checks each of the hive's agents. It reads the agent's record in
|
||||
`agent-closures`, then fetches the closure's narinfo from
|
||||
`<url>/<cache>/<hash>.narinfo` with the pull token as the basic-auth password.
|
||||
It checks that a signature by the public key covers the narinfo, in Rust, with
|
||||
no help from the host's nix daemon. It downloads nothing. The result is a log
|
||||
line (at info when it changes) and the gauge
|
||||
`hyperhive.agent.closure_pullable{agent}` (the `hive` label comes from the
|
||||
collector, as for every `hyperhive.*` metric): `1` when the hive can pull the
|
||||
latest closure, else `0` with a `reason` label. The first check that fails
|
||||
decides the reason:
|
||||
|
||||
| `reason` | Meaning |
|
||||
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `no-credentials` | One of the four leaves is missing or unusable (a URL that isn't `https` is unusable). The swarm has no cache, the controller is too old to copy them, or you haven't seeded them yet. |
|
||||
| `no-record` | `agent-closures` holds no build for the agent. |
|
||||
| `rev-mismatch` | The record's build came from another `internal/meta` commit than its head, or the hive can't read the head. |
|
||||
| `not-cached` | The record says the builder didn't push the closure, or the cache answers 404 for the narinfo. |
|
||||
| `fetch-failed` | The request failed, or the cache answered with a status other than 200 or 404 (a refused token is a 401 or 403). |
|
||||
| `bad-signature` | The narinfo describes another path, or no signature by the public key covers it. |
|
||||
|
||||
A failed read of the store, the queue or the forge leaves the last result in
|
||||
place and raises a warning after three failed passes. A hive reports nothing
|
||||
before its first pass.
|
||||
|
||||
Mixed versions:
|
||||
|
||||
| controller | hive | what happens |
|
||||
| ---------- | ---- | ----------------------------------------------------------------------------------- |
|
||||
| new | new | the copies exist and the hive reports per agent |
|
||||
| old | new | the controller copies nothing, so the hive reports `no-credentials` for every agent |
|
||||
| new | old | the controller writes copies nobody reads |
|
||||
| any | any | a swarm with no cache seeds no leaves, and every hive reports `no-credentials` |
|
||||
|
||||
The signature check follows Nix's fingerprint format but hasn't run against a
|
||||
live attic. A `bad-signature` on every agent in the first pass after a deploy
|
||||
means check attic's fingerprint format; it says nothing about an attacker. A
|
||||
`fetch-failed` on every agent points at the URL, the token or the network.
|
||||
|
||||
## Credentials
|
||||
|
||||
| What | Where | Who reads it |
|
||||
|
|
|
|||
Loading…
Reference in a new issue