vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

default swarm.attic domain and cache to the swarm's own cache on every host

Browse source
This commit is contained in:
atlas 2026-10-27 19:00:59 +01:00 • committed by mara
commit 606817a994
10 changed files with 80 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

33
docs/swarm/builder.md
View file

@ -258,16 +258,18 @@ grant the forge refuses fails the build with no verdict.
## Binary cache ## Binary cache
With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes `services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own
each green meta build's closure to that cache, an attic server inside or outside the cache (`attic.<swarm domain>` and `swarm`) on every host, so the builder pushes
swarm, before it replies. It runs `attic push swarm:<cache> <out path>` once per build, each green meta build's closure to it, or to the cache you override them with,
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
not `attic watch`, and the builder pushes again when the controller runs a build again. The not `attic watch`, and the builder pushes again when the controller runs a build again. The
push has ten minutes, or less when the reply is due sooner, because it must end push has ten minutes, or less when the reply is due sooner, because it must end
inside the 61 minutes the controller waits. `attic` checks the server's certificate inside the 61 minutes the controller waits. `attic` checks the server's certificate
against the system roots and pins nothing, so a Let's Encrypt certificate against the system roots and pins nothing, so a Let's Encrypt certificate
works. works.
Set both options or neither: evaluation refuses a domain without a cache name. Set both options to a name or both to null: evaluation refuses one without the
other. Null in both is how a swarm opts out of pushing.
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`, The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
starting with a letter or digit, and evaluation refuses any other name. starting with a letter or digit, and evaluation refuses any other name.
@ -281,8 +283,8 @@ The token is in no argument or variable, so no process listing shows it.
The reply's `cached` says whether the push finished. A push that fails, times out The reply's `cached` says whether the push finished. A push that fails, times out
or can't read the token leaves the build green with `cached` false and logs a or can't read the token leaves the build green with `cached` false and logs a
warning with the reason; the deploy goes ahead and the next build of the agent warning with the reason; the deploy goes ahead and the next build of the agent
tries again. With no cache set, the builder pushes nothing and `cached` is always tries again. With no attic behind the defaults, every push fails this way. With
false. both options null, the builder pushes nothing and `cached` is always false.
Mixed versions: Mixed versions:
@ -315,11 +317,9 @@ mints nothing. Both tokens last ten years and nothing renews them. To replace on
write the new token to its path in the store: nothing mints over a value that write the new token to its path in the store: nothing mints over a value that
exists. A new signing key invalidates both tokens. exists. A new signing key invalidates both tokens.
With the switch on, `swarm.attic.domain` and `swarm.attic.cache` default to `swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
`attic.<swarm domain>` and `swarm`, so the builder on that host pushes to it. and `swarm` on every host, whether or not it runs the cache, so a builder on any
Naming either option yourself wins over its default. The defaults apply only on host pushes to it.
the host that runs the cache: on any other host, name both options too, with the
same values.
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
path nobody added or fetched for 14 days goes at the next garbage collection, path nobody added or fetched for 14 days goes at the next garbage collection,
@ -336,9 +336,10 @@ to its journal only; its journal doesn't reach the swarm log store.
The cache can be an attic tenant on an instance outside the swarm. To point The cache can be an attic tenant on an instance outside the swarm. To point
the swarm at an existing tenant: the swarm at an existing tenant:
1. On the builder host, set `services.hyperhive.swarm.attic.domain` to the 1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the
instance's host name (no scheme, `host:port` if it isn't on 443) and instance's host name (no scheme, `host:port` if it isn't on 443) and
`services.hyperhive.swarm.attic.cache` to the cache's name on it. `services.hyperhive.swarm.attic.cache` to the cache's name on it. Set
both options on every host that reads them.
2. Seed the push token in the swarm secret store at 2. Seed the push token in the swarm secret store at
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI, `secret/swarm/services/attic/push-token`, field `value`. Use the store's UI,
or `bao kv put` with `value=-` and the token on stdin, so the token is in no or `bao kv put` with `value=-` and the token on stdin, so the token is in no
@ -350,9 +351,9 @@ pins nothing, so a Let's Encrypt certificate needs no setup. The builder
reads the token again on every push, so a new token takes effect on the next reads the token again on every push, so a new token takes effect on the next
build. build.
Where you host the shared services, `deploy.attic.enable` defaults on and runs To use an external cache, override the two defaults and don't set
the swarm's own cache. To use an external one, set it to `false` with `deploy.attic.enable` anywhere. Where you host the shared services with
`lib.mkForce`, then set the two options by hand and seed the token as above. `allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there.
The swarm then renders no container, vhost or unit of its own cache, and The swarm then renders no container, vhost or unit of its own cache, and
creates no secret in the store. The store's host still mints the cache's creates no secret in the store. The store's host still mints the cache's
leaf and writes its role, which nothing logs in to. leaf and writes its role, which nothing logs in to.

2
docs/swarm/credentials-reference.md
View file

@ -43,7 +43,7 @@ the rest of the cell says how.
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run | | `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build | | `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it | | `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated | | `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated |
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes | | `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |

7
docs/swarm/services.md
View file

@ -437,12 +437,13 @@ for the CI runner. What it builds and why are in [`builder.md`](builder.md).
The swarm's attic server, in a `swarm-attic` container on the host that sets The swarm's attic server, in a `swarm-attic` container on the host that sets
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has `deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has
the cache's store identity, as for the builder: the host that runs the store has the cache's store identity, as for the builder: the host that runs the store has
it. Set `deploy.attic.enable = lib.mkForce false` to use an external cache it. For an external cache, override `swarm.attic.{domain,cache}` and don't set
instead and name `swarm.attic.{domain,cache}` yourself. `deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
`allSwarmServices`.
The container shares the host's network namespace and atticd listens on loopback The container shares the host's network namespace and atticd listens on loopback
only. The gateway serves it at `swarm.attic.domain` (default only. The gateway serves it at `swarm.attic.domain` (default
`attic.<swarm domain>`) over TLS, with no login in front, because attic checks `attic.<swarm domain>`, on every host) over TLS, with no login in front, because attic checks
its own tokens. Storage is local disk in the container's state. its own tokens. Storage is local disk in the container's state.
It needs no manual secret. Two host units, under the cache's own `attic` leaf, It needs no manual secret. Two host units, under the cache's own `attic` leaf,

7
nix/host-modules/deploy.nix
View file

@ -521,9 +521,10 @@ in
{option}`services.hyperhive.deploy.bao.atticClientCertFile` and {option}`services.hyperhive.deploy.bao.atticClientCertFile` and
`atticClientKeyFile`, minted on the store's host). `atticClientKeyFile`, minted on the store's host).
With it on, {option}`services.hyperhive.swarm.attic.domain` and `cache` {option}`services.hyperhive.swarm.attic.domain` and `cache` default to
default to the cache's own name and cache; naming either yourself wins. this cache on every host. For an external cache, override both and
With it off, those two options name an external cache or stay unset. leave this unset, or set it to `lib.mkForce false` under
`allSwarmServices`.
''; '';
}; };

28
nix/host-modules/swarm-attic-options.nix
View file

@ -1,7 +1,7 @@
# The binary cache the swarm pushes agent closures to, as every host sees it: # The binary cache the swarm pushes agent closures to, as every host sees it:
# which attic server, and which cache on it. A host that sets # which attic server, and which cache on it. Both default to the swarm's own,
# `deploy.attic.enable` runs the swarm's own and fills both in (./swarm-attic.nix); # which the host that sets `deploy.attic.enable` runs (./swarm-attic.nix); an
# otherwise the instance is the operator's, outside the swarm. # operator with an instance outside the swarm overrides both.
{ lib, config, ... }: { lib, config, ... }:
let let
swarmDomain = config.services.hyperhive.swarm.domain; swarmDomain = config.services.hyperhive.swarm.domain;
@ -35,8 +35,7 @@ in
defaultText = lib.literalExpression ''"attic.''${services.hyperhive.swarm.domain}"''; defaultText = lib.literalExpression ''"attic.''${services.hyperhive.swarm.domain}"'';
description = '' description = ''
Read-only: the name the swarm's own cache is served under, which Read-only: the name the swarm's own cache is served under, which
{option}`services.hyperhive.deploy.attic.enable` defaults {option}`services.hyperhive.swarm.attic.domain` defaults to. A sibling of the
{option}`services.hyperhive.swarm.attic.domain` to. A sibling of the
swarm's other service names, so the swarm-services sub-CA can issue for it, swarm's other service names, so the swarm-services sub-CA can issue for it,
and in `swarm.serviceDomains` on every host whether or not one runs it. and in `swarm.serviceDomains` on every host whether or not one runs it.
''; '';
@ -44,7 +43,8 @@ in
domain = lib.mkOption { domain = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = null; default = config.services.hyperhive.swarm.attic.ownDomain;
defaultText = lib.literalExpression "services.hyperhive.swarm.attic.ownDomain";
example = "cache.example.org"; example = "cache.example.org";
description = '' description = ''
Name of the attic server, with no scheme: the swarm builder pushes to Name of the attic server, with no scheme: the swarm builder pushes to
@ -53,13 +53,15 @@ in
{option}`services.hyperhive.swarm.attic.cache`; evaluation fails when only {option}`services.hyperhive.swarm.attic.cache`; evaluation fails when only
one is set. one is set.
Defaults to the swarm's own cache, the same on every host. For a cache
outside the swarm, override both options and leave
{option}`services.hyperhive.deploy.attic.enable` unset. With no cache
behind the default, a push fails and the closure record says it is not
cached.
Null pushes nothing: the swarm builder does not push, and the closure Null pushes nothing: the swarm builder does not push, and the closure
record says it is not cached. record says it is not cached.
{option}`services.hyperhive.deploy.attic.enable` defaults this to
`attic.<swarm.domain>` on the host that runs the swarm's own cache. Every
other host reads it from the same option, so name it there too.
The push token is not an option. It is at The push token is not an option. It is at
`secret/swarm/services/attic/push-token` in the swarm secret store `secret/swarm/services/attic/push-token` in the swarm secret store
(field `value`), and the builder reads it on every push. The swarm's own (field `value`), and the builder reads it on every push. The swarm's own
@ -70,12 +72,12 @@ in
cache = lib.mkOption { cache = lib.mkOption {
# attic's own rule for a cache name; the builder checks the same at start. # attic's own rule for a cache name; the builder checks the same at start.
type = lib.types.nullOr (lib.types.strMatching "^[A-Za-z0-9][A-Za-z0-9_+-]{0,49}$"); type = lib.types.nullOr (lib.types.strMatching "^[A-Za-z0-9][A-Za-z0-9_+-]{0,49}$");
default = null; default = "swarm";
example = "grid"; example = "grid";
description = '' description = ''
Name of the cache on {option}`services.hyperhive.swarm.attic.domain` Name of the cache on {option}`services.hyperhive.swarm.attic.domain`
that the swarm builder pushes closures to, `swarm` by default on the that the swarm builder pushes closures to, the swarm's own cache by
host that runs the swarm's own cache. The push token must be allowed to default. The push token must be allowed to
push to it. The name follows attic's rule: 1 to 50 letters, push to it. The name follows attic's rule: 1 to 50 letters,
digits, `-`, `_` or `+`, starting with a letter or digit. digits, `-`, `_` or `+`, starting with a letter or digit.
''; '';

10
nix/host-modules/swarm-attic.nix
View file

@ -14,9 +14,8 @@
# and attic's own tokens are the authentication. Storage is local disk under the # and attic's own tokens are the authentication. Storage is local disk under the
# container's state. # container's state.
# #
# The two `swarm.attic` options every host reads (./swarm-attic-options.nix) get # The two `swarm.attic` options every host reads (./swarm-attic-options.nix)
# their values from here, at `mkDefault`: naming them yourself wins, and with # default to this cache's names on every host, whether or not it runs here.
# the switch off nothing is set.
{ {
pkgs, pkgs,
lib, lib,
@ -168,11 +167,6 @@ in
}; };
config = lib.mkIf cfg.enable { config = lib.mkIf cfg.enable {
services.hyperhive.swarm.attic = {
domain = lib.mkDefault atticCfg.ownDomain;
cache = lib.mkDefault "swarm";
};
assertions = [ assertions = [
{ {
assertion = haveClientIdentity; assertion = haveClientIdentity;

2
nix/host-modules/swarm-required-services.nix
View file

@ -132,7 +132,7 @@ in
# The binary cache, on the same terms: its units log in to the store as the # The binary cache, on the same terms: its units log in to the store as the
# cache's own principal. An operator running an external cache sets this to # cache's own principal. An operator running an external cache sets this to
# `false` and names `swarm.attic.{domain,cache}` itself. # `lib.mkForce false` and overrides `swarm.attic.{domain,cache}`.
config.services.hyperhive.deploy.attic.enable = lib.mkDefault ( config.services.hyperhive.deploy.attic.enable = lib.mkDefault (
deployCfg.allSwarmServices deployCfg.allSwarmServices
&& deployCfg.bao.atticClientCertFile != null && deployCfg.bao.atticClientCertFile != null

40
nix/module-eval/builder.nix
View file

@ -45,6 +45,7 @@ let
deploy.bao.enable = true; deploy.bao.enable = true;
deploy.builder.enable = true; deploy.builder.enable = true;
swarm.attic.domain = "cache.example.org"; swarm.attic.domain = "cache.example.org";
swarm.attic.cache = null;
}; };
refusedHalfCache = refusedHalfCache =
m: m:
@ -67,13 +68,23 @@ let
svc = inner.systemd.services.swarm-builder; svc = inner.systemd.services.swarm-builder;
env = svc.environment; env = svc.environment;
hasOnPath = name: s: lib.any (p: lib.getName p == name) s.path; hasOnPath = name: s: lib.any (p: lib.getName p == name) s.path;
# A builder with no cache named anywhere. # A builder that opts out of pushing: both options null.
withBuilderNoCache = hive { withBuilderNoCache = hive {
deploy.bao.enable = true; deploy.bao.enable = true;
deploy.builder.enable = true; deploy.builder.enable = true;
swarm.attic.domain = null;
swarm.attic.cache = null;
}; };
withoutCache = withBuilderNoCache.containers.swarm-builder.config.systemd.services.swarm-builder; withoutCache = withBuilderNoCache.containers.swarm-builder.config.systemd.services.swarm-builder;
# A builder on a host that doesn't run the cache and names nothing.
builderDefaults = hive {
deploy.bao.enable = true;
deploy.builder.enable = true;
};
defaultsEnv =
builderDefaults.containers.swarm-builder.config.systemd.services.swarm-builder.environment;
# Hosting the shared services runs the swarm's own cache, which names itself. # Hosting the shared services runs the swarm's own cache, which names itself.
atticHere = servicesHere; atticHere = servicesHere;
atticDeploy = atticHere.services.hyperhive.deploy.attic; atticDeploy = atticHere.services.hyperhive.deploy.attic;
@ -83,7 +94,7 @@ let
atticTokens = atticHere.systemd.services.swarm-bao-attic-tokens; atticTokens = atticHere.systemd.services.swarm-bao-attic-tokens;
atticVhost = atticHere.services.hyperhive.gateway.vhosts."attic.t.local"; atticVhost = atticHere.services.hyperhive.gateway.vhosts."attic.t.local";
# The cache switched on while the operator names the options by hand. # The cache switched on while the operator overrides the options.
atticByHand = hive { atticByHand = hive {
deploy.bao.enable = true; deploy.bao.enable = true;
deploy.attic.enable = true; deploy.attic.enable = true;
@ -310,10 +321,10 @@ let
} }
{ {
# The switch is on wherever the shared services are and the host holds the # The switch is on wherever the shared services are and the host holds the
# cache's leaf, and the two options every host reads come from the cache. # cache's leaf.
# The builder on the same host pushes to it. `bare` is the control that the # The builder on the same host pushes to it. `bare` is the control that no
# lookup reads null where nothing runs a cache. # cache renders where nothing enables it.
name = "hosting the shared services runs the swarm's cache and fills the cache options, which the builder follows"; name = "hosting the shared services runs the swarm's cache, and the builder pushes to it";
ok = ok =
atticDeploy.enable atticDeploy.enable
&& atticHere.containers ? swarm-attic && atticHere.containers ? swarm-attic
@ -327,10 +338,23 @@ let
&& &&
atticHere.containers.swarm-builder.config.systemd.services.swarm-builder.environment.SWARM_BUILDER_ATTIC_CACHE atticHere.containers.swarm-builder.config.systemd.services.swarm-builder.environment.SWARM_BUILDER_ATTIC_CACHE
== "swarm" == "swarm"
&& (atticSwarm bare).domain == null
&& (atticSwarm bare).cache == null
&& !(hasAtticHost bare); && !(hasAtticHost bare);
} }
{
# The options default to the swarm's own cache on every host, so a
# builder host that runs no cache pushes to it. `atticHere` is the control
# that the cache host reads the same values.
name = "a builder host that doesn't run the cache pushes to the swarm's own by default";
ok =
!(hasAtticHost builderDefaults)
&& (atticSwarm builderDefaults).domain == "attic.t.local"
&& (atticSwarm builderDefaults).cache == "swarm"
&& (atticSwarm bare).domain == (atticSwarm atticHere).domain
&& (atticSwarm bare).cache == (atticSwarm atticHere).cache
&& defaultsEnv.SWARM_BUILDER_ATTIC_DOMAIN == "attic.t.local"
&& defaultsEnv.SWARM_BUILDER_ATTIC_CACHE == "swarm"
&& !(builderDefaults.systemd.services ? swarm-bao-attic-tokens);
}
{ {
# Names the operator gives win over the cache's own, and the gateway then # Names the operator gives win over the cache's own, and the gateway then
# serves the cache under the operator's name. `atticHere` is the control # serves the cache under the operator's name. `atticHere` is the control

4
nix/module-eval/swarm-services-switch.nix
View file

@ -304,12 +304,12 @@ let
} }
{ {
# The cache's units assert the store identity the store's host mints, as # The cache's units assert the store identity the store's host mints, as
# the router's do; its options stay as unset as they were. # the router's do; its options keep the same defaults.
name = "the swarm's cache stays off where the shared services are hosted without the store"; name = "the swarm's cache stays off where the shared services are hosted without the store";
ok = ok =
!swarmServicesBaoElsewhere.services.hyperhive.deploy.attic.enable !swarmServicesBaoElsewhere.services.hyperhive.deploy.attic.enable
&& !(swarmServicesBaoElsewhere.containers ? swarm-attic) && !(swarmServicesBaoElsewhere.containers ? swarm-attic)
&& swarmServicesBaoElsewhere.services.hyperhive.swarm.attic.domain == null && swarmServicesBaoElsewhere.services.hyperhive.swarm.attic.domain == "attic.t.local"
&& swarmServicesHere.services.hyperhive.deploy.attic.enable && swarmServicesHere.services.hyperhive.deploy.attic.enable
&& swarmServicesBaoElsewhere.services.hyperhive.deploy.nats.enable; && swarmServicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
} }

4
swarm-builder/README.md
View file

@ -31,8 +31,8 @@ the platform it builds for. The controller records the answer; the service
writes nothing. It takes the same one-build slot as the PR checks. See writes nothing. It takes the same one-build slot as the PR checks. See
[`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate). [`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate).
Where `services.hyperhive.swarm.attic.{domain,cache}` are set, it first pushes a Where `services.hyperhive.swarm.attic.{domain,cache}` are not null (they default
green build's closure to that cache with `attic push`, and the answer says to the swarm's own cache), it first pushes a green build's closure to that cache with `attic push`, and the answer says
whether that worked. A failed push leaves the build green. See whether that worked. A failed push leaves the build green. See
[`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache). [`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache).