From 606817a9949facba01a471d6af1004a66a30ef64 Mon Sep 17 00:00:00 2001 From: atlas Date: Tue, 27 Oct 2026 19:00:59 +0100 Subject: [PATCH] default swarm.attic domain and cache to the swarm's own cache on every host --- docs/swarm/builder.md | 33 ++++++++-------- docs/swarm/credentials-reference.md | 2 +- docs/swarm/services.md | 7 ++-- nix/host-modules/deploy.nix | 7 ++-- nix/host-modules/swarm-attic-options.nix | 28 +++++++------- nix/host-modules/swarm-attic.nix | 10 +---- nix/host-modules/swarm-required-services.nix | 2 +- nix/module-eval/builder.nix | 40 ++++++++++++++++---- nix/module-eval/swarm-services-switch.nix | 4 +- swarm-builder/README.md | 4 +- 10 files changed, 80 insertions(+), 57 deletions(-) diff --git a/docs/swarm/builder.md b/docs/swarm/builder.md index 3f6941dc..e90780d2 100644 --- a/docs/swarm/builder.md +++ b/docs/swarm/builder.md @@ -258,16 +258,18 @@ grant the forge refuses fails the build with no verdict. ## Binary cache -With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes -each green meta build's closure to that cache, an attic server inside or outside the -swarm, before it replies. It runs `attic push swarm: ` once per build, +`services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own +cache (`attic.` and `swarm`) on every host, so the builder pushes +each green meta build's closure to it, or to the cache you override them with, +before it replies. It runs `attic push swarm: ` once per build, not `attic watch`, and the builder pushes again when the controller runs a build again. The push has ten minutes, or less when the reply is due sooner, because it must end inside the 61 minutes the controller waits. `attic` checks the server's certificate against the system roots and pins nothing, so a Let's Encrypt certificate works. -Set both options or neither: evaluation refuses a domain without a cache name. +Set both options to a name or both to null: evaluation refuses one without the +other. Null in both is how a swarm opts out of pushing. The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`, starting with a letter or digit, and evaluation refuses any other name. @@ -281,8 +283,8 @@ The token is in no argument or variable, so no process listing shows it. The reply's `cached` says whether the push finished. A push that fails, times out or can't read the token leaves the build green with `cached` false and logs a warning with the reason; the deploy goes ahead and the next build of the agent -tries again. With no cache set, the builder pushes nothing and `cached` is always -false. +tries again. With no attic behind the defaults, every push fails this way. With +both options null, the builder pushes nothing and `cached` is always false. Mixed versions: @@ -315,11 +317,9 @@ mints nothing. Both tokens last ten years and nothing renews them. To replace on write the new token to its path in the store: nothing mints over a value that exists. A new signing key invalidates both tokens. -With the switch on, `swarm.attic.domain` and `swarm.attic.cache` default to -`attic.` and `swarm`, so the builder on that host pushes to it. -Naming either option yourself wins over its default. The defaults apply only on -the host that runs the cache: on any other host, name both options too, with the -same values. +`swarm.attic.domain` and `swarm.attic.cache` default to `attic.` +and `swarm` on every host, whether or not it runs the cache, so a builder on any +host pushes to it. Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store path nobody added or fetched for 14 days goes at the next garbage collection, @@ -336,9 +336,10 @@ to its journal only; its journal doesn't reach the swarm log store. The cache can be an attic tenant on an instance outside the swarm. To point the swarm at an existing tenant: -1. On the builder host, set `services.hyperhive.swarm.attic.domain` to the +1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the instance's host name (no scheme, `host:port` if it isn't on 443) and - `services.hyperhive.swarm.attic.cache` to the cache's name on it. + `services.hyperhive.swarm.attic.cache` to the cache's name on it. Set + both options on every host that reads them. 2. Seed the push token in the swarm secret store at `secret/swarm/services/attic/push-token`, field `value`. Use the store's UI, or `bao kv put` with `value=-` and the token on stdin, so the token is in no @@ -350,9 +351,9 @@ pins nothing, so a Let's Encrypt certificate needs no setup. The builder reads the token again on every push, so a new token takes effect on the next build. -Where you host the shared services, `deploy.attic.enable` defaults on and runs -the swarm's own cache. To use an external one, set it to `false` with -`lib.mkForce`, then set the two options by hand and seed the token as above. +To use an external cache, override the two defaults and don't set +`deploy.attic.enable` anywhere. Where you host the shared services with +`allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there. The swarm then renders no container, vhost or unit of its own cache, and creates no secret in the store. The store's host still mints the cache's leaf and writes its role, which nothing logs in to. diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index f0f75003..6199f6b8 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -43,7 +43,7 @@ the rest of the cell says how. | `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review | | `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build | -| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it | +| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it | | `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated | | `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | diff --git a/docs/swarm/services.md b/docs/swarm/services.md index 2247dbbe..7f248268 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -437,12 +437,13 @@ for the CI runner. What it builds and why are in [`builder.md`](builder.md). The swarm's attic server, in a `swarm-attic` container on the host that sets `deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has the cache's store identity, as for the builder: the host that runs the store has -it. Set `deploy.attic.enable = lib.mkForce false` to use an external cache -instead and name `swarm.attic.{domain,cache}` yourself. +it. For an external cache, override `swarm.attic.{domain,cache}` and don't set +`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under +`allSwarmServices`. The container shares the host's network namespace and atticd listens on loopback only. The gateway serves it at `swarm.attic.domain` (default -`attic.`) over TLS, with no login in front, because attic checks +`attic.`, on every host) over TLS, with no login in front, because attic checks its own tokens. Storage is local disk in the container's state. It needs no manual secret. Two host units, under the cache's own `attic` leaf, diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index 4640b9d6..d0c1bfce 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -521,9 +521,10 @@ in {option}`services.hyperhive.deploy.bao.atticClientCertFile` and `atticClientKeyFile`, minted on the store's host). - With it on, {option}`services.hyperhive.swarm.attic.domain` and `cache` - default to the cache's own name and cache; naming either yourself wins. - With it off, those two options name an external cache or stay unset. + {option}`services.hyperhive.swarm.attic.domain` and `cache` default to + this cache on every host. For an external cache, override both and + leave this unset, or set it to `lib.mkForce false` under + `allSwarmServices`. ''; }; diff --git a/nix/host-modules/swarm-attic-options.nix b/nix/host-modules/swarm-attic-options.nix index 5adbf9e0..37848437 100644 --- a/nix/host-modules/swarm-attic-options.nix +++ b/nix/host-modules/swarm-attic-options.nix @@ -1,7 +1,7 @@ # The binary cache the swarm pushes agent closures to, as every host sees it: -# which attic server, and which cache on it. A host that sets -# `deploy.attic.enable` runs the swarm's own and fills both in (./swarm-attic.nix); -# otherwise the instance is the operator's, outside the swarm. +# which attic server, and which cache on it. Both default to the swarm's own, +# which the host that sets `deploy.attic.enable` runs (./swarm-attic.nix); an +# operator with an instance outside the swarm overrides both. { lib, config, ... }: let swarmDomain = config.services.hyperhive.swarm.domain; @@ -35,8 +35,7 @@ in defaultText = lib.literalExpression ''"attic.''${services.hyperhive.swarm.domain}"''; description = '' Read-only: the name the swarm's own cache is served under, which - {option}`services.hyperhive.deploy.attic.enable` defaults - {option}`services.hyperhive.swarm.attic.domain` to. A sibling of the + {option}`services.hyperhive.swarm.attic.domain` defaults to. A sibling of the swarm's other service names, so the swarm-services sub-CA can issue for it, and in `swarm.serviceDomains` on every host whether or not one runs it. ''; @@ -44,7 +43,8 @@ in domain = lib.mkOption { type = lib.types.nullOr lib.types.str; - default = null; + default = config.services.hyperhive.swarm.attic.ownDomain; + defaultText = lib.literalExpression "services.hyperhive.swarm.attic.ownDomain"; example = "cache.example.org"; description = '' Name of the attic server, with no scheme: the swarm builder pushes to @@ -53,13 +53,15 @@ in {option}`services.hyperhive.swarm.attic.cache`; evaluation fails when only one is set. + Defaults to the swarm's own cache, the same on every host. For a cache + outside the swarm, override both options and leave + {option}`services.hyperhive.deploy.attic.enable` unset. With no cache + behind the default, a push fails and the closure record says it is not + cached. + Null pushes nothing: the swarm builder does not push, and the closure record says it is not cached. - {option}`services.hyperhive.deploy.attic.enable` defaults this to - `attic.` on the host that runs the swarm's own cache. Every - other host reads it from the same option, so name it there too. - The push token is not an option. It is at `secret/swarm/services/attic/push-token` in the swarm secret store (field `value`), and the builder reads it on every push. The swarm's own @@ -70,12 +72,12 @@ in cache = lib.mkOption { # attic's own rule for a cache name; the builder checks the same at start. type = lib.types.nullOr (lib.types.strMatching "^[A-Za-z0-9][A-Za-z0-9_+-]{0,49}$"); - default = null; + default = "swarm"; example = "grid"; description = '' Name of the cache on {option}`services.hyperhive.swarm.attic.domain` - that the swarm builder pushes closures to, `swarm` by default on the - host that runs the swarm's own cache. The push token must be allowed to + that the swarm builder pushes closures to, the swarm's own cache by + default. The push token must be allowed to push to it. The name follows attic's rule: 1 to 50 letters, digits, `-`, `_` or `+`, starting with a letter or digit. ''; diff --git a/nix/host-modules/swarm-attic.nix b/nix/host-modules/swarm-attic.nix index 2ef51c22..903672d1 100644 --- a/nix/host-modules/swarm-attic.nix +++ b/nix/host-modules/swarm-attic.nix @@ -14,9 +14,8 @@ # and attic's own tokens are the authentication. Storage is local disk under the # container's state. # -# The two `swarm.attic` options every host reads (./swarm-attic-options.nix) get -# their values from here, at `mkDefault`: naming them yourself wins, and with -# the switch off nothing is set. +# The two `swarm.attic` options every host reads (./swarm-attic-options.nix) +# default to this cache's names on every host, whether or not it runs here. { pkgs, lib, @@ -168,11 +167,6 @@ in }; config = lib.mkIf cfg.enable { - services.hyperhive.swarm.attic = { - domain = lib.mkDefault atticCfg.ownDomain; - cache = lib.mkDefault "swarm"; - }; - assertions = [ { assertion = haveClientIdentity; diff --git a/nix/host-modules/swarm-required-services.nix b/nix/host-modules/swarm-required-services.nix index a7477fdb..de1f7e76 100644 --- a/nix/host-modules/swarm-required-services.nix +++ b/nix/host-modules/swarm-required-services.nix @@ -132,7 +132,7 @@ in # The binary cache, on the same terms: its units log in to the store as the # cache's own principal. An operator running an external cache sets this to - # `false` and names `swarm.attic.{domain,cache}` itself. + # `lib.mkForce false` and overrides `swarm.attic.{domain,cache}`. config.services.hyperhive.deploy.attic.enable = lib.mkDefault ( deployCfg.allSwarmServices && deployCfg.bao.atticClientCertFile != null diff --git a/nix/module-eval/builder.nix b/nix/module-eval/builder.nix index e4cdc06b..1f33645e 100644 --- a/nix/module-eval/builder.nix +++ b/nix/module-eval/builder.nix @@ -45,6 +45,7 @@ let deploy.bao.enable = true; deploy.builder.enable = true; swarm.attic.domain = "cache.example.org"; + swarm.attic.cache = null; }; refusedHalfCache = m: @@ -67,13 +68,23 @@ let svc = inner.systemd.services.swarm-builder; env = svc.environment; hasOnPath = name: s: lib.any (p: lib.getName p == name) s.path; - # A builder with no cache named anywhere. + # A builder that opts out of pushing: both options null. withBuilderNoCache = hive { deploy.bao.enable = true; deploy.builder.enable = true; + swarm.attic.domain = null; + swarm.attic.cache = null; }; withoutCache = withBuilderNoCache.containers.swarm-builder.config.systemd.services.swarm-builder; + # A builder on a host that doesn't run the cache and names nothing. + builderDefaults = hive { + deploy.bao.enable = true; + deploy.builder.enable = true; + }; + defaultsEnv = + builderDefaults.containers.swarm-builder.config.systemd.services.swarm-builder.environment; + # Hosting the shared services runs the swarm's own cache, which names itself. atticHere = servicesHere; atticDeploy = atticHere.services.hyperhive.deploy.attic; @@ -83,7 +94,7 @@ let atticTokens = atticHere.systemd.services.swarm-bao-attic-tokens; atticVhost = atticHere.services.hyperhive.gateway.vhosts."attic.t.local"; - # The cache switched on while the operator names the options by hand. + # The cache switched on while the operator overrides the options. atticByHand = hive { deploy.bao.enable = true; deploy.attic.enable = true; @@ -310,10 +321,10 @@ let } { # The switch is on wherever the shared services are and the host holds the - # cache's leaf, and the two options every host reads come from the cache. - # The builder on the same host pushes to it. `bare` is the control that the - # lookup reads null where nothing runs a cache. - name = "hosting the shared services runs the swarm's cache and fills the cache options, which the builder follows"; + # cache's leaf. + # The builder on the same host pushes to it. `bare` is the control that no + # cache renders where nothing enables it. + name = "hosting the shared services runs the swarm's cache, and the builder pushes to it"; ok = atticDeploy.enable && atticHere.containers ? swarm-attic @@ -327,10 +338,23 @@ let && atticHere.containers.swarm-builder.config.systemd.services.swarm-builder.environment.SWARM_BUILDER_ATTIC_CACHE == "swarm" - && (atticSwarm bare).domain == null - && (atticSwarm bare).cache == null && !(hasAtticHost bare); } + { + # The options default to the swarm's own cache on every host, so a + # builder host that runs no cache pushes to it. `atticHere` is the control + # that the cache host reads the same values. + name = "a builder host that doesn't run the cache pushes to the swarm's own by default"; + ok = + !(hasAtticHost builderDefaults) + && (atticSwarm builderDefaults).domain == "attic.t.local" + && (atticSwarm builderDefaults).cache == "swarm" + && (atticSwarm bare).domain == (atticSwarm atticHere).domain + && (atticSwarm bare).cache == (atticSwarm atticHere).cache + && defaultsEnv.SWARM_BUILDER_ATTIC_DOMAIN == "attic.t.local" + && defaultsEnv.SWARM_BUILDER_ATTIC_CACHE == "swarm" + && !(builderDefaults.systemd.services ? swarm-bao-attic-tokens); + } { # Names the operator gives win over the cache's own, and the gateway then # serves the cache under the operator's name. `atticHere` is the control diff --git a/nix/module-eval/swarm-services-switch.nix b/nix/module-eval/swarm-services-switch.nix index 1df72a4d..d372845e 100644 --- a/nix/module-eval/swarm-services-switch.nix +++ b/nix/module-eval/swarm-services-switch.nix @@ -304,12 +304,12 @@ let } { # The cache's units assert the store identity the store's host mints, as - # the router's do; its options stay as unset as they were. + # the router's do; its options keep the same defaults. name = "the swarm's cache stays off where the shared services are hosted without the store"; ok = !swarmServicesBaoElsewhere.services.hyperhive.deploy.attic.enable && !(swarmServicesBaoElsewhere.containers ? swarm-attic) - && swarmServicesBaoElsewhere.services.hyperhive.swarm.attic.domain == null + && swarmServicesBaoElsewhere.services.hyperhive.swarm.attic.domain == "attic.t.local" && swarmServicesHere.services.hyperhive.deploy.attic.enable && swarmServicesBaoElsewhere.services.hyperhive.deploy.nats.enable; } diff --git a/swarm-builder/README.md b/swarm-builder/README.md index 8a3864be..6449986a 100644 --- a/swarm-builder/README.md +++ b/swarm-builder/README.md @@ -31,8 +31,8 @@ the platform it builds for. The controller records the answer; the service writes nothing. It takes the same one-build slot as the PR checks. See [`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate). -Where `services.hyperhive.swarm.attic.{domain,cache}` are set, it first pushes a -green build's closure to that cache with `attic push`, and the answer says +Where `services.hyperhive.swarm.attic.{domain,cache}` are not null (they default +to the swarm's own cache), it first pushes a green build's closure to that cache with `attic push`, and the answer says whether that worked. A failed push leaves the build green. See [`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache).