default swarm.attic domain and cache to the swarm's own cache on every host
This commit is contained in:
atlas
mara
parent
61292e865e
commit
606817a994
10 changed files with 80 additions and 57 deletions
|
|
@ -258,16 +258,18 @@ grant the forge refuses fails the build with no verdict.
|
|||
|
||||
## Binary cache
|
||||
|
||||
With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes
|
||||
each green meta build's closure to that cache, an attic server inside or outside the
|
||||
swarm, before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
|
||||
`services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own
|
||||
cache (`attic.<swarm domain>` and `swarm`) on every host, so the builder pushes
|
||||
each green meta build's closure to it, or to the cache you override them with,
|
||||
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
|
||||
not `attic watch`, and the builder pushes again when the controller runs a build again. The
|
||||
push has ten minutes, or less when the reply is due sooner, because it must end
|
||||
inside the 61 minutes the controller waits. `attic` checks the server's certificate
|
||||
against the system roots and pins nothing, so a Let's Encrypt certificate
|
||||
works.
|
||||
|
||||
Set both options or neither: evaluation refuses a domain without a cache name.
|
||||
Set both options to a name or both to null: evaluation refuses one without the
|
||||
other. Null in both is how a swarm opts out of pushing.
|
||||
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
|
||||
starting with a letter or digit, and evaluation refuses any other name.
|
||||
|
||||
|
|
@ -281,8 +283,8 @@ The token is in no argument or variable, so no process listing shows it.
|
|||
The reply's `cached` says whether the push finished. A push that fails, times out
|
||||
or can't read the token leaves the build green with `cached` false and logs a
|
||||
warning with the reason; the deploy goes ahead and the next build of the agent
|
||||
tries again. With no cache set, the builder pushes nothing and `cached` is always
|
||||
false.
|
||||
tries again. With no attic behind the defaults, every push fails this way. With
|
||||
both options null, the builder pushes nothing and `cached` is always false.
|
||||
|
||||
Mixed versions:
|
||||
|
||||
|
|
@ -315,11 +317,9 @@ mints nothing. Both tokens last ten years and nothing renews them. To replace on
|
|||
write the new token to its path in the store: nothing mints over a value that
|
||||
exists. A new signing key invalidates both tokens.
|
||||
|
||||
With the switch on, `swarm.attic.domain` and `swarm.attic.cache` default to
|
||||
`attic.<swarm domain>` and `swarm`, so the builder on that host pushes to it.
|
||||
Naming either option yourself wins over its default. The defaults apply only on
|
||||
the host that runs the cache: on any other host, name both options too, with the
|
||||
same values.
|
||||
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
|
||||
and `swarm` on every host, whether or not it runs the cache, so a builder on any
|
||||
host pushes to it.
|
||||
|
||||
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
|
||||
path nobody added or fetched for 14 days goes at the next garbage collection,
|
||||
|
|
@ -336,9 +336,10 @@ to its journal only; its journal doesn't reach the swarm log store.
|
|||
The cache can be an attic tenant on an instance outside the swarm. To point
|
||||
the swarm at an existing tenant:
|
||||
|
||||
1. On the builder host, set `services.hyperhive.swarm.attic.domain` to the
|
||||
1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the
|
||||
instance's host name (no scheme, `host:port` if it isn't on 443) and
|
||||
`services.hyperhive.swarm.attic.cache` to the cache's name on it.
|
||||
`services.hyperhive.swarm.attic.cache` to the cache's name on it. Set
|
||||
both options on every host that reads them.
|
||||
2. Seed the push token in the swarm secret store at
|
||||
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI,
|
||||
or `bao kv put` with `value=-` and the token on stdin, so the token is in no
|
||||
|
|
@ -350,9 +351,9 @@ pins nothing, so a Let's Encrypt certificate needs no setup. The builder
|
|||
reads the token again on every push, so a new token takes effect on the next
|
||||
build.
|
||||
|
||||
Where you host the shared services, `deploy.attic.enable` defaults on and runs
|
||||
the swarm's own cache. To use an external one, set it to `false` with
|
||||
`lib.mkForce`, then set the two options by hand and seed the token as above.
|
||||
To use an external cache, override the two defaults and don't set
|
||||
`deploy.attic.enable` anywhere. Where you host the shared services with
|
||||
`allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there.
|
||||
The swarm then renders no container, vhost or unit of its own cache, and
|
||||
creates no secret in the store. The store's host still mints the cache's
|
||||
leaf and writes its role, which nothing logs in to.
|
||||
|
|
|
|||
|
|
@ -43,7 +43,7 @@ the rest of the cell says how.
|
|||
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
|
||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
|
||||
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
|
||||
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it |
|
||||
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
|
||||
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated |
|
||||
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
|
|
|
|||
|
|
@ -437,12 +437,13 @@ for the CI runner. What it builds and why are in [`builder.md`](builder.md).
|
|||
The swarm's attic server, in a `swarm-attic` container on the host that sets
|
||||
`deploy.attic.enable`. `allSwarmServices` defaults it on wherever the host has
|
||||
the cache's store identity, as for the builder: the host that runs the store has
|
||||
it. Set `deploy.attic.enable = lib.mkForce false` to use an external cache
|
||||
instead and name `swarm.attic.{domain,cache}` yourself.
|
||||
it. For an external cache, override `swarm.attic.{domain,cache}` and don't set
|
||||
`deploy.attic.enable` anywhere, or set it to `lib.mkForce false` under
|
||||
`allSwarmServices`.
|
||||
|
||||
The container shares the host's network namespace and atticd listens on loopback
|
||||
only. The gateway serves it at `swarm.attic.domain` (default
|
||||
`attic.<swarm domain>`) over TLS, with no login in front, because attic checks
|
||||
`attic.<swarm domain>`, on every host) over TLS, with no login in front, because attic checks
|
||||
its own tokens. Storage is local disk in the container's state.
|
||||
|
||||
It needs no manual secret. Two host units, under the cache's own `attic` leaf,
|
||||
|
|
|
|||
|
|
@ -521,9 +521,10 @@ in
|
|||
{option}`services.hyperhive.deploy.bao.atticClientCertFile` and
|
||||
`atticClientKeyFile`, minted on the store's host).
|
||||
|
||||
With it on, {option}`services.hyperhive.swarm.attic.domain` and `cache`
|
||||
default to the cache's own name and cache; naming either yourself wins.
|
||||
With it off, those two options name an external cache or stay unset.
|
||||
{option}`services.hyperhive.swarm.attic.domain` and `cache` default to
|
||||
this cache on every host. For an external cache, override both and
|
||||
leave this unset, or set it to `lib.mkForce false` under
|
||||
`allSwarmServices`.
|
||||
'';
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# The binary cache the swarm pushes agent closures to, as every host sees it:
|
||||
# which attic server, and which cache on it. A host that sets
|
||||
# `deploy.attic.enable` runs the swarm's own and fills both in (./swarm-attic.nix);
|
||||
# otherwise the instance is the operator's, outside the swarm.
|
||||
# which attic server, and which cache on it. Both default to the swarm's own,
|
||||
# which the host that sets `deploy.attic.enable` runs (./swarm-attic.nix); an
|
||||
# operator with an instance outside the swarm overrides both.
|
||||
{ lib, config, ... }:
|
||||
let
|
||||
swarmDomain = config.services.hyperhive.swarm.domain;
|
||||
|
|
@ -35,8 +35,7 @@ in
|
|||
defaultText = lib.literalExpression ''"attic.''${services.hyperhive.swarm.domain}"'';
|
||||
description = ''
|
||||
Read-only: the name the swarm's own cache is served under, which
|
||||
{option}`services.hyperhive.deploy.attic.enable` defaults
|
||||
{option}`services.hyperhive.swarm.attic.domain` to. A sibling of the
|
||||
{option}`services.hyperhive.swarm.attic.domain` defaults to. A sibling of the
|
||||
swarm's other service names, so the swarm-services sub-CA can issue for it,
|
||||
and in `swarm.serviceDomains` on every host whether or not one runs it.
|
||||
'';
|
||||
|
|
@ -44,7 +43,8 @@ in
|
|||
|
||||
domain = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
default = config.services.hyperhive.swarm.attic.ownDomain;
|
||||
defaultText = lib.literalExpression "services.hyperhive.swarm.attic.ownDomain";
|
||||
example = "cache.example.org";
|
||||
description = ''
|
||||
Name of the attic server, with no scheme: the swarm builder pushes to
|
||||
|
|
@ -53,13 +53,15 @@ in
|
|||
{option}`services.hyperhive.swarm.attic.cache`; evaluation fails when only
|
||||
one is set.
|
||||
|
||||
Defaults to the swarm's own cache, the same on every host. For a cache
|
||||
outside the swarm, override both options and leave
|
||||
{option}`services.hyperhive.deploy.attic.enable` unset. With no cache
|
||||
behind the default, a push fails and the closure record says it is not
|
||||
cached.
|
||||
|
||||
Null pushes nothing: the swarm builder does not push, and the closure
|
||||
record says it is not cached.
|
||||
|
||||
{option}`services.hyperhive.deploy.attic.enable` defaults this to
|
||||
`attic.<swarm.domain>` on the host that runs the swarm's own cache. Every
|
||||
other host reads it from the same option, so name it there too.
|
||||
|
||||
The push token is not an option. It is at
|
||||
`secret/swarm/services/attic/push-token` in the swarm secret store
|
||||
(field `value`), and the builder reads it on every push. The swarm's own
|
||||
|
|
@ -70,12 +72,12 @@ in
|
|||
cache = lib.mkOption {
|
||||
# attic's own rule for a cache name; the builder checks the same at start.
|
||||
type = lib.types.nullOr (lib.types.strMatching "^[A-Za-z0-9][A-Za-z0-9_+-]{0,49}$");
|
||||
default = null;
|
||||
default = "swarm";
|
||||
example = "grid";
|
||||
description = ''
|
||||
Name of the cache on {option}`services.hyperhive.swarm.attic.domain`
|
||||
that the swarm builder pushes closures to, `swarm` by default on the
|
||||
host that runs the swarm's own cache. The push token must be allowed to
|
||||
that the swarm builder pushes closures to, the swarm's own cache by
|
||||
default. The push token must be allowed to
|
||||
push to it. The name follows attic's rule: 1 to 50 letters,
|
||||
digits, `-`, `_` or `+`, starting with a letter or digit.
|
||||
'';
|
||||
|
|
|
|||
|
|
@ -14,9 +14,8 @@
|
|||
# and attic's own tokens are the authentication. Storage is local disk under the
|
||||
# container's state.
|
||||
#
|
||||
# The two `swarm.attic` options every host reads (./swarm-attic-options.nix) get
|
||||
# their values from here, at `mkDefault`: naming them yourself wins, and with
|
||||
# the switch off nothing is set.
|
||||
# The two `swarm.attic` options every host reads (./swarm-attic-options.nix)
|
||||
# default to this cache's names on every host, whether or not it runs here.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
|
|
@ -168,11 +167,6 @@ in
|
|||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
services.hyperhive.swarm.attic = {
|
||||
domain = lib.mkDefault atticCfg.ownDomain;
|
||||
cache = lib.mkDefault "swarm";
|
||||
};
|
||||
|
||||
assertions = [
|
||||
{
|
||||
assertion = haveClientIdentity;
|
||||
|
|
|
|||
|
|
@ -132,7 +132,7 @@ in
|
|||
|
||||
# The binary cache, on the same terms: its units log in to the store as the
|
||||
# cache's own principal. An operator running an external cache sets this to
|
||||
# `false` and names `swarm.attic.{domain,cache}` itself.
|
||||
# `lib.mkForce false` and overrides `swarm.attic.{domain,cache}`.
|
||||
config.services.hyperhive.deploy.attic.enable = lib.mkDefault (
|
||||
deployCfg.allSwarmServices
|
||||
&& deployCfg.bao.atticClientCertFile != null
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ let
|
|||
deploy.bao.enable = true;
|
||||
deploy.builder.enable = true;
|
||||
swarm.attic.domain = "cache.example.org";
|
||||
swarm.attic.cache = null;
|
||||
};
|
||||
refusedHalfCache =
|
||||
m:
|
||||
|
|
@ -67,13 +68,23 @@ let
|
|||
svc = inner.systemd.services.swarm-builder;
|
||||
env = svc.environment;
|
||||
hasOnPath = name: s: lib.any (p: lib.getName p == name) s.path;
|
||||
# A builder with no cache named anywhere.
|
||||
# A builder that opts out of pushing: both options null.
|
||||
withBuilderNoCache = hive {
|
||||
deploy.bao.enable = true;
|
||||
deploy.builder.enable = true;
|
||||
swarm.attic.domain = null;
|
||||
swarm.attic.cache = null;
|
||||
};
|
||||
withoutCache = withBuilderNoCache.containers.swarm-builder.config.systemd.services.swarm-builder;
|
||||
|
||||
# A builder on a host that doesn't run the cache and names nothing.
|
||||
builderDefaults = hive {
|
||||
deploy.bao.enable = true;
|
||||
deploy.builder.enable = true;
|
||||
};
|
||||
defaultsEnv =
|
||||
builderDefaults.containers.swarm-builder.config.systemd.services.swarm-builder.environment;
|
||||
|
||||
# Hosting the shared services runs the swarm's own cache, which names itself.
|
||||
atticHere = servicesHere;
|
||||
atticDeploy = atticHere.services.hyperhive.deploy.attic;
|
||||
|
|
@ -83,7 +94,7 @@ let
|
|||
atticTokens = atticHere.systemd.services.swarm-bao-attic-tokens;
|
||||
atticVhost = atticHere.services.hyperhive.gateway.vhosts."attic.t.local";
|
||||
|
||||
# The cache switched on while the operator names the options by hand.
|
||||
# The cache switched on while the operator overrides the options.
|
||||
atticByHand = hive {
|
||||
deploy.bao.enable = true;
|
||||
deploy.attic.enable = true;
|
||||
|
|
@ -310,10 +321,10 @@ let
|
|||
}
|
||||
{
|
||||
# The switch is on wherever the shared services are and the host holds the
|
||||
# cache's leaf, and the two options every host reads come from the cache.
|
||||
# The builder on the same host pushes to it. `bare` is the control that the
|
||||
# lookup reads null where nothing runs a cache.
|
||||
name = "hosting the shared services runs the swarm's cache and fills the cache options, which the builder follows";
|
||||
# cache's leaf.
|
||||
# The builder on the same host pushes to it. `bare` is the control that no
|
||||
# cache renders where nothing enables it.
|
||||
name = "hosting the shared services runs the swarm's cache, and the builder pushes to it";
|
||||
ok =
|
||||
atticDeploy.enable
|
||||
&& atticHere.containers ? swarm-attic
|
||||
|
|
@ -327,10 +338,23 @@ let
|
|||
&&
|
||||
atticHere.containers.swarm-builder.config.systemd.services.swarm-builder.environment.SWARM_BUILDER_ATTIC_CACHE
|
||||
== "swarm"
|
||||
&& (atticSwarm bare).domain == null
|
||||
&& (atticSwarm bare).cache == null
|
||||
&& !(hasAtticHost bare);
|
||||
}
|
||||
{
|
||||
# The options default to the swarm's own cache on every host, so a
|
||||
# builder host that runs no cache pushes to it. `atticHere` is the control
|
||||
# that the cache host reads the same values.
|
||||
name = "a builder host that doesn't run the cache pushes to the swarm's own by default";
|
||||
ok =
|
||||
!(hasAtticHost builderDefaults)
|
||||
&& (atticSwarm builderDefaults).domain == "attic.t.local"
|
||||
&& (atticSwarm builderDefaults).cache == "swarm"
|
||||
&& (atticSwarm bare).domain == (atticSwarm atticHere).domain
|
||||
&& (atticSwarm bare).cache == (atticSwarm atticHere).cache
|
||||
&& defaultsEnv.SWARM_BUILDER_ATTIC_DOMAIN == "attic.t.local"
|
||||
&& defaultsEnv.SWARM_BUILDER_ATTIC_CACHE == "swarm"
|
||||
&& !(builderDefaults.systemd.services ? swarm-bao-attic-tokens);
|
||||
}
|
||||
{
|
||||
# Names the operator gives win over the cache's own, and the gateway then
|
||||
# serves the cache under the operator's name. `atticHere` is the control
|
||||
|
|
|
|||
|
|
@ -304,12 +304,12 @@ let
|
|||
}
|
||||
{
|
||||
# The cache's units assert the store identity the store's host mints, as
|
||||
# the router's do; its options stay as unset as they were.
|
||||
# the router's do; its options keep the same defaults.
|
||||
name = "the swarm's cache stays off where the shared services are hosted without the store";
|
||||
ok =
|
||||
!swarmServicesBaoElsewhere.services.hyperhive.deploy.attic.enable
|
||||
&& !(swarmServicesBaoElsewhere.containers ? swarm-attic)
|
||||
&& swarmServicesBaoElsewhere.services.hyperhive.swarm.attic.domain == null
|
||||
&& swarmServicesBaoElsewhere.services.hyperhive.swarm.attic.domain == "attic.t.local"
|
||||
&& swarmServicesHere.services.hyperhive.deploy.attic.enable
|
||||
&& swarmServicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -31,8 +31,8 @@ the platform it builds for. The controller records the answer; the service
|
|||
writes nothing. It takes the same one-build slot as the PR checks. See
|
||||
[`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate).
|
||||
|
||||
Where `services.hyperhive.swarm.attic.{domain,cache}` are set, it first pushes a
|
||||
green build's closure to that cache with `attic push`, and the answer says
|
||||
Where `services.hyperhive.swarm.attic.{domain,cache}` are not null (they default
|
||||
to the swarm's own cache), it first pushes a green build's closure to that cache with `attic push`, and the answer says
|
||||
whether that worked. A failed push leaves the build green. See
|
||||
[`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache).
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue