Watch
0
0
Fork
You've already forked hyperhive
0

push green meta builds to the attic cache and record cached

This commit is contained in:
atlas 2026-10-27 10:12:35 +01:00
commit 0849afec80
16 changed files with 779 additions and 31 deletions

View file

@ -103,9 +103,10 @@ status has no link.
hive's. A green check can therefore still fail on a hive, and the reverse. hive's. A green check can therefore still fail on a hive, and the reverse.
- The agent base is the builder's host's, not the hive's. Hives on another - The agent base is the builder's host's, not the hive's. Hives on another
hyperhive or nixpkgs pin build something else. hyperhive or nixpkgs pin build something else.
- The builder pushes nothing. The closure stays in the builder - The check pushes nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches. hive's own build can reuse whatever matches. A [meta build](#deploy-gate)
can push to [a binary cache](#binary-cache).
- The PR check gates nothing. A merge deploys whatever the status says. The - The PR check gates nothing. A merge deploys whatever the status says. The
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`. [deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
- A lost webhook delivery isn't retried: the head isn't built until the next - A lost webhook delivery isn't retried: the head isn't built until the next
@ -247,13 +248,48 @@ On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A failed build writes nothing and leaves the previous key. holds no grant on it. A failed build writes nothing and leaves the previous key.
Nothing reads the bucket yet. The closure stays unrooted in the builder host's The record's `cached` field says whether the builder pushed the closure to
store. [the binary cache](#binary-cache). Nothing reads the bucket yet. The closure stays
unrooted in the builder host's store.
The controller gives the `builder` forge account read on `internal/meta` when it The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, and again before each build, since the repo is private. A updates the flake, and again before each build, since the repo is private. A
grant the forge refuses fails the build with no verdict. grant the forge refuses fails the build with no verdict.
## Binary cache
With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes
each green meta build's closure to that cache, an attic server outside the swarm,
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
not `attic watch`, and the builder pushes again when the controller runs a build again. The
push has ten minutes, or less when the reply is due sooner, because it must end
inside the 61 minutes the controller waits. `attic` checks the server's certificate
against the system roots and pins nothing, so a Let's Encrypt certificate
works.
The push token is the operator's to write. The builder reads it on every push
from `secret/swarm/services/attic/push-token` (field `value`), writes an `attic`
config file holding it in the unit's private runtime directory (`/run/swarm-builder`,
mode `0700`, the file `0600`), points `attic` at it with `XDG_CONFIG_HOME` and
removes it after the push. The token is in no argument or variable, so no
process listing shows it.
The reply's `cached` says whether the push finished. A push that fails, times out
or can't read the token leaves the build green with `cached` false and logs a
warning with the reason; the deploy goes ahead and the next build of the agent
tries again. With no cache set, the builder pushes nothing and `cached` is always
false.
Mixed versions:
| controller | builder | what happens |
| ---------- | ------- | ------------------------------------------------------------- |
| new | new | `cached` in the reply and in the `agent-closures` record |
| new | old | the reply has no `cached`, which reads as `false` |
| old | new | the controller ignores `cached`; its record has no such field |
A record written before the field existed also reads as `false`.
## Credentials ## Credentials
| What | Where | Who reads it | | What | Where | Who reads it |
@ -261,6 +297,7 @@ grant the forge refuses fails the build with no verdict.
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential | | store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it | | queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build | | forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push |
The token belongs to the `builder` forge account, which swarm-controller The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config creates, puts in the `hives` team of `agent-configs` (read on every config
@ -270,7 +307,7 @@ gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers status. The same `git` fetches a config's own inputs, so the builder registers
the helper for the forge's URL only, and the helper answers only an `https` the helper for the forge's URL only, and the helper answers only an `https`
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
status with its own forge token. Details per path: status with its own forge token. Details per path:
[`credentials-reference.md`](credentials-reference.md). [`credentials-reference.md`](credentials-reference.md).

View file

@ -41,6 +41,7 @@ the rest of the cell says how.
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run | | `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build | | `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
| `swarm/services/attic/push-token` | you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
@ -158,7 +159,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` | | `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` | | `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` | | `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` | | `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` | | `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |

View file

@ -62,6 +62,7 @@
./glue-swarm-otel-oidc-client.nix ./glue-swarm-otel-oidc-client.nix
./glue-victorialogs-otel-oidc-client.nix ./glue-victorialogs-otel-oidc-client.nix
./glue-victoriametrics-otel-oidc-client.nix ./glue-victoriametrics-otel-oidc-client.nix
./swarm-attic-options.nix
./swarm-authelia-service.nix ./swarm-authelia-service.nix
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao-service.nix ./swarm-bao-service.nix

View file

@ -0,0 +1,37 @@
# The binary cache the swarm pushes agent closures to, as every host sees it:
# which attic server, and which cache on it. The swarm runs no attic of its
# own yet; the instance is the operator's, outside the swarm.
{ lib, ... }:
{
options.services.hyperhive.swarm.attic = {
domain = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "cache.example.org";
description = ''
Name of the attic server, with no scheme: the swarm builder pushes to
`https://<domain>`, and trusts the certificate the system roots do, so
a Let's Encrypt certificate works and none is pinned. Set together with
{option}`services.hyperhive.swarm.attic.cache`.
Null pushes nothing: the swarm builder does not push, and the closure
record says it is not cached.
The push token is not an option. The operator writes it to
`secret/swarm/services/attic/push-token` in the swarm secret store
(field `value`), and the builder reads it on every push.
'';
};
cache = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "grid";
description = ''
Name of the cache on {option}`services.hyperhive.swarm.attic.domain`
that the swarm builder pushes closures to. The push token must be
allowed to push to it.
'';
};
};
}

View file

@ -692,6 +692,12 @@ let
# to fetch config repos. # to fetch config repos.
builderForgeTokenLeaf = "swarm/services/builder/forge/token"; builderForgeTokenLeaf = "swarm/services/builder/forge/token";
# The token the swarm builder pushes closures to the binary cache with, the nix
# half of `swarm_secret_client::builder::attic_push_token_path`. The operator
# writes it by hand; the builder reads it on every push, and nothing else in
# the swarm reaches it.
atticPushTokenLeaf = "swarm/services/attic/push-token";
# The CI run poller's forge token, the nix half of # The CI run poller's forge token, the nix half of
# `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it # `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it
# and reads it back before deciding whether to re-mint; the poller reads it # and reads it back before deciding whether to re-mint; the poller reads it
@ -1146,14 +1152,16 @@ let
} }
]; ];
# The swarm builder: its forge token, read once per build, and its own queue # The swarm builder: its forge token, read once per build, the cache push token,
# client secret, read at start. Both by the daemon inside its container. # read once per push, and its own queue client secret, read at start. All by the
# daemon inside its container.
builderReaders = [ builderReaders = [
{ {
name = "swarm-builder"; name = "swarm-builder";
cn = baoDeploy.builderCommonName; cn = baoDeploy.builderCommonName;
policyText = policyText =
readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}" readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}"
+ readStanza "${credentialMountPath}/data/${atticPushTokenLeaf}"
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client"; + readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client";
} }
]; ];
@ -2081,8 +2089,8 @@ in
description = '' description = ''
Subject the store's `swarm-builder` cert-auth role accepts: the Subject the store's `swarm-builder` cert-auth role accepts: the
identity the swarm builder presents from inside its container. Its identity the swarm builder presents from inside its container. Its
grant is two paths, read only: `swarm/services/builder/forge/token` grant is three paths, read only: `swarm/services/builder/forge/token`,
and its own queue client secret. `swarm/services/attic/push-token` and its own queue client secret.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings. ⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
''; '';

View file

@ -17,8 +17,8 @@
# #
# The container holds one credential, its own store leaf, which nspawn hands in # The container holds one credential, its own store leaf, which nspawn hands in
# as a system credential with no copy on disk. The daemon reads everything else # as a system credential with no copy on disk. The daemon reads everything else
# from the store with it: its queue client secret at start and its forge token # from the store with it: its queue client secret at start, its forge token
# on every build. # on every build and, where a cache is set, its cache push token on every push.
{ {
pkgs, pkgs,
lib, lib,
@ -33,6 +33,7 @@ let
natsCfg = hyperhiveCfg.swarm.nats; natsCfg = hyperhiveCfg.swarm.nats;
autheliaCfg = hyperhiveCfg.swarm.authelia; autheliaCfg = hyperhiveCfg.swarm.authelia;
forgeCfg = hyperhiveCfg.swarm.forge; forgeCfg = hyperhiveCfg.swarm.forge;
atticCfg = hyperhiveCfg.swarm.attic;
gatewayCfg = hyperhiveCfg.gateway; gatewayCfg = hyperhiveCfg.gateway;
networkCfg = hyperhiveCfg.network; networkCfg = hyperhiveCfg.network;
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls; tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
@ -89,6 +90,9 @@ let
hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake; hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake;
nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake; nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake;
# Either half alone reaches the daemon too, which refuses to start on it.
pushesToCache = atticCfg.domain != null || atticCfg.cache != null;
privateNetwork = true; privateNetwork = true;
in in
{ {
@ -230,8 +234,9 @@ in
wantedBy = [ "multi-user.target" ]; wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ]; wants = [ "network-online.target" ];
after = [ "network-online.target" ]; after = [ "network-online.target" ];
# nix fetches the config repo with `git`. # nix fetches the config repo with `git`, and a green build is pushed
path = [ pkgs.git ]; # with `attic` where a cache is set.
path = [ pkgs.git ] ++ lib.optional pushesToCache pkgs.attic-client;
environment = { environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = credential "builder-bao-cert"; BAO_CLIENT_CERT = credential "builder-bao-cert";
@ -252,6 +257,12 @@ in
SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}"; SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}";
SWARM_BUILDER_CONFIG_ORG = "agent-configs"; SWARM_BUILDER_CONFIG_ORG = "agent-configs";
} }
// lib.optionalAttrs (atticCfg.domain != null) {
SWARM_BUILDER_ATTIC_DOMAIN = atticCfg.domain;
}
// lib.optionalAttrs (atticCfg.cache != null) {
SWARM_BUILDER_ATTIC_CACHE = atticCfg.cache;
}
// lib.optionalAttrs (storeFiles ? builder-bao-server-ca) { // lib.optionalAttrs (storeFiles ? builder-bao-server-ca) {
BAO_CACERT = credential "builder-bao-server-ca"; BAO_CACERT = credential "builder-bao-server-ca";
} }
@ -264,6 +275,10 @@ in
ExecStart = "${cfg.package}/bin/swarm-builder"; ExecStart = "${cfg.package}/bin/swarm-builder";
DynamicUser = true; DynamicUser = true;
CacheDirectory = unit; CacheDirectory = unit;
# /run/swarm-builder, mode 0700 and owned by the dynamic user: the
# push writes its token-bearing `attic` config under it.
RuntimeDirectory = unit;
RuntimeDirectoryMode = "0700";
# Bare ids: the system credentials nspawn handed in. # Bare ids: the system credentials nspawn handed in.
LoadCredential = lib.attrNames storeFiles; LoadCredential = lib.attrNames storeFiles;
Restart = "on-failure"; Restart = "on-failure";

View file

@ -1743,17 +1743,18 @@ let
]; ];
} }
{ {
# The builder reads its forge token and its own queue client secret, # The builder reads its forge token, the cache push token and its own queue
# each read-only and spelled to the leaf: `services/*` would reach every # client secret, each read-only and spelled to the leaf: `services/*` would
# swarm service's secret. # reach every swarm service's secret.
name = "the swarm builder's grant is its forge token and its queue client secret, and nothing else"; name = "the swarm builder's grant is its forge token, the cache push token and its queue client secret, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-builder-policy.script; s = baoGrantHere.systemd.services.swarm-bao-builder-policy.script;
in in
lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"read\"]\n}" s lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"read\"]\n}" s
&& lib.hasInfix "path \"secret/data/swarm/services/attic/push-token\" {\n capabilities = [\"read\"]\n}" s
&& lib.hasInfix "path \"secret/data/swarm/services/swarm-builder/oidc/client\" {\n capabilities = [\"read\"]\n}" s && lib.hasInfix "path \"secret/data/swarm/services/swarm-builder/oidc/client\" {\n capabilities = [\"read\"]\n}" s
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2 && lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 3
&& !(lib.hasInfix "secret/data/swarm/services/*" s) && !(lib.hasInfix "secret/data/swarm/services/*" s)
&& lib.hasInfix "auth/cert/certs/swarm-builder" s && lib.hasInfix "auth/cert/certs/swarm-builder" s
&& lib.hasInfix "allowed_common_names=swarm-builder" s && lib.hasInfix "allowed_common_names=swarm-builder" s

View file

@ -29,6 +29,8 @@ let
withBuilder = hive { withBuilder = hive {
deploy.bao.enable = true; deploy.bao.enable = true;
deploy.builder.enable = true; deploy.builder.enable = true;
swarm.attic.domain = "cache.example.org";
swarm.attic.cache = "grid";
}; };
servicesHere = shared.swarmServicesWithController; servicesHere = shared.swarmServicesWithController;
@ -48,6 +50,8 @@ let
inner = container.config; inner = container.config;
svc = inner.systemd.services.swarm-builder; svc = inner.systemd.services.swarm-builder;
env = svc.environment; env = svc.environment;
hasOnPath = name: s: lib.any (p: lib.getName p == name) s.path;
withoutCache = servicesHere.containers.swarm-builder.config.systemd.services.swarm-builder;
# The check flake on the inputs the builder's unit names, called as # The check flake on the inputs the builder's unit names, called as
# `nix build` calls it. `hyperhive` is entered through the filtered source's # `nix build` calls it. `hyperhive` is entered through the filtered source's
@ -116,6 +120,25 @@ let
&& svc.serviceConfig.DynamicUser && svc.serviceConfig.DynamicUser
&& !(lib.any (lib.hasSuffix "TOKEN") (lib.attrNames env)); && !(lib.any (lib.hasSuffix "TOKEN") (lib.attrNames env));
} }
{
# The closure is pushed with `attic` from the unit, whose config file holds
# the push token: it goes in a runtime directory only the dynamic user can
# enter, and the token is in no variable. The host without a cache is the
# control: same unit, none of it, with `git` still on the path to show
# the lookup finds what is there.
name = "the builder pushes to the configured cache with attic from a private runtime directory, and pushes nowhere without one";
ok =
env.SWARM_BUILDER_ATTIC_DOMAIN == "cache.example.org"
&& env.SWARM_BUILDER_ATTIC_CACHE == "grid"
&& hasOnPath "attic-client" svc
&& svc.serviceConfig.RuntimeDirectory == "swarm-builder"
&& svc.serviceConfig.RuntimeDirectoryMode == "0700"
&& !(lib.any (lib.hasInfix "TOKEN") (lib.attrNames env))
&& hasOnPath "git" withoutCache
&& !(hasOnPath "attic-client" withoutCache)
&& !(withoutCache.environment ? SWARM_BUILDER_ATTIC_DOMAIN)
&& !(withoutCache.environment ? SWARM_BUILDER_ATTIC_CACHE);
}
{ {
# The minting unit goes inactive after each run, so a start that pulls it # The minting unit goes inactive after each run, so a start that pulls it
# runs it again and mints a leaf that is missing; an active # runs it again and mints a leaf that is missing; an active

View file

@ -19,7 +19,7 @@ the answer into a `swarm-builder` commit status on the PR's head.
as an untrusted client, so a config can neither set daemon options nor read as an untrusted client, so a config can neither set daemon options nor read
this service's environment. this service's environment.
- Nothing is pushed and nothing is recorded: the closure stays in the host's - Nothing is pushed and nothing is recorded: the closure stays in the host's
store until it is collected. store until it is collected. (A meta build can push, see below.)
## Meta builds ## Meta builds
@ -31,16 +31,21 @@ the platform it builds for. The controller records the answer; the service
writes nothing. It takes the same one-build slot as the PR checks. See writes nothing. It takes the same one-build slot as the PR checks. See
[`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate). [`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate).
Where `services.hyperhive.swarm.attic.{domain,cache}` are set, it first pushes a
green build's closure to that cache with `attic push`, and the answer says
whether that worked. A failed push leaves the build green. See
[`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache).
## Credentials ## Credentials
The container holds one credential: its own store certificate, which nspawn The container holds one credential: its own store certificate, which nspawn
hands in at start without a copy on disk. With it the service reads its hands in at start without a copy on disk. With it the service reads its
queue client secret at start and again when authelia refuses it, and its queue client secret at start and again when authelia refuses it, and its
read-only forge token at every build. read-only forge token at every build, and the cache push token at every push.
`git` gets the token through a credential helper reading the child's `git` gets the token through a credential helper reading the child's
environment, never in a URL or an argument, and the helper answers only an environment, never in a URL or an argument, and the helper answers only an
`https` request for the forge's host. Its store role reads exactly `https` request for the forge's host. Its store role reads exactly
those two paths. those three paths.
## Archetypes ## Archetypes

View file

@ -16,6 +16,8 @@
//! and an `internal/meta` commit, and the answer is the store path of that //! and an `internal/meta` commit, and the answer is the store path of that
//! agent's toplevel, built at the flake's own lock. They take the build slot //! agent's toplevel, built at the flake's own lock. They take the build slot
//! PR checks take. The controller records the path; the builder writes nothing. //! PR checks take. The controller records the path; the builder writes nothing.
//! When a cache is set (see [`push`]), a green meta build is pushed to it before
//! the reply, and the reply says whether that worked.
//! //!
//! The store identity comes from the `BAO_*` environment. The queue client //! The store identity comes from the `BAO_*` environment. The queue client
//! secret is read at start and again when authelia refuses it, the forge token //! secret is read at start and again when authelia refuses it, the forge token
@ -39,6 +41,7 @@ use swarm_wire::builder::{
mod archetypes; mod archetypes;
mod nix_build; mod nix_build;
mod push;
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix` /// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
/// writes it, matching the leaf's subject. /// writes it, matching the leaf's subject.
@ -64,6 +67,17 @@ const MAX_SLOT_WAIT: Duration = REPLY_DEADLINE
.saturating_sub(BUILD_DEADLINE) .saturating_sub(BUILD_DEADLINE)
.saturating_sub(SLOT_MARGIN); .saturating_sub(SLOT_MARGIN);
/// What is held back from a meta build's reply deadline for publishing the reply
/// after its push.
const REPLY_MARGIN: Duration = Duration::from_secs(10);
/// How long the push of a meta build that ends at now may run: at most
/// [`push::PUSH_DEADLINE`], and never past `reply_by`, so a push cannot make the
/// reply later than the controller waits.
fn push_budget(reply_by: tokio::time::Instant) -> Duration {
push::PUSH_DEADLINE.min(reply_by.saturating_duration_since(tokio::time::Instant::now()))
}
#[tokio::main] #[tokio::main]
async fn main() -> Result<()> { async fn main() -> Result<()> {
hive_log::init(); hive_log::init();
@ -83,6 +97,7 @@ async fn main() -> Result<()> {
deadline: BUILD_DEADLINE, deadline: BUILD_DEADLINE,
lock_deadline: LOCK_DEADLINE, lock_deadline: LOCK_DEADLINE,
system: var("SYSTEM")?, system: var("SYSTEM")?,
cache: push_target()?,
}); });
let secret = let secret =
swarm_secret_client::read_with_startup_retry("reading the queue client secret", || { swarm_secret_client::read_with_startup_retry("reading the queue client secret", || {
@ -300,7 +315,7 @@ async fn answer_meta_builds(
&slot, &slot,
received, received,
|| busy_meta(&job), || busy_meta(&job),
handle_meta(&job, &builder), handle_meta(&job, &builder, received + REPLY_DEADLINE - REPLY_MARGIN),
) )
.await .await
else { else {
@ -383,19 +398,36 @@ fn busy_meta(job: &BuildFromMeta) -> FromMetaReply {
} }
/// Build one meta request and log the build. /// Build one meta request and log the build.
async fn handle_meta(job: &BuildFromMeta, builder: &nix_build::Builder) -> FromMetaReply { async fn handle_meta(
job: &BuildFromMeta,
builder: &nix_build::Builder,
reply_by: tokio::time::Instant,
) -> FromMetaReply {
let reply = match within(TOKEN_DEADLINE, forge_token()).await { let reply = match within(TOKEN_DEADLINE, forge_token()).await {
Ok(token) => builder.build_from_meta(job, &token).await, Ok(token) => builder.build_from_meta(job, &token).await,
Err(e) => FromMetaReply::Unavailable { Err(e) => FromMetaReply::Unavailable {
reason: format!("{e:#}"), reason: format!("{e:#}"),
}, },
}; };
let reply = push::cache_built(
reply,
builder.cache.as_ref(),
attic_push_token(),
push_budget(reply_by),
&job.agent,
)
.await;
match &reply { match &reply {
FromMetaReply::Built { out_path, system } => tracing::info!( FromMetaReply::Built {
out_path,
system,
cached,
} => tracing::info!(
agent = %job.agent, agent = %job.agent,
meta_rev = %job.meta_rev, meta_rev = %job.meta_rev,
%out_path, %out_path,
%system, %system,
cached,
"agent meta build built" "agent meta build built"
), ),
FromMetaReply::Failed { error_tail } => tracing::warn!( FromMetaReply::Failed { error_tail } => tracing::warn!(
@ -458,6 +490,23 @@ fn log_build(job: &BuildAgent, reply: &Reply) {
} }
} }
/// The token the builder pushes closures to the cache with, read on every push so
/// a rotation takes effect on the next one.
async fn attic_push_token() -> Result<String> {
let path = swarm_secret_client::builder::attic_push_token_path()?;
let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE)
.await
.context("logging in to the secret store")?;
let stored: Option<swarm_secret_client::builder::AtticPushToken> = store
.read_optional(&path)
.await
.with_context(|| format!("reading {path}"))?;
match stored {
Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()),
_ => bail!("nothing at {path}: the cache push token has not been written"),
}
}
/// The forge token the builder fetches config repos with, read on every /// The forge token the builder fetches config repos with, read on every
/// build so a rotation takes effect on the next one. /// build so a rotation takes effect on the next one.
async fn forge_token() -> Result<String> { async fn forge_token() -> Result<String> {
@ -495,6 +544,17 @@ async fn client_secret(client_id: &str) -> Result<String> {
} }
} }
/// The cache the unit's environment names, if any. systemd sets
/// `RUNTIME_DIRECTORY` for the unit's runtime directory.
fn push_target() -> Result<Option<push::Remote>> {
let var = |name: &str| std::env::var(name).ok();
push::Remote::from_settings(
var(&format!("{PREFIX}_ATTIC_DOMAIN")),
var(&format!("{PREFIX}_ATTIC_CACHE")),
var("RUNTIME_DIRECTORY"),
)
}
/// The forge's base URL, which must be `https`: the credential helper offers /// The forge's base URL, which must be `https`: the credential helper offers
/// the token to `https` requests for its host alone. /// the token to `https` requests for its host alone.
fn forge_url() -> Result<String> { fn forge_url() -> Result<String> {
@ -580,6 +640,7 @@ mod tests {
FromMetaReply::Built { FromMetaReply::Built {
out_path: "/nix/store/x".to_owned(), out_path: "/nix/store/x".to_owned(),
system: "s".to_owned(), system: "s".to_owned(),
cached: false,
} }
}) })
.await; .await;
@ -594,6 +655,7 @@ mod tests {
FromMetaReply::Built { FromMetaReply::Built {
out_path: "/nix/store/x".to_owned(), out_path: "/nix/store/x".to_owned(),
system: "s".to_owned(), system: "s".to_owned(),
cached: false,
} }
}) })
.await; .await;
@ -616,6 +678,20 @@ mod tests {
assert_eq!(prompt.expect("read"), "t"); assert_eq!(prompt.expect("read"), "t");
} }
#[tokio::test(start_paused = true)]
async fn a_push_gets_the_deadline_unless_the_reply_is_due_sooner() {
let now = Instant::now();
assert_eq!(
push_budget(now + Duration::from_hours(1)),
push::PUSH_DEADLINE
);
assert_eq!(
push_budget(now + Duration::from_secs(90)),
Duration::from_secs(90)
);
assert_eq!(push_budget(now), Duration::ZERO);
}
#[test] #[test]
fn a_build_started_at_the_slot_limit_ends_inside_the_reply_deadline() { fn a_build_started_at_the_slot_limit_ends_inside_the_reply_deadline() {
assert!(MAX_SLOT_WAIT > Duration::ZERO); assert!(MAX_SLOT_WAIT > Duration::ZERO);

View file

@ -59,6 +59,9 @@ pub struct Builder {
pub lock_deadline: Duration, pub lock_deadline: Duration,
/// The platform this host builds for, reported with each meta build. /// The platform this host builds for, reported with each meta build.
pub system: String, pub system: String,
/// Where a meta build's closure is pushed after it succeeds; `None` pushes
/// nothing.
pub cache: Option<crate::push::Remote>,
} }
impl Builder { impl Builder {
@ -88,6 +91,7 @@ impl Builder {
Reply::Built { out_path } => FromMetaReply::Built { Reply::Built { out_path } => FromMetaReply::Built {
out_path, out_path,
system: self.system.clone(), system: self.system.clone(),
cached: false,
}, },
Reply::Failed { error_tail } => FromMetaReply::Failed { error_tail }, Reply::Failed { error_tail } => FromMetaReply::Failed { error_tail },
}, },
@ -437,6 +441,7 @@ mod tests {
deadline: BUILD_DEADLINE, deadline: BUILD_DEADLINE,
lock_deadline: LOCK_DEADLINE, lock_deadline: LOCK_DEADLINE,
system: "x86_64-linux".to_owned(), system: "x86_64-linux".to_owned(),
cache: None,
} }
} }
@ -514,6 +519,7 @@ mod tests {
FromMetaReply::Built { FromMetaReply::Built {
out_path: format!("{} {}", BUILD_FLAGS.join(" "), b.meta_attr(&meta_job())), out_path: format!("{} {}", BUILD_FLAGS.join(" "), b.meta_attr(&meta_job())),
system: "x86_64-linux".to_owned(), system: "x86_64-linux".to_owned(),
cached: false,
} }
); );
} }

437
swarm-builder/src/push.rs Normal file
View file

@ -0,0 +1,437 @@
//! The push of a built closure to the binary cache, with `attic push`.
//!
//! The cache is an attic instance outside the swarm. Its push token is read
//! from the secret store per push and written to an `attic` config file in a
//! directory only the builder's unit can enter, which `attic` finds through
//! `XDG_CONFIG_HOME`. `attic login <name> <url> <token>` is not used: it puts
//! the token in the process list. `attic` verifies the server's certificate
//! against the system roots.
//!
//! A push that does not happen or does not finish leaves the build green with
//! `cached` false: the closure is still in the host's store, and the next build
//! of the agent pushes again.
use std::future::Future;
use std::path::{Path, PathBuf};
use std::process::Stdio;
use std::time::Duration;
use anyhow::{Result, bail};
use swarm_wire::builder::FromMetaReply;
/// The longest one push may run, when the reply deadline leaves that much.
pub const PUSH_DEADLINE: Duration = Duration::from_mins(10);
/// The server name in the config file, and in the cache argument of the push.
const SERVER: &str = "swarm";
/// Where the builder pushes: one cache on one attic server.
pub struct Remote {
/// The `attic` binary: a name found in the unit's path in production.
pub attic: PathBuf,
/// The server's name, without scheme: `host` or `host:port`.
pub domain: String,
/// The cache's name on that server.
pub cache: String,
/// A directory only the builder can enter, which the config files go in.
pub state_dir: PathBuf,
}
impl Remote {
/// The cache the environment names: `None` when neither value is set, so
/// the builder pushes nothing.
///
/// # Errors
/// When only one of the two is set, or a value would bend the config file
/// or the command.
pub fn from_settings(
domain: Option<String>,
cache: Option<String>,
state_dir: Option<String>,
) -> Result<Option<Self>> {
let (domain, cache) = match (domain, cache) {
(None, None) => return Ok(None),
(Some(domain), Some(cache)) => (domain, cache),
_ => bail!("the cache domain and the cache name must be set together"),
};
let plain = |s: &str| {
!s.is_empty()
&& s.bytes()
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_' | b':'))
};
if !plain(&domain) || !plain(&cache) {
bail!("the cache domain {domain:?} and name {cache:?} must be plain names");
}
let Some(state_dir) = state_dir else {
bail!("a cache is set, but the unit has no runtime directory to hold its config");
};
Ok(Some(Self {
attic: PathBuf::from("attic"),
domain,
cache,
state_dir: PathBuf::from(state_dir),
}))
}
/// Push `out_path` with `token`. The result is what `attic` printed.
///
/// # Errors
/// Why the push did not finish: the config could not be written, `attic`
/// could not start, or it exited with an error.
pub async fn push(&self, token: &str, out_path: &str) -> Result<String, String> {
if !out_path.starts_with("/nix/store/") || out_path.contains(char::is_whitespace) {
return Err(format!("{out_path:?} is not a store path"));
}
let home = ConfigHome::write(&self.state_dir, &self.domain, token)
.map_err(|e| format!("writing the attic config: {e}"))?;
let output = self
.command(&home.0, out_path)
.output()
.await
.map_err(|e| format!("starting attic: {e}"))?;
let said = |bytes: &[u8]| {
crate::nix_build::last_lines(&String::from_utf8_lossy(bytes))
.unwrap_or_default()
.replace(token, "<redacted>")
};
if output.status.success() {
Ok(said(&output.stdout))
} else {
Err(format!(
"attic exited with {}: {}",
output.status,
said(&output.stderr)
))
}
}
/// `attic push` of `out_path`, reading its config from `config_home`. The
/// token is in the file there and in no argument or variable.
fn command(&self, config_home: &Path, out_path: &str) -> tokio::process::Command {
let mut cmd = tokio::process::Command::new(&self.attic);
cmd.arg("push")
.arg(format!("{SERVER}:{}", self.cache))
.arg(out_path)
.env("XDG_CONFIG_HOME", config_home)
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.kill_on_drop(true);
cmd
}
}
/// Numbers the config directories of one process's pushes.
static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
/// The config directory of one push, removed with the token in it on drop.
struct ConfigHome(PathBuf);
impl ConfigHome {
/// `<parent>/push-<n>/attic/config.toml`, readable by this user alone.
fn write(parent: &Path, domain: &str, token: &str) -> std::io::Result<Self> {
use std::io::Write as _;
use std::os::unix::fs::{DirBuilderExt as _, OpenOptionsExt as _};
// A quote or a control character would end the TOML string early.
if token.is_empty() || token.bytes().any(|b| b == b'"' || b == b'\\' || b < b' ') {
return Err(std::io::Error::other(
"the push token is empty or holds a character the config cannot",
));
}
let home = Self(parent.join(format!(
"push-{}-{}",
std::process::id(),
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
)));
let dir = home.0.join("attic");
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o700)
.create(&dir)?;
let mut file = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(dir.join("config.toml"))?;
write!(
file,
"default-server = \"{SERVER}\"\n\n[servers.{SERVER}]\n\
endpoint = \"https://{domain}\"\ntoken = \"{token}\"\n"
)?;
Ok(home)
}
}
impl Drop for ConfigHome {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
/// `reply` with `cached` set by pushing its closure, when it is a built one and
/// a cache is set. `read_token` is not polled without a cache. The token read
/// and the push together get `budget`.
///
/// Whatever happens to the push, a built reply stays built: a failure is logged
/// and leaves `cached` false.
pub async fn cache_built(
reply: FromMetaReply,
cache: Option<&Remote>,
read_token: impl Future<Output = Result<String>>,
budget: Duration,
agent: &str,
) -> FromMetaReply {
let (
Some(cache),
FromMetaReply::Built {
out_path, system, ..
},
) = (cache, &reply)
else {
return reply;
};
let pushed = tokio::time::timeout(budget, async {
let token = read_token
.await
.map_err(|e| format!("reading the push token: {e:#}"))?;
cache.push(&token, out_path).await
})
.await
.unwrap_or_else(|_| {
Err(format!(
"the push did not finish within {} s",
budget.as_secs()
))
});
match pushed {
Ok(_) => {
tracing::info!(%agent, %out_path, cache = %cache.cache, "closure pushed to the cache");
FromMetaReply::Built {
out_path: out_path.clone(),
system: system.clone(),
cached: true,
}
}
Err(reason) => {
tracing::warn!(%agent, %out_path, cache = %cache.cache, %reason, "closure not pushed to the cache");
reply
}
}
}
#[cfg(test)]
mod tests {
use std::os::unix::fs::PermissionsExt as _;
use std::sync::atomic::{AtomicBool, Ordering};
use super::*;
const TOKEN: &str = "eyJ0b2tlbiI6InNlY3JldCJ9.c2lnbmVk";
const OUT: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
fn built() -> FromMetaReply {
FromMetaReply::Built {
out_path: OUT.to_owned(),
system: "x86_64-linux".to_owned(),
cached: false,
}
}
fn cached(reply: &FromMetaReply) -> bool {
match reply {
FromMetaReply::Built { cached, .. } => *cached,
other => panic!("not built: {other:?}"),
}
}
/// A cache whose `attic` is the program `attic`: `true` and `false` stand
/// in for a push that works and one that fails, and `echo` for one that
/// says what it was run with.
fn cache_with(attic: &str) -> Remote {
Remote {
attic: PathBuf::from(attic),
domain: "cache.example.org".to_owned(),
cache: "grid".to_owned(),
state_dir: std::env::temp_dir(),
}
}
fn token() -> impl Future<Output = Result<String>> {
std::future::ready(Ok(TOKEN.to_owned()))
}
#[tokio::test]
async fn a_push_that_works_marks_the_build_cached() {
let cache = cache_with("true");
let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await;
assert!(cached(&reply), "{reply:?}");
}
#[tokio::test]
async fn a_push_that_fails_leaves_the_build_green_and_not_cached() {
let cache = cache_with("false");
let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await;
assert_eq!(reply, built());
}
#[tokio::test]
async fn a_push_that_cannot_start_leaves_the_build_green_and_not_cached() {
let cache = cache_with("/nonexistent/attic");
let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await;
assert_eq!(reply, built());
}
#[tokio::test]
async fn an_unreadable_token_leaves_the_build_green_and_not_cached() {
let cache = cache_with("true");
let reply = cache_built(
built(),
Some(&cache),
async { anyhow::bail!("nothing at the path") },
PUSH_DEADLINE,
"atlas",
)
.await;
assert_eq!(reply, built());
}
#[tokio::test(start_paused = true)]
async fn a_token_read_that_outlasts_the_budget_leaves_the_build_green() {
let cache = cache_with("true");
let started = tokio::time::Instant::now();
let reply = cache_built(
built(),
Some(&cache),
std::future::pending(),
Duration::from_secs(5),
"atlas",
)
.await;
assert_eq!(reply, built());
assert_eq!(started.elapsed(), Duration::from_secs(5));
}
#[tokio::test]
async fn without_a_cache_nothing_is_read_or_pushed() {
let polled = std::sync::Arc::new(AtomicBool::new(false));
let tracked = |polled: &std::sync::Arc<AtomicBool>| {
let polled = std::sync::Arc::clone(polled);
async move {
polled.store(true, Ordering::SeqCst);
token().await
}
};
let reply = cache_built(built(), None, tracked(&polled), PUSH_DEADLINE, "atlas").await;
assert_eq!(reply, built());
assert!(!polled.load(Ordering::SeqCst), "the token was read");
// Control: with a cache the same future runs.
let cache = cache_with("true");
cache_built(
built(),
Some(&cache),
tracked(&polled),
PUSH_DEADLINE,
"atlas",
)
.await;
assert!(polled.load(Ordering::SeqCst));
}
#[tokio::test]
async fn a_build_that_is_not_built_is_not_pushed() {
let cache = cache_with("true");
let failed = FromMetaReply::Failed {
error_tail: "e".to_owned(),
};
let reply = cache_built(
failed.clone(),
Some(&cache),
token(),
PUSH_DEADLINE,
"atlas",
)
.await;
assert_eq!(reply, failed);
}
#[tokio::test]
async fn the_push_command_has_no_token_in_it() {
// `echo` prints the arguments it is given.
let cache = cache_with("echo");
let said = cache.push(TOKEN, OUT).await.expect("echo succeeds");
assert_eq!(said, format!("push swarm:grid {OUT}"));
assert!(!said.contains(TOKEN));
let cmd = cache.command(Path::new("/run/home"), OUT);
let cmd = cmd.as_std();
let args: Vec<_> = cmd.get_args().collect();
assert_eq!(args, ["push", "swarm:grid", OUT]);
let env: Vec<_> = cmd.get_envs().collect();
assert_eq!(
env,
[(
std::ffi::OsStr::new("XDG_CONFIG_HOME"),
Some(std::ffi::OsStr::new("/run/home"))
)]
);
}
#[test]
fn the_token_is_in_a_private_config_file_that_is_removed_after() {
let parent = std::env::temp_dir().join(format!("push-test-{}", std::process::id()));
std::fs::create_dir_all(&parent).expect("parent");
let home = ConfigHome::write(&parent, "cache.example.org", TOKEN).expect("written");
let file = home.0.join("attic/config.toml");
let text = std::fs::read_to_string(&file).expect("reads");
assert!(text.contains(&format!("token = \"{TOKEN}\"")), "{text}");
assert!(
text.contains("endpoint = \"https://cache.example.org\""),
"{text}"
);
let mode = |p: &Path| std::fs::metadata(p).expect("stat").permissions().mode() & 0o777;
assert_eq!(mode(&file), 0o600);
assert_eq!(mode(&home.0), 0o700);
let root = home.0.clone();
drop(home);
assert!(!root.exists());
std::fs::remove_dir_all(&parent).expect("cleanup");
}
#[test]
fn a_token_that_would_end_the_config_string_is_refused() {
let parent = std::env::temp_dir();
for bad in ["", "a\"b", "a\\b", "a\nb"] {
assert!(ConfigHome::write(&parent, "h", bad).is_err(), "{bad:?}");
}
}
#[tokio::test]
async fn a_path_that_is_not_a_store_path_is_not_pushed() {
let cache = cache_with("true");
assert!(cache.push(TOKEN, "--help").await.is_err());
assert!(cache.push(TOKEN, "/nix/store/x y").await.is_err());
assert!(cache.push(TOKEN, OUT).await.is_ok());
}
#[test]
fn the_settings_name_a_cache_only_when_both_are_set() {
let state = || Some("/run/swarm-builder".to_owned());
let some = |s: &str| Some(s.to_owned());
assert!(
Remote::from_settings(None, None, None)
.expect("ok")
.is_none()
);
let cache = Remote::from_settings(some("cache.example.org"), some("grid"), state())
.expect("ok")
.expect("set");
assert_eq!(cache.domain, "cache.example.org");
assert_eq!(cache.cache, "grid");
assert!(Remote::from_settings(some("cache.example.org"), None, state()).is_err());
assert!(Remote::from_settings(None, some("grid"), state()).is_err());
assert!(Remote::from_settings(some("https://cache"), some("grid"), state()).is_err());
assert!(Remote::from_settings(some("cache"), some("a\"b"), state()).is_err());
assert!(Remote::from_settings(some("cache"), some("grid"), None).is_err());
}
}

View file

@ -306,13 +306,18 @@ pub(crate) async fn run(
} }
Outcome::Failed(error_tail) Outcome::Failed(error_tail)
} }
Ok(FromMetaReply::Built { out_path, system }) => { Ok(FromMetaReply::Built {
out_path,
system,
cached,
}) => {
let built_rev = meta_rev.clone(); let built_rev = meta_rev.clone();
let closure = AgentClosure { let closure = AgentClosure {
meta_rev, meta_rev,
store_path: out_path.clone(), store_path: out_path.clone(),
system, system,
built_unix: unix_now(), built_unix: unix_now(),
cached,
}; };
match queue.record(&agent, &closure).await { match queue.record(&agent, &closure).await {
Ok(()) => { Ok(()) => {
@ -415,9 +420,14 @@ mod tests {
} }
fn built() -> Self { fn built() -> Self {
Self::built_cached(false)
}
fn built_cached(cached: bool) -> Self {
Self::answering(Ok(FromMetaReply::Built { Self::answering(Ok(FromMetaReply::Built {
out_path: "/nix/store/x-toplevel".to_owned(), out_path: "/nix/store/x-toplevel".to_owned(),
system: "x86_64-linux".to_owned(), system: "x86_64-linux".to_owned(),
cached,
})) }))
} }
} }
@ -470,6 +480,22 @@ mod tests {
assert_eq!(closure.store_path, "/nix/store/x-toplevel"); assert_eq!(closure.store_path, "/nix/store/x-toplevel");
assert_eq!(closure.system, "x86_64-linux"); assert_eq!(closure.system, "x86_64-linux");
assert!(closure.built_unix > 0); assert!(closure.built_unix > 0);
assert!(!closure.cached);
}
/// The record says what the builder said about the cache, both ways.
#[tokio::test]
async fn the_record_carries_whether_the_builder_cached_the_closure() {
for cached in [true, false] {
let queue = FakeQueue::built_cached(cached);
let outcome = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await;
assert_eq!(outcome, Outcome::Done);
let recorded = queue.recorded.lock().unwrap();
let [(_, closure)] = recorded.as_slice() else {
panic!("one record: {recorded:?}")
};
assert_eq!(closure.cached, cached);
}
} }
/// Any `nix` failure, a timeout included, fails the node at once with what /// Any `nix` failure, a timeout included, fails the node at once with what
@ -749,6 +775,7 @@ mod tests {
store_path: "/nix/store/a".to_owned(), store_path: "/nix/store/a".to_owned(),
system: "x86_64-linux".to_owned(), system: "x86_64-linux".to_owned(),
built_unix: 1, built_unix: 1,
cached: false,
}; };
client.record("atlas", &closure).await.expect("written"); client.record("atlas", &closure).await.expect("written");
closure.store_path = "/nix/store/b".to_owned(); closure.store_path = "/nix/store/b".to_owned();

View file

@ -2,8 +2,9 @@
//! secret live in the store, and what each object holds. //! secret live in the store, and what each object holds.
//! //!
//! swarm-controller mints the forge token; the builder reads it, and its queue //! swarm-controller mints the forge token; the builder reads it, and its queue
//! client secret, under its own certificate. Its read stanza in //! client secret, under its own certificate. It also reads the cache push
//! `swarm-bao.nix` names exactly these two paths. //! token, which the operator writes. Its read stanza in `swarm-bao.nix` names
//! exactly these three paths.
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
@ -26,6 +27,30 @@ pub fn forge_token_path() -> Result<String, Error> {
Ok(format!("{prefix}/forge/token")) Ok(format!("{prefix}/forge/token"))
} }
/// The path holding the token the builder pushes closures to the binary cache
/// with, as an [`AtticPushToken`]. The operator writes it by hand: the cache is
/// an instance outside the swarm, and no swarm unit mints the token.
///
/// Under the cache's own name segment rather than the builder's, since the
/// token belongs to the cache's tenant.
///
/// # Errors
/// Never in practice: the name segment is a constant. The `Result` is
/// [`principal_prefix`]'s.
pub fn attic_push_token_path() -> Result<String, Error> {
let prefix = principal_prefix(Kind::Service, "attic")?;
Ok(format!("{prefix}/push-token"))
}
/// What [`attic_push_token_path`] holds.
///
/// No `Debug`: `value` writes to the binary cache.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct AtticPushToken {
/// The token, under the field name the operator writes.
pub value: String,
}
/// The path holding the queue client secret of the builder's OIDC client /// The path holding the queue client secret of the builder's OIDC client
/// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's. /// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's.
/// ///
@ -60,5 +85,9 @@ mod tests {
client_secret_path("swarm-builder").expect("a plain id is legal"), client_secret_path("swarm-builder").expect("a plain id is legal"),
"swarm/services/swarm-builder/oidc/client" "swarm/services/swarm-builder/oidc/client"
); );
assert_eq!(
attic_push_token_path().expect("a constant segment is legal"),
"swarm/services/attic/push-token"
);
} }
} }

View file

@ -32,6 +32,10 @@ pub struct AgentClosure {
pub system: String, pub system: String,
/// When the controller recorded the build, in unix seconds. /// When the controller recorded the build, in unix seconds.
pub built_unix: i64, pub built_unix: i64,
/// Whether the builder pushed the closure to the binary cache. A record
/// written before the field existed reads as false.
#[serde(default)]
pub cached: bool,
} }
#[cfg(test)] #[cfg(test)]
@ -51,12 +55,14 @@ mod tests {
store_path: "/nix/store/x".to_owned(), store_path: "/nix/store/x".to_owned(),
system: "x86_64-linux".to_owned(), system: "x86_64-linux".to_owned(),
built_unix: 7, built_unix: 7,
cached: true,
}; };
let wire = serde_json::json!({ let wire = serde_json::json!({
"meta_rev": "abc", "meta_rev": "abc",
"store_path": "/nix/store/x", "store_path": "/nix/store/x",
"system": "x86_64-linux", "system": "x86_64-linux",
"built_unix": 7, "built_unix": 7,
"cached": true,
}); });
assert_eq!(serde_json::to_value(&value).expect("encodes"), wire); assert_eq!(serde_json::to_value(&value).expect("encodes"), wire);
assert_eq!( assert_eq!(
@ -64,4 +70,16 @@ mod tests {
value value
); );
} }
#[test]
fn a_record_without_the_field_reads_as_not_cached() {
let old = serde_json::json!({
"meta_rev": "abc",
"store_path": "/nix/store/x",
"system": "x86_64-linux",
"built_unix": 7,
});
let record = serde_json::from_value::<AgentClosure>(old).expect("decodes");
assert!(!record.cached);
}
} }

View file

@ -140,8 +140,16 @@ pub struct BuildFromMeta {
#[serde(tag = "state", rename_all = "kebab-case")] #[serde(tag = "state", rename_all = "kebab-case")]
pub enum FromMetaReply { pub enum FromMetaReply {
/// The toplevel built; `out_path` is its store path in the builder host's /// The toplevel built; `out_path` is its store path in the builder host's
/// store and `system` the platform the builder host builds for. /// store and `system` the platform the builder host builds for. `cached`
Built { out_path: String, system: String }, /// is whether the builder pushed the closure to the binary cache; a failed
/// push leaves the build green with `cached` false. A builder that predates
/// the field answers without it, which reads as false.
Built {
out_path: String,
system: String,
#[serde(default)]
cached: bool,
},
/// The build did not succeed: the last lines `nix` printed, or why it /// The build did not succeed: the last lines `nix` printed, or why it
/// stopped (it timed out, or could not start). The controller holds the /// stopped (it timed out, or could not start). The controller holds the
/// deploy back and builds again on its next round. /// deploy back and builds again on its next round.
@ -253,8 +261,14 @@ mod tests {
FromMetaReply::Built { FromMetaReply::Built {
out_path: "/nix/store/x".to_owned(), out_path: "/nix/store/x".to_owned(),
system: "x86_64-linux".to_owned(), system: "x86_64-linux".to_owned(),
cached: true,
}, },
json!({"state": "built", "out_path": "/nix/store/x", "system": "x86_64-linux"}), json!({
"state": "built",
"out_path": "/nix/store/x",
"system": "x86_64-linux",
"cached": true,
}),
), ),
( (
FromMetaReply::Failed { FromMetaReply::Failed {
@ -277,6 +291,19 @@ mod tests {
} }
} }
#[test]
fn a_built_reply_from_a_builder_without_the_field_reads_as_not_cached() {
let old = json!({"state": "built", "out_path": "/nix/store/x", "system": "s"});
assert_eq!(
serde_json::from_value::<FromMetaReply>(old).expect("decodes"),
FromMetaReply::Built {
out_path: "/nix/store/x".to_owned(),
system: "s".to_owned(),
cached: false,
}
);
}
#[test] #[test]
fn the_subjects_are_pinned() { fn the_subjects_are_pinned() {
assert_eq!(SUBJECT, "$SWARM.builder.build"); assert_eq!(SUBJECT, "$SWARM.builder.build");