diff --git a/docs/swarm/builder.md b/docs/swarm/builder.md index 98ec002c..87e4da60 100644 --- a/docs/swarm/builder.md +++ b/docs/swarm/builder.md @@ -103,9 +103,10 @@ status has no link. hive's. A green check can therefore still fail on a hive, and the reverse. - The agent base is the builder's host's, not the hive's. Hives on another hyperhive or nixpkgs pin build something else. -- The builder pushes nothing. The closure stays in the builder +- The check pushes nothing. The closure stays in the builder host's store until it's collected, so on a host that also runs the hive, the - hive's own build can reuse whatever matches. + hive's own build can reuse whatever matches. A [meta build](#deploy-gate) + can push to [a binary cache](#binary-cache). - The PR check gates nothing. A merge deploys whatever the status says. The [deploy gate](#deploy-gate) is a separate build, from `internal/meta`. - A lost webhook delivery isn't retried: the head isn't built until the next @@ -247,13 +248,48 @@ On a built reply the controller writes the agent's key in the `agent-closures` KV bucket: the `internal/meta` commit, the closure's store path, the builder's platform and the time. Only the controller writes the bucket, and the builder holds no grant on it. A failed build writes nothing and leaves the previous key. -Nothing reads the bucket yet. The closure stays unrooted in the builder host's -store. +The record's `cached` field says whether the builder pushed the closure to +[the binary cache](#binary-cache). Nothing reads the bucket yet. The closure stays +unrooted in the builder host's store. The controller gives the `builder` forge account read on `internal/meta` when it updates the flake, and again before each build, since the repo is private. A grant the forge refuses fails the build with no verdict. +## Binary cache + +With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes +each green meta build's closure to that cache, an attic server outside the swarm, +before it replies. It runs `attic push swarm: ` once per build, +not `attic watch`, and the builder pushes again when the controller runs a build again. The +push has ten minutes, or less when the reply is due sooner, because it must end +inside the 61 minutes the controller waits. `attic` checks the server's certificate +against the system roots and pins nothing, so a Let's Encrypt certificate +works. + +The push token is the operator's to write. The builder reads it on every push +from `secret/swarm/services/attic/push-token` (field `value`), writes an `attic` +config file holding it in the unit's private runtime directory (`/run/swarm-builder`, +mode `0700`, the file `0600`), points `attic` at it with `XDG_CONFIG_HOME` and +removes it after the push. The token is in no argument or variable, so no +process listing shows it. + +The reply's `cached` says whether the push finished. A push that fails, times out +or can't read the token leaves the build green with `cached` false and logs a +warning with the reason; the deploy goes ahead and the next build of the agent +tries again. With no cache set, the builder pushes nothing and `cached` is always +false. + +Mixed versions: + +| controller | builder | what happens | +| ---------- | ------- | ------------------------------------------------------------- | +| new | new | `cached` in the reply and in the `agent-closures` record | +| new | old | the reply has no `cached`, which reads as `false` | +| old | new | the controller ignores `cached`; its record has no such field | + +A record written before the field existed also reads as `false`. + ## Credentials | What | Where | Who reads it | @@ -261,6 +297,7 @@ grant the forge refuses fails the build with no verdict. | store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential | | queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it | | forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build | +| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push | The token belongs to the `builder` forge account, which swarm-controller creates, puts in the `hives` team of `agent-configs` (read on every config @@ -270,7 +307,7 @@ gets the token through a credential helper reading the child's environment: it's in no argument, URL or lock, so no line `nix` prints can carry it into a status. The same `git` fetches a config's own inputs, so the builder registers the helper for the forge's URL only, and the helper answers only an `https` -request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the +request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the status with its own forge token. Details per path: [`credentials-reference.md`](credentials-reference.md). diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 93349050..282250ab 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -41,6 +41,7 @@ the rest of the cell says how. | `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review | | `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build | +| `swarm/services/attic/push-token` | you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass | | _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated | @@ -158,7 +159,7 @@ login minter's, the forge host's and Grafana's, which reach more than one: | `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services//oidc/client` | | `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services//oidc/client` | | `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services//oidc/client` | -| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services//oidc/client` | +| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services//oidc/client` | | `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services//oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` and `swarm/services//oidc/client` | diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index cd5d6363..2adcf238 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -62,6 +62,7 @@ ./glue-swarm-otel-oidc-client.nix ./glue-victorialogs-otel-oidc-client.nix ./glue-victoriametrics-otel-oidc-client.nix + ./swarm-attic-options.nix ./swarm-authelia-service.nix ./swarm-authelia.nix ./swarm-bao-service.nix diff --git a/nix/host-modules/swarm-attic-options.nix b/nix/host-modules/swarm-attic-options.nix new file mode 100644 index 00000000..481ffa0d --- /dev/null +++ b/nix/host-modules/swarm-attic-options.nix @@ -0,0 +1,37 @@ +# The binary cache the swarm pushes agent closures to, as every host sees it: +# which attic server, and which cache on it. The swarm runs no attic of its +# own yet; the instance is the operator's, outside the swarm. +{ lib, ... }: +{ + options.services.hyperhive.swarm.attic = { + domain = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "cache.example.org"; + description = '' + Name of the attic server, with no scheme: the swarm builder pushes to + `https://`, and trusts the certificate the system roots do, so + a Let's Encrypt certificate works and none is pinned. Set together with + {option}`services.hyperhive.swarm.attic.cache`. + + Null pushes nothing: the swarm builder does not push, and the closure + record says it is not cached. + + The push token is not an option. The operator writes it to + `secret/swarm/services/attic/push-token` in the swarm secret store + (field `value`), and the builder reads it on every push. + ''; + }; + + cache = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "grid"; + description = '' + Name of the cache on {option}`services.hyperhive.swarm.attic.domain` + that the swarm builder pushes closures to. The push token must be + allowed to push to it. + ''; + }; + }; +} diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index c27b4ac1..5531c8bd 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -692,6 +692,12 @@ let # to fetch config repos. builderForgeTokenLeaf = "swarm/services/builder/forge/token"; + # The token the swarm builder pushes closures to the binary cache with, the nix + # half of `swarm_secret_client::builder::attic_push_token_path`. The operator + # writes it by hand; the builder reads it on every push, and nothing else in + # the swarm reaches it. + atticPushTokenLeaf = "swarm/services/attic/push-token"; + # The CI run poller's forge token, the nix half of # `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it # and reads it back before deciding whether to re-mint; the poller reads it @@ -1146,14 +1152,16 @@ let } ]; - # The swarm builder: its forge token, read once per build, and its own queue - # client secret, read at start. Both by the daemon inside its container. + # The swarm builder: its forge token, read once per build, the cache push token, + # read once per push, and its own queue client secret, read at start. All by the + # daemon inside its container. builderReaders = [ { name = "swarm-builder"; cn = baoDeploy.builderCommonName; policyText = readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}" + + readStanza "${credentialMountPath}/data/${atticPushTokenLeaf}" + readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client"; } ]; @@ -2081,8 +2089,8 @@ in description = '' Subject the store's `swarm-builder` cert-auth role accepts: the identity the swarm builder presents from inside its container. Its - grant is two paths, read only: `swarm/services/builder/forge/token` - and its own queue client secret. + grant is three paths, read only: `swarm/services/builder/forge/token`, + `swarm/services/attic/push-token` and its own queue client secret. ⚠️ Reserved as a hive name by ./swarm.nix, like its siblings. ''; diff --git a/nix/host-modules/swarm-builder.nix b/nix/host-modules/swarm-builder.nix index d8ddf1ac..49ff166a 100644 --- a/nix/host-modules/swarm-builder.nix +++ b/nix/host-modules/swarm-builder.nix @@ -17,8 +17,8 @@ # # The container holds one credential, its own store leaf, which nspawn hands in # as a system credential with no copy on disk. The daemon reads everything else -# from the store with it: its queue client secret at start and its forge token -# on every build. +# from the store with it: its queue client secret at start, its forge token +# on every build and, where a cache is set, its cache push token on every push. { pkgs, lib, @@ -33,6 +33,7 @@ let natsCfg = hyperhiveCfg.swarm.nats; autheliaCfg = hyperhiveCfg.swarm.authelia; forgeCfg = hyperhiveCfg.swarm.forge; + atticCfg = hyperhiveCfg.swarm.attic; gatewayCfg = hyperhiveCfg.gateway; networkCfg = hyperhiveCfg.network; tlsCfg = hyperhiveCfg.deploy.hive-controller.tls; @@ -89,6 +90,9 @@ let hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake; nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake; + # Either half alone reaches the daemon too, which refuses to start on it. + pushesToCache = atticCfg.domain != null || atticCfg.cache != null; + privateNetwork = true; in { @@ -230,8 +234,9 @@ in wantedBy = [ "multi-user.target" ]; wants = [ "network-online.target" ]; after = [ "network-online.target" ]; - # nix fetches the config repo with `git`. - path = [ pkgs.git ]; + # nix fetches the config repo with `git`, and a green build is pushed + # with `attic` where a cache is set. + path = [ pkgs.git ] ++ lib.optional pushesToCache pkgs.attic-client; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; BAO_CLIENT_CERT = credential "builder-bao-cert"; @@ -252,6 +257,12 @@ in SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}"; SWARM_BUILDER_CONFIG_ORG = "agent-configs"; } + // lib.optionalAttrs (atticCfg.domain != null) { + SWARM_BUILDER_ATTIC_DOMAIN = atticCfg.domain; + } + // lib.optionalAttrs (atticCfg.cache != null) { + SWARM_BUILDER_ATTIC_CACHE = atticCfg.cache; + } // lib.optionalAttrs (storeFiles ? builder-bao-server-ca) { BAO_CACERT = credential "builder-bao-server-ca"; } @@ -264,6 +275,10 @@ in ExecStart = "${cfg.package}/bin/swarm-builder"; DynamicUser = true; CacheDirectory = unit; + # /run/swarm-builder, mode 0700 and owned by the dynamic user: the + # push writes its token-bearing `attic` config under it. + RuntimeDirectory = unit; + RuntimeDirectoryMode = "0700"; # Bare ids: the system credentials nspawn handed in. LoadCredential = lib.attrNames storeFiles; Restart = "on-failure"; diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 1c3ad40b..2c9b74cb 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -1743,17 +1743,18 @@ let ]; } { - # The builder reads its forge token and its own queue client secret, - # each read-only and spelled to the leaf: `services/*` would reach every - # swarm service's secret. - name = "the swarm builder's grant is its forge token and its queue client secret, and nothing else"; + # The builder reads its forge token, the cache push token and its own queue + # client secret, each read-only and spelled to the leaf: `services/*` would + # reach every swarm service's secret. + name = "the swarm builder's grant is its forge token, the cache push token and its queue client secret, and nothing else"; ok = let s = baoGrantHere.systemd.services.swarm-bao-builder-policy.script; in lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"read\"]\n}" s + && lib.hasInfix "path \"secret/data/swarm/services/attic/push-token\" {\n capabilities = [\"read\"]\n}" s && lib.hasInfix "path \"secret/data/swarm/services/swarm-builder/oidc/client\" {\n capabilities = [\"read\"]\n}" s - && lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2 + && lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 3 && !(lib.hasInfix "secret/data/swarm/services/*" s) && lib.hasInfix "auth/cert/certs/swarm-builder" s && lib.hasInfix "allowed_common_names=swarm-builder" s diff --git a/nix/module-eval/builder.nix b/nix/module-eval/builder.nix index 4cfcb075..0ec09f6d 100644 --- a/nix/module-eval/builder.nix +++ b/nix/module-eval/builder.nix @@ -29,6 +29,8 @@ let withBuilder = hive { deploy.bao.enable = true; deploy.builder.enable = true; + swarm.attic.domain = "cache.example.org"; + swarm.attic.cache = "grid"; }; servicesHere = shared.swarmServicesWithController; @@ -48,6 +50,8 @@ let inner = container.config; svc = inner.systemd.services.swarm-builder; env = svc.environment; + hasOnPath = name: s: lib.any (p: lib.getName p == name) s.path; + withoutCache = servicesHere.containers.swarm-builder.config.systemd.services.swarm-builder; # The check flake on the inputs the builder's unit names, called as # `nix build` calls it. `hyperhive` is entered through the filtered source's @@ -116,6 +120,25 @@ let && svc.serviceConfig.DynamicUser && !(lib.any (lib.hasSuffix "TOKEN") (lib.attrNames env)); } + { + # The closure is pushed with `attic` from the unit, whose config file holds + # the push token: it goes in a runtime directory only the dynamic user can + # enter, and the token is in no variable. The host without a cache is the + # control: same unit, none of it, with `git` still on the path to show + # the lookup finds what is there. + name = "the builder pushes to the configured cache with attic from a private runtime directory, and pushes nowhere without one"; + ok = + env.SWARM_BUILDER_ATTIC_DOMAIN == "cache.example.org" + && env.SWARM_BUILDER_ATTIC_CACHE == "grid" + && hasOnPath "attic-client" svc + && svc.serviceConfig.RuntimeDirectory == "swarm-builder" + && svc.serviceConfig.RuntimeDirectoryMode == "0700" + && !(lib.any (lib.hasInfix "TOKEN") (lib.attrNames env)) + && hasOnPath "git" withoutCache + && !(hasOnPath "attic-client" withoutCache) + && !(withoutCache.environment ? SWARM_BUILDER_ATTIC_DOMAIN) + && !(withoutCache.environment ? SWARM_BUILDER_ATTIC_CACHE); + } { # The minting unit goes inactive after each run, so a start that pulls it # runs it again and mints a leaf that is missing; an active diff --git a/swarm-builder/README.md b/swarm-builder/README.md index 97994501..8a3864be 100644 --- a/swarm-builder/README.md +++ b/swarm-builder/README.md @@ -19,7 +19,7 @@ the answer into a `swarm-builder` commit status on the PR's head. as an untrusted client, so a config can neither set daemon options nor read this service's environment. - Nothing is pushed and nothing is recorded: the closure stays in the host's - store until it is collected. + store until it is collected. (A meta build can push, see below.) ## Meta builds @@ -31,16 +31,21 @@ the platform it builds for. The controller records the answer; the service writes nothing. It takes the same one-build slot as the PR checks. See [`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate). +Where `services.hyperhive.swarm.attic.{domain,cache}` are set, it first pushes a +green build's closure to that cache with `attic push`, and the answer says +whether that worked. A failed push leaves the build green. See +[`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache). + ## Credentials The container holds one credential: its own store certificate, which nspawn hands in at start without a copy on disk. With it the service reads its queue client secret at start and again when authelia refuses it, and its -read-only forge token at every build. +read-only forge token at every build, and the cache push token at every push. `git` gets the token through a credential helper reading the child's environment, never in a URL or an argument, and the helper answers only an `https` request for the forge's host. Its store role reads exactly -those two paths. +those three paths. ## Archetypes diff --git a/swarm-builder/src/main.rs b/swarm-builder/src/main.rs index 4d274b65..c8f1470e 100644 --- a/swarm-builder/src/main.rs +++ b/swarm-builder/src/main.rs @@ -16,6 +16,8 @@ //! and an `internal/meta` commit, and the answer is the store path of that //! agent's toplevel, built at the flake's own lock. They take the build slot //! PR checks take. The controller records the path; the builder writes nothing. +//! When a cache is set (see [`push`]), a green meta build is pushed to it before +//! the reply, and the reply says whether that worked. //! //! The store identity comes from the `BAO_*` environment. The queue client //! secret is read at start and again when authelia refuses it, the forge token @@ -39,6 +41,7 @@ use swarm_wire::builder::{ mod archetypes; mod nix_build; +mod push; /// The cert-auth role this service logs in to the store as. `swarm-bao.nix` /// writes it, matching the leaf's subject. @@ -64,6 +67,17 @@ const MAX_SLOT_WAIT: Duration = REPLY_DEADLINE .saturating_sub(BUILD_DEADLINE) .saturating_sub(SLOT_MARGIN); +/// What is held back from a meta build's reply deadline for publishing the reply +/// after its push. +const REPLY_MARGIN: Duration = Duration::from_secs(10); + +/// How long the push of a meta build that ends at now may run: at most +/// [`push::PUSH_DEADLINE`], and never past `reply_by`, so a push cannot make the +/// reply later than the controller waits. +fn push_budget(reply_by: tokio::time::Instant) -> Duration { + push::PUSH_DEADLINE.min(reply_by.saturating_duration_since(tokio::time::Instant::now())) +} + #[tokio::main] async fn main() -> Result<()> { hive_log::init(); @@ -83,6 +97,7 @@ async fn main() -> Result<()> { deadline: BUILD_DEADLINE, lock_deadline: LOCK_DEADLINE, system: var("SYSTEM")?, + cache: push_target()?, }); let secret = swarm_secret_client::read_with_startup_retry("reading the queue client secret", || { @@ -300,7 +315,7 @@ async fn answer_meta_builds( &slot, received, || busy_meta(&job), - handle_meta(&job, &builder), + handle_meta(&job, &builder, received + REPLY_DEADLINE - REPLY_MARGIN), ) .await else { @@ -383,19 +398,36 @@ fn busy_meta(job: &BuildFromMeta) -> FromMetaReply { } /// Build one meta request and log the build. -async fn handle_meta(job: &BuildFromMeta, builder: &nix_build::Builder) -> FromMetaReply { +async fn handle_meta( + job: &BuildFromMeta, + builder: &nix_build::Builder, + reply_by: tokio::time::Instant, +) -> FromMetaReply { let reply = match within(TOKEN_DEADLINE, forge_token()).await { Ok(token) => builder.build_from_meta(job, &token).await, Err(e) => FromMetaReply::Unavailable { reason: format!("{e:#}"), }, }; + let reply = push::cache_built( + reply, + builder.cache.as_ref(), + attic_push_token(), + push_budget(reply_by), + &job.agent, + ) + .await; match &reply { - FromMetaReply::Built { out_path, system } => tracing::info!( + FromMetaReply::Built { + out_path, + system, + cached, + } => tracing::info!( agent = %job.agent, meta_rev = %job.meta_rev, %out_path, %system, + cached, "agent meta build built" ), FromMetaReply::Failed { error_tail } => tracing::warn!( @@ -458,6 +490,23 @@ fn log_build(job: &BuildAgent, reply: &Reply) { } } +/// The token the builder pushes closures to the cache with, read on every push so +/// a rotation takes effect on the next one. +async fn attic_push_token() -> Result { + let path = swarm_secret_client::builder::attic_push_token_path()?; + let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE) + .await + .context("logging in to the secret store")?; + let stored: Option = store + .read_optional(&path) + .await + .with_context(|| format!("reading {path}"))?; + match stored { + Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()), + _ => bail!("nothing at {path}: the cache push token has not been written"), + } +} + /// The forge token the builder fetches config repos with, read on every /// build so a rotation takes effect on the next one. async fn forge_token() -> Result { @@ -495,6 +544,17 @@ async fn client_secret(client_id: &str) -> Result { } } +/// The cache the unit's environment names, if any. systemd sets +/// `RUNTIME_DIRECTORY` for the unit's runtime directory. +fn push_target() -> Result> { + let var = |name: &str| std::env::var(name).ok(); + push::Remote::from_settings( + var(&format!("{PREFIX}_ATTIC_DOMAIN")), + var(&format!("{PREFIX}_ATTIC_CACHE")), + var("RUNTIME_DIRECTORY"), + ) +} + /// The forge's base URL, which must be `https`: the credential helper offers /// the token to `https` requests for its host alone. fn forge_url() -> Result { @@ -580,6 +640,7 @@ mod tests { FromMetaReply::Built { out_path: "/nix/store/x".to_owned(), system: "s".to_owned(), + cached: false, } }) .await; @@ -594,6 +655,7 @@ mod tests { FromMetaReply::Built { out_path: "/nix/store/x".to_owned(), system: "s".to_owned(), + cached: false, } }) .await; @@ -616,6 +678,20 @@ mod tests { assert_eq!(prompt.expect("read"), "t"); } + #[tokio::test(start_paused = true)] + async fn a_push_gets_the_deadline_unless_the_reply_is_due_sooner() { + let now = Instant::now(); + assert_eq!( + push_budget(now + Duration::from_hours(1)), + push::PUSH_DEADLINE + ); + assert_eq!( + push_budget(now + Duration::from_secs(90)), + Duration::from_secs(90) + ); + assert_eq!(push_budget(now), Duration::ZERO); + } + #[test] fn a_build_started_at_the_slot_limit_ends_inside_the_reply_deadline() { assert!(MAX_SLOT_WAIT > Duration::ZERO); diff --git a/swarm-builder/src/nix_build.rs b/swarm-builder/src/nix_build.rs index 2e0c1da0..e20124e7 100644 --- a/swarm-builder/src/nix_build.rs +++ b/swarm-builder/src/nix_build.rs @@ -59,6 +59,9 @@ pub struct Builder { pub lock_deadline: Duration, /// The platform this host builds for, reported with each meta build. pub system: String, + /// Where a meta build's closure is pushed after it succeeds; `None` pushes + /// nothing. + pub cache: Option, } impl Builder { @@ -88,6 +91,7 @@ impl Builder { Reply::Built { out_path } => FromMetaReply::Built { out_path, system: self.system.clone(), + cached: false, }, Reply::Failed { error_tail } => FromMetaReply::Failed { error_tail }, }, @@ -437,6 +441,7 @@ mod tests { deadline: BUILD_DEADLINE, lock_deadline: LOCK_DEADLINE, system: "x86_64-linux".to_owned(), + cache: None, } } @@ -514,6 +519,7 @@ mod tests { FromMetaReply::Built { out_path: format!("{} {}", BUILD_FLAGS.join(" "), b.meta_attr(&meta_job())), system: "x86_64-linux".to_owned(), + cached: false, } ); } diff --git a/swarm-builder/src/push.rs b/swarm-builder/src/push.rs new file mode 100644 index 00000000..30493f7a --- /dev/null +++ b/swarm-builder/src/push.rs @@ -0,0 +1,437 @@ +//! The push of a built closure to the binary cache, with `attic push`. +//! +//! The cache is an attic instance outside the swarm. Its push token is read +//! from the secret store per push and written to an `attic` config file in a +//! directory only the builder's unit can enter, which `attic` finds through +//! `XDG_CONFIG_HOME`. `attic login ` is not used: it puts +//! the token in the process list. `attic` verifies the server's certificate +//! against the system roots. +//! +//! A push that does not happen or does not finish leaves the build green with +//! `cached` false: the closure is still in the host's store, and the next build +//! of the agent pushes again. + +use std::future::Future; +use std::path::{Path, PathBuf}; +use std::process::Stdio; +use std::time::Duration; + +use anyhow::{Result, bail}; +use swarm_wire::builder::FromMetaReply; + +/// The longest one push may run, when the reply deadline leaves that much. +pub const PUSH_DEADLINE: Duration = Duration::from_mins(10); + +/// The server name in the config file, and in the cache argument of the push. +const SERVER: &str = "swarm"; + +/// Where the builder pushes: one cache on one attic server. +pub struct Remote { + /// The `attic` binary: a name found in the unit's path in production. + pub attic: PathBuf, + /// The server's name, without scheme: `host` or `host:port`. + pub domain: String, + /// The cache's name on that server. + pub cache: String, + /// A directory only the builder can enter, which the config files go in. + pub state_dir: PathBuf, +} + +impl Remote { + /// The cache the environment names: `None` when neither value is set, so + /// the builder pushes nothing. + /// + /// # Errors + /// When only one of the two is set, or a value would bend the config file + /// or the command. + pub fn from_settings( + domain: Option, + cache: Option, + state_dir: Option, + ) -> Result> { + let (domain, cache) = match (domain, cache) { + (None, None) => return Ok(None), + (Some(domain), Some(cache)) => (domain, cache), + _ => bail!("the cache domain and the cache name must be set together"), + }; + let plain = |s: &str| { + !s.is_empty() + && s.bytes() + .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_' | b':')) + }; + if !plain(&domain) || !plain(&cache) { + bail!("the cache domain {domain:?} and name {cache:?} must be plain names"); + } + let Some(state_dir) = state_dir else { + bail!("a cache is set, but the unit has no runtime directory to hold its config"); + }; + Ok(Some(Self { + attic: PathBuf::from("attic"), + domain, + cache, + state_dir: PathBuf::from(state_dir), + })) + } + + /// Push `out_path` with `token`. The result is what `attic` printed. + /// + /// # Errors + /// Why the push did not finish: the config could not be written, `attic` + /// could not start, or it exited with an error. + pub async fn push(&self, token: &str, out_path: &str) -> Result { + if !out_path.starts_with("/nix/store/") || out_path.contains(char::is_whitespace) { + return Err(format!("{out_path:?} is not a store path")); + } + let home = ConfigHome::write(&self.state_dir, &self.domain, token) + .map_err(|e| format!("writing the attic config: {e}"))?; + let output = self + .command(&home.0, out_path) + .output() + .await + .map_err(|e| format!("starting attic: {e}"))?; + let said = |bytes: &[u8]| { + crate::nix_build::last_lines(&String::from_utf8_lossy(bytes)) + .unwrap_or_default() + .replace(token, "") + }; + if output.status.success() { + Ok(said(&output.stdout)) + } else { + Err(format!( + "attic exited with {}: {}", + output.status, + said(&output.stderr) + )) + } + } + + /// `attic push` of `out_path`, reading its config from `config_home`. The + /// token is in the file there and in no argument or variable. + fn command(&self, config_home: &Path, out_path: &str) -> tokio::process::Command { + let mut cmd = tokio::process::Command::new(&self.attic); + cmd.arg("push") + .arg(format!("{SERVER}:{}", self.cache)) + .arg(out_path) + .env("XDG_CONFIG_HOME", config_home) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .kill_on_drop(true); + cmd + } +} + +/// Numbers the config directories of one process's pushes. +static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + +/// The config directory of one push, removed with the token in it on drop. +struct ConfigHome(PathBuf); + +impl ConfigHome { + /// `/push-/attic/config.toml`, readable by this user alone. + fn write(parent: &Path, domain: &str, token: &str) -> std::io::Result { + use std::io::Write as _; + use std::os::unix::fs::{DirBuilderExt as _, OpenOptionsExt as _}; + // A quote or a control character would end the TOML string early. + if token.is_empty() || token.bytes().any(|b| b == b'"' || b == b'\\' || b < b' ') { + return Err(std::io::Error::other( + "the push token is empty or holds a character the config cannot", + )); + } + let home = Self(parent.join(format!( + "push-{}-{}", + std::process::id(), + NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + ))); + let dir = home.0.join("attic"); + std::fs::DirBuilder::new() + .recursive(true) + .mode(0o700) + .create(&dir)?; + let mut file = std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o600) + .open(dir.join("config.toml"))?; + write!( + file, + "default-server = \"{SERVER}\"\n\n[servers.{SERVER}]\n\ + endpoint = \"https://{domain}\"\ntoken = \"{token}\"\n" + )?; + Ok(home) + } +} + +impl Drop for ConfigHome { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +/// `reply` with `cached` set by pushing its closure, when it is a built one and +/// a cache is set. `read_token` is not polled without a cache. The token read +/// and the push together get `budget`. +/// +/// Whatever happens to the push, a built reply stays built: a failure is logged +/// and leaves `cached` false. +pub async fn cache_built( + reply: FromMetaReply, + cache: Option<&Remote>, + read_token: impl Future>, + budget: Duration, + agent: &str, +) -> FromMetaReply { + let ( + Some(cache), + FromMetaReply::Built { + out_path, system, .. + }, + ) = (cache, &reply) + else { + return reply; + }; + let pushed = tokio::time::timeout(budget, async { + let token = read_token + .await + .map_err(|e| format!("reading the push token: {e:#}"))?; + cache.push(&token, out_path).await + }) + .await + .unwrap_or_else(|_| { + Err(format!( + "the push did not finish within {} s", + budget.as_secs() + )) + }); + match pushed { + Ok(_) => { + tracing::info!(%agent, %out_path, cache = %cache.cache, "closure pushed to the cache"); + FromMetaReply::Built { + out_path: out_path.clone(), + system: system.clone(), + cached: true, + } + } + Err(reason) => { + tracing::warn!(%agent, %out_path, cache = %cache.cache, %reason, "closure not pushed to the cache"); + reply + } + } +} + +#[cfg(test)] +mod tests { + use std::os::unix::fs::PermissionsExt as _; + use std::sync::atomic::{AtomicBool, Ordering}; + + use super::*; + + const TOKEN: &str = "eyJ0b2tlbiI6InNlY3JldCJ9.c2lnbmVk"; + const OUT: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas"; + + fn built() -> FromMetaReply { + FromMetaReply::Built { + out_path: OUT.to_owned(), + system: "x86_64-linux".to_owned(), + cached: false, + } + } + + fn cached(reply: &FromMetaReply) -> bool { + match reply { + FromMetaReply::Built { cached, .. } => *cached, + other => panic!("not built: {other:?}"), + } + } + + /// A cache whose `attic` is the program `attic`: `true` and `false` stand + /// in for a push that works and one that fails, and `echo` for one that + /// says what it was run with. + fn cache_with(attic: &str) -> Remote { + Remote { + attic: PathBuf::from(attic), + domain: "cache.example.org".to_owned(), + cache: "grid".to_owned(), + state_dir: std::env::temp_dir(), + } + } + + fn token() -> impl Future> { + std::future::ready(Ok(TOKEN.to_owned())) + } + + #[tokio::test] + async fn a_push_that_works_marks_the_build_cached() { + let cache = cache_with("true"); + let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await; + assert!(cached(&reply), "{reply:?}"); + } + + #[tokio::test] + async fn a_push_that_fails_leaves_the_build_green_and_not_cached() { + let cache = cache_with("false"); + let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await; + assert_eq!(reply, built()); + } + + #[tokio::test] + async fn a_push_that_cannot_start_leaves_the_build_green_and_not_cached() { + let cache = cache_with("/nonexistent/attic"); + let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await; + assert_eq!(reply, built()); + } + + #[tokio::test] + async fn an_unreadable_token_leaves_the_build_green_and_not_cached() { + let cache = cache_with("true"); + let reply = cache_built( + built(), + Some(&cache), + async { anyhow::bail!("nothing at the path") }, + PUSH_DEADLINE, + "atlas", + ) + .await; + assert_eq!(reply, built()); + } + + #[tokio::test(start_paused = true)] + async fn a_token_read_that_outlasts_the_budget_leaves_the_build_green() { + let cache = cache_with("true"); + let started = tokio::time::Instant::now(); + let reply = cache_built( + built(), + Some(&cache), + std::future::pending(), + Duration::from_secs(5), + "atlas", + ) + .await; + assert_eq!(reply, built()); + assert_eq!(started.elapsed(), Duration::from_secs(5)); + } + + #[tokio::test] + async fn without_a_cache_nothing_is_read_or_pushed() { + let polled = std::sync::Arc::new(AtomicBool::new(false)); + let tracked = |polled: &std::sync::Arc| { + let polled = std::sync::Arc::clone(polled); + async move { + polled.store(true, Ordering::SeqCst); + token().await + } + }; + let reply = cache_built(built(), None, tracked(&polled), PUSH_DEADLINE, "atlas").await; + assert_eq!(reply, built()); + assert!(!polled.load(Ordering::SeqCst), "the token was read"); + + // Control: with a cache the same future runs. + let cache = cache_with("true"); + cache_built( + built(), + Some(&cache), + tracked(&polled), + PUSH_DEADLINE, + "atlas", + ) + .await; + assert!(polled.load(Ordering::SeqCst)); + } + + #[tokio::test] + async fn a_build_that_is_not_built_is_not_pushed() { + let cache = cache_with("true"); + let failed = FromMetaReply::Failed { + error_tail: "e".to_owned(), + }; + let reply = cache_built( + failed.clone(), + Some(&cache), + token(), + PUSH_DEADLINE, + "atlas", + ) + .await; + assert_eq!(reply, failed); + } + + #[tokio::test] + async fn the_push_command_has_no_token_in_it() { + // `echo` prints the arguments it is given. + let cache = cache_with("echo"); + let said = cache.push(TOKEN, OUT).await.expect("echo succeeds"); + assert_eq!(said, format!("push swarm:grid {OUT}")); + assert!(!said.contains(TOKEN)); + + let cmd = cache.command(Path::new("/run/home"), OUT); + let cmd = cmd.as_std(); + let args: Vec<_> = cmd.get_args().collect(); + assert_eq!(args, ["push", "swarm:grid", OUT]); + let env: Vec<_> = cmd.get_envs().collect(); + assert_eq!( + env, + [( + std::ffi::OsStr::new("XDG_CONFIG_HOME"), + Some(std::ffi::OsStr::new("/run/home")) + )] + ); + } + + #[test] + fn the_token_is_in_a_private_config_file_that_is_removed_after() { + let parent = std::env::temp_dir().join(format!("push-test-{}", std::process::id())); + std::fs::create_dir_all(&parent).expect("parent"); + let home = ConfigHome::write(&parent, "cache.example.org", TOKEN).expect("written"); + let file = home.0.join("attic/config.toml"); + let text = std::fs::read_to_string(&file).expect("reads"); + assert!(text.contains(&format!("token = \"{TOKEN}\"")), "{text}"); + assert!( + text.contains("endpoint = \"https://cache.example.org\""), + "{text}" + ); + let mode = |p: &Path| std::fs::metadata(p).expect("stat").permissions().mode() & 0o777; + assert_eq!(mode(&file), 0o600); + assert_eq!(mode(&home.0), 0o700); + let root = home.0.clone(); + drop(home); + assert!(!root.exists()); + std::fs::remove_dir_all(&parent).expect("cleanup"); + } + + #[test] + fn a_token_that_would_end_the_config_string_is_refused() { + let parent = std::env::temp_dir(); + for bad in ["", "a\"b", "a\\b", "a\nb"] { + assert!(ConfigHome::write(&parent, "h", bad).is_err(), "{bad:?}"); + } + } + + #[tokio::test] + async fn a_path_that_is_not_a_store_path_is_not_pushed() { + let cache = cache_with("true"); + assert!(cache.push(TOKEN, "--help").await.is_err()); + assert!(cache.push(TOKEN, "/nix/store/x y").await.is_err()); + assert!(cache.push(TOKEN, OUT).await.is_ok()); + } + + #[test] + fn the_settings_name_a_cache_only_when_both_are_set() { + let state = || Some("/run/swarm-builder".to_owned()); + let some = |s: &str| Some(s.to_owned()); + assert!( + Remote::from_settings(None, None, None) + .expect("ok") + .is_none() + ); + let cache = Remote::from_settings(some("cache.example.org"), some("grid"), state()) + .expect("ok") + .expect("set"); + assert_eq!(cache.domain, "cache.example.org"); + assert_eq!(cache.cache, "grid"); + assert!(Remote::from_settings(some("cache.example.org"), None, state()).is_err()); + assert!(Remote::from_settings(None, some("grid"), state()).is_err()); + assert!(Remote::from_settings(some("https://cache"), some("grid"), state()).is_err()); + assert!(Remote::from_settings(some("cache"), some("a\"b"), state()).is_err()); + assert!(Remote::from_settings(some("cache"), some("grid"), None).is_err()); + } +} diff --git a/swarm-controller/src/meta_build.rs b/swarm-controller/src/meta_build.rs index 5829e3ba..aeee6c7c 100644 --- a/swarm-controller/src/meta_build.rs +++ b/swarm-controller/src/meta_build.rs @@ -306,13 +306,18 @@ pub(crate) async fn run( } Outcome::Failed(error_tail) } - Ok(FromMetaReply::Built { out_path, system }) => { + Ok(FromMetaReply::Built { + out_path, + system, + cached, + }) => { let built_rev = meta_rev.clone(); let closure = AgentClosure { meta_rev, store_path: out_path.clone(), system, built_unix: unix_now(), + cached, }; match queue.record(&agent, &closure).await { Ok(()) => { @@ -415,9 +420,14 @@ mod tests { } fn built() -> Self { + Self::built_cached(false) + } + + fn built_cached(cached: bool) -> Self { Self::answering(Ok(FromMetaReply::Built { out_path: "/nix/store/x-toplevel".to_owned(), system: "x86_64-linux".to_owned(), + cached, })) } } @@ -470,6 +480,22 @@ mod tests { assert_eq!(closure.store_path, "/nix/store/x-toplevel"); assert_eq!(closure.system, "x86_64-linux"); assert!(closure.built_unix > 0); + assert!(!closure.cached); + } + + /// The record says what the builder said about the cache, both ways. + #[tokio::test] + async fn the_record_carries_whether_the_builder_cached_the_closure() { + for cached in [true, false] { + let queue = FakeQueue::built_cached(cached); + let outcome = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await; + assert_eq!(outcome, Outcome::Done); + let recorded = queue.recorded.lock().unwrap(); + let [(_, closure)] = recorded.as_slice() else { + panic!("one record: {recorded:?}") + }; + assert_eq!(closure.cached, cached); + } } /// Any `nix` failure, a timeout included, fails the node at once with what @@ -749,6 +775,7 @@ mod tests { store_path: "/nix/store/a".to_owned(), system: "x86_64-linux".to_owned(), built_unix: 1, + cached: false, }; client.record("atlas", &closure).await.expect("written"); closure.store_path = "/nix/store/b".to_owned(); diff --git a/swarm-secret-client/src/builder.rs b/swarm-secret-client/src/builder.rs index e5e69701..2e3444c2 100644 --- a/swarm-secret-client/src/builder.rs +++ b/swarm-secret-client/src/builder.rs @@ -2,8 +2,9 @@ //! secret live in the store, and what each object holds. //! //! swarm-controller mints the forge token; the builder reads it, and its queue -//! client secret, under its own certificate. Its read stanza in -//! `swarm-bao.nix` names exactly these two paths. +//! client secret, under its own certificate. It also reads the cache push +//! token, which the operator writes. Its read stanza in `swarm-bao.nix` names +//! exactly these three paths. use serde::{Deserialize, Serialize}; @@ -26,6 +27,30 @@ pub fn forge_token_path() -> Result { Ok(format!("{prefix}/forge/token")) } +/// The path holding the token the builder pushes closures to the binary cache +/// with, as an [`AtticPushToken`]. The operator writes it by hand: the cache is +/// an instance outside the swarm, and no swarm unit mints the token. +/// +/// Under the cache's own name segment rather than the builder's, since the +/// token belongs to the cache's tenant. +/// +/// # Errors +/// Never in practice: the name segment is a constant. The `Result` is +/// [`principal_prefix`]'s. +pub fn attic_push_token_path() -> Result { + let prefix = principal_prefix(Kind::Service, "attic")?; + Ok(format!("{prefix}/push-token")) +} + +/// What [`attic_push_token_path`] holds. +/// +/// No `Debug`: `value` writes to the binary cache. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct AtticPushToken { + /// The token, under the field name the operator writes. + pub value: String, +} + /// The path holding the queue client secret of the builder's OIDC client /// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's. /// @@ -60,5 +85,9 @@ mod tests { client_secret_path("swarm-builder").expect("a plain id is legal"), "swarm/services/swarm-builder/oidc/client" ); + assert_eq!( + attic_push_token_path().expect("a constant segment is legal"), + "swarm/services/attic/push-token" + ); } } diff --git a/swarm-wire/src/agent_closures.rs b/swarm-wire/src/agent_closures.rs index 2830ff2b..68aea9c8 100644 --- a/swarm-wire/src/agent_closures.rs +++ b/swarm-wire/src/agent_closures.rs @@ -32,6 +32,10 @@ pub struct AgentClosure { pub system: String, /// When the controller recorded the build, in unix seconds. pub built_unix: i64, + /// Whether the builder pushed the closure to the binary cache. A record + /// written before the field existed reads as false. + #[serde(default)] + pub cached: bool, } #[cfg(test)] @@ -51,12 +55,14 @@ mod tests { store_path: "/nix/store/x".to_owned(), system: "x86_64-linux".to_owned(), built_unix: 7, + cached: true, }; let wire = serde_json::json!({ "meta_rev": "abc", "store_path": "/nix/store/x", "system": "x86_64-linux", "built_unix": 7, + "cached": true, }); assert_eq!(serde_json::to_value(&value).expect("encodes"), wire); assert_eq!( @@ -64,4 +70,16 @@ mod tests { value ); } + + #[test] + fn a_record_without_the_field_reads_as_not_cached() { + let old = serde_json::json!({ + "meta_rev": "abc", + "store_path": "/nix/store/x", + "system": "x86_64-linux", + "built_unix": 7, + }); + let record = serde_json::from_value::(old).expect("decodes"); + assert!(!record.cached); + } } diff --git a/swarm-wire/src/builder.rs b/swarm-wire/src/builder.rs index ea71bb87..2a5d67b3 100644 --- a/swarm-wire/src/builder.rs +++ b/swarm-wire/src/builder.rs @@ -140,8 +140,16 @@ pub struct BuildFromMeta { #[serde(tag = "state", rename_all = "kebab-case")] pub enum FromMetaReply { /// The toplevel built; `out_path` is its store path in the builder host's - /// store and `system` the platform the builder host builds for. - Built { out_path: String, system: String }, + /// store and `system` the platform the builder host builds for. `cached` + /// is whether the builder pushed the closure to the binary cache; a failed + /// push leaves the build green with `cached` false. A builder that predates + /// the field answers without it, which reads as false. + Built { + out_path: String, + system: String, + #[serde(default)] + cached: bool, + }, /// The build did not succeed: the last lines `nix` printed, or why it /// stopped (it timed out, or could not start). The controller holds the /// deploy back and builds again on its next round. @@ -253,8 +261,14 @@ mod tests { FromMetaReply::Built { out_path: "/nix/store/x".to_owned(), system: "x86_64-linux".to_owned(), + cached: true, }, - json!({"state": "built", "out_path": "/nix/store/x", "system": "x86_64-linux"}), + json!({ + "state": "built", + "out_path": "/nix/store/x", + "system": "x86_64-linux", + "cached": true, + }), ), ( FromMetaReply::Failed { @@ -277,6 +291,19 @@ mod tests { } } + #[test] + fn a_built_reply_from_a_builder_without_the_field_reads_as_not_cached() { + let old = json!({"state": "built", "out_path": "/nix/store/x", "system": "s"}); + assert_eq!( + serde_json::from_value::(old).expect("decodes"), + FromMetaReply::Built { + out_path: "/nix/store/x".to_owned(), + system: "s".to_owned(), + cached: false, + } + ); + } + #[test] fn the_subjects_are_pinned() { assert_eq!(SUBJECT, "$SWARM.builder.build");