push green meta builds to the attic cache and record cached
This commit is contained in:
atlas
parent
c4e8d5b540
commit
0849afec80
16 changed files with 779 additions and 31 deletions
|
|
@ -103,9 +103,10 @@ status has no link.
|
|||
hive's. A green check can therefore still fail on a hive, and the reverse.
|
||||
- The agent base is the builder's host's, not the hive's. Hives on another
|
||||
hyperhive or nixpkgs pin build something else.
|
||||
- The builder pushes nothing. The closure stays in the builder
|
||||
- The check pushes nothing. The closure stays in the builder
|
||||
host's store until it's collected, so on a host that also runs the hive, the
|
||||
hive's own build can reuse whatever matches.
|
||||
hive's own build can reuse whatever matches. A [meta build](#deploy-gate)
|
||||
can push to [a binary cache](#binary-cache).
|
||||
- The PR check gates nothing. A merge deploys whatever the status says. The
|
||||
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
|
||||
- A lost webhook delivery isn't retried: the head isn't built until the next
|
||||
|
|
@ -247,13 +248,48 @@ On a built reply the controller writes the agent's key in the `agent-closures`
|
|||
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
|
||||
platform and the time. Only the controller writes the bucket, and the builder
|
||||
holds no grant on it. A failed build writes nothing and leaves the previous key.
|
||||
Nothing reads the bucket yet. The closure stays unrooted in the builder host's
|
||||
store.
|
||||
The record's `cached` field says whether the builder pushed the closure to
|
||||
[the binary cache](#binary-cache). Nothing reads the bucket yet. The closure stays
|
||||
unrooted in the builder host's store.
|
||||
|
||||
The controller gives the `builder` forge account read on `internal/meta` when it
|
||||
updates the flake, and again before each build, since the repo is private. A
|
||||
grant the forge refuses fails the build with no verdict.
|
||||
|
||||
## Binary cache
|
||||
|
||||
With `services.hyperhive.swarm.attic.domain` and `cache` set, the builder pushes
|
||||
each green meta build's closure to that cache, an attic server outside the swarm,
|
||||
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
|
||||
not `attic watch`, and the builder pushes again when the controller runs a build again. The
|
||||
push has ten minutes, or less when the reply is due sooner, because it must end
|
||||
inside the 61 minutes the controller waits. `attic` checks the server's certificate
|
||||
against the system roots and pins nothing, so a Let's Encrypt certificate
|
||||
works.
|
||||
|
||||
The push token is the operator's to write. The builder reads it on every push
|
||||
from `secret/swarm/services/attic/push-token` (field `value`), writes an `attic`
|
||||
config file holding it in the unit's private runtime directory (`/run/swarm-builder`,
|
||||
mode `0700`, the file `0600`), points `attic` at it with `XDG_CONFIG_HOME` and
|
||||
removes it after the push. The token is in no argument or variable, so no
|
||||
process listing shows it.
|
||||
|
||||
The reply's `cached` says whether the push finished. A push that fails, times out
|
||||
or can't read the token leaves the build green with `cached` false and logs a
|
||||
warning with the reason; the deploy goes ahead and the next build of the agent
|
||||
tries again. With no cache set, the builder pushes nothing and `cached` is always
|
||||
false.
|
||||
|
||||
Mixed versions:
|
||||
|
||||
| controller | builder | what happens |
|
||||
| ---------- | ------- | ------------------------------------------------------------- |
|
||||
| new | new | `cached` in the reply and in the `agent-closures` record |
|
||||
| new | old | the reply has no `cached`, which reads as `false` |
|
||||
| old | new | the controller ignores `cached`; its record has no such field |
|
||||
|
||||
A record written before the field existed also reads as `false`.
|
||||
|
||||
## Credentials
|
||||
|
||||
| What | Where | Who reads it |
|
||||
|
|
@ -261,6 +297,7 @@ grant the forge refuses fails the build with no verdict.
|
|||
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
||||
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
|
||||
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
||||
| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push |
|
||||
|
||||
The token belongs to the `builder` forge account, which swarm-controller
|
||||
creates, puts in the `hives` team of `agent-configs` (read on every config
|
||||
|
|
@ -270,7 +307,7 @@ gets the token through a credential helper reading the child's environment:
|
|||
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
||||
status. The same `git` fetches a config's own inputs, so the builder registers
|
||||
the helper for the forge's URL only, and the helper answers only an `https`
|
||||
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the
|
||||
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
|
||||
status with its own forge token. Details per path:
|
||||
[`credentials-reference.md`](credentials-reference.md).
|
||||
|
||||
|
|
|
|||
|
|
@ -41,6 +41,7 @@ the rest of the cell says how.
|
|||
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
|
||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
|
||||
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
|
||||
| `swarm/services/attic/push-token` | you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with no `swarm.attic` cache set, nothing reads it |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
||||
|
|
@ -158,7 +159,7 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
|||
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
|
||||
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
|
||||
| `swarm-notify-router`, in its container | `notifyRouterClientCertFile` / `notifyRouterClientKeyFile` | `swarm/services/<router client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token`, `swarm/services/attic/push-token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||
|
||||
|
|
|
|||
|
|
@ -62,6 +62,7 @@
|
|||
./glue-swarm-otel-oidc-client.nix
|
||||
./glue-victorialogs-otel-oidc-client.nix
|
||||
./glue-victoriametrics-otel-oidc-client.nix
|
||||
./swarm-attic-options.nix
|
||||
./swarm-authelia-service.nix
|
||||
./swarm-authelia.nix
|
||||
./swarm-bao-service.nix
|
||||
|
|
|
|||
37
nix/host-modules/swarm-attic-options.nix
Normal file
37
nix/host-modules/swarm-attic-options.nix
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# The binary cache the swarm pushes agent closures to, as every host sees it:
|
||||
# which attic server, and which cache on it. The swarm runs no attic of its
|
||||
# own yet; the instance is the operator's, outside the swarm.
|
||||
{ lib, ... }:
|
||||
{
|
||||
options.services.hyperhive.swarm.attic = {
|
||||
domain = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "cache.example.org";
|
||||
description = ''
|
||||
Name of the attic server, with no scheme: the swarm builder pushes to
|
||||
`https://<domain>`, and trusts the certificate the system roots do, so
|
||||
a Let's Encrypt certificate works and none is pinned. Set together with
|
||||
{option}`services.hyperhive.swarm.attic.cache`.
|
||||
|
||||
Null pushes nothing: the swarm builder does not push, and the closure
|
||||
record says it is not cached.
|
||||
|
||||
The push token is not an option. The operator writes it to
|
||||
`secret/swarm/services/attic/push-token` in the swarm secret store
|
||||
(field `value`), and the builder reads it on every push.
|
||||
'';
|
||||
};
|
||||
|
||||
cache = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "grid";
|
||||
description = ''
|
||||
Name of the cache on {option}`services.hyperhive.swarm.attic.domain`
|
||||
that the swarm builder pushes closures to. The push token must be
|
||||
allowed to push to it.
|
||||
'';
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
@ -692,6 +692,12 @@ let
|
|||
# to fetch config repos.
|
||||
builderForgeTokenLeaf = "swarm/services/builder/forge/token";
|
||||
|
||||
# The token the swarm builder pushes closures to the binary cache with, the nix
|
||||
# half of `swarm_secret_client::builder::attic_push_token_path`. The operator
|
||||
# writes it by hand; the builder reads it on every push, and nothing else in
|
||||
# the swarm reaches it.
|
||||
atticPushTokenLeaf = "swarm/services/attic/push-token";
|
||||
|
||||
# The CI run poller's forge token, the nix half of
|
||||
# `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it
|
||||
# and reads it back before deciding whether to re-mint; the poller reads it
|
||||
|
|
@ -1146,14 +1152,16 @@ let
|
|||
}
|
||||
];
|
||||
|
||||
# The swarm builder: its forge token, read once per build, and its own queue
|
||||
# client secret, read at start. Both by the daemon inside its container.
|
||||
# The swarm builder: its forge token, read once per build, the cache push token,
|
||||
# read once per push, and its own queue client secret, read at start. All by the
|
||||
# daemon inside its container.
|
||||
builderReaders = [
|
||||
{
|
||||
name = "swarm-builder";
|
||||
cn = baoDeploy.builderCommonName;
|
||||
policyText =
|
||||
readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}"
|
||||
+ readStanza "${credentialMountPath}/data/${atticPushTokenLeaf}"
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
|
@ -2081,8 +2089,8 @@ in
|
|||
description = ''
|
||||
Subject the store's `swarm-builder` cert-auth role accepts: the
|
||||
identity the swarm builder presents from inside its container. Its
|
||||
grant is two paths, read only: `swarm/services/builder/forge/token`
|
||||
and its own queue client secret.
|
||||
grant is three paths, read only: `swarm/services/builder/forge/token`,
|
||||
`swarm/services/attic/push-token` and its own queue client secret.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
'';
|
||||
|
|
|
|||
|
|
@ -17,8 +17,8 @@
|
|||
#
|
||||
# The container holds one credential, its own store leaf, which nspawn hands in
|
||||
# as a system credential with no copy on disk. The daemon reads everything else
|
||||
# from the store with it: its queue client secret at start and its forge token
|
||||
# on every build.
|
||||
# from the store with it: its queue client secret at start, its forge token
|
||||
# on every build and, where a cache is set, its cache push token on every push.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
|
|
@ -33,6 +33,7 @@ let
|
|||
natsCfg = hyperhiveCfg.swarm.nats;
|
||||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||||
forgeCfg = hyperhiveCfg.swarm.forge;
|
||||
atticCfg = hyperhiveCfg.swarm.attic;
|
||||
gatewayCfg = hyperhiveCfg.gateway;
|
||||
networkCfg = hyperhiveCfg.network;
|
||||
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
|
||||
|
|
@ -89,6 +90,9 @@ let
|
|||
hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake;
|
||||
nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake;
|
||||
|
||||
# Either half alone reaches the daemon too, which refuses to start on it.
|
||||
pushesToCache = atticCfg.domain != null || atticCfg.cache != null;
|
||||
|
||||
privateNetwork = true;
|
||||
in
|
||||
{
|
||||
|
|
@ -230,8 +234,9 @@ in
|
|||
wantedBy = [ "multi-user.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
# nix fetches the config repo with `git`.
|
||||
path = [ pkgs.git ];
|
||||
# nix fetches the config repo with `git`, and a green build is pushed
|
||||
# with `attic` where a cache is set.
|
||||
path = [ pkgs.git ] ++ lib.optional pushesToCache pkgs.attic-client;
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
BAO_CLIENT_CERT = credential "builder-bao-cert";
|
||||
|
|
@ -252,6 +257,12 @@ in
|
|||
SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}";
|
||||
SWARM_BUILDER_CONFIG_ORG = "agent-configs";
|
||||
}
|
||||
// lib.optionalAttrs (atticCfg.domain != null) {
|
||||
SWARM_BUILDER_ATTIC_DOMAIN = atticCfg.domain;
|
||||
}
|
||||
// lib.optionalAttrs (atticCfg.cache != null) {
|
||||
SWARM_BUILDER_ATTIC_CACHE = atticCfg.cache;
|
||||
}
|
||||
// lib.optionalAttrs (storeFiles ? builder-bao-server-ca) {
|
||||
BAO_CACERT = credential "builder-bao-server-ca";
|
||||
}
|
||||
|
|
@ -264,6 +275,10 @@ in
|
|||
ExecStart = "${cfg.package}/bin/swarm-builder";
|
||||
DynamicUser = true;
|
||||
CacheDirectory = unit;
|
||||
# /run/swarm-builder, mode 0700 and owned by the dynamic user: the
|
||||
# push writes its token-bearing `attic` config under it.
|
||||
RuntimeDirectory = unit;
|
||||
RuntimeDirectoryMode = "0700";
|
||||
# Bare ids: the system credentials nspawn handed in.
|
||||
LoadCredential = lib.attrNames storeFiles;
|
||||
Restart = "on-failure";
|
||||
|
|
|
|||
|
|
@ -1743,17 +1743,18 @@ let
|
|||
];
|
||||
}
|
||||
{
|
||||
# The builder reads its forge token and its own queue client secret,
|
||||
# each read-only and spelled to the leaf: `services/*` would reach every
|
||||
# swarm service's secret.
|
||||
name = "the swarm builder's grant is its forge token and its queue client secret, and nothing else";
|
||||
# The builder reads its forge token, the cache push token and its own queue
|
||||
# client secret, each read-only and spelled to the leaf: `services/*` would
|
||||
# reach every swarm service's secret.
|
||||
name = "the swarm builder's grant is its forge token, the cache push token and its queue client secret, and nothing else";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-builder-policy.script;
|
||||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"read\"]\n}" s
|
||||
&& lib.hasInfix "path \"secret/data/swarm/services/attic/push-token\" {\n capabilities = [\"read\"]\n}" s
|
||||
&& lib.hasInfix "path \"secret/data/swarm/services/swarm-builder/oidc/client\" {\n capabilities = [\"read\"]\n}" s
|
||||
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2
|
||||
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 3
|
||||
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
|
||||
&& lib.hasInfix "auth/cert/certs/swarm-builder" s
|
||||
&& lib.hasInfix "allowed_common_names=swarm-builder" s
|
||||
|
|
|
|||
|
|
@ -29,6 +29,8 @@ let
|
|||
withBuilder = hive {
|
||||
deploy.bao.enable = true;
|
||||
deploy.builder.enable = true;
|
||||
swarm.attic.domain = "cache.example.org";
|
||||
swarm.attic.cache = "grid";
|
||||
};
|
||||
|
||||
servicesHere = shared.swarmServicesWithController;
|
||||
|
|
@ -48,6 +50,8 @@ let
|
|||
inner = container.config;
|
||||
svc = inner.systemd.services.swarm-builder;
|
||||
env = svc.environment;
|
||||
hasOnPath = name: s: lib.any (p: lib.getName p == name) s.path;
|
||||
withoutCache = servicesHere.containers.swarm-builder.config.systemd.services.swarm-builder;
|
||||
|
||||
# The check flake on the inputs the builder's unit names, called as
|
||||
# `nix build` calls it. `hyperhive` is entered through the filtered source's
|
||||
|
|
@ -116,6 +120,25 @@ let
|
|||
&& svc.serviceConfig.DynamicUser
|
||||
&& !(lib.any (lib.hasSuffix "TOKEN") (lib.attrNames env));
|
||||
}
|
||||
{
|
||||
# The closure is pushed with `attic` from the unit, whose config file holds
|
||||
# the push token: it goes in a runtime directory only the dynamic user can
|
||||
# enter, and the token is in no variable. The host without a cache is the
|
||||
# control: same unit, none of it, with `git` still on the path to show
|
||||
# the lookup finds what is there.
|
||||
name = "the builder pushes to the configured cache with attic from a private runtime directory, and pushes nowhere without one";
|
||||
ok =
|
||||
env.SWARM_BUILDER_ATTIC_DOMAIN == "cache.example.org"
|
||||
&& env.SWARM_BUILDER_ATTIC_CACHE == "grid"
|
||||
&& hasOnPath "attic-client" svc
|
||||
&& svc.serviceConfig.RuntimeDirectory == "swarm-builder"
|
||||
&& svc.serviceConfig.RuntimeDirectoryMode == "0700"
|
||||
&& !(lib.any (lib.hasInfix "TOKEN") (lib.attrNames env))
|
||||
&& hasOnPath "git" withoutCache
|
||||
&& !(hasOnPath "attic-client" withoutCache)
|
||||
&& !(withoutCache.environment ? SWARM_BUILDER_ATTIC_DOMAIN)
|
||||
&& !(withoutCache.environment ? SWARM_BUILDER_ATTIC_CACHE);
|
||||
}
|
||||
{
|
||||
# The minting unit goes inactive after each run, so a start that pulls it
|
||||
# runs it again and mints a leaf that is missing; an active
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ the answer into a `swarm-builder` commit status on the PR's head.
|
|||
as an untrusted client, so a config can neither set daemon options nor read
|
||||
this service's environment.
|
||||
- Nothing is pushed and nothing is recorded: the closure stays in the host's
|
||||
store until it is collected.
|
||||
store until it is collected. (A meta build can push, see below.)
|
||||
|
||||
## Meta builds
|
||||
|
||||
|
|
@ -31,16 +31,21 @@ the platform it builds for. The controller records the answer; the service
|
|||
writes nothing. It takes the same one-build slot as the PR checks. See
|
||||
[`docs/swarm/builder.md`](../docs/swarm/builder.md#deploy-gate).
|
||||
|
||||
Where `services.hyperhive.swarm.attic.{domain,cache}` are set, it first pushes a
|
||||
green build's closure to that cache with `attic push`, and the answer says
|
||||
whether that worked. A failed push leaves the build green. See
|
||||
[`docs/swarm/builder.md`](../docs/swarm/builder.md#binary-cache).
|
||||
|
||||
## Credentials
|
||||
|
||||
The container holds one credential: its own store certificate, which nspawn
|
||||
hands in at start without a copy on disk. With it the service reads its
|
||||
queue client secret at start and again when authelia refuses it, and its
|
||||
read-only forge token at every build.
|
||||
read-only forge token at every build, and the cache push token at every push.
|
||||
`git` gets the token through a credential helper reading the child's
|
||||
environment, never in a URL or an argument, and the helper answers only an
|
||||
`https` request for the forge's host. Its store role reads exactly
|
||||
those two paths.
|
||||
those three paths.
|
||||
|
||||
## Archetypes
|
||||
|
||||
|
|
|
|||
|
|
@ -16,6 +16,8 @@
|
|||
//! and an `internal/meta` commit, and the answer is the store path of that
|
||||
//! agent's toplevel, built at the flake's own lock. They take the build slot
|
||||
//! PR checks take. The controller records the path; the builder writes nothing.
|
||||
//! When a cache is set (see [`push`]), a green meta build is pushed to it before
|
||||
//! the reply, and the reply says whether that worked.
|
||||
//!
|
||||
//! The store identity comes from the `BAO_*` environment. The queue client
|
||||
//! secret is read at start and again when authelia refuses it, the forge token
|
||||
|
|
@ -39,6 +41,7 @@ use swarm_wire::builder::{
|
|||
|
||||
mod archetypes;
|
||||
mod nix_build;
|
||||
mod push;
|
||||
|
||||
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
|
||||
/// writes it, matching the leaf's subject.
|
||||
|
|
@ -64,6 +67,17 @@ const MAX_SLOT_WAIT: Duration = REPLY_DEADLINE
|
|||
.saturating_sub(BUILD_DEADLINE)
|
||||
.saturating_sub(SLOT_MARGIN);
|
||||
|
||||
/// What is held back from a meta build's reply deadline for publishing the reply
|
||||
/// after its push.
|
||||
const REPLY_MARGIN: Duration = Duration::from_secs(10);
|
||||
|
||||
/// How long the push of a meta build that ends at now may run: at most
|
||||
/// [`push::PUSH_DEADLINE`], and never past `reply_by`, so a push cannot make the
|
||||
/// reply later than the controller waits.
|
||||
fn push_budget(reply_by: tokio::time::Instant) -> Duration {
|
||||
push::PUSH_DEADLINE.min(reply_by.saturating_duration_since(tokio::time::Instant::now()))
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
hive_log::init();
|
||||
|
|
@ -83,6 +97,7 @@ async fn main() -> Result<()> {
|
|||
deadline: BUILD_DEADLINE,
|
||||
lock_deadline: LOCK_DEADLINE,
|
||||
system: var("SYSTEM")?,
|
||||
cache: push_target()?,
|
||||
});
|
||||
let secret =
|
||||
swarm_secret_client::read_with_startup_retry("reading the queue client secret", || {
|
||||
|
|
@ -300,7 +315,7 @@ async fn answer_meta_builds(
|
|||
&slot,
|
||||
received,
|
||||
|| busy_meta(&job),
|
||||
handle_meta(&job, &builder),
|
||||
handle_meta(&job, &builder, received + REPLY_DEADLINE - REPLY_MARGIN),
|
||||
)
|
||||
.await
|
||||
else {
|
||||
|
|
@ -383,19 +398,36 @@ fn busy_meta(job: &BuildFromMeta) -> FromMetaReply {
|
|||
}
|
||||
|
||||
/// Build one meta request and log the build.
|
||||
async fn handle_meta(job: &BuildFromMeta, builder: &nix_build::Builder) -> FromMetaReply {
|
||||
async fn handle_meta(
|
||||
job: &BuildFromMeta,
|
||||
builder: &nix_build::Builder,
|
||||
reply_by: tokio::time::Instant,
|
||||
) -> FromMetaReply {
|
||||
let reply = match within(TOKEN_DEADLINE, forge_token()).await {
|
||||
Ok(token) => builder.build_from_meta(job, &token).await,
|
||||
Err(e) => FromMetaReply::Unavailable {
|
||||
reason: format!("{e:#}"),
|
||||
},
|
||||
};
|
||||
let reply = push::cache_built(
|
||||
reply,
|
||||
builder.cache.as_ref(),
|
||||
attic_push_token(),
|
||||
push_budget(reply_by),
|
||||
&job.agent,
|
||||
)
|
||||
.await;
|
||||
match &reply {
|
||||
FromMetaReply::Built { out_path, system } => tracing::info!(
|
||||
FromMetaReply::Built {
|
||||
out_path,
|
||||
system,
|
||||
cached,
|
||||
} => tracing::info!(
|
||||
agent = %job.agent,
|
||||
meta_rev = %job.meta_rev,
|
||||
%out_path,
|
||||
%system,
|
||||
cached,
|
||||
"agent meta build built"
|
||||
),
|
||||
FromMetaReply::Failed { error_tail } => tracing::warn!(
|
||||
|
|
@ -458,6 +490,23 @@ fn log_build(job: &BuildAgent, reply: &Reply) {
|
|||
}
|
||||
}
|
||||
|
||||
/// The token the builder pushes closures to the cache with, read on every push so
|
||||
/// a rotation takes effect on the next one.
|
||||
async fn attic_push_token() -> Result<String> {
|
||||
let path = swarm_secret_client::builder::attic_push_token_path()?;
|
||||
let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE)
|
||||
.await
|
||||
.context("logging in to the secret store")?;
|
||||
let stored: Option<swarm_secret_client::builder::AtticPushToken> = store
|
||||
.read_optional(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
match stored {
|
||||
Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()),
|
||||
_ => bail!("nothing at {path}: the cache push token has not been written"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The forge token the builder fetches config repos with, read on every
|
||||
/// build so a rotation takes effect on the next one.
|
||||
async fn forge_token() -> Result<String> {
|
||||
|
|
@ -495,6 +544,17 @@ async fn client_secret(client_id: &str) -> Result<String> {
|
|||
}
|
||||
}
|
||||
|
||||
/// The cache the unit's environment names, if any. systemd sets
|
||||
/// `RUNTIME_DIRECTORY` for the unit's runtime directory.
|
||||
fn push_target() -> Result<Option<push::Remote>> {
|
||||
let var = |name: &str| std::env::var(name).ok();
|
||||
push::Remote::from_settings(
|
||||
var(&format!("{PREFIX}_ATTIC_DOMAIN")),
|
||||
var(&format!("{PREFIX}_ATTIC_CACHE")),
|
||||
var("RUNTIME_DIRECTORY"),
|
||||
)
|
||||
}
|
||||
|
||||
/// The forge's base URL, which must be `https`: the credential helper offers
|
||||
/// the token to `https` requests for its host alone.
|
||||
fn forge_url() -> Result<String> {
|
||||
|
|
@ -580,6 +640,7 @@ mod tests {
|
|||
FromMetaReply::Built {
|
||||
out_path: "/nix/store/x".to_owned(),
|
||||
system: "s".to_owned(),
|
||||
cached: false,
|
||||
}
|
||||
})
|
||||
.await;
|
||||
|
|
@ -594,6 +655,7 @@ mod tests {
|
|||
FromMetaReply::Built {
|
||||
out_path: "/nix/store/x".to_owned(),
|
||||
system: "s".to_owned(),
|
||||
cached: false,
|
||||
}
|
||||
})
|
||||
.await;
|
||||
|
|
@ -616,6 +678,20 @@ mod tests {
|
|||
assert_eq!(prompt.expect("read"), "t");
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn a_push_gets_the_deadline_unless_the_reply_is_due_sooner() {
|
||||
let now = Instant::now();
|
||||
assert_eq!(
|
||||
push_budget(now + Duration::from_hours(1)),
|
||||
push::PUSH_DEADLINE
|
||||
);
|
||||
assert_eq!(
|
||||
push_budget(now + Duration::from_secs(90)),
|
||||
Duration::from_secs(90)
|
||||
);
|
||||
assert_eq!(push_budget(now), Duration::ZERO);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_build_started_at_the_slot_limit_ends_inside_the_reply_deadline() {
|
||||
assert!(MAX_SLOT_WAIT > Duration::ZERO);
|
||||
|
|
|
|||
|
|
@ -59,6 +59,9 @@ pub struct Builder {
|
|||
pub lock_deadline: Duration,
|
||||
/// The platform this host builds for, reported with each meta build.
|
||||
pub system: String,
|
||||
/// Where a meta build's closure is pushed after it succeeds; `None` pushes
|
||||
/// nothing.
|
||||
pub cache: Option<crate::push::Remote>,
|
||||
}
|
||||
|
||||
impl Builder {
|
||||
|
|
@ -88,6 +91,7 @@ impl Builder {
|
|||
Reply::Built { out_path } => FromMetaReply::Built {
|
||||
out_path,
|
||||
system: self.system.clone(),
|
||||
cached: false,
|
||||
},
|
||||
Reply::Failed { error_tail } => FromMetaReply::Failed { error_tail },
|
||||
},
|
||||
|
|
@ -437,6 +441,7 @@ mod tests {
|
|||
deadline: BUILD_DEADLINE,
|
||||
lock_deadline: LOCK_DEADLINE,
|
||||
system: "x86_64-linux".to_owned(),
|
||||
cache: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -514,6 +519,7 @@ mod tests {
|
|||
FromMetaReply::Built {
|
||||
out_path: format!("{} {}", BUILD_FLAGS.join(" "), b.meta_attr(&meta_job())),
|
||||
system: "x86_64-linux".to_owned(),
|
||||
cached: false,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
|
|
|||
437
swarm-builder/src/push.rs
Normal file
437
swarm-builder/src/push.rs
Normal file
|
|
@ -0,0 +1,437 @@
|
|||
//! The push of a built closure to the binary cache, with `attic push`.
|
||||
//!
|
||||
//! The cache is an attic instance outside the swarm. Its push token is read
|
||||
//! from the secret store per push and written to an `attic` config file in a
|
||||
//! directory only the builder's unit can enter, which `attic` finds through
|
||||
//! `XDG_CONFIG_HOME`. `attic login <name> <url> <token>` is not used: it puts
|
||||
//! the token in the process list. `attic` verifies the server's certificate
|
||||
//! against the system roots.
|
||||
//!
|
||||
//! A push that does not happen or does not finish leaves the build green with
|
||||
//! `cached` false: the closure is still in the host's store, and the next build
|
||||
//! of the agent pushes again.
|
||||
|
||||
use std::future::Future;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Stdio;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use swarm_wire::builder::FromMetaReply;
|
||||
|
||||
/// The longest one push may run, when the reply deadline leaves that much.
|
||||
pub const PUSH_DEADLINE: Duration = Duration::from_mins(10);
|
||||
|
||||
/// The server name in the config file, and in the cache argument of the push.
|
||||
const SERVER: &str = "swarm";
|
||||
|
||||
/// Where the builder pushes: one cache on one attic server.
|
||||
pub struct Remote {
|
||||
/// The `attic` binary: a name found in the unit's path in production.
|
||||
pub attic: PathBuf,
|
||||
/// The server's name, without scheme: `host` or `host:port`.
|
||||
pub domain: String,
|
||||
/// The cache's name on that server.
|
||||
pub cache: String,
|
||||
/// A directory only the builder can enter, which the config files go in.
|
||||
pub state_dir: PathBuf,
|
||||
}
|
||||
|
||||
impl Remote {
|
||||
/// The cache the environment names: `None` when neither value is set, so
|
||||
/// the builder pushes nothing.
|
||||
///
|
||||
/// # Errors
|
||||
/// When only one of the two is set, or a value would bend the config file
|
||||
/// or the command.
|
||||
pub fn from_settings(
|
||||
domain: Option<String>,
|
||||
cache: Option<String>,
|
||||
state_dir: Option<String>,
|
||||
) -> Result<Option<Self>> {
|
||||
let (domain, cache) = match (domain, cache) {
|
||||
(None, None) => return Ok(None),
|
||||
(Some(domain), Some(cache)) => (domain, cache),
|
||||
_ => bail!("the cache domain and the cache name must be set together"),
|
||||
};
|
||||
let plain = |s: &str| {
|
||||
!s.is_empty()
|
||||
&& s.bytes()
|
||||
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_' | b':'))
|
||||
};
|
||||
if !plain(&domain) || !plain(&cache) {
|
||||
bail!("the cache domain {domain:?} and name {cache:?} must be plain names");
|
||||
}
|
||||
let Some(state_dir) = state_dir else {
|
||||
bail!("a cache is set, but the unit has no runtime directory to hold its config");
|
||||
};
|
||||
Ok(Some(Self {
|
||||
attic: PathBuf::from("attic"),
|
||||
domain,
|
||||
cache,
|
||||
state_dir: PathBuf::from(state_dir),
|
||||
}))
|
||||
}
|
||||
|
||||
/// Push `out_path` with `token`. The result is what `attic` printed.
|
||||
///
|
||||
/// # Errors
|
||||
/// Why the push did not finish: the config could not be written, `attic`
|
||||
/// could not start, or it exited with an error.
|
||||
pub async fn push(&self, token: &str, out_path: &str) -> Result<String, String> {
|
||||
if !out_path.starts_with("/nix/store/") || out_path.contains(char::is_whitespace) {
|
||||
return Err(format!("{out_path:?} is not a store path"));
|
||||
}
|
||||
let home = ConfigHome::write(&self.state_dir, &self.domain, token)
|
||||
.map_err(|e| format!("writing the attic config: {e}"))?;
|
||||
let output = self
|
||||
.command(&home.0, out_path)
|
||||
.output()
|
||||
.await
|
||||
.map_err(|e| format!("starting attic: {e}"))?;
|
||||
let said = |bytes: &[u8]| {
|
||||
crate::nix_build::last_lines(&String::from_utf8_lossy(bytes))
|
||||
.unwrap_or_default()
|
||||
.replace(token, "<redacted>")
|
||||
};
|
||||
if output.status.success() {
|
||||
Ok(said(&output.stdout))
|
||||
} else {
|
||||
Err(format!(
|
||||
"attic exited with {}: {}",
|
||||
output.status,
|
||||
said(&output.stderr)
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
/// `attic push` of `out_path`, reading its config from `config_home`. The
|
||||
/// token is in the file there and in no argument or variable.
|
||||
fn command(&self, config_home: &Path, out_path: &str) -> tokio::process::Command {
|
||||
let mut cmd = tokio::process::Command::new(&self.attic);
|
||||
cmd.arg("push")
|
||||
.arg(format!("{SERVER}:{}", self.cache))
|
||||
.arg(out_path)
|
||||
.env("XDG_CONFIG_HOME", config_home)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.kill_on_drop(true);
|
||||
cmd
|
||||
}
|
||||
}
|
||||
|
||||
/// Numbers the config directories of one process's pushes.
|
||||
static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
|
||||
|
||||
/// The config directory of one push, removed with the token in it on drop.
|
||||
struct ConfigHome(PathBuf);
|
||||
|
||||
impl ConfigHome {
|
||||
/// `<parent>/push-<n>/attic/config.toml`, readable by this user alone.
|
||||
fn write(parent: &Path, domain: &str, token: &str) -> std::io::Result<Self> {
|
||||
use std::io::Write as _;
|
||||
use std::os::unix::fs::{DirBuilderExt as _, OpenOptionsExt as _};
|
||||
// A quote or a control character would end the TOML string early.
|
||||
if token.is_empty() || token.bytes().any(|b| b == b'"' || b == b'\\' || b < b' ') {
|
||||
return Err(std::io::Error::other(
|
||||
"the push token is empty or holds a character the config cannot",
|
||||
));
|
||||
}
|
||||
let home = Self(parent.join(format!(
|
||||
"push-{}-{}",
|
||||
std::process::id(),
|
||||
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
|
||||
)));
|
||||
let dir = home.0.join("attic");
|
||||
std::fs::DirBuilder::new()
|
||||
.recursive(true)
|
||||
.mode(0o700)
|
||||
.create(&dir)?;
|
||||
let mut file = std::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.create_new(true)
|
||||
.mode(0o600)
|
||||
.open(dir.join("config.toml"))?;
|
||||
write!(
|
||||
file,
|
||||
"default-server = \"{SERVER}\"\n\n[servers.{SERVER}]\n\
|
||||
endpoint = \"https://{domain}\"\ntoken = \"{token}\"\n"
|
||||
)?;
|
||||
Ok(home)
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ConfigHome {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.0);
|
||||
}
|
||||
}
|
||||
|
||||
/// `reply` with `cached` set by pushing its closure, when it is a built one and
|
||||
/// a cache is set. `read_token` is not polled without a cache. The token read
|
||||
/// and the push together get `budget`.
|
||||
///
|
||||
/// Whatever happens to the push, a built reply stays built: a failure is logged
|
||||
/// and leaves `cached` false.
|
||||
pub async fn cache_built(
|
||||
reply: FromMetaReply,
|
||||
cache: Option<&Remote>,
|
||||
read_token: impl Future<Output = Result<String>>,
|
||||
budget: Duration,
|
||||
agent: &str,
|
||||
) -> FromMetaReply {
|
||||
let (
|
||||
Some(cache),
|
||||
FromMetaReply::Built {
|
||||
out_path, system, ..
|
||||
},
|
||||
) = (cache, &reply)
|
||||
else {
|
||||
return reply;
|
||||
};
|
||||
let pushed = tokio::time::timeout(budget, async {
|
||||
let token = read_token
|
||||
.await
|
||||
.map_err(|e| format!("reading the push token: {e:#}"))?;
|
||||
cache.push(&token, out_path).await
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|_| {
|
||||
Err(format!(
|
||||
"the push did not finish within {} s",
|
||||
budget.as_secs()
|
||||
))
|
||||
});
|
||||
match pushed {
|
||||
Ok(_) => {
|
||||
tracing::info!(%agent, %out_path, cache = %cache.cache, "closure pushed to the cache");
|
||||
FromMetaReply::Built {
|
||||
out_path: out_path.clone(),
|
||||
system: system.clone(),
|
||||
cached: true,
|
||||
}
|
||||
}
|
||||
Err(reason) => {
|
||||
tracing::warn!(%agent, %out_path, cache = %cache.cache, %reason, "closure not pushed to the cache");
|
||||
reply
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
use super::*;
|
||||
|
||||
const TOKEN: &str = "eyJ0b2tlbiI6InNlY3JldCJ9.c2lnbmVk";
|
||||
const OUT: &str = "/nix/store/0123456789abcdfghijklmnpqrsvwxyz-nixos-system-atlas";
|
||||
|
||||
fn built() -> FromMetaReply {
|
||||
FromMetaReply::Built {
|
||||
out_path: OUT.to_owned(),
|
||||
system: "x86_64-linux".to_owned(),
|
||||
cached: false,
|
||||
}
|
||||
}
|
||||
|
||||
fn cached(reply: &FromMetaReply) -> bool {
|
||||
match reply {
|
||||
FromMetaReply::Built { cached, .. } => *cached,
|
||||
other => panic!("not built: {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// A cache whose `attic` is the program `attic`: `true` and `false` stand
|
||||
/// in for a push that works and one that fails, and `echo` for one that
|
||||
/// says what it was run with.
|
||||
fn cache_with(attic: &str) -> Remote {
|
||||
Remote {
|
||||
attic: PathBuf::from(attic),
|
||||
domain: "cache.example.org".to_owned(),
|
||||
cache: "grid".to_owned(),
|
||||
state_dir: std::env::temp_dir(),
|
||||
}
|
||||
}
|
||||
|
||||
fn token() -> impl Future<Output = Result<String>> {
|
||||
std::future::ready(Ok(TOKEN.to_owned()))
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_push_that_works_marks_the_build_cached() {
|
||||
let cache = cache_with("true");
|
||||
let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await;
|
||||
assert!(cached(&reply), "{reply:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_push_that_fails_leaves_the_build_green_and_not_cached() {
|
||||
let cache = cache_with("false");
|
||||
let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await;
|
||||
assert_eq!(reply, built());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_push_that_cannot_start_leaves_the_build_green_and_not_cached() {
|
||||
let cache = cache_with("/nonexistent/attic");
|
||||
let reply = cache_built(built(), Some(&cache), token(), PUSH_DEADLINE, "atlas").await;
|
||||
assert_eq!(reply, built());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_unreadable_token_leaves_the_build_green_and_not_cached() {
|
||||
let cache = cache_with("true");
|
||||
let reply = cache_built(
|
||||
built(),
|
||||
Some(&cache),
|
||||
async { anyhow::bail!("nothing at the path") },
|
||||
PUSH_DEADLINE,
|
||||
"atlas",
|
||||
)
|
||||
.await;
|
||||
assert_eq!(reply, built());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn a_token_read_that_outlasts_the_budget_leaves_the_build_green() {
|
||||
let cache = cache_with("true");
|
||||
let started = tokio::time::Instant::now();
|
||||
let reply = cache_built(
|
||||
built(),
|
||||
Some(&cache),
|
||||
std::future::pending(),
|
||||
Duration::from_secs(5),
|
||||
"atlas",
|
||||
)
|
||||
.await;
|
||||
assert_eq!(reply, built());
|
||||
assert_eq!(started.elapsed(), Duration::from_secs(5));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn without_a_cache_nothing_is_read_or_pushed() {
|
||||
let polled = std::sync::Arc::new(AtomicBool::new(false));
|
||||
let tracked = |polled: &std::sync::Arc<AtomicBool>| {
|
||||
let polled = std::sync::Arc::clone(polled);
|
||||
async move {
|
||||
polled.store(true, Ordering::SeqCst);
|
||||
token().await
|
||||
}
|
||||
};
|
||||
let reply = cache_built(built(), None, tracked(&polled), PUSH_DEADLINE, "atlas").await;
|
||||
assert_eq!(reply, built());
|
||||
assert!(!polled.load(Ordering::SeqCst), "the token was read");
|
||||
|
||||
// Control: with a cache the same future runs.
|
||||
let cache = cache_with("true");
|
||||
cache_built(
|
||||
built(),
|
||||
Some(&cache),
|
||||
tracked(&polled),
|
||||
PUSH_DEADLINE,
|
||||
"atlas",
|
||||
)
|
||||
.await;
|
||||
assert!(polled.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_build_that_is_not_built_is_not_pushed() {
|
||||
let cache = cache_with("true");
|
||||
let failed = FromMetaReply::Failed {
|
||||
error_tail: "e".to_owned(),
|
||||
};
|
||||
let reply = cache_built(
|
||||
failed.clone(),
|
||||
Some(&cache),
|
||||
token(),
|
||||
PUSH_DEADLINE,
|
||||
"atlas",
|
||||
)
|
||||
.await;
|
||||
assert_eq!(reply, failed);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_push_command_has_no_token_in_it() {
|
||||
// `echo` prints the arguments it is given.
|
||||
let cache = cache_with("echo");
|
||||
let said = cache.push(TOKEN, OUT).await.expect("echo succeeds");
|
||||
assert_eq!(said, format!("push swarm:grid {OUT}"));
|
||||
assert!(!said.contains(TOKEN));
|
||||
|
||||
let cmd = cache.command(Path::new("/run/home"), OUT);
|
||||
let cmd = cmd.as_std();
|
||||
let args: Vec<_> = cmd.get_args().collect();
|
||||
assert_eq!(args, ["push", "swarm:grid", OUT]);
|
||||
let env: Vec<_> = cmd.get_envs().collect();
|
||||
assert_eq!(
|
||||
env,
|
||||
[(
|
||||
std::ffi::OsStr::new("XDG_CONFIG_HOME"),
|
||||
Some(std::ffi::OsStr::new("/run/home"))
|
||||
)]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_token_is_in_a_private_config_file_that_is_removed_after() {
|
||||
let parent = std::env::temp_dir().join(format!("push-test-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&parent).expect("parent");
|
||||
let home = ConfigHome::write(&parent, "cache.example.org", TOKEN).expect("written");
|
||||
let file = home.0.join("attic/config.toml");
|
||||
let text = std::fs::read_to_string(&file).expect("reads");
|
||||
assert!(text.contains(&format!("token = \"{TOKEN}\"")), "{text}");
|
||||
assert!(
|
||||
text.contains("endpoint = \"https://cache.example.org\""),
|
||||
"{text}"
|
||||
);
|
||||
let mode = |p: &Path| std::fs::metadata(p).expect("stat").permissions().mode() & 0o777;
|
||||
assert_eq!(mode(&file), 0o600);
|
||||
assert_eq!(mode(&home.0), 0o700);
|
||||
let root = home.0.clone();
|
||||
drop(home);
|
||||
assert!(!root.exists());
|
||||
std::fs::remove_dir_all(&parent).expect("cleanup");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_token_that_would_end_the_config_string_is_refused() {
|
||||
let parent = std::env::temp_dir();
|
||||
for bad in ["", "a\"b", "a\\b", "a\nb"] {
|
||||
assert!(ConfigHome::write(&parent, "h", bad).is_err(), "{bad:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_path_that_is_not_a_store_path_is_not_pushed() {
|
||||
let cache = cache_with("true");
|
||||
assert!(cache.push(TOKEN, "--help").await.is_err());
|
||||
assert!(cache.push(TOKEN, "/nix/store/x y").await.is_err());
|
||||
assert!(cache.push(TOKEN, OUT).await.is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_settings_name_a_cache_only_when_both_are_set() {
|
||||
let state = || Some("/run/swarm-builder".to_owned());
|
||||
let some = |s: &str| Some(s.to_owned());
|
||||
assert!(
|
||||
Remote::from_settings(None, None, None)
|
||||
.expect("ok")
|
||||
.is_none()
|
||||
);
|
||||
let cache = Remote::from_settings(some("cache.example.org"), some("grid"), state())
|
||||
.expect("ok")
|
||||
.expect("set");
|
||||
assert_eq!(cache.domain, "cache.example.org");
|
||||
assert_eq!(cache.cache, "grid");
|
||||
assert!(Remote::from_settings(some("cache.example.org"), None, state()).is_err());
|
||||
assert!(Remote::from_settings(None, some("grid"), state()).is_err());
|
||||
assert!(Remote::from_settings(some("https://cache"), some("grid"), state()).is_err());
|
||||
assert!(Remote::from_settings(some("cache"), some("a\"b"), state()).is_err());
|
||||
assert!(Remote::from_settings(some("cache"), some("grid"), None).is_err());
|
||||
}
|
||||
}
|
||||
|
|
@ -306,13 +306,18 @@ pub(crate) async fn run(
|
|||
}
|
||||
Outcome::Failed(error_tail)
|
||||
}
|
||||
Ok(FromMetaReply::Built { out_path, system }) => {
|
||||
Ok(FromMetaReply::Built {
|
||||
out_path,
|
||||
system,
|
||||
cached,
|
||||
}) => {
|
||||
let built_rev = meta_rev.clone();
|
||||
let closure = AgentClosure {
|
||||
meta_rev,
|
||||
store_path: out_path.clone(),
|
||||
system,
|
||||
built_unix: unix_now(),
|
||||
cached,
|
||||
};
|
||||
match queue.record(&agent, &closure).await {
|
||||
Ok(()) => {
|
||||
|
|
@ -415,9 +420,14 @@ mod tests {
|
|||
}
|
||||
|
||||
fn built() -> Self {
|
||||
Self::built_cached(false)
|
||||
}
|
||||
|
||||
fn built_cached(cached: bool) -> Self {
|
||||
Self::answering(Ok(FromMetaReply::Built {
|
||||
out_path: "/nix/store/x-toplevel".to_owned(),
|
||||
system: "x86_64-linux".to_owned(),
|
||||
cached,
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
|
@ -470,6 +480,22 @@ mod tests {
|
|||
assert_eq!(closure.store_path, "/nix/store/x-toplevel");
|
||||
assert_eq!(closure.system, "x86_64-linux");
|
||||
assert!(closure.built_unix > 0);
|
||||
assert!(!closure.cached);
|
||||
}
|
||||
|
||||
/// The record says what the builder said about the cache, both ways.
|
||||
#[tokio::test]
|
||||
async fn the_record_carries_whether_the_builder_cached_the_closure() {
|
||||
for cached in [true, false] {
|
||||
let queue = FakeQueue::built_cached(cached);
|
||||
let outcome = run_with(&FakeHead::reading(Ok(Some(REV))), &queue).await;
|
||||
assert_eq!(outcome, Outcome::Done);
|
||||
let recorded = queue.recorded.lock().unwrap();
|
||||
let [(_, closure)] = recorded.as_slice() else {
|
||||
panic!("one record: {recorded:?}")
|
||||
};
|
||||
assert_eq!(closure.cached, cached);
|
||||
}
|
||||
}
|
||||
|
||||
/// Any `nix` failure, a timeout included, fails the node at once with what
|
||||
|
|
@ -749,6 +775,7 @@ mod tests {
|
|||
store_path: "/nix/store/a".to_owned(),
|
||||
system: "x86_64-linux".to_owned(),
|
||||
built_unix: 1,
|
||||
cached: false,
|
||||
};
|
||||
client.record("atlas", &closure).await.expect("written");
|
||||
closure.store_path = "/nix/store/b".to_owned();
|
||||
|
|
|
|||
|
|
@ -2,8 +2,9 @@
|
|||
//! secret live in the store, and what each object holds.
|
||||
//!
|
||||
//! swarm-controller mints the forge token; the builder reads it, and its queue
|
||||
//! client secret, under its own certificate. Its read stanza in
|
||||
//! `swarm-bao.nix` names exactly these two paths.
|
||||
//! client secret, under its own certificate. It also reads the cache push
|
||||
//! token, which the operator writes. Its read stanza in `swarm-bao.nix` names
|
||||
//! exactly these three paths.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
|
|
@ -26,6 +27,30 @@ pub fn forge_token_path() -> Result<String, Error> {
|
|||
Ok(format!("{prefix}/forge/token"))
|
||||
}
|
||||
|
||||
/// The path holding the token the builder pushes closures to the binary cache
|
||||
/// with, as an [`AtticPushToken`]. The operator writes it by hand: the cache is
|
||||
/// an instance outside the swarm, and no swarm unit mints the token.
|
||||
///
|
||||
/// Under the cache's own name segment rather than the builder's, since the
|
||||
/// token belongs to the cache's tenant.
|
||||
///
|
||||
/// # Errors
|
||||
/// Never in practice: the name segment is a constant. The `Result` is
|
||||
/// [`principal_prefix`]'s.
|
||||
pub fn attic_push_token_path() -> Result<String, Error> {
|
||||
let prefix = principal_prefix(Kind::Service, "attic")?;
|
||||
Ok(format!("{prefix}/push-token"))
|
||||
}
|
||||
|
||||
/// What [`attic_push_token_path`] holds.
|
||||
///
|
||||
/// No `Debug`: `value` writes to the binary cache.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct AtticPushToken {
|
||||
/// The token, under the field name the operator writes.
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
/// The path holding the queue client secret of the builder's OIDC client
|
||||
/// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's.
|
||||
///
|
||||
|
|
@ -60,5 +85,9 @@ mod tests {
|
|||
client_secret_path("swarm-builder").expect("a plain id is legal"),
|
||||
"swarm/services/swarm-builder/oidc/client"
|
||||
);
|
||||
assert_eq!(
|
||||
attic_push_token_path().expect("a constant segment is legal"),
|
||||
"swarm/services/attic/push-token"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -32,6 +32,10 @@ pub struct AgentClosure {
|
|||
pub system: String,
|
||||
/// When the controller recorded the build, in unix seconds.
|
||||
pub built_unix: i64,
|
||||
/// Whether the builder pushed the closure to the binary cache. A record
|
||||
/// written before the field existed reads as false.
|
||||
#[serde(default)]
|
||||
pub cached: bool,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
|
@ -51,12 +55,14 @@ mod tests {
|
|||
store_path: "/nix/store/x".to_owned(),
|
||||
system: "x86_64-linux".to_owned(),
|
||||
built_unix: 7,
|
||||
cached: true,
|
||||
};
|
||||
let wire = serde_json::json!({
|
||||
"meta_rev": "abc",
|
||||
"store_path": "/nix/store/x",
|
||||
"system": "x86_64-linux",
|
||||
"built_unix": 7,
|
||||
"cached": true,
|
||||
});
|
||||
assert_eq!(serde_json::to_value(&value).expect("encodes"), wire);
|
||||
assert_eq!(
|
||||
|
|
@ -64,4 +70,16 @@ mod tests {
|
|||
value
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_record_without_the_field_reads_as_not_cached() {
|
||||
let old = serde_json::json!({
|
||||
"meta_rev": "abc",
|
||||
"store_path": "/nix/store/x",
|
||||
"system": "x86_64-linux",
|
||||
"built_unix": 7,
|
||||
});
|
||||
let record = serde_json::from_value::<AgentClosure>(old).expect("decodes");
|
||||
assert!(!record.cached);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -140,8 +140,16 @@ pub struct BuildFromMeta {
|
|||
#[serde(tag = "state", rename_all = "kebab-case")]
|
||||
pub enum FromMetaReply {
|
||||
/// The toplevel built; `out_path` is its store path in the builder host's
|
||||
/// store and `system` the platform the builder host builds for.
|
||||
Built { out_path: String, system: String },
|
||||
/// store and `system` the platform the builder host builds for. `cached`
|
||||
/// is whether the builder pushed the closure to the binary cache; a failed
|
||||
/// push leaves the build green with `cached` false. A builder that predates
|
||||
/// the field answers without it, which reads as false.
|
||||
Built {
|
||||
out_path: String,
|
||||
system: String,
|
||||
#[serde(default)]
|
||||
cached: bool,
|
||||
},
|
||||
/// The build did not succeed: the last lines `nix` printed, or why it
|
||||
/// stopped (it timed out, or could not start). The controller holds the
|
||||
/// deploy back and builds again on its next round.
|
||||
|
|
@ -253,8 +261,14 @@ mod tests {
|
|||
FromMetaReply::Built {
|
||||
out_path: "/nix/store/x".to_owned(),
|
||||
system: "x86_64-linux".to_owned(),
|
||||
cached: true,
|
||||
},
|
||||
json!({"state": "built", "out_path": "/nix/store/x", "system": "x86_64-linux"}),
|
||||
json!({
|
||||
"state": "built",
|
||||
"out_path": "/nix/store/x",
|
||||
"system": "x86_64-linux",
|
||||
"cached": true,
|
||||
}),
|
||||
),
|
||||
(
|
||||
FromMetaReply::Failed {
|
||||
|
|
@ -277,6 +291,19 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_built_reply_from_a_builder_without_the_field_reads_as_not_cached() {
|
||||
let old = json!({"state": "built", "out_path": "/nix/store/x", "system": "s"});
|
||||
assert_eq!(
|
||||
serde_json::from_value::<FromMetaReply>(old).expect("decodes"),
|
||||
FromMetaReply::Built {
|
||||
out_path: "/nix/store/x".to_owned(),
|
||||
system: "s".to_owned(),
|
||||
cached: false,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_subjects_are_pinned() {
|
||||
assert_eq!(SUBJECT, "$SWARM.builder.build");
|
||||
|
|
|
|||
Loading…
Reference in a new issue