copy the cache's pull facts to each hive and create them in the in-swarm cache
the in-swarm cache stores its address, name and public key beside its tokens, create-only. swarm-controller copies the four cache facts to swarm/hives/<hive>/attic/* every five minutes, with read on the sources and write on the copies. part 1 of 3: nothing reads the copies until the hive reader and probe land.
This commit is contained in:
atlas
mara
parent
c79c41e004
commit
1e5070796d
15 changed files with 640 additions and 67 deletions
|
|
@ -311,12 +311,21 @@ principal and create what's missing.
|
|||
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
|
||||
the builder reads, at `secret/swarm/services/attic/push-token`, and a
|
||||
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
|
||||
It also stores the facts a hive pulls with: the cache's address at
|
||||
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
|
||||
`secret/swarm/services/attic/cache`, and the public key it signs with at
|
||||
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
|
||||
|
||||
A unit creates each secret only while the store holds none, so a second start
|
||||
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
|
||||
write the new token to its path in the store: nothing mints over a value that
|
||||
exists. A new signing key invalidates both tokens.
|
||||
|
||||
The unit creates the address, name and key once and never updates them. After
|
||||
you change `swarm.attic.domain`, or after the server's database goes and the
|
||||
cache gets a new key, delete the stale leaf and restart
|
||||
`swarm-bao-attic-tokens`.
|
||||
|
||||
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
|
||||
and `swarm` on every host, whether or not it runs the cache, so a builder on any
|
||||
host pushes to it.
|
||||
|
|
@ -345,6 +354,14 @@ the swarm at an existing tenant:
|
|||
or `bao kv put` with `value=-` and the token on stdin, so the token is in no
|
||||
command line. The token needs push rights on that cache. Until it
|
||||
exists, every push fails and builds stay green with `cached` false.
|
||||
Seed the hives' pull token the same way at
|
||||
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
|
||||
3. Seed the three facts hives pull with, each as field `value`:
|
||||
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
|
||||
slash), `secret/swarm/services/attic/cache` (the cache's name) and
|
||||
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
|
||||
cache's page or `attic cache info`). The controller copies the leaves that
|
||||
exist.
|
||||
|
||||
The builder checks the instance's certificate against the system roots and
|
||||
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
|
||||
|
|
@ -358,6 +375,14 @@ The swarm then renders no container, vhost or unit of its own cache, and
|
|||
creates no secret in the store. The store's host still mints the cache's
|
||||
leaf and writes its role, which nothing logs in to.
|
||||
|
||||
## What hives do with the cache
|
||||
|
||||
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
|
||||
leaves (`pull-token`, `url`, `cache`, `public-key`) to
|
||||
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
|
||||
own prefix. It writes a copy only when the value differs, skips a leaf the
|
||||
store doesn't hold, and leaves a copy whose source disappeared.
|
||||
|
||||
## Credentials
|
||||
|
||||
| What | Where | Who reads it |
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ the rest of the cell says how.
|
|||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
| store path | minter | reader | automatic re-mint | automatic re-pull |
|
||||
| ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, at agent creation and every 5 minutes | the agent | ✅ when missing or rejected by the homeserver | ✅ the daemon restarts within 5 minutes of a rejection |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
|
||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
|
||||
|
|
@ -44,7 +44,11 @@ the rest of the cell says how.
|
|||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
|
||||
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
|
||||
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
|
||||
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years | nothing yet | ❌ write a new token | must be stated |
|
||||
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ write a new token | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ❌ as the URL. A cache recreated with a new key leaves the stored one stale: every hive then reports `bad-signature` | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
|
||||
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
|
|
|||
|
|
@ -80,7 +80,8 @@ let
|
|||
policies = [ "swarm-bao-grafana-oidc-policy" ];
|
||||
};
|
||||
# ./swarm-attic.nix, under the cache's leaf, which may create the signing
|
||||
# key and the two tokens only once that policy is written.
|
||||
# key, the two tokens and the cache's public facts only once that policy is
|
||||
# written.
|
||||
swarm-bao-attic-secret = {
|
||||
here = deployCfg.attic.enable;
|
||||
policies = [ "swarm-bao-attic-policy" ];
|
||||
|
|
|
|||
|
|
@ -330,7 +330,7 @@ in
|
|||
[ -s ${pkiDir}/notify-router.pem ] || ${signLeaf} ${pkiDir} notify-router \
|
||||
${lib.escapeShellArg deployCfg.bao.notifyRouterCommonName} "" clientAuth
|
||||
|
||||
# The in-swarm cache's, which creates and reads its three secrets.
|
||||
# The in-swarm cache's, which creates and reads its six secrets.
|
||||
# Minted here for the matrix-ctl leaf's reason: the cache is a swarm
|
||||
# service, so on a cache host without the store this is the file an
|
||||
# operator copies.
|
||||
|
|
|
|||
|
|
@ -13,14 +13,22 @@
|
|||
#
|
||||
# `prelude` expects `./atomic-write-secret.nix` and `./refresh-consumer.nix`
|
||||
# above it, and `bao`, `coreutils` and `systemd` on the caller's `path`.
|
||||
# `ensureJwt` also wants `openssl` there; `provision` wants `atticadm`, `attic`
|
||||
# and `curl`.
|
||||
# `ensureJwt` also wants `openssl` there; `provision` wants `atticadm`, `attic`,
|
||||
# `curl` and `sed`.
|
||||
{ lib }:
|
||||
{
|
||||
# Store paths in the form `bao kv` takes, mount first.
|
||||
jwtPath,
|
||||
pullPath,
|
||||
pushPath,
|
||||
# The cache's public facts, created only while absent: the server's address
|
||||
# (scheme included), the cache's name, and the key its narinfo are signed
|
||||
# with. Hives pull with them.
|
||||
urlPath,
|
||||
cachePath,
|
||||
publicKeyPath,
|
||||
# The address `urlPath` holds, which clients reach the server at.
|
||||
url,
|
||||
# The cache's name on the server.
|
||||
cache,
|
||||
# Where the host reaches atticd, scheme included.
|
||||
|
|
@ -98,9 +106,9 @@ in
|
|||
refresh_consumer ${q machine} ${q unit} ${q envFile}
|
||||
'';
|
||||
|
||||
# The cache and the two tokens, once atticd answers. The tokens are signed
|
||||
# here with the key the store holds; the cache is created through the API by a
|
||||
# token that lives for five minutes and is never stored.
|
||||
# The cache, its two tokens and its public facts, once atticd answers. The
|
||||
# tokens are signed here with the key the store holds; the cache is created
|
||||
# through the API by a token that lives for five minutes and is never stored.
|
||||
provision = ''
|
||||
up=0
|
||||
for _ in $(seq 1 30); do
|
||||
|
|
@ -161,5 +169,16 @@ in
|
|||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# What hives pull with. The key is the one the server reports for the cache;
|
||||
# `attic` prints it on stderr.
|
||||
mint_url() { printf '%s' ${q url}; }
|
||||
mint_cache() { printf '%s' ${q cache}; }
|
||||
mint_public_key() {
|
||||
attic cache info ${q "local:${cache}"} 2>&1 | sed -n 's/^ *Public Key: *//p'
|
||||
}
|
||||
ensure_secret ${q urlPath} mint_url >/dev/null || exit 1
|
||||
ensure_secret ${q cachePath} mint_cache >/dev/null || exit 1
|
||||
ensure_secret ${q publicKeyPath} mint_public_key >/dev/null || exit 1
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,7 +6,8 @@
|
|||
# Two host units run under the cache's own store leaf, the shape
|
||||
# ./swarm-grafana.nix gives its admin password. One creates the key atticd signs
|
||||
# tokens with, if the store holds none, and hands it to the container. The other
|
||||
# creates the cache and the push and pull tokens, each only while absent
|
||||
# creates the cache, the push and pull tokens, and the facts hives pull with
|
||||
# (its address, name and public key), each only while absent
|
||||
# (./lib/attic-init.nix). Neither needs a manual step.
|
||||
#
|
||||
# The container shares the host's network namespace, as ./swarm-victorialogs.nix
|
||||
|
|
@ -73,6 +74,10 @@ let
|
|||
jwtPath = "secret/swarm/services/attic/jwt-secret";
|
||||
pullPath = "secret/swarm/services/attic/pull-token";
|
||||
pushPath = "secret/swarm/services/attic/push-token";
|
||||
urlPath = "secret/swarm/services/attic/url";
|
||||
cachePath = "secret/swarm/services/attic/cache";
|
||||
publicKeyPath = "secret/swarm/services/attic/public-key";
|
||||
url = "https://${atticCfg.domain}";
|
||||
cache = atticCfg.cache;
|
||||
inherit
|
||||
endpoint
|
||||
|
|
@ -295,10 +300,11 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# The cache and its two tokens, once atticd answers. The push token is the
|
||||
# path the swarm builder reads; the pull token is for the hives.
|
||||
# The cache, its two tokens and its public facts, once atticd answers. The
|
||||
# push token is the path the swarm builder reads; the pull token and the
|
||||
# facts are for the hives, which the controller copies them to.
|
||||
systemd.services.swarm-bao-attic-tokens = {
|
||||
description = "create the swarm cache, and its push and pull tokens in the secret store, if absent";
|
||||
description = "create the swarm cache, and its tokens and public facts in the secret store, if absent";
|
||||
after = [
|
||||
"container@${machine}.service"
|
||||
"swarm-bao-attic-secret.service"
|
||||
|
|
@ -311,6 +317,7 @@ in
|
|||
cfg.clientPackage
|
||||
pkgs.coreutils
|
||||
pkgs.curl
|
||||
pkgs.gnused
|
||||
];
|
||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||
serviceConfig = storeRetry.serviceConfig // {
|
||||
|
|
|
|||
|
|
@ -482,6 +482,14 @@ let
|
|||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
|
||||
${lib.concatMapStrings readStanza (
|
||||
map (leaf: "${credentialMountPath}/data/${leaf}") atticFactLeaves
|
||||
)}
|
||||
${lib.concatMapStrings (name: ''
|
||||
path "${credentialMountPath}/data/swarm/hives/+/attic/${name}" {
|
||||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
'') atticHiveFactNames}
|
||||
path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" {
|
||||
capabilities = ["create", "update"]
|
||||
}
|
||||
|
|
@ -718,6 +726,28 @@ let
|
|||
atticJwtLeaf = "swarm/services/attic/jwt-secret";
|
||||
atticPullTokenLeaf = "swarm/services/attic/pull-token";
|
||||
|
||||
# The cache's three public facts, beside the pull token: its address, its
|
||||
# name, and the key its narinfo are signed with. The in-swarm cache's host
|
||||
# unit creates them; for an external cache the operator writes them.
|
||||
# `swarm-controller` reads all four and copies each to
|
||||
# `swarm/hives/<hive>/attic/<name>`
|
||||
# (`swarm_secret_client::attic::Leaf`).
|
||||
atticUrlLeaf = "swarm/services/attic/url";
|
||||
atticCacheLeaf = "swarm/services/attic/cache";
|
||||
atticPublicKeyLeaf = "swarm/services/attic/public-key";
|
||||
atticFactLeaves = [
|
||||
atticPullTokenLeaf
|
||||
atticUrlLeaf
|
||||
atticCacheLeaf
|
||||
atticPublicKeyLeaf
|
||||
];
|
||||
atticHiveFactNames = [
|
||||
"pull-token"
|
||||
"url"
|
||||
"cache"
|
||||
"public-key"
|
||||
];
|
||||
|
||||
# The CI run poller's forge token, the nix half of
|
||||
# `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it
|
||||
# and reads it back before deciding whether to re-mint; the poller reads it
|
||||
|
|
@ -1199,7 +1229,7 @@ let
|
|||
}
|
||||
];
|
||||
|
||||
# The in-swarm cache's three secrets, create and read and nothing more: its
|
||||
# The in-swarm cache's six secrets, create and read and nothing more: its
|
||||
# host units create each one only while it is absent, so none needs `update`,
|
||||
# and a rewrite of a signing key or a token an operator holds is refused. The
|
||||
# cache container's journal forwarder reads its client secret under the same
|
||||
|
|
@ -1215,11 +1245,13 @@ let
|
|||
capabilities = ["create", "read"]
|
||||
}
|
||||
'')
|
||||
(
|
||||
[
|
||||
atticJwtLeaf
|
||||
atticPushTokenLeaf
|
||||
atticPullTokenLeaf
|
||||
]
|
||||
++ atticFactLeaves
|
||||
)
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.attic.otel.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
|
@ -2474,8 +2506,8 @@ in
|
|||
description = ''
|
||||
Subject the store's `swarm-attic` cert-auth role accepts: the identity
|
||||
the in-swarm cache's host units present. Its grant is create and read on
|
||||
three paths, `swarm/services/attic/jwt-secret`, `push-token` and
|
||||
`pull-token`, and nothing else.
|
||||
six paths under `swarm/services/attic/`, `jwt-secret`, `push-token`,
|
||||
`pull-token`, `url`, `cache` and `public-key`, and nothing else.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
'';
|
||||
|
|
|
|||
|
|
@ -1722,6 +1722,38 @@ let
|
|||
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
|
||||
&& !(lib.hasInfix "secret/metadata/swarm/hives" s);
|
||||
}
|
||||
{
|
||||
# `attic_copy` reads the cache's four facts and writes each into every
|
||||
# hive's `attic/` prefix. Pinned as whole stanzas, per leaf: a read of
|
||||
# the services prefix would reach every service's secrets, and `*` under
|
||||
# `hives/` every hive's queue credential. The read side has no
|
||||
# `create`/`update`, so the controller cannot replace the cache's token.
|
||||
name = "the controller reads the cache's four facts and writes a copy of each into every hive";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||
leaves = [
|
||||
"pull-token"
|
||||
"url"
|
||||
"cache"
|
||||
"public-key"
|
||||
];
|
||||
grants = grantsIn s;
|
||||
in
|
||||
lib.all (
|
||||
l:
|
||||
(grantFor grants "secret/data/swarm/services/attic/${l}").caps == [ "read" ]
|
||||
&&
|
||||
(grantFor grants "secret/data/swarm/hives/+/attic/${l}").caps == [
|
||||
"create"
|
||||
"read"
|
||||
"update"
|
||||
]
|
||||
) leaves
|
||||
&& lib.hasInfix "path \"secret/data/swarm/hives/+/attic/pull-token\" {\n capabilities = [\"create\", \"read\", \"update\"]\n}" s
|
||||
&& (grantFor grants "secret/data/swarm/services/attic/jwt-secret") == null
|
||||
&& (grantFor grants "secret/data/swarm/services/attic/push-token") == null;
|
||||
}
|
||||
{
|
||||
# `forge::runner_token` writes the runner's registration token and keeps
|
||||
# the last value in memory, so its stanza has no `read`. Pinned as the
|
||||
|
|
@ -1988,11 +2020,11 @@ let
|
|||
lib.hasInfix "/notify-router.pem ]" s && lib.hasInfix "swarm-notify-router \"\" clientAuth" s;
|
||||
}
|
||||
{
|
||||
# The cache's host units create each of its three secrets and read them
|
||||
# The cache's host units create each of its six secrets and read them
|
||||
# back, and may not replace one: no `update`. Its container's journal
|
||||
# forwarder reads its own client secret, read only. The notification
|
||||
# router's grant is the control that `grantsIn` reads a one-path grant too.
|
||||
name = "the cache's grant is create and read on its signing key and its two tokens, read on its forwarder client, and nothing else";
|
||||
name = "the cache's grant is create and read on its signing key, its two tokens and its three public facts, read on its forwarder client, and nothing else";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-attic-policy.script;
|
||||
|
|
@ -2005,24 +2037,33 @@ let
|
|||
(grantFor grants "secret/data/swarm/services/attic/jwt-secret").caps == caps
|
||||
&& (grantFor grants "secret/data/swarm/services/attic/push-token").caps == caps
|
||||
&& (grantFor grants "secret/data/swarm/services/attic/pull-token").caps == caps
|
||||
&& (grantFor grants "secret/data/swarm/services/attic/url").caps == caps
|
||||
&& (grantFor grants "secret/data/swarm/services/attic/cache").caps == caps
|
||||
&& (grantFor grants "secret/data/swarm/services/attic/public-key").caps == caps
|
||||
&&
|
||||
(grantFor grants "secret/data/swarm/services/swarm-attic-collector/oidc/client").caps == [ "read" ]
|
||||
&& lib.length grants == 4
|
||||
&& lib.length grants == 7
|
||||
&& lib.length (grantsIn baoGrantHere.systemd.services.swarm-bao-notify-router-policy.script) == 1
|
||||
&& lib.hasInfix "auth/cert/certs/swarm-attic" s
|
||||
&& lib.hasInfix "allowed_common_names=swarm-attic" s
|
||||
&& lib.hasInfix "token_policies=swarm-attic" s;
|
||||
}
|
||||
{
|
||||
# The signing key and the pull token are the cache's alone; the push
|
||||
# token is also the builder's, and is read there.
|
||||
name = "no granting unit but the cache's names its signing key or its pull token";
|
||||
# The signing key is the cache's alone; the push token is also the
|
||||
# builder's, and is read there. The pull token is also the controller's,
|
||||
# which only reads it (pinned above).
|
||||
name = "no granting unit but the cache's names its signing key, and only it and the controller name its pull token";
|
||||
ok =
|
||||
lib.filter (
|
||||
u:
|
||||
lib.hasInfix "services/attic/jwt-secret" baoGrantHere.systemd.services.${u}.script
|
||||
|| lib.hasInfix "services/attic/pull-token" baoGrantHere.systemd.services.${u}.script
|
||||
) grantingUnitNames == [ "swarm-bao-attic-policy" ];
|
||||
u: lib.hasInfix "services/attic/jwt-secret" baoGrantHere.systemd.services.${u}.script
|
||||
) grantingUnitNames == [ "swarm-bao-attic-policy" ]
|
||||
&&
|
||||
lib.filter (
|
||||
u: lib.hasInfix "services/attic/pull-token" baoGrantHere.systemd.services.${u}.script
|
||||
) grantingUnitNames == [
|
||||
"swarm-bao-controller-policy"
|
||||
"swarm-bao-attic-policy"
|
||||
];
|
||||
}
|
||||
{
|
||||
name = "the PKI unit signs the cache's leaf under its own subject";
|
||||
|
|
|
|||
|
|
@ -13,6 +13,10 @@ let
|
|||
jwtPath = "secret/swarm/services/attic/jwt-secret";
|
||||
pullPath = "secret/swarm/services/attic/pull-token";
|
||||
pushPath = "secret/swarm/services/attic/push-token";
|
||||
urlPath = "secret/swarm/services/attic/url";
|
||||
cachePath = "secret/swarm/services/attic/cache";
|
||||
publicKeyPath = "secret/swarm/services/attic/public-key";
|
||||
url = "http://cache.example";
|
||||
cache = "grid";
|
||||
endpoint = "http://127.0.0.1:8910";
|
||||
configFile = "@DIR@/server.toml";
|
||||
|
|
|
|||
|
|
@ -84,7 +84,9 @@ tokenfile=\$(sed -n 's/^token-file = "\(.*\)"$/\1/p' "\$XDG_CONFIG_HOME/attic/co
|
|||
token=\$(cat "\$tokenfile")
|
||||
echo "attic \$* | token=\$token" >>"$calls"
|
||||
case "\$1 \$2" in
|
||||
"cache info") [ -f "\$DIR/cache-exists" ] ;;
|
||||
"cache info")
|
||||
[ -f "\$DIR/cache-exists" ] && echo " Public Key: grid:FAKEKEY" >&2
|
||||
;;
|
||||
"cache create") touch "\$DIR/cache-exists" ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
|
|
@ -123,6 +125,9 @@ stored() { cat "$dir/store/$1" 2>/dev/null || true; }
|
|||
JWT=secret_swarm_services_attic_jwt-secret
|
||||
PULL=secret_swarm_services_attic_pull-token
|
||||
PUSH=secret_swarm_services_attic_push-token
|
||||
URL=secret_swarm_services_attic_url
|
||||
CACHE=secret_swarm_services_attic_cache
|
||||
KEY=secret_swarm_services_attic_public-key
|
||||
ENVFILE=$dir/secrets/server.env
|
||||
|
||||
expect() { # <description> <got> <want>
|
||||
|
|
@ -179,6 +184,9 @@ expect "exit status" $? 0
|
|||
expect "pull token stored" "$(stored $PULL)" "tok-swarm-hives-pull-1"
|
||||
expect "push token stored" "$(stored $PUSH)" "tok-swarm-builder-push-2"
|
||||
expect "cache created once" "$(n '^attic cache create')" 1
|
||||
expect "the address is stored" "$(stored $URL)" "http://cache.example"
|
||||
expect "the cache name is stored" "$(stored $CACHE)" "grid"
|
||||
expect "the cache's public key is stored" "$(stored $KEY)" "grid:FAKEKEY"
|
||||
expect "cache created by the short-lived token" "$(n '^attic cache create local:grid | token=tok-swarm-attic-init-3')" 1
|
||||
expect "tokens minted: pull, push, the creator" "$(n '^atticadm')" 3
|
||||
expect "pull token reads only the cache" "$(n 'make-token --sub swarm-hives-pull --validity 10y --pull grid$')" 1
|
||||
|
|
@ -197,6 +205,7 @@ expect "no cache create" "$(n '^attic cache create')" 0
|
|||
expect "the pull token looked the cache up" "$(n "^attic cache info local:grid | token=$pull")" 1
|
||||
expect "pull token unchanged" "$(stored $PULL)" "$pull"
|
||||
expect "push token unchanged" "$(stored $PUSH)" "$push"
|
||||
expect "address, name and key unchanged" "$(stored $URL)$(stored $CACHE)$(stored $KEY)" "http://cache.examplegridgrid:FAKEKEY"
|
||||
|
||||
case="an empty server gets its cache back with the stored tokens"
|
||||
rm -f "$dir/cache-exists"
|
||||
|
|
@ -223,7 +232,7 @@ run "$ENSURE_JWT"
|
|||
run "$PROVISION"
|
||||
expect "no token among the client's arguments" "$(grep -c -- '^attic [^|]*tok-' "$calls" || true)" 0
|
||||
# The control: the client did see tokens, through its config file.
|
||||
expect "tokens seen through the config file" "$(grep -c -- '| token=tok-' "$calls" || true)" 3
|
||||
expect "tokens seen through the config file" "$(grep -c -- '| token=tok-' "$calls" || true)" 4
|
||||
|
||||
if [ "$failures" -ne 0 ]; then
|
||||
echo "$failures of $count checks failed" >&2
|
||||
|
|
|
|||
323
swarm-controller/src/attic_copy.rs
Normal file
323
swarm-controller/src/attic_copy.rs
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
//! The cache facts a hive pulls with, copied to where the hive may read them.
|
||||
//!
|
||||
//! The cache's operator keeps four facts under `swarm/services/attic/`
|
||||
//! ([`swarm_secret_client::attic`]). A hive's store identity reads
|
||||
//! `swarm/hives/<hive>/*` and nothing under `swarm/services/`, so every
|
||||
//! [`RECONCILE_INTERVAL`] this writes each fact that is stored to every hive's
|
||||
//! `attic/` prefix, and leaves a copy that already holds the value alone.
|
||||
//!
|
||||
//! A fact that is not stored is not copied, and a copy whose source is gone
|
||||
//! stays: the pass never deletes. A swarm with no cache stores none, so the
|
||||
//! pass writes nothing.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use swarm_secret_client::SecretStore;
|
||||
use swarm_secret_client::attic::{Fact, Leaf, hive_path, service_path};
|
||||
|
||||
/// How often [`spawn`] copies.
|
||||
const RECONCILE_INTERVAL: Duration = Duration::from_mins(5);
|
||||
|
||||
/// A pass that has not finished by now is given up on; the next tick retries.
|
||||
const PASS_DEADLINE: Duration = Duration::from_mins(2);
|
||||
|
||||
/// The two store calls a pass makes, so a test can stand in for the store.
|
||||
pub(crate) trait FactStore {
|
||||
/// The fact at `path`; `None` when nothing is stored there.
|
||||
async fn read(&self, path: &str) -> Result<Option<String>>;
|
||||
/// Store `value` at `path`, replacing what is there.
|
||||
async fn write(&self, path: &str, value: &str) -> Result<()>;
|
||||
}
|
||||
|
||||
impl FactStore for SecretStore {
|
||||
async fn read(&self, path: &str) -> Result<Option<String>> {
|
||||
Ok(self.read_optional::<Fact>(path).await?.map(|f| f.value))
|
||||
}
|
||||
|
||||
async fn write(&self, path: &str, value: &str) -> Result<()> {
|
||||
let fact = Fact {
|
||||
value: value.to_owned(),
|
||||
};
|
||||
Ok(SecretStore::write(self, path, &fact).await?)
|
||||
}
|
||||
}
|
||||
|
||||
/// What one pass did.
|
||||
#[derive(Debug, Default, PartialEq, Eq)]
|
||||
pub(crate) struct Copied {
|
||||
/// Copies written because they were absent or held another value.
|
||||
pub written: usize,
|
||||
/// Hives whose copies could not all be checked or written.
|
||||
pub failed: usize,
|
||||
}
|
||||
|
||||
/// The stored facts, as `(leaf, value)`. An empty value counts as not stored.
|
||||
async fn sources(store: &impl FactStore) -> Result<Vec<(Leaf, String)>> {
|
||||
let mut found = Vec::new();
|
||||
for leaf in Leaf::ALL {
|
||||
let path = service_path(leaf)?;
|
||||
let stored = store
|
||||
.read(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
if let Some(value) = stored.filter(|v| !v.trim().is_empty()) {
|
||||
found.push((leaf, value));
|
||||
}
|
||||
}
|
||||
Ok(found)
|
||||
}
|
||||
|
||||
/// Bring `hive`'s copies up to `sources`; how many were written.
|
||||
async fn copy_hive(
|
||||
store: &impl FactStore,
|
||||
hive: &str,
|
||||
sources: &[(Leaf, String)],
|
||||
) -> Result<usize> {
|
||||
let mut written = 0;
|
||||
for (leaf, value) in sources {
|
||||
let path = hive_path(hive, *leaf)?;
|
||||
let current = store
|
||||
.read(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
if current.as_deref() == Some(value.as_str()) {
|
||||
continue;
|
||||
}
|
||||
store
|
||||
.write(&path, value)
|
||||
.await
|
||||
.with_context(|| format!("writing {path}"))?;
|
||||
written += 1;
|
||||
}
|
||||
Ok(written)
|
||||
}
|
||||
|
||||
/// One pass over `hives`.
|
||||
///
|
||||
/// # Errors
|
||||
/// When a source cannot be read, which stops the pass before any copy: a
|
||||
/// failed read is not an absent fact. A hive that fails is counted in
|
||||
/// [`Copied::failed`] and logged, and the others are still copied.
|
||||
pub(crate) async fn copy_pass(store: &impl FactStore, hives: &[String]) -> Result<Copied> {
|
||||
let sources = sources(store).await?;
|
||||
let mut done = Copied::default();
|
||||
for hive in hives {
|
||||
match copy_hive(store, hive, &sources).await {
|
||||
Ok(n) => done.written += n,
|
||||
Err(e) => {
|
||||
tracing::warn!(hive, error = %format!("{e:#}"), "attic facts: copy failed");
|
||||
done.failed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(done)
|
||||
}
|
||||
|
||||
/// Copy now and every [`RECONCILE_INTERVAL`] after. A pass that fails is
|
||||
/// logged and retried on the next tick; it never stops the daemon.
|
||||
pub fn spawn(hives: Vec<String>) {
|
||||
let hives = Arc::new(hives);
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
let pass = async {
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
copy_pass(&store, &hives).await
|
||||
};
|
||||
match tokio::time::timeout(PASS_DEADLINE, pass).await {
|
||||
Ok(Ok(done)) if done.written > 0 || done.failed > 0 => tracing::info!(
|
||||
written = done.written,
|
||||
failed = done.failed,
|
||||
"attic facts: copied to the hives"
|
||||
),
|
||||
Ok(Ok(_)) => tracing::debug!("attic facts: every hive's copy is current"),
|
||||
Ok(Err(e)) => tracing::warn!(
|
||||
error = %format!("{e:#}"),
|
||||
retry_in_s = RECONCILE_INTERVAL.as_secs(),
|
||||
"attic facts: pass failed; retrying next tick"
|
||||
),
|
||||
Err(_) => tracing::warn!(
|
||||
deadline_s = PASS_DEADLINE.as_secs(),
|
||||
"attic facts: pass timed out; retrying next tick"
|
||||
),
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Mutex;
|
||||
|
||||
use super::*;
|
||||
|
||||
/// An in-memory store that counts writes and can refuse reads or writes
|
||||
/// under a prefix.
|
||||
#[derive(Default)]
|
||||
struct Memory {
|
||||
kv: Mutex<BTreeMap<String, String>>,
|
||||
writes: Mutex<Vec<String>>,
|
||||
refuse: Option<&'static str>,
|
||||
}
|
||||
|
||||
impl Memory {
|
||||
fn with(pairs: &[(&str, &str)]) -> Self {
|
||||
let m = Self::default();
|
||||
for (path, value) in pairs {
|
||||
m.kv.lock()
|
||||
.unwrap()
|
||||
.insert((*path).to_owned(), (*value).to_owned());
|
||||
}
|
||||
m
|
||||
}
|
||||
|
||||
fn get(&self, path: &str) -> Option<String> {
|
||||
self.kv.lock().unwrap().get(path).cloned()
|
||||
}
|
||||
}
|
||||
|
||||
impl FactStore for Memory {
|
||||
async fn read(&self, path: &str) -> Result<Option<String>> {
|
||||
Ok(self.get(path))
|
||||
}
|
||||
|
||||
async fn write(&self, path: &str, value: &str) -> Result<()> {
|
||||
if self.refuse.is_some_and(|p| path.starts_with(p)) {
|
||||
anyhow::bail!("denied");
|
||||
}
|
||||
self.writes.lock().unwrap().push(path.to_owned());
|
||||
self.kv
|
||||
.lock()
|
||||
.unwrap()
|
||||
.insert(path.to_owned(), value.to_owned());
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
const SOURCES: [(&str, &str); 4] = [
|
||||
("swarm/services/attic/pull-token", "tok"),
|
||||
("swarm/services/attic/url", "https://cache.example"),
|
||||
("swarm/services/attic/cache", "swarm"),
|
||||
("swarm/services/attic/public-key", "swarm:AAAA"),
|
||||
];
|
||||
|
||||
fn hives() -> Vec<String> {
|
||||
vec!["alpha".to_owned(), "beta".to_owned()]
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn present_facts_reach_every_hive() {
|
||||
let store = Memory::with(&SOURCES);
|
||||
let done = copy_pass(&store, &hives()).await.expect("pass");
|
||||
assert_eq!(
|
||||
done,
|
||||
Copied {
|
||||
written: 8,
|
||||
failed: 0
|
||||
}
|
||||
);
|
||||
for hive in hives() {
|
||||
assert_eq!(
|
||||
store
|
||||
.get(&format!("swarm/hives/{hive}/attic/pull-token"))
|
||||
.as_deref(),
|
||||
Some("tok")
|
||||
);
|
||||
assert_eq!(
|
||||
store
|
||||
.get(&format!("swarm/hives/{hive}/attic/public-key"))
|
||||
.as_deref(),
|
||||
Some("swarm:AAAA")
|
||||
);
|
||||
}
|
||||
// Presence control for the absence cases below: copies are the only
|
||||
// thing a pass writes under a hive.
|
||||
assert!(store.get("swarm/hives/alpha/forge/token").is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn absent_facts_write_nothing() {
|
||||
let store = Memory::default();
|
||||
let done = copy_pass(&store, &hives()).await.expect("pass");
|
||||
assert_eq!(done, Copied::default());
|
||||
assert!(store.writes.lock().unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn only_the_stored_facts_are_copied_and_an_empty_one_is_not_stored() {
|
||||
let store = Memory::with(&[
|
||||
("swarm/services/attic/pull-token", "tok"),
|
||||
("swarm/services/attic/url", " \n"),
|
||||
]);
|
||||
copy_pass(&store, &hives()).await.expect("pass");
|
||||
assert_eq!(
|
||||
store.get("swarm/hives/alpha/attic/pull-token").as_deref(),
|
||||
Some("tok")
|
||||
);
|
||||
assert!(store.get("swarm/hives/alpha/attic/url").is_none());
|
||||
assert!(store.get("swarm/hives/alpha/attic/cache").is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_current_copy_is_not_rewritten_and_a_changed_value_is() {
|
||||
let store = Memory::with(&SOURCES);
|
||||
copy_pass(&store, &hives()).await.expect("first");
|
||||
let again = copy_pass(&store, &hives()).await.expect("second");
|
||||
assert_eq!(again, Copied::default(), "unchanged values are not written");
|
||||
|
||||
store.kv.lock().unwrap().insert(
|
||||
"swarm/services/attic/pull-token".to_owned(),
|
||||
"rotated".to_owned(),
|
||||
);
|
||||
let rotated = copy_pass(&store, &hives()).await.expect("third");
|
||||
assert_eq!(
|
||||
rotated,
|
||||
Copied {
|
||||
written: 2,
|
||||
failed: 0
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
store.get("swarm/hives/beta/attic/pull-token").as_deref(),
|
||||
Some("rotated")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_removed_source_leaves_the_copies() {
|
||||
let store = Memory::with(&SOURCES);
|
||||
copy_pass(&store, &hives()).await.expect("first");
|
||||
store
|
||||
.kv
|
||||
.lock()
|
||||
.unwrap()
|
||||
.remove("swarm/services/attic/pull-token");
|
||||
copy_pass(&store, &hives()).await.expect("second");
|
||||
assert_eq!(
|
||||
store.get("swarm/hives/alpha/attic/pull-token").as_deref(),
|
||||
Some("tok")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn one_hive_failing_does_not_stop_the_others() {
|
||||
let mut store = Memory::with(&SOURCES);
|
||||
store.refuse = Some("swarm/hives/alpha/");
|
||||
let done = copy_pass(&store, &hives()).await.expect("pass");
|
||||
assert_eq!(
|
||||
done,
|
||||
Copied {
|
||||
written: 4,
|
||||
failed: 1
|
||||
}
|
||||
);
|
||||
assert!(store.get("swarm/hives/beta/attic/cache").is_some());
|
||||
assert!(store.get("swarm/hives/alpha/attic/cache").is_none());
|
||||
}
|
||||
}
|
||||
|
|
@ -37,6 +37,7 @@ mod agents_md;
|
|||
mod agents_md_initial;
|
||||
mod alerts;
|
||||
mod archetypes;
|
||||
mod attic_copy;
|
||||
mod auth;
|
||||
mod builder;
|
||||
mod claude_login;
|
||||
|
|
|
|||
|
|
@ -367,7 +367,8 @@ fn spawn_subagent_streams(status: Option<&Arc<status::StatusReader>>, hives: &[H
|
|||
|
||||
/// The per-agent background passes: subagent and terminal streams, and
|
||||
/// schedule keys, and the permissions bucket and its import, and the
|
||||
/// archetypes agents are created from.
|
||||
/// archetypes agents are created from. Beside them, the
|
||||
/// copy of the cache facts to each hive.
|
||||
/// Lifted out of `main` for `clippy::too_many_lines`.
|
||||
pub(crate) fn spawn_per_agent_upkeep(
|
||||
status: Option<&Arc<status::StatusReader>>,
|
||||
|
|
@ -376,6 +377,7 @@ pub(crate) fn spawn_per_agent_upkeep(
|
|||
spawn_subagent_streams(status, hives);
|
||||
spawn_schedule_reconcile(status, hives);
|
||||
spawn_agent_permissions(status, hives);
|
||||
crate::attic_copy::spawn(hives.iter().map(|h| h.name.clone()).collect());
|
||||
if let Some(status) = status {
|
||||
archetypes::spawn(status.queue_client());
|
||||
}
|
||||
|
|
|
|||
104
swarm-secret-client/src/attic.rs
Normal file
104
swarm-secret-client/src/attic.rs
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
//! The binary cache agreement: the four facts a hive needs to pull from the
|
||||
//! swarm's cache, where the cache's operator keeps them, and where the
|
||||
//! controller copies them for each hive.
|
||||
//!
|
||||
//! The cache's own unit (or, for an external cache, the operator) writes them
|
||||
//! under `swarm/services/attic/`. No hive may read that prefix, so
|
||||
//! `swarm-controller` copies each leaf to `swarm/hives/<hive>/attic/<leaf>`,
|
||||
//! which the hive's own `swarm/hives/<hive>/*` stanza reads. The nix half of
|
||||
//! the source paths is `swarm-bao.nix`'s `atticFactLeaves`.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::{
|
||||
Error,
|
||||
path::{Kind, principal_prefix},
|
||||
};
|
||||
|
||||
/// One of the facts a hive pulls with.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum Leaf {
|
||||
/// The read-only token, as a basic-auth password.
|
||||
PullToken,
|
||||
/// The cache server's base URL, scheme included and no trailing slash.
|
||||
Url,
|
||||
/// The cache's name on that server.
|
||||
Cache,
|
||||
/// The cache's public signing key, `<name>:<base64>`.
|
||||
PublicKey,
|
||||
}
|
||||
|
||||
impl Leaf {
|
||||
/// Every leaf, in the order the controller copies them.
|
||||
pub const ALL: [Leaf; 4] = [Leaf::PullToken, Leaf::Url, Leaf::Cache, Leaf::PublicKey];
|
||||
|
||||
/// The last path segment, under both prefixes.
|
||||
#[must_use]
|
||||
pub fn name(self) -> &'static str {
|
||||
match self {
|
||||
Leaf::PullToken => "pull-token",
|
||||
Leaf::Url => "url",
|
||||
Leaf::Cache => "cache",
|
||||
Leaf::PublicKey => "public-key",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Where the cache's operator keeps `leaf`.
|
||||
///
|
||||
/// # Errors
|
||||
/// Never in practice: the name segments are constants. The `Result` is
|
||||
/// [`principal_prefix`]'s.
|
||||
pub fn service_path(leaf: Leaf) -> Result<String, Error> {
|
||||
let prefix = principal_prefix(Kind::Service, "attic")?;
|
||||
Ok(format!("{prefix}/{}", leaf.name()))
|
||||
}
|
||||
|
||||
/// Where `hive` reads its copy of `leaf`.
|
||||
///
|
||||
/// # Errors
|
||||
/// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`,
|
||||
/// which keeps one hive's name from addressing another hive's secret.
|
||||
pub fn hive_path(hive: &str, leaf: Leaf) -> Result<String, Error> {
|
||||
let prefix = principal_prefix(Kind::Hive, hive)?;
|
||||
Ok(format!("{prefix}/attic/{}", leaf.name()))
|
||||
}
|
||||
|
||||
/// What every leaf holds: one string under the field `bao kv get
|
||||
/// -field=value` reads.
|
||||
///
|
||||
/// No `Debug`: for [`Leaf::PullToken`] `value` reads from the cache.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct Fact {
|
||||
/// The fact itself.
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_paths_are_the_ones_the_store_grants() {
|
||||
// Spelled out: `swarm-bao.nix` grants each literal.
|
||||
let sources = Leaf::ALL.map(|l| service_path(l).expect("constant segments"));
|
||||
assert_eq!(
|
||||
sources,
|
||||
[
|
||||
"swarm/services/attic/pull-token",
|
||||
"swarm/services/attic/url",
|
||||
"swarm/services/attic/cache",
|
||||
"swarm/services/attic/public-key",
|
||||
]
|
||||
);
|
||||
assert_eq!(
|
||||
hive_path("alpha", Leaf::PullToken).expect("a plain name"),
|
||||
"swarm/hives/alpha/attic/pull-token"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_hive_name_cannot_address_another_hives_copy() {
|
||||
assert!(hive_path("alpha/../beta", Leaf::Url).is_err());
|
||||
}
|
||||
}
|
||||
|
|
@ -6,7 +6,7 @@
|
|||
//! deployment's environment into a logged-in client ([`client`]), and, per kind
|
||||
//! of secret, the path it lives at together with the fields it holds
|
||||
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`grafana`],
|
||||
//! [`acp`], [`ci`], [`review_classifier`], [`builder`], [`ci_metrics`],
|
||||
//! [`acp`], [`attic`], [`ci`], [`review_classifier`], [`builder`], [`ci_metrics`],
|
||||
//! [`claude_login`]).
|
||||
//! Each of those is a thing the controller and a hive must say identically, so
|
||||
//! it is said once here.
|
||||
|
|
@ -30,6 +30,7 @@
|
|||
//! needs only the provider and the store links nothing else.
|
||||
|
||||
pub mod acp;
|
||||
pub mod attic;
|
||||
pub mod builder;
|
||||
pub mod cert;
|
||||
#[cfg(feature = "otel")]
|
||||
|
|
|
|||
Loading…
Reference in a new issue