nix: re-run the host's store fetches on a ten-minute timer
Seven host oneshots fetch a secret from the swarm secret store into a file their consumer loads at start: swarm-bao-grafana-oidc, swarm-bao-otel-oidc, swarm-bao-matrix-oidc, swarm-bao-ci-runner, swarm-bao-forge-oidc, swarm-bao-hive-otel-oidc and swarm-bao-forwarder-oidc. Each was RemainAfterExit with no timer, so it ran at boot or deploy and never again: a secret re-minted in the store reached its consumer only through an operator restarting the fetch. lib/store-refresh-timer.nix gives each one a timer (OnUnitInactiveSec 10min, RandomizedDelaySec 1min, the agent's own store re-read interval), and each drops RemainAfterExit so a timer start is not a no-op. Each script already writes the file only when secret_differs, and refresh_consumer/refresh_unit restart the consumer only when the file is newer than its last start, so a run reading an unchanged value restarts nothing. Two docs said these units do not run on a timer; credentials.md's ci-runner row said the runner token is not re-pulled. Part of #5017 (slice T).
This commit is contained in:
parent
c2dc18413b
commit
2f05e2db28
12 changed files with 146 additions and 13 deletions
|
|
@ -160,8 +160,8 @@ needs `deploy.forgejo.enable` or a hive on its host.
|
||||||
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
|
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
|
||||||
and nothing else, retrying every 30 seconds until the token is there. It
|
and nothing else, retrying every 30 seconds until the token is there. It
|
||||||
writes the token to a root-only file inside the container, and restarts
|
writes the token to a root-only file inside the container, and restarts
|
||||||
the runner when the value changes. It runs at boot and on deploy, not on a
|
the runner when the value changes. It runs at boot, on deploy, and every
|
||||||
timer.
|
ten minutes on a timer.
|
||||||
3. **`swarm-ci-runner-register.service`** in the container is the only unit
|
3. **`swarm-ci-runner-register.service`** in the container is the only unit
|
||||||
that loads that file, with `LoadCredential=`. systemd hands a unit's
|
that loads that file, with `LoadCredential=`. systemd hands a unit's
|
||||||
credentials to all of its processes, and for a `host` label act_runner
|
credentials to all of its processes, and for a `host` label act_runner
|
||||||
|
|
|
||||||
|
|
@ -134,8 +134,9 @@ reads its secret from the swarm's secret store under the hive's own leaf
|
||||||
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
|
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
|
||||||
`swarm/hives/<hive>/oidc/client`, which the publisher on authelia's host
|
`swarm/hives/<hive>/oidc/client`, which the publisher on authelia's host
|
||||||
writes. It runs at boot and retries every 30s until the store answers; when a
|
writes. It runs at boot and retries every 30s until the store answers; when a
|
||||||
run lands a changed value it restarts the collector. It doesn't run on a timer,
|
run lands a changed value it restarts the collector. A timer runs it again
|
||||||
so a secret rotated later arrives the next time the unit runs. On a hive that
|
every ten minutes, so a secret rotated later reaches the collector within
|
||||||
|
about ten minutes. On a hive that
|
||||||
runs the store the leaf is already there; elsewhere it's the credential you
|
runs the store the leaf is already there; elsewhere it's the credential you
|
||||||
place — see [secret placement](../swarm/secrets.md).
|
place — see [secret placement](../swarm/secrets.md).
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -72,7 +72,7 @@ of the cell says how.
|
||||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held until hive-c0re restarts |
|
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held until hive-c0re restarts |
|
||||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed |
|
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
||||||
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
||||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||||
|
|
|
||||||
|
|
@ -68,6 +68,7 @@ let
|
||||||
|
|
||||||
baoCfg = config.services.hyperhive.swarm.bao;
|
baoCfg = config.services.hyperhive.swarm.bao;
|
||||||
storeRetry = import ../lib/store-retry.nix { };
|
storeRetry = import ../lib/store-retry.nix { };
|
||||||
|
storeRefreshTimer = import ../lib/store-refresh-timer.nix { };
|
||||||
atomicWriteSecret = import ../lib/atomic-write-secret.nix { };
|
atomicWriteSecret = import ../lib/atomic-write-secret.nix { };
|
||||||
refreshConsumer = import ../lib/refresh-consumer.nix { };
|
refreshConsumer = import ../lib/refresh-consumer.nix { };
|
||||||
havePublisherIdentity =
|
havePublisherIdentity =
|
||||||
|
|
@ -1033,8 +1034,9 @@ in
|
||||||
# later attempt lands re-runs `forgejo-sso-source` (below).
|
# later attempt lands re-runs `forgejo-sso-source` (below).
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
# No `RemainAfterExit`: the timer below starts this again, and
|
||||||
|
# starting an active unit is a no-op.
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
RemainAfterExit = true;
|
|
||||||
SyslogIdentifier = "swarm-bao-forge-oidc";
|
SyslogIdentifier = "swarm-bao-forge-oidc";
|
||||||
TimeoutStartSec = 30;
|
TimeoutStartSec = 30;
|
||||||
};
|
};
|
||||||
|
|
@ -1113,6 +1115,8 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
systemd.timers.swarm-bao-forge-oidc = storeRefreshTimer "re-read the forge's OIDC client secret from the swarm secret store";
|
||||||
|
|
||||||
# The reverse direction of the unit above: publish the swarm-controller's
|
# The reverse direction of the unit above: publish the swarm-controller's
|
||||||
# freshly-minted forge token OUT of the container into the swarm secret
|
# freshly-minted forge token OUT of the container into the swarm secret
|
||||||
# store, where the controller reads it wherever it runs. On the host and
|
# store, where the controller reads it wherever it runs. On the host and
|
||||||
|
|
|
||||||
|
|
@ -51,6 +51,7 @@ let
|
||||||
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
||||||
refreshConsumer = import ./lib/refresh-consumer.nix { };
|
refreshConsumer = import ./lib/refresh-consumer.nix { };
|
||||||
storeRetry = import ./lib/store-retry.nix { };
|
storeRetry = import ./lib/store-retry.nix { };
|
||||||
|
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
|
|
||||||
# ⚠️ tuwunel does NOT read the host path directly, and this indirection
|
# ⚠️ tuwunel does NOT read the host path directly, and this indirection
|
||||||
# is not ceremony. Upstream's own words: "under systemd the path must be
|
# is not ceremony. Upstream's own words: "under systemd the path must be
|
||||||
|
|
@ -617,8 +618,9 @@ in
|
||||||
# later attempt lands restarts tuwunel (below).
|
# later attempt lands restarts tuwunel (below).
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
# No `RemainAfterExit`: the timer below starts this again, and
|
||||||
|
# starting an active unit is a no-op.
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
RemainAfterExit = true;
|
|
||||||
SyslogIdentifier = "swarm-bao-matrix-oidc";
|
SyslogIdentifier = "swarm-bao-matrix-oidc";
|
||||||
TimeoutStartSec = 30;
|
TimeoutStartSec = 30;
|
||||||
};
|
};
|
||||||
|
|
@ -689,6 +691,8 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
systemd.timers.swarm-bao-matrix-oidc = storeRefreshTimer "re-read the homeserver's OIDC client secret from the swarm secret store";
|
||||||
|
|
||||||
# ⚠️ Deliberately NO `networking.hosts` entry for authelia's name. This
|
# ⚠️ Deliberately NO `networking.hosts` entry for authelia's name. This
|
||||||
# container resolves through the hive's dnsmasq at `bridgeIp` (see the
|
# container resolves through the hive's dnsmasq at `bridgeIp` (see the
|
||||||
# static resolv.conf below), and every `gateway.localNames` entry —
|
# static resolv.conf below), and every `gateway.localNames` entry —
|
||||||
|
|
|
||||||
27
nix/host-modules/lib/store-refresh-timer.nix
Normal file
27
nix/host-modules/lib/store-refresh-timer.nix
Normal file
|
|
@ -0,0 +1,27 @@
|
||||||
|
# Timer for a host oneshot that fetches a credential from the swarm secret
|
||||||
|
# store: it starts the oneshot again 10 minutes after each run, so a value
|
||||||
|
# re-minted in the store lands on the host and restarts its consumer with no
|
||||||
|
# operator step. Ten minutes matches the agent's own store re-read
|
||||||
|
# (../../agent-modules/bao.nix).
|
||||||
|
#
|
||||||
|
# ⚠️ The oneshot must not be `RemainAfterExit`: starting a unit that stays
|
||||||
|
# `active (exited)` is a no-op, so the timer would fire and fetch nothing.
|
||||||
|
# A run that reads an unchanged value writes nothing and restarts nothing —
|
||||||
|
# its `secret_differs` gate and ./refresh-consumer.nix see to that.
|
||||||
|
#
|
||||||
|
# Gate the timer exactly as its service is gated: a timer whose unit does
|
||||||
|
# not exist fails to load.
|
||||||
|
#
|
||||||
|
# Pure function — NOT a NixOS module. Call it from a module's `let`:
|
||||||
|
#
|
||||||
|
# storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
|
# systemd.timers.swarm-bao-foo = storeRefreshTimer "re-read foo's secret from the swarm secret store";
|
||||||
|
{ }:
|
||||||
|
description: {
|
||||||
|
inherit description;
|
||||||
|
wantedBy = [ "timers.target" ];
|
||||||
|
timerConfig = {
|
||||||
|
OnUnitInactiveSec = "10min";
|
||||||
|
RandomizedDelaySec = "1min";
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
@ -348,6 +348,7 @@ in
|
||||||
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
||||||
refreshConsumer = import ./lib/refresh-consumer.nix { };
|
refreshConsumer = import ./lib/refresh-consumer.nix { };
|
||||||
storeRetry = import ./lib/store-retry.nix { };
|
storeRetry = import ./lib/store-retry.nix { };
|
||||||
|
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
# Reachable from agent containers and nowhere else: this opens
|
# Reachable from agent containers and nowhere else: this opens
|
||||||
|
|
@ -766,8 +767,9 @@ in
|
||||||
# a later attempt lands restarts it (below).
|
# a later attempt lands restarts it (below).
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
# No `RemainAfterExit`: the timer below starts this again, and
|
||||||
|
# starting an active unit is a no-op.
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
RemainAfterExit = true;
|
|
||||||
SyslogIdentifier = fetchUnit;
|
SyslogIdentifier = fetchUnit;
|
||||||
TimeoutStartSec = 30;
|
TimeoutStartSec = 30;
|
||||||
};
|
};
|
||||||
|
|
@ -828,6 +830,10 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
systemd.timers.${fetchUnit} = lib.mkIf fetches (
|
||||||
|
storeRefreshTimer "re-read this hive's telemetry OIDC client secret from the swarm secret store"
|
||||||
|
);
|
||||||
|
|
||||||
assertions = [
|
assertions = [
|
||||||
{
|
{
|
||||||
# A hive authenticates to the swarm's collector as itself — there
|
# A hive authenticates to the swarm's collector as itself — there
|
||||||
|
|
|
||||||
|
|
@ -1367,6 +1367,7 @@ let
|
||||||
# rather than a convenience for nginx.
|
# rather than a convenience for nginx.
|
||||||
privateNetwork = false;
|
privateNetwork = false;
|
||||||
storeRetry = import ./lib/store-retry.nix { };
|
storeRetry = import ./lib/store-retry.nix { };
|
||||||
|
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
|
|
@ -2566,8 +2567,9 @@ in
|
||||||
# which this boot waits on.
|
# which this boot waits on.
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
# No `RemainAfterExit`: the timer below starts this again, and
|
||||||
|
# starting an active unit is a no-op.
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
RemainAfterExit = true;
|
|
||||||
SyslogIdentifier = "swarm-bao-forwarder-oidc";
|
SyslogIdentifier = "swarm-bao-forwarder-oidc";
|
||||||
TimeoutStartSec = 30;
|
TimeoutStartSec = 30;
|
||||||
};
|
};
|
||||||
|
|
@ -2639,6 +2641,8 @@ in
|
||||||
refresh_consumer ${lib.escapeShellArg cfg.machine} opentelemetry-collector.service ${lib.escapeShellArg forwarderHostSecretPath}
|
refresh_consumer ${lib.escapeShellArg cfg.machine} opentelemetry-collector.service ${lib.escapeShellArg forwarderHostSecretPath}
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
systemd.timers.swarm-bao-forwarder-oidc = storeRefreshTimer "re-read the secret store forwarder's OIDC client secret from the store";
|
||||||
})
|
})
|
||||||
|
|
||||||
(lib.mkIf (deployCfg.bao.enable && haveServerTls) {
|
(lib.mkIf (deployCfg.bao.enable && haveServerTls) {
|
||||||
|
|
|
||||||
|
|
@ -91,6 +91,7 @@ let
|
||||||
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
||||||
refreshConsumer = import ./lib/refresh-consumer.nix { };
|
refreshConsumer = import ./lib/refresh-consumer.nix { };
|
||||||
storeRetry = import ./lib/store-retry.nix { };
|
storeRetry = import ./lib/store-retry.nix { };
|
||||||
|
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
|
|
||||||
privateNetwork = true;
|
privateNetwork = true;
|
||||||
in
|
in
|
||||||
|
|
@ -206,8 +207,9 @@ in
|
||||||
];
|
];
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
# No `RemainAfterExit`: the timer below starts this again, and
|
||||||
|
# starting an active unit is a no-op.
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
RemainAfterExit = true;
|
|
||||||
SyslogIdentifier = "swarm-bao-ci-runner";
|
SyslogIdentifier = "swarm-bao-ci-runner";
|
||||||
TimeoutStartSec = 30;
|
TimeoutStartSec = 30;
|
||||||
};
|
};
|
||||||
|
|
@ -268,6 +270,8 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
systemd.timers.swarm-bao-ci-runner = storeRefreshTimer "re-read the CI runner's registration token from the swarm secret store";
|
||||||
|
|
||||||
systemd.services."container@${machine}" = caTrust.containerOrdering;
|
systemd.services."container@${machine}" = caTrust.containerOrdering;
|
||||||
|
|
||||||
containers.${machine} = {
|
containers.${machine} = {
|
||||||
|
|
|
||||||
|
|
@ -278,6 +278,7 @@ let
|
||||||
# reaches this at 127.0.0.1:<port>.
|
# reaches this at 127.0.0.1:<port>.
|
||||||
privateNetwork = false;
|
privateNetwork = false;
|
||||||
storeRetry = import ./lib/store-retry.nix { };
|
storeRetry = import ./lib/store-retry.nix { };
|
||||||
|
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
# What the service IS to every hive is `swarm.grafana` in
|
# What the service IS to every hive is `swarm.grafana` in
|
||||||
|
|
@ -576,8 +577,9 @@ in
|
||||||
# Grafana (below).
|
# Grafana (below).
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
# No `RemainAfterExit`: the timer below starts this again, and
|
||||||
|
# starting an active unit is a no-op.
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
RemainAfterExit = true;
|
|
||||||
SyslogIdentifier = "swarm-bao-grafana-oidc";
|
SyslogIdentifier = "swarm-bao-grafana-oidc";
|
||||||
# What actually bounds each attempt below. Stated here rather than left
|
# What actually bounds each attempt below. Stated here rather than left
|
||||||
# to systemd's default, so the number a boot waits on is in the file
|
# to systemd's default, so the number a boot waits on is in the file
|
||||||
|
|
@ -665,6 +667,8 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
systemd.timers.swarm-bao-grafana-oidc = storeRefreshTimer "re-read Grafana's OIDC client secret from the swarm secret store";
|
||||||
|
|
||||||
# The admin password: created in the store if it holds none, then copied
|
# The admin password: created in the store if it holds none, then copied
|
||||||
# into the container, under Grafana's own leaf. Same shape as
|
# into the container, under Grafana's own leaf. Same shape as
|
||||||
# `swarm-bao-nats-callout-xkey` in ./swarm-nats.nix. The grant is
|
# `swarm-bao-nats-callout-xkey` in ./swarm-nats.nix. The grant is
|
||||||
|
|
|
||||||
|
|
@ -303,6 +303,7 @@ let
|
||||||
# its own trust material.
|
# its own trust material.
|
||||||
privateNetwork = false;
|
privateNetwork = false;
|
||||||
storeRetry = import ./lib/store-retry.nix { };
|
storeRetry = import ./lib/store-retry.nix { };
|
||||||
|
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
# What the collector IS to the swarm — the client it is registered as, where
|
# What the collector IS to the swarm — the client it is registered as, where
|
||||||
|
|
@ -453,8 +454,9 @@ in
|
||||||
# the collector (below).
|
# the collector (below).
|
||||||
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
serviceConfig = storeRetry.serviceConfig // {
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
# No `RemainAfterExit`: the timer below starts this again, and
|
||||||
|
# starting an active unit is a no-op.
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
RemainAfterExit = true;
|
|
||||||
SyslogIdentifier = "swarm-bao-otel-oidc";
|
SyslogIdentifier = "swarm-bao-otel-oidc";
|
||||||
# What actually bounds each attempt below. Stated here rather than left
|
# What actually bounds each attempt below. Stated here rather than left
|
||||||
# to systemd's default, so the number a boot waits on is in the file
|
# to systemd's default, so the number a boot waits on is in the file
|
||||||
|
|
@ -532,6 +534,10 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
systemd.timers.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity (
|
||||||
|
storeRefreshTimer "re-read the swarm collector's OIDC client secret from the swarm secret store"
|
||||||
|
);
|
||||||
|
|
||||||
# Where the delivery unit above lands the secret. `mkDefault`, so a
|
# Where the delivery unit above lands the secret. `mkDefault`, so a
|
||||||
# deployment delivering it some other way just sets the option — and
|
# deployment delivering it some other way just sets the option — and
|
||||||
# `mkIf haveClientIdentity` so a host with no store identity is left with
|
# `mkIf haveClientIdentity` so a host with no store identity is left with
|
||||||
|
|
|
||||||
|
|
@ -73,6 +73,43 @@ let
|
||||||
vals = lib.concatMap (s: builtins.attrValues (s.serviceConfig or { })) (builtins.attrValues svcs);
|
vals = lib.concatMap (s: builtins.attrValues (s.serviceConfig or { })) (builtins.attrValues svcs);
|
||||||
in
|
in
|
||||||
builtins.foldl' (acc: v: builtins.seq v acc) true vals;
|
builtins.foldl' (acc: v: builtins.seq v acc) true vals;
|
||||||
|
|
||||||
|
# The host fetches that restart their consumer when the store's value
|
||||||
|
# changes (../host-modules/lib/refresh-consumer.nix), each on a fixture
|
||||||
|
# that renders it.
|
||||||
|
storeRefreshed = [
|
||||||
|
{
|
||||||
|
unit = "swarm-bao-grafana-oidc";
|
||||||
|
machine = allLocal;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
unit = "swarm-bao-otel-oidc";
|
||||||
|
machine = allLocal;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
unit = "swarm-bao-matrix-oidc";
|
||||||
|
machine = allLocal;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
unit = "swarm-bao-forge-oidc";
|
||||||
|
machine = allLocal;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
unit = "swarm-bao-hive-otel-oidc";
|
||||||
|
machine = allLocal;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
unit = "swarm-bao-forwarder-oidc";
|
||||||
|
machine = allLocal;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
unit = "swarm-bao-ci-runner";
|
||||||
|
machine = withCi;
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
remainsAfterExit = u: u.serviceConfig.RemainAfterExit or false;
|
||||||
|
|
||||||
cases = [
|
cases = [
|
||||||
{
|
{
|
||||||
# The domain is the stub's swarm domain, which both fixtures share.
|
# The domain is the stub's swarm domain, which both fixtures share.
|
||||||
|
|
@ -621,6 +658,42 @@ let
|
||||||
}
|
}
|
||||||
&& lib.all (a: a.assertion) (noDomain { deploy.hive-controller.enable = false; }).assertions;
|
&& lib.all (a: a.assertion) (noDomain { deploy.hive-controller.enable = false; }).assertions;
|
||||||
}
|
}
|
||||||
];
|
{
|
||||||
|
# Control: a store fetch with no timer still renders `RemainAfterExit`,
|
||||||
|
# so the cases below read a real value rather than a missing one.
|
||||||
|
name = "Grafana's admin-password fetch stays active after it exits and has no timer";
|
||||||
|
ok =
|
||||||
|
allLocal.systemd.services ? swarm-bao-grafana-admin
|
||||||
|
&& remainsAfterExit allLocal.systemd.services.swarm-bao-grafana-admin
|
||||||
|
&& !(allLocal.systemd.timers ? swarm-bao-grafana-admin);
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# A timer whose unit does not exist fails to load, so each timer is
|
||||||
|
# gated exactly as its fetch is.
|
||||||
|
name = "a host running none of the store-refreshed fetches renders none of their timers";
|
||||||
|
ok = lib.all (
|
||||||
|
{ unit, ... }: !(bare.systemd.services ? ${unit}) && !(bare.systemd.timers ? ${unit})
|
||||||
|
) storeRefreshed;
|
||||||
|
}
|
||||||
|
]
|
||||||
|
# Starting a unit that stays `active (exited)` is a no-op, so a timer aimed
|
||||||
|
# at one fires and fetches nothing.
|
||||||
|
++ map (
|
||||||
|
{ unit, machine }:
|
||||||
|
{
|
||||||
|
name = "${unit} re-runs on a ten-minute timer and does not stay active after it exits";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = machine.systemd.services;
|
||||||
|
t = machine.systemd.timers;
|
||||||
|
in
|
||||||
|
s ? ${unit}
|
||||||
|
&& t ? ${unit}
|
||||||
|
&& !(remainsAfterExit s.${unit})
|
||||||
|
&& lib.elem "timers.target" t.${unit}.wantedBy
|
||||||
|
&& t.${unit}.timerConfig.OnUnitInactiveSec == "10min"
|
||||||
|
&& (t.${unit}.timerConfig.Unit or "${unit}.service") == "${unit}.service";
|
||||||
|
}
|
||||||
|
) storeRefreshed;
|
||||||
in
|
in
|
||||||
runGroup "core-toggle" cases
|
runGroup "core-toggle" cases
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue