`refresh_unit` read the credential file's mtime with
`stat -c %.6Y | tr -d .`. coreutils prints that fraction with the
locale's decimal separator, so on a host whose locale uses a comma
(muede-lpt2: `1791232888,193159`) `tr` stripped nothing, the `-le`
against the unit's start errored, the "unchanged" early return was
skipped, and every run try-restarted the consumer. Every unit on a
store-refresh timer was restarted every ~10 minutes; for
`swarm-bao-ci-runner` that is `gitea-runner-swarm`, which killed every
CI job running at the time.
The mtime is now read with `date -r … +%s%6N`, the same digit-only
format the start time already used (`date -d … +%s%6N` on systemd's
fixed English timestamp), so neither side has a separator for a locale
to change.
A value that still does not read as digits — an unparsable start time,
a missing file — now restarts nothing and logs at `err`. Skipping on
doubt rather than restarting on doubt: the file's mtime is kept, so a
genuinely changed credential still restarts its consumer on the first
run that can read both times, whereas restarting on doubt restarts it
on every run, which is this bug. Both reads also fall back to empty
instead of aborting the caller's `set -e` script.
`script-test-refresh-consumer` runs the rendered helper under
`de_DE.UTF-8` against a stub `systemctl`, and checks first that `stat`
really prints a comma there. The helper on main fails 14 of its
assertions.
Fixes#5230
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.
`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.
otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.
nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.
Refs #5015
The restart decision was a shell variable set by comparing the fetched
value with the file just before overwriting it. A run that wrote the
file and then failed before the restart (the matrix unit's registration
render, or `systemctl --machine` finding no bus yet) left a retry that
saw an unchanged file and never restarted the consumer.
The file is now written only when the value differs, so its mtime marks
the last real change, and `refresh_consumer <machine> <unit> <path>`
compares that mtime with the consumer's ActiveEnterTimestamp on every
run, the shape the openbao client-CA refresh in swarm-bao.nix already
uses. A consumer that started after the last change is left alone; a
running one is try-restarted, a failed one reset and started, all with
--no-block, and nothing happens while the container is down.
The helper's comment block also exceeded the 30-line limit
(`comment-block lint` failed on d871467d); its per-function notes now
sit beside the functions.
module-eval-bao-grants asserts the gated write, the path the refresh is
keyed on, and the mtime-vs-start comparison for each consumer.
Refs #4662
The previous commit's comments said a unit in auto-restart keeps its
start job, so anything ordered after it waits for the whole 24h retry
window. That is wrong under the default RestartMode=normal: each failed
attempt passes through `failed`, which ends that start job. `After=`
dependents proceed after one attempt, `Requires=` dependents fail with
`dependency`, and the retries continue as fresh start jobs. The
2026-09-24 journal shows it with the already-2880 swarm-services-cert:
nginx got "Dependency failed" 1ms after the first failure, and
switch-to-configuration exited before the first restart was scheduled.
The comments in lib/store-retry.nix, glue-matrix-bao-token.nix,
glue-queue-agent-credential.nix, swarm-otel.nix and swarm-grafana.nix
now say that, and so does docs/swarm/credentials.md.
Because dependents start after one attempt, a consumer that loads its
credential at start never sees a value a later attempt lands, or a
rotated one. nix/host-modules/lib/refresh-consumer.nix adds
`secret_differs` and `refresh_consumer`, and the four fetch units whose
consumers take a start-time copy call them after the write, only when
the value changed:
- swarm-bao-matrix-token -> tuwunel.service in hive-matrix
- swarm-bao-otel-oidc -> opentelemetry-collector.service in swarm-otel
- swarm-bao-grafana-oidc -> grafana.service in the grafana container
- swarm-bao-forwarder-oidc -> opentelemetry-collector.service in swarm-bao
A running consumer is try-restarted, a failed one is reset and started,
all with --no-block. Inline in the fetch script rather than a
PathChanged path unit because the fetch script is the only writer and
already knows whether the value changed, and it is the same shape as
this PR's nginx hook and swarm-bao-nats-tls's restart of nats.
module-eval-bao-grants gains one case per consumer.
Refs #4662
