be7eaa8181
`refresh_unit` read the credential file's mtime with `stat -c %.6Y | tr -d .`. coreutils prints that fraction with the locale's decimal separator, so on a host whose locale uses a comma (muede-lpt2: `1791232888,193159`) `tr` stripped nothing, the `-le` against the unit's start errored, the "unchanged" early return was skipped, and every run try-restarted the consumer. Every unit on a store-refresh timer was restarted every ~10 minutes; for `swarm-bao-ci-runner` that is `gitea-runner-swarm`, which killed every CI job running at the time. The mtime is now read with `date -r … +%s%6N`, the same digit-only format the start time already used (`date -d … +%s%6N` on systemd's fixed English timestamp), so neither side has a separator for a locale to change. A value that still does not read as digits — an unparsable start time, a missing file — now restarts nothing and logs at `err`. Skipping on doubt rather than restarting on doubt: the file's mtime is kept, so a genuinely changed credential still restarts its consumer on the first run that can read both times, whereas restarting on doubt restarts it on every run, which is this bug. Both reads also fall back to empty instead of aborting the caller's `set -e` script. `script-test-refresh-consumer` runs the rendered helper under `de_DE.UTF-8` against a stub `systemctl`, and checks first that `stat` really prints a comma there. The helper on main fails 14 of its assertions. Fixes #5230
83 lines
3.8 KiB
Nix
83 lines
3.8 KiB
Nix
# Shared shell steps for a host oneshot that fetches a credential into a
|
|
# file a service loads at start (`LoadCredential`, or a config file expanded
|
|
# once while parsing). Such a service never sees a value that lands after it
|
|
# started — late or rotated — until it restarts.
|
|
#
|
|
# The file's mtime is the record of a change, so write the file only when
|
|
# `secret_differs` says so. The restart decision is then re-derived from
|
|
# the file on every run: a run that wrote the file but failed before the
|
|
# restart leaves a retry that still sees the file newer than the service.
|
|
#
|
|
# Pure function — NOT a NixOS module. Call it from a module's `let`:
|
|
#
|
|
# refreshConsumer = import ./lib/refresh-consumer.nix { };
|
|
# script = ''
|
|
# ${refreshConsumer}
|
|
# if secret_differs "$path" "$secret"; then
|
|
# atomic_write_secret 0400 root:root "$path" "$secret"
|
|
# fi
|
|
# refresh_consumer my-machine my.service "$path" # a unit in a container
|
|
# refresh_unit my.service "$path" # a host unit
|
|
# '';
|
|
#
|
|
# Requires `systemd` and `coreutils` on the caller's `path`.
|
|
{ }:
|
|
''
|
|
# True when $1 is missing or holds something other than $2. `$(< path)`
|
|
# is a bash builtin, so the value never becomes an argument in /proc;
|
|
# both sides lose their trailing newlines, which is how
|
|
# `atomic_write_secret` writes and `$(bao …)` reads.
|
|
secret_differs() {
|
|
[ ! -f "$1" ] || [ "$(< "$1")" != "$2" ]
|
|
}
|
|
|
|
# <unit> <path> [systemctl option…]. Nothing when <unit> last entered
|
|
# `active` after <path> was last written. Otherwise a running <unit> is
|
|
# restarted and a failed one — it may have hit its start limit without the
|
|
# credential — is reset and started; one stopped on purpose stays stopped.
|
|
# The options are passed to every `systemctl` call, which is how
|
|
# `refresh_consumer` reaches into a container. `--no-block` because a
|
|
# caller ordered `Before=` its consumer must not wait on a job that waits
|
|
# on the caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix).
|
|
refresh_unit() {
|
|
local unit="$1" path="$2" started started_us=0 written_us
|
|
shift 2
|
|
# Empty for a unit that has never been active, which `date` would
|
|
# otherwise read as today's midnight.
|
|
started="$(systemctl "$@" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")"
|
|
if [ -n "$started" ]; then
|
|
started_us="$(date -u -d "$started" +%s%6N)" || started_us=
|
|
fi
|
|
written_us="$(date -u -r "$path" +%s%6N)" || written_us=
|
|
# Both are bare microsecond counts, with no decimal separator for a
|
|
# locale to change. One that is not is logged at `err` and nothing is
|
|
# restarted: the mtime stays, so the first run that can read both still
|
|
# restarts <unit>, where restarting on doubt restarts it on every run.
|
|
if [[ ! "$started_us" =~ ^[0-9]+$ || ! "$written_us" =~ ^[0-9]+$ ]]; then
|
|
echo "<3>cannot compare the mtime of $path ('$written_us') with the last start of $unit ('$started') — leaving $unit as it is" >&2
|
|
return 0
|
|
fi
|
|
if [ "$written_us" -le "$started_us" ]; then
|
|
return 0
|
|
fi
|
|
if systemctl "$@" is-failed --quiet "$unit"; then
|
|
echo "$path changed after $unit last started, and $unit had failed — starting it"
|
|
systemctl "$@" reset-failed "$unit"
|
|
systemctl "$@" start --no-block "$unit"
|
|
else
|
|
echo "$path changed after $unit last started — restarting it if it runs"
|
|
systemctl "$@" try-restart --no-block "$unit"
|
|
fi
|
|
}
|
|
|
|
# <machine> <unit> <path>. `refresh_unit` inside `container@<machine>`, and
|
|
# nothing while that container is not active: a container that starts
|
|
# later loads the file as it starts.
|
|
refresh_consumer() {
|
|
local machine="$1" unit="$2" path="$3"
|
|
if ! systemctl is-active --quiet "container@$machine.service"; then
|
|
return 0
|
|
fi
|
|
refresh_unit "$unit" "$path" --machine="$machine"
|
|
}
|
|
''
|