refresh-consumer: key the restart on the file's mtime, not a pre-write compare

The restart decision was a shell variable set by comparing the fetched
value with the file just before overwriting it. A run that wrote the
file and then failed before the restart (the matrix unit's registration
render, or `systemctl --machine` finding no bus yet) left a retry that
saw an unchanged file and never restarted the consumer.

The file is now written only when the value differs, so its mtime marks
the last real change, and `refresh_consumer <machine> <unit> <path>`
compares that mtime with the consumer's ActiveEnterTimestamp on every
run, the shape the openbao client-CA refresh in swarm-bao.nix already
uses. A consumer that started after the last change is left alone; a
running one is try-restarted, a failed one reset and started, all with
--no-block, and nothing happens while the container is down.

The helper's comment block also exceeded the 30-line limit
(`comment-block lint` failed on d871467d); its per-function notes now
sit beside the functions.

module-eval-bao-grants asserts the gated write, the path the refresh is
keyed on, and the mtime-vs-start comparison for each consumer.

Refs #4662

nix/host-modules/lib/refresh-consumer.nix

@@ -1,56 +1,64 @@
 # Shared shell steps for a host oneshot that fetches a credential into a
 # file a service inside a container loads at start (`LoadCredential`, or a
 # config file expanded once while parsing). Such a service never sees a
-# value that lands after it started — a late fetch or a rotation — until it
-# is restarted, so the fetch restarts it, and only when the value changed:
-# a routine re-fetch of the same value leaves it alone.
+# value that lands after it started — late or rotated — until it restarts.
 #
-#   secret_differs <path> <value>
-#     True when <path> is missing or holds something other than <value>.
-#     Call it BEFORE `atomic_write_secret` writes <path>. Compares with
-#     `$(< path)`, a bash builtin, so the value never becomes an argument in
-#     /proc; both sides lose their trailing newlines, which is how
-#     `atomic_write_secret` writes and `$(bao …)` reads.
-#
-#   refresh_consumer <machine> <unit>
-#     Nothing while `container@<machine>` is not active: a container that
-#     starts later loads the file as it starts. Otherwise a running
-#     consumer is restarted, and a failed one —
-#     a consumer that found no credential may have hit its start limit — is
-#     reset and started. A consumer stopped on purpose stays stopped.
-#     `--no-block` throughout: a caller ordered `Before=` its consumer must
-#     not wait on a job that waits on the caller (the deadlock stated at
-#     `swarm-services-cert`'s propagation in ../hive-tls.nix).
+# The file's mtime is the record of a change, so write the file only when
+# `secret_differs` says so. The restart decision is then re-derived from
+# the file on every run: a run that wrote the file but failed before the
+# restart leaves a retry that still sees the file newer than the service.
 #
 # Pure function — NOT a NixOS module. Call it from a module's `let`:
 #
 #   refreshConsumer = import ./lib/refresh-consumer.nix { };
 #   script = ''
 #     ${refreshConsumer}
-#     changed=0
-#     if secret_differs "$path" "$secret"; then changed=1; fi
-#     atomic_write_secret 0400 root:root "$path" "$secret"
-#     if [ "$changed" = 1 ]; then refresh_consumer my-machine my.service; fi
+#     if secret_differs "$path" "$secret"; then
+#       atomic_write_secret 0400 root:root "$path" "$secret"
+#     fi
+#     refresh_consumer my-machine my.service "$path"
 #   '';
 #
-# Requires `systemd` on the caller's `path`.
+# Requires `systemd` and `coreutils` on the caller's `path`.
 { }:
 ''
+  # True when $1 is missing or holds something other than $2. `$(< path)`
+  # is a bash builtin, so the value never becomes an argument in /proc;
+  # both sides lose their trailing newlines, which is how
+  # `atomic_write_secret` writes and `$(bao …)` reads.
   secret_differs() {
     [ ! -f "$1" ] || [ "$(< "$1")" != "$2" ]
   }
 
+  # <machine> <unit> <path>. Nothing while `container@<machine>` is not
+  # active (a container that starts later loads the file as it starts), or
+  # when <unit> last entered `active` after <path> was last written.
+  # Otherwise a running consumer is restarted and a failed one — it may
+  # have hit its start limit without the credential — is reset and started;
+  # one stopped on purpose stays stopped. `--no-block` because a caller
+  # ordered `Before=` its consumer must not wait on a job that waits on the
+  # caller (see `swarm-services-cert`'s propagation in ../hive-tls.nix).
   refresh_consumer() {
-    local machine="$1" unit="$2"
+    local machine="$1" unit="$2" path="$3" started started_us=0 written_us
     if ! systemctl is-active --quiet "container@$machine.service"; then
       return 0
     fi
+    # Empty for a unit that has never been active, which `date` would
+    # otherwise read as today's midnight.
+    started="$(systemctl --machine="$machine" show --timestamp=us+utc -p ActiveEnterTimestamp --value "$unit")"
+    if [ -n "$started" ]; then
+      started_us="$(date -u -d "$started" +%s%6N)"
+    fi
+    written_us="$(stat -c %.6Y "$path" | tr -d .)"
+    if [ "$written_us" -le "$started_us" ]; then
+      return 0
+    fi
     if systemctl --machine="$machine" is-failed --quiet "$unit"; then
-      echo "the credential for $unit in $machine changed and $unit had failed — starting it"
+      echo "$path changed after $unit in $machine last started, and $unit had failed — starting it"
       systemctl --machine="$machine" reset-failed "$unit"
       systemctl --machine="$machine" start --no-block "$unit"
     else
-      echo "the credential for $unit in $machine changed — restarting it if it runs"
+      echo "$path changed after $unit in $machine last started — restarting it if it runs"
       systemctl --machine="$machine" try-restart --no-block "$unit"
     fi
   }