docs: extract best-effort oneshot service contract + matrix-avatar (#718 batch 6)

Two new sections housing rationale that was inline in
`harness-base.nix`:

- `docs/conventions.md::Best-effort oneshot services` — shape
  contract shared by `tea-login`, `forge-avatar-sync`, and
  `matrix-avatar-sync`: always exit 0, no `set -e`, skip silently
  on missing prerequisites, wired to multi-user.target,
  re-runnable. Plus the artefact-under-agent-home + service-stays-
  root posture and the operator-visible-via-journalctl trade-off.

- `docs/persistence.md::matrix-avatar-sync` — the two-step `media
  upload` → `set avatar_url` matrix-spec dance + why
  `RemainAfterExit = false` (so the `.path` watcher's re-fire on
  token appearance actually re-executes the unit).

In-code comments trim to one-liner purpose + pointers; the script
bodies stay (they're the actual implementation) but their inline
`#` rationale comments collapse where the docs cover them.

`description = ''…''` blocks (operator-facing options docs)
preserved per iris #718.

`nix flake check` clean; `nix fmt` clean.

docs/conventions.md

@@ -206,3 +206,53 @@ Stage and commit when work *looks* ready, then run validation
 (`cargo check`, `nix flake check`, real deploy). Failures get a
 follow-up commit rather than an amend. The commit history is the
 work log; rewriting it loses signal.
+
+## Best-effort oneshot services
+
+The harness ships a family of one-shot systemd services that
+configure agent-side surfaces from values hive-c0re writes into
+the state dir at provisioning time:
+
+- `tea-login` — writes `~/.config/tea/config.yml` from the
+  `forge-token` written by `hive-c0re::forge::ensure_user_for`,
+  so `tea repos create` / `tea pulls create` work without
+  interactive prompts.
+- `forge-avatar-sync` — uploads `hyperhive.icon` SVG to the
+  agent's Forgejo profile, so the icon shows up on commits / PRs /
+  issue comments.
+- `matrix-avatar-sync` — same idea for matrix profile avatars
+  (two-step `media upload` → `set avatar_url` dance — see
+  `docs/persistence.md::matrix-avatar-sync` for the protocol
+  detail).
+
+Shape contract — every one of these:
+
+1. **Always `exit 0`**, even on internal failure. A non-zero exit
+   would mark the unit `failed`, which in turn aborts
+   `nixos-container update` and blocks rebuilds. The agent's
+   capability surface is not allowed to gate the container
+   build.
+2. **No `set -e`** in the script body. Subshell failures must not
+   propagate. Use `... || true` on every external call that can
+   fail (forge unreachable, missing icon, parse error, etc.)
+3. **Skip silently when prerequisites are missing**: no token
+   file, no icon, no reachable upstream → `echo` a short skip
+   line + `exit 0`. The next boot tries again.
+4. **Wired to `multi-user.target`** so they run on every boot
+   (lets a rotated token / new icon take effect without
+   `systemctl restart` gymnastics).
+5. **Re-runnable**: a second invocation produces the same final
+   state (idempotent uploads, idempotent config rewrites). Used
+   by the `.path` watchers that re-fire on token appearance
+   (#571 — see `docs/persistence.md::matrix-avatar-sync`).
+
+The artefact lives under the agent user's home where applicable
+(`~/.config/tea/config.yml`) and is chown'd to that user, but the
+service itself stays root-owned so the bootstrap ordering doesn't
+need a user-existence check before each fire.
+
+This pattern keeps the rebuild path resilient: any failure inside
+these services degrades the corresponding surface (no tea config,
+no avatar) but never blocks the container from coming up. The
+operator notices through `journalctl -u <unit>` rather than a
+broken switch-to-configuration.

docs/persistence.md

@@ -259,3 +259,25 @@ restart. The `.path` unit makes the appearance of the token re-fire
 the service so the daemon comes alive in the same boot cycle as
 provisioning. `matrix-avatar-sync.path` uses the same pattern for
 the icon-upload oneshot.
+
+### matrix-avatar-sync (two-step media + profile dance)
+
+Mirrors the forge-avatar oneshot's shape (`docs/conventions.md::
+Best-effort oneshot services`) but differs in protocol: matrix
+avatars are a two-step `POST /media/r0/upload` → `PUT
+/profile/<user_id>/avatar_url` dance, both authenticated by the
+`access_token` written by `hive-c0re::matrix::ensure_user_for` to
+`<state>/matrix-token`.
+
+Triggered by EITHER boot (`wantedBy = multi-user.target`) OR the
+sibling `matrix-avatar-sync.path` firing on token appearance. Both
+paths re-run the oneshot idempotently — running the avatar set
+twice is harmless.
+
+Critically: **`RemainAfterExit = false`** (not the more common
+`true` for oneshots). systemd treats `RemainAfterExit = true`
+oneshots as "still running" after the first exit, so the second
+trigger from the `.path` watcher becomes a no-op. Setting it to
+`false` lets re-fires actually re-execute. The trade-off is the
+service unit shows `inactive (dead)` between fires — visible in
+`journalctl` but harmless; the `.path` unit drives the lifecycle.

nix/templates/harness-base.nix

@@ -984,11 +984,9 @@ in
       (pkgs.callPackage ../packages/hive-forge-tools.nix { })
     ];
 
-    # One-shot: write tea's config.yml from the seeded forge token so
+    # One-shot: tea config.yml from the seeded forge token. Shape
-    # the agent can use `tea` without interactive prompts. Runs on
+    # contract (always exit 0, no set -e, skip-silently, re-runnable):
-    # every boot so a rotated token (hive-c0re remints on each agent
+    # docs/conventions.md::Best-effort oneshot services.
-    # rebuild) is always reflected. *Always* exits 0 — never fail a
-    # NixOS switch-to-configuration over a missing/temperamental forge.
     systemd.services.tea-login = {
       description = "configure tea CLI from hive-forge token (best-effort)";
       wantedBy = [ "multi-user.target" ];
@@ -1005,12 +1003,9 @@ in
       environment.HOME_DIR = homeDir;
       environment.AGENT_USER = userName;
       script = ''
-                # No `set -e`: any subshell failure must not propagate.
+                # No `set -e`: best-effort posture (see docs pointer above).
-                # A failed unit aborts `nixos-container update` which blocks rebuilds.
                 FORGE_URL=${lib.escapeShellArg config.hyperhive.forge.url}
-                # $HYPERHIVE_STATE_DIR is set system-wide by the meta flake
+                # $HYPERHIVE_STATE_DIR is system-wide via the meta flake.
-                # (systemd.globalEnvironment, /agents/<name>/state per agent
-                # including manager post-#604).
                 TOKEN_FILE="$HYPERHIVE_STATE_DIR/forge-token"
                 if [ ! -f "$TOKEN_FILE" ]; then
                   echo "tea-login: no forge-token at $TOKEN_FILE; skipping"
@@ -1027,15 +1022,8 @@ in
                   echo "tea-login: could not resolve username from forge API; skipping"
                   exit 0
                 fi
-                # tea reads config from ~/.config/tea/config.yml. The
+                # Config under the agent user's home, chown'd to them;
-                # agent user's home is $HOME_DIR (set by NixOS via
+                # service stays root-owned (see docs pointer above).
-                # hyperhive.user.name). Write the config under that
-                # home + chown to the agent user so tea reads it when
-                # invoked as that user. Still runs as root (this
-                # service stays root-owned to avoid bootstrap
-                # ordering issues — see comment near serviceConfig
-                # below), but the artefact it produces is for the
-                # agent user.
                 CONFIG="$HOME_DIR/.config/tea/config.yml"
                 mkdir -p "$(dirname "$CONFIG")" || true
                 cat > "$CONFIG" << EOF
@@ -1059,11 +1047,8 @@ in
       '';
     };
 
-    # One-shot: upload the agent's configured icon to its Forgejo user avatar
+    # One-shot: hyperhive.icon → Forgejo profile avatar. Shape contract:
-    # so the icon shows up on commits / PRs / issue comments in the forge.
+    # docs/conventions.md::Best-effort oneshot services.
-    # Only runs when `/etc/hyperhive/icon.svg` is present (set via
-    # `hyperhive.icon`). No-op when the forge is unreachable or the icon
-    # is not set. *Always* exits 0.
     systemd.services.forge-avatar-sync = {
       description = "sync agent icon to Forgejo user avatar (best-effort)";
       wantedBy = [ "multi-user.target" ];
@@ -1158,36 +1143,20 @@ in
       pathConfig.PathExistsGlob = "/agents/*/state/matrix-token";
     };
 
-    # Path-trigger sibling so matrix-avatar-sync fires the moment
+    # Path-trigger sibling: re-fires matrix-avatar-sync the moment
-    # `<state>/matrix-token` appears (#571 closes argus's first-boot
+    # `<state>/matrix-token` appears. Same first-boot-ordering pattern
-    # ordering nag on #567). On a clean boot hive-c0re's matrix
+    # as hive-matrix-daemon above.
-    # provisioning runs concurrently with the agent container coming
-    # up; without this the oneshot would skip silently because the
-    # token didn't exist yet, and avatar would only land on the next
-    # restart. With the path watcher the appearance of the token
-    # triggers a re-fire of the service so the avatar is set in the
-    # same boot cycle as provisioning completes. The glob matches
-    # every agent (manager sees its own state at `/agents/hm1nd/state/`
-    # via the `/agents` bind).
     systemd.paths.matrix-avatar-sync = {
       description = "trigger matrix-avatar-sync when matrix-token appears";
       wantedBy = [ "multi-user.target" ];
       pathConfig.PathExistsGlob = "/agents/*/state/matrix-token";
     };
 
-    # One-shot: upload the agent's configured icon to its matrix profile
+    # One-shot: hyperhive.icon → matrix profile avatar (two-step media
-    # avatar so the icon shows up next to messages in matrix rooms (#548
+    # upload + set avatar_url). Shape contract:
-    # phase 2.5). Mirrors the forge-avatar-sync flow above, only differs
+    # docs/conventions.md::Best-effort oneshot services. Protocol +
-    # in protocol: matrix avatars are a two-step `media upload` → `set
+    # why RemainAfterExit = false:
-    # avatar_url` dance, both authenticated by the access_token written
+    # docs/persistence.md::matrix-avatar-sync.
-    # by hive-c0re's `matrix::ensure_user_for`. No-op when the icon
-    # isn't configured, the matrix token isn't present, or the
-    # homeserver isn't reachable. *Always* exits 0.
-    #
-    # Triggered by EITHER boot (`wantedBy = multi-user.target`) OR
-    # the path-watcher above (`matrix-avatar-sync.path`) firing on
-    # token appearance (#571). Both paths re-run the oneshot
-    # idempotently — running the avatar set twice is harmless.
     systemd.services.matrix-avatar-sync = {
       description = "sync agent icon to matrix profile avatar (best-effort)";
       wantedBy = [ "multi-user.target" ];
@@ -1195,11 +1164,8 @@ in
       # equivalent prerequisite; we just need the homeserver up.
       serviceConfig = {
         Type = "oneshot";
-        # NB: NOT `RemainAfterExit = true` — we want re-runs from
+        # RemainAfterExit = false so the .path trigger can re-fire
-        # the path trigger to actually re-execute. With
+        # the unit (see docs/persistence.md::matrix-avatar-sync).
-        # RemainAfterExit, systemd treats the service as "still
-        # running" after the first exit and the second trigger
-        # becomes a no-op.
         RemainAfterExit = false;
       };
       path = [
@@ -1276,9 +1242,9 @@ in
       '';
     };
 
-    # Write declared dashboardLinks to the state dir so hive-c0re can read
+    # Write declared dashboardLinks to the state dir so hive-c0re can
-    # them without accessing the container's /etc/ from the host.
+    # read them without accessing the container's /etc/ from the host.
-    # Runs every boot; idempotent (overwrite). Always exits 0.
+    # Best-effort oneshot (always exit 0): docs/conventions.md.
     systemd.services.hive-dashboard-links = lib.mkIf (config.hyperhive.dashboardLinks != [ ]) {
       description = "write declarative dashboardLinks to agent state dir";
       wantedBy = [ "multi-user.target" ];