feat(#3125): reshape the hive-to-swarm OTEL hop by domain

Drops swarm.otel.url (a loopback default an operator had to override on a
split host) in favor of swarm.otel.domain -- the same
gateway.localNames + nginx-vhost-through-the-gateway shape every other
swarm service (authelia, grafana, victoriametrics, ui) already uses. The
hive tier's exporter now reaches it as https://<domain> unconditionally,
resolved locally by dnsmasq on a co-located host and over the real network
otherwise, instead of a config knob nobody sets until they hit the silent
drop.

Costs CA trust on the hive tier: otel.nix wires
lib/hive-ca-trust.nix's trustBundle with hostUnit = true on the
opentelemetry-collector host unit, the same flag #3441/#3442 added for
swarm-controller and hive-c0re.

mara, #3125 comment 58363: "go c".

docs/network.md

@@ -177,8 +177,11 @@ namespace.
 ### Reaching host services (`exposeHostPorts`)
 
 By default agents can only reach the host on 80/443 (+53 DNS), so a
-host-side service on another port — e.g. a dev OTEL collector for
+host-side service on another port — e.g. a dev OTLP collector you want
-`services.hyperhive.otel.endpoint` (see `docs/observability.md`) — is unreachable.
+agents to reach directly — is unreachable. (hyperhive's own telemetry
+needs none of this: `otel.enable` opens its collector's port itself, and
+`otel.endpoint` is the *upstream*, which no agent ever dials. See
+`docs/observability.md`.)
 
 `services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
 listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an

docs/observability.md

@@ -16,18 +16,25 @@ services.hyperhive.otel = {
 };
 ```
 
-`enable` is the single gate. `endpoint` is where telemetry goes upstream —
+`enable` is the single gate. `endpoint` is where telemetry ends up after it
-required when enabled *unless* this host runs the swarm's own metrics store
+leaves the swarm — optional, because the swarm's own metrics store
-(`swarm.victoriametrics.enable`), which is a destination in its own right. With
+(`swarm.victoriametrics.enable`) is a destination in its own right. With both,
-both, telemetry goes to both. See
+telemetry goes to both. See
 [`swarm/services.md`](swarm/services.md#metrics-victoriametrics--grafana).
 
 **There is exactly one way telemetry leaves a hive: through the collector that
 `enable` starts on the host.** Agents never talk to `endpoint` themselves —
 they export unauthenticated to a bridge address only their own containers can
-reach, and the collector forwards upstream with the auth header. So the
+reach. That collector forwards to the swarm's
-upstream credential exists in one place, on the host, and no agent ever holds
+([`swarm/services.md`](swarm/services.md#telemetry-collector-otel)), which is
-a copy.
+the single process holding the upstream credential and the only writer to the
+swarm's store. No agent holds a copy, and neither does this hive.
+
+The hive collector reaches the swarm collector by its gateway name
+(`swarm.otel.domain`, default `otel.<swarm domain>`) — the same DNS-and-CA-trust
+shape every hive-to-swarm-service hop uses, not a URL an operator has to point
+anywhere. A hive that does not run the swarm's services still resolves that
+name through the gateway; nothing here needs setting for the split-host case.
 
 ⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
 same host as the agents and restarts on failure, and telemetry is not the
@@ -42,8 +49,8 @@ port on the bridge interface only. So "unauthenticated to a bridge address"
 means *reachable from an agent container*, not *presents a credential*.
 
 The consequence, stated because it is a choice rather than an oversight: **any
-agent can push arbitrary OTLP, and the collector forwards it upstream under the
+agent can push arbitrary OTLP, and it is forwarded on under the operator's
-operator's credential.** It cannot tell a container's genuine Claude Code stats
+credential.** Neither tier can tell a container's genuine Claude Code stats
 from anything else shaped like OTLP arriving on that port — including data
 smuggled out in resource attributes on an otherwise-legitimate export.
 
@@ -61,33 +68,44 @@ closed by this design, and nothing here should be read as closing it.
 
 Master switch. When true, all other options below take effect.
 
-### `services.hyperhive.otel.endpoint` — string, required when enabled unless the swarm store runs here
+### `services.hyperhive.otel.endpoint` — string, default `""`
 
-Upstream OTLP endpoint URL. Set as `OTEL_EXPORTER_OTLP_ENDPOINT` for every
+Upstream OTLP endpoint URL, read by the swarm's collector. Example:
-agent. Example: `"https://collector.example.com/otel"`.
+`"https://collector.example.com/otel"`.
 
-Leave it empty **only** on a host running `swarm.victoriametrics.enable` — the
+Leave it empty on a swarm running its own metrics store — that store is then
-local store is then the destination and the collector writes there instead.
+the destination. With neither, the swarm collector is refused at eval:
-With neither, `enable` is refused at eval: telemetry with nowhere to go is a
+telemetry with nowhere to go is a misconfiguration, not a quiet no-op.
-misconfiguration, not a quiet no-op.
+
+Not what agents are handed. Their endpoint is this hive's own collector,
+derived from the bridge address, so setting this changes where telemetry
+*ends up* and never what a container is told.
 
 ### `services.hyperhive.otel.protocol` — enum, default `"http/protobuf"`
 
-OTLP wire protocol, passed as `OTEL_EXPORTER_OTLP_PROTOCOL`. Accepted values:
+Wire protocol for the **upstream** link, honoured by the swarm collector's
+exporter. Accepted values:
 - `"http/protobuf"` (default)
 - `"http/json"`
 - `"grpc"`
 
+Agents are not affected: their first hop is this hive's collector, whose
+OTLP/HTTP receiver takes protobuf whatever the upstream wants.
+
 ### `services.hyperhive.otel.headersCredential` — string or null, default `null`
 
 Absolute path to a secret file on the host holding the upstream auth header as
 `NAME=value` (e.g. `Authorization=Bearer <token>`).
 
-**Only the host collector reads it.** It arrives as an `EnvironmentFile` on the
+**Only the swarm's collector reads it** — the one tier that talks to the
-collector's unit, so the value is never read by nix, never copied into the
+upstream. It arrives as an `EnvironmentFile` on that unit, so the value is
-store or the generated config, never passed in argv — and **never forwarded
+never read by nix, never copied into the store or the generated config, never
-into an agent container**. An agent cannot read the hive's upstream credential
+passed in argv — and reaches **neither an agent container nor a hive's own
-because it is never given one.
+collector**. An agent cannot read the upstream credential because it is never
+given one.
+
+Set it on the host running the swarm's services; a hive that only forwards has
+no use for it.
 
 Leave `null` if the upstream needs no auth header; the collector then sends
 none rather than an empty one.
@@ -130,46 +148,56 @@ metrics on process exit, so interval tuning is not required for metrics to be
 exported. A lower value gives more frequent intermediate flushes within
 long-running turns — cosmetic, not a correctness knob.
 
-## The host collector
+## The two collectors
 
-`enable` starts an OpenTelemetry collector on the host. It is not optional and
+Telemetry crosses two collectors, and which one you configure depends on what
-there is no second path — that is the whole point:
+the host is:
+
+| | runs where | receives from | does |
+|---|---|---|---|
+| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
+| **swarm tier** — `swarm.otel.enable` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
+
+An all-local host runs both, and needs nothing said about the hop between them.
 
 ```nix
 services.hyperhive.otel = {
   enable            = true;
   endpoint          = "https://collector.example.com/otel";  # the upstream
-  headersCredential = "/run/secrets/otel-headers";           # only the host reads it
+  headersCredential = "/run/secrets/otel-headers";           # only the swarm tier reads it
 };
 ```
 
-**Why it isn't a knob.** Exporting straight to `endpoint` means every agent
+**Why the hive tier isn't optional.** Exporting straight to `endpoint` means
-needs the credential to authenticate — and the harness delivers that token into
+every agent needs the credential to authenticate — and the harness delivers
-the agent's own `~/.claude/settings.json`, a file the agent can read. `0600`
+that token into the agent's own `~/.claude/settings.json`, a file the agent can
-protects it from other containers, not from the agent itself. As long as the
+read. `0600` protects it from other containers, not from the agent itself. As
-direct path stays *selectable*, that hole stays selectable; an option that can
+long as the direct path stays *selectable*, that hole stays selectable; an
-reintroduce it is a hole with extra steps.
+option that can reintroduce it is a hole with extra steps.
 
-**`endpoint` keeps meaning "where telemetry goes upstream."** The collector
+**Why the tiers stay separate on one box.** They are not collapsed when
-does not redefine it — the agent-facing value is *derived*
+co-located: an all-local hive is a statement about *where* processes run, not
+about the shape of the deployment. A boundary that disappears locally is one
+the local deployment stops testing.
+
+**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
+redefines it — the agent-facing value is *derived*
 (`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
 keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
 automatically; there is nothing to open by hand.
 
-What the collector *added* is a second destination: on a host running the
-swarm's metrics store it writes there too, so `endpoint` is no longer the only
-place telemetry can land — and no longer the only way to have one.
-
 ### `services.hyperhive.otel.collector.port` — port, default `4318`
 
-The OTLP/HTTP port the collector listens on, bound to the bridge IP only.
+The OTLP/HTTP port the hive tier listens on, bound to the bridge IP only. The
+swarm tier has its own (`swarm.otel.port`, default `4319`) — they share a
+network namespace when co-located, so the two must differ.
 
 ### `services.hyperhive.otel.collector.upstreamHeaderName` — string, default `"Authorization"`
 
-Name of the header the collector sends upstream. The **value** comes from the
+Name of the header the swarm tier sends upstream. The **value** comes from the
 credential file at runtime (`EnvironmentFile` → `${env:<name>}`), never from
 nix — so header names are config and header values are secrets, which is the
-only split the collector's static header map can express.
+only split a static header map can express.
 
 ⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
 `otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC

docs/swarm/services.md

@@ -93,15 +93,42 @@ login form is switched off whenever SSO is configured. If you enable
 Grafana on a host with no authelia, the form stays on and Grafana's
 default `admin`/`admin` applies; change it before exposing that host.
 
-**Where the data comes from.** With `otel.enable` on, the hive's OTEL
+**Where the data comes from.** The swarm's OTEL collector, below.
-collector writes into this store as well as to any upstream endpoint —
-both, not one or the other, since a local store is for looking at this
-swarm and an upstream is for whoever aggregates across swarms. That also
-means `otel.endpoint` is no longer required when the store runs here: a
-hive with a local store already has somewhere for telemetry to go. See
-[`../observability.md`](../observability.md).
 
 Neither container is reachable except through the gateway: both bind
 loopback, and VictoriaMetrics' write endpoint takes no credential, so
 the collector is the only intended writer.
 
+### Telemetry collector (OTEL)
+
+The swarm's collector receives from every hive's own collector and is the
+only process that decides where telemetry goes: it writes the store above
+and exports to `otel.endpoint`, doing both when both are configured. It
+also holds the upstream credential, which is why no hive and no agent
+needs one.
+
+It follows `swarm.enableRequiredServices` like the services above, in a
+`swarm-otel` container. Its `swarm.otel.port` defaults to `4319` rather
+than OTLP's usual `4318`, which the hive tier already uses — swarm
+containers share the host's network namespace, so two collectors on one
+port is a coin toss at runtime rather than an error at build time.
+
+Every hive's own collector reaches this one by its gateway name,
+`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
+by-domain-through-the-gateway shape every other swarm service uses, not a
+loopback URL an operator has to redirect. There is nothing to set on a hive
+that does not run the swarm's services; the name resolves through the
+gateway either way.
+
+| Option | When you'd touch it |
+|---|---|
+| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
+| `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
+
+With neither `otel.endpoint` nor the store enabled, this collector is
+refused at eval — a tier that receives samples and drops them looks
+healthy while losing data.
+
+Agent-side configuration, and what a hive's own collector does, are in
+[`../observability.md`](../observability.md).
+

frontend/packages/swarm-ui/src/pages/CreateAgentPage.css

@@ -1,16 +1,19 @@
 /* <CreateAgentPage> — the create-agent form. Input/button chrome now
-   comes from the shared `ui/` form kit (`TextField`/`Button`); this
+   comes from the shared `ui/` form kit (`TextField`/`SelectField`/
-   file only owns the page's own layout + copy. Reuses the same base16
+   `Button`); this file only owns the page's own layout + copy. Column,
-   slots (../theme.css) every other component draws from. */
+   not row: with a second field (hive) added, a row layout put fields of
+   different natural widths on one baseline and looked misaligned —
+   mara: "make it a col". Reuses the same base16 slots (../theme.css)
+   every other component draws from. */
 .create-agent-intro {
   margin: 0 0 1.5em;
   color: var(--muted);
 }
 .create-agent-form {
   display: flex;
-  align-items: flex-end;
+  flex-direction: column;
+  align-items: flex-start;
   gap: 0.75em;
-  flex-wrap: wrap;
 }
 .create-agent-result {
   margin-top: 1em;

frontend/packages/swarm-ui/src/pages/CreateAgentPage.tsx

@@ -16,19 +16,35 @@
 // one path segment deep — a real bug, filed separately rather than
 // fixed here, but reason enough to avoid a nested route today.
 //
-// First real form in this package — its name field + submit button now
+// `hive` field added because agent creation had no way to record which
-// come from the shared `ui/` form kit (`TextField`/`Button`) rather than
+// hive an agent runs on. `POST /api/agents` now requires it, so the
-// page-scoped input/button chrome, so a hive-picker soon landing on this
+// roster fetched off `GET /api/hives` backs a required select here
-// same page has something to reuse instead of copying this page's CSS.
+// rather than a free-text field — a spawn target has to be one of the
-import { useState } from 'preact/hooks';
+// swarm's actual hives, same reasoning `Ident.parse` gets client-side
+// pattern validation for `name`.
+//
+// Its name field, hive select, and submit button all come from the
+// shared `ui/` form kit (`TextField`/`SelectField`/`Button`) rather than
+// page-scoped input/button chrome — see that kit's own comments for why.
+import { useEffect, useState } from 'preact/hooks';
 import { Link } from 'wouter-preact';
 import { ApiErrorPanel } from '@hive/shared/api-error-panel.js';
 import { readApiError, type ProblemDetails } from '@hive/shared/api-error.js';
 import { Panel } from '../ui/panel/Panel.js';
 import { TextField } from '../ui/text-field/TextField.js';
+import { SelectField, type SelectOption } from '../ui/select-field/SelectField.js';
 import { Button } from '../ui/button/Button.js';
 import './CreateAgentPage.css';
 
+// Mirrors swarm-controller's `HiveEntry` — same shape `OverviewPage`
+// consumes off `/api/hives/status`, but this page hits the plain
+// `/api/hives` roster (no status/freshness needed, just "what hives
+// exist to spawn into").
+interface HiveEntry {
+  name: string;
+  domain: string;
+}
+
 interface CreateAgentResponse {
   node_id: number;
 }
@@ -56,8 +72,37 @@ const NAME_PATTERN = '[a-z0-9\\-]{1,63}';
 
 export function CreateAgentPage() {
   const [name, setName] = useState('');
+  const [hive, setHive] = useState('');
+  // `null` = still loading, `[]` = loaded but empty (a real, if unusual,
+  // swarm state) — distinct from "not fetched yet" so the placeholder
+  // option's label can tell the two apart.
+  const [hives, setHives] = useState<HiveEntry[] | null>(null);
+  const [hivesError, setHivesError] = useState<ProblemDetails | null>(null);
   const [result, setResult] = useState<SubmitState>({ status: 'idle' });
 
+  // Loaded once, not re-fetched on submit: the roster changes rarely
+  // enough (a nix-level swarm config change) that staleness within one
+  // page visit isn't worth a second round-trip per keystroke/submit.
+  useEffect(() => {
+    (async () => {
+      const r = await fetch('/api/hives');
+      if (!r.ok) {
+        setHivesError(await readApiError(r));
+        return;
+      }
+      const data = (await r.json()) as HiveEntry[];
+      setHives(data);
+      // Single-hive swarms are the common case — default it rather than
+      // making the operator pick the only option. Multi-hive swarms get
+      // no default (the placeholder stays selected), so `required` on
+      // the select below forces an explicit choice per mara's scope.
+      if (data.length === 1) setHive(data[0].name);
+    })().catch((e: unknown) => setHivesError({ detail: String(e) }));
+  }, []);
+
+  const hiveOptions: SelectOption[] = (hives ?? []).map((h) => ({ value: h.name, label: h.name }));
+  const hivesReady = hives !== null && hives.length > 0;
+
   async function submit(e: Event) {
     e.preventDefault();
     setResult({ status: 'submitting' });
@@ -65,7 +110,7 @@ export function CreateAgentPage() {
       const r = await fetch('/api/agents', {
         method: 'POST',
         headers: { 'content-type': 'application/json' },
-        body: JSON.stringify({ name }),
+        body: JSON.stringify({ name, hive }),
       });
       if (!r.ok) {
         setResult({ status: 'error', problem: await readApiError(r) });
@@ -85,6 +130,7 @@ export function CreateAgentPage() {
         Create a new agent's swarm-level identity. This only queues the job — check{' '}
         <Link href="/jobs">jobs</Link> to watch it settle.
       </p>
+      {hivesError && <ApiErrorPanel context="failed to load the hive list" problem={hivesError} />}
       <form class="create-agent-form" onSubmit={submit}>
         <TextField
           id="agent-name"
@@ -95,7 +141,17 @@ export function CreateAgentPage() {
           required
           onInput={setName}
         />
-        <Button variant="primary" type="submit" disabled={result.status === 'submitting'}>
+        <SelectField
+          id="agent-hive"
+          label="hive"
+          value={hive}
+          onChange={setHive}
+          options={hiveOptions}
+          required
+          disabled={!hivesReady}
+          placeholder={!hives ? 'loading…' : hives.length === 0 ? 'no hives configured' : 'select a hive'}
+        />
+        <Button variant="primary" type="submit" disabled={result.status === 'submitting' || !hivesReady}>
           {result.status === 'submitting' ? 'creating…' : 'create'}
         </Button>
       </form>

frontend/packages/swarm-ui/src/ui/select-field/SelectField.tsx

@@ -18,6 +18,8 @@ export function SelectField({
   onChange,
   options,
   required,
+  disabled,
+  placeholder,
 }: {
   id: string;
   label: string;
@@ -25,6 +27,13 @@ export function SelectField({
   onChange: (value: string) => void;
   options: SelectOption[];
   required?: boolean;
+  disabled?: boolean;
+  // Rendered as a disabled, always-first `value=""` option — for a
+  // required select with no default (e.g. a multi-hive roster: no
+  // single right answer to pre-select), so the control shows something
+  // other than silently landing on whatever option happens to be first.
+  // Omit when the field always has a real default (`value` never `''`).
+  placeholder?: string;
 }) {
   return (
     <FormField label={label} htmlFor={id}>
@@ -33,8 +42,14 @@ export function SelectField({
         class="ui-form-control"
         value={value}
         required={required}
+        disabled={disabled}
         onChange={(e) => onChange((e.target as HTMLSelectElement).value)}
       >
+        {placeholder !== undefined && (
+          <option value="" disabled>
+            {placeholder}
+          </option>
+        )}
         {options.map((o) => (
           <option key={o.value} value={o.value}>
             {o.label}

nix/host-modules/default.nix

@@ -27,6 +27,7 @@
     ./swarm-nats.nix
     ./swarm-controller.nix
     ./swarm-grafana.nix
+    ./swarm-otel.nix
     ./swarm-snapshot-store.nix
     ./swarm-ui.nix
     ./swarm-victoriametrics.nix

nix/host-modules/otel.nix

@@ -1,28 +1,69 @@
-# Hive-wide OTEL stats export. Set ONCE here at host level; the
+# Hive-wide OTEL stats export, and the HIVE tier of the collector pair.
-# meta-flake renderer (`hive-c0re/src/meta.rs::otel_config`) reads the
+# Set ONCE here at host level; the meta-flake renderer
-# HYPERHIVE_OTEL_* env exported off hive-c0re's unit (see
+# (`hive-c0re/src/meta.rs::otel_config`) reads the HYPERHIVE_OTEL_* env
-# ./hive-c0re) and injects the matching `hyperhive.otel.*` build-time
+# exported off hive-c0re's unit (see ./hive-c0re) and injects the matching
-# config into EVERY agent (mirroring the CA-cert injection), so each
+# `hyperhive.otel.*` build-time config into EVERY agent (mirroring the
-# agent's harness exports its own Claude Code stats directly to the
+# CA-cert injection), so each agent's harness exports its own Claude Code
-# collector. There is no per-agent opt-in — this is the single switch
+# stats directly to the collector. There is no per-agent opt-in — this is
-# for the whole hive.
+# the single switch for the whole hive.
+#
+# This tier receives from this hive's agents and forwards to the swarm's
+# collector (./swarm-otel.nix). It holds no credential and picks no
+# destination: an agent's samples cross a hive boundary exactly once, and
+# what happens after that is the swarm's decision, not a hive's. The
+# upstream options declared below describe that far end and are read one
+# tier up — they stay here because they mean what they have always meant.
 {
+  pkgs,
   lib,
   config,
   ...
 }:
+let
+  # This tier now reaches the swarm's collector by name through the
+  # gateway (`swarm-otel.nix`'s `domain`) instead of a loopback URL, so it
+  # needs the same hive-CA trust every other host consumer of an `https://`
+  # swarm-service name needs — see `swarm-controller.nix` for the sibling
+  # wiring this copies.
+  #
+  # `hostUnit`: `opentelemetry-collector` is a host systemd service, not a
+  # container, so it reads the CA from the host path and the bundle oneshot
+  # waits on `hive-tls-ca.service` itself. `enable`: `imports` is
+  # unconditional at the host's top level, so without it a hive with this
+  # tier off would still get a bundle oneshot and a phantom
+  # `opentelemetry-collector` service holding an `SSL_CERT_FILE`.
+  caTrust = import ./lib/hive-ca-trust.nix {
+    inherit lib;
+    tlsCfg = config.services.hyperhive.tls;
+    gatewayCfg = config.services.hyperhive.gateway;
+  };
+in
 {
+  imports = [
+    (caTrust.trustBundle {
+      inherit pkgs;
+      name = "hive-otel";
+      consumers = [ "opentelemetry-collector" ];
+      hostUnit = true;
+      enable = config.services.hyperhive.otel.enable;
+    })
+  ];
+
   options.services.hyperhive.otel = {
     enable = lib.mkEnableOption ''
       hive-wide export of every agent's Claude Code stats (token usage,
       cost, tool calls) to an OTLP endpoint via Claude Code's built-in
       OpenTelemetry. One switch for all agents.
 
-      Enabling this also runs a collector on the host: there is exactly
+      Enabling this also runs this hive's collector on the host: there is
-      one way telemetry leaves this hive, and it is through that
+      exactly one way telemetry leaves this hive, and it is through that
       collector. Agents export unauthenticated to a bridge address only
-      their own containers can reach, and the collector is the only
+      their own containers can reach.
-      holder of the upstream credential — an agent never sees it.
+
+      That collector forwards to the swarm's
+      ({option}`services.hyperhive.swarm.otel.enable`), which holds the
+      upstream credential and writes the swarm's store. So an agent never
+      sees the credential, and neither does this tier.
 
       ⚠️ The collector is therefore in the path of all telemetry. It runs
       on the same host as the agents and restarts on failure, and
@@ -36,15 +77,19 @@
       default = "";
       example = "https://collector.example.com/otel";
       description = ''
-        Upstream OTLP endpoint, set as `OTEL_EXPORTER_OTLP_ENDPOINT` for
+        Upstream OTLP endpoint: where telemetry ultimately goes, after it
-        every agent.
+        has left the swarm.
 
-        Required when `enable` is true, **unless** this host runs the
+        Read by the swarm's collector
-        swarm's metrics store
+        ({option}`services.hyperhive.swarm.otel.enable`), which is the only
-        ({option}`services.hyperhive.swarm.victoriametrics.enable`) — that
+        tier that holds the upstream credential. An agent is handed the
-        store is a destination in its own right, and with both configured
+        *first* hop instead — this hive's own collector — so this value is
-        telemetry goes to both. With neither, `enable` is refused rather
+        never given to a container.
-        than silently exporting nowhere.
+
+        Optional. Leave it empty and the swarm's own metrics store
+        ({option}`services.hyperhive.swarm.victoriametrics.enable`) is the
+        destination; that is a complete deployment, not a degraded one.
+        Set both and telemetry goes to both.
       '';
     };
 
@@ -56,7 +101,12 @@
       ];
       default = "http/protobuf";
       description = ''
-        OTLP wire protocol, set as `OTEL_EXPORTER_OTLP_PROTOCOL`.
+        OTLP wire protocol for the **upstream** link, honoured by the
+        swarm collector's exporter.
+
+        Not what agents speak: their first hop is this hive's collector,
+        whose OTLP/HTTP receiver takes protobuf whatever the upstream
+        wants (see `hive-c0re/environment.nix`).
       '';
     };
 
@@ -73,12 +123,12 @@
         upstream auth header as `NAME=value` (e.g.
         `Authorization=Bearer <token>`).
 
-        **Only the host-side collector reads this.** It reaches the
+        **Only the swarm's collector reads this** — the one tier that
-        collector as an `EnvironmentFile`, so the value is never read by
+        talks to the upstream. It arrives as an `EnvironmentFile`, so the
-        nix, never copied into the store or the generated config, and
+        value is never read by nix, never copied into the store or the
-        never passed in argv — and it is never forwarded into an agent
+        generated config, and never passed in argv; and it reaches
-        container, which is the point of the collector existing. Must be
+        neither an agent container nor this hive's own collector, which
-        absolute.
+        is the point of the tiers existing. Must be absolute.
 
         Leave null if the upstream needs no auth header; the collector
         then sends none rather than an empty one.
@@ -161,131 +211,67 @@
     };
   };
 
-  config = lib.mkMerge [
+  config = lib.mkIf config.services.hyperhive.otel.enable (
-    (lib.mkIf config.services.hyperhive.c0re.enable {
+    let
-      assertions = lib.optionals config.services.hyperhive.otel.enable [
+      otel = config.services.hyperhive.otel;
-        {
+      listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
-          # Telemetry has to go SOMEWHERE, but "somewhere" stopped meaning
+      swarmName = "otlphttp/swarm";
-          # "an upstream endpoint" once the swarm grew its own store: a hive
+    in
-          # running `swarm.victoriametrics` is a complete destination on its
+    {
-          # own, and requiring an external endpoint as well would make the
+      # Reachable from agent containers and nowhere else: this opens
-          # all-local mode impossible to express.
+      # the port on the bridge interface only.
-          #
+      services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
-          # This only ever relaxes the old rule — every config that passed
-          # before still passes.
-          assertion =
-            config.services.hyperhive.otel.endpoint != ""
-            || config.services.hyperhive.swarm.victoriametrics.enable;
-          message = ''
-            services.hyperhive.otel.enable is true but telemetry has nowhere
-            to go: services.hyperhive.otel.endpoint is empty and
-            services.hyperhive.swarm.victoriametrics.enable is false.
 
-            Set the endpoint to export upstream, or enable the swarm's
+      services.opentelemetry-collector = {
-            metrics store to keep telemetry on this host.
+        enable = true;
-          '';
+        # `validateConfigFile` defaults to `isStorePath configFile`,
-        }
+        # and `configFile` is null on the `settings` path — so the
-      ];
+        # upstream default is OFF for exactly the way this module
-    })
+        # configures it. Turning it on runs `otelcol validate` at
+        # build time, which is the collector checking its own config.
+        # ⚠️ It is a PARSER, not a wiring check, and the gap is wider
+        # than "no sample was sent": measured 2026-08-15, `validate`
+        # ACCEPTS a receiver naming an auth extension that is absent
+        # from the build, and the collector then dies at startup with
+        # `Failed to start component`. So a green build does not
+        # prove this config STARTS, never mind that a sample arrives.
+        validateConfigFile = true;
+        settings = {
+          receivers.otlp.protocols.http.endpoint = listen;
 
-    (lib.mkIf config.services.hyperhive.otel.enable (
+          # One destination, and it is the swarm's collector. This tier
-      (
+          # holds no upstream credential and writes no store: it receives
-        let
+          # from this hive's agents and forwards, which is the whole of
-          otel = config.services.hyperhive.otel;
+          # its job. Everything that decides where telemetry ultimately
-          listen = "${config.services.hyperhive.network.bridgeIp}:${toString otel.collector.port}";
+          # goes lives one tier up, in ./swarm-otel.nix.
-          # `otel.protocol` describes the UPSTREAM link and always did.
+          exporters.${swarmName} = {
-          # Inserting a collector splits the path in two, and the
+            # ⚠️ Plain `endpoint`, and the exporter beside this one in
-          # upstream half is the one that has to keep honouring it — so
+            # ./swarm-otel.nix warns against exactly that spelling — read
-          # the exporter is chosen by it, rather than the option quietly
+            # both before "fixing" either. The difference is the far end,
-          # becoming "how agents talk to the collector". The agent half
+            # not the exporter: `endpoint` is a BASE that otlphttp appends
-          # is pinned to OTLP/HTTP by the receiver below (derived in
+            # `/v1/metrics` to, which is precisely the path an OTLP/HTTP
-          # hive-c0re/environment.nix).
+            # receiver serves. VictoriaMetrics is the odd one out, serving
-          vmCfg = config.services.hyperhive.swarm.victoriametrics;
+            # OTLP at `/opentelemetry/api/v1/push`, and that is why the
-          # Two independent destinations, either of which may be absent: an
+            # store exporter needs `metrics_endpoint` while this one must
-          # upstream the operator named, and the swarm's own store when this
+            # not have it.
-          # host runs it. The assertion above guarantees at least one.
-          upstreamConfigured = otel.endpoint != "";
-          localStore = vmCfg.enable;
-
-          storeName = "otlphttp/victoriametrics";
-          store = {
-            # ⚠️ `metrics_endpoint`, NOT `endpoint`, and the difference is
-            # invisible until you read the far end: `endpoint` is a BASE that
-            # otlphttp appends `/v1/metrics` to, while VictoriaMetrics serves
-            # OTLP at `/opentelemetry/api/v1/push`. With `endpoint` the
-            # collector still answers 200 to its own clients and the samples
-            # are silently posted to a path that does not exist.
-            # `metrics_endpoint` is used verbatim.
             #
-            # Measured end-to-end rather than read: a real sample crossed a
+            # By name through the gateway, not a loopback literal: a
-            # real collector into a real store, and the same probe with
+            # loopback literal is correct only while listener and caller
-            # `endpoint` never arrived — see `state/probe-3265-collector-to-vm.sh`.
+            # share a netns, an assumption that has cost this project two
-            metrics_endpoint = "http://127.0.0.1:${toString vmCfg.port}/opentelemetry/api/v1/push";
+            # outages, and it is exactly the split-host case a swarm
+            # service name exists to make a config fact rather than a code
+            # change. `https://` because that name resolves through the
+            # gateway even on a co-located host — see `caTrust` above for
+            # the trust half that makes this verify.
+            endpoint = "https://${config.services.hyperhive.swarm.otel.domain}";
           };
 
-          grpcUpstream = otel.protocol == "grpc";
+          service.pipelines.metrics = {
-          upstreamName = if grpcUpstream then "otlp" else "otlphttp";
+            receivers = [ "otlp" ];
-          upstream = {
+            exporters = [ swarmName ];
-            endpoint = otel.endpoint;
-          }
-          // lib.optionalAttrs (otel.headersCredential != null) {
-            # The value is interpolated by the collector at runtime from
-            # its environment, never by nix. `EnvironmentFile` below is
-            # what puts it there. No credential configured means no
-            # header at all — an upstream that needs no auth is a
-            # legitimate deployment, and rendering `${env:…}` for a
-            # variable nothing sets would send the literal.
-            headers.${otel.collector.upstreamHeaderName} = "\${env:${otel.collector.upstreamHeaderName}}";
-          }
-          // lib.optionalAttrs (otel.protocol == "http/json") { encoding = "json"; };
-        in
-        {
-          # Reachable from agent containers and nowhere else: this opens
-          # the port on the bridge interface only.
-          services.hyperhive.network.exposeHostPorts = [ otel.collector.port ];
-
-          services.opentelemetry-collector = {
-            enable = true;
-            # `validateConfigFile` defaults to `isStorePath configFile`,
-            # and `configFile` is null on the `settings` path — so the
-            # upstream default is OFF for exactly the way this module
-            # configures it. Turning it on runs `otelcol validate` at
-            # build time, which is the collector checking its own config.
-            # ⚠️ It is a PARSER, not a wiring check, and the gap is wider
-            # than "no sample was sent": measured 2026-08-15, `validate`
-            # ACCEPTS a receiver naming an auth extension that is absent
-            # from the build, and the collector then dies at startup with
-            # `Failed to start component`. So a green build does not
-            # prove this config STARTS, never mind that a sample arrives.
-            validateConfigFile = true;
-            settings = {
-              receivers.otlp.protocols.http.endpoint = listen;
-              exporters =
-                lib.optionalAttrs upstreamConfigured { ${upstreamName} = upstream; }
-                // lib.optionalAttrs localStore { ${storeName} = store; };
-              service.pipelines.metrics = {
-                receivers = [ "otlp" ];
-                # Fan-out, not a choice: with both configured the same
-                # samples go upstream AND into the swarm's store. A local
-                # store is for looking at this swarm; an upstream is for
-                # whoever aggregates across swarms, and neither replaces
-                # the other.
-                exporters = lib.optional upstreamConfigured upstreamName ++ lib.optional localStore storeName;
-              };
-            };
           };
-
+        };
-          # The credential file is already `NAME=value`, which is
+      };
-          # systemd's EnvironmentFile format — so the secret reaches the
+    }
-          # process as an environment variable without ever being read by
+  );
-          # nix, written to the store, or passed in argv.
-          systemd.services.opentelemetry-collector.serviceConfig =
-            lib.optionalAttrs (otel.headersCredential != null)
-              {
-                EnvironmentFile = otel.headersCredential;
-              };
-        }
-      )
-    ))
-  ];
 }

nix/host-modules/swarm-otel.nix

@@ -0,0 +1,291 @@
+# The swarm's telemetry collector: one per swarm, in a `swarm-otel`
+# nixos-container beside the swarm's other shared services.
+#
+# Two tiers, and they are separate on purpose:
+#
+#   - `otel.nix` is the **hive** tier. It receives from this hive's agents
+#     on the bridge and forwards, and it holds no upstream credential.
+#   - this is the **swarm** tier. It is the only holder of the upstream
+#     credential, the only writer to the swarm's metrics store, and the
+#     place that will stamp `hive=` from the authenticated connection
+#     rather than from anything a sender can choose.
+#
+# On a host that runs both, both processes run. They are not collapsed:
+# all-local is a statement about *where* processes run, not about what
+# shape the deployment has, and a local tier boundary that disappears is
+# one the local deployment stops testing. `hive=` attribution is the
+# property that would differ, and the ingest auth that makes it
+# unforgeable is built on this boundary existing.
+#
+# A container rather than a second host unit, for the same reason every
+# sibling swarm service is one — and because `services.opentelemetry-collector`
+# is a singleton NixOS option, already spoken for on the host by the hive
+# tier. A container gets its own evaluation and therefore its own collector.
+{
+  pkgs,
+  lib,
+  config,
+  ...
+}:
+let
+  cfg = config.services.hyperhive.swarm.otel;
+  swarmCfg = config.services.hyperhive.swarm;
+  otelCfg = config.services.hyperhive.otel;
+  vmCfg = config.services.hyperhive.swarm.victoriametrics;
+  hyperhiveCfg = config.services.hyperhive;
+  gatewayCfg = hyperhiveCfg.gateway;
+  swarmDomain = hyperhiveCfg.swarm.domain;
+
+  # Total on a null swarm domain for the same reason every sibling module is:
+  # the required-domain assertion in hive-network.nix should be what an
+  # operator sees, not a coercion error from here.
+  domainBase = if swarmDomain == null then "invalid" else swarmDomain;
+in
+{
+  options.services.hyperhive.swarm.otel = {
+    enable = lib.mkOption {
+      type = lib.types.bool;
+      default = false;
+      description = ''
+        Run the swarm's telemetry collector on this host.
+
+        Asserted from `swarm.enableRequiredServices` in
+        ./swarm-required-services.nix, with the metrics pair this
+        collector feeds: a swarm has one of these, and it belongs
+        wherever the shared services live rather than on every hive.
+
+        A hive that does not run it still runs its own hive-tier collector
+        (`services.hyperhive.otel.enable`) and reaches this one by name, at
+        {option}`services.hyperhive.swarm.otel.domain`.
+      '';
+    };
+
+    machine = lib.mkOption {
+      type = lib.types.str;
+      readOnly = true;
+      default = "swarm-otel";
+      description = ''
+        Name of the nixos-container this collector runs in — also the
+        `machinectl` name, so other modules may read it rather than
+        repeating the literal.
+      '';
+    };
+
+    port = lib.mkOption {
+      type = lib.types.port;
+      default = 4319;
+      description = ''
+        Port this collector's OTLP/HTTP receiver listens on.
+
+        ⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
+        hive tier already uses it (`services.hyperhive.otel.collector.port`)
+        and every swarm container shares the host's network namespace. Two
+        listeners claiming one port on one host is not a build failure —
+        it is a runtime coin toss over which one gets it, with nothing in
+        any log saying so. The same collision cost a release when grafana
+        and the forge both defaulted to 3000.
+      '';
+    };
+
+    telemetryPort = lib.mkOption {
+      type = lib.types.port;
+      default = 8889;
+      description = ''
+        Port this collector serves its **own** metrics on — queue depth,
+        refused and dropped samples, exporter failures. How you find out
+        that telemetry is being lost, so it is worth keeping rather than
+        switching off.
+
+        ⚠️ **Deliberately not 8889's neighbour 8888**, which is the
+        collector's built-in default and therefore what the hive tier
+        already binds. Two collectors share a network namespace whenever
+        they are co-located, and unlike the OTLP port this one appears
+        nowhere in either config — it is a default inside the binary, so
+        nothing that compares configured ports can see the clash. The
+        second collector to start simply dies with
+        `bind: address already in use`.
+      '';
+    };
+
+    domain = lib.mkOption {
+      type = lib.types.str;
+      default = "otel.${domainBase}";
+      defaultText = lib.literalExpression ''"otel.''${services.hyperhive.swarm.domain}"'';
+      description = ''
+        Name the gateway serves this on. A sibling of the swarm's other
+        service names, so the swarm-services sub-CA can issue for it — see
+        `hive-tls.nix` for why a service name being a sibling rather than a
+        child decides which CA may sign it.
+
+        This is what the **hive** tier's exporter reaches — the hive
+        collector is a plain producer against this name exactly like every
+        other client of a swarm service, resolved locally by dnsmasq on a
+        co-located host and over the real network otherwise. There is no
+        separate loopback-vs-remote knob to get wrong: `swarm-nats` is the
+        deliberate exception to this pattern (its cross-hive reach is the
+        wireguard mesh, not the gateway), everything else in this swarm
+        addresses its siblings by name.
+      '';
+    };
+  };
+
+  config = lib.mkIf (config.services.hyperhive.enable && cfg.enable) {
+    # The gateway name, inside `cfg.enable` — that guard is the load-bearing
+    # part. Every hive in a swarm may know this collector exists, but only
+    # the host that RUNS it may claim the name; a client hive declaring the
+    # vhost would answer for a service it does not have.
+    services.hyperhive.gateway.localNames = [ cfg.domain ];
+
+    # OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
+    # every sibling swarm service: TLS terminates here, then plain http to
+    # the co-located container over loopback (shared netns, like the store
+    # this collector writes to).
+    services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
+      listen = gatewayCfg.lib.listen;
+      extraConfig = gatewayCfg.lib.securityHeaders;
+      locations."/" = {
+        proxyPass = "http://127.0.0.1:${toString cfg.port}";
+      };
+    };
+
+    assertions = [
+      {
+        # The tier exists to hold the upstream credential and to write the
+        # swarm's store. With neither, it is a process that receives
+        # samples and drops them — which looks healthy and loses data.
+        assertion = otelCfg.endpoint != "" || vmCfg.enable;
+        message = ''
+          services.hyperhive.swarm.otel.enable is true but this collector
+          has nowhere to send what it receives:
+          services.hyperhive.otel.endpoint is empty and
+          services.hyperhive.swarm.victoriametrics.enable is false.
+
+          Set the endpoint to export upstream, or enable the swarm's
+          metrics store.
+        '';
+      }
+    ];
+
+    containers.${cfg.machine} = {
+      autoStart = true;
+      ephemeral = false;
+      # Shared host netns, like every sibling swarm service: the hive tier
+      # reaches this collector, and this collector reaches the metrics
+      # store, without either crossing a network boundary that would need
+      # its own trust material.
+      privateNetwork = false;
+
+      # The upstream credential is operator-provided and lives on the host.
+      # Read-only, and only when one is configured — binding a path that
+      # does not exist makes nixos-container refuse to start the container,
+      # which is a stall several layers from its cause.
+      bindMounts = lib.optionalAttrs (otelCfg.headersCredential != null) {
+        ${otelCfg.headersCredential} = {
+          hostPath = otelCfg.headersCredential;
+          isReadOnly = true;
+        };
+      };
+
+      config =
+        { ... }:
+        {
+          # This tier is the one that resolves an operator-configured
+          # hostname: `otel.endpoint` is an external URL, and reaching it
+          # is the entire reason this container holds a credential. The
+          # `/etc/resolv.conf` nixos-containers copies in is a snapshot
+          # taken once at boot, so without this the upstream export
+          # depends on the host's file having been right at that instant.
+          imports = [
+            (import ./swarm-container-resolver.nix {
+              inherit (config.services.hyperhive.network) bridgeIp;
+              dnsConsumers = [ "opentelemetry-collector.service" ];
+            })
+          ];
+
+          system.stateVersion = config.system.stateVersion;
+          networking.firewall.enable = false;
+          # Keep the host-copied /etc/resolv.conf intact — same reasoning
+          # as the sibling swarm containers.
+          networking.resolvconf.enable = lib.mkForce false;
+
+          services.opentelemetry-collector = {
+            enable = true;
+            package = pkgs.opentelemetry-collector-contrib;
+            # Runs `otelcol validate` at build time. ⚠️ A parser, not a
+            # wiring check: it accepts a receiver naming an absent
+            # extension, and the collector then dies at startup. A green
+            # build does not prove this config starts, never mind that a
+            # sample arrives — which is why this module's gate pushes a
+            # real sample through both tiers into the store.
+            validateConfigFile = true;
+            settings = {
+              receivers.otlp.protocols.http.endpoint = "127.0.0.1:${toString cfg.port}";
+
+              exporters =
+                lib.optionalAttrs vmCfg.enable {
+                  # `metrics_endpoint`, NOT `endpoint`: the latter is a
+                  # base that otlphttp appends `/v1/metrics` to, while
+                  # VictoriaMetrics serves OTLP at
+                  # `/opentelemetry/api/v1/push`. With `endpoint` the
+                  # collector answers 200 to its own clients and posts the
+                  # samples to a path that does not exist. Measured
+                  # end-to-end, not read — `state/probe-3265-collector-to-vm.sh`.
+                  "otlphttp/victoriametrics".metrics_endpoint =
+                    "http://127.0.0.1:${toString vmCfg.port}/opentelemetry/api/v1/push";
+                }
+                // lib.optionalAttrs (otelCfg.endpoint != "") {
+                  ${if otelCfg.protocol == "grpc" then "otlp" else "otlphttp"} = {
+                    endpoint = otelCfg.endpoint;
+                  }
+                  // lib.optionalAttrs (otelCfg.headersCredential != null) {
+                    # Interpolated by the collector from its environment at
+                    # runtime, never by nix: `EnvironmentFile` below is what
+                    # puts it there, so the value is not read into the store.
+                    headers.${otelCfg.collector.upstreamHeaderName} = "\${env:${otelCfg.collector.upstreamHeaderName}}";
+                  }
+                  // lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; };
+                };
+
+              # Moves this collector's self-metrics off the built-in
+              # default of `localhost:8888`, which the hive tier holds.
+              #
+              # ⚠️ `metrics.address` is the spelling that looks right and
+              # is REJECTED by this collector version — measured, not
+              # read: `'migration.MetricsConfigV030' has invalid keys:
+              # address`. `readers` is the schema it accepts, and the
+              # difference is a startup failure rather than a warning.
+              service.telemetry.metrics.readers = [
+                {
+                  pull.exporter.prometheus = {
+                    host = "127.0.0.1";
+                    port = cfg.telemetryPort;
+                  };
+                }
+              ];
+
+              service.pipelines.metrics = {
+                receivers = [ "otlp" ];
+                # Fan-out, not a choice: with both configured the same
+                # samples go upstream AND into the swarm's store. The store
+                # is for looking at this swarm; the upstream is for whoever
+                # aggregates across swarms, and neither replaces the other.
+                exporters =
+                  lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
+                  ++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
+              };
+            };
+          };
+
+          # The credential file is already `NAME=value`, systemd's
+          # EnvironmentFile format — so the secret reaches the process as an
+          # environment variable without being read by nix, written to the
+          # store, or passed in argv.
+          systemd.services.opentelemetry-collector.serviceConfig =
+            lib.optionalAttrs (otelCfg.headersCredential != null)
+              {
+                EnvironmentFile = otelCfg.headersCredential;
+              };
+        };
+    };
+  };
+}

nix/host-modules/swarm-required-services.nix

@@ -70,6 +70,12 @@ in
     # exactly one still sets it directly, which `mkDefault` allows.
     victoriametrics.enable = lib.mkDefault swarmCfg.enableRequiredServices;
     grafana.enable = lib.mkDefault swarmCfg.enableRequiredServices;
+
+    # The collector that feeds the pair above, and the only tier holding
+    # the upstream credential. Same rule as the rest: once per swarm,
+    # optional, and a hive that is not the service host is a *client* of
+    # it (by name, `swarm.otel.domain`) rather than a second one.
+    otel.enable = lib.mkDefault swarmCfg.enableRequiredServices;
   };
 
   # The collector that feeds the pair above (note: no `swarm.` prefix,