vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity
hyperhive/nix/host-modules/swarm-otel-service.nix
atlas 0f7b403803 agents get a todo when assigned to an issue or PR, rewritten when unassigned and cleared on close 
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).

- swarm-controller registers instance-wide hooks for `issues` +
  `issue_assign` and `pull_request` + `pull_request_assign`, updates the
  event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
  verified body unchanged in the SWARM_EVENTS stream on
  $SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
  A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
  flows (source -> match -> todo sink, stored as nodes plus edges and checked
  to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
  work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
  key, rewritten) and closed (prefix-clears every holder's router todos). A
  per-issue snapshot in its notify-router-state bucket holds the last
  assignees and the todo holders; only agents in the agent-status bucket get
  ops, and an assignee that was not one is left out of the snapshot so the
  next assign event retries it. After the first event that must be retried,
  the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
  op to its todo store as subsystem `router`, then acks; a refused op stops
  the rest of its batch the same way. The store's (subsystem, key) dedup
  makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
  $SWARM.events.forge.*; each agent gets three {agent} templates (extended
  consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
  swarm-notify-router bao policy (read on its own queue client secret), an
  authelia machine client, the secret publisher entry, a journal forwarder
  route. No manual step.

hive-forge-notify is unchanged, so agents also still get forge's own
notifications.

Refs #5571
2026-10-22 07:28:53 +02:00

477 lines
21 KiB
Nix
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# The swarm's telemetry collector as every hive sees it: the name it is
# reached on, its receiver ports, the published targets it scrapes, the client
# it is registered as and the audiences that client may present, identical on
# every host. What the host running it decides, and the container itself, are
# in ./swarm-otel.nix.
{
lib,
config,
...
}:
let
cfg = config.services.hyperhive.swarm.otel;
vmCfg = config.services.hyperhive.swarm.victoriametrics;
vlCfg = config.services.hyperhive.swarm.victorialogs;
hyperhiveCfg = config.services.hyperhive;
baoCfg = hyperhiveCfg.swarm.bao;
swarmDomain = hyperhiveCfg.swarm.domain;
# Total on a null swarm domain for the same reason every sibling module is:
# the required-domain assertion in hive-network.nix should be what an
# operator sees, not a coercion error from here.
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
# Each store's OTLP route, by domain: the audiences `audience` below registers
# the client for. ⚠️ ./swarm-otel.nix binds the same strings for the
# exporters that request those tokens; the two copies must agree.
metricsPushUrl = "https://${vmCfg.domain}/opentelemetry/api/v1/push";
logsPushUrl = "https://${vlCfg.domain}/insert/opentelemetry/v1/logs";
pushAudiences = {
victoriametrics = metricsPushUrl;
victorialogs = logsPushUrl;
};
in
{
# `enable` is `services.hyperhive.deploy.swarm-otel.enable` — see ./deploy.nix.
# ⚠️ That is the SWARM collector. The per-hive one keeps its own
# `services.hyperhive.otel.collector.enable` (./otel.nix) and is a different
# option entirely — every hive runs that one.
options.services.hyperhive.swarm.otel = {
machine = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-otel";
description = ''
Name of the nixos-container this collector runs in — also the
`machinectl` name, so other modules may read it rather than
repeating the literal.
'';
};
port = lib.mkOption {
type = lib.types.port;
default = 4319;
description = ''
First port of this collector's receiver range. Every hive in
{option}`services.hyperhive.swarm.hives` gets its **own**
authenticated receiver — that is what makes the `hive` label
unforgeable — so the range is one port per hive, starting here, in
sorted-name order.
⚠️ Internal. No client is ever told a port: a hive reaches its own
receiver as `https://''${domain}/<hive>`, and the gateway routes on
that path. So adding a hive, which renumbers the ones after it, is
harmless — nginx is rendered from this same evaluation and moves
with it.
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
hive tier already uses it (`services.hyperhive.swarm.hiveOtel.collectorPort`)
and every swarm container shares the host's network namespace. Two
listeners claiming one port on one host is not a build failure —
it is a runtime coin toss over which one gets it, with nothing in
any log saying so. On the host running this collector, an
assertion rejects a derived range that overlaps any other port
this module or the hive tier declares, which is as far as a module
can see.
'';
};
telemetryPort = lib.mkOption {
type = lib.types.port;
default = 8889;
description = ''
Port this collector serves its **own** metrics on — queue depth,
refused and dropped samples, exporter failures. How you find out
that telemetry is being lost, so it is worth keeping rather than
switching off.
⚠️ **Deliberately not 8889's neighbour 8888**, which is the
collector's built-in default and therefore what the hive tier
already binds. Two collectors share a network namespace whenever
they are co-located, and unlike the OTLP port this one appears
nowhere in either config — it is a default inside the binary, so
nothing that compares configured ports can see the clash. The
second collector to start simply dies with
`bind: address already in use`.
'';
};
producerPort = lib.mkOption {
type = lib.types.port;
default = 4390;
description = ''
Port the swarm tier's own OTLP/HTTP receiver for a
**swarm-level** producer (`swarm-controller`'s vcs/jobq counters
today) listens on, at `127.0.0.1`. Reached through the gateway at
`https://''${domain}/''${producerName}/`, same shape as a hive's
own receiver — a swarm-level producer may not share a host with
this collector (the swarm services need not run on the same host
as the swarm controller), so loopback-only reachability is not a
supported shape here, as it is not for a hive.
Authenticated by `oidc/''${producerName}`, checking for the
audience `services.hyperhive.swarm.controller.queueClientId`
requests — not the per-hive `hiveClientPrefix` namespace, because
a swarm-level producer is explicitly not a hive (see that
option's own doc comment). The `swarm` label this receiver's
samples get is still the one the receiver's own *existence*
establishes, same as before — there is no hive identity being
forged or attributed here, just a caller proving it is the one
principal allowed to push into this pipeline.
⚠️ Deliberately NOT derived from `port + (number of hives)`: that
range grows every time a hive is added, and a fixed offset from
it would silently start colliding once the hive count caught up.
Kept as its own reserved value instead, and the port-collision
assertion still rejects a real collision (including one hive
growth eventually causes) rather than starting a collector that
quietly drops one receiver's samples.
'';
};
storeProducerPort = lib.mkOption {
type = lib.types.port;
default = 4391;
description = ''
Port the receiver for the **secret store's own forwarder** listens
on, at `127.0.0.1`. Reached through the gateway at
`https://''${domain}/''${storeProducerName}/`, the same
path-per-producer shape a hive and the swarm-tier producer above
both use.
Its own receiver, and not a second sender into
{option}`services.hyperhive.swarm.otel.producerPort`, because an
`oidc` authenticator checks exactly ONE audience: that receiver
admits the audience `swarm-controller` registers for itself, and
the store's forwarder is a different principal with a client of its
own. Sharing the route would mean sharing that client — which is
the thing the store exists to make unnecessary.
⚠️ A reserved value rather than an offset from
{option}`services.hyperhive.swarm.otel.port`, for the reason stated
on `producerPort`: that range grows with the hive count and would
eventually walk into any fixed offset from it. The port-collision
assertion covers this port too.
'';
};
serviceProducers = lib.mkOption {
internal = true;
type = lib.types.attrsOf (
lib.types.submodule {
options = {
clientId = lib.mkOption {
type = lib.types.str;
description = ''
The producer's OAuth2 client id: the one audience its
receiver's `oidc/<name>` authenticator accepts, and the one
the forwarder requests.
'';
};
port = lib.mkOption {
type = lib.types.port;
description = ''
Loopback port of the producer's receiver on the collector's
host. In the port-collision assertion with every other
receiver there.
'';
};
};
}
);
default = { };
description = ''
Swarm services whose own container forwards to this collector, keyed
by the route name: each gets a receiver `otlp/<name>`, an
authenticator `oidc/<name>` and a gateway location
`https://''${domain}/<name>/`, and its records land in the swarm
tier's pipelines with no `hive` label.
A receiver per producer because an `oidc` authenticator checks one
audience, and each service mints its token as its own client.
Declared below for every host, under no `enable`: the collector's
host renders the receiver whether or not the service runs there.
Every name must contain a fragment of
`nix/reserved-hive-fragments.nix`, so no hive can own its components.
'';
};
storeProducerName = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = baoCfg.machine;
defaultText = lib.literalExpression "services.hyperhive.swarm.bao.machine";
description = ''
Path and component name of the store forwarder's own receiver
(`otlp/''${storeProducerName}`, `oidc/''${storeProducerName}`).
Published so `swarm-bao.nix` addresses the receiver by name rather
than repeating the string, exactly as `producerName` above is
published for `glue-controller-env.nix`.
Read-only and derived from the store's container name: two
spellings of a name both ends have to agree on is a 404 on a
request that authenticated perfectly.
'';
};
producerName = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm";
description = ''
The swarm-tier producer's own component/path/authenticator name
— the same reserved literal this module's collector components
(`otlp/swarm`, `oidc/swarm`, `resource/swarm`, `metrics/swarm`)
are already named after internally. Published as an option so a
sibling module (`glue-controller-env.nix`) can address the receiver
by name — `https://''${domain}/''${producerName}/` — instead of
repeating the string. Read-only for the same reason
`clientId`/`machine`/`domain` are: two spellings of a name that
has to match on both ends is a mismatch waiting to happen.
'';
};
domain = lib.mkOption {
type = lib.types.str;
default = "otel.${domainBase}";
defaultText = lib.literalExpression ''"otel.''${services.hyperhive.swarm.domain}"'';
description = ''
Name the gateway serves this on. A sibling of the swarm's other
service names, so the swarm-services sub-CA can issue for it — see
`hive-tls.nix` for why a service name being a sibling rather than a
child decides which CA may sign it.
This is what the **hive** tier's exporter reaches — the hive
collector is a plain producer against this name exactly like every
other client of a swarm service, resolved locally by dnsmasq on a
co-located host and over the real network otherwise. There is no
separate loopback-vs-remote knob to get wrong: everything in this
swarm addresses its siblings by name.
'';
};
scrapeTargets = lib.mkOption {
type = lib.types.attrsOf lib.types.str;
default = { };
example = lib.literalExpression ''{ forgejo = "127.0.0.1:3000"; }'';
description = ''
Prometheus exposition endpoints this collector scrapes, as
`<job name> = "<host>:<port>"`.
A path and query may follow the port —
`"127.0.0.1:8202/v1/sys/metrics?format=prometheus"` — for an
exporter that does not serve `/metrics`. Both are optional and
omitted when absent, so a bare `host:port` is scraped exactly as
before.
**A service declares its own entry, from its own module, under its
own `enable`.** Do not assemble the list here: an entry then exists
only where the service that named it runs, so a target is never
declared on a host that does not serve it. An entry that also needs
the collector on the same host (the loopback address is only honest
then) is declared from a glue module instead, which reads both
switches. Every swarm service is a `publishedScrapeTargets` entry
instead, which a collector on any host scrapes.
⚠️ **That constrains the TARGET, not the SCRAPER, and the difference
is a silent gap.** Nothing here puts the collector on the same host.
Services are co-located by *default* — `swarm-required-services.nix`
derives their `enable` flags from one `lib.mkDefault` — not by
construction, and an operator may split them.
When they are split the target is simply never scraped: the
service's host declares an entry no local collector reads, and the
collector's host never enabled that service so has no entry at all.
No error, no eval failure, no warning.
**No assertion can catch this.** Two hosts are separate NixOS
evaluations with no shared context, so neither can see what the
other runs. A service that would be seriously wrong to lose should
say so in its own contribution, because saying it is the only
mechanism available.
Samples land in a swarm-level pipeline that stamps `swarm` and
**never** `hive`: a swarm service does not belong to a hive, and
`hive` stays a property of which authenticated receiver accepted a
push, not something a scrape can acquire.
Empty by default, in which case no scrape receiver, processor or
pipeline is emitted at all — an enabled scraper with nothing to
scrape is the inert configuration this option exists to avoid.
'';
};
publishedScrapeTargets = lib.mkOption {
type = lib.types.attrsOf lib.types.str;
default = { };
example = lib.literalExpression ''{ forge = "https://forge.example.com/metrics"; }'';
description = ''
Prometheus endpoints this collector scrapes **by name, with a
credential**, as `<job name> = "<url>"`.
**Entries are declared swarm-wide**: from this module's `config`,
unconditionally, keyed on the service's `swarm.<svc>.domain`, so
every host evaluates the same set. Unlike `scrapeTargets`, an entry
is not declared under the service's own `enable`: the collector
reads this list on its host and authelia's registration reads it on
authelia's host, and neither need be the service's host. A swarm
that does not run the service anywhere scrapes a name that answers
nothing, which shows as a failing job rather than a missing one.
The collector's OAuth2 client derives its permitted audiences from
these URLs, and authelia derives from each one the access rule that
admits only that client to it, so a target and the authorisation to
reach it are **one declaration**.
Two lists that must agree would be a drift to maintain, and its
failure mode is the bad one: a target whose audience was forgotten
authenticates against nothing and reads as a scrape failure rather
than a config mistake.
⚠️ A **full URL**, not `host:port`. Authelia validates a bearer
token against the address being requested, so the string here is
also the audience the token is minted for; a near miss (a trailing
slash, `http` for `https`) presents as a valid token rejected at
the target, several layers from its cause.
⚠️ Deliberately separate from `scrapeTargets`. An entry there is
trusted because the scraper and the target share a host — that
option is loopback-and-unauthenticated by contract. An entry here
is trusted because it presents a credential. One shape for both
would leave a reader unable to tell which of those a given target
relies on.
'';
};
clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-collector";
description = ''
OAuth2 client id this collector authenticates as. Published so
authelia's `access_control` rules can name it without carrying a
second copy of the string, exactly as
`services.hyperhive.swarm.authelia.hiveClientPrefix` is published
for the queue's responder.
Two spellings drifting apart is not a build failure: authelia
refuses a rule naming an unregistered client in its startup
validator, so the swarm's SSO service fails to *restart* — long
after the change that caused it evaluated cleanly.
'';
};
audience = lib.mkOption {
type = lib.types.listOf lib.types.str;
readOnly = true;
default = lib.attrValues cfg.publishedScrapeTargets ++ lib.attrValues pushAudiences;
description = ''
Every audience this collector's OAuth2 client is permitted to
present a token for — the scrape targets it reads with a credential,
plus the stores it pushes to. Published read-only so
./glue-swarm-otel-oidc-client.nix can register the client wherever
authelia runs without restating the derivation: this option and
that registration are the same fact seen from two hosts, and a
second formula for it would be free to drift from this one.
'';
};
};
config.services.hyperhive.swarm.otel.serviceProducers = {
# The forwarder in ./swarm-bao.nix.
${cfg.storeProducerName} = {
clientId = baoCfg.otel.clientId;
port = cfg.storeProducerPort;
};
# The forwarder in ./swarm-nats.nix, which addresses this route by the
# queue container's name.
swarm-nats = {
clientId = hyperhiveCfg.swarm.nats.clientId;
port = lib.mkDefault 4392;
};
# The forwarders in ./swarm-builder.nix, ./swarm-claude-login.nix,
# ./swarm-notify-router.nix and ./swarm-review-classifier.nix, which
# address their routes by container name.
swarm-builder = {
clientId = hyperhiveCfg.swarm.builder.clientId;
port = lib.mkDefault 4393;
};
swarm-claude-login = {
clientId = hyperhiveCfg.swarm.claude-login.clientId;
port = lib.mkDefault 4394;
};
swarm-review-classifier = {
clientId = hyperhiveCfg.swarm.review-classifier.clientId;
port = lib.mkDefault 4395;
};
swarm-notify-router = {
clientId = hyperhiveCfg.swarm.notify-router.clientId;
port = lib.mkDefault 4404;
};
# The forwarders in ./swarm-ci-runner.nix, ./swarm-grafana.nix and
# ./swarm-victorialogs.nix, each authenticated as a client of its own.
swarm-ci-runner = {
clientId = hyperhiveCfg.swarm.ci-runner.otel.clientId;
port = lib.mkDefault 4396;
};
swarm-grafana = {
clientId = hyperhiveCfg.swarm.grafana.otel.clientId;
port = lib.mkDefault 4397;
};
swarm-victorialogs = {
clientId = hyperhiveCfg.swarm.victorialogs.otel.clientId;
port = lib.mkDefault 4398;
};
# The forwarders in ./hive-forge/default.nix and ./hive-matrix.nix, each
# authenticated as a client of its own.
hive-forge = {
clientId = hyperhiveCfg.swarm.forge.otel.clientId;
port = lib.mkDefault 4399;
};
hive-matrix = {
clientId = hyperhiveCfg.swarm.matrix.otel.clientId;
port = lib.mkDefault 4400;
};
# The forwarder in ./swarm-victoriametrics.nix, which reaches this route
# through its host's nginx.
swarm-victoriametrics = {
clientId = hyperhiveCfg.swarm.victoriametrics.otel.clientId;
port = lib.mkDefault 4401;
};
# The CI run poller in ./swarm-ci-metrics.nix, a host unit that pushes
# its run samples here directly.
swarm-ci-metrics = {
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
port = lib.mkDefault 4403;
};
# The forwarder in ./swarm-authelia.nix.
swarm-authelia = {
clientId = hyperhiveCfg.swarm.authelia.otel.clientId;
port = lib.mkDefault 4402;
};
};
# Not under any `enable`: see `publishedScrapeTargets` for why every host
# declares every entry.
#
# ⚠️ Each URL must be exactly what the collector requests, because authelia
# checks the token's audience against it: it has to agree with the gated
# `= /metrics` location on the service's vhost. A near miss is a correctly
# minted token refused at the target.
config.services.hyperhive.swarm.otel.publishedScrapeTargets = {
# The vhost location is in ./hive-forge/default.nix.
forgejo = "https://${hyperhiveCfg.swarm.forge.domain}/metrics";
# The vhost location is in ./swarm-authelia.nix.
authelia = "https://${hyperhiveCfg.swarm.authelia.domain}/metrics";
# The vhost location is in ./swarm-victoriametrics.nix.
victoriametrics = "https://${vmCfg.domain}/metrics";
# The vhost location is in ./swarm-victorialogs.nix.
victorialogs = "https://${hyperhiveCfg.swarm.victorialogs.domain}/metrics";
# The vhost location is in ./swarm-grafana.nix.
grafana = "https://${hyperhiveCfg.swarm.grafana.domain}/metrics";
# The vhost location is in ./swarm-nats.nix.
nats = "https://${hyperhiveCfg.swarm.nats.domain}/metrics";
};
}
View git blame Copy permalink