vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

agents get a todo when assigned to an issue or PR, rewritten when unassigned and cleared on close

Browse source 
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).

- swarm-controller registers instance-wide hooks for `issues` +
  `issue_assign` and `pull_request` + `pull_request_assign`, updates the
  event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
  verified body unchanged in the SWARM_EVENTS stream on
  $SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
  A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
  flows (source -> match -> todo sink, stored as nodes plus edges and checked
  to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
  work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
  key, rewritten) and closed (prefix-clears every holder's router todos). A
  per-issue snapshot in its notify-router-state bucket holds the last
  assignees and the todo holders; only agents in the agent-status bucket get
  ops, and an assignee that was not one is left out of the snapshot so the
  next assign event retries it. After the first event that must be retried,
  the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
  op to its todo store as subsystem `router`, then acks; a refused op stops
  the rest of its batch the same way. The store's (subsystem, key) dedup
  makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
  $SWARM.events.forge.*; each agent gets three {agent} templates (extended
  consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
  swarm-notify-router bao policy (read on its own queue client secret), an
  authelia machine client, the secret publisher entry, a journal forwarder
  route. No manual step.

hive-forge-notify is unchanged, so agents also still get forge's own
notifications.

Refs #5571
This commit is contained in:
atlas 2026-10-22 03:03:26 +02:00 • committed by mara
commit 0f7b403803
53 changed files with 3587 additions and 96 deletions
Download patch file Download diff file Expand all files Collapse all files

2
nix/host-modules/default.nix
View file

@ -52,6 +52,7 @@
./glue-ci-metrics-otel-oidc-client.nix
./glue-ci-runner-otel-oidc-client.nix
./glue-claude-login-oidc-client.nix
./glue-notify-router-oidc-client.nix
./glue-forge-otel-oidc-client.nix
./glue-grafana-otel-oidc-client.nix
./glue-matrix-otel-oidc-client.nix
@ -68,6 +69,7 @@
./swarm-bao.nix
./swarm-builder.nix
./swarm-claude-login.nix
./swarm-notify-router.nix
./swarm-ci-metrics.nix
./swarm-ci-runner.nix
./swarm-review-classifier.nix

19
nix/host-modules/deploy.nix
View file

@ -406,6 +406,25 @@ in
'';
};
notify-router.enable = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Run the notification router in a `swarm-notify-router` container on
this host. It reads the forge events swarm-controller keeps from its
webhooks and queues todos for the agents they concern: assigned,
unassigned, and cleared when the issue or pull request closes. Run
it on one host: it handles each issue's events in order.
On by default where
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
host has the router's store identity (the pair
{option}`services.hyperhive.deploy.bao.notifyRouterClientCertFile`
and `notifyRouterClientKeyFile`, minted on the store's host).
'';
};
builder.enable = lib.mkOption {
type = lib.types.bool;
default = false;

9
nix/host-modules/glue-bao-tls.nix
View file

@ -121,6 +121,8 @@ in
ciMetricsClientKeyFile = lib.mkDefault "${pkiDir}/ci-metrics-key.pem";
claudeLoginClientCertFile = lib.mkDefault "${pkiDir}/claude-login.pem";
claudeLoginClientKeyFile = lib.mkDefault "${pkiDir}/claude-login-key.pem";
notifyRouterClientCertFile = lib.mkDefault "${pkiDir}/notify-router.pem";
notifyRouterClientKeyFile = lib.mkDefault "${pkiDir}/notify-router-key.pem";
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
@ -326,6 +328,13 @@ in
[ -s ${pkiDir}/claude-login.pem ] || ${signLeaf} ${pkiDir} claude-login \
${lib.escapeShellArg deployCfg.bao.claudeLoginCommonName} "" clientAuth
# The notification router's, which reads its queue client secret.
# Minted here for the matrix-ctl leaf's reason: the router is a swarm
# service, so on a router host without the store this is the file an
# operator copies.
[ -s ${pkiDir}/notify-router.pem ] || ${signLeaf} ${pkiDir} notify-router \
${lib.escapeShellArg deployCfg.bao.notifyRouterCommonName} "" clientAuth
# The forge's token publisher. Minted here for the matrix-ctl leaf's
# reason: the forge is a swarm singleton, so on a forge host without the
# store this is the file an operator copies.

39
nix/host-modules/glue-notify-router-oidc-client.nix Normal file
View file

@ -0,0 +1,39 @@
# Glue: register the notification router as an OIDC client wherever authelia
# runs.
#
# ONE PAIRING PER FILE — swarm-notify-router ← authelia, and nothing else.
# Deleting this leaves a router whose token requests authelia answers with a
# 401, and no minted secret for swarm-secret-publish to put in the store.
#
# ⚠️ Gated on authelia being HERE, and NOT on this host running the router,
# for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS
# host's provider config. Registered whether or not any host enables the
# router: nothing in `swarm.*` records that, so a swarm without one carries
# one unused client and one unused minted secret.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
clientId = hyperhiveCfg.swarm.notify-router.clientId;
in
{
config = lib.mkIf deployCfg.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = clientId;
description = "HyperHive notification router";
# `client_credentials`: a daemon authenticating as itself.
kind = "machine";
audience = [ clientId ];
# The container's journal forwarder mints its token as this client, for
# the swarm collector's receiver, which verifies it offline against
# `/jwks.json`; authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
}
];
};
}

49
nix/host-modules/swarm-bao.nix
View file

@ -1148,6 +1148,16 @@ let
}
];
# The notification router: its own queue client secret, read at start by the
# daemon inside its container, and nothing else.
notifyRouterReaders = [
{
name = "swarm-notify-router";
cn = baoDeploy.notifyRouterCommonName;
policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.notify-router.clientId}/oidc/client";
}
];
# The forge host's one store identity. The forge's
# `forgejo-swarm-controller-account`, in its container, writes the
# controller's forge token under it, the one writer of that path and
@ -2401,6 +2411,18 @@ in
'';
};
notifyRouterCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-notify-router";
description = ''
Subject the store's `swarm-notify-router` cert-auth role accepts: the
identity the notification router presents from inside its container.
Its grant is read on its own queue client secret.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
'';
};
claudeLoginClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -2426,6 +2448,31 @@ in
'';
};
notifyRouterClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/notify-router.pem";
description = ''
Certificate the notification router presents to the store. Its subject
must be {option}`services.hyperhive.deploy.bao.notifyRouterCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
On a router host without the store, it is a file the operator copies
from the store's host. A path, never a value.
'';
};
notifyRouterClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/notify-router-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.notifyRouterClientCertFile`.
A path, never a value.
'';
};
forgePublisherClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -3524,6 +3571,8 @@ in
systemd.services.swarm-bao-claude-login-policy = readerPolicyUnit "write the claude login minter's credentials-writer bao policy and cert-auth role" claudeLoginWriters;
systemd.services.swarm-bao-notify-router-policy = readerPolicyUnit "write the notification router's client-secret-reader bao policy and cert-auth role" notifyRouterReaders;
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
# A FOURTH sibling, same shape and same reasons as the two above. This

13
nix/host-modules/swarm-nats.nix
View file

@ -867,6 +867,9 @@ in
"--builder-client ${lib.escapeShellArg config.services.hyperhive.swarm.builder.clientId}"
# Hears login steps and answers each by reply, and nothing else.
"--claude-login-client ${lib.escapeShellArg config.services.hyperhive.swarm.claude-login.clientId}"
# Drains its consumer on the forge events stream and queues
# agents' todo ops (`swarm-nats-auth`'s `notify_router_permissions`).
"--notify-router-client ${lib.escapeShellArg config.services.hyperhive.swarm.notify-router.clientId}"
# What an agent may publish to, keyed on its own name: its
# terminal rows and its turn-state header
# (`hive-agent::swarm_agent_state`), two subject families so a
@ -914,6 +917,16 @@ in
# (`swarm_queue_client::agents_md_initial`), on the same terms: the
# harness reads the text its `AGENTS.md` starts with.
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.API.DIRECT.GET.KV_agents-md-initial.\$\$KV.agents-md-initial.{agent}"}"
# Its own durable consumer on the notification router's
# `ROUTER_TODOS` stream (`swarm_wire::router_todo`): create it
# in the extended form whose last tokens are the filter, which
# the server checks against the payload, so the consumer
# delivers this agent's subject alone; pull from it; ack what
# it delivered. No bare `CONSUMER.CREATE`, no `STREAM.INFO`
# and no publish on the subjects: the router is their writer.
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.API.CONSUMER.CREATE.ROUTER_TODOS.todo-{agent}.\$\$SWARM.router.todo.{agent}"}"
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.API.CONSUMER.MSG.NEXT.ROUTER_TODOS.todo-{agent}"}"
"--agent-token-publish-subject ${lib.escapeShellArg "\$\$JS.ACK.ROUTER_TODOS.todo-{agent}.>"}"
# Every agent also reads the operator list, the one key of the
# `operator-profiles` bucket, with one direct get. That grant
# names no agent, so the responder adds it itself

235
nix/host-modules/swarm-notify-router.nix Normal file
View file

@ -0,0 +1,235 @@
# The swarm notification router, in a `swarm-notify-router` container on the
# host that sets `deploy.notify-router.enable`, which `deploy.allSwarmServices`
# defaults on where the host has the router's store identity. It reads the
# forge events swarm-controller keeps from its webhooks and queues todo ops on
# each agent's subject of `ROUTER_TODOS`, which the agent's harness applies.
# docs/swarm/notify-router.md has the design.
#
# Its own network namespace on the bridge, as ./swarm-claude-login.nix has: it
# reaches the swarm services by name and nothing on the host's loopback. It
# talks only to the queue, the store and authelia, so it can run on any host.
#
# The container holds one credential, its own store leaf, which nspawn hands in
# as a system credential with no copy on disk. The daemon reads its queue
# client secret with it.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
cfg = hyperhiveCfg.deploy.notify-router;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
natsCfg = hyperhiveCfg.swarm.nats;
autheliaCfg = hyperhiveCfg.swarm.authelia;
gatewayCfg = hyperhiveCfg.gateway;
networkCfg = hyperhiveCfg.network;
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
machine = "swarm-notify-router";
unit = "swarm-notify-router";
clientId = hyperhiveCfg.swarm.notify-router.clientId;
# The container's journal forwarder pushes to its own receiver on the swarm
# collector, the route `serviceProducers.swarm-notify-router` in
# ./swarm-otel-service.nix declares, as ./swarm-claude-login.nix does.
swarmOtelCfg = hyperhiveCfg.swarm.otel;
gatewayPortSuffix = lib.optionalString (
gatewayCfg.httpsPort != 443
) ":${toString gatewayCfg.httpsPort}";
forwarderEndpoint = "https://${swarmOtelCfg.domain}${gatewayPortSuffix}/${machine}";
forwarderUnit = "swarm-journal-forwarder-oidc";
# Loopback in the container's own netns, and distinct from the other service
# containers' forwarders all the same.
forwarderTelemetryPort = 8902;
# The daemon reaches authelia, the store and the queue by name over TLS, and
# in self-signed mode their leaves chain to the hive CA.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
caBundleModule = caTrust.trustBundle {
inherit pkgs;
name = machine;
consumers = [
unit
# Go: `SSL_CERT_FILE` replaces its trust store, for the swarm
# collector's and authelia's names.
"opentelemetry-collector"
];
};
haveClientIdentity =
baoDeploy.notifyRouterClientCertFile != null && baoDeploy.notifyRouterClientKeyFile != null;
# Credential id → host source, as ./swarm-claude-login.nix hands its leaf in.
storeFiles = lib.filterAttrs (_: source: source != null) {
notify-router-bao-cert = baoDeploy.notifyRouterClientCertFile;
notify-router-bao-key = baoDeploy.notifyRouterClientKeyFile;
notify-router-bao-server-ca = baoDeploy.serverCaFile;
};
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
credential = id: "/run/credentials/${unit}.service/${id}";
forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}";
privateNetwork = true;
in
{
options.services.hyperhive.swarm.notify-router.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-notify-router";
description = ''
The OAuth2 client id the notification router presents to the swarm
queue. Read-only: the host running authelia registers it, the queue's
auth-callout responder grants it the router's subjects, and the secret
publisher carries its secret to the store, all from this one value.
'';
};
options.services.hyperhive.deploy.notify-router.package = lib.mkOption {
type = lib.types.package;
defaultText = lib.literalExpression "hyperhive.packages.\${system}.swarm-notify-router";
description = "The notification router daemon package.";
};
config = lib.mkIf cfg.enable {
assertions = [
{
assertion = haveClientIdentity;
message = ''
services.hyperhive.deploy.notify-router.enable requires the router's
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.notifyRouterClientCertFile
services.hyperhive.deploy.bao.notifyRouterClientKeyFile
The router reads its queue client secret with it.
Both default to the pair glue-bao-tls.nix mints when
services.hyperhive.deploy.bao.enable is true (the host that runs the
store). Otherwise copy notify-router.pem and notify-router-key.pem
from the store host's /var/lib/swarm-bao-pki/ and name them in those
two options.
'';
}
{
assertion = lib.all oneWord (lib.attrValues storeFiles);
message = ''
The notification router's store identity paths must not contain
whitespace: nspawn's --load-credential receives them through
nixos-container's unquoted extra flags.
'';
}
];
# The container hangs off the bridge, takes its address from the gateway's
# DHCP and resolves through its dnsmasq.
services.hyperhive.network.enable = lib.mkDefault true;
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
# nspawn fails to start the container when a credential source is
# missing, so where this host mints the leaf the container waits for it.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
{ after = [ "swarm-bao-pki.service" ]; }
];
containers.${machine} = {
autoStart = true;
# The daemon keeps its state in the queue: its consumer's position and
# its per-issue snapshots.
ephemeral = false;
# No `--link-journal=host`: the journal stays in the guest, where the
# container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
inherit privateNetwork;
hostBridge = networkCfg.bridgeName;
bindMounts = caTrust.bindMount;
config =
{ lib, ... }:
{
imports = [
../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
caBundleModule
];
services.hyperhive.swarmContainer = {
inherit privateNetwork;
writesOwnResolvConf = false;
};
networking.nameservers = [ networkCfg.bridgeIp ];
networking.interfaces.eth0.useDHCP = true;
users.users.${unit} = {
isSystemUser = true;
group = unit;
};
users.groups.${unit} = { };
# Authenticated as the daemon's own OIDC client, whose secret its
# store identity already reads: one identity for the whole container.
services.hyperhive.swarmJournalForwarder = {
enable = true;
serviceName = "notify-router";
endpoint = forwarderEndpoint;
inherit clientId;
tokenUrl = "${autheliaCfg.url}/api/oidc/token";
telemetryPort = forwarderTelemetryPort;
store = {
package = baoDeploy.package;
addr = "https://${baoCfg.domain}:${toString baoCfg.port}";
secretPath = "secret/swarm/services/${clientId}/oidc/client";
clientCertFile = forwarderCredential "notify-router-bao-cert";
clientKeyFile = forwarderCredential "notify-router-bao-key";
caFile =
if storeFiles ? notify-router-bao-server-ca then
forwarderCredential "notify-router-bao-server-ca"
else
null;
};
};
systemd.services.${forwarderUnit}.serviceConfig.LoadCredential = lib.attrNames storeFiles;
systemd.services.${unit} = {
description = "route forge events to agents' todo lists";
wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ];
after = [ "network-online.target" ];
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = credential "notify-router-bao-cert";
BAO_CLIENT_KEY = credential "notify-router-bao-key";
SWARM_NOTIFY_ROUTER_NATS_URL = "tls://${natsCfg.domain}:${toString natsCfg.port}";
SWARM_NOTIFY_ROUTER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
SWARM_NOTIFY_ROUTER_OIDC_CLIENT_ID = clientId;
}
// lib.optionalAttrs (storeFiles ? notify-router-bao-server-ca) {
BAO_CACERT = credential "notify-router-bao-server-ca";
}
// lib.optionalAttrs caTrust.useSelfSigned {
SWARM_NOTIFY_ROUTER_OIDC_CA_FILE = caTrust.bundlePathFor machine;
};
serviceConfig = {
ExecStart = "${cfg.package}/bin/swarm-notify-router";
User = unit;
Group = unit;
UMask = "0077";
# Bare ids: the system credentials nspawn handed in.
LoadCredential = lib.attrNames storeFiles;
Restart = "on-failure";
RestartSec = "10s";
SyslogIdentifier = unit;
};
};
};
};
};
}

10
nix/host-modules/swarm-otel-service.nix
View file

@ -391,9 +391,9 @@ in
clientId = hyperhiveCfg.swarm.nats.clientId;
port = lib.mkDefault 4392;
};
# The forwarders in ./swarm-builder.nix, ./swarm-claude-login.nix and
# ./swarm-review-classifier.nix, which address their routes by container
# name.
# The forwarders in ./swarm-builder.nix, ./swarm-claude-login.nix,
# ./swarm-notify-router.nix and ./swarm-review-classifier.nix, which
# address their routes by container name.
swarm-builder = {
clientId = hyperhiveCfg.swarm.builder.clientId;
port = lib.mkDefault 4393;
@ -406,6 +406,10 @@ in
clientId = hyperhiveCfg.swarm.review-classifier.clientId;
port = lib.mkDefault 4395;
};
swarm-notify-router = {
clientId = hyperhiveCfg.swarm.notify-router.clientId;
port = lib.mkDefault 4404;
};
# The forwarders in ./swarm-ci-runner.nix, ./swarm-grafana.nix and
# ./swarm-victorialogs.nix, each authenticated as a client of its own.
swarm-ci-runner = {

7
nix/host-modules/swarm-otel.nix
View file

@ -205,9 +205,10 @@ let
# nixpkgs hardcodes `--link-journal=try-guest`, which keeps this container's
# journal in the guest, so the host directory holds the host's units only.
# Every container with a `serviceProducers` entry (`swarm-bao`, `swarm-nats`,
# `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`,
# `swarm-ci-runner`, `swarm-grafana`, `swarm-victorialogs`, `hive-forge`,
# `hive-matrix`, `swarm-victoriametrics`, `swarm-authelia`) forwards its own
# `swarm-builder`, `swarm-claude-login`, `swarm-notify-router`,
# `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
# `swarm-victorialogs`, `hive-forge`, `hive-matrix`, `swarm-victoriametrics`,
# `swarm-authelia`) forwards its own
# journal from its own collector, so this receiver never sees their units.
hostJournalDir = "/var/log/journal";
hostJournalMount = "/run/host-journal";

7
nix/host-modules/swarm-required-services.nix
View file

@ -137,4 +137,11 @@ in
&& deployCfg.bao.claudeLoginClientCertFile != null
&& deployCfg.bao.claudeLoginClientKeyFile != null
);
# The notification router, defaulted on the same terms.
config.services.hyperhive.deploy.notify-router.enable = lib.mkDefault (
deployCfg.allSwarmServices
&& deployCfg.bao.notifyRouterClientCertFile != null
&& deployCfg.bao.notifyRouterClientKeyFile != null
);
}

2
nix/host-modules/swarm-secret-publisher.nix
View file

@ -58,6 +58,8 @@ let
hyperhiveCfg.swarm.builder.clientId
# The claude login minter, which reads it itself inside its container.
hyperhiveCfg.swarm.claude-login.clientId
# The notification router, which reads it itself inside its container.
hyperhiveCfg.swarm.notify-router.clientId
# The journal forwarders in the CI runner's, Grafana's, the log store's,
# the forge's and the homeserver's containers, each reading it inside its
# container.

1
nix/host-modules/swarm.nix
View file

@ -61,6 +61,7 @@ let
deployCfg.bao.builderCommonName
deployCfg.bao.ciMetricsCommonName
deployCfg.bao.claudeLoginCommonName
deployCfg.bao.notifyRouterCommonName
deployCfg.bao.forgePublisherCommonName
]
# The per-hive subjects, spelled out per hive rather than as the prefix.