0f7b403803
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).
- swarm-controller registers instance-wide hooks for `issues` +
`issue_assign` and `pull_request` + `pull_request_assign`, updates the
event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
verified body unchanged in the SWARM_EVENTS stream on
$SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
flows (source -> match -> todo sink, stored as nodes plus edges and checked
to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
key, rewritten) and closed (prefix-clears every holder's router todos). A
per-issue snapshot in its notify-router-state bucket holds the last
assignees and the todo holders; only agents in the agent-status bucket get
ops, and an assignee that was not one is left out of the snapshot so the
next assign event retries it. After the first event that must be retried,
the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
op to its todo store as subsystem `router`, then acks; a refused op stops
the rest of its batch the same way. The store's (subsystem, key) dedup
makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
$SWARM.events.forge.*; each agent gets three {agent} templates (extended
consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
swarm-notify-router bao policy (read on its own queue client secret), an
authelia machine client, the secret publisher entry, a journal forwarder
route. No manual step.
hive-forge-notify is unchanged, so agents also still get forge's own
notifications.
Refs #5571
195 lines
8.9 KiB
Nix
195 lines
8.9 KiB
Nix
# What a host decides about the swarm's secret publisher: whether it runs, and
|
||
# the store identity it presents, plus the list of secrets it copies. The unit
|
||
# itself is ../container-modules/swarm-secret-publisher.nix, inside authelia's
|
||
# container beside the mint whose secrets it copies; ./swarm-authelia.nix
|
||
# imports it there and hands the identity in as system credentials.
|
||
{
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
|
||
|
||
# The client id agent containers present, per hive — composed exactly as
|
||
# ./swarm-authelia.nix composes it, from the same two read-only options, so a
|
||
# rename there cannot leave this spelling behind.
|
||
agentClientId =
|
||
hive:
|
||
"${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}${hyperhiveCfg.swarm.authelia.agentClientSuffix}";
|
||
|
||
# The hive's own client id, which hive-c0re publishes the hive's status with.
|
||
# Composed as ./swarm-authelia.nix composes it, for the reason above.
|
||
hiveClientId = hive: "${hyperhiveCfg.swarm.authelia.hiveClientPrefix}${hive}";
|
||
|
||
# The swarm's own services, as opposed to its hives. One client for the whole
|
||
# swarm rather than one per hive, so one value in the store rather than a copy
|
||
# each: `swarm/services/<id>/oidc/client`.
|
||
#
|
||
# The ids come from `swarm.*`, which is identical on every host — that is what
|
||
# lets this host name a service's client while running none of them, and it is
|
||
# the same read each service's own module registers its client with. A swarm
|
||
# that runs no Grafana, or no collector, mints no secret for the one it lacks,
|
||
# so its entry skips below rather than needing a condition here.
|
||
serviceClientIds = [
|
||
hyperhiveCfg.swarm.grafana.oidc.clientId
|
||
hyperhiveCfg.swarm.otel.clientId
|
||
# The secret store's own journal forwarder. A third swarm service by this
|
||
# list's own rule — a client registered once for the whole swarm, whose
|
||
# secret one host mints and another reads — even though the principal
|
||
# behind it runs inside the store's container rather than beside it.
|
||
# Missing from here, authelia mints the value and nothing carries it, so
|
||
# the reader waits on a path that is never written.
|
||
hyperhiveCfg.swarm.bao.otel.clientId
|
||
# The store's browser UI, whose `oidc` auth method the store's host
|
||
# configures with this secret.
|
||
hyperhiveCfg.swarm.bao.ui.oidc.clientId
|
||
# The queue's auth-callout responder, which reads it itself on the
|
||
# queue's host.
|
||
hyperhiveCfg.swarm.nats.clientId
|
||
# The homeserver, read by `swarm-bao-matrix-oidc` on its host.
|
||
hyperhiveCfg.swarm.matrix.sso.clientId
|
||
# The forge's login source, read by `swarm-bao-forge-oidc` on its host.
|
||
hyperhiveCfg.swarm.forge.sso.clientId
|
||
# The review classifier, which reads it itself inside its container.
|
||
hyperhiveCfg.swarm.review-classifier.clientId
|
||
# The swarm builder, which reads it itself inside its container.
|
||
hyperhiveCfg.swarm.builder.clientId
|
||
# The claude login minter, which reads it itself inside its container.
|
||
hyperhiveCfg.swarm.claude-login.clientId
|
||
# The notification router, which reads it itself inside its container.
|
||
hyperhiveCfg.swarm.notify-router.clientId
|
||
# The journal forwarders in the CI runner's, Grafana's, the log store's,
|
||
# the forge's and the homeserver's containers, each reading it inside its
|
||
# container.
|
||
hyperhiveCfg.swarm.ci-runner.otel.clientId
|
||
hyperhiveCfg.swarm.grafana.otel.clientId
|
||
hyperhiveCfg.swarm.victorialogs.otel.clientId
|
||
hyperhiveCfg.swarm.forge.otel.clientId
|
||
hyperhiveCfg.swarm.matrix.otel.clientId
|
||
# The metrics store container's journal forwarder, whose host reads it.
|
||
hyperhiveCfg.swarm.victoriametrics.otel.clientId
|
||
# The CI run poller, which reads it itself on its host.
|
||
hyperhiveCfg.swarm.ci-metrics.clientId
|
||
# Authelia's container's journal forwarder, which reads it back under the
|
||
# publisher's own identity.
|
||
hyperhiveCfg.swarm.authelia.otel.clientId
|
||
];
|
||
|
||
# The swarm controller's OIDC client, which it reads back from the store
|
||
# under its own certificate.
|
||
controllerClientId = hyperhiveCfg.swarm.controller.queueClientId;
|
||
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
|
||
|
||
secrets =
|
||
# `client_id` beside the value: an agent's id is derived per hive and has
|
||
# to be reconstructable from the store alone.
|
||
map (hive: {
|
||
clientId = agentClientId hive;
|
||
path = "secret/swarm/hives/${hive}/queue/agent";
|
||
withClientId = true;
|
||
}) hiveNames
|
||
# Each hive's own client, which that hive's hive-c0re reads under its own
|
||
# certificate. The nix half of
|
||
# `swarm_secret_client::queue::hive_client_path`.
|
||
++ map (hive: {
|
||
clientId = hiveClientId hive;
|
||
path = "secret/swarm/hives/${hive}/oidc/client";
|
||
}) hiveNames
|
||
# No `client_id`: a service's client id is the swarm-wide option both
|
||
# ends already read.
|
||
++ map (id: {
|
||
clientId = id;
|
||
path = "secret/swarm/services/${id}/oidc/client";
|
||
}) serviceClientIds
|
||
# Under `controller/`, which no hive reads: this client may read every
|
||
# hive's status. The nix half of
|
||
# `swarm_secret_client::queue::controller_client_path`.
|
||
++ [
|
||
{
|
||
clientId = controllerClientId;
|
||
path = controllerSecretPath;
|
||
}
|
||
];
|
||
in
|
||
{
|
||
options.services.hyperhive.deploy.swarm-secret-publisher = {
|
||
enable = lib.mkOption {
|
||
type = lib.types.bool;
|
||
default = false;
|
||
description = ''
|
||
Publish the OIDC client secrets this host mints into the swarm's
|
||
secret store, so a hive that does not run authelia can read its
|
||
own and its agents' credentials and a swarm service or the
|
||
controller can read its own — from wherever it runs, this host
|
||
included. The unit, `swarm-secret-publish`, runs inside authelia's
|
||
container, so it exists only where authelia runs.
|
||
|
||
Defaults to whether this host mints the OIDC secrets (set by
|
||
`glue-secret-publisher-authelia.nix`), which is the only half of the
|
||
question that is a property of *this* host.
|
||
|
||
⚠️ Deliberately **not** `deploy.bao.enable`. That asks whether the
|
||
store stands here, and a publisher beside a remote store, holding a
|
||
leaf issued out of band, is a deployment this exists to serve —
|
||
defaulting on co-location would leave exactly that shape silently
|
||
publishing nothing. Whether the wiring is complete is the client
|
||
identity's job (see `baoClientCertFile`), not this option's.
|
||
|
||
Turning it off leaves every hive but this one without its agents'
|
||
credential, every hive unable to publish its status, the swarm
|
||
controller unable to start, the swarm's Grafana without any login at
|
||
all, its collector pushing unauthenticated, the queue's responder
|
||
unable to start, the homeserver unable to start, and the forge's login
|
||
source without its secret.
|
||
Each secret has exactly one route and this is the producer's
|
||
end of it, so the honest reason to set it false is a deployment
|
||
delivering those secrets by some other mechanism it owns.
|
||
'';
|
||
};
|
||
|
||
baoClientCertFile = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = null;
|
||
description = ''
|
||
Client certificate this publisher presents to the store. Its subject
|
||
must be {option}`services.hyperhive.deploy.bao.secretPublisherCommonName`
|
||
— cert auth matches on the CN, and the role accepts nothing else.
|
||
|
||
A path on the host. ./swarm-authelia.nix hands it into authelia's
|
||
container as an nspawn system credential, and nspawn refuses to
|
||
start the container while a credential's source is missing: set,
|
||
the file has to exist before authelia's container starts.
|
||
|
||
No default: a module that guessed would be holding the CA opinion
|
||
./swarm-bao.nix deliberately does not hold.
|
||
./glue-secret-publisher-bao-identity.nix points it at the leaf
|
||
./glue-bao-tls.nix mints, where this host mints one.
|
||
'';
|
||
};
|
||
|
||
baoClientKeyFile = lib.mkOption {
|
||
type = lib.types.nullOr lib.types.str;
|
||
default = null;
|
||
description = ''
|
||
Private key for {option}`services.hyperhive.deploy.swarm-secret-publisher.baoClientCertFile`.
|
||
Both or neither — the unit does not exist unless each is set. A host
|
||
path, handed in like the certificate.
|
||
'';
|
||
};
|
||
|
||
secrets = lib.mkOption {
|
||
type = lib.types.listOf lib.types.attrs;
|
||
readOnly = true;
|
||
internal = true;
|
||
default = secrets;
|
||
description = ''
|
||
Every secret the publisher copies: the minted client's id, the store
|
||
path its reader fetches, and whether `client_id` is stored beside the
|
||
value. Derived from `swarm.*` alone, so it is the same list on every
|
||
host; ./swarm-authelia.nix hands it to the unit in its container.
|
||
'';
|
||
};
|
||
};
|
||
}
|