0f7b403803
A new swarm service, swarm-notify-router, in its own container on the host
that sets deploy.notify-router.enable (defaulted on by allSwarmServices where
the host has the router's store leaf, like the claude login minter).
- swarm-controller registers instance-wide hooks for `issues` +
`issue_assign` and `pull_request` + `pull_request_assign`, updates the
event list of a hook already registered with fewer (leaving its active flag as it is), and keeps each
verified body unchanged in the SWARM_EVENTS stream on
$SWARM.events.forge.<X-Forgejo-Event-Type>, deduped by X-Forgejo-Delivery.
A delivery the stream did not keep answers 503.
- swarm-notify-router drains its durable consumer on that stream, runs its
flows (source -> match -> todo sink, stored as nodes plus edges and checked
to be a tree) and queues each op on $SWARM.router.todo.<agent> in the
work-queue stream ROUTER_TODOS. Built-in flows: assigned, unassigned (same
key, rewritten) and closed (prefix-clears every holder's router todos). A
per-issue snapshot in its notify-router-state bucket holds the last
assignees and the todo holders; only agents in the agent-status bucket get
ops, and an assignee that was not one is left out of the snapshot so the
next assign event retries it. After the first event that must be retried,
the rest of the batch is nakked so redelivery keeps stream order.
- hive-agent creates its own durable consumer todo-<agent> and applies each
op to its todo store as subsystem `router`, then acks; a refused op stops
the rest of its batch the same way. The store's (subsystem, key) dedup
makes a redelivery harmless. New Todos::clear_prefix.
- swarm-nats-auth: a --notify-router-client arm; the controller may publish
$SWARM.events.forge.*; each agent gets three {agent} templates (extended
consumer create whose filter the server checks, MSG.NEXT, ACK).
- Credentials: a notify-router store leaf minted by glue-bao-tls, a
swarm-notify-router bao policy (read on its own queue client secret), an
authelia machine client, the secret publisher entry, a journal forwarder
route. No manual step.
hive-forge-notify is unchanged, so agents also still get forge's own
notifications.
Refs #5571
481 lines
20 KiB
Nix
481 lines
20 KiB
Nix
# "What does THIS host deploy?"
|
|
#
|
|
# Separated from `services.hyperhive.swarm.*` because those are two
|
|
# different kinds of fact and only one of them varies per machine:
|
|
#
|
|
# swarm.* — swarm-wide truth. The swarm's name, domain, hives, peers,
|
|
# and where each service lives. **Identical on every
|
|
# host**, byte for byte; a hive needs all of it to be a
|
|
# *client* of the swarm.
|
|
# deploy.* — this machine's deployment decisions. Necessarily different
|
|
# on every host, because that is what a deployment is.
|
|
#
|
|
# The `enable` toggles live here rather than under `swarm.*` so the
|
|
# namespace that is identical everywhere does not carry the one thing
|
|
# that must differ per host.
|
|
#
|
|
# Flat and named for the thing deployed — `deploy.forgejo`, not
|
|
# `deploy.swarmServices.forgejo`: grouping by "swarm service" re-encodes
|
|
# the service-side taxonomy into a layer that does not care about it.
|
|
#
|
|
# ⚠️ Each entry is an attrset with an `enable`, not a bare bool, so a
|
|
# service that grows a second *deployment* decision has somewhere to put
|
|
# it — `deploy.nats = { enable; authPackage; }` is then an ordinary
|
|
# addition rather than a migration, where a bare bool leaves it
|
|
# unrepresentable.
|
|
#
|
|
# ⚠️ The renames below are deliberately in this one file rather than
|
|
# spread across the service modules, so all of them have a single home
|
|
# and a single file to delete when the deprecation window closes.
|
|
{ config, lib, ... }:
|
|
let
|
|
# The swarm UI's own switch, in both spellings. Accepted and ignored: a
|
|
# definition only produces a warning, so a host that still sets one keeps
|
|
# evaluating. `null` is "not set".
|
|
swarmUiSwitchIgnored = lib.mkOption {
|
|
type = lib.types.nullOr lib.types.bool;
|
|
default = null;
|
|
visible = false;
|
|
description = ''
|
|
No effect. swarm-ui runs wherever swarm-controller runs.
|
|
'';
|
|
};
|
|
swarmUiSwitchWarning =
|
|
path: value:
|
|
lib.optional (value != null)
|
|
"services.hyperhive.${path} no longer has any effect: swarm-ui now runs wherever swarm-controller runs. Remove this setting; services.hyperhive.deploy.swarm-controller.enable decides it.";
|
|
|
|
# The removed status-publishing secret below.
|
|
statusSecretRemoved = ''
|
|
hive-c0re reads this hive's status-publishing client secret from the
|
|
swarm's secret store, under the hive's own certificate
|
|
(deploy.bao.clientCertFile), and swarm-secret-publish in authelia's container
|
|
puts it there. Remove this definition; the file it named is read by
|
|
nothing now and can be deleted.
|
|
'';
|
|
in
|
|
{
|
|
imports = [
|
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "victoriametrics" "port" ] ''
|
|
VictoriaMetrics listens on a unix socket in a network namespace of its
|
|
own, and the gateway reaches it there, so it binds no host port. Remove
|
|
this definition.
|
|
'')
|
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "authelia" "port" ] ''
|
|
authelia listens on a unix socket in a network namespace of its own, and
|
|
the gateway reaches it there, so it binds no port. Remove this
|
|
definition.
|
|
'')
|
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "authelia" "metricsPort" ] ''
|
|
authelia's metrics listener is a unix socket the gateway scrapes, so it
|
|
binds no port. Remove this definition.
|
|
'')
|
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "authelia" "bridgePort" ] ''
|
|
swarm-authelia-bridge listens on a unix socket the gateway proxies to, so
|
|
it binds no port. Remove this definition.
|
|
'')
|
|
(lib.mkRemovedOptionModule [
|
|
"services"
|
|
"hyperhive"
|
|
"deploy"
|
|
"hive-controller"
|
|
"statusPublish"
|
|
"clientSecretFile"
|
|
] statusSecretRemoved)
|
|
|
|
# Same type, same meaning, new path — so a rename carries it exactly
|
|
# and existing configs keep evaluating with one warning naming both
|
|
# paths.
|
|
#
|
|
# The runner moved from `deploy.forgejo.ci` to `deploy.ci-runner` with
|
|
# every option keeping its type and meaning, so the old paths forward.
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "forgejo" "ci" "enable" ]
|
|
[ "services" "hyperhive" "deploy" "ci-runner" "enable" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "forgejo" "ci" "name" ]
|
|
[ "services" "hyperhive" "deploy" "ci-runner" "name" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "forgejo" "ci" "concurrency" ]
|
|
[ "services" "hyperhive" "deploy" "ci-runner" "concurrency" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "forgejo" "ci" "labels" ]
|
|
[ "services" "hyperhive" "deploy" "ci-runner" "labels" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "forgejo" "ci" "package" ]
|
|
[ "services" "hyperhive" "deploy" "ci-runner" "package" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "forgejo" "ci" "jobTimeout" ]
|
|
[ "services" "hyperhive" "deploy" "ci-runner" "jobTimeout" ]
|
|
)
|
|
|
|
# The gateway's services-issuer pair is a swarm-service identity, not a
|
|
# hive's, so it sits with the store's other host-unit readers.
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "baoClientCertFile" ]
|
|
[ "services" "hyperhive" "deploy" "bao" "servicesIssuerClientCertFile" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "baoClientKeyFile" ]
|
|
[ "services" "hyperhive" "deploy" "bao" "servicesIssuerClientKeyFile" ]
|
|
)
|
|
|
|
# What hive-c0re hands this hive's agents: a per-host choice, so it
|
|
# sits with the controller's other deployment options.
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "c0re" "claudeCodePackage" ]
|
|
[ "services" "hyperhive" "deploy" "hive-controller" "claudeCodePackage" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "c0re" "dashboardPort" ]
|
|
[ "services" "hyperhive" "deploy" "hive-controller" "dashboardPort" ]
|
|
)
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "otel" "extraResourceAttributes" ]
|
|
[ "services" "hyperhive" "deploy" "hive-controller" "otel" "extraResourceAttributes" ]
|
|
)
|
|
|
|
# The homeserver's port and its firewall switch, removed rather than
|
|
# renamed: tuwunel listens on a unix socket only the gateway reaches, so
|
|
# there is no host port to choose or open.
|
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "matrix" "httpPort" ] ''
|
|
The homeserver no longer listens on a TCP port: it runs in a network
|
|
namespace of its own and the gateway reaches it over a unix socket.
|
|
Remove this definition. Clients and federation peers reach it at
|
|
services.hyperhive.swarm.matrix.gatewayHost.
|
|
'')
|
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "matrix" "openFirewall" ] ''
|
|
The homeserver no longer listens on a host port, so there is no port
|
|
to open. Remove this definition. Reach from outside the host goes
|
|
through the gateway: services.hyperhive.gateway.openFirewall.
|
|
'')
|
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "matrix" "gui" "enable" ] ''
|
|
The matrix web client has no on/off switch: every host that runs the
|
|
homeserver serves it at services.hyperhive.swarm.matrix.gatewayHost,
|
|
and the swarm UI links to it. Remove this definition.
|
|
'')
|
|
|
|
# The hive CA's lifetime and its leaves', removed rather than renamed: no
|
|
# hive CA is generated, and the store's per-hive role sets the gateway
|
|
# leaf's lifetime.
|
|
(lib.mkRemovedOptionModule
|
|
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "caValidityDays" ]
|
|
''
|
|
No hive CA is generated any more: the gateway leaf is issued by the
|
|
secret store's pki mount, whose per-hive role sets its lifetime
|
|
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
|
|
definition. See docs/swarm/ca.md.
|
|
''
|
|
)
|
|
(lib.mkRemovedOptionModule
|
|
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "leafValidityDays" ]
|
|
''
|
|
No hive CA is generated any more: the gateway leaf is issued by the
|
|
secret store's pki mount, whose per-hive role sets its lifetime
|
|
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
|
|
definition. See docs/swarm/ca.md.
|
|
''
|
|
)
|
|
|
|
# `deploy.swarm-ui` keeps only its ignored `enable` (declared below); its
|
|
# `package` is the controller's UI package.
|
|
(lib.mkRenamedOptionModule
|
|
[ "services" "hyperhive" "deploy" "swarm-ui" "package" ]
|
|
[ "services" "hyperhive" "deploy" "swarm-controller" "ui" "package" ]
|
|
)
|
|
];
|
|
|
|
# ⚠️ Only `deploy.forgejo.enable` is declared here. The rest of
|
|
# `deploy.forgejo` is in ./hive-forge/default.nix: the knobs live with the
|
|
# module that reads them; this file stays the registry of toggles.
|
|
options.services.hyperhive.swarm.ui.enable = swarmUiSwitchIgnored;
|
|
|
|
config.warnings =
|
|
swarmUiSwitchWarning "deploy.swarm-ui.enable" config.services.hyperhive.deploy.swarm-ui.enable
|
|
++ swarmUiSwitchWarning "swarm.ui.enable" config.services.hyperhive.swarm.ui.enable;
|
|
|
|
options.services.hyperhive.deploy = {
|
|
swarm-ui.enable = swarmUiSwitchIgnored;
|
|
forgejo.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the swarm's forge in a `hive-forge` container on this host. A
|
|
swarm has one forge, so one host turns this on. It is the
|
|
canonical store for the meta flake and every agent's config repo,
|
|
so the swarm needs it, but *this* host running it is a decision
|
|
like any other shared service's.
|
|
|
|
With it off, this hive is a *client*: it reaches the swarm's
|
|
forge at {option}`services.hyperhive.swarm.forge.domain`, which
|
|
the operator's DNS must resolve to the forge's host. No container
|
|
is created, and an existing one's state stays on disk under
|
|
`/var/lib/nixos-containers/hive-forge/`.
|
|
|
|
{option}`services.hyperhive.deploy.allSwarmServices` turns it on,
|
|
and `singleHostSwarm` through that.
|
|
'';
|
|
};
|
|
|
|
grafana.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
description = ''
|
|
Run the swarm's metrics UI on this host.
|
|
|
|
Off by default and not derived from
|
|
{option}`services.hyperhive.deploy.hive-controller.enable`: a swarm
|
|
has one Grafana, so running it is a decision about this host rather
|
|
than about whether it runs a hive.
|
|
'';
|
|
};
|
|
|
|
victoriametrics.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
description = ''
|
|
Run the swarm's metrics store on this host.
|
|
|
|
Paired with
|
|
{option}`services.hyperhive.deploy.grafana.enable`: a store
|
|
with no UI is unreadable and a UI with no store is empty, so there
|
|
is no sensible deployment that takes one and not the other from
|
|
that switch. Set either directly to run exactly one.
|
|
'';
|
|
};
|
|
|
|
victorialogs.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
description = ''
|
|
Run the swarm's log store on this host.
|
|
|
|
A hive that is not the service host is a *client* of this store,
|
|
not a second one.
|
|
'';
|
|
};
|
|
|
|
bao.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the swarm's secret store in a `swarm-bao` container on this
|
|
host. A swarm has one store and it has to exist somewhere.
|
|
|
|
*Where* it runs is a separate question from *that* it runs: set
|
|
this directly to put the store on a host of its own, and clients
|
|
still reach it by name at
|
|
{option}`services.hyperhive.swarm.bao.domain` rather than at a
|
|
local address.
|
|
|
|
With it off, this hive is a *client*: it still reads its own
|
|
secrets from whoever runs the store, authenticating with its own
|
|
client certificate. Every hive needs the client half; only one
|
|
runs the server half, which is why the two live in different
|
|
namespaces.
|
|
'';
|
|
};
|
|
|
|
authelia.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the swarm's authelia in a `swarm-authelia` container on this
|
|
host. A swarm has one SSO provider, and this says it lives here.
|
|
|
|
With it off, this hive is a *client*:
|
|
{option}`services.hyperhive.swarm.authelia.url` still points at
|
|
whoever runs it, and no container is created. That asymmetry is
|
|
why the two live in different namespaces — every hive needs the
|
|
client half, only one runs the server half.
|
|
'';
|
|
};
|
|
|
|
swarm-otel.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
description = ''
|
|
Run the **swarm's** telemetry collector on this host.
|
|
|
|
A swarm has one of these, and it belongs wherever the shared
|
|
services live rather than on every hive.
|
|
|
|
Named `swarm-otel` rather than `otel` because there are two
|
|
collectors and the tier is the whole distinction:
|
|
{option}`services.hyperhive.otel.collector.enable` is the **hive-tier** one,
|
|
which every hive runs. A bare `deploy.otel` would not say which
|
|
it meant. A hive that does not run the swarm collector still runs
|
|
its own, and reaches this one by name at
|
|
{option}`services.hyperhive.swarm.otel.domain`.
|
|
'';
|
|
};
|
|
|
|
matrix.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
description = ''
|
|
Run the swarm's matrix homeserver — matrix-tuwunel, in a
|
|
`hive-matrix` container — on this host.
|
|
|
|
Set it directly to put the homeserver somewhere other than the
|
|
host holding the rest of the swarm's services.
|
|
|
|
Client-side settings stay in
|
|
{option}`services.hyperhive.swarm.matrix.*`, which every hive
|
|
agrees on; this is only the decision to run it here.
|
|
'';
|
|
};
|
|
|
|
nats.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
description = ''
|
|
Run the swarm's message queue in a `swarm-nats` container on this
|
|
host. A swarm has one queue, so at most one host turns this on —
|
|
but *which* host is its own decision, not necessarily the one
|
|
running the swarm's other shared services.
|
|
|
|
Off by default, and off means *absent*: no container is created
|
|
and nothing else in the evaluated config changes.
|
|
'';
|
|
};
|
|
|
|
ci-runner.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the swarm's Forgejo Actions runner in a `swarm-ci-runner`
|
|
container on this host. It reaches the forge at
|
|
{option}`services.hyperhive.swarm.forge.domain` and registers with
|
|
the token swarm-controller publishes to the secret store, so it
|
|
needs neither the forge nor a hive on this host.
|
|
|
|
Not derived from
|
|
{option}`services.hyperhive.deploy.allSwarmServices`: a runner
|
|
executes pull-request code, so running one is a decision of its own.
|
|
'';
|
|
};
|
|
|
|
review-classifier.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the review classifier in a `swarm-review-classifier` container on
|
|
this host. It answers the swarm UI's review try-outs with `claude` on
|
|
the Anthropic API key the operator writes to the secret store at
|
|
`secret/swarm/services/review-classifier/anthropic-api-key`, field
|
|
`api_key`.
|
|
|
|
On by default where
|
|
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
|
host has the classifier's store identity (the pair
|
|
{option}`services.hyperhive.deploy.bao.reviewClassifierClientCertFile`
|
|
and `reviewClassifierClientKeyFile`, minted on the store's host). The
|
|
key is never written for you: until the operator writes it the service
|
|
runs and every run fails with "no API key…".
|
|
'';
|
|
};
|
|
|
|
claude-login.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the claude login minter in a `swarm-claude-login` container on
|
|
this host. It runs `claude auth login` for the agent an operator logs
|
|
in from its card in the swarm UI, and writes the credentials to that
|
|
agent's `secret/swarm/agents/<agent>/claude-credentials`, which the
|
|
agent's harness installs. Run it on one host: a login in progress
|
|
lives in the service's memory.
|
|
|
|
On by default where
|
|
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
|
host has the minter's store identity (the pair
|
|
{option}`services.hyperhive.deploy.bao.claudeLoginClientCertFile`
|
|
and `claudeLoginClientKeyFile`, minted on the store's host).
|
|
'';
|
|
};
|
|
|
|
notify-router.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the notification router in a `swarm-notify-router` container on
|
|
this host. It reads the forge events swarm-controller keeps from its
|
|
webhooks and queues todos for the agents they concern: assigned,
|
|
unassigned, and cleared when the issue or pull request closes. Run
|
|
it on one host: it handles each issue's events in order.
|
|
|
|
On by default where
|
|
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
|
host has the router's store identity (the pair
|
|
{option}`services.hyperhive.deploy.bao.notifyRouterClientCertFile`
|
|
and `notifyRouterClientKeyFile`, minted on the store's host).
|
|
'';
|
|
};
|
|
|
|
builder.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run the swarm builder in a `swarm-builder` container on this host. It
|
|
builds each agent config PR's head on this host's agent base, through
|
|
this host's nix daemon, and swarm-controller writes the result as a
|
|
`swarm-builder` commit status on the PR.
|
|
|
|
On by default where
|
|
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
|
host has the builder's store identity (the pair
|
|
{option}`services.hyperhive.deploy.bao.builderClientCertFile` and
|
|
`builderClientKeyFile`, minted on the store's host). It evaluates
|
|
unreviewed config PRs and builds them through this host's nix daemon,
|
|
from a container with its own network namespace.
|
|
'';
|
|
};
|
|
|
|
hive-controller.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
example = true;
|
|
description = ''
|
|
Run a hive on this host: the `hive-c0re` coordinator that owns every
|
|
agent container here, its `hive-priv` helper, and the bridge,
|
|
resolver and gateway the agents hang off.
|
|
|
|
Every hive turns this on. A host that only runs the swarm's shared
|
|
services ({option}`services.hyperhive.deploy.allSwarmServices`)
|
|
leaves it off.
|
|
'';
|
|
};
|
|
|
|
swarm-controller.enable = lib.mkOption {
|
|
type = lib.types.bool;
|
|
default = false;
|
|
description = ''
|
|
Run the swarm-controller daemon on this host.
|
|
|
|
Off by default and deliberately not derived from
|
|
{option}`services.hyperhive.deploy.hive-controller.enable`: a swarm
|
|
has one controller, so running it is a decision about this host
|
|
rather than about whether it runs a hive.
|
|
|
|
Also serves the swarm UI from this host. The UI is a view onto the
|
|
controller's state and reaches it over that daemon's unix socket,
|
|
so the host that runs the controller is the only host that can
|
|
serve it.
|
|
'';
|
|
};
|
|
};
|
|
}
|