From bed4358e191ef69fb3add4a987b10cab6d3f478f Mon Sep 17 00:00:00 2001 From: atlas Date: Sun, 4 Oct 2026 17:26:13 +0200 Subject: [PATCH] swarm options: no swarm.* value depends on a deploy option or this host's hive - swarm.serviceDomains lists every swarm service name on every host. The store's issuing role is built from it, so the store now admits a leaf for grafana, metrics, otel, logs, the bao UI and the apex whichever host runs the store. Each gateway's own leaf stays narrowed by localServiceDomains. - swarm.matrix.apiUrl defaults to http:// (null when gatewayHost is null) instead of loopback on the matrix host. - The authelia vhost proxies /bridge/ to swarm-authelia-bridge with the prefix stripped, and deploy.swarm-controller.authBridgeUrl defaults to ${swarm.authelia.url}/bridge on every host. swarm.authelia.bridgeUrl is removed. - swarm-authelia-bridge admits only swarm-controller's OIDC client on every request (403 for any other active token), configured by SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID (renamed from ..._OPERATOR_READER_CLIENT_ID, which gated ListOperators alone). - swarm.peerHives is removed; swarm-wireguard.nix filters this host's own hive out of swarm.hives itself. Closes #4933 --- docs/integrations/matrix.md | 8 +- docs/networking/gateway.md | 1 + docs/swarm/README.md | 6 +- docs/swarm/ca.md | 3 +- docs/swarm/services.md | 8 + docs/swarm/sso.md | 7 +- hive-c0re/src/server.rs | 11 +- nix/host-modules/hive-c0re/default.nix | 6 +- nix/host-modules/hive-c0re/environment.nix | 13 +- nix/host-modules/hive-matrix-service.nix | 18 +- nix/host-modules/swarm-authelia-service.nix | 31 +-- nix/host-modules/swarm-authelia.nix | 15 +- nix/host-modules/swarm-controller.nix | 18 +- nix/host-modules/swarm-wireguard.nix | 12 +- nix/host-modules/swarm.nix | 79 +++----- nix/module-eval/core-toggle.nix | 16 ++ nix/module-eval/nats-authelia.nix | 24 ++- swarm-authelia-bridge/README.md | 17 +- swarm-authelia-bridge/src/main.rs | 207 ++++++++------------ swarm-controller/src/auth.rs | 6 +- swarm-controller/src/main.rs | 3 +- 21 files changed, 226 insertions(+), 283 deletions(-) diff --git a/docs/integrations/matrix.md b/docs/integrations/matrix.md index 6ce2b3cf..ccab6ab5 100644 --- a/docs/integrations/matrix.md +++ b/docs/integrations/matrix.md @@ -82,10 +82,10 @@ pin the value before the homeserver mints any ids under the default. ## Default-closed firewall -`deploy.matrix.openFirewall` defaults to `false`: the host -reaches the homeserver on loopback, and agent containers reach it -at `chat.` via the gateway — so the firewall hole only -matters for access from *outside* the host. Flip to `true` when +`deploy.matrix.openFirewall` defaults to `false`: hive-c0re and agent +containers on every hive reach the homeserver at `chat.` +via the gateway, so the firewall hole only matters for direct access to +`httpPort` from *outside* the host. Flip to `true` when announcing the homeserver to other hives or when an external matrix client needs to reach the client-server API directly. diff --git a/docs/networking/gateway.md b/docs/networking/gateway.md index e6dc0cf0..58708513 100644 --- a/docs/networking/gateway.md +++ b/docs/networking/gateway.md @@ -14,6 +14,7 @@ You rarely switch it on yourself. `gateway.enable` defaults to off, and every mo | --- | --- | --- | | `/` | swarm-ui dist (static), behind an authelia subrequest | `swarm-ui.nix`, `deploy.swarm-ui.enable` | | `auth./` | authelia (`9091`) | `swarm-authelia.nix`, `deploy.authelia.enable` | +| `auth./bridge/*` | swarm-authelia-bridge (`9092`), prefix stripped; answers swarm-controller's client only | `swarm-authelia.nix`, `deploy.authelia.enable` | | `forge./` | forgejo (`3000`) | `hive-forge/`, `deploy.forgejo.enable` | | `chat./_matrix/*` | tuwunel (`8008`) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` | | `chat./` | fluffychat-web static | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` | diff --git a/docs/swarm/README.md b/docs/swarm/README.md index bb0a9c7f..bc43d1e6 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -472,9 +472,9 @@ services.hyperhive = { host's routing table). - It opens UDP port 51820 (or `listenPort`) on the host firewall. - `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly - from `services.hyperhive.swarm.peerHives` to build `wg-hive`'s - `allowedIPs`, so intra-swarm traffic can route over the mesh address - rather than the public domain. + from `services.hyperhive.swarm.hives`, minus this host's own hive, to + build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over + the mesh address rather than the public domain. - It sets `persistentKeepalive = 25` by default; override or null to disable (not needed when both sides have public IPs and no NAT). diff --git a/docs/swarm/ca.md b/docs/swarm/ca.md index ca76dcfe..a7e340a2 100644 --- a/docs/swarm/ca.md +++ b/docs/swarm/ca.md @@ -54,7 +54,8 @@ copies from. Every hive does this with its own identity, so holding the swarm root's private key stopped being what decides whether a hive can serve its swarm's names. -`services.hyperhive.swarm.serviceDomains` populates the role's +`services.hyperhive.swarm.serviceDomains`, every swarm service name and the +same list on every host, populates the role's `allowed_domains`, so asking for a name nobody configured is a refusal from the store naming that name — not a certificate quietly issued for it. A host's own SANs are narrower still: it asks only for the names it diff --git a/docs/swarm/services.md b/docs/swarm/services.md index b0dfdff8..40181a0d 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -106,6 +106,14 @@ there is one IdP and one auth path. Agent subjects come from swarm-controller's agent-creation job, written into the users database by `swarm-authelia-bridge`; human ones come from `swarmctl user add` → [setup.md § 2](../getting-started/setup.md#2--your-sso-account). + +The bridge listens on loopback on the authelia host, and the gateway +publishes it as +`https:///bridge/`, which is where +`deploy.swarm-controller.authBridgeUrl` points by default on every host. +It answers only swarm-controller's OIDC client +(`swarm.controller.queueClientId`) and refuses every other token `403`. + On first boot this module seeds the users database with a disabled placeholder subject, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia generates session and storage keys in the container on first boot and never rotates them automatically; replacing one invalidates data already written (sessions, the encrypted store), so diff --git a/docs/swarm/sso.md b/docs/swarm/sso.md index 90caeca8..cc04bdf9 100644 --- a/docs/swarm/sso.md +++ b/docs/swarm/sso.md @@ -249,9 +249,10 @@ HTML instead of the failure that actually happened: - `/api/oidc/token`, `/.well-known/openid-configuration` — a client parsing an error page as its JSON document -The gateway routes authelia's `/api/` and `/.well-known/` prefixes -**without** error interception. The split is by _audience_, not by an enumerated path -list: a human gets the page, every machine caller gets the status. +The gateway routes authelia's `/api/` and `/.well-known/` prefixes, and +the bridge's `/bridge/`, **without** error interception. The split is by +_audience_, not by an enumerated path list: a human gets the page, every +machine caller gets the status. Enumerating endpoints individually would leave the next one added silently intercepted. diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 5436c96f..729fbd46 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -451,17 +451,16 @@ async fn handle_set_resource_limits( /// Guard: matrix provisioning needs a homeserver to provision against. /// -/// Answers "is one configured", not "is one running here" — the message names -/// both ways to get there, since a hive that talks to someone else's -/// homeserver never enables the local container at all. +/// Answers "is one configured", not "is one running here": the address is +/// the swarm's homeserver, wherever it runs. fn require_matrix_present() -> Result<()> { if crate::matrix::is_present() { return Ok(()); } anyhow::bail!( - "no matrix homeserver configured — set services.hyperhive.deploy.matrix.enable = true to run one \ - here, or services.hyperhive.swarm.matrix.apiUrl to point at an existing one, before \ - provisioning matrix users" + "no matrix homeserver configured — set services.hyperhive.swarm.matrix.apiUrl (it defaults \ + to the swarm.matrix.gatewayHost vhost, and is null when that is) before provisioning \ + matrix users" ) } diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index 10788432..a81cabfd 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -127,9 +127,9 @@ in ./options.nix ./theme.nix # Hive-CA trust for this daemon's outbound TLS. Nothing it is given by - # default is https — the forge and matrix URLs resolve to plain http or - # loopback, and the queue's TLS is verified against - # `HIVE_C0RE_OIDC_CA_FILE` (../hive-tls.nix), not this bundle — so this + # default is https — the forge and matrix URLs are plain http, and the + # queue's TLS is verified against `HIVE_C0RE_OIDC_CA_FILE` + # (../hive-tls.nix), not this bundle — so this # changes nothing on an all-local hive. It matters for the split-host # shape the options invite: `swarm.matrix.apiUrl`'s own example is # `https://matrix.example.com`, and pointing it at another hive's diff --git a/nix/host-modules/hive-c0re/environment.nix b/nix/host-modules/hive-c0re/environment.nix index 85cd8f84..77751537 100644 --- a/nix/host-modules/hive-c0re/environment.nix +++ b/nix/host-modules/hive-c0re/environment.nix @@ -180,15 +180,10 @@ in // lib.optionalAttrs (config.services.hyperhive.swarm.matrix.apiUrl != null) { # Client-server API base hive-c0re uses to provision matrix (register # agent users, create the hive space + chat room, invite members). - # Supplied by `services.hyperhive.swarm.matrix.apiUrl`, which the matrix - # module fills in with its own loopback listener when it is the thing - # running tuwunel — and which the operator sets by hand when the - # homeserver lives on another machine. - # - # NOT the agent-facing HIVE_MATRIX_URL above: that one is the gateway - # vhost, and it is absent whenever there is no vhost. Reusing it here - # would silently stop provisioning on a hive that runs matrix without - # one. + # Supplied by `services.hyperhive.swarm.matrix.apiUrl`, which defaults to + # the same gateway vhost as HIVE_MATRIX_URL above but is its own option, + # so an operator can point provisioning elsewhere without changing what + # agents are handed. HIVE_MATRIX_API_URL = config.services.hyperhive.swarm.matrix.apiUrl; } // { diff --git a/nix/host-modules/hive-matrix-service.nix b/nix/host-modules/hive-matrix-service.nix index bd62a9d1..1014e8ce 100644 --- a/nix/host-modules/hive-matrix-service.nix +++ b/nix/host-modules/hive-matrix-service.nix @@ -10,7 +10,6 @@ let cfg = config.services.hyperhive.swarm.matrix; swarmDomain = config.services.hyperhive.swarm.domain; - deployCfg = config.services.hyperhive.deploy; in { options.services.hyperhive.swarm.matrix = { @@ -65,11 +64,10 @@ in apiUrl = lib.mkOption { type = lib.types.nullOr lib.types.str; - default = if deployCfg.matrix.enable then "http://127.0.0.1:${toString cfg.httpPort}" else null; + default = if cfg.gatewayHost == null then null else "http://${cfg.gatewayHost}"; defaultText = lib.literalExpression '' - if services.hyperhive.deploy.matrix.enable - then "http://127.0.0.1:''${toString services.hyperhive.swarm.matrix.httpPort}" - else null + if services.hyperhive.swarm.matrix.gatewayHost == null then null + else "http://''${services.hyperhive.swarm.matrix.gatewayHost}" ''; example = "https://matrix.example.com"; description = '' @@ -79,12 +77,10 @@ in `hyperhive.matrix.url`, which is the gateway vhost handed to each agent's `hive-matrix-daemon`. - Defaults to the loopback listener **only when this module is the - thing running tuwunel** — in that case the address is not a - guess, it is where this module just put the container. Set it - explicitly (with `enable = false`) when the homeserver runs on - another machine; "everything on one host" is a special case of - the full deployment, not the assumption. + Defaults to the homeserver's gateway vhost (`gatewayHost`), the + same name on every hive whichever host runs tuwunel, and `null` + when there is no vhost. Set it explicitly when the homeserver is + reachable only some other way. `null` means hive-c0re has no homeserver to provision against and matrix provisioning no-ops. There is deliberately no diff --git a/nix/host-modules/swarm-authelia-service.nix b/nix/host-modules/swarm-authelia-service.nix index 1545dda4..f548df4d 100644 --- a/nix/host-modules/swarm-authelia-service.nix +++ b/nix/host-modules/swarm-authelia-service.nix @@ -10,7 +10,6 @@ let cfg = config.services.hyperhive.swarm.authelia; swarmDomain = config.services.hyperhive.swarm.domain; - deployCfg = config.services.hyperhive.deploy; # The unit name upstream's `services.authelia.instances.` derives, # which `unit` publishes. ./swarm-authelia.nix binds the same two names for @@ -432,32 +431,10 @@ in (`127.0.0.1:''${bridgePort}`) — one above authelia's own default `port` (9091), outside hyperhive's other claimed ranges. - Reachable directly from this host's other processes (this - container shares the host netns, same as authelia's own `port`) - without going through the gateway — this is an internal - service-to-service endpoint, not something meant to be exposed - publicly. - ''; - }; - - bridgeUrl = lib.mkOption { - type = lib.types.nullOr lib.types.str; - readOnly = true; - default = if deployCfg.authelia.enable then "http://127.0.0.1:${toString cfg.bridgePort}" else null; - defaultText = lib.literalExpression ''if enable then "http://127.0.0.1:''${bridgePort}" else null''; - description = '' - Where `swarm-authelia-bridge` answers, **as seen from this - host** — correct only when a caller (`swarm-controller`) also - runs on this host, the same co-location assumption - `swarm.nix`'s `clientSecretFile` documents for its own - cross-host case. `null` when this host doesn't run - `swarm-authelia` at all. - - A split-host swarm has no automated delivery for this address: - the operator points `swarm-controller`'s own option at wherever - this host has made the bridge reachable (a firewall rule, a - different bind address), the same manual-copy shape used - throughout this codebase's other cross-host cases. + Not published on its own: the authelia vhost proxies + `/bridge/` to it, and that is the address `swarm-controller` + uses from any host. The bridge answers only + `swarm-controller`'s OIDC client. ''; }; }; diff --git a/nix/host-modules/swarm-authelia.nix b/nix/host-modules/swarm-authelia.nix index 12168038..52c1acc4 100644 --- a/nix/host-modules/swarm-authelia.nix +++ b/nix/host-modules/swarm-authelia.nix @@ -759,6 +759,13 @@ in locations."/.well-known/" = { proxyPass = "http://127.0.0.1:${toString cfg.port}"; }; + # `swarm-authelia-bridge`, a separate process from authelia, also a + # machine surface. The trailing slash on `proxyPass` strips the + # prefix, so `/bridge/requests` reaches the bridge as `/requests`. + # The bridge itself refuses every client but swarm-controller's. + locations."/bridge/" = { + proxyPass = "http://127.0.0.1:${toString cfg.bridgePort}/"; + }; }; # Order the container after the host CA generator, so the bind source @@ -924,10 +931,10 @@ in # includes `bridgeClient`) — same file this container's own # `renderClient` reads the digest half of. SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE = "${clientsDir}/${bridgeClientId}.secret"; - # The operator list names who holds operator rights, so the - # bridge answers it to the controller's client and refuses every - # other active token (each hive and its agents hold one). - SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID = hyperhiveCfg.swarm.controller.queueClientId; + # The one client the bridge answers. Every other active token + # (each hive and its agents hold one) is refused, on every + # request, since `/bridge/` is reachable from any host. + SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID = hyperhiveCfg.swarm.controller.queueClientId; SWARM_AUTHELIA_BRIDGE_OPERATOR_GROUP = operatorGroup; }; }; diff --git a/nix/host-modules/swarm-controller.nix b/nix/host-modules/swarm-controller.nix index 56a303bb..ae577574 100644 --- a/nix/host-modules/swarm-controller.nix +++ b/nix/host-modules/swarm-controller.nix @@ -587,17 +587,14 @@ in authBridgeUrl = lib.mkOption { type = lib.types.nullOr lib.types.str; - default = if deployCfg.authelia.enable then autheliaCfg.bridgeUrl else null; - defaultText = lib.literalExpression '' - authelia's own `bridgeUrl` when this host also runs - `swarm-authelia`, else null - ''; - example = "http://127.0.0.1:9092"; + default = "${autheliaCfg.url}/bridge"; + defaultText = lib.literalExpression ''"''${services.hyperhive.swarm.authelia.url}/bridge"''; + example = "https://auth.example.com/bridge"; description = '' Where `swarm-authelia-bridge` (the only writer of authelia's - users database) answers — see that option's own doc comment for - the cross-host caveat, since this default is only correct when - this host also runs `swarm-authelia`. + users database) answers. The default is the `/bridge/` location + on the authelia vhost, the same address from every host whether + or not it runs `swarm-authelia`; the daemon appends `/requests`. No new credential to configure: the bearer token presented to the bridge is minted from THIS daemon's own existing queue OIDC @@ -879,8 +876,7 @@ in environment = { SWARM_CONTROLLER_SOCKET = deployCfg.swarm-controller.socketPath; # The swarm's hive directory, JSON-encoded — the full directory - # (this daemon has no "self" hive to exclude, unlike - # `swarm.peerHives`, `swarm.hives` minus this hive) rather than + # (this daemon has no "self" hive to exclude) rather than # peers-minus-self. Consumed by `GET /api/hives` # (swarm-controller/src/main.rs::load_hives). # The one blacklist of names an agent may not take, shared verbatim diff --git a/nix/host-modules/swarm-wireguard.nix b/nix/host-modules/swarm-wireguard.nix index fb913f8e..2df2b82a 100644 --- a/nix/host-modules/swarm-wireguard.nix +++ b/nix/host-modules/swarm-wireguard.nix @@ -114,12 +114,14 @@ networking.wireguard.interfaces = let wgCfg = config.services.hyperhive.deploy.wireguard; - # `peerHives` is `swarm.hives` minus this hive (../swarm.nix) — - # a mesh that included our own entry would configure a tunnel to - # ourselves. + # `swarm.hives` minus this host's own hive: a mesh that included + # our own entry would configure a tunnel to ourselves. meshPeers = lib.filterAttrs ( - _: p: p.wireguardPublicKey != null && p.wireguardAddress != null - ) config.services.hyperhive.swarm.peerHives; + name: p: + name != config.services.hyperhive.hiveName + && p.wireguardPublicKey != null + && p.wireguardAddress != null + ) config.services.hyperhive.swarm.hives; in { wg-hive = { diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 891a769b..129e00da 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -1,6 +1,6 @@ # The swarm's directory: one entry per hive, **including this one**, # identical on every host in the swarm. `services.hyperhive.hiveName` -# says which entry is us, and `peerHives` below derives the rest. +# says which entry is us. # # Why a directory rather than a per-host peer list: every field here is # intrinsic to the hive it describes — none of them says anything about @@ -78,41 +78,31 @@ let # assertion encoded an intended shape, not a property of the code, and # it rejected the supported migration path: a hive pinning its old # `forge.` while joining a swarm at a different apex. + # + # Every name is listed whether or not this host runs the service: the + # store's issuing role is built from this list on the store's host and + # has to admit a leaf for any gateway in the swarm. What one gateway's + # leaf carries is `localServiceDomains`. + # + # Membership is also what `gateway.lib.tlsFor` consults to pick the + # services leaf over the hive one, so a gateway-published swarm name + # missing here is served the hive certificate, which cannot cover a name + # under a different apex. serviceDomains' = [ swarmCfg.forge.domain swarmCfg.matrix.gatewayHost swarmCfg.authelia.domain - ] - # The swarm UI's name is a SIBLING of the other three, not a parent of - # them — the apex is as much a name needing a certificate as - # `forge.` is, and no CA in the hierarchy issues for it - # implicitly. Left out, its vhost falls back to the hive leaf and the - # swarm's front page opens with a name mismatch. - ++ lib.optional (deployCfg.swarm-ui.enable && swarmCfg.domain != null) swarmCfg.domain - # Every swarm service that claims a gateway name belongs here, and - # these three were missing it. Membership is what `gateway.lib.tlsFor` - # consults to pick the services leaf over the hive one, so a name - # absent from this list is served the HIVE certificate — which cannot - # cover a name under a different apex however the sub-CA is set up. - # - # ⚠️ How that stayed invisible: a mismatch a browser shows as a - # click-through warning is one a machine client rejects outright. The - # metrics UI and store looked fine for as long as only people opened - # them; the collector's exporter — same defect, no human in the loop — - # failed every POST and dropped the samples. - ++ lib.optional deployCfg.grafana.enable swarmCfg.grafana.domain - ++ lib.optional deployCfg.victoriametrics.enable swarmCfg.victoriametrics.domain - ++ lib.optional deployCfg.swarm-otel.enable swarmCfg.otel.domain - # VictoriaLogs' vhost is new (was previously unpublished entirely — see - # swarm-victorialogs.nix's file-top comment) and needs the same - # membership every gateway-published swarm service needs: absent from - # this list, `gateway.lib.tlsFor` falls back to the hive leaf, which - # cannot cover a name under a different apex — see the ⚠️ above this - # list for what that looked like the last time a name was missed here. - ++ lib.optional deployCfg.victorialogs.enable swarmCfg.victorialogs.domain - # The store's browser UI, not the store: `swarm.bao.domain` is mTLS and - # never gets a gateway certificate. - ++ lib.optional deployCfg.bao.enable swarmCfg.bao.ui.domain; + # The swarm UI's apex is a sibling of the other names, not a parent: + # no CA in the hierarchy issues for it implicitly. + swarmCfg.domain + swarmCfg.grafana.domain + swarmCfg.victoriametrics.domain + swarmCfg.otel.domain + swarmCfg.victorialogs.domain + # The store's browser UI, not the store: `swarm.bao.domain` is mTLS and + # never gets a gateway certificate. + swarmCfg.bao.ui.domain + ]; # Hives whose entry still carries the removed `certFingerprint`. Scanned # here, at top level, because that is the only place an assertion about a @@ -274,28 +264,17 @@ in ''; }; - options.services.hyperhive.swarm.peerHives = lib.mkOption { - type = lib.types.attrsOf (lib.types.attrsOf lib.types.unspecified); - readOnly = true; - internal = true; - description = '' - Read-only: `hives` minus this host's own entry. Derived once here - rather than in each consumer, because "everything that isn't me" - is a filter four different modules were re-implementing and only - one of them has to be wrong for a hive to peer with itself. - ''; - }; - options.services.hyperhive.swarm.serviceDomains = lib.mkOption { type = lib.types.listOf lib.types.str; readOnly = true; internal = true; description = '' - Read-only: the public hostnames of the swarm's own services, in a - stable sorted order. Second derived set alongside `peerHives`, and - here for the same reason — the CA that name-constrains these and - the leaf that carries them as SANs must agree exactly, and two - modules each assembling the list is how they stop agreeing. + Read-only: the public hostnames of every swarm service, in a + stable sorted order, identical on every host whichever services it + runs. Derived once here because the CA that name-constrains these, + the store role that issues for them and the leaf that carries them + as SANs must agree exactly, and two modules each assembling the + list is how they stop agreeing. Sorted and deduplicated deliberately: consumers compare this list against what they issued last time to decide whether to re-issue, @@ -326,8 +305,6 @@ in }; config = { - services.hyperhive.swarm.peerHives = lib.filterAttrs (name: _: name != cfg.hiveName) swarmCfg.hives; - services.hyperhive.swarm.serviceDomains = lib.sort (a: b: a < b) ( lib.unique (lib.filter (d: d != null && d != "") serviceDomains') ); diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index 07029155..d572e151 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -197,6 +197,22 @@ let && !(lib.hasInfix "auth.t.local" u) && lib.elem "auth.t.local" s; } + { + # The store's issuing role is built from this list on the store's host, + # so it cannot depend on which services the evaluating host runs. The + # last conjunct is the control: grafana runs on neither `bare` nor the + # forge host, and is still listed. + name = "the swarm's service names are the same on hosts running different services"; + ok = + let + s = h: h.services.hyperhive.swarm.serviceDomains; + in + s (hive { + deploy.forgejo.enable = true; + }) == s bare + && s allLocal == s bare + && lib.elem "grafana.t.local" (s bare); + } { # The narrowing's floor: no names left means no request, rather than a # request for an empty SAN set that the store would refuse. Its own diff --git a/nix/module-eval/nats-authelia.nix b/nix/module-eval/nats-authelia.nix index be098da3..fe9feca3 100644 --- a/nix/module-eval/nats-authelia.nix +++ b/nix/module-eval/nats-authelia.nix @@ -264,19 +264,33 @@ let && !(lib.hasInfix ''echo "users: {}"'' s); } { - # The bridge refuses its operator list to every client but the one - # named here, so this has to be the client the controller authenticates - # as. A drift is a controller whose admin sync is refused every pass. - name = "the authelia bridge answers its operator list to the controller's client only"; + # The bridge refuses every client but the one named here, so this has + # to be the client the controller authenticates as. A drift is a + # controller whose every bridge call is refused. + name = "the authelia bridge answers the controller's client only"; ok = let env = autheliaColocated.containers.swarm-authelia.config.systemd.services.swarm-authelia-bridge.environment; in - env.SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID + env.SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID == autheliaColocated.services.hyperhive.swarm.controller.queueClientId && env.SWARM_AUTHELIA_BRIDGE_OPERATOR_GROUP == "admins"; } + { + # One address from every host, and a location the IdP host serves: a + # controller off the authelia host has no other way to the bridge. + name = "the auth bridge address is the authelia vhost's /bridge/ on every host"; + ok = + let + url = h: h.services.hyperhive.deploy.swarm-controller.authBridgeUrl; + authelia = autheliaColocated.services.hyperhive.swarm.authelia; + location = autheliaColocated.services.nginx.virtualHosts.${authelia.domain}.locations."/bridge/"; + in + url autheliaNotColocated == "${autheliaNotColocated.services.hyperhive.swarm.authelia.url}/bridge" + && url autheliaColocated == url autheliaNotColocated + && location.proxyPass == "http://127.0.0.1:${toString authelia.bridgePort}/"; + } { # Config NAMES the IdP; it never computes where the IdP is. Both arms # matter together: the address is the swarm's name on a hive that runs diff --git a/swarm-authelia-bridge/README.md b/swarm-authelia-bridge/README.md index 03699d89..ce81bfa2 100644 --- a/swarm-authelia-bridge/README.md +++ b/swarm-authelia-bridge/README.md @@ -25,18 +25,17 @@ simply owns the file it writes. No elevated privilege anywhere. `ListOperators` (the operator group's usernames). - Bearer-authenticated via authelia's own OIDC token introspection (RFC 7662), against `swarm-controller`'s **existing** machine-client identity - (already minted for the queue connection) — no new credential. The two - agent verbs admit any active token. ⚠️ `ListOperators` admits only the - client named by `SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID` - (`swarm-controller`'s) and answers every other active token `403`: - authelia issues machine tokens to every hive and its agents too. + (already minted for the queue connection) — no new credential. ⚠️ Every + request admits only the client named by + `SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID` (`swarm-controller`'s) and + answers every other active token `403`: authelia issues machine tokens to + every hive and its agents too. - No restart of authelia after a write — relies on `authentication_backend.file.watch`, confirmed working against the pinned 4.39.20 build during this design work. -- Network-facing rather than unix-socket-only: `swarm-authelia`'s container - shares the host netns, so the same listener serves both a co-located - `swarm-controller` (loopback) and a split-host one (bind wider + firewall) - with no separate transport. +- Listens on loopback. The authelia vhost proxies `/bridge/` to it with + the prefix stripped, so `swarm-controller` on any host calls + `https:///bridge/requests`. ## One store, two writers diff --git a/swarm-authelia-bridge/src/main.rs b/swarm-authelia-bridge/src/main.rs index 1dde363f..0068fb7a 100644 --- a/swarm-authelia-bridge/src/main.rs +++ b/swarm-authelia-bridge/src/main.rs @@ -11,13 +11,10 @@ //! with root/`CAP_CHOWN`/a shared group (all examined and rejected — see //! `swarmctl/README.md`'s own identical analysis of this same file). //! -//! Network-facing, not unix-socket-only: `swarm-authelia`'s container -//! shares the host netns (`privateNetwork = false`, same as `hive-forge`/ -//! `hive-matrix`), so one listener serves a co-located `swarm-controller` -//! (loopback) and a split-host one (bind wider, firewall it) with the -//! same code path — no separate transport for the cross-host case. +//! Listens on loopback; the authelia vhost proxies `/bridge/` to it, so a +//! `swarm-controller` on any host reaches it at the same address. //! Bearer-authenticated via authelia's own OIDC token introspection (RFC -//! 7662) against `swarm-controller`'s **existing** machine-client +//! 7662), and admits only `swarm-controller`'s **existing** machine-client //! identity (already minted for the queue connection) — no new //! credential to mint or deliver, and always local/fast to verify since //! this process runs right next to the authelia it asks. @@ -25,7 +22,8 @@ //! One endpoint, `POST /requests`, body = //! [`swarm_authelia_bridge_sock::BridgeRequest`] verbatim: ensure an agent //! exists as an authelia subject, list the ones that do, and list the -//! operator group (to `swarm-controller` only, see `permit`). Not a +//! operator group. Every request is answered to `swarm-controller` only, +//! see `classify`. Not a //! wholesale-replace-the-file API — this process owns rendering //! `users.yml` internally; see `store` module. @@ -53,8 +51,8 @@ struct Config { introspection_url: String, client_id: String, client_secret: String, - /// The one client [`BridgeRequest::ListOperators`] answers. - operator_reader_client_id: String, + /// The one client this bridge answers: `swarm-controller`'s. + controller_client_id: String, /// The authelia group whose members are operators. operator_group: String, } @@ -68,7 +66,7 @@ impl Config { introspection_url: env_var("SWARM_AUTHELIA_BRIDGE_INTROSPECTION_URL")?, client_id: env_var("SWARM_AUTHELIA_BRIDGE_CLIENT_ID")?, client_secret: read_secret_file(&env_var("SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE")?)?, - operator_reader_client_id: env_var("SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID")?, + controller_client_id: env_var("SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID")?, operator_group: env_var("SWARM_AUTHELIA_BRIDGE_OPERATOR_GROUP")?, }) } @@ -146,28 +144,20 @@ async fn handle_request( headers: HeaderMap, Json(req): Json, ) -> Response { - let caller = match authorize(&state, &headers).await { - Ok(caller) => caller, - Err(resp) => return resp, - }; - respond(&state.config, &state.write_lock, caller.as_deref(), req).await + if let Err(resp) = authorize(&state, &headers).await { + return resp; + } + respond(&state.config, &state.write_lock, req).await } -/// Everything after authentication: the per-request [`permit`] check, then -/// the operation. Takes the config and the write lock rather than -/// [`AppState`], whose HTTP client only [`authorize`] uses. +/// Everything after authentication: the operation itself. Takes the config +/// and the write lock rather than [`AppState`], whose HTTP client only +/// [`authorize`] uses. async fn respond( config: &Config, write_lock: &tokio::sync::Mutex<()>, - caller: Option<&str>, req: BridgeRequest, ) -> Response { - if let Err(refusal) = permit(&req, caller, &config.operator_reader_client_id) { - // The client id is not a credential; naming it is what lets an - // operator see which client asked for something it may not have. - tracing::warn!(client_id = ?caller, request = ?req, "refused a request scoped to another client"); - return refuse(refusal); - } let (op, result) = match req { BridgeRequest::EnsureAgentIdentity { name } => { ("ensure_identity", handle(config, write_lock, name).await) @@ -194,10 +184,10 @@ async fn respond( /// "the op itself failed," a different case from "the caller was never let /// in." /// -/// Anything short of an active token is refused; **which status** depends -/// on whose credential failed, see [`Refusal`]. `Ok` carries the client the -/// token was issued to, for [`permit`]. -async fn authorize(state: &AppState, headers: &HeaderMap) -> Result, Response> { +/// Anything short of an active token from the controller's client is +/// refused; **which status** depends on whose credential failed, see +/// [`Refusal`]. +async fn authorize(state: &AppState, headers: &HeaderMap) -> Result<(), Response> { let Some(auth) = headers.get(axum::http::header::AUTHORIZATION) else { return Err(refuse(Refusal::Caller)); }; @@ -215,41 +205,37 @@ async fn authorize(state: &AppState, headers: &HeaderMap) -> Result Result, Refusal> { +/// Fail-closed: `Ok(())` is reachable only from an active token issued to +/// `controller`. authelia also issues machine tokens to every hive and every +/// hive's agents, and those are as active as the controller's, so they are +/// refused `403`; a token attributed to no client is refused too, rather +/// than compared as a blank. Split out of [`authorize`] so the rule is +/// testable without an authelia — and it is the *real* rule, not a +/// restatement of it, because this is the function the request path calls. +fn classify(verdict: introspect::Verdict, controller: &str) -> Result<(), Refusal> { match verdict { - introspect::Verdict::Active { client_id } => Ok(client_id), + introspect::Verdict::Active { client_id } if client_id.as_deref() == Some(controller) => { + Ok(()) + } + introspect::Verdict::Active { client_id } => { + // The client id is not a credential; naming it is what lets an + // operator see which client was refused. + tracing::warn!( + ?client_id, + "refused an active token issued to another client" + ); + Err(Refusal::Forbidden) + } introspect::Verdict::Inactive => Err(Refusal::Caller), introspect::Verdict::Unavailable => Err(Refusal::Bridge), } } -/// Whether an admitted caller may make this particular request. -/// -/// [`BridgeRequest::ListOperators`] answers only `operator_reader`, the -/// controller's client: authelia also issues machine tokens to every hive and -/// every hive's agents, and those are as active as the controller's. A caller -/// with no client id is refused it too, rather than compared as a blank. -fn permit( - req: &BridgeRequest, - client_id: Option<&str>, - operator_reader: &str, -) -> Result<(), Refusal> { - match req { - BridgeRequest::EnsureAgentIdentity { .. } | BridgeRequest::ListAgentIdentities => Ok(()), - BridgeRequest::ListOperators if client_id == Some(operator_reader) => Ok(()), - BridgeRequest::ListOperators => Err(Refusal::Forbidden), - } -} - /// Whose credential was at fault when a request was refused. /// /// The distinction is the whole point: these used to be one 401, so a @@ -262,13 +248,14 @@ enum Refusal { /// This bridge could not get an answer out of authelia at all, so the /// request was never evaluated. Bridge, - /// The token is valid, but its client may not make this request. + /// The token is valid, but was issued to a client other than the + /// controller's. Forbidden, } impl Refusal { - /// `401` for the caller; `503` for us; `403` for a valid caller asking - /// for something scoped to another client. + /// `401` for the caller; `503` for us; `403` for a valid token issued + /// to a client other than the controller's. /// /// 401 would be wrong for [`Refusal::Bridge`] under RFC 7235 — nothing /// about the caller's credential was determined — and it is also the @@ -456,34 +443,58 @@ async fn generate_password(bin: &std::path::Path) -> Result { #[cfg(test)] mod tests { use super::{ - BridgeRequest, Config, Refusal, StatusCode, classify, introspect::Verdict, permit, respond, + BridgeRequest, Config, Refusal, StatusCode, classify, introspect::Verdict, respond, }; + const CONTROLLER: &str = "swarm-controller"; + + fn active(client_id: Option<&str>) -> Verdict { + Verdict::Active { + client_id: client_id.map(str::to_owned), + } + } + /// Fail-closed, asserted per variant rather than as one negation: /// splitting blame out of a single denial is exactly the change that /// could have widened admission, so each refusing variant is named on /// its own line. #[test] fn only_an_active_token_is_admitted() { + assert_eq!(classify(active(Some(CONTROLLER)), CONTROLLER), Ok(())); assert_eq!( - classify(Verdict::Active { - client_id: Some("swarm-controller".to_owned()) - }), - Ok(Some("swarm-controller".to_owned())) - ); - assert_eq!( - classify(Verdict::Inactive), + classify(Verdict::Inactive, CONTROLLER), Err(Refusal::Caller), "authelia evaluated the token and said no" ); assert_eq!( - classify(Verdict::Unavailable), + classify(Verdict::Unavailable, CONTROLLER), Err(Refusal::Bridge), "a bridge that cannot ask must not admit — this variant reports a \ different fault, it is not a softer denial" ); } + /// The trust boundary. Every caller below holds a token authelia calls + /// active — a hive and a hive's agents are issued machine tokens too — + /// and only the controller's client is admitted. The first assertion is + /// the control: the same verdict shape with the controller's id passes. + #[test] + fn only_the_controllers_client_is_admitted() { + assert_eq!(classify(active(Some(CONTROLLER)), CONTROLLER), Ok(())); + for other in ["hive-alpha", "hive-alpha-agent", "swarm-authelia-bridge"] { + assert_eq!( + classify(active(Some(other)), CONTROLLER), + Err(Refusal::Forbidden), + "{other} holds an active token but is not the controller" + ); + } + assert_eq!( + classify(active(None), CONTROLLER), + Err(Refusal::Forbidden), + "a token authelia attributed to no client is not the controller's" + ); + } + /// The whole point of the split, asserted on the value rather than /// on message text — the previous code picked its status by matching on /// a log message's wording, which is what let the two collapse. @@ -521,51 +532,8 @@ mod tests { } } - const CONTROLLER: &str = "swarm-controller"; - - /// The trust boundary on the operator list. Every caller below holds a - /// token authelia calls active — a hive and a hive's agents are issued - /// machine tokens too — and only the controller's client is answered. #[test] - fn only_the_controller_may_list_operators() { - let req = BridgeRequest::ListOperators; - assert_eq!(permit(&req, Some(CONTROLLER), CONTROLLER), Ok(())); - for other in ["hive-alpha", "hive-alpha-agent", "swarm-authelia-bridge"] { - assert_eq!( - permit(&req, Some(other), CONTROLLER), - Err(Refusal::Forbidden), - "{other} holds an active token but is not the controller" - ); - } - assert_eq!( - permit(&req, None, CONTROLLER), - Err(Refusal::Forbidden), - "a token authelia attributed to no client is not the controller's" - ); - } - - /// The control for the test above: the agent verbs stay open to any - /// admitted caller, so the refusal there comes from the scoping and not - /// from `permit` refusing everyone who is not the controller. - #[test] - fn the_agent_verbs_stay_open_to_any_active_caller() { - for req in [ - BridgeRequest::EnsureAgentIdentity { - name: "atlas".to_owned(), - }, - BridgeRequest::ListAgentIdentities, - ] { - assert_eq!( - permit(&req, Some("hive-alpha"), CONTROLLER), - Ok(()), - "{req:?}" - ); - assert_eq!(permit(&req, None, CONTROLLER), Ok(()), "{req:?}"); - } - } - - #[test] - fn a_scoped_refusal_is_403_not_401() { + fn a_refused_client_is_403_not_401() { assert_eq!( Refusal::Forbidden.status(), StatusCode::FORBIDDEN, @@ -598,29 +566,16 @@ mod tests { introspection_url: String::new(), client_id: String::new(), client_secret: String::new(), - operator_reader_client_id: CONTROLLER.to_owned(), + controller_client_id: CONTROLLER.to_owned(), operator_group: "admins".to_owned(), } } - /// The request path, not only [`permit`]: a dispatch that skipped the - /// check would answer the hive below with the operator list. #[tokio::test] - async fn the_request_path_answers_the_operator_list_to_the_controller_only() { + async fn the_operator_list_is_the_operator_groups_members() { let config = config_with_one_operator(); let lock = tokio::sync::Mutex::new(()); - for caller in [Some("hive-alpha"), None] { - let refused = respond(&config, &lock, caller, BridgeRequest::ListOperators).await; - assert_eq!(refused.status(), StatusCode::FORBIDDEN, "{caller:?}"); - } - - let answered = respond( - &config, - &lock, - Some(CONTROLLER), - BridgeRequest::ListOperators, - ) - .await; + let answered = respond(&config, &lock, BridgeRequest::ListOperators).await; assert_eq!(answered.status(), StatusCode::OK); let body = axum::body::to_bytes(answered.into_body(), usize::MAX) .await diff --git a/swarm-controller/src/auth.rs b/swarm-controller/src/auth.rs index db551dc0..ab556395 100644 --- a/swarm-controller/src/auth.rs +++ b/swarm-controller/src/auth.rs @@ -11,9 +11,9 @@ //! connection attempt: no window in which this process holds a token //! that outlives its intended use. //! -//! `None` when this deployment did not wire a bridge up — the bridge only -//! exists on hosts that also run `swarm-authelia`, so a controller split -//! from it can neither create an identity nor read the roster. +//! `None` when this deployment set no bridge address; the default is the +//! `/bridge/` location on the authelia vhost, from every host. Without one, +//! this daemon can neither create an identity nor read the roster. //! //! Each caller answers that absence in its own terms rather than this //! module inventing a shared one: `SwarmNodeKind::CreateIdentity` fails diff --git a/swarm-controller/src/main.rs b/swarm-controller/src/main.rs index a1e90b92..ee0717a8 100644 --- a/swarm-controller/src/main.rs +++ b/swarm-controller/src/main.rs @@ -741,8 +741,7 @@ struct AppState { /// Env var the controller's NixOS module sets from /// `services.hyperhive.swarm.hives`, JSON-encoded — the full directory -/// (this daemon has no "self" to exclude) rather than peers-minus-self -/// (`services.hyperhive.swarm.peerHives`, which other consumers use). +/// (this daemon has no "self" to exclude) rather than peers-minus-self. const HIVES_ENV: &str = "SWARM_CONTROLLER_HIVES"; /// Parses [`HIVES_ENV`] into the swarm's hive directory. Unset or