vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm options: no swarm.* value depends on a deploy option or this host's hive

Browse source 
- swarm.serviceDomains lists every swarm service name on every host. The
  store's issuing role is built from it, so the store now admits a leaf
  for grafana, metrics, otel, logs, the bao UI and the apex whichever host
  runs the store. Each gateway's own leaf stays narrowed by
  localServiceDomains.
- swarm.matrix.apiUrl defaults to http://<swarm.matrix.gatewayHost>
  (null when gatewayHost is null) instead of loopback on the matrix host.
- The authelia vhost proxies /bridge/ to swarm-authelia-bridge with the
  prefix stripped, and deploy.swarm-controller.authBridgeUrl defaults to
  ${swarm.authelia.url}/bridge on every host. swarm.authelia.bridgeUrl is
  removed.
- swarm-authelia-bridge admits only swarm-controller's OIDC client on
  every request (403 for any other active token), configured by
  SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID (renamed from
  ..._OPERATOR_READER_CLIENT_ID, which gated ListOperators alone).
- swarm.peerHives is removed; swarm-wireguard.nix filters this host's own
  hive out of swarm.hives itself.

Closes #4933
This commit is contained in:
atlas 2026-10-04 17:26:13 +02:00
commit bed4358e19
21 changed files with 226 additions and 283 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/integrations/matrix.md
View file

@ -82,10 +82,10 @@ pin the value before the homeserver mints any ids under the default.
## Default-closed firewall
`deploy.matrix.openFirewall` defaults to `false`: the host
reaches the homeserver on loopback, and agent containers reach it
at `chat.<swarm-domain>` via the gateway — so the firewall hole only
matters for access from *outside* the host. Flip to `true` when
`deploy.matrix.openFirewall` defaults to `false`: hive-c0re and agent
containers on every hive reach the homeserver at `chat.<swarm-domain>`
via the gateway, so the firewall hole only matters for direct access to
`httpPort` from *outside* the host. Flip to `true` when
announcing the homeserver to other hives or when an external matrix
client needs to reach the client-server API directly.

1
docs/networking/gateway.md
View file

@ -14,6 +14,7 @@ You rarely switch it on yourself. `gateway.enable` defaults to off, and every mo
| --- | --- | --- |
| `<swarm>/` | swarm-ui dist (static), behind an authelia subrequest | `swarm-ui.nix`, `deploy.swarm-ui.enable` |
| `auth.<swarm>/` | authelia (`9091`) | `swarm-authelia.nix`, `deploy.authelia.enable` |
| `auth.<swarm>/bridge/*` | swarm-authelia-bridge (`9092`), prefix stripped; answers swarm-controller's client only | `swarm-authelia.nix`, `deploy.authelia.enable` |
| `forge.<swarm>/` | forgejo (`3000`) | `hive-forge/`, `deploy.forgejo.enable` |
| `chat.<swarm>/_matrix/*` | tuwunel (`8008`) | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |
| `chat.<swarm>/` | fluffychat-web static | `hive-matrix.nix`, `swarm.matrix.gatewayHost != null` |

6
docs/swarm/README.md
View file

@ -472,9 +472,9 @@ services.hyperhive = {
host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- `swarm-wireguard.nix` reads each entry's `wireguardAddress` directly
from `services.hyperhive.swarm.peerHives` to build `wg-hive`'s
`allowedIPs`, so intra-swarm traffic can route over the mesh address
rather than the public domain.
from `services.hyperhive.swarm.hives`, minus this host's own hive, to
build `wg-hive`'s `allowedIPs`, so intra-swarm traffic can route over
the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
disable (not needed when both sides have public IPs and no NAT).

3
docs/swarm/ca.md
View file

@ -54,7 +54,8 @@ copies from. Every hive does this with its own identity, so holding the
swarm root's private key stopped being what decides whether a hive can
serve its swarm's names.
`services.hyperhive.swarm.serviceDomains` populates the role's
`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
same list on every host, populates the role's
`allowed_domains`, so asking for a name nobody configured is a refusal
from the store naming that name — not a certificate quietly issued for
it. A host's own SANs are narrower still: it asks only for the names it

8
docs/swarm/services.md
View file

@ -106,6 +106,14 @@ there is one IdP and one auth path.
Agent subjects come from swarm-controller's agent-creation job, written
into the users database by `swarm-authelia-bridge`; human ones come from
`swarmctl user add` → [setup.md § 2](../getting-started/setup.md#2--your-sso-account).
The bridge listens on loopback on the authelia host, and the gateway
publishes it as
`https://<swarm.authelia.domain>/bridge/`, which is where
`deploy.swarm-controller.authBridgeUrl` points by default on every host.
It answers only swarm-controller's OIDC client
(`swarm.controller.queueClientId`) and refuses every other token `403`.
On first boot this module seeds the users database with a disabled placeholder subject, so authelia has a non-empty store to start against before any real account exists (`swarm-authelia.nix:873`). Authelia generates session and storage keys in the container on
first boot and never rotates them automatically; replacing one
invalidates data already written (sessions, the encrypted store), so

7
docs/swarm/sso.md
View file

@ -249,9 +249,10 @@ HTML instead of the failure that actually happened:
- `/api/oidc/token`, `/.well-known/openid-configuration` — a client
parsing an error page as its JSON document
The gateway routes authelia's `/api/` and `/.well-known/` prefixes
**without** error interception. The split is by _audience_, not by an enumerated path
list: a human gets the page, every machine caller gets the status.
The gateway routes authelia's `/api/` and `/.well-known/` prefixes, and
the bridge's `/bridge/`, **without** error interception. The split is by
_audience_, not by an enumerated path list: a human gets the page, every
machine caller gets the status.
Enumerating endpoints individually would leave the next one added
silently intercepted.

11
hive-c0re/src/server.rs
View file

@ -451,17 +451,16 @@ async fn handle_set_resource_limits(
/// Guard: matrix provisioning needs a homeserver to provision against.
///
/// Answers "is one configured", not "is one running here" — the message names
/// both ways to get there, since a hive that talks to someone else's
/// homeserver never enables the local container at all.
/// Answers "is one configured", not "is one running here": the address is
/// the swarm's homeserver, wherever it runs.
fn require_matrix_present() -> Result<()> {
if crate::matrix::is_present() {
return Ok(());
}
anyhow::bail!(
"no matrix homeserver configured — set services.hyperhive.deploy.matrix.enable = true to run one \
here, or services.hyperhive.swarm.matrix.apiUrl to point at an existing one, before \
provisioning matrix users"
"no matrix homeserver configured — set services.hyperhive.swarm.matrix.apiUrl (it defaults \
to the swarm.matrix.gatewayHost vhost, and is null when that is) before provisioning \
matrix users"
)
}

6
nix/host-modules/hive-c0re/default.nix
View file

@ -127,9 +127,9 @@ in
./options.nix
./theme.nix
# Hive-CA trust for this daemon's outbound TLS. Nothing it is given by
# default is https — the forge and matrix URLs resolve to plain http or
# loopback, and the queue's TLS is verified against
# `HIVE_C0RE_OIDC_CA_FILE` (../hive-tls.nix), not this bundle — so this
# default is https — the forge and matrix URLs are plain http, and the
# queue's TLS is verified against `HIVE_C0RE_OIDC_CA_FILE`
# (../hive-tls.nix), not this bundle — so this
# changes nothing on an all-local hive. It matters for the split-host
# shape the options invite: `swarm.matrix.apiUrl`'s own example is
# `https://matrix.example.com`, and pointing it at another hive's

13
nix/host-modules/hive-c0re/environment.nix
View file

@ -180,15 +180,10 @@ in
// lib.optionalAttrs (config.services.hyperhive.swarm.matrix.apiUrl != null) {
# Client-server API base hive-c0re uses to provision matrix (register
# agent users, create the hive space + chat room, invite members).
# Supplied by `services.hyperhive.swarm.matrix.apiUrl`, which the matrix
# module fills in with its own loopback listener when it is the thing
# running tuwunel — and which the operator sets by hand when the
# homeserver lives on another machine.
#
# NOT the agent-facing HIVE_MATRIX_URL above: that one is the gateway
# vhost, and it is absent whenever there is no vhost. Reusing it here
# would silently stop provisioning on a hive that runs matrix without
# one.
# Supplied by `services.hyperhive.swarm.matrix.apiUrl`, which defaults to
# the same gateway vhost as HIVE_MATRIX_URL above but is its own option,
# so an operator can point provisioning elsewhere without changing what
# agents are handed.
HIVE_MATRIX_API_URL = config.services.hyperhive.swarm.matrix.apiUrl;
}
// {

18
nix/host-modules/hive-matrix-service.nix
View file

@ -10,7 +10,6 @@
let
cfg = config.services.hyperhive.swarm.matrix;
swarmDomain = config.services.hyperhive.swarm.domain;
deployCfg = config.services.hyperhive.deploy;
in
{
options.services.hyperhive.swarm.matrix = {
@ -65,11 +64,10 @@ in
apiUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = if deployCfg.matrix.enable then "http://127.0.0.1:${toString cfg.httpPort}" else null;
default = if cfg.gatewayHost == null then null else "http://${cfg.gatewayHost}";
defaultText = lib.literalExpression ''
if services.hyperhive.deploy.matrix.enable
then "http://127.0.0.1:''${toString services.hyperhive.swarm.matrix.httpPort}"
else null
if services.hyperhive.swarm.matrix.gatewayHost == null then null
else "http://''${services.hyperhive.swarm.matrix.gatewayHost}"
'';
example = "https://matrix.example.com";
description = ''
@ -79,12 +77,10 @@ in
`hyperhive.matrix.url`, which is the gateway vhost handed to
each agent's `hive-matrix-daemon`.
Defaults to the loopback listener **only when this module is the
thing running tuwunel** — in that case the address is not a
guess, it is where this module just put the container. Set it
explicitly (with `enable = false`) when the homeserver runs on
another machine; "everything on one host" is a special case of
the full deployment, not the assumption.
Defaults to the homeserver's gateway vhost (`gatewayHost`), the
same name on every hive whichever host runs tuwunel, and `null`
when there is no vhost. Set it explicitly when the homeserver is
reachable only some other way.
`null` means hive-c0re has no homeserver to provision against
and matrix provisioning no-ops. There is deliberately no

31
nix/host-modules/swarm-authelia-service.nix
View file

@ -10,7 +10,6 @@
let
cfg = config.services.hyperhive.swarm.authelia;
swarmDomain = config.services.hyperhive.swarm.domain;
deployCfg = config.services.hyperhive.deploy;
# The unit name upstream's `services.authelia.instances.<name>` derives,
# which `unit` publishes. ./swarm-authelia.nix binds the same two names for
@ -432,32 +431,10 @@ in
(`127.0.0.1:''${bridgePort}`) — one above authelia's own default
`port` (9091), outside hyperhive's other claimed ranges.
Reachable directly from this host's other processes (this
container shares the host netns, same as authelia's own `port`)
without going through the gateway — this is an internal
service-to-service endpoint, not something meant to be exposed
publicly.
'';
};
bridgeUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
readOnly = true;
default = if deployCfg.authelia.enable then "http://127.0.0.1:${toString cfg.bridgePort}" else null;
defaultText = lib.literalExpression ''if enable then "http://127.0.0.1:''${bridgePort}" else null'';
description = ''
Where `swarm-authelia-bridge` answers, **as seen from this
host** — correct only when a caller (`swarm-controller`) also
runs on this host, the same co-location assumption
`swarm.nix`'s `clientSecretFile` documents for its own
cross-host case. `null` when this host doesn't run
`swarm-authelia` at all.
A split-host swarm has no automated delivery for this address:
the operator points `swarm-controller`'s own option at wherever
this host has made the bridge reachable (a firewall rule, a
different bind address), the same manual-copy shape used
throughout this codebase's other cross-host cases.
Not published on its own: the authelia vhost proxies
`/bridge/` to it, and that is the address `swarm-controller`
uses from any host. The bridge answers only
`swarm-controller`'s OIDC client.
'';
};
};

15
nix/host-modules/swarm-authelia.nix
View file

@ -759,6 +759,13 @@ in
locations."/.well-known/" = {
proxyPass = "http://127.0.0.1:${toString cfg.port}";
};
# `swarm-authelia-bridge`, a separate process from authelia, also a
# machine surface. The trailing slash on `proxyPass` strips the
# prefix, so `/bridge/requests` reaches the bridge as `/requests`.
# The bridge itself refuses every client but swarm-controller's.
locations."/bridge/" = {
proxyPass = "http://127.0.0.1:${toString cfg.bridgePort}/";
};
};
# Order the container after the host CA generator, so the bind source
@ -924,10 +931,10 @@ in
# includes `bridgeClient`) — same file this container's own
# `renderClient` reads the digest half of.
SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE = "${clientsDir}/${bridgeClientId}.secret";
# The operator list names who holds operator rights, so the
# bridge answers it to the controller's client and refuses every
# other active token (each hive and its agents hold one).
SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID = hyperhiveCfg.swarm.controller.queueClientId;
# The one client the bridge answers. Every other active token
# (each hive and its agents hold one) is refused, on every
# request, since `/bridge/` is reachable from any host.
SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID = hyperhiveCfg.swarm.controller.queueClientId;
SWARM_AUTHELIA_BRIDGE_OPERATOR_GROUP = operatorGroup;
};
};

18
nix/host-modules/swarm-controller.nix
View file

@ -587,17 +587,14 @@ in
authBridgeUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = if deployCfg.authelia.enable then autheliaCfg.bridgeUrl else null;
defaultText = lib.literalExpression ''
authelia's own `bridgeUrl` when this host also runs
`swarm-authelia`, else null
'';
example = "http://127.0.0.1:9092";
default = "${autheliaCfg.url}/bridge";
defaultText = lib.literalExpression ''"''${services.hyperhive.swarm.authelia.url}/bridge"'';
example = "https://auth.example.com/bridge";
description = ''
Where `swarm-authelia-bridge` (the only writer of authelia's
users database) answers — see that option's own doc comment for
the cross-host caveat, since this default is only correct when
this host also runs `swarm-authelia`.
users database) answers. The default is the `/bridge/` location
on the authelia vhost, the same address from every host whether
or not it runs `swarm-authelia`; the daemon appends `/requests`.
No new credential to configure: the bearer token presented to
the bridge is minted from THIS daemon's own existing queue OIDC
@ -879,8 +876,7 @@ in
environment = {
SWARM_CONTROLLER_SOCKET = deployCfg.swarm-controller.socketPath;
# The swarm's hive directory, JSON-encoded — the full directory
# (this daemon has no "self" hive to exclude, unlike
# `swarm.peerHives`, `swarm.hives` minus this hive) rather than
# (this daemon has no "self" hive to exclude) rather than
# peers-minus-self. Consumed by `GET /api/hives`
# (swarm-controller/src/main.rs::load_hives).
# The one blacklist of names an agent may not take, shared verbatim

12
nix/host-modules/swarm-wireguard.nix
View file

@ -114,12 +114,14 @@
networking.wireguard.interfaces =
let
wgCfg = config.services.hyperhive.deploy.wireguard;
# `peerHives` is `swarm.hives` minus this hive (../swarm.nix) —
# a mesh that included our own entry would configure a tunnel to
# ourselves.
# `swarm.hives` minus this host's own hive: a mesh that included
# our own entry would configure a tunnel to ourselves.
meshPeers = lib.filterAttrs (
_: p: p.wireguardPublicKey != null && p.wireguardAddress != null
) config.services.hyperhive.swarm.peerHives;
name: p:
name != config.services.hyperhive.hiveName
&& p.wireguardPublicKey != null
&& p.wireguardAddress != null
) config.services.hyperhive.swarm.hives;
in
{
wg-hive = {

75
nix/host-modules/swarm.nix
View file

@ -1,6 +1,6 @@
# The swarm's directory: one entry per hive, **including this one**,
# identical on every host in the swarm. `services.hyperhive.hiveName`
# says which entry is us, and `peerHives` below derives the rest.
# says which entry is us.
#
# Why a directory rather than a per-host peer list: every field here is
# intrinsic to the hive it describes — none of them says anything about
@ -78,41 +78,31 @@ let
# assertion encoded an intended shape, not a property of the code, and
# it rejected the supported migration path: a hive pinning its old
# `forge.<hive domain>` while joining a swarm at a different apex.
#
# Every name is listed whether or not this host runs the service: the
# store's issuing role is built from this list on the store's host and
# has to admit a leaf for any gateway in the swarm. What one gateway's
# leaf carries is `localServiceDomains`.
#
# Membership is also what `gateway.lib.tlsFor` consults to pick the
# services leaf over the hive one, so a gateway-published swarm name
# missing here is served the hive certificate, which cannot cover a name
# under a different apex.
serviceDomains' = [
swarmCfg.forge.domain
swarmCfg.matrix.gatewayHost
swarmCfg.authelia.domain
]
# The swarm UI's name is a SIBLING of the other three, not a parent of
# them — the apex is as much a name needing a certificate as
# `forge.<apex>` is, and no CA in the hierarchy issues for it
# implicitly. Left out, its vhost falls back to the hive leaf and the
# swarm's front page opens with a name mismatch.
++ lib.optional (deployCfg.swarm-ui.enable && swarmCfg.domain != null) swarmCfg.domain
# Every swarm service that claims a gateway name belongs here, and
# these three were missing it. Membership is what `gateway.lib.tlsFor`
# consults to pick the services leaf over the hive one, so a name
# absent from this list is served the HIVE certificate — which cannot
# cover a name under a different apex however the sub-CA is set up.
#
# ⚠️ How that stayed invisible: a mismatch a browser shows as a
# click-through warning is one a machine client rejects outright. The
# metrics UI and store looked fine for as long as only people opened
# them; the collector's exporter — same defect, no human in the loop —
# failed every POST and dropped the samples.
++ lib.optional deployCfg.grafana.enable swarmCfg.grafana.domain
++ lib.optional deployCfg.victoriametrics.enable swarmCfg.victoriametrics.domain
++ lib.optional deployCfg.swarm-otel.enable swarmCfg.otel.domain
# VictoriaLogs' vhost is new (was previously unpublished entirely — see
# swarm-victorialogs.nix's file-top comment) and needs the same
# membership every gateway-published swarm service needs: absent from
# this list, `gateway.lib.tlsFor` falls back to the hive leaf, which
# cannot cover a name under a different apex — see the ⚠️ above this
# list for what that looked like the last time a name was missed here.
++ lib.optional deployCfg.victorialogs.enable swarmCfg.victorialogs.domain
# The swarm UI's apex is a sibling of the other names, not a parent:
# no CA in the hierarchy issues for it implicitly.
swarmCfg.domain
swarmCfg.grafana.domain
swarmCfg.victoriametrics.domain
swarmCfg.otel.domain
swarmCfg.victorialogs.domain
# The store's browser UI, not the store: `swarm.bao.domain` is mTLS and
# never gets a gateway certificate.
++ lib.optional deployCfg.bao.enable swarmCfg.bao.ui.domain;
swarmCfg.bao.ui.domain
];
# Hives whose entry still carries the removed `certFingerprint`. Scanned
# here, at top level, because that is the only place an assertion about a
@ -274,28 +264,17 @@ in
'';
};
options.services.hyperhive.swarm.peerHives = lib.mkOption {
type = lib.types.attrsOf (lib.types.attrsOf lib.types.unspecified);
readOnly = true;
internal = true;
description = ''
Read-only: `hives` minus this host's own entry. Derived once here
rather than in each consumer, because "everything that isn't me"
is a filter four different modules were re-implementing and only
one of them has to be wrong for a hive to peer with itself.
'';
};
options.services.hyperhive.swarm.serviceDomains = lib.mkOption {
type = lib.types.listOf lib.types.str;
readOnly = true;
internal = true;
description = ''
Read-only: the public hostnames of the swarm's own services, in a
stable sorted order. Second derived set alongside `peerHives`, and
here for the same reason — the CA that name-constrains these and
the leaf that carries them as SANs must agree exactly, and two
modules each assembling the list is how they stop agreeing.
Read-only: the public hostnames of every swarm service, in a
stable sorted order, identical on every host whichever services it
runs. Derived once here because the CA that name-constrains these,
the store role that issues for them and the leaf that carries them
as SANs must agree exactly, and two modules each assembling the
list is how they stop agreeing.
Sorted and deduplicated deliberately: consumers compare this list
against what they issued last time to decide whether to re-issue,
@ -326,8 +305,6 @@ in
};
config = {
services.hyperhive.swarm.peerHives = lib.filterAttrs (name: _: name != cfg.hiveName) swarmCfg.hives;
services.hyperhive.swarm.serviceDomains = lib.sort (a: b: a < b) (
lib.unique (lib.filter (d: d != null && d != "") serviceDomains')
);

16
nix/module-eval/core-toggle.nix
View file

@ -197,6 +197,22 @@ let
&& !(lib.hasInfix "auth.t.local" u)
&& lib.elem "auth.t.local" s;
}
{
# The store's issuing role is built from this list on the store's host,
# so it cannot depend on which services the evaluating host runs. The
# last conjunct is the control: grafana runs on neither `bare` nor the
# forge host, and is still listed.
name = "the swarm's service names are the same on hosts running different services";
ok =
let
s = h: h.services.hyperhive.swarm.serviceDomains;
in
s (hive {
deploy.forgejo.enable = true;
}) == s bare
&& s allLocal == s bare
&& lib.elem "grafana.t.local" (s bare);
}
{
# The narrowing's floor: no names left means no request, rather than a
# request for an empty SAN set that the store would refuse. Its own

24
nix/module-eval/nats-authelia.nix
View file

@ -264,19 +264,33 @@ let
&& !(lib.hasInfix ''echo "users: {}"'' s);
}
{
# The bridge refuses its operator list to every client but the one
# named here, so this has to be the client the controller authenticates
# as. A drift is a controller whose admin sync is refused every pass.
name = "the authelia bridge answers its operator list to the controller's client only";
# The bridge refuses every client but the one named here, so this has
# to be the client the controller authenticates as. A drift is a
# controller whose every bridge call is refused.
name = "the authelia bridge answers the controller's client only";
ok =
let
env =
autheliaColocated.containers.swarm-authelia.config.systemd.services.swarm-authelia-bridge.environment;
in
env.SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID
env.SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID
== autheliaColocated.services.hyperhive.swarm.controller.queueClientId
&& env.SWARM_AUTHELIA_BRIDGE_OPERATOR_GROUP == "admins";
}
{
# One address from every host, and a location the IdP host serves: a
# controller off the authelia host has no other way to the bridge.
name = "the auth bridge address is the authelia vhost's /bridge/ on every host";
ok =
let
url = h: h.services.hyperhive.deploy.swarm-controller.authBridgeUrl;
authelia = autheliaColocated.services.hyperhive.swarm.authelia;
location = autheliaColocated.services.nginx.virtualHosts.${authelia.domain}.locations."/bridge/";
in
url autheliaNotColocated == "${autheliaNotColocated.services.hyperhive.swarm.authelia.url}/bridge"
&& url autheliaColocated == url autheliaNotColocated
&& location.proxyPass == "http://127.0.0.1:${toString authelia.bridgePort}/";
}
{
# Config NAMES the IdP; it never computes where the IdP is. Both arms
# matter together: the address is the swarm's name on a hive that runs

17
swarm-authelia-bridge/README.md
View file

@ -25,18 +25,17 @@ simply owns the file it writes. No elevated privilege anywhere.
`ListOperators` (the operator group's usernames).
- Bearer-authenticated via authelia's own OIDC token introspection (RFC
7662), against `swarm-controller`'s **existing** machine-client identity
(already minted for the queue connection) — no new credential. The two
agent verbs admit any active token. ⚠️ `ListOperators` admits only the
client named by `SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID`
(`swarm-controller`'s) and answers every other active token `403`:
authelia issues machine tokens to every hive and its agents too.
(already minted for the queue connection) — no new credential. ⚠️ Every
request admits only the client named by
`SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID` (`swarm-controller`'s) and
answers every other active token `403`: authelia issues machine tokens to
every hive and its agents too.
- No restart of authelia after a write — relies on
`authentication_backend.file.watch`, confirmed working against the
pinned 4.39.20 build during this design work.
- Network-facing rather than unix-socket-only: `swarm-authelia`'s container
shares the host netns, so the same listener serves both a co-located
`swarm-controller` (loopback) and a split-host one (bind wider + firewall)
with no separate transport.
- Listens on loopback. The authelia vhost proxies `/bridge/` to it with
the prefix stripped, so `swarm-controller` on any host calls
`https://<swarm.authelia.domain>/bridge/requests`.
## One store, two writers

207
swarm-authelia-bridge/src/main.rs
View file

@ -11,13 +11,10 @@
//! with root/`CAP_CHOWN`/a shared group (all examined and rejected — see
//! `swarmctl/README.md`'s own identical analysis of this same file).
//!
//! Network-facing, not unix-socket-only: `swarm-authelia`'s container
//! shares the host netns (`privateNetwork = false`, same as `hive-forge`/
//! `hive-matrix`), so one listener serves a co-located `swarm-controller`
//! (loopback) and a split-host one (bind wider, firewall it) with the
//! same code path — no separate transport for the cross-host case.
//! Listens on loopback; the authelia vhost proxies `/bridge/` to it, so a
//! `swarm-controller` on any host reaches it at the same address.
//! Bearer-authenticated via authelia's own OIDC token introspection (RFC
//! 7662) against `swarm-controller`'s **existing** machine-client
//! 7662), and admits only `swarm-controller`'s **existing** machine-client
//! identity (already minted for the queue connection) — no new
//! credential to mint or deliver, and always local/fast to verify since
//! this process runs right next to the authelia it asks.
@ -25,7 +22,8 @@
//! One endpoint, `POST /requests`, body =
//! [`swarm_authelia_bridge_sock::BridgeRequest`] verbatim: ensure an agent
//! exists as an authelia subject, list the ones that do, and list the
//! operator group (to `swarm-controller` only, see `permit`). Not a
//! operator group. Every request is answered to `swarm-controller` only,
//! see `classify`. Not a
//! wholesale-replace-the-file API — this process owns rendering
//! `users.yml` internally; see `store` module.
@ -53,8 +51,8 @@ struct Config {
introspection_url: String,
client_id: String,
client_secret: String,
/// The one client [`BridgeRequest::ListOperators`] answers.
operator_reader_client_id: String,
/// The one client this bridge answers: `swarm-controller`'s.
controller_client_id: String,
/// The authelia group whose members are operators.
operator_group: String,
}
@ -68,7 +66,7 @@ impl Config {
introspection_url: env_var("SWARM_AUTHELIA_BRIDGE_INTROSPECTION_URL")?,
client_id: env_var("SWARM_AUTHELIA_BRIDGE_CLIENT_ID")?,
client_secret: read_secret_file(&env_var("SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE")?)?,
operator_reader_client_id: env_var("SWARM_AUTHELIA_BRIDGE_OPERATOR_READER_CLIENT_ID")?,
controller_client_id: env_var("SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID")?,
operator_group: env_var("SWARM_AUTHELIA_BRIDGE_OPERATOR_GROUP")?,
})
}
@ -146,28 +144,20 @@ async fn handle_request(
headers: HeaderMap,
Json(req): Json<BridgeRequest>,
) -> Response {
let caller = match authorize(&state, &headers).await {
Ok(caller) => caller,
Err(resp) => return resp,
};
respond(&state.config, &state.write_lock, caller.as_deref(), req).await
if let Err(resp) = authorize(&state, &headers).await {
return resp;
}
respond(&state.config, &state.write_lock, req).await
}
/// Everything after authentication: the per-request [`permit`] check, then
/// the operation. Takes the config and the write lock rather than
/// [`AppState`], whose HTTP client only [`authorize`] uses.
/// Everything after authentication: the operation itself. Takes the config
/// and the write lock rather than [`AppState`], whose HTTP client only
/// [`authorize`] uses.
async fn respond(
config: &Config,
write_lock: &tokio::sync::Mutex<()>,
caller: Option<&str>,
req: BridgeRequest,
) -> Response {
if let Err(refusal) = permit(&req, caller, &config.operator_reader_client_id) {
// The client id is not a credential; naming it is what lets an
// operator see which client asked for something it may not have.
tracing::warn!(client_id = ?caller, request = ?req, "refused a request scoped to another client");
return refuse(refusal);
}
let (op, result) = match req {
BridgeRequest::EnsureAgentIdentity { name } => {
("ensure_identity", handle(config, write_lock, name).await)
@ -194,10 +184,10 @@ async fn respond(
/// "the op itself failed," a different case from "the caller was never let
/// in."
///
/// Anything short of an active token is refused; **which status** depends
/// on whose credential failed, see [`Refusal`]. `Ok` carries the client the
/// token was issued to, for [`permit`].
async fn authorize(state: &AppState, headers: &HeaderMap) -> Result<Option<String>, Response> {
/// Anything short of an active token from the controller's client is
/// refused; **which status** depends on whose credential failed, see
/// [`Refusal`].
async fn authorize(state: &AppState, headers: &HeaderMap) -> Result<(), Response> {
let Some(auth) = headers.get(axum::http::header::AUTHORIZATION) else {
return Err(refuse(Refusal::Caller));
};
@ -215,41 +205,37 @@ async fn authorize(state: &AppState, headers: &HeaderMap) -> Result<Option<Strin
token,
)
.await;
classify(verdict).map_err(refuse)
classify(verdict, &state.config.controller_client_id).map_err(refuse)
}
/// The admission decision, and the only place that makes it.
///
/// Fail-closed: `Ok(())` is reachable from exactly one variant. Split out
/// of [`authorize`] so the rule is testable without an authelia — and it is
/// the *real* rule, not a restatement of it, because this is the function
/// the request path calls.
fn classify(verdict: introspect::Verdict) -> Result<Option<String>, Refusal> {
/// Fail-closed: `Ok(())` is reachable only from an active token issued to
/// `controller`. authelia also issues machine tokens to every hive and every
/// hive's agents, and those are as active as the controller's, so they are
/// refused `403`; a token attributed to no client is refused too, rather
/// than compared as a blank. Split out of [`authorize`] so the rule is
/// testable without an authelia — and it is the *real* rule, not a
/// restatement of it, because this is the function the request path calls.
fn classify(verdict: introspect::Verdict, controller: &str) -> Result<(), Refusal> {
match verdict {
introspect::Verdict::Active { client_id } => Ok(client_id),
introspect::Verdict::Active { client_id } if client_id.as_deref() == Some(controller) => {
Ok(())
}
introspect::Verdict::Active { client_id } => {
// The client id is not a credential; naming it is what lets an
// operator see which client was refused.
tracing::warn!(
?client_id,
"refused an active token issued to another client"
);
Err(Refusal::Forbidden)
}
introspect::Verdict::Inactive => Err(Refusal::Caller),
introspect::Verdict::Unavailable => Err(Refusal::Bridge),
}
}
/// Whether an admitted caller may make this particular request.
///
/// [`BridgeRequest::ListOperators`] answers only `operator_reader`, the
/// controller's client: authelia also issues machine tokens to every hive and
/// every hive's agents, and those are as active as the controller's. A caller
/// with no client id is refused it too, rather than compared as a blank.
fn permit(
req: &BridgeRequest,
client_id: Option<&str>,
operator_reader: &str,
) -> Result<(), Refusal> {
match req {
BridgeRequest::EnsureAgentIdentity { .. } | BridgeRequest::ListAgentIdentities => Ok(()),
BridgeRequest::ListOperators if client_id == Some(operator_reader) => Ok(()),
BridgeRequest::ListOperators => Err(Refusal::Forbidden),
}
}
/// Whose credential was at fault when a request was refused.
///
/// The distinction is the whole point: these used to be one 401, so a
@ -262,13 +248,14 @@ enum Refusal {
/// This bridge could not get an answer out of authelia at all, so the
/// request was never evaluated.
Bridge,
/// The token is valid, but its client may not make this request.
/// The token is valid, but was issued to a client other than the
/// controller's.
Forbidden,
}
impl Refusal {
/// `401` for the caller; `503` for us; `403` for a valid caller asking
/// for something scoped to another client.
/// `401` for the caller; `503` for us; `403` for a valid token issued
/// to a client other than the controller's.
///
/// 401 would be wrong for [`Refusal::Bridge`] under RFC 7235 — nothing
/// about the caller's credential was determined — and it is also the
@ -456,34 +443,58 @@ async fn generate_password(bin: &std::path::Path) -> Result<String> {
#[cfg(test)]
mod tests {
use super::{
BridgeRequest, Config, Refusal, StatusCode, classify, introspect::Verdict, permit, respond,
BridgeRequest, Config, Refusal, StatusCode, classify, introspect::Verdict, respond,
};
const CONTROLLER: &str = "swarm-controller";
fn active(client_id: Option<&str>) -> Verdict {
Verdict::Active {
client_id: client_id.map(str::to_owned),
}
}
/// Fail-closed, asserted per variant rather than as one negation:
/// splitting blame out of a single denial is exactly the change that
/// could have widened admission, so each refusing variant is named on
/// its own line.
#[test]
fn only_an_active_token_is_admitted() {
assert_eq!(classify(active(Some(CONTROLLER)), CONTROLLER), Ok(()));
assert_eq!(
classify(Verdict::Active {
client_id: Some("swarm-controller".to_owned())
}),
Ok(Some("swarm-controller".to_owned()))
);
assert_eq!(
classify(Verdict::Inactive),
classify(Verdict::Inactive, CONTROLLER),
Err(Refusal::Caller),
"authelia evaluated the token and said no"
);
assert_eq!(
classify(Verdict::Unavailable),
classify(Verdict::Unavailable, CONTROLLER),
Err(Refusal::Bridge),
"a bridge that cannot ask must not admit — this variant reports a \
different fault, it is not a softer denial"
);
}
/// The trust boundary. Every caller below holds a token authelia calls
/// active — a hive and a hive's agents are issued machine tokens too —
/// and only the controller's client is admitted. The first assertion is
/// the control: the same verdict shape with the controller's id passes.
#[test]
fn only_the_controllers_client_is_admitted() {
assert_eq!(classify(active(Some(CONTROLLER)), CONTROLLER), Ok(()));
for other in ["hive-alpha", "hive-alpha-agent", "swarm-authelia-bridge"] {
assert_eq!(
classify(active(Some(other)), CONTROLLER),
Err(Refusal::Forbidden),
"{other} holds an active token but is not the controller"
);
}
assert_eq!(
classify(active(None), CONTROLLER),
Err(Refusal::Forbidden),
"a token authelia attributed to no client is not the controller's"
);
}
/// The whole point of the split, asserted on the value rather than
/// on message text — the previous code picked its status by matching on
/// a log message's wording, which is what let the two collapse.
@ -521,51 +532,8 @@ mod tests {
}
}
const CONTROLLER: &str = "swarm-controller";
/// The trust boundary on the operator list. Every caller below holds a
/// token authelia calls active — a hive and a hive's agents are issued
/// machine tokens too — and only the controller's client is answered.
#[test]
fn only_the_controller_may_list_operators() {
let req = BridgeRequest::ListOperators;
assert_eq!(permit(&req, Some(CONTROLLER), CONTROLLER), Ok(()));
for other in ["hive-alpha", "hive-alpha-agent", "swarm-authelia-bridge"] {
assert_eq!(
permit(&req, Some(other), CONTROLLER),
Err(Refusal::Forbidden),
"{other} holds an active token but is not the controller"
);
}
assert_eq!(
permit(&req, None, CONTROLLER),
Err(Refusal::Forbidden),
"a token authelia attributed to no client is not the controller's"
);
}
/// The control for the test above: the agent verbs stay open to any
/// admitted caller, so the refusal there comes from the scoping and not
/// from `permit` refusing everyone who is not the controller.
#[test]
fn the_agent_verbs_stay_open_to_any_active_caller() {
for req in [
BridgeRequest::EnsureAgentIdentity {
name: "atlas".to_owned(),
},
BridgeRequest::ListAgentIdentities,
] {
assert_eq!(
permit(&req, Some("hive-alpha"), CONTROLLER),
Ok(()),
"{req:?}"
);
assert_eq!(permit(&req, None, CONTROLLER), Ok(()), "{req:?}");
}
}
#[test]
fn a_scoped_refusal_is_403_not_401() {
fn a_refused_client_is_403_not_401() {
assert_eq!(
Refusal::Forbidden.status(),
StatusCode::FORBIDDEN,
@ -598,29 +566,16 @@ mod tests {
introspection_url: String::new(),
client_id: String::new(),
client_secret: String::new(),
operator_reader_client_id: CONTROLLER.to_owned(),
controller_client_id: CONTROLLER.to_owned(),
operator_group: "admins".to_owned(),
}
}
/// The request path, not only [`permit`]: a dispatch that skipped the
/// check would answer the hive below with the operator list.
#[tokio::test]
async fn the_request_path_answers_the_operator_list_to_the_controller_only() {
async fn the_operator_list_is_the_operator_groups_members() {
let config = config_with_one_operator();
let lock = tokio::sync::Mutex::new(());
for caller in [Some("hive-alpha"), None] {
let refused = respond(&config, &lock, caller, BridgeRequest::ListOperators).await;
assert_eq!(refused.status(), StatusCode::FORBIDDEN, "{caller:?}");
}
let answered = respond(
&config,
&lock,
Some(CONTROLLER),
BridgeRequest::ListOperators,
)
.await;
let answered = respond(&config, &lock, BridgeRequest::ListOperators).await;
assert_eq!(answered.status(), StatusCode::OK);
let body = axum::body::to_bytes(answered.into_body(), usize::MAX)
.await

6
swarm-controller/src/auth.rs
View file

@ -11,9 +11,9 @@
//! connection attempt: no window in which this process holds a token
//! that outlives its intended use.
//!
//! `None` when this deployment did not wire a bridge up — the bridge only
//! exists on hosts that also run `swarm-authelia`, so a controller split
//! from it can neither create an identity nor read the roster.
//! `None` when this deployment set no bridge address; the default is the
//! `/bridge/` location on the authelia vhost, from every host. Without one,
//! this daemon can neither create an identity nor read the roster.
//!
//! Each caller answers that absence in its own terms rather than this
//! module inventing a shared one: `SwarmNodeKind::CreateIdentity` fails

3
swarm-controller/src/main.rs
View file

@ -741,8 +741,7 @@ struct AppState {
/// Env var the controller's NixOS module sets from
/// `services.hyperhive.swarm.hives`, JSON-encoded — the full directory
/// (this daemon has no "self" to exclude) rather than peers-minus-self
/// (`services.hyperhive.swarm.peerHives`, which other consumers use).
/// (this daemon has no "self" to exclude) rather than peers-minus-self.
const HIVES_ENV: &str = "SWARM_CONTROLLER_HIVES";
/// Parses [`HIVES_ENV`] into the swarm's hive directory. Unset or