feat(swarm-controller): aggregate per-hive status from the swarm queue
The controller connects to the swarm queue as its own client and serves what each hive last said about itself at GET /api/hives/status. THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream KV bucket (history 1) and the controller reads it per request, keeping no copy. A cache here would be a second answer to the same question, free to disagree with the first, and the disagreement surfaces as a hive reading healthy on a dashboard while the bucket says otherwise. Whichever side arrives first creates the bucket; both want the same shape. Rows come from the roster rather than from the bucket, so an empty bucket renders as a swarm nobody has heard from instead of a healthy one, and `never_reported` stays distinct from `stale` - went quiet is a fault, never spoke is usually a deployment that has not happened. Freshness is derived at read time and never stored as a flag, because a stored `healthy` boolean goes stale silently the moment nothing arrives, which is the failure this endpoint is designed against. The timestamp is the NATS server's, applied when the value landed, so a publisher cannot make itself look fresher than it is. Authentication is per connection attempt, not per process. Authelia issues `client_credentials` tokens that expire in 3599s, and auth happens at CONNECT, so a long-lived connection is fine but a reconnect an hour later needs a token minted an hour later. `with_auth_callback` is re-run by async-nats for each attempt, which handles expiry by construction rather than by a timer - the alternative fails in the way this subsystem exists to prevent, with the controller still serving while its data quietly stops updating. Three failure shapes are deliberate: - A half-set environment is fatal; an absent one is not. Silently behaving like an unconfigured host is how every hive ends up reading `never_reported` with nothing to point at. - The endpoint answers 503 rather than an empty list when the store cannot be read. "I cannot reach the store" and "every hive is silent" are different answers, and rendering the second turns a local fault into an apparent swarm-wide outage. - `retry_on_initial_connect` makes the daemon and the queue bootable in either order, and the status handler refuses when the client is not Connected rather than issuing a request into it - a request made in that window does not fail, it waits, so every poll would hang and learn nothing. `Pending` is the state a never-connected client is in, which is why the test is `!= Connected` and not `== Disconnected`. The rendering rules are a pure function over a map, so the semantics are tested against a table rather than against a running server. The KV read, the credential rotation and the 503 paths are covered behaviourally instead: a real NATS server with a rotating token endpoint, asserting that the controller recovers only when the credential rotates, and mutation-tested by holding the credential wrong for the same window.
This commit is contained in:
atlas
parent
b3f46e4f43
commit
8891b46943
7 changed files with 1030 additions and 17 deletions
|
|
@ -33,6 +33,39 @@ let
|
|||
SWARMCTL_AUTHELIA_UNIT = autheliaCfg.unit;
|
||||
};
|
||||
|
||||
natsCfg = config.services.hyperhive.swarm.nats;
|
||||
|
||||
# The controller's own OAuth2 client. It is NOT a hive: the per-hive
|
||||
# clients the roster issues belong to hives, and the responder's client
|
||||
# belongs to the responder. One identity per principal — the rule is that
|
||||
# a principal's credentials all derive from the same identity, not that
|
||||
# the swarm has one.
|
||||
queueClientId = "swarm-controller";
|
||||
|
||||
# Both halves have to be here: authelia to have minted the secret, and the
|
||||
# queue to connect to. Same guard, and the same reasoning, as `autheliaEnv`
|
||||
# above — a value set on a host that runs neither would point at a file
|
||||
# that does not exist and produce a daemon that retries forever.
|
||||
queueLocal = autheliaCfg.enable && natsCfg.enable;
|
||||
|
||||
# `LoadCredential` and not a copy-oneshot, which is where this deliberately
|
||||
# differs from the callout responder: that one delivers INTO a container,
|
||||
# so it has to copy across a filesystem boundary. The controller is a plain
|
||||
# host unit, so systemd can hand it the file directly — fewer moving parts,
|
||||
# and the secret never gains a second on-disk copy to forget about.
|
||||
queueEnv = lib.optionalAttrs queueLocal {
|
||||
# The queue container shares the host netns, so loopback is correct here
|
||||
# and is not the `localhost`-means-the-wrong-thing trap that applies
|
||||
# inside agent containers.
|
||||
SWARM_CONTROLLER_NATS_URL = "nats://127.0.0.1:${toString natsCfg.port}";
|
||||
SWARM_CONTROLLER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
|
||||
SWARM_CONTROLLER_OIDC_CLIENT_ID = queueClientId;
|
||||
# `%d` is systemd's credentials directory: root reads the plaintext at
|
||||
# unit start and the daemon's own user sees it 0400, without the unit
|
||||
# ever being able to read the rest of authelia's state dir.
|
||||
SWARM_CONTROLLER_OIDC_CLIENT_SECRET_FILE = "%d/queue-client.secret";
|
||||
};
|
||||
|
||||
# Wrapped rather than documented: every one of these values is derived
|
||||
# from an option this deployment already set, so making the operator
|
||||
# re-supply them on the command line would be asking them to repeat the
|
||||
|
|
@ -113,6 +146,25 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
staleAfterSeconds = lib.mkOption {
|
||||
type = lib.types.ints.positive;
|
||||
default = 120;
|
||||
description = ''
|
||||
How old a hive's last status snapshot may be before
|
||||
`GET /api/hives/status` reports it as `stale` rather than `fresh`.
|
||||
|
||||
This is a statement about how often hives *publish*, not about how
|
||||
patient a reader is — set it above the publishing cadence or every
|
||||
hive reads stale between offers. It is an option and not a
|
||||
constant precisely because that cadence is a property of the
|
||||
deployment.
|
||||
|
||||
Freshness is derived when the endpoint is read, never stored, so
|
||||
changing this takes effect for the next request; no hive has to
|
||||
re-publish anything.
|
||||
'';
|
||||
};
|
||||
|
||||
links = lib.mkOption {
|
||||
type = lib.types.listOf (
|
||||
lib.types.submodule {
|
||||
|
|
@ -174,6 +226,23 @@ in
|
|||
# exactly what it must not inherit.
|
||||
environment.systemPackages = [ swarmctlConfigured ];
|
||||
|
||||
# One declaration, two readers — the controller knows which client id it
|
||||
# authenticates under, so making the operator restate it in authelia's
|
||||
# client list would be a second source of truth for a string whose
|
||||
# mismatch is an opaque 401 from the token endpoint. Same shape as the
|
||||
# queue's own client declaration.
|
||||
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf autheliaCfg.enable [
|
||||
{
|
||||
id = queueClientId;
|
||||
description = "HyperHive swarm controller";
|
||||
# `client_credentials`: a daemon authenticating as itself, with
|
||||
# nobody to redirect. Declared rather than inferred from an empty
|
||||
# redirect list, because authelia permits only the grants a client
|
||||
# names and an omitted `grant_types` means authorization-code alone.
|
||||
kind = "machine";
|
||||
}
|
||||
];
|
||||
|
||||
systemd.services.swarm-controller = {
|
||||
description = "hyperhive swarm-level controller daemon";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
|
|
@ -181,6 +250,14 @@ in
|
|||
|
||||
serviceConfig = {
|
||||
ExecStart = "${cfg.package}/bin/swarm-controller";
|
||||
|
||||
# Only when the queue is actually reachable from here. An absent
|
||||
# credential is not a failure: the daemon logs that no queue is
|
||||
# configured and serves its HTTP surface, which is the correct
|
||||
# behaviour on the hosts that do not run one.
|
||||
LoadCredential = lib.mkIf queueLocal [
|
||||
"queue-client.secret:${autheliaCfg.hostClientSecretDir}/${queueClientId}.secret"
|
||||
];
|
||||
User = "swarm-controller";
|
||||
Group = "swarm-controller";
|
||||
Restart = "on-failure";
|
||||
|
|
@ -218,23 +295,36 @@ in
|
|||
];
|
||||
};
|
||||
|
||||
environment.SWARM_CONTROLLER_SOCKET = cfg.socketPath;
|
||||
# The swarm's hive directory, JSON-encoded — same shape hive-c0re
|
||||
# already builds for HYPERHIVE_PEERS (../hive-c0re/environment.nix),
|
||||
# just the full directory (this daemon has no "self" hive to
|
||||
# exclude, unlike a per-hive c0re's peer list) rather than
|
||||
# peers-minus-self. Consumed by `GET /api/hives`
|
||||
# (swarm-controller/src/main.rs::load_hives).
|
||||
environment.SWARM_CONTROLLER_HIVES = builtins.toJSON (
|
||||
lib.mapAttrsToList (name: h: {
|
||||
inherit name;
|
||||
inherit (h) domain;
|
||||
}) config.services.hyperhive.swarm.hives
|
||||
);
|
||||
# The merged links list — see `links`' description above for who
|
||||
# contributes to it. Consumed by `GET /api/links`
|
||||
# (swarm-controller/src/main.rs::load_links).
|
||||
environment.SWARM_CONTROLLER_LINKS = builtins.toJSON cfg.links;
|
||||
# Queue coordinates (`queueEnv`) merge in last and are present only
|
||||
# where the queue and its IdP both run. The daemon refuses a PARTIAL
|
||||
# set rather than treating it as absent, which is why they are built
|
||||
# as one attrset and never assigned individually.
|
||||
environment = {
|
||||
SWARM_CONTROLLER_SOCKET = cfg.socketPath;
|
||||
# The swarm's hive directory, JSON-encoded — same shape hive-c0re
|
||||
# already builds for HYPERHIVE_PEERS (../hive-c0re/environment.nix),
|
||||
# just the full directory (this daemon has no "self" hive to
|
||||
# exclude, unlike a per-hive c0re's peer list) rather than
|
||||
# peers-minus-self. Consumed by `GET /api/hives`
|
||||
# (swarm-controller/src/main.rs::load_hives).
|
||||
SWARM_CONTROLLER_HIVES = builtins.toJSON (
|
||||
lib.mapAttrsToList (name: h: {
|
||||
inherit name;
|
||||
inherit (h) domain;
|
||||
}) config.services.hyperhive.swarm.hives
|
||||
);
|
||||
# The merged links list — see `links`' description above for who
|
||||
# contributes to it. Consumed by `GET /api/links`
|
||||
# (swarm-controller/src/main.rs::load_links).
|
||||
SWARM_CONTROLLER_LINKS = builtins.toJSON cfg.links;
|
||||
# Staleness threshold for `GET /api/hives/status` — see
|
||||
# `staleAfterSeconds`' description. Set unconditionally rather
|
||||
# than inside `queueEnv`: it is not a queue coordinate, and
|
||||
# nothing about it is unsafe to define on a host whose queue is
|
||||
# off (the daemon just has nothing to apply it to).
|
||||
SWARM_CONTROLLER_STALE_AFTER_SECS = toString cfg.staleAfterSeconds;
|
||||
}
|
||||
// queueEnv;
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue