swarm: default every queue URL to the queue's name on every hive
A remote hive dialled nothing until an operator copied the queue's URL into it, though the URL is the same string everywhere. statusPublish.natsUrl, queue.agentNatsUrl and controller.queue.natsUrl now default to tls://<swarm.nats.domain>:<port> unconditionally. The statusPublish assertion treated a URL without a secret as a half config. With the URL a default on every hive, only the secret claims publishing: the assertion now refuses a secret without a URL or token endpoint, and hive-c0re's status environment is gated on the secret too, so a hive without one publishes nothing instead of reading a missing credential.
This commit is contained in:
parent
0081d75c86
commit
a5259146dc
8 changed files with 220 additions and 129 deletions
|
|
@ -338,10 +338,8 @@ nothing has to re-publish.
|
||||||
|
|
||||||
### Making a hive report
|
### Making a hive report
|
||||||
|
|
||||||
Three options, on the **hive**, set together or not at all — a
|
Three options on the **hive**. They sit in two namespaces, because two of them
|
||||||
half-configured hive is an eval error rather than one that quietly never
|
are facts about _this machine_ and one is the swarm's single address:
|
||||||
reports. They sit in two namespaces, because two of them are facts about
|
|
||||||
_this machine_ and one is the swarm's single address:
|
|
||||||
|
|
||||||
| option | what to set it to |
|
| option | what to set it to |
|
||||||
| ------------------------------------------------------- | --------------------------------------------- |
|
| ------------------------------------------------------- | --------------------------------------------- |
|
||||||
|
|
@ -349,10 +347,14 @@ _this machine_ and one is the swarm's single address:
|
||||||
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
|
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` |
|
||||||
| `deploy.hive-controller.statusPublish.clientSecretFile` | path to this hive's client secret, plaintext |
|
| `deploy.hive-controller.statusPublish.clientSecretFile` | path to this hive's client secret, plaintext |
|
||||||
|
|
||||||
On a host that runs the queue and the IdP itself, all three default to
|
The queue URL and the token endpoint default to the swarm's own addresses on
|
||||||
the local ones and there is nothing to set. Any other hive needs them
|
every hive, so there is nothing to set for them. The secret is what turns
|
||||||
spelled out, and needs the secret to physically be there: the swarm does
|
publishing on: a hive without it doesn't publish. A secret without the other
|
||||||
not distribute it. Copy `hive-<hiveName>.secret` out of the swarm host's
|
two is an eval error rather than a hive that quietly never reports.
|
||||||
|
|
||||||
|
On a host that runs the queue and the IdP, the secret defaults to the local one.
|
||||||
|
Any other hive needs the secret to physically be there, because the swarm
|
||||||
|
doesn't distribute it. Copy `hive-<hiveName>.secret` out of the swarm host's
|
||||||
`deploy.authelia.hostClientSecretDir` with whatever secret management the
|
`deploy.authelia.hostClientSecretDir` with whatever secret management the
|
||||||
deployment already uses.
|
deployment already uses.
|
||||||
|
|
||||||
|
|
@ -390,11 +392,12 @@ with the credential itself and aren't configurable.
|
||||||
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
|
| `deploy.hive-controller.queue.agentNatsUrl` | where the queue listens, as an agent **container** reaches it |
|
||||||
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` — the same one the hive uses |
|
| `swarm.statusPublish.tokenEndpoint` | the swarm IdP's `/api/oidc/token` — the same one the hive uses |
|
||||||
|
|
||||||
On a host that runs the queue, `agentNatsUrl` defaults to the same
|
On every hive, `agentNatsUrl` defaults to the same
|
||||||
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. Inside a
|
`tls://<swarm.nats.domain>:<swarm.nats.port>` as the hive's own. On the queue's
|
||||||
container the name resolves to the bridge address, where the firewall opens the
|
host the name resolves inside a container to the bridge address, where the
|
||||||
port. ⚠️ **Never a loopback address here**: an agent has its own network
|
firewall opens the port; on any other hive it resolves through the host's DNS,
|
||||||
namespace, so `127.0.0.1` reaches the agent.
|
like the store's name. ⚠️ **Never a loopback address here**: an agent has its
|
||||||
|
own network namespace, so `127.0.0.1` reaches the agent.
|
||||||
|
|
||||||
The harness sees four variables, and treats them as all-or-none:
|
The harness sees four variables, and treats them as all-or-none:
|
||||||
`HIVE_AGENT_NATS_URL` and `HIVE_AGENT_OIDC_TOKEN_ENDPOINT` from the two options
|
`HIVE_AGENT_NATS_URL` and `HIVE_AGENT_OIDC_TOKEN_ENDPOINT` from the two options
|
||||||
|
|
|
||||||
|
|
@ -133,12 +133,12 @@ in
|
||||||
./options.nix
|
./options.nix
|
||||||
./theme.nix
|
./theme.nix
|
||||||
# Hive-CA trust for this daemon's outbound TLS. Nothing it is given by
|
# Hive-CA trust for this daemon's outbound TLS. Nothing it is given by
|
||||||
# default is https — the forge, matrix and queue URLs all resolve to
|
# default is https — the forge and matrix URLs resolve to plain http or
|
||||||
# plain http or loopback — so this changes nothing on an all-local
|
# loopback, and the queue's TLS is verified against
|
||||||
# hive. It matters for the split-host shape the options invite:
|
# `HIVE_C0RE_OIDC_CA_FILE` (../hive-tls.nix), not this bundle — so this
|
||||||
# `swarm.matrix.apiUrl`'s own example is `https://matrix.example.com`,
|
# changes nothing on an all-local hive. It matters for the split-host
|
||||||
# and pointing it (or `deploy.hive-controller.statusPublish.natsUrl`)
|
# shape the options invite: `swarm.matrix.apiUrl`'s own example is
|
||||||
# at another hive's
|
# `https://matrix.example.com`, and pointing it at another hive's
|
||||||
# gateway means verifying a leaf signed by a CA generated at runtime,
|
# gateway means verifying a leaf signed by a CA generated at runtime,
|
||||||
# which no build-time trust store can contain.
|
# which no build-time trust store can contain.
|
||||||
#
|
#
|
||||||
|
|
|
||||||
|
|
@ -257,24 +257,30 @@ in
|
||||||
# environment is a deployment bug the daemon refuses to treat as
|
# environment is a deployment bug the daemon refuses to treat as
|
||||||
# "no queue configured", because the failure it would otherwise
|
# "no queue configured", because the failure it would otherwise
|
||||||
# produce is a hive that comes up fine and silently never reports.
|
# produce is a hive that comes up fine and silently never reports.
|
||||||
# The three-option version of that same rule is asserted at eval in
|
# The rule that a secret needs the other two options is asserted at
|
||||||
# ./../swarm.nix, so this can only ever emit a complete set.
|
# eval in ./../swarm.nix, so this can only ever emit a complete set.
|
||||||
#
|
#
|
||||||
# ⚠️ The guard and the value beside it read different namespaces on
|
# ⚠️ Gated on the SECRET as well as the URL. The URL defaults to the
|
||||||
# purpose: where the queue is and where its secret sits are this
|
# queue's name on every hive, so on its own it says where the queue is,
|
||||||
# machine's, the token endpoint is the swarm's one address. The
|
# not that this hive publishes; the secret is that claim. Gated on the
|
||||||
# assertion covers all three, which is what keeps the guard honest.
|
# URL alone, a hive with no secret would be handed a secret path that
|
||||||
lib.optionalAttrs (config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null) {
|
# `LoadCredential` in ./default.nix never fills.
|
||||||
HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl;
|
lib.optionalAttrs
|
||||||
HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
|
(
|
||||||
# The identity swarm-authelia.nix already declares for every entry in
|
config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl != null
|
||||||
# `swarm.hives` — the hive does not choose its own name here, it uses
|
&& config.services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile != null
|
||||||
# the one the roster gave it.
|
)
|
||||||
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
|
{
|
||||||
# `%d` is systemd's credentials directory — see the LoadCredential in
|
HIVE_C0RE_NATS_URL = config.services.hyperhive.deploy.hive-controller.statusPublish.natsUrl;
|
||||||
# ./default.nix. The daemon reads a path, never a value.
|
HIVE_C0RE_OIDC_TOKEN_ENDPOINT = config.services.hyperhive.swarm.statusPublish.tokenEndpoint;
|
||||||
HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
|
# The identity swarm-authelia.nix already declares for every entry in
|
||||||
}
|
# `swarm.hives` — the hive does not choose its own name here, it uses
|
||||||
|
# the one the roster gave it.
|
||||||
|
HIVE_C0RE_OIDC_CLIENT_ID = "hive-${config.services.hyperhive.hiveName}";
|
||||||
|
# `%d` is systemd's credentials directory — see the LoadCredential in
|
||||||
|
# ./default.nix. The daemon reads a path, never a value.
|
||||||
|
HIVE_C0RE_OIDC_CLIENT_SECRET_FILE = "%d/swarm-status-client.secret";
|
||||||
|
}
|
||||||
// {
|
// {
|
||||||
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
|
# Where ../glue-queue-agent-credential.nix lands the AGENTS' queue
|
||||||
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
|
# credential. Read by `hive_c0re::lifecycle::host_config`, which stats the
|
||||||
|
|
|
||||||
|
|
@ -86,31 +86,13 @@ in
|
||||||
config.services.hyperhive.deploy.nats.autoGenerateCallout =
|
config.services.hyperhive.deploy.nats.autoGenerateCallout =
|
||||||
lib.mkDefault cfg.deploy.singleHostSwarm;
|
lib.mkDefault cfg.deploy.singleHostSwarm;
|
||||||
|
|
||||||
|
# ⚠️ Every `swarm.*` default this mode sets lives INSIDE this attrset, not
|
||||||
|
# as a second `config.services.hyperhive.swarm.…` path beside it — written
|
||||||
|
# that way the two definitions of `swarm` collide and the nested one is
|
||||||
|
# silently lost. The gate caught exactly that on the controller's queue URL,
|
||||||
|
# when this mode still set it: mode on, `natsUrl` still "".
|
||||||
config.services.hyperhive.swarm = {
|
config.services.hyperhive.swarm = {
|
||||||
ca.autoConfigure = lib.mkDefault cfg.deploy.singleHostSwarm;
|
ca.autoConfigure = lib.mkDefault cfg.deploy.singleHostSwarm;
|
||||||
# The controller's queue coordinates. Kept with the mode, not in the
|
|
||||||
# options' own `default`, because the controller's minted client secret
|
|
||||||
# only exists on the host authelia ran its first boot on — so they belong
|
|
||||||
# to the mode that asserts this box is the whole deployment.
|
|
||||||
#
|
|
||||||
# Deriving them from `deploy.nats` / `deploy.authelia`
|
|
||||||
# inside those defaults is the mixing this file exists to prevent: the
|
|
||||||
# option would be describing a deployment shape instead of describing
|
|
||||||
# itself, and "what does all-local turn on?" would stop having one
|
|
||||||
# answer.
|
|
||||||
#
|
|
||||||
# ⚠️ Must live INSIDE this attrset, not as a second
|
|
||||||
# `config.services.hyperhive.swarm.…` path beside it — written that
|
|
||||||
# way the two definitions of `swarm` collide and the nested one is
|
|
||||||
# silently lost. The gate caught exactly that: mode on, `natsUrl`
|
|
||||||
# still "".
|
|
||||||
#
|
|
||||||
# The *requirement* stays in `swarm-controller.nix` as an assertion:
|
|
||||||
# needing a queue is the controller's own property in every topology,
|
|
||||||
# and only the convenience is local.
|
|
||||||
controller.queue.natsUrl = lib.mkIf cfg.deploy.singleHostSwarm (
|
|
||||||
lib.mkDefault "tls://${config.services.hyperhive.swarm.nats.domain}:${toString config.services.hyperhive.swarm.nats.port}"
|
|
||||||
);
|
|
||||||
};
|
};
|
||||||
|
|
||||||
# The controller is asserted by the MODE and by nothing else. Its own
|
# The controller is asserted by the MODE and by nothing else. Its own
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,7 @@ let
|
||||||
cfg = config.services.hyperhive.swarm.controller;
|
cfg = config.services.hyperhive.swarm.controller;
|
||||||
deployCfg = config.services.hyperhive.deploy;
|
deployCfg = config.services.hyperhive.deploy;
|
||||||
autheliaCfg = config.services.hyperhive.swarm.authelia;
|
autheliaCfg = config.services.hyperhive.swarm.authelia;
|
||||||
|
natsCfg = config.services.hyperhive.swarm.nats;
|
||||||
|
|
||||||
# Where the secret store is, and whether this host holds the controller's
|
# Where the secret store is, and whether this host holds the controller's
|
||||||
# own leaf for it. ⚠️ The controller's pair, NOT `deploy.bao.clientCertFile`
|
# own leaf for it. ⚠️ The controller's pair, NOT `deploy.bao.clientCertFile`
|
||||||
|
|
@ -364,18 +365,18 @@ in
|
||||||
queue = {
|
queue = {
|
||||||
natsUrl = lib.mkOption {
|
natsUrl = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "";
|
default = "tls://${natsCfg.domain}:${toString natsCfg.port}";
|
||||||
|
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
|
||||||
example = "tls://nats.example.com:4222";
|
example = "tls://nats.example.com:4222";
|
||||||
description = ''
|
description = ''
|
||||||
Where the controller reaches the swarm queue.
|
Where the controller reaches the swarm queue.
|
||||||
|
|
||||||
|
Defaults to the queue's name, which is the same URL on every
|
||||||
|
host: the queue's own host resolves it locally, and any other
|
||||||
|
through the operator's DNS.
|
||||||
|
|
||||||
Empty means unset, which the assertion below refuses — a
|
Empty means unset, which the assertion below refuses — a
|
||||||
controller with no queue is not a lighter controller.
|
controller with no queue is not a lighter controller.
|
||||||
|
|
||||||
`singleHostSwarm` fills this in with
|
|
||||||
`tls://<swarm.nats.domain>:<port>`. That derivation lives with the
|
|
||||||
mode rather than here, so this option describes itself rather than
|
|
||||||
a deployment shape.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -764,8 +765,8 @@ in
|
||||||
message = ''
|
message = ''
|
||||||
services.hyperhive.swarm.controller.queue.natsUrl is unset.
|
services.hyperhive.swarm.controller.queue.natsUrl is unset.
|
||||||
|
|
||||||
`singleHostSwarm` fills it in with the queue's name. A controller
|
It defaults to the queue's name; something in this configuration
|
||||||
in any other deployment has to be told the URL.
|
set it to "". Remove that setting, or set the URL.
|
||||||
'';
|
'';
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
|
|
|
||||||
|
|
@ -115,16 +115,15 @@ let
|
||||||
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
|
lib.filterAttrs (_: hive: hive.certFingerprint != null) swarmCfg.hives
|
||||||
);
|
);
|
||||||
|
|
||||||
# Whether the queue and its minted secret are on THIS host — the two
|
# Whether this hive's minted status secret is on THIS host, which is the
|
||||||
# coordinates that are a statement about this machine's disk and
|
# one status-publishing coordinate that is a statement about this
|
||||||
# netns, from one condition so a partial set is unrepresentable rather
|
# machine's disk.
|
||||||
# than merely detected.
|
|
||||||
#
|
#
|
||||||
# ⚠️ The token endpoint is NOT one of them any more. It is the swarm's
|
# ⚠️ Neither the token endpoint nor the queue's URL is gated on it. Both
|
||||||
# one address, derived from `swarm.authelia.url` like
|
# are the swarm's one address: the endpoint derived from
|
||||||
# ./swarm-controller.nix's own `queue.tokenEndpoint` already is, so it
|
# `swarm.authelia.url`, the queue from its name, which resolves on every
|
||||||
# is correct for a remote provider and does not ask where anything
|
# hive. So they are correct wherever the queue and the IdP run, and do
|
||||||
# runs.
|
# not ask where anything runs.
|
||||||
queueLocal = deployCfg.nats.enable && deployCfg.authelia.enable && cfg.hiveName != null;
|
queueLocal = deployCfg.nats.enable && deployCfg.authelia.enable && cfg.hiveName != null;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
|
|
@ -373,18 +372,16 @@ in
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Deliberately an assertion and not a silent "then publish
|
# Deliberately an assertion and not a silent "then publish
|
||||||
# nothing": a half-set trio is a config an operator believes is
|
# nothing": a secret without the coordinates it is presented at is
|
||||||
# working, and its runtime failure mode is the expensive one —
|
# a config an operator believes is working, and its runtime failure
|
||||||
# the daemon comes up fine, never connects, and the hive reads
|
# mode is the expensive one — the daemon comes up fine, never
|
||||||
# `never_reported` on a dashboard nobody is watching yet.
|
# connects, and the hive reads `never_reported` on a dashboard
|
||||||
|
# nobody is watching yet.
|
||||||
#
|
#
|
||||||
# Safe to add to an existing deployment: every `statusPublish`
|
# It encodes a property of the code rather than an intended shape —
|
||||||
# default is either all-local or all-null, so no config that
|
# hive-c0re genuinely cannot publish with two of three coordinates —
|
||||||
# evaluates today can be caught by this. It also encodes a
|
# which is the distinction the `serviceDomains'` comment at the top
|
||||||
# property of the code rather than an intended shape — hive-c0re
|
# of this file was written about.
|
||||||
# genuinely cannot publish with two of three coordinates — which
|
|
||||||
# is the distinction the `serviceDomains'` comment at the top of
|
|
||||||
# this file was written about.
|
|
||||||
#
|
#
|
||||||
# The three coordinates live in two namespaces now: the token
|
# The three coordinates live in two namespaces now: the token
|
||||||
# endpoint is the swarm's one address, the other two are this
|
# endpoint is the swarm's one address, the other two are this
|
||||||
|
|
@ -392,23 +389,21 @@ in
|
||||||
# operator told only the option names would look for them under
|
# operator told only the option names would look for them under
|
||||||
# one prefix and find one of them.
|
# one prefix and find one of them.
|
||||||
#
|
#
|
||||||
# ⚠️ Asymmetric on purpose. A token endpoint on its own is the
|
# ⚠️ Asymmetric on purpose. The token endpoint and the queue's URL
|
||||||
# normal state of every hive in a swarm that has an IdP — it says
|
# both default to the swarm's one address on every hive: they say
|
||||||
# where the IdP is, not that this hive publishes. Only the two
|
# where the IdP and the queue are, not that this hive publishes, so
|
||||||
# per-host coordinates are a claim to be publishing, and they
|
# either without the secret is publishing off rather than a half-set
|
||||||
# need the endpoint to mean anything.
|
# trio. Only the client secret is a claim to be publishing, and it
|
||||||
|
# needs the other two to mean anything.
|
||||||
assertion =
|
assertion =
|
||||||
let
|
deployCfg.hive-controller.statusPublish.clientSecretFile == null
|
||||||
local = lib.filter (v: v != null) [
|
|| (
|
||||||
deployCfg.hive-controller.statusPublish.natsUrl
|
deployCfg.hive-controller.statusPublish.natsUrl != null
|
||||||
deployCfg.hive-controller.statusPublish.clientSecretFile
|
&& swarmCfg.statusPublish.tokenEndpoint != null
|
||||||
];
|
);
|
||||||
in
|
|
||||||
builtins.length local == 0
|
|
||||||
|| (builtins.length local == 2 && swarmCfg.statusPublish.tokenEndpoint != null);
|
|
||||||
message = ''
|
message = ''
|
||||||
This hive's status-publishing coordinates have to be set
|
This hive has a status-publishing client secret but not the
|
||||||
together or not at all — it has only some of them.
|
coordinates to publish with it.
|
||||||
|
|
||||||
Currently:
|
Currently:
|
||||||
deploy.hive-controller.statusPublish.natsUrl
|
deploy.hive-controller.statusPublish.natsUrl
|
||||||
|
|
@ -419,7 +414,7 @@ in
|
||||||
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
|
= ${toString deployCfg.hive-controller.statusPublish.clientSecretFile}
|
||||||
|
|
||||||
Set the missing ones to publish this hive's status to the
|
Set the missing ones to publish this hive's status to the
|
||||||
swarm, or set all three to null to turn publishing off.
|
swarm, or set clientSecretFile to null to turn publishing off.
|
||||||
'';
|
'';
|
||||||
}
|
}
|
||||||
(nameGuards.mustNotEqual {
|
(nameGuards.mustNotEqual {
|
||||||
|
|
@ -543,21 +538,22 @@ in
|
||||||
options.services.hyperhive.deploy.hive-controller.statusPublish = {
|
options.services.hyperhive.deploy.hive-controller.statusPublish = {
|
||||||
natsUrl = lib.mkOption {
|
natsUrl = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.str;
|
type = lib.types.nullOr lib.types.str;
|
||||||
default =
|
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
|
||||||
if queueLocal then "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}" else null;
|
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
|
||||||
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
|
|
||||||
example = "tls://nats.example.com:4222";
|
example = "tls://nats.example.com:4222";
|
||||||
description = ''
|
description = ''
|
||||||
Where the swarm queue listens, as seen from *this* hive.
|
Where the swarm queue listens, as seen from *this* hive.
|
||||||
|
|
||||||
Defaults to the queue's name when this host runs the queue and the
|
Defaults to the queue's name on every hive. The queue's own host
|
||||||
IdP. A hive that is not the swarm host sets the same URL: the name
|
resolves it locally, and any other hive through the operator's
|
||||||
resolves through the operator's DNS there. TLS only, and by name,
|
DNS. TLS only, and by name, since the queue's certificate carries
|
||||||
since the queue's certificate carries the name and no address.
|
the name and no address.
|
||||||
|
|
||||||
Null disables status publishing: this hive computes its own
|
A URL alone does not publish.
|
||||||
readiness as always, and simply offers it to nobody. The swarm
|
{option}`services.hyperhive.deploy.hive-controller.statusPublish.clientSecretFile`
|
||||||
controller then reports it `never_reported`, which is the honest
|
is what turns publishing on. Without it this hive computes its own
|
||||||
|
readiness as always and offers it to nobody, and the swarm
|
||||||
|
controller reports it `never_reported`, which is the honest
|
||||||
reading.
|
reading.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
@ -579,6 +575,9 @@ in
|
||||||
readable, and one in the environment is readable by anything
|
readable, and one in the environment is readable by anything
|
||||||
that can open {file}`/proc/<pid>/environ`.
|
that can open {file}`/proc/<pid>/environ`.
|
||||||
|
|
||||||
|
Setting it is what makes this hive publish its status; null
|
||||||
|
turns publishing off.
|
||||||
|
|
||||||
Defaults to authelia's own minted secret when the IdP runs on
|
Defaults to authelia's own minted secret when the IdP runs on
|
||||||
this host. On any other hive the secret has to get here somehow,
|
this host. On any other hive the secret has to get here somehow,
|
||||||
and the swarm does not distribute it — copy it out of the swarm
|
and the swarm does not distribute it — copy it out of the swarm
|
||||||
|
|
@ -598,25 +597,24 @@ in
|
||||||
# owns `queue.agentCredentialDir` in the same namespace.
|
# owns `queue.agentCredentialDir` in the same namespace.
|
||||||
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
|
options.services.hyperhive.deploy.hive-controller.queue.agentNatsUrl = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.str;
|
type = lib.types.nullOr lib.types.str;
|
||||||
default =
|
default = "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}";
|
||||||
if queueLocal then "tls://${swarmCfg.nats.domain}:${toString swarmCfg.nats.port}" else null;
|
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}"'';
|
||||||
defaultText = lib.literalExpression ''"tls://''${swarm.nats.domain}:''${swarm.nats.port}" when this host runs the queue and the IdP, else null'';
|
|
||||||
example = "tls://nats.example.com:4222";
|
example = "tls://nats.example.com:4222";
|
||||||
description = ''
|
description = ''
|
||||||
Where the swarm queue listens, as an agent *container* on this host
|
Where the swarm queue listens, as an agent *container* on this host
|
||||||
reaches it.
|
reaches it.
|
||||||
|
|
||||||
Defaults to the queue's name when this host runs the queue. The agent
|
Defaults to the queue's name on every hive. The agent resolves it
|
||||||
resolves it through the bridge, where dnsmasq answers it with the
|
through the bridge: on the queue's own host dnsmasq answers it with
|
||||||
bridge address. ⚠️ Never a loopback address: inside an agent's network
|
the bridge address, and elsewhere it forwards to the operator's DNS,
|
||||||
namespace `127.0.0.1` is the agent, not this host, and the queue's
|
as for the store's name. ⚠️ Never a loopback address: inside an
|
||||||
certificate names no address anyway.
|
agent's network namespace `127.0.0.1` is the agent, not this host,
|
||||||
|
and the queue's certificate names no address anyway.
|
||||||
|
|
||||||
Null means this hive's agents have not been given the queue's address.
|
Null means this hive's agents have not been given the queue's address.
|
||||||
Together with
|
Together with
|
||||||
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
|
{option}`services.hyperhive.swarm.statusPublish.tokenEndpoint` it is
|
||||||
what decides whether the harness is handed queue coordinates at all; a
|
what decides whether the harness is handed queue coordinates at all.
|
||||||
hive whose queue is elsewhere names the address its containers route to.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -42,6 +42,10 @@ let
|
||||||
# renders the same absences `bare` does.
|
# renders the same absences `bare` does.
|
||||||
centralToggleOff = hive { enable = false; };
|
centralToggleOff = hive { enable = false; };
|
||||||
|
|
||||||
|
# Every hive defaults the agents' queue address to the queue's name, so the
|
||||||
|
# only hive without one is a hive told to have none.
|
||||||
|
noAgentQueue = hive { deploy.hive-controller.queue.agentNatsUrl = null; };
|
||||||
|
|
||||||
withCi = hive { deploy.forgejo.ci.enable = true; };
|
withCi = hive { deploy.forgejo.ci.enable = true; };
|
||||||
|
|
||||||
# A priority collision is a property of the *option*, not
|
# A priority collision is a property of the *option*, not
|
||||||
|
|
@ -149,13 +153,13 @@ let
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The absence arm, and what makes the two above able to fail: a hive
|
# The absence arm, and what makes the two above able to fail: a hive
|
||||||
# with no queue address must forward neither coordinate, because half a
|
# with no queue address for its agents must forward neither coordinate,
|
||||||
# pair reaches the harness as a partial configuration rather than as
|
# because half a pair reaches the harness as a partial configuration
|
||||||
# none.
|
# rather than as none.
|
||||||
name = "a hive with no swarm queue forwards no agent queue coordinates";
|
name = "a hive with no agent queue address forwards no agent queue coordinates";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
e = bare.systemd.services.hive-c0re.environment;
|
e = noAgentQueue.systemd.services.hive-c0re.environment;
|
||||||
in
|
in
|
||||||
!(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT);
|
!(e ? HIVE_AGENT_NATS_URL) && !(e ? HIVE_AGENT_OIDC_TOKEN_ENDPOINT);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -47,6 +47,40 @@ let
|
||||||
deploy.nats.autoGenerateCallout = true;
|
deploy.nats.autoGenerateCallout = true;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# A hive that is not the queue's host, with nothing about the queue's
|
||||||
|
# address set by hand: what every hive but one in a multi-host swarm looks
|
||||||
|
# like. The controller is on too, since it may run away from the queue.
|
||||||
|
#
|
||||||
|
# The two secrets are the ones a hive away from authelia already has to be
|
||||||
|
# handed, and neither is an address; without them this hive would fail
|
||||||
|
# assertions that have nothing to do with the queue.
|
||||||
|
remoteSecrets = {
|
||||||
|
deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
|
||||||
|
deploy.swarm-controller.queue.clientSecretFile = "/var/lib/secrets/swarm-controller.secret";
|
||||||
|
};
|
||||||
|
remote = hive (lib.recursiveUpdate remoteSecrets { deploy.swarm-controller.enable = true; });
|
||||||
|
|
||||||
|
# The same hive given its status secret by hand, which is what turns
|
||||||
|
# publishing on away from the IdP's host.
|
||||||
|
remotePublishing = hive (
|
||||||
|
lib.recursiveUpdate remoteSecrets {
|
||||||
|
deploy.hive-controller.statusPublish.clientSecretFile = "/var/lib/secrets/hive-h1.secret";
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
# A real half-config: the secret, with the URL it would be presented at
|
||||||
|
# taken away.
|
||||||
|
remoteSecretNoUrl = hive (
|
||||||
|
lib.recursiveUpdate remoteSecrets {
|
||||||
|
deploy.hive-controller.statusPublish.clientSecretFile = "/var/lib/secrets/hive-h1.secret";
|
||||||
|
deploy.hive-controller.statusPublish.natsUrl = null;
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
failedAssertions = m: lib.filter (a: !a.assertion) m.assertions;
|
||||||
|
refusedStatusSecret =
|
||||||
|
m: lib.any (a: lib.hasInfix "status-publishing client secret" a.message) (failedAssertions m);
|
||||||
|
|
||||||
policyScript = allLocal.systemd.services.swarm-bao-nats-tls-policy.script;
|
policyScript = allLocal.systemd.services.swarm-bao-nats-tls-policy.script;
|
||||||
leafUnit = allLocal.systemd.services.swarm-bao-nats-tls;
|
leafUnit = allLocal.systemd.services.swarm-bao-nats-tls;
|
||||||
natsContainer = allLocal.containers.swarm-nats.config;
|
natsContainer = allLocal.containers.swarm-nats.config;
|
||||||
|
|
@ -78,6 +112,8 @@ let
|
||||||
containerUnits = lib.concatLists (
|
containerUnits = lib.concatLists (
|
||||||
lib.mapAttrsToList (c: cc: fromUnits c cc.config.systemd.services) machine.containers
|
lib.mapAttrsToList (c: cc: fromUnits c cc.config.systemd.services) machine.containers
|
||||||
);
|
);
|
||||||
|
# The responder runs beside the queue only, so a hive without one has
|
||||||
|
# none to read.
|
||||||
responder =
|
responder =
|
||||||
map
|
map
|
||||||
(u: {
|
(u: {
|
||||||
|
|
@ -85,7 +121,9 @@ let
|
||||||
value = u;
|
value = u;
|
||||||
})
|
})
|
||||||
(
|
(
|
||||||
urlsIn machine.containers.swarm-nats.config.systemd.services.swarm-nats-auth.serviceConfig.ExecStart
|
lib.optionals (machine.containers ? swarm-nats) (
|
||||||
|
urlsIn machine.containers.swarm-nats.config.systemd.services.swarm-nats-auth.serviceConfig.ExecStart
|
||||||
|
)
|
||||||
);
|
);
|
||||||
in
|
in
|
||||||
lib.listToAttrs (fromUnits "host" machine.systemd.services ++ containerUnits ++ responder);
|
lib.listToAttrs (fromUnits "host" machine.systemd.services ++ containerUnits ++ responder);
|
||||||
|
|
@ -212,6 +250,65 @@ let
|
||||||
!(lib.elem "swarm-bao-nats-tls-policy.service" u.after)
|
!(lib.elem "swarm-bao-nats-tls-policy.service" u.after)
|
||||||
&& !(lib.elem "swarm-bao-nats-tls-policy.service" u.wants);
|
&& !(lib.elem "swarm-bao-nats-tls-policy.service" u.wants);
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
# Every hive dials the queue by the same name, so a hive away from it
|
||||||
|
# needs no URL of its own. Control and property in one: the scan must
|
||||||
|
# reach the controller and the agents' address here, and every URL it
|
||||||
|
# finds, like the options it reads through, is the name.
|
||||||
|
name = "a hive that is not the queue's host dials tls://<the queue's name>:4222 with nothing set";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = clientUrls remote;
|
||||||
|
d = remote.services.hyperhive.deploy;
|
||||||
|
in
|
||||||
|
!d.nats.enable
|
||||||
|
&& s ? "host/hive-c0re/HIVE_AGENT_NATS_URL"
|
||||||
|
&& s ? "host/swarm-controller/SWARM_CONTROLLER_NATS_URL"
|
||||||
|
&& lib.all (u: u == natsUrl) (lib.attrValues s)
|
||||||
|
&& d.hive-controller.statusPublish.natsUrl == natsUrl
|
||||||
|
&& d.hive-controller.queue.agentNatsUrl == natsUrl
|
||||||
|
&& remote.services.hyperhive.swarm.controller.queue.natsUrl == natsUrl;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# The URL is set and the secret is not, which is every such hive until
|
||||||
|
# an operator places one: publishing is off, not misconfigured.
|
||||||
|
name = "that hive evaluates without an assertion failure";
|
||||||
|
ok = failedAssertions remote == [ ];
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# Off means off: hive-c0re is handed no status coordinates, rather
|
||||||
|
# than a secret path nothing fills.
|
||||||
|
name = "without its status secret, that hive's hive-c0re is given no status coordinates";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = remote.systemd.services.hive-c0re;
|
||||||
|
in
|
||||||
|
!(s.environment ? HIVE_C0RE_NATS_URL)
|
||||||
|
&& !(s.environment ? HIVE_C0RE_OIDC_CLIENT_SECRET_FILE)
|
||||||
|
&& !(lib.any (lib.hasPrefix "swarm-status-client.secret:") (
|
||||||
|
lib.toList (s.serviceConfig.LoadCredential or [ ])
|
||||||
|
));
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# The secret alone turns publishing on, at the default URL.
|
||||||
|
name = "given its status secret, that hive publishes to the queue's name";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = remotePublishing.systemd.services.hive-c0re;
|
||||||
|
in
|
||||||
|
failedAssertions remotePublishing == [ ]
|
||||||
|
&& s.environment.HIVE_C0RE_NATS_URL == natsUrl
|
||||||
|
&& s.environment.HIVE_C0RE_OIDC_CLIENT_SECRET_FILE == "%d/swarm-status-client.secret"
|
||||||
|
&& lib.elem "swarm-status-client.secret:/var/lib/secrets/hive-h1.secret" (
|
||||||
|
lib.toList s.serviceConfig.LoadCredential
|
||||||
|
);
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# What the assertion was written for is still refused: a secret with
|
||||||
|
# nowhere to present it.
|
||||||
|
name = "a status secret without a queue URL is refused at eval";
|
||||||
|
ok = refusedStatusSecret remoteSecretNoUrl && !(refusedStatusSecret remote);
|
||||||
|
}
|
||||||
];
|
];
|
||||||
in
|
in
|
||||||
runGroup "nats-tls" cases
|
runGroup "nats-tls" cases
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue