Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
Author SHA1 Message Date
atlas
b32c92a446 swarm-secret-client: say why the queue credential's client id is required
Pinning a decision rather than changing behaviour, because I was one
edit away from reversing it and left no reason on the field.

The sibling `matrix::Credential::homeserver` is an `Option`, which reads
like the house style to copy. It is not: that field is optional because
it was added to objects already in the store, and KV2 keeps those
versions forever. This path has never been written, so there is nothing
to stay compatible with.

Making it optional would also defeat the field. The id rides with the
secret so a reader never has to spell `hive-<name>-agent` itself, and
the only thing a reader holding `None` can do is exactly that. An object
without an id is not a usable credential, so failing to decode is the
behaviour we want.

Refs #3853
2026-09-12 11:22:33 +02:00
atlas
d6a79b4e63 module-eval: cover the secret publisher, and catch that it cannot evaluate
Five cases: the leaf and the pairing that points at it, that the push
hands bao a path rather than the secret, a path per hive in the roster,
that a publisher holding an identity renders on a host with no store,
and the control that renders none without an identity.

They fail on this branch, and that is the finding. The suite was green
at 100 cases with the publisher already committed, because no fixture
enabled both authelia and a store identity — so the module's `config`
never activated and its script was never evaluated. An imported module
whose config never fires is as unmeasured as an unimported one.

What they surface: `swarm.authelia.agentClientSuffix` does not exist on
main. The agent principal these secrets belong to is minted by the PR
for the agent queue principal, whose branch adds both the option and the
`agentClients` list authelia mints from. Delivery of a secret nothing
mints cannot evaluate, let alone run — so that PR lands first and this
one rebases onto it.

The argv case strips comments before matching, which it earned: a
`script` renders its own comments into the text, and this unit's
comments name the hazard verbatim so the next editor does not
reintroduce it. Matching the raw text read that warning and failed —
a check the artifact defeats by describing the thing it is checked for.

Refs #3853
2026-09-12 11:22:33 +02:00
atlas
eaa52ef200 swarm: publish minted OIDC client secrets into the swarm store
A hive that does not host authelia has no path to its own agent queue
client secret. The mint writes the plaintext to a host directory whose
other reader lives in a different container, so the host that mints is
the only place both trees are addressable — which is where this unit
runs.

Four pieces, in the order they depend on each other: the leaf
(glue-bao-tls.nix signs it, because the thing that owns a private key
owns issuing from it), the module declaring its own cert/key options,
the one-pairing glue file pointing them at that leaf, and the imports.

The unit is gated on holding a client identity, never on
deploy.bao.enable — that option is the co-location assumption itself,
and the publisher is the case that assumption excludes.

The secret is passed to bao as `value=@<path>`, never as an argv
element: bao is an external binary, so an argument is world-readable in
/proc for the life of the call.

Refs #3853
2026-09-12 11:22:33 +02:00
atlas
98f2a94d82 swarm-bao: write the secret publisher's policy and cert-auth role
A sibling unit rather than more script in swarm-bao-controller-policy, because
that unit's name is an operator-facing string: docs/getting-started/setup.md
tells a reader to run `systemctl status swarm-bao-controller-policy`. Widening
it to two principals makes the name wrong; renaming it makes the instruction
wrong.

`after` and not `requires`. The controller's unit creates the KV and cert-auth
mounts this one writes into, so the ordering is real — but a failed oneshot
still counts as finished, so `requires` would neither wait for its success nor
re-run this unit when the sibling's own retry eventually lands. Ordering plus
this unit's `Restart=on-failure` is what converges.

Four module-eval cases, because the unit arrived with every claim about it in
prose and the suite still reporting the same count: the grant is write-only and
reaches the hive prefix alone (pinned as the whole capability list, since an
added capability is what a presence check misses, with negative arms for the
agent prefix, the bare swarm prefix and the policy path); it is ordered after
the unit that creates the mounts; it renders on the host; and the control, that
it does not render inside the store's container.

Refs #3853
2026-09-12 10:56:50 +02:00
atlas
a597b6bb1c swarm-bao: the secret publisher's policy — one grant, under the hive prefix only
Write-only `create`/`update` on `secret/data/swarm/hives/*`, and nothing else.
It copies secrets in and never reads one back; a read capability would let a
file-copier recover every hive's credentials rather than merely replace them.

`hives/` and not `swarm/*` because this principal has no business with an
agent's or a service's credentials, and the hive prefix is the only one whose
paths it produces. `secret/data/` is KV v2's ACL prefix, inserted by the engine
rather than written by the caller — the same trap the controller's grant
documents one binding up.

Named outside `hive-*`: the controller may create policies under that prefix,
and a policy it can rewrite is not a constraint on it.

The unit that writes this lands next. Refs #3853
2026-09-12 10:56:50 +02:00
atlas
e8ff633c6b swarm-bao, swarm: give the secret publisher its own subject, reserved like the controller's
The unit that will copy authelia's minted OIDC client secrets into the store
needs an identity of its own. Not the controller's: that grant includes
rewriting every hive's policy and login role, which a unit whose whole job is
copying one file has no business holding.

The subject joins `certAuthCns`, so it is unrepresentable as a hive name for
the same reason the controller's is — cert auth trusts the CA, and a hive's
own leaf carries its name as the CN.

The module-eval case collides with the SECOND list element and leaves the
controller's subject at its default. A list with one consulted element and one
dead one is indistinguishable from the first element's case, so without this
the addition could be inert and nothing would say so.

Refs #3853
2026-09-12 10:56:50 +02:00
atlas
395ecbdf41 swarm-secret-client: name the hive queue credential, and grant a hive its own kind
The agreement half of delivering the agent queue principal's client secret
through the store. No producer yet, so nothing writes this path — the unit
that does lands in the same PR, with the write grant it needs.

queue.rs is the sibling matrix.rs prescribes for a second kind of secret
rather than another field on a shared struct. Keyed per HIVE, not per agent:
the queue identity is minted once per hive at deploy time and says which hive
an agent belongs to, never which agent.

The client id rides with the secret for matrix.rs's stated reason — a
credential has to be reconstructable from the store alone, and deriving
`hive-<name>-agent` on the reading side is the split spelling the authelia
module warns denies every agent as a timeout.

policy.rs's render() takes the hive name now and emits a second, narrow
stanza for that hive's own path. The agent stanza is untouched: an agent's
path does not name its hive, so narrowing it still needs the enumeration
docs/trust-boundary/security.md rejects. A hive path does name its principal,
so scoping it costs nothing and drifts nowhere.

every_hive_gets_a_byte_identical_document is replaced rather than deleted.
Its surviving half is that the text is a function of the deploy-time name
alone, so a re-emission cannot drift; the new arms are that one hive's
document cannot reach another's path, and that a name which could close the
stanza is refused — live again now that a name reaches the document text.

Refs #3853
2026-09-12 10:56:50 +02:00
12 changed files with 711 additions and 56 deletions

View file

@ -26,9 +26,11 @@
./glue-bao-tls.nix ./glue-bao-tls.nix
./glue-controller-bao-identity.nix ./glue-controller-bao-identity.nix
./glue-matrix-bao-token.nix ./glue-matrix-bao-token.nix
./glue-secret-publisher-bao-identity.nix
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao.nix ./swarm-bao.nix
./swarm-ca.nix ./swarm-ca.nix
./swarm-secret-publisher.nix
./swarm-nats.nix ./swarm-nats.nix
./swarm-controller.nix ./swarm-controller.nix
./swarm-grafana.nix ./swarm-grafana.nix

View file

@ -140,6 +140,13 @@ in
# this hive's name. # this hive's name.
[ -s ${pkiDir}/controller.pem ] || ${signLeaf} ${pkiDir} controller \ [ -s ${pkiDir}/controller.pem ] || ${signLeaf} ${pkiDir} controller \
${lib.escapeShellArg deployCfg.bao.controllerCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.controllerCommonName} "" clientAuth
# The secret publisher's, minted here for the reason the controller's
# line above gives — and it serves that case more often, not less: the
# publisher runs beside AUTHELIA, which is the one host guaranteed not
# to be this one whenever the store has a host of its own.
[ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \
${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth
''; '';
}; };
}; };

View file

@ -0,0 +1,42 @@
# Glue: point the secret publisher at the bao leaf minted for it.
#
# ONE PAIRING PER FILE — publisher ← bao, and nothing else. Deleting this
# leaves a publisher that takes operator-provided certificate paths, which is
# what any deployment not minting its own already does.
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, because the thing that owns a private key owns issuing from it. What
# belongs here is the pairing: which paths this host's publisher reads.
#
# ⚠️ Gated on the leaf existing, not on the store being enabled — the same rule
# ./glue-controller-bao-identity.nix states. A publisher on the store's own
# host is one deployment; a publisher beside a remote authelia holding a leaf
# issued out of band is another, and both want this wiring.
#
# Everything is `mkDefault`. An operator naming their own paths wins.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoDeploy = deployCfg.bao;
# Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own
# path rather than repeating that file's directory literal: an operator who
# moves the PKI moves both, and the two cannot drift apart.
haveMintedPki = baoDeploy.clientCertFile != null;
pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null;
in
{
config =
lib.mkIf (hyperhiveCfg.enable && deployCfg.swarm-secret-publisher.enable && haveMintedPki)
{
services.hyperhive.deploy.swarm-secret-publisher = {
baoClientCertFile = lib.mkDefault "${pkiDir}/secret-publisher.pem";
baoClientKeyFile = lib.mkDefault "${pkiDir}/secret-publisher-key.pem";
};
};
}

View file

@ -203,6 +203,30 @@ let
} }
''; '';
# The identity that copies authelia's minted OIDC client secrets into the
# store. Named outside `hive-*` for the same reason as the controller's: the
# controller may create policies under that prefix, and a policy it can
# rewrite is not a constraint on anything.
secretPublisherPolicyName = "swarm-secret-publisher";
secretPublisherCn = baoDeploy.secretPublisherCommonName;
# One grant, and every narrowing in it is load-bearing.
#
# `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than
# written by the caller — same trap as the controller's grant above.
#
# `hives/` and not `swarm/*`: this principal has no business with an agent's
# or a service's credentials, and the hive prefix is the only one whose paths
# it produces.
#
# Write-only. It copies secrets in and never reads one back; a read
# capability would let a file-copier recover every hive's credentials.
secretPublisherPolicyText = ''
path "${credentialMountPath}/data/swarm/hives/*" {
capabilities = ["create", "update"]
}
'';
# The KV v2 engine the controller writes agent credentials through. Named # The KV v2 engine the controller writes agent credentials through. Named
# once because the grant above and the `secrets enable` in the bootstrap unit # once because the grant above and the `secrets enable` in the bootstrap unit
# have to agree: a policy pointing at a mount nobody created is precisely the # have to agree: a policy pointing at a mount nobody created is precisely the
@ -554,6 +578,29 @@ in
''; '';
}; };
secretPublisherCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-secret-publisher";
example = "swarm-secret-publisher.svc";
description = ''
Subject the store's secret-publisher cert-auth role accepts — the
identity that copies authelia's minted OIDC client secrets into the
store so a hive that does not host authelia can read its own.
A **second** identity rather than reuse of
{option}`services.hyperhive.deploy.bao.controllerCommonName`: the
controller's grant includes rewriting every hive's policy and login
role, which is not something a unit whose whole job is copying one file
should be able to do. This one's grant is `create`/`update` under the
hive prefix and nothing else.
⚠️ Same collision as its sibling above, and the same answer: ./swarm.nix
feeds this value into the guard on
{option}`services.hyperhive.swarm.hives`, so a hive named after it fails
evaluation rather than silently receiving the publisher's grants.
'';
};
serverCaFile = lib.mkOption { serverCaFile = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = null; default = null;
@ -960,6 +1007,58 @@ in
''; '';
}; };
# A SIBLING rather than more script in the unit above, because that unit's
# name is an operator-facing string: ../../docs/getting-started/setup.md
# tells a reader to run `systemctl status swarm-bao-controller-policy`.
# Widening it to two principals would make the name wrong, and renaming it
# would make that instruction wrong.
#
# `after` and not `requires`: the unit above creates the KV and cert-auth
# mounts this one writes into, but a failed oneshot still counts as
# finished, so `requires` would neither wait for its success nor re-run
# this one when its own retry eventually lands. Ordering plus this unit's
# own retry is what actually converges.
systemd.services.swarm-bao-secret-publisher-policy = lib.mkIf haveBootstrapToken {
description = "write the swarm secret publisher's bao policy and cert-auth role";
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
];
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
# Same unseal wait as its sibling above, for the reason stated there:
# under `seal = "shamir"` a human unseals by hand, which can take a day.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
export BAO_TOKEN
printf '%s' ${lib.escapeShellArg secretPublisherPolicyText} |
bao policy write ${lib.escapeShellArg secretPublisherPolicyName} -
''
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
bao write auth/cert/certs/${lib.escapeShellArg secretPublisherPolicyName} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg secretPublisherCn} \
token_policies=${lib.escapeShellArg secretPublisherPolicyName} \
display_name=${lib.escapeShellArg secretPublisherCn}
'';
};
containers.${cfg.machine} = { containers.${cfg.machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;

View file

@ -0,0 +1,172 @@
# The unit that copies authelia's minted OIDC client secrets into the swarm's
# secret store, so a hive that does not host authelia can read its own.
#
# ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a
# separate thing. `deploy.authelia.hostClientSecretDir`'s own description says
# why: the plaintext's other reader lives in a **different container**, and
# containers that share this host's network namespace still have separate
# filesystem roots, so "the host is the only place both trees are addressable".
# A delivery step therefore runs on the host that mints — not on the store's
# host, and not on the reading hive's.
#
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit whose entire job is copying one
# file has no business holding it. ./swarm-bao.nix grants this one
# `create`/`update` under the hive prefix and nothing else.
#
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself. (`/knowledge/secret-hygiene.md`:
# "a path keeps the secret out of the store, and reading it into a shell
# variable puts it straight into argv".)
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoDeploy = deployCfg.bao;
autheliaCfg = hyperhiveCfg.swarm.authelia;
cfg = deployCfg.swarm-secret-publisher;
# A reader is defined by holding a certificate the store accepts, never by
# standing next to it — the rule ./glue-matrix-bao-token.nix states in full.
# `deploy.bao.enable` here would be the co-location assumption itself.
haveClientIdentity = cfg.baoClientCertFile != null && cfg.baoClientKeyFile != null;
# Both halves have to be here: the mint (authelia, for the plaintext) and an
# identity (for the store). Neither implies the other.
active = hyperhiveCfg.enable && cfg.enable && deployCfg.authelia.enable && haveClientIdentity;
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
# The client id agent containers present, per hive — composed exactly as
# ./swarm-authelia.nix composes it, from the same two read-only options, so a
# rename there cannot leave this spelling behind.
agentClientId = hive: "${autheliaCfg.hiveClientPrefix}${hive}${autheliaCfg.agentClientSuffix}";
in
{
options.services.hyperhive.deploy.swarm-secret-publisher = {
enable = lib.mkOption {
type = lib.types.bool;
default = deployCfg.authelia.enable;
defaultText = lib.literalExpression "deploy.authelia.enable";
description = ''
Publish the OIDC client secrets this host mints into the swarm's
secret store, so hives that do not run authelia can read their own.
Defaults to whether this host mints them, which is the only half of
the question that is a property of *this* host.
⚠️ Deliberately **not** `deploy.bao.enable`. That asks whether the
store stands here, and a publisher beside a remote store, holding a
leaf issued out of band, is a deployment this exists to serve —
defaulting on co-location would leave exactly that shape silently
publishing nothing. Whether the wiring is complete is the client
identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every non-co-located hive without a delivery
path, which is the state this exists to end — so the honest reason
to set it false is a deployment delivering those secrets by some
other mechanism it owns.
'';
};
baoClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = ''
Client certificate this publisher presents to the store. Its subject
must be {option}`services.hyperhive.deploy.bao.secretPublisherCommonName`
— cert auth matches on the CN, and the role accepts nothing else.
No default: a module that guessed would be holding the CA opinion
./swarm-bao.nix deliberately does not hold.
./glue-secret-publisher-bao-identity.nix points it at the leaf
./glue-bao-tls.nix mints, where this host mints one.
'';
};
baoClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = ''
Private key for {option}`services.hyperhive.deploy.swarm-secret-publisher.baoClientCertFile`.
Both or neither — the unit does not exist unless each is set.
'';
};
};
config = lib.mkIf active {
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-secret-publish" ];
# Re-publish when authelia rotates a secret. The mint writes the file, so
# the file is the event — there is no signal from authelia to subscribe to.
systemd.paths.swarm-secret-publish = {
description = "watch for minted OIDC client secrets to publish";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = deployCfg.authelia.hostClientSecretDir;
Unit = "swarm-secret-publish.service";
};
};
systemd.services.swarm-secret-publish = {
description = "publish minted OIDC client secrets to the swarm secret store";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
path = [
baoDeploy.package
pkgs.coreutils
];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
# Bounded here rather than left to systemd's default, so the number a
# boot waits on is in the file that waits. A sealed store answers on
# the port and never answers the write.
TimeoutStartSec = 60;
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
BAO_CLIENT_CERT = cfg.baoClientCertFile;
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
published=0
skipped=0
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"}
if [ -s "$src" ]; then
# `value=@$src` hands bao the PATH: bao opens the file itself, so
# the plaintext is never an argument of this process. Writing it as
# `value="$(cat "$src")"` would publish it to /proc for anyone on
# the host to read.
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/queue/agent"} \
value=@"$src" \
client_id=${lib.escapeShellArg (agentClientId hive)}
published=$((published + 1))
else
# Not an error: authelia mints on its FIRST BOOT, so an absent file
# is "not yet", and the path unit above re-runs this when it lands.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') hiveNames}
echo "published $published client secret(s), skipped $skipped"
'';
};
};
}

View file

@ -43,7 +43,10 @@ let
# every role added beside these widens what a hive name must not collide # every role added beside these widens what a hive name must not collide
# with. A hive's own leaf carries its name as the CN, so a hive named after # with. A hive's own leaf carries its name as the CN, so a hive named after
# one of these presents a certificate that role accepts. # one of these presents a certificate that role accepts.
certAuthCns = [ deployCfg.bao.controllerCommonName ]; certAuthCns = [
deployCfg.bao.controllerCommonName
deployCfg.bao.secretPublisherCommonName
];
# Public hostnames of the swarm's own services, in declaration order. # Public hostnames of the swarm's own services, in declaration order.
# `serviceDomains` below is this set sorted + deduplicated. # `serviceDomains` below is this set sorted + deduplicated.

View file

@ -327,6 +327,32 @@ let
deploy.bao.serverCaFile = lib.mkForce "/etc/pki/store-server-ca.pem"; deploy.bao.serverCaFile = lib.mkForce "/etc/pki/store-server-ca.pem";
}; };
# The IdP and the store on one machine: the shape where minted plaintext and
# a store identity are both present without an operator placing anything.
# Two hives in the roster, because the publisher walks it — an arm written
# against a single-hive fixture passes on a hardcoded name.
secretPublisherHere = hive {
deploy.bao.enable = true;
deploy.authelia.enable = true;
swarm.hives.h2.domain = "h2.t.local";
};
# The IdP with no store on the box and a leaf placed by hand, which is the
# deployment this unit exists for: authelia is the one host the store is
# guaranteed not to share once either has a machine of its own.
#
# ⚠️ `enable` is deliberately NOT set here. It used to be, with a comment
# saying the default asked whether both ran on this host — which documented
# the co-location bug instead of catching it. Leaving it unset is what makes
# this fixture exercise the default rather than mask it.
secretPublisherRemote = hive {
deploy.authelia.enable = true;
deploy.swarm-secret-publisher.baoClientCertFile = "/etc/pki/publisher.pem";
deploy.swarm-secret-publisher.baoClientKeyFile = "/etc/pki/publisher-key.pem";
};
# The same IdP with the identity taken away. Minting the secrets is not being
# able to publish them, and this is the arm that separates the two.
secretPublisherNoIdentity = hive { deploy.authelia.enable = true; };
# The host's `bao` wrapper, pulled apart once so each case below names one # The host's `bao` wrapper, pulled apart once so each case below names one
# property instead of a conjunction — a failing conjunction says only that # property instead of a conjunction — a failing conjunction says only that
# something is wrong. # something is wrong.
@ -539,6 +565,15 @@ let
deploy.bao.controllerCommonName = "ctl"; deploy.bao.controllerCommonName = "ctl";
}; };
# The reserved subjects are a LIST, and a list with one consulted element and
# one dead one looks identical from the first element's case. This fixture
# collides with the SECOND, leaving the controller's at its default.
hiveNamedAfterPublisherSubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao.secretPublisherCommonName = "pubctl";
swarm.hives.pubctl.domain = "p.t.local";
};
hiveNameWithComposedWord = hive { hiveNameWithComposedWord = hive {
deploy.swarm-otel.enable = false; deploy.swarm-otel.enable = false;
swarm.hives."h1-agent".domain = "a.t.local"; swarm.hives."h1-agent".domain = "a.t.local";
@ -565,6 +600,17 @@ let
equalityGuardFired hiveNamedAfterCertSubject equalityGuardFired hiveNamedAfterCertSubject
&& lib.any (a: !a.assertion && lib.hasInfix "'ctl'" a.message) hiveNamedAfterCertSubject.assertions; && lib.any (a: !a.assertion && lib.hasInfix "'ctl'" a.message) hiveNamedAfterCertSubject.assertions;
} }
{
# Every cert-auth subject is reserved, not just the first one in the
# list. Without this case the second element could be dead and the case
# above would still pass.
name = "a hive named after the secret publisher's subject is refused too";
ok =
equalityGuardFired hiveNamedAfterPublisherSubject
&& lib.any (
a: !a.assertion && lib.hasInfix "'pubctl'" a.message
) hiveNamedAfterPublisherSubject.assertions;
}
{ {
# Without this the case above proves nothing: an arm that fires for every # Without this the case above proves nothing: an arm that fires for every
# roster is not a guard, and `hives` is non-empty in both fixtures. # roster is not a guard, and `hives` is non-empty in both fixtures.
@ -821,6 +867,51 @@ let
in in
lib.hasInfix "sys/policies/acl/hive-*" s && !(lib.hasInfix "sys/policies/acl/*" s); lib.hasInfix "sys/policies/acl/hive-*" s && !(lib.hasInfix "sys/policies/acl/*" s);
} }
{
# Same host-side reasoning as the controller's granting unit above: the
# write needs a client certificate and the host is the side that has one.
name = "a store host with a placed bootstrap token renders the publisher's granting unit too";
ok =
let
u = baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy;
in
u.unitConfig.ConditionPathExists == "/run/secrets/bao-bootstrap.token"
&& lib.hasInfix "swarm-secret-publisher" u.script;
}
{
# The control for the case above, and the same one the controller's unit
# has: rendered on the host means NOT rendered in the container, where it
# would have neither an identity nor a route to the store.
name = "the publisher's granting unit is not rendered inside the store's container";
ok =
!(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-secret-publisher-policy);
}
{
# The whole point of a second principal. `hives/` and not `swarm/`, so it
# cannot touch an agent's or a service's credentials; and no `read`, so a
# unit whose job is copying a file cannot recover what is already there.
# Pinned as the full capability list, because an added capability is
# exactly what a presence check misses.
name = "the publisher's grant is write-only and reaches the hive prefix alone";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/hives/*\" {\n capabilities = [\"create\", \"update\"]" s
&& !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/*" s)
&& !(lib.hasInfix "sys/policies/acl" s);
}
{
# The ordering is load-bearing and invisible at runtime: the controller's
# unit creates the KV and cert-auth mounts this one writes into, so
# without it a cold boot races and fails with "route entry not found",
# which names neither unit.
name = "the publisher's granting unit is ordered after the one that creates the mounts";
ok = lib.elem "swarm-bao-controller-policy.service" (
baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy.after
);
}
{ {
# The policy authorising this route lives in another file, and nothing # The policy authorising this route lives in another file, and nothing
# else relates the grants to the paths the code actually writes. # else relates the grants to the paths the code actually writes.
@ -934,6 +1025,98 @@ let
&& builtins.elem "bao-client.pem:/var/lib/swarm-bao-pki/controller.pem" s.swarm-controller.serviceConfig.LoadCredential && builtins.elem "bao-client.pem:/var/lib/swarm-bao-pki/controller.pem" s.swarm-controller.serviceConfig.LoadCredential
&& builtins.elem "bao-client-key.pem:/var/lib/swarm-bao-pki/controller-key.pem" s.swarm-controller.serviceConfig.LoadCredential; && builtins.elem "bao-client-key.pem:/var/lib/swarm-bao-pki/controller-key.pem" s.swarm-controller.serviceConfig.LoadCredential;
} }
{
# The same hole the controller's case above names, open a second time: the
# PKI script grew a third leaf and no case read it.
name = "the store mints a leaf for the secret publisher, and the publisher is pointed at it";
ok =
let
m = secretPublisherHere;
p = m.services.hyperhive.deploy.swarm-secret-publisher;
in
lib.hasInfix "secret-publisher.pem" m.systemd.services.swarm-bao-pki.script
&& p.baoClientCertFile == "/var/lib/swarm-bao-pki/secret-publisher.pem"
&& p.baoClientKeyFile == "/var/lib/swarm-bao-pki/secret-publisher-key.pem";
}
{
# mara caught this by reading, which means no arm existed for it: the
# default asked `authelia.enable && bao.enable`, so the split deployment
# this unit is FOR defaulted off and published nothing, silently.
#
# The second clause is the control. Without it this passes on a default
# of plain `true`, which would be a different bug with the same symptom
# — an IdP-less host claiming it publishes secrets it never mints.
name = "the publisher defaults on where secrets are minted, whether or not the store is local";
ok =
secretPublisherRemote.services.hyperhive.deploy.swarm-secret-publisher.enable
&& !bare.services.hyperhive.deploy.swarm-secret-publisher.enable;
}
{
# The one security property of this unit, and why its push cannot be
# rewritten into the obvious shape: `bao` is an external binary, so an
# argument is world-readable in /proc for the life of the call.
# `value=@<path>` hands it the path and bao opens the file itself.
#
# The second arm is what makes the first mean anything — `value=@` can
# sit one line above a command substitution that put the plaintext in
# argv anyway.
#
# ⚠️ Comments are stripped first, and that is not tidiness. A `script`
# renders its own comments into the text, and this unit's comments name
# the hazard verbatim so the next editor does not reintroduce it. Without
# the strip this case reads that warning and fails — a check the artifact
# defeats by DESCRIBING the thing it is checked for.
name = "the publisher hands bao the secret's path, never the secret";
ok =
let
s = secretPublisherHere.systemd.services.swarm-secret-publish.script;
code = lib.concatStringsSep "\n" (
lib.filter (l: builtins.match "[[:space:]]*#.*" l == null) (lib.splitString "\n" s)
);
in
lib.hasInfix "value=@" code && !(lib.hasInfix "$(cat" code);
}
{
# Two ends of a wire nothing at eval time carries end to end: this is the
# path `swarm_secret_client::queue` resolves for the reader. Both hives
# are asserted, so a publisher that knew one name rather than the roster
# fails here rather than on the second hive ever added to a swarm.
name = "the publisher writes every hive in the roster to that hive's own queue path";
ok =
let
s = secretPublisherHere.systemd.services.swarm-secret-publish.script;
in
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s
&& lib.hasInfix "secret/swarm/hives/h2/queue/agent" s;
}
{
# The doctrine three glue files state, as a property a rewrite has to
# keep: a client is defined by holding a certificate the store accepts,
# never by standing next to the store. Gating this on `deploy.bao.enable`
# would have left the unit rendering only on the one deployment that has
# no use for it.
name = "a publisher holding an identity runs on a host with no store";
ok =
let
m = secretPublisherRemote;
in
!m.services.hyperhive.deploy.bao.enable
&& (m.systemd.services ? swarm-secret-publish)
&& (m.systemd.paths ? swarm-secret-publish);
}
{
# What makes the arm above able to fail. Minting the secrets is not being
# able to publish them: with no certificate the unit would fail a TLS
# handshake on every rotation, so it must not exist at all.
name = "an IdP host with no store identity renders no publisher";
ok =
let
m = secretPublisherNoIdentity;
in
m.services.hyperhive.deploy.authelia.enable
&& !(m.systemd.services ? swarm-secret-publish)
&& !(m.systemd.paths ? swarm-secret-publish);
}
{ {
# A hive's cert-auth role carries the authority by value, so the daemon # A hive's cert-auth role carries the authority by value, so the daemon
# has to be handed the file rather than a path into the store's own # has to be handed the file rather than a path into the store's own

View file

@ -165,11 +165,11 @@ async fn provision(hives: &[String]) -> Result<(), Unreachable> {
Ok(()) Ok(())
} }
/// One hive's policy: its own name, the shared document. /// One hive's policy: its own name, its own document.
async fn write_policy_for(store: &SecretStore, hive: &str) -> Result<()> { async fn write_policy_for(store: &SecretStore, hive: &str) -> Result<()> {
let name = policy::hive_object_name(hive)?; let name = policy::hive_object_name(hive)?;
store store
.write_policy(&name, &policy::render()) .write_policy(&name, &policy::render(hive)?)
.await .await
.with_context(|| format!("writing the read policy {name}"))?; .with_context(|| format!("writing the read policy {name}"))?;
tracing::info!(hive, policy = %name, "hive read policy in place"); tracing::info!(hive, policy = %name, "hive read policy in place");

View file

@ -305,7 +305,7 @@ mod tests {
let request = WriteAclPolicy { let request = WriteAclPolicy {
name: "hive-pr1ma".to_owned(), name: "hive-pr1ma".to_owned(),
policy: crate::policy::render(), policy: crate::policy::render("pr1ma").expect("a plain name is legal"),
}; };
assert_eq!(request.path(), "sys/policies/acl/hive-pr1ma"); assert_eq!(request.path(), "sys/policies/acl/hive-pr1ma");
} }
@ -316,7 +316,7 @@ mod tests {
let request = WriteAclPolicy { let request = WriteAclPolicy {
name: "hive-pr1ma".to_owned(), name: "hive-pr1ma".to_owned(),
policy: crate::policy::render(), policy: crate::policy::render("pr1ma").expect("a plain name is legal"),
}; };
let body = request let body = request
.body() .body()
@ -324,7 +324,10 @@ mod tests {
.expect("a policy write sends one"); .expect("a policy write sends one");
let sent: serde_json::Value = let sent: serde_json::Value =
serde_json::from_slice(&body).expect("the body is the JSON the store parses"); serde_json::from_slice(&body).expect("the body is the JSON the store parses");
assert_eq!(sent["policy"], crate::policy::render()); assert_eq!(
sent["policy"],
crate::policy::render("pr1ma").expect("legal")
);
assert!( assert!(
sent.get("name").is_none(), sent.get("name").is_none(),
"`name` addresses the policy in the path; sending it too would make \ "`name` addresses the policy in the path; sending it too would make \

View file

@ -5,8 +5,8 @@
//! the rules every path obeys ([`path`]), the translation from this //! the rules every path obeys ([`path`]), the translation from this
//! deployment's environment into a logged-in client ([`client`]), and, per kind //! deployment's environment into a logged-in client ([`client`]), and, per kind
//! of secret, the path it lives at together with the fields it holds //! of secret, the path it lives at together with the fields it holds
//! ([`matrix`]). Each of those is a thing the controller and a hive must say //! ([`matrix`], [`queue`]). Each of those is a thing the controller and a hive
//! identically, so it is said once here. //! must say identically, so it is said once here.
//! //!
//! [`policy`] is the same kind of agreement seen from the other side: which of //! [`policy`] is the same kind of agreement seen from the other side: which of
//! those paths a hive's own token may read. It belongs here rather than in the //! those paths a hive's own token may read. It belongs here rather than in the
@ -21,6 +21,7 @@ pub mod client;
pub mod matrix; pub mod matrix;
pub mod path; pub mod path;
pub mod policy; pub mod policy;
pub mod queue;
pub use client::SecretStore; pub use client::SecretStore;

View file

@ -1,16 +1,25 @@
//! The read agreement: which credentials a hive's own token may fetch. //! The read agreement: which credentials a hive's own token may fetch.
//! //!
//! The mirror of [`crate::matrix`]. That module says where a credential lives; //! The mirror of [`crate::matrix`] and [`crate::queue`]. Those modules say
//! this one says who is allowed to read it, and the two have to agree on the //! where a credential lives; this one says who is allowed to read it, and the
//! same path or a delivery fails with a 403 that names nothing. //! two have to agree on the same path or a delivery fails with a 403 that names
//! nothing.
//! //!
//! ⚠️ Every hive gets the same document, and it grants read on **every** //! The document has one stanza per kind a hive reads, and the two are scoped
//! agent's credentials rather than on the ones that hive hosts. That is a //! differently on purpose:
//! decision, not an oversight: an agent's path does not name its hive, so a //!
//! per-hive grant has to be enumerated and re-emitted, and an enumeration that //! ⚠️ The **agent** stanza grants read on *every* agent's credentials rather
//! can silently drift advertises a boundary it does not hold. A wide grant that //! than on the ones that hive hosts. That is a decision, not an oversight: an
//! says so beats a narrow one that only looks narrow. The narrower shapes, and //! agent's path does not name its hive, so a per-hive grant has to be
//! what they would cost, are in `docs/trust-boundary/security.md`. //! enumerated and re-emitted, and an enumeration that can silently drift
//! advertises a boundary it does not hold. A wide grant that says so beats a
//! narrow one that only looks narrow. The narrower shapes, and what they would
//! cost, are in `docs/trust-boundary/security.md`.
//!
//! The **hive** stanza has no such problem and is therefore narrow: that path
//! names its principal, so scoping it to the reader's own name costs nothing
//! and drifts nowhere. Do not widen it to match its neighbour — the asymmetry
//! is the point.
//! //!
//! Rendering stays separate from writing so the text can be asserted with no //! Rendering stays separate from writing so the text can be asserted with no
//! store to talk to. //! store to talk to.
@ -37,25 +46,46 @@ pub fn hive_object_name(hive: &str) -> Result<String, Error> {
Ok(format!("{HIVE_PREFIX}{hive}")) Ok(format!("{HIVE_PREFIX}{hive}"))
} }
/// Render the document every hive's policy holds: read on every agent's /// One read stanza. The only shape this module emits, so "read-only" is a
/// credentials. /// property of the renderer rather than of each call site.
fn read_stanza(path: &str) -> String {
format!("path \"{path}\" {{\n capabilities = [\"read\"]\n}}\n")
}
/// Render `hive`'s policy document: read on every agent's credentials, and on
/// this hive's own.
/// ///
/// Takes no arguments because it depends on nothing — same text for every /// The name is the only input, and it is deploy-time — so the document is
/// hive, unchanged by which agents exist. That is what makes it a deploy-time /// still a deploy-time object rather than derived state with a re-emission to
/// object rather than derived state with a re-emission to get wrong. /// get wrong. Nothing about which agents exist changes the text.
/// ///
/// Read-only: the controller mints these and never reads one back. /// Read-only: the controller mints these and never reads one back.
/// ⚠️ Still the agent kind alone. The other kinds are deliberately absent: a ///
/// hive has no business reading a service's or the controller's credentials, /// ⚠️ The service and controller kinds are deliberately absent: a hive has no
/// and what a hive may read of its *own* kind is a boundary question this /// business reading a service's or the controller's credentials. Adding either
/// module's header answers only for agents. Widening it is a decision, not a /// is a boundary decision, not a consequence of the namespace growing.
/// consequence of the namespace growing. ///
#[must_use] /// The hive's *own* kind is granted, and that is the decision the agent-only
pub fn render() -> String { /// version of this grant said had to be made rather than assumed: a hive holds
format!( /// the queue credential its own agents authenticate with, so it has to read the
"path \"{MOUNT}/data/{ROOT}/{}/*\" {{\n capabilities = [\"read\"]\n}}\n", /// one principal named after itself — and only that one, which is why the path
/// interpolates the name instead of widening to the whole kind.
///
/// # Errors
/// [`Error::PathSegment`] when `hive` holds anything but `[A-Za-z0-9_-]` — the
/// name is interpolated into a policy path, so a name that could close the
/// stanza could grant itself anything.
pub fn render(hive: &str) -> Result<String, Error> {
checked_segment("hive", hive)?;
let agents = read_stanza(&format!(
"{MOUNT}/data/{ROOT}/{}/*",
<&str>::from(Kind::Agent) <&str>::from(Kind::Agent)
) ));
let own = read_stanza(&format!(
"{MOUNT}/data/{ROOT}/{}/{hive}/*",
<&str>::from(Kind::Hive)
));
Ok(format!("{agents}{own}"))
} }
#[cfg(test)] #[cfg(test)]
@ -63,19 +93,19 @@ mod tests {
use super::*; use super::*;
#[test] #[test]
fn the_document_grants_read_over_the_whole_agent_prefix() { fn the_document_grants_the_whole_agent_prefix_and_this_hive_alone() {
assert_eq!( assert_eq!(
render(), render("pr1ma").expect("a plain name is legal"),
"path \"secret/data/swarm/agents/*\" {\n capabilities = [\"read\"]\n}\n" "path \"secret/data/swarm/agents/*\" {\n capabilities = [\"read\"]\n}\n\
path \"secret/data/swarm/hives/pr1ma/*\" {\n capabilities = [\"read\"]\n}\n"
); );
} }
#[test] #[test]
fn the_grant_is_read_only() { fn the_grant_is_read_only() {
// The half of the old policy that survives the widening: a hive reads // A hive reads credentials; a hive that could write one could hand
// credentials, and a hive that could write one could hand itself an // itself an agent's identity.
// agent's identity. let p = render("pr1ma").expect("legal");
let p = render();
assert!(!p.contains("create")); assert!(!p.contains("create"));
assert!(!p.contains("update")); assert!(!p.contains("update"));
assert!(!p.contains("delete")); assert!(!p.contains("delete"));
@ -83,17 +113,36 @@ mod tests {
} }
#[test] #[test]
fn no_name_reaches_the_document_at_all() { fn one_hives_document_does_not_reach_another_hives_path() {
// Why the injection cases that used to live here are gone rather than // Replaces `every_hive_gets_a_byte_identical_document`: the hive stanza
// relaxed: nothing interpolates into the text any more, so there is no // is per-reader now, so identical text is no longer the property. This
// stanza for a name to close. `hive_object_name` still validates, // is what that test was protecting — that a document says only what the
// because a name does reach the policy's *identifier*. // deploy-time name puts in it.
let p = render(); let a = render("alpha").expect("legal");
assert!(!p.contains("pr1ma")); assert!(a.contains("swarm/hives/alpha/*"));
assert_eq!(p.matches("path \"").count(), 1, "one stanza, no per-agent"); assert!(!a.contains("beta"));
assert!(!a.contains("swarm/hives/*"), "the hive stanza stays narrow");
}
#[test]
fn the_same_name_still_renders_byte_identically() {
// The half of the old property that survives: the text is a function of
// the deploy-time name alone, so a re-emission cannot drift.
assert_eq!(
render("pr1ma").expect("legal"),
render("pr1ma").expect("legal")
);
}
#[test]
fn a_name_that_could_close_the_stanza_is_refused() {
// The name reaches the document now, which it did not before — so the
// injection case is live again in the policy TEXT, not just in the
// policy's identifier.
assert!(render("alpha/*\" { capabilities = [\"root\"] }").is_err());
assert!( assert!(
hive_object_name("atlas/*\" { capabilities = [\"root\"] }").is_err(), hive_object_name("atlas/*\" { capabilities = [\"root\"] }").is_err(),
"the object NAME is still a place a name can do damage" "the object NAME is a place a name can do damage too"
); );
} }
@ -102,13 +151,7 @@ mod tests {
// The control for the case above: if every name were refused, that // The control for the case above: if every name were refused, that
// assertion would pass while proving nothing. // assertion would pass while proving nothing.
assert!(hive_object_name("a-b_C9").is_ok()); assert!(hive_object_name("a-b_C9").is_ok());
} assert!(render("a-b_C9").is_ok());
#[test]
fn every_hive_gets_a_byte_identical_document() {
// The property the deploy-time write depends on: nothing about a hive
// or its agents changes the text, so there is nothing to re-emit.
assert_eq!(render(), render());
} }
#[test] #[test]

View file

@ -0,0 +1,100 @@
//! The queue agreement: where a hive's agent-container credential lives in the
//! store, and what the object at that path holds.
//!
//! The sibling of [`crate::matrix`], and it differs from it in one way worth
//! reading before using either: a matrix credential is keyed per **agent**,
//! this one per **hive**. Agents are created at runtime, so the queue
//! identity they present is minted once per hive at deploy time and says which
//! hive an agent belongs to, never which agent.
use serde::{Deserialize, Serialize};
use crate::{
Error,
path::{Kind, principal_prefix},
};
/// The path holding the client secret that agent containers on `hive` present
/// to the swarm queue.
///
/// # Errors
/// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`,
/// which is what keeps one hive's name from addressing another hive's secret.
pub fn agent_client_path(hive: &str) -> Result<String, Error> {
let prefix = principal_prefix(Kind::Hive, hive)?;
Ok(format!("{prefix}/queue/agent"))
}
/// What the path holds: the client secret, plus the client id it belongs to.
///
/// The id rides with the secret for the same reason the homeserver rides with
/// a matrix token — a credential has to be reconstructable from the store
/// alone. Deriving it on the reading side instead would mean spelling
/// `hive-<name>-agent` in a second place, and the authelia module's own option
/// says what a split spelling costs: every agent is denied as a timeout.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Credential {
/// The secret itself. Named to match [`crate::matrix::Credential::value`]
/// so a nix-side reader spells `bao kv get -field=value` for either kind.
pub value: String,
/// The OIDC client id the secret authenticates.
///
/// Required, unlike [`crate::matrix::Credential::homeserver`]: that field
/// is optional because it was added to objects already in the store, and
/// this path is new. An object without an id is not a usable credential,
/// so failing to decode is the wanted behaviour — the alternative is a
/// reader holding `None` whose only recourse is to spell the id itself.
pub client_id: String,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_hive_name_lands_under_its_own_principal_prefix() {
assert_eq!(
agent_client_path("alpha").expect("a plain name is legal"),
"swarm/hives/alpha/queue/agent"
);
}
#[test]
fn a_traversal_in_the_hive_name_is_refused() {
let e = agent_client_path("../beta").expect_err("a traversal is not");
assert!(matches!(e, Error::PathSegment { kind: "hive", .. }), "{e}");
}
#[test]
fn two_hives_never_share_a_path() {
assert_ne!(
agent_client_path("alpha").expect("legal"),
agent_client_path("beta").expect("legal")
);
}
#[test]
fn the_object_round_trips_through_the_store_representation() {
let c = Credential {
value: "s3cr3t".to_owned(),
client_id: "hive-alpha-agent".to_owned(),
};
let json = serde_json::to_string(&c).expect("serialises");
assert_eq!(
serde_json::from_str::<Credential>(&json).expect("deserialises"),
c
);
}
#[test]
fn the_field_names_the_nix_reader_asks_for_are_the_ones_written() {
let json = serde_json::to_value(Credential {
value: "s3cr3t".to_owned(),
client_id: "hive-alpha-agent".to_owned(),
})
.expect("serialises");
assert_eq!(json["value"], "s3cr3t");
assert_eq!(json["client_id"], "hive-alpha-agent");
}
}