swarm-bao: write the first grant from the host, not the container
`swarm-bao-controller-policy` creates the `swarm-controller` policy and cert-auth role — the credential every hive logs in with. It has never succeeded on any deployment, and the reason is where it ran. Inside the container it had neither of the two things the store demands. Its `BAO_ADDR` was the public DNS name, which from that netns resolves to the hive bridge: `dial tcp 10.42.0.1:8200: connect: connection refused`. And every API listener carries `tls_require_and_verify_client_cert`, while `tlsDir` holds the server's leaf and the CA that signs clients — no client identity at all. Fixing only the address moves the failure one hop. The comment above the unit asserted the opposite — that in there the store is "reachable without a client certificate at all, which is the point". The listener config decides that, and says otherwise. That belief is what put the unit in the container, so it goes with it. On the host all four coordinates already exist: `baoCli` carries the address, the CA, the certificate and the key, so the unit needs no `environment` block at all. `bootstrapTokenFile` was always a host path — the container only saw it through a bind mount. Nothing new crosses the boundary; the mount gets no wider. The retry bound is resized with it. 10 attempts at 30s is five minutes, and under `seal = "shamir"` an operator unseals by hand, so it would give up before a human arrived — permanently, because `start-limit-hit` does not self-heal. That is the same silent no-bootstrap this issue is about. 2880 × 30s covers a day, inside a 25h window. module-eval follows the unit to the host and gains an arm asserting it is NOT rendered inside the container: the move is the fix, so the side it landed on is worth pinning.
This commit is contained in:
atlas
mara
parent
20da007351
commit
16182c670e
2 changed files with 106 additions and 95 deletions
|
|
@ -856,6 +856,88 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
# The swarm's first grant, written from the HOST. Every API listener sets
|
||||
# `tls_require_and_verify_client_cert`, so a client needs an identity
|
||||
# wherever it runs — and only the host has one. The bootstrap token is a
|
||||
# host path too; the container saw it through a bind mount.
|
||||
systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken {
|
||||
description = "write the swarm controller's bao policy and cert-auth role";
|
||||
after = [ "container@${cfg.machine}.service" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
# The wrapper rather than the package: it carries the address, the CA
|
||||
# and this host's certificate, which is what makes running here cheaper
|
||||
# than shipping an identity the other way.
|
||||
path = [
|
||||
baoCli
|
||||
pkgs.coreutils
|
||||
];
|
||||
# Named but not placed is a legitimate state: all-local supplies the
|
||||
# path as a default and the operator drops the file there after
|
||||
# `bao operator init`. Skipping rather than failing is also what makes
|
||||
# deleting the token at the end of that procedure safe.
|
||||
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||
# A store that is up is not necessarily unsealed — under
|
||||
# `seal = "shamir"` an operator unseals BY HAND, so early attempts fail
|
||||
# for as long as that takes, which can be a day.
|
||||
#
|
||||
# `StartLimit*` are `[Unit]` settings that systemd ignores under
|
||||
# `[Service]`, which is what these two options render to. The window has
|
||||
# to exceed `RestartSec × burst`, or it closes between attempts and the
|
||||
# burst is never reached: 2880 × 30s is 24h, inside a 25h window.
|
||||
# Bounding it tighter would give up before a human could unseal, and
|
||||
# `start-limit-hit` does not self-heal.
|
||||
startLimitBurst = 2880;
|
||||
startLimitIntervalSec = 90000;
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 30;
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||
export BAO_TOKEN
|
||||
|
||||
# Idempotent on purpose: a rebuild re-asserts the policy rather
|
||||
# than failing on one that already exists.
|
||||
printf '%s' ${lib.escapeShellArg controllerPolicyText} |
|
||||
bao policy write ${lib.escapeShellArg controllerPolicyName} -
|
||||
''
|
||||
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
|
||||
|
||||
# The policy above grants paths under `auth/cert/`, and nothing
|
||||
# in this tree creates that mount. Without this, the grant names
|
||||
# a location that does not exist and every certificate login
|
||||
# fails — the controller's own, and the per-hive ones it later
|
||||
# issues against the same mount.
|
||||
#
|
||||
# Asked rather than attempted: `auth enable` errors on a mount
|
||||
# that already exists, and recognising that would tie a rebuild
|
||||
# to an error string we have never seen this store emit.
|
||||
mounted="$(bao auth list -format=json)"
|
||||
case "$mounted" in
|
||||
*'"cert/"'*) ;;
|
||||
*) bao auth enable cert ;;
|
||||
esac
|
||||
|
||||
# `certificate=` is the CA, so this role trusts every leaf that
|
||||
# CA signed and `allowed_common_names` is the whole of what
|
||||
# narrows it to one identity. ⚠️ The same CA signs each hive's
|
||||
# reader leaf with CN = the hive's name, so a hive named
|
||||
# `${controllerCn}` would satisfy this role.
|
||||
#
|
||||
# Named outside the `hive-*` namespace the policy grants, so the
|
||||
# controller cannot rewrite the role that constrains it.
|
||||
bao write auth/cert/certs/${lib.escapeShellArg controllerPolicyName} \
|
||||
certificate=@${tlsDir}/client-ca.pem \
|
||||
allowed_common_names=${lib.escapeShellArg controllerCn} \
|
||||
token_policies=${lib.escapeShellArg controllerPolicyName} \
|
||||
display_name=${lib.escapeShellArg controllerCn}
|
||||
'';
|
||||
};
|
||||
|
||||
containers.${cfg.machine} = {
|
||||
autoStart = true;
|
||||
ephemeral = false;
|
||||
|
|
@ -1076,84 +1158,6 @@ in
|
|||
# Restarting is an operator action with an unseal on the far side of
|
||||
# it, which is why nothing here tries to be clever about it.
|
||||
|
||||
# The swarm's first grant, written from in here because this is
|
||||
# where the store is reachable without a client certificate — which
|
||||
# is the point, since no role exists yet to issue one against.
|
||||
systemd.services.swarm-bao-controller-policy = lib.mkIf haveBootstrapToken {
|
||||
description = "write the swarm controller's bao policy and cert-auth role";
|
||||
after = [ "openbao.service" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
path = [
|
||||
baoDeploy.package
|
||||
pkgs.coreutils
|
||||
];
|
||||
# Named but not placed is a legitimate state: all-local supplies
|
||||
# the path as a default and the operator drops the file there
|
||||
# after `bao operator init`. Skipping rather than failing is also
|
||||
# what makes deleting the token at the end of that procedure safe.
|
||||
unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile;
|
||||
# A store that is up is not necessarily unsealed — under
|
||||
# `seal = "shamir"` an operator unseals by hand after every
|
||||
# restart — so early attempts legitimately fail. Bounded,
|
||||
# because a token that is wrong rather than early would
|
||||
# otherwise retry forever.
|
||||
#
|
||||
# `StartLimit*` are `[Unit]` settings that systemd ignores under
|
||||
# `[Service]`, which is what these two options render to. The
|
||||
# interval also has to exceed `RestartSec × burst`, or the window
|
||||
# closes between attempts and the burst is never reached.
|
||||
startLimitBurst = 10;
|
||||
startLimitIntervalSec = 600;
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 30;
|
||||
};
|
||||
environment.BAO_ADDR = "https://${cfg.domain}:${toString cfg.port}";
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})"
|
||||
export BAO_TOKEN
|
||||
|
||||
# Idempotent on purpose: a rebuild re-asserts the policy rather
|
||||
# than failing on one that already exists.
|
||||
printf '%s' ${lib.escapeShellArg controllerPolicyText} |
|
||||
bao policy write ${lib.escapeShellArg controllerPolicyName} -
|
||||
''
|
||||
+ lib.optionalString (baoDeploy.clientCaFile != null) ''
|
||||
|
||||
# The policy above grants paths under `auth/cert/`, and nothing
|
||||
# in this tree creates that mount. Without this, the grant names
|
||||
# a location that does not exist and every certificate login
|
||||
# fails — the controller's own, and the per-hive ones it later
|
||||
# issues against the same mount.
|
||||
#
|
||||
# Asked rather than attempted: `auth enable` errors on a mount
|
||||
# that already exists, and recognising that would tie a rebuild
|
||||
# to an error string we have never seen this store emit.
|
||||
mounted="$(bao auth list -format=json)"
|
||||
case "$mounted" in
|
||||
*'"cert/"'*) ;;
|
||||
*) bao auth enable cert ;;
|
||||
esac
|
||||
|
||||
# `certificate=` is the CA, so this role trusts every leaf that
|
||||
# CA signed and `allowed_common_names` is the whole of what
|
||||
# narrows it to one identity. ⚠️ The same CA signs each hive's
|
||||
# reader leaf with CN = the hive's name, so a hive named
|
||||
# `${controllerCn}` would satisfy this role.
|
||||
#
|
||||
# Named outside the `hive-*` namespace the policy grants, so the
|
||||
# controller cannot rewrite the role that constrains it.
|
||||
bao write auth/cert/certs/${lib.escapeShellArg controllerPolicyName} \
|
||||
certificate=@${tlsDir}/client-ca.pem \
|
||||
allowed_common_names=${lib.escapeShellArg controllerCn} \
|
||||
token_policies=${lib.escapeShellArg controllerPolicyName} \
|
||||
display_name=${lib.escapeShellArg controllerCn}
|
||||
'';
|
||||
};
|
||||
};
|
||||
};
|
||||
})
|
||||
|
|
|
|||
Loading…
Reference in a new issue