hive-agent: read the per-agent queue secret from bao in process
The harness now reads swarm/agents/<agent>/queue from the store itself, under the agent's own store certificate, and holds it in memory only. It reads once before the first connect and again on every reconnect attempt (async-nats `ConnectOptions::with_auth_callback`), so an agent whose secret was re-minted reconnects with the new value instead of being refused until the container restarts. hive-agent-queue-credential.service, the /run file it wrote, and HIVE_AGENT_QUEUE_AGENT_SECRET_FILE are gone; queue-identity.nix now hands hive-agent.service the store address, its certificate paths and the agent name. A failed or empty read before the first connect still falls back to the hive's shared client. Each read is bounded by a 10s timeout, and retries wait out the existing reconnect backoff (500ms doubling, capped at 60s). Closes #4783
This commit is contained in:
atlas
mara
parent
f7437a4773
commit
ccb5bd3b38
10 changed files with 770 additions and 529 deletions
|
|
@ -5,15 +5,14 @@
|
|||
# to `swarm/agents/<agent>/forge-token`
|
||||
# (`swarm_secret_client::forge::agent_token_path`); no hive is in that chain.
|
||||
# This unit logs in to the store with the certificate ./bao.nix already proves
|
||||
# it can log in with, and reads its own path. The shape is ./queue-identity.nix's,
|
||||
# for the same reason: a hive handing the token over would be the hive reading
|
||||
# a secret on the agent's behalf.
|
||||
# it can log in with, and reads its own path. A hive handing the token over
|
||||
# would be the hive reading a secret on the agent's behalf.
|
||||
#
|
||||
# Unlike the queue credential this one rotates: the controller replaces the
|
||||
# token when the forge's copy stops matching the stored one. So the unit is
|
||||
# re-run by a timer, and it swaps the file in by rename, only when the value
|
||||
# changed, so a reader never sees half a token and a watcher on the file
|
||||
# (forge-avatar-sync.path) fires only on a real change.
|
||||
# The token rotates: the controller replaces it when the forge's copy stops
|
||||
# matching the stored one. So the unit is re-run by a timer, and it swaps the
|
||||
# file in by rename, only when the value changed, so a reader never sees half
|
||||
# a token and a watcher on the file (forge-avatar-sync.path) fires only on a
|
||||
# real change.
|
||||
#
|
||||
# Consumers read `services.hyperhive.agent.forge.tokenFile` first and fall back
|
||||
# to `<state>/forge-token`, the file the hive wrote before this existed.
|
||||
|
|
@ -79,7 +78,9 @@ in
|
|||
description = "fetch this agent's own forge token from the secret store";
|
||||
after = [
|
||||
"network.target"
|
||||
# Ordering only, for the reason ./queue-identity.nix gives.
|
||||
# Ordering only: that unit reports a store this agent cannot reach as
|
||||
# itself, and should get to say so before this one reports a path it
|
||||
# could not read.
|
||||
"hive-agent-bao-identity.service"
|
||||
];
|
||||
before = [ "hive-forge-notify.service" ];
|
||||
|
|
@ -93,10 +94,10 @@ in
|
|||
startLimitIntervalSec = 300;
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
# Not `RemainAfterExit`, unlike the queue fetch: the timer below has to
|
||||
# be able to start this unit again, and an active unit cannot be
|
||||
# started. `RuntimeDirectoryPreserve` is what keeps the directory, and
|
||||
# the token in it, alive between runs instead.
|
||||
# Not `RemainAfterExit`: the timer below has to be able to start this
|
||||
# unit again, and an active unit cannot be started.
|
||||
# `RuntimeDirectoryPreserve` is what keeps the directory, and the token
|
||||
# in it, alive between runs instead.
|
||||
RemainAfterExit = false;
|
||||
TimeoutStartSec = 30;
|
||||
Restart = "on-failure";
|
||||
|
|
@ -167,8 +168,8 @@ in
|
|||
fi
|
||||
export BAO_TOKEN
|
||||
|
||||
# 🩸 Degrades rather than fails, for the reason ./queue-identity.nix
|
||||
# gives: the policy stanza that let the login read `bao-mtls` covers
|
||||
# 🩸 Degrades where ./bao.nix's check fails, because one policy stanza
|
||||
# governs both reads: the one that let the login read `bao-mtls` covers
|
||||
# this path too, so a refusal here is a token not minted yet. The
|
||||
# file already in place, if any, is kept: a store that is briefly
|
||||
# unreachable must not take a working token away.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
# This agent's own credential for the swarm queue, fetched from the store by
|
||||
# the agent itself.
|
||||
# This agent's own credential for the swarm queue, read from the store by the
|
||||
# harness itself.
|
||||
#
|
||||
# The credential beside it in ./queue.nix is keyed per **hive**: one OIDC
|
||||
# client minted at deploy time and handed to every agent container on the
|
||||
|
|
@ -9,17 +9,21 @@
|
|||
# (`swarm_secret_client::queue::agent_queue_path`), at swarm level, with no
|
||||
# hive anywhere in the chain.
|
||||
#
|
||||
# That is also why this unit *fetches* rather than being handed a credential:
|
||||
# a hive courier in the path would be the hive vouching for which agent this
|
||||
# is, which is the property the per-agent credential exists to remove. The
|
||||
# agent authenticates to the store as itself, with the certificate ./bao.nix
|
||||
# already proves it can log in with, and reads its own path.
|
||||
# The agent authenticates to the store as itself, with the certificate
|
||||
# ./bao.nix already proves it can log in with, and reads its own path. A hive
|
||||
# courier in the path would be the hive vouching for which agent this is,
|
||||
# which is the property the per-agent credential exists to remove.
|
||||
#
|
||||
# Nothing here writes the secret anywhere. The harness reads it into memory
|
||||
# before its first queue connection and again on every reconnect attempt
|
||||
# (`hive-agent`'s `swarm_queue`), so a re-minted secret is picked up without a
|
||||
# restart. What this module hands the harness is the store's address and the
|
||||
# paths of its certificate.
|
||||
#
|
||||
# ⛔ The store certificate is for reaching the store and nothing else. It is
|
||||
# never presented to the queue: what goes to the queue is the secret read
|
||||
# back from this path.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
|
|
@ -29,32 +33,17 @@ let
|
|||
|
||||
# This container's agent name — the same string `swarm-controller` minted
|
||||
# the credential under, because the agent's unix user is named for the
|
||||
# agent (see ./user.nix).
|
||||
# agent (see ./user.nix). The harness builds the path and the cert-auth
|
||||
# role from it.
|
||||
agentName = config.services.hyperhive.agent.user.name;
|
||||
|
||||
# The same three ids ./bao.nix loads. Both units present the same
|
||||
# certificate because there is one store identity per agent; the ids are
|
||||
# `hive_c0re::lifecycle::agent_identity`'s and a rename is a rename there
|
||||
# too.
|
||||
# The same three ids ./bao.nix loads, because there is one store identity
|
||||
# per agent; the ids are `hive_c0re::lifecycle::agent_identity`'s and a
|
||||
# rename is a rename there too.
|
||||
certCredential = "hive-agent-bao-cert";
|
||||
keyCredential = "hive-agent-bao-key";
|
||||
serverCaCredential = "hive-agent-bao-server-ca";
|
||||
|
||||
unitName = "hive-agent-queue-credential";
|
||||
|
||||
# The nix half of `swarm_secret_client::queue::agent_queue_path` plus
|
||||
# `path::MOUNT`, spelled exactly as ./bao.nix spells its own sibling path.
|
||||
queuePath = "secret/swarm/agents/${agentName}/queue";
|
||||
|
||||
# `RuntimeDirectory=` under the unit's own `User=`, so the file is owned by
|
||||
# the agent and readable by the harness without a mode change. /run and not
|
||||
# the state dir on purpose: a secret fetched at boot has no business
|
||||
# surviving one.
|
||||
runtimeDir = "${unitName}";
|
||||
secretFile = "/run/${runtimeDir}/secret";
|
||||
# bao's stderr, in the unit's own `0700` directory rather than `/tmp`.
|
||||
errFile = "/run/${runtimeDir}/bao.err";
|
||||
|
||||
# The store's address is the whole switch, exactly as in ./bao.nix — and
|
||||
# deliberately *not* the queue coordinates in ./queue.nix. Those are the
|
||||
# hive's, and gating a swarm-minted per-agent credential on a hive-level
|
||||
|
|
@ -63,167 +52,27 @@ let
|
|||
configured = cfg.addr != null;
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.agent.queue.agentSecretFile = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
readOnly = true;
|
||||
default = secretFile;
|
||||
description = ''
|
||||
Path this agent's own swarm-queue secret is fetched to, for a consumer
|
||||
outside the harness unit. Read-only for the same reason
|
||||
{option}`services.hyperhive.agent.queue.clientSecretFile` is: it is a
|
||||
fact about where the fetch writes, not a knob.
|
||||
|
||||
🩸 A PATH and never a value. The file is `0400` to the agent user and
|
||||
is read at the moment it is needed; nothing in this tree puts its
|
||||
contents in an environment variable, where `/proc/<pid>/environ` would
|
||||
publish them to every process in the container.
|
||||
|
||||
The file exists only once the swarm has minted a credential for this
|
||||
agent. An agent created before its swarm did so has none, and
|
||||
`${unitName}.service` says so in the journal rather than failing —
|
||||
see that unit.
|
||||
'';
|
||||
};
|
||||
|
||||
config = lib.mkIf configured {
|
||||
systemd.services.${unitName} = {
|
||||
description = "fetch this agent's own swarm-queue credential from the secret store";
|
||||
after = [
|
||||
"network.target"
|
||||
# Ordering only, not a requirement: that unit is the one that reports
|
||||
# a store this agent cannot reach as itself, and it should get to say
|
||||
# so before this one reports a path it could not read.
|
||||
"hive-agent-bao-identity.service"
|
||||
systemd.services.hive-agent = {
|
||||
# Bare ids, no paths: the terse `LoadCredential=` form that inherits a
|
||||
# credential the service *manager* received, which is what the container
|
||||
# manager passed in. ./bao.nix and ./queue.nix state the same shape.
|
||||
serviceConfig.LoadCredential = [
|
||||
certCredential
|
||||
keyCredential
|
||||
serverCaCredential
|
||||
];
|
||||
before = [ "hive-agent.service" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
path = [
|
||||
pkgs.openbao
|
||||
pkgs.coreutils
|
||||
];
|
||||
# Same sizing and the same `[Unit]`-not-`[Service]` placement as
|
||||
# ./bao.nix's check: a few short attempts cover a store that comes up
|
||||
# alongside this container, and a longer window only delays the report.
|
||||
startLimitBurst = 4;
|
||||
startLimitIntervalSec = 300;
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
# Keeps the unit active, which is what keeps `RuntimeDirectory=`
|
||||
# from being removed out from under the harness.
|
||||
RemainAfterExit = true;
|
||||
TimeoutStartSec = 30;
|
||||
Restart = "on-failure";
|
||||
RestartSec = 15;
|
||||
User = agentName;
|
||||
Group = agentName;
|
||||
RuntimeDirectory = runtimeDir;
|
||||
RuntimeDirectoryMode = "0700";
|
||||
UMask = "0377";
|
||||
# Bare ids, no paths: the terse `LoadCredential=` form that inherits
|
||||
# a credential the service *manager* received, which is what the
|
||||
# container manager passed in. ./bao.nix and ./queue.nix state the
|
||||
# same shape.
|
||||
LoadCredential = [
|
||||
certCredential
|
||||
keyCredential
|
||||
serverCaCredential
|
||||
];
|
||||
};
|
||||
# Paths and a name only. `%d` is this unit's own credentials directory.
|
||||
environment = {
|
||||
HIVE_AGENT_NAME = agentName;
|
||||
BAO_ADDR = cfg.addr;
|
||||
# `%d` is `$CREDENTIALS_DIRECTORY`, per-unit and owned by `User=`.
|
||||
BAO_CLIENT_CERT = "%d/${certCredential}";
|
||||
BAO_CLIENT_KEY = "%d/${keyCredential}";
|
||||
# Named even when no CA was delivered: a bare `LoadCredential=` is
|
||||
# non-fatal when absent, and the harness treats a missing or empty file
|
||||
# as "use the container's own trust store".
|
||||
BAO_CACERT = "%d/${serverCaCredential}";
|
||||
};
|
||||
script = ''
|
||||
set -euo pipefail
|
||||
|
||||
# No identity delivered at all. ./bao.nix's check reports this as the
|
||||
# failure it is; there is nothing for this unit to add, and failing
|
||||
# here too would be the same cause stated twice.
|
||||
for id in ${lib.escapeShellArg certCredential} ${lib.escapeShellArg keyCredential}; do
|
||||
if [ ! -s "$CREDENTIALS_DIRECTORY/$id" ]; then
|
||||
echo "this agent has no store identity, so it cannot fetch its own queue credential." >&2
|
||||
exit 0
|
||||
fi
|
||||
if [ ! -r "$CREDENTIALS_DIRECTORY/$id" ]; then
|
||||
echo "cannot read $CREDENTIALS_DIRECTORY/$id, so this agent cannot present its store identity." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Only when one was delivered — absent means verify the store's
|
||||
# listener against the container's own trust store, which is what a
|
||||
# deployment with a real CA wants. ./bao.nix says the same.
|
||||
if [ -s "$CREDENTIALS_DIRECTORY/${serverCaCredential}" ]; then
|
||||
export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}"
|
||||
fi
|
||||
|
||||
# `UMask=0377` makes every file this script creates `0400`, so only
|
||||
# the redirect that creates a file can write to it: `$err` is removed
|
||||
# before each redirect into it.
|
||||
err=${lib.escapeShellArg errFile}
|
||||
trap 'rm -f "$err"' EXIT
|
||||
|
||||
# Cert auth is a login, not a transport setting: the `BAO_CLIENT_*`
|
||||
# variables above only pick the certificate the handshake presents.
|
||||
# `-token-only` answers on stdout and skips the token helper, which
|
||||
# is a `sh` this unit's `path` does not carry.
|
||||
rm -f "$err"
|
||||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||||
# The redirect creates `$err` before bao starts, so no file means
|
||||
# bao never ran. bao prints `Code: <status>` only for an HTTP
|
||||
# answer, and `remote error: tls:` only for an alert the store sent.
|
||||
re='Code: ([0-9]{3})'
|
||||
if [ ! -e "$err" ]; then
|
||||
echo "could not create $err, so bao never ran and the store at $BAO_ADDR was not asked." >&2
|
||||
elif [[ "$(<"$err")" =~ $re ]]; then
|
||||
case "''${BASH_REMATCH[1]}" in
|
||||
4*) echo "the swarm secret store at $BAO_ADDR refused this agent's certificate login with HTTP ''${BASH_REMATCH[1]}:" >&2 ;;
|
||||
*) echo "the swarm secret store at $BAO_ADDR failed this agent's certificate login with HTTP ''${BASH_REMATCH[1]}:" >&2 ;;
|
||||
esac
|
||||
elif [[ "$(<"$err")" == *"remote error: tls:"* ]]; then
|
||||
echo "the swarm secret store at $BAO_ADDR refused this agent's certificate in the TLS handshake:" >&2
|
||||
else
|
||||
echo "bao got no answer from the swarm secret store at $BAO_ADDR (network, DNS, or TLS on this side):" >&2
|
||||
fi
|
||||
if [ -s "$err" ]; then cat "$err" >&2; fi
|
||||
exit 1
|
||||
fi
|
||||
export BAO_TOKEN
|
||||
|
||||
# 🩸 Degrades where ./bao.nix's check fails, and the reason is that
|
||||
# the two reads are governed by the *same* policy stanza:
|
||||
# `swarm_secret_client::policy::render_agent` grants read on
|
||||
# `secret/data/swarm/agents/<agent>/*`, which covers this path and
|
||||
# the `bao-mtls` one beside it alike. So a refusal this unit sees and
|
||||
# that check did not cannot be a policy that drifted — it is an
|
||||
# object that has not been minted, which is the ordinary state of
|
||||
# every agent created before its swarm knew to mint one. A unit that
|
||||
# failed at every boot over that would be loud about a deployment
|
||||
# doing nothing wrong.
|
||||
#
|
||||
# ⚠️ Written by redirect into the runtime directory, never echoed:
|
||||
# the field is the secret itself.
|
||||
rm -f "$err"
|
||||
if ! bao kv get -field=value ${lib.escapeShellArg queuePath} > ${lib.escapeShellArg secretFile} 2>"$err"; then
|
||||
rm -f ${lib.escapeShellArg secretFile}
|
||||
echo "no per-agent queue credential at ${queuePath} yet; this agent falls back to its hive's shared one." >&2
|
||||
if [ -s "$err" ]; then cat "$err" >&2; fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "fetched this agent's own queue credential from ${queuePath}."
|
||||
'';
|
||||
};
|
||||
|
||||
# The harness reads a path and never a value, the same shape ./queue.nix
|
||||
# hands it the hive-scoped secret in. Not `%d` here: this credential is
|
||||
# not a systemd credential at all — it is a file this container fetched
|
||||
# for itself, which is the whole point.
|
||||
systemd.services.hive-agent = {
|
||||
after = [ "${unitName}.service" ];
|
||||
environment.HIVE_AGENT_QUEUE_AGENT_SECRET_FILE = secretFile;
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -40,8 +40,15 @@ let
|
|||
|
||||
agentNoBao = agentWith { };
|
||||
|
||||
# Both at once, for the property that only exists when both are configured:
|
||||
# the store certificate never reaches a queue variable.
|
||||
agentQueueBao = agentWith {
|
||||
hyperhive.queue.natsUrl = "nats://10.42.0.1:4222";
|
||||
hyperhive.queue.tokenEndpoint = "https://auth.t.local/api/oidc/token";
|
||||
services.hyperhive.agent.bao.addr = "https://bao.t.local:8200";
|
||||
};
|
||||
|
||||
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
|
||||
agentQueueCredential = machine: machine.systemd.services.hive-agent-queue-credential;
|
||||
cases = [
|
||||
{
|
||||
# Both ids or neither: the secret authenticates nobody without the id it
|
||||
|
|
@ -166,91 +173,62 @@ let
|
|||
ok = !(agentNoBao.systemd.services ? hive-agent-bao-identity);
|
||||
}
|
||||
{
|
||||
# The per-agent credential's fetch rides on the same store identity the
|
||||
# check above proves, because there is one identity per agent. A fetch
|
||||
# unit loading different ids would be a second certificate nothing
|
||||
# mints.
|
||||
name = "the queue-credential fetch presents the agent's own store identity";
|
||||
# The harness reads its per-agent queue secret from the store itself,
|
||||
# under the same store identity the check above proves, because there is
|
||||
# one identity per agent.
|
||||
name = "the harness presents the agent's own store identity";
|
||||
ok =
|
||||
let
|
||||
u = agentQueueCredential agentBao;
|
||||
u = agentHarness agentBao;
|
||||
in
|
||||
builtins.elem "hive-agent-bao-cert" u.serviceConfig.LoadCredential
|
||||
&& builtins.elem "hive-agent-bao-key" u.serviceConfig.LoadCredential
|
||||
&& u.environment.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
|
||||
&& u.environment.BAO_CLIENT_KEY == "%d/hive-agent-bao-key";
|
||||
&& u.environment.BAO_CLIENT_KEY == "%d/hive-agent-bao-key"
|
||||
&& u.environment.BAO_ADDR == agentBao.services.hyperhive.agent.bao.addr;
|
||||
}
|
||||
{
|
||||
# Same 403-not-a-miss reason as every other reader here: the path
|
||||
# `swarm_secret_client::queue::agent_queue_path` builds is the one this
|
||||
# agent's own policy stanza covers. Built from the agent's own name,
|
||||
# because the name is what makes it this agent's credential and not a
|
||||
# neighbour's — which is the entire point of minting one per agent.
|
||||
name = "the queue-credential fetch reads the agent's own per-agent path";
|
||||
# `swarm_secret_client` builds both the store path and the cert-auth role
|
||||
# from this name, so it has to be the name the credential was minted
|
||||
# under.
|
||||
name = "the harness is told the agent name its queue secret is minted under";
|
||||
ok =
|
||||
let
|
||||
m = agentBao;
|
||||
name = m.services.hyperhive.agent.user.name;
|
||||
in
|
||||
lib.hasInfix "secret/swarm/agents/${name}/queue" (agentQueueCredential m).script;
|
||||
(agentHarness agentBao).environment.HIVE_AGENT_NAME == agentBao.services.hyperhive.agent.user.name;
|
||||
}
|
||||
{
|
||||
# The per-agent queue secret lives in the harness's memory only: no unit
|
||||
# fetches it to a file, and the harness is handed no file to read it
|
||||
# from.
|
||||
name = "no unit writes the per-agent queue secret to disk";
|
||||
ok =
|
||||
!(agentBao.systemd.services ? hive-agent-queue-credential)
|
||||
&& !((agentHarness agentBao).environment ? HIVE_AGENT_QUEUE_AGENT_SECRET_FILE);
|
||||
}
|
||||
{
|
||||
# ⛔ The store certificate authenticates against the store and nothing
|
||||
# else. It must never reach the queue, so nothing in this unit may hand
|
||||
# a `BAO_CLIENT_*` path to anything queue-shaped.
|
||||
name = "the queue-credential fetch never points the queue at the store certificate";
|
||||
# else. With both the store and the queue configured, no queue variable
|
||||
# may name one of its paths.
|
||||
name = "the harness never points the queue at the store certificate";
|
||||
ok =
|
||||
let
|
||||
u = agentQueueCredential agentBao;
|
||||
harness = agentHarness agentBao;
|
||||
e = (agentHarness agentQueueBao).environment;
|
||||
queueVars = lib.filterAttrs (
|
||||
n: _: lib.hasPrefix "HIVE_AGENT_OIDC_" n || n == "HIVE_AGENT_NATS_URL"
|
||||
) e;
|
||||
in
|
||||
!(lib.hasInfix "nats" u.script)
|
||||
&& !(lib.any (lib.hasPrefix "BAO_") (builtins.attrNames harness.environment));
|
||||
}
|
||||
{
|
||||
# A secret fetched at boot has no business surviving one, and the
|
||||
# directory has to be the unit's own so the file is owned by the agent
|
||||
# rather than needing a mode change. `RemainAfterExit` is what keeps
|
||||
# systemd from removing it out from under the harness.
|
||||
name = "the fetched credential lands in a runtime directory the unit keeps alive";
|
||||
ok =
|
||||
let
|
||||
u = agentQueueCredential agentBao;
|
||||
c = u.serviceConfig;
|
||||
in
|
||||
c.RuntimeDirectory == "hive-agent-queue-credential"
|
||||
&& c.RemainAfterExit
|
||||
&& lib.hasInfix "/run/hive-agent-queue-credential/secret" u.script;
|
||||
}
|
||||
{
|
||||
# 🩸 Degrades where the identity check fails, and the reason is that
|
||||
# both reads are governed by one policy stanza: a refusal this unit
|
||||
# sees and that check did not is an object not yet minted, not a policy
|
||||
# that drifted. An agent created before its swarm minted one is a
|
||||
# deployment doing nothing wrong.
|
||||
name = "a per-agent credential that was never minted does not fail the unit";
|
||||
ok = lib.hasInfix "exit 0" (agentQueueCredential agentBao).script;
|
||||
}
|
||||
{
|
||||
# The harness reads a path and never a value — the same discipline
|
||||
# ../agent-modules/queue.nix keeps for the hive-scoped secret. Not
|
||||
# `%d`: this one is not a systemd credential, it is a file the
|
||||
# container fetched for itself.
|
||||
name = "the harness is handed the fetched credential as a path";
|
||||
ok =
|
||||
let
|
||||
m = agentBao;
|
||||
in
|
||||
(agentHarness m).environment.HIVE_AGENT_QUEUE_AGENT_SECRET_FILE
|
||||
== m.services.hyperhive.agent.queue.agentSecretFile;
|
||||
queueVars != { } && !(lib.any (lib.hasInfix "hive-agent-bao") (builtins.attrValues queueVars));
|
||||
}
|
||||
{
|
||||
# The absence arm. An agent whose swarm gave it no store has nothing to
|
||||
# log in with, so there is nothing to fetch with either — and the
|
||||
# harness is then told no path rather than one that never fills.
|
||||
name = "an agent told no store address fetches no queue credential";
|
||||
# log in with, so the harness is handed no store coordinates at all.
|
||||
name = "an agent told no store address hands the harness no store identity";
|
||||
ok =
|
||||
!(agentNoBao.systemd.services ? hive-agent-queue-credential)
|
||||
&& !((agentHarness agentNoBao).environment ? HIVE_AGENT_QUEUE_AGENT_SECRET_FILE);
|
||||
let
|
||||
u = agentHarness agentNoBao;
|
||||
in
|
||||
!(u.environment ? BAO_ADDR)
|
||||
&& !(u.environment ? HIVE_AGENT_NAME)
|
||||
&& !(builtins.elem "hive-agent-bao-cert" (u.serviceConfig.LoadCredential or [ ]));
|
||||
}
|
||||
];
|
||||
in
|
||||
|
|
|
|||
|
|
@ -1,9 +1,8 @@
|
|||
# `checks.script-test-agent-bao-fetch` — runs the two agent units that log in
|
||||
# to the swarm secret store and fetch a secret, ../agent-modules/forge-token.nix
|
||||
# and ../agent-modules/queue-identity.nix, against a stub `bao`. The cases are
|
||||
# in ./agent-bao-fetch.sh.
|
||||
# `checks.script-test-agent-bao-fetch` — runs the agent unit that logs in to
|
||||
# the swarm secret store and fetches a secret, ../agent-modules/forge-token.nix,
|
||||
# against a stub `bao`. The cases are in ./agent-bao-fetch.sh.
|
||||
#
|
||||
# What runs is each unit's rendered `ExecStart`, on the unit's own `PATH` with
|
||||
# What runs is the unit's rendered `ExecStart`, on the unit's own `PATH` with
|
||||
# the stub in front. The one edit is the unit's `/run/<unit>/` prefix, moved
|
||||
# under the build directory because the sandbox has no writable `/run`.
|
||||
{
|
||||
|
|
@ -71,7 +70,6 @@ in
|
|||
pkgs.runCommand "hyperhive-script-test-agent-bao-fetch"
|
||||
(
|
||||
unitEnv "FORGE" "hive-agent-forge-token"
|
||||
// unitEnv "QUEUE" "hive-agent-queue-credential"
|
||||
// {
|
||||
FAKE_BAO_BIN = "${fakeBao}/bin";
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ if (: >"$probe") 2>/dev/null; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# setup FORGE|QUEUE <description>
|
||||
# setup FORGE <description>
|
||||
setup() {
|
||||
prefix=$1
|
||||
case="$prefix: $2"
|
||||
|
|
@ -111,125 +111,108 @@ expect_no_file() {
|
|||
if [ -e "$1" ]; then fail "$1 left behind"; fi
|
||||
}
|
||||
|
||||
for prefix in FORGE QUEUE; do
|
||||
if [ "$prefix" = FORGE ]; then
|
||||
secret_name=token
|
||||
path=secret/swarm/agents/a1/forge-token
|
||||
missing_msg="no forge token at $path yet"
|
||||
else
|
||||
secret_name=secret
|
||||
path=secret/swarm/agents/a1/queue
|
||||
missing_msg="no per-agent queue credential at $path yet"
|
||||
fi
|
||||
login_call="login -method=cert -token-only"
|
||||
kv_call="kv get -field=value $path"
|
||||
secret_name=token
|
||||
path=secret/swarm/agents/a1/forge-token
|
||||
missing_msg="no forge token at $path yet"
|
||||
login_call="login -method=cert -token-only"
|
||||
kv_call="kv get -field=value $path"
|
||||
|
||||
setup "$prefix" "no store identity delivered"
|
||||
: >"$creds/hive-agent-bao-cert"
|
||||
go
|
||||
expect_rc 0
|
||||
expect_err "has no store identity"
|
||||
expect_calls
|
||||
setup FORGE "no store identity delivered"
|
||||
: >"$creds/hive-agent-bao-cert"
|
||||
go
|
||||
expect_rc 0
|
||||
expect_err "has no store identity"
|
||||
expect_calls
|
||||
|
||||
setup "$prefix" "a credential that cannot be read"
|
||||
chmod 0000 "$creds/hive-agent-bao-key"
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "cannot read $creds/hive-agent-bao-key"
|
||||
expect_calls
|
||||
setup FORGE "a credential that cannot be read"
|
||||
chmod 0000 "$creds/hive-agent-bao-key"
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "cannot read $creds/hive-agent-bao-key"
|
||||
expect_calls
|
||||
|
||||
setup "$prefix" "bao's stderr file cannot be created"
|
||||
chmod 0500 "$rt"
|
||||
go
|
||||
chmod 0700 "$rt"
|
||||
expect_rc 1
|
||||
expect_err "could not create $rt/bao.err, so bao never ran"
|
||||
expect_calls
|
||||
setup FORGE "bao's stderr file cannot be created"
|
||||
chmod 0500 "$rt"
|
||||
go
|
||||
chmod 0700 "$rt"
|
||||
expect_rc 1
|
||||
expect_err "could not create $rt/bao.err, so bao never ran"
|
||||
expect_calls
|
||||
|
||||
setup "$prefix" "the store refuses the login with an HTTP 4xx"
|
||||
login_rc=2
|
||||
login_err=$'Error authenticating: Error making API request.\n\nURL: PUT '"$bao_addr"$'/v1/auth/cert/login\nCode: 403. Errors:\n\n* permission denied\n'
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "refused this agent's certificate login with HTTP 403:"
|
||||
expect_err "* permission denied"
|
||||
expect_calls "$login_call"
|
||||
setup FORGE "the store refuses the login with an HTTP 4xx"
|
||||
login_rc=2
|
||||
login_err=$'Error authenticating: Error making API request.\n\nURL: PUT '"$bao_addr"$'/v1/auth/cert/login\nCode: 403. Errors:\n\n* permission denied\n'
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "refused this agent's certificate login with HTTP 403:"
|
||||
expect_err "* permission denied"
|
||||
expect_calls "$login_call"
|
||||
|
||||
setup "$prefix" "the store fails the login with another HTTP status"
|
||||
login_rc=2
|
||||
login_err=$'Error authenticating: Error making API request.\n\nCode: 500. Errors:\n\n* internal error\n'
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "failed this agent's certificate login with HTTP 500:"
|
||||
expect_err "* internal error"
|
||||
expect_calls "$login_call"
|
||||
setup FORGE "the store fails the login with another HTTP status"
|
||||
login_rc=2
|
||||
login_err=$'Error authenticating: Error making API request.\n\nCode: 500. Errors:\n\n* internal error\n'
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "failed this agent's certificate login with HTTP 500:"
|
||||
expect_err "* internal error"
|
||||
expect_calls "$login_call"
|
||||
|
||||
setup "$prefix" "the store refuses the certificate with a TLS alert"
|
||||
login_rc=2
|
||||
login_err="Error authenticating: Put \"$bao_addr/v1/auth/cert/login\": remote error: tls: unknown certificate authority"
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "refused this agent's certificate in the TLS handshake:"
|
||||
expect_err "remote error: tls: unknown certificate authority"
|
||||
expect_calls "$login_call"
|
||||
setup FORGE "the store refuses the certificate with a TLS alert"
|
||||
login_rc=2
|
||||
login_err="Error authenticating: Put \"$bao_addr/v1/auth/cert/login\": remote error: tls: unknown certificate authority"
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "refused this agent's certificate in the TLS handshake:"
|
||||
expect_err "remote error: tls: unknown certificate authority"
|
||||
expect_calls "$login_call"
|
||||
|
||||
setup "$prefix" "the store does not answer"
|
||||
login_rc=2
|
||||
login_err="Error authenticating: Put \"$bao_addr/v1/auth/cert/login\": dial tcp: lookup bao.t.local: no such host"
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "got no answer from the swarm secret store at $bao_addr"
|
||||
expect_err "no such host"
|
||||
expect_calls "$login_call"
|
||||
setup FORGE "the store does not answer"
|
||||
login_rc=2
|
||||
login_err="Error authenticating: Put \"$bao_addr/v1/auth/cert/login\": dial tcp: lookup bao.t.local: no such host"
|
||||
go
|
||||
expect_rc 1
|
||||
expect_err "got no answer from the swarm secret store at $bao_addr"
|
||||
expect_err "no such host"
|
||||
expect_calls "$login_call"
|
||||
|
||||
# Only the forge unit keeps its runtime directory between runs; the queue
|
||||
# unit starts each run with an empty one.
|
||||
setup "$prefix" "nothing minted at the agent's path yet"
|
||||
if [ "$prefix" = FORGE ]; then
|
||||
printf old >"$rt/$secret_name"
|
||||
chmod 0400 "$rt/$secret_name"
|
||||
fi
|
||||
kv_rc=2
|
||||
kv_err="No value found at secret/data/swarm/agents/a1"
|
||||
go
|
||||
expect_rc 0
|
||||
expect_err "$missing_msg"
|
||||
expect_err "No value found"
|
||||
expect_calls "$login_call" "$kv_call"
|
||||
if [ "$prefix" = FORGE ]; then
|
||||
expect_file "$rt/$secret_name" old
|
||||
else
|
||||
expect_no_file "$rt/$secret_name"
|
||||
fi
|
||||
# The unit keeps its runtime directory between runs.
|
||||
setup FORGE "nothing minted at the agent's path yet"
|
||||
printf old >"$rt/$secret_name"
|
||||
chmod 0400 "$rt/$secret_name"
|
||||
kv_rc=2
|
||||
kv_err="No value found at secret/data/swarm/agents/a1"
|
||||
go
|
||||
expect_rc 0
|
||||
expect_err "$missing_msg"
|
||||
expect_err "No value found"
|
||||
expect_calls "$login_call" "$kv_call"
|
||||
expect_file "$rt/$secret_name" old
|
||||
|
||||
setup "$prefix" "a first fetch writes the secret 0400"
|
||||
go
|
||||
expect_rc 0
|
||||
expect_out "fetched this agent's"
|
||||
expect_no_err "Permission denied"
|
||||
expect_calls "$login_call" "$kv_call"
|
||||
expect_file "$rt/$secret_name" the-secret
|
||||
if [ "$(stat -c %a "$rt/$secret_name")" != 400 ]; then fail "$secret_name is mode $(stat -c %a "$rt/$secret_name"), expected 400"; fi
|
||||
expect_no_file "$rt/bao.err"
|
||||
expect_no_file "$rt/token.new"
|
||||
setup FORGE "a first fetch writes the secret 0400"
|
||||
go
|
||||
expect_rc 0
|
||||
expect_out "fetched this agent's"
|
||||
expect_no_err "Permission denied"
|
||||
expect_calls "$login_call" "$kv_call"
|
||||
expect_file "$rt/$secret_name" the-secret
|
||||
if [ "$(stat -c %a "$rt/$secret_name")" != 400 ]; then fail "$secret_name is mode $(stat -c %a "$rt/$secret_name"), expected 400"; fi
|
||||
expect_no_file "$rt/bao.err"
|
||||
expect_no_file "$rt/token.new"
|
||||
|
||||
# `UMask=0377` makes every file the script creates 0400, the login's own
|
||||
# `bao.err` included, and a redirect into a 0400 file fails before bao starts.
|
||||
setup "$prefix" "0400 files already in place do not block the fetch"
|
||||
printf stale >"$rt/bao.err"
|
||||
chmod 0400 "$rt/bao.err"
|
||||
if [ "$prefix" = FORGE ]; then
|
||||
printf stale >"$rt/token.new"
|
||||
chmod 0400 "$rt/token.new"
|
||||
fi
|
||||
go
|
||||
expect_rc 0
|
||||
expect_out "fetched this agent's"
|
||||
expect_no_err "Permission denied"
|
||||
expect_no_err "refused"
|
||||
expect_calls "$login_call" "$kv_call"
|
||||
expect_file "$rt/$secret_name" the-secret
|
||||
done
|
||||
# `UMask=0377` makes every file the script creates 0400, the login's own
|
||||
# `bao.err` included, and a redirect into a 0400 file fails before bao starts.
|
||||
setup FORGE "0400 files already in place do not block the fetch"
|
||||
printf stale >"$rt/bao.err"
|
||||
chmod 0400 "$rt/bao.err"
|
||||
printf stale >"$rt/token.new"
|
||||
chmod 0400 "$rt/token.new"
|
||||
go
|
||||
expect_rc 0
|
||||
expect_out "fetched this agent's"
|
||||
expect_no_err "Permission denied"
|
||||
expect_no_err "refused"
|
||||
expect_calls "$login_call" "$kv_call"
|
||||
expect_file "$rt/$secret_name" the-secret
|
||||
|
||||
setup FORGE "an unchanged token is left in place"
|
||||
printf the-secret >"$rt/token"
|
||||
|
|
|
|||
Loading…
Reference in a new issue