swarm: courier an agent's store identity into its container, and log in with it
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the hop that carries it the rest of the way, and the in-container consumer that proves the hop works. Host side, `lifecycle::agent_identity` reads the row under *this hive's* own certificate — the hive is a principal the store already knows — and stages the leaf and its key `0600` under a new `agent-identity/<name>` state dir, deliberately outside every bind-mounted tree. Both files go in as systemd credentials rather than binds, the same answer and the same mode reason as the queue secret beside it: the staged key is unreadable to the unprivileged agent user, and the container manager reads a `--load-credential` source as root before re-exposing it under the consuming unit's own `User=`. The agent is never asked to authenticate in order to obtain the thing it authenticates with. Container side, `hive-agent-bao-identity.service` logs in with that certificate and reads the agent's own path back, failing the unit when either step does not succeed. It fails loudly where the hive-side readers degrade quietly, because a refused certificate means an agent that believes it reaches the store and never does — a cause only the login itself can name. The address is the whole switch, no separate `enable`, matching how `queue.nix` and `logs.nix` already gate themselves. A hive with a store forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a hive without one forwards nothing and no agent does. That is what keeps the delivery from landing in a container with nothing to read it. The hive can now reach an agent's identity, so hive privilege covers agent privilege. Accepted, not mitigated: the alternative is an agent fetching its own credential with a credential it does not yet have. Refs #4137
This commit is contained in:
atlas
parent
8aafe4eaee
commit
837e658d4a
10 changed files with 716 additions and 7 deletions
|
|
@ -64,10 +64,12 @@ let
|
|||
# the meta flake hands a container, so what this evaluates is what an
|
||||
# agent gets.
|
||||
#
|
||||
# Note `hyperhive`, not `services.hyperhive`: an agent container's options
|
||||
# live at the top level.
|
||||
agent =
|
||||
extra:
|
||||
# Takes a whole module rather than a settings attrset, so a fixture can
|
||||
# reach either spelling of the agent tier. `user.name` is the one option
|
||||
# with no usable default, set here at its real path and at `mkDefault` so a
|
||||
# fixture naming its own agent still wins.
|
||||
agentWith =
|
||||
module:
|
||||
(nixosSystem {
|
||||
system = pkgs.stdenv.hostPlatform.system;
|
||||
modules = [
|
||||
|
|
@ -79,11 +81,20 @@ let
|
|||
};
|
||||
boot.loader.grub.enable = false;
|
||||
system.stateVersion = "25.11";
|
||||
hyperhive = lib.recursiveUpdate { user.name = "a1"; } extra;
|
||||
services.hyperhive.agent.user.name = lib.mkDefault "a1";
|
||||
}
|
||||
module
|
||||
];
|
||||
}).config;
|
||||
|
||||
# Note `hyperhive`, not `services.hyperhive`: the agent tier's options moved
|
||||
# under `services.hyperhive.agent`, and ../agent-modules/renamed-options.nix
|
||||
# keeps the top-level spelling reaching them. ⚠️ That shim covers the
|
||||
# options that existed when the tier moved and nothing since, so a fixture
|
||||
# for an option added afterwards has to go through [`agentWith`] and name
|
||||
# the real path.
|
||||
agent = extra: agentWith { hyperhive = extra; };
|
||||
|
||||
allLocal = hive { deploy.singleHostSwarm = true; };
|
||||
bare = hive { };
|
||||
withCi = hive { deploy.forgejo.ci.enable = true; };
|
||||
|
|
@ -605,6 +616,12 @@ let
|
|||
};
|
||||
agentNoQueue = agent { };
|
||||
|
||||
# The agent side of the swarm secret store. The address is the whole switch —
|
||||
# it is both what generates the login check and what that check points at —
|
||||
# so it and the empty fixture beside it are the two arms worth having.
|
||||
agentBao = agentWith { services.hyperhive.agent.bao.addr = "https://bao.t.local:8200"; };
|
||||
agentNoBao = agentWith { };
|
||||
|
||||
# The memory-pressure pair. `claudeMemoryMaxBytes` is the container's own
|
||||
# cap, rendered per agent by meta.rs — the capped arm is the one every
|
||||
# real deploy gets, the uncapped arm is a hive that set `infinity` or a
|
||||
|
|
@ -636,6 +653,7 @@ let
|
|||
};
|
||||
agentHarness = machine: machine.systemd.services.hive-agent;
|
||||
agentSubagentDaemon = machine: machine.systemd.services.hive-subagent-daemon;
|
||||
agentBaoIdentity = machine: machine.systemd.services.hive-agent-bao-identity;
|
||||
agentSettings = machine: machine.services.opentelemetry-collector.settings;
|
||||
|
||||
# This hive's own collector, which is a HOST service — unlike the swarm
|
||||
|
|
@ -1766,6 +1784,112 @@ let
|
|||
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_SECRET_FILE)
|
||||
&& !(u.environment ? HIVE_AGENT_OIDC_CLIENT_ID_FILE);
|
||||
}
|
||||
{
|
||||
# The three ids `hive_c0re::lifecycle::agent_identity` forwards under.
|
||||
# Neither end can discover the other's spelling, and a mismatch is a
|
||||
# credential that is simply not there — which this unit then reports as
|
||||
# a hive that delivered nothing.
|
||||
name = "an agent with the store enabled imports every half of its identity";
|
||||
ok =
|
||||
let
|
||||
c = (agentBaoIdentity agentBao).serviceConfig.LoadCredential;
|
||||
in
|
||||
builtins.elem "hive-agent-bao-cert" c
|
||||
&& builtins.elem "hive-agent-bao-key" c
|
||||
&& builtins.elem "hive-agent-bao-server-ca" c;
|
||||
}
|
||||
{
|
||||
# `%d` and not a path under the agent's state dir, for the reason the
|
||||
# queue arm above gives: the host file is `0600` to the hive daemon, so
|
||||
# the only copy this unprivileged unit can open is the one systemd puts
|
||||
# in its own credentials directory. The address is the option's value
|
||||
# rather than a literal that agrees with it today.
|
||||
name = "the identity check presents its certificate out of the credentials directory";
|
||||
ok =
|
||||
let
|
||||
e = (agentBaoIdentity agentBao).environment;
|
||||
in
|
||||
e.BAO_CLIENT_CERT == "%d/hive-agent-bao-cert"
|
||||
&& e.BAO_CLIENT_KEY == "%d/hive-agent-bao-key"
|
||||
&& e.BAO_ADDR == agentBao.services.hyperhive.agent.bao.addr;
|
||||
}
|
||||
{
|
||||
# Same 403-not-a-miss reason as the hive-side readers: the path
|
||||
# `swarm_secret_client::mtls::identity_path` builds is the one this
|
||||
# agent's own policy stanza covers, and a path outside it is refused
|
||||
# however correct it looks. Built from the agent's own name rather than
|
||||
# from a literal, because the name is what makes it this agent's path
|
||||
# and not some other agent's.
|
||||
name = "the identity check reads the agent's own path";
|
||||
ok =
|
||||
let
|
||||
m = agentBao;
|
||||
name = m.services.hyperhive.agent.user.name;
|
||||
in
|
||||
lib.hasInfix "secret/swarm/agents/${name}/bao-mtls" (agentBaoIdentity m).script;
|
||||
}
|
||||
{
|
||||
# The whole point of the unit, and the thing a quieter default would
|
||||
# undo: every arm of the check ends the unit non-zero, so an agent that
|
||||
# cannot authenticate as itself says so at boot instead of at whichever
|
||||
# pull needed the store first.
|
||||
name = "the identity check fails the unit rather than degrading";
|
||||
ok =
|
||||
let
|
||||
u = agentBaoIdentity agentBao;
|
||||
in
|
||||
lib.hasInfix "exit 1" u.script
|
||||
&& !(lib.hasInfix "exit 0" u.script)
|
||||
&& u.serviceConfig.Restart == "on-failure";
|
||||
}
|
||||
{
|
||||
# Nothing about the identity may be printed, and the read-back is where
|
||||
# that could slip: `bao kv get` on this path answers with certificate
|
||||
# material, and the object beside it is a private key. The check needs
|
||||
# only whether the read succeeded.
|
||||
#
|
||||
# The path goes through `lib.escapeShellArg` here for the same reason the
|
||||
# module passes it through one — that helper decides whether an argument
|
||||
# needs quotes at all, and this one (only `[a-z0-9/-]`) comes back bare.
|
||||
# Spelling the quotes in by hand asserts a rendering nixpkgs chooses
|
||||
# rather than the redirect this property is about.
|
||||
name = "the identity check discards what it reads back";
|
||||
ok =
|
||||
let
|
||||
m = agentBao;
|
||||
name = m.services.hyperhive.agent.user.name;
|
||||
arg = lib.escapeShellArg "secret/swarm/agents/${name}/bao-mtls";
|
||||
in
|
||||
lib.hasInfix "bao kv get -field=cert ${arg} >/dev/null" (agentBaoIdentity m).script;
|
||||
}
|
||||
{
|
||||
# The absence arm, and what makes the four above able to fail. An agent
|
||||
# whose swarm never minted an identity has nothing to log in with, and a
|
||||
# failed unit at every boot would be the loudest possible statement
|
||||
# about a deployment that never asked for one.
|
||||
name = "an agent told no store address runs no identity check";
|
||||
ok = !(agentNoBao.systemd.services ? hive-agent-bao-identity);
|
||||
}
|
||||
{
|
||||
# Where the store is and where the agent is told it is, one agreement
|
||||
# spanning two modules. Asserted against the hive's own `BAO_ADDR`
|
||||
# rather than a literal, because an agent pointed at a different
|
||||
# spelling of the same store presents a certificate to a listener whose
|
||||
# name it cannot verify.
|
||||
name = "an agent is told the same store address its hive uses";
|
||||
ok =
|
||||
let
|
||||
e = allLocal.systemd.services.hive-c0re.environment;
|
||||
in
|
||||
e.HIVE_AGENT_BAO_ADDR == e.BAO_ADDR;
|
||||
}
|
||||
{
|
||||
# A hive with no certificate of its own can collect no agent's identity,
|
||||
# so forwarding an address would name a store nothing in the container
|
||||
# can reach. The same gate the `BAO_*` pair beside it sits behind.
|
||||
name = "a hive with no store identity forwards no store address to its agents";
|
||||
ok = !(bare.systemd.services.hive-c0re.environment ? HIVE_AGENT_BAO_ADDR);
|
||||
}
|
||||
{
|
||||
# Two thirds of the container's cap, and a SOFT ceiling: the daemon's
|
||||
# cgroup holds every nested claude, so `MemoryHigh=` throttles the
|
||||
|
|
|
|||
Loading…
Reference in a new issue