Watch
0
0
Fork
You've already forked hyperhive
0

sign ssh certificates on demand from ssh itself, warn before setup, say what to do when a host key changes

This commit is contained in:
atlas 2026-11-16 19:42:44 +01:00 • committed by mara
commit 3d7541c0af
11 changed files with 521 additions and 67 deletions

View file

@ -5,6 +5,13 @@ key onto it. The agent holds no standing SSH key. While you allow it, the agent
gets a certificate from the swarm's SSH certificate authority (CA) that lasts
an hour. The machine trusts the CA, so it needs no list of keys.
> **Warning:** SSH access with no constraints on the machine makes the agent
> unconstrained there. Whatever the agent's own sandbox stops it doing, it can
> do on a machine it can log in to. Switch SSH on only for machines you would let
> the agent control completely. What the agent can do on the machine is up to
> the account you let it use: give it an account with only the rights you want
> it to have.
Two things have to be true before an agent gets in:
1. **You switch SSH on for that agent**, in the swarm UI.
@ -16,9 +23,14 @@ certificate unit. The switch alone decides whether the swarm signs.
## 1. Switch it on for an agent
Open the agent's card in the swarm UI and find the **ssh** row. Select it and
tick **allow the agent to get SSH certificates**. The default is off. The
agent has a certificate within about 25 minutes of the change. Untick the box
to stop new certificates. The row also shows the three forms of the target
tick **allow the agent to get SSH certificates**. The default is off. Untick the
box to stop new certificates.
The change takes seconds, not minutes. The swarm updates the agent's login
within a few seconds. The agent gets its certificate the first time it runs
`ssh` after that, so there is nothing to wait for on the agent. If it ran `ssh`
just before you ticked the box, the next run within ten seconds still finds the
refusal remembered; after that it signs. The row also shows the three forms of the target
setup below, already filled in with the swarm's CA key and the agent's name.
The agent then runs `ssh user@host`, with no other setup. It offers its
@ -29,6 +41,25 @@ shows on the first connection, keeps it in its state directory (so it survives
restarts), and refuses a changed key afterwards. A swapped machine on the first
contact would go unnoticed. The row's explainer says so.
### When a host's key changes
The agent refuses a machine whose key differs from the one it kept, and `ssh`
says the host identification changed. A rebuilt or replaced machine does this.
Verify the new key out of band first: read the machine's host key on the machine
itself (`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`) and compare it with
the fingerprint the agent reports. If it matches, tell the agent to forget the
old one. On the hive host, as root:
```bash
ssh-keygen -R <host> -f /agents/<agent>/state/ssh_known_hosts
```
The next connection records the new key. You can also delete the line, or the
whole file, at that path. To work inside the agent's container instead, see
[`hivectl`](../hives/hivectl.md) for the shell it opens as the agent user. If the
key changed and you didn't change the machine, don't clear it: something else
answers at that address.
This doesn't open the network. If you set the hive's `egress.forward` to
`reject`, the hive refuses the agent's direct connection to the machine's port
until a separate allowance exists; the default `log` lets it through. See
@ -77,8 +108,8 @@ bao read -field=public_key ssh-agents/config/ca
entry, or the `cert-authority` line) and deploy the machine. The next login
attempt fails.
- **One agent from every machine:** untick the switch. The swarm stops signing
for it within five minutes; an agent already signed in can still sign
for up to an hour. Destroying the agent cuts its store identity off within
for it within seconds; a token the agent got while the switch was on can still
sign for up to an hour. Destroying the agent cuts its store identity off within
minutes. Either way a certificate already signed stays valid until it
expires, at most three hours, so removing the principal on a machine is what
cuts it off there at once.

View file

@ -81,11 +81,16 @@ and attaches it to the agent's cert-auth role `hive-agent-<name>` as a second
- the key is `/run/hive-agent-ssh-cert/key` (directory `0700`, files `0400`,
`UMask=0377`), generated when absent; the certificate is
`key-cert.pub` beside it;
- each run logs in to the store with the cert identity `hive-agent-bao-identity`
keeps, signs `key.pub` through the role, checks the answer parses as a
certificate and renames it into place. A `403` from the signing call is the
switch being off: the unit prints one line and exits 0, and the next tick asks
again. Any other failure fails the unit, which retries;
- the unit's script is also the **helper** `ssh` runs: one text, two entries. Each
run logs in afresh to the store with the cert identity
`hive-agent-bao-identity` keeps, signs `key.pub` through the role, checks the
answer parses as a certificate and renames it into place. A `403` from the
signing call is the switch being off: the unit prints one line and exits 0,
and the next tick asks again. Any other failure fails the unit, which retries.
One run at a time (`flock` on the directory), the unit's or an `ssh`'s;
- the unit copies the store's CA out of its credentials into
`server-ca.pem` beside the key, so that an `ssh` run, which has no
credentials directory, can reach the store;
- the timer fires 20 minutes after each run, a third of the 60-minute lifetime,
from the same file as the role's TTL;
- the system `/etc/ssh/ssh_config` has one `Host *` block: `IdentityFile` and
@ -94,7 +99,16 @@ and attaches it to the agent's cert-auth role `hive-agent-<name>` as a second
`UserKnownHostsFile /agents/<name>/state/ssh_known_hosts`. The host binds the
state directory in, so the pins survive a restart; the container root
doesn't. The agent trusts the first key a host shows and refuses a changed
one.
one;
- the same config starts with `Match exec "HIVE_SSH_CERT_ENSURE=1 <helper> </dev/null"`,
with no settings of its own. `ssh` runs the command while it reads the config,
before it loads an identity. With `HIVE_SSH_CERT_ENSURE` the helper is silent and
its exit status says whether a usable certificate exists: it exits 0 at once
when the certificate has more than ten minutes left, asks the store nothing
for ten seconds after a refusal, and otherwise logs in and signs. Nothing in
the config depends on the answer; the point is the certificate it leaves
behind. Every `ssh` on the agent runs this, so a loop of them while the switch
is off costs one login per ten seconds.
## The target module

View file

@ -857,10 +857,12 @@ checkbox that does a `PUT` of `ssh` with the stored permissions beside it, and
a folded **on the target** section. That section reads `GET /api/ssh/ca` and
shows the swarm's CA key and the agent's name as the lines for a NixOS machine,
a home-manager user and a plain `authorized_keys` file, with a copy button; the
card builds the text in `frontend/packages/swarm-ui/src/agent/detail/ssh.ts`. A `PUT`
card builds the text in `frontend/packages/swarm-ui/src/agent/detail/ssh.ts`. Above the checkbox the card shows a warning (SSH access to an
unconstrained machine leaves the agent with no limits there), the explainer with the delays,
and how to forget a host key that changed. The card says the change takes seconds, and why: the controller rewrites the role
when the field changes and the agent signs on its next `ssh`. A `PUT`
that leaves `ssh` out, and the `DELETE` of **use the default**, keep it. The
controller attaches the signing policy to the agent's login role within five
minutes; see [the SSH CA](../swarm/ssh-ca.md#the-switch-and-the-signing-policy).
controller attaches the signing policy to the agent's login role at once; see [the SSH CA](../swarm/ssh-ca.md#the-switch-and-the-signing-policy).
A saved change isn't instant. The hive reads the grant about every 30 seconds
and the agent takes it at its next turn. A change to `manage_root_agent`

View file

@ -19,6 +19,7 @@ import { Hint } from "../../ui/hint/Hint.js";
import { Preformatted } from "../../ui/preformatted/Preformatted.js";
import type { GrantState } from "./AgentPermissions.js";
import {
forgetHostKeyCommand,
sshBody,
sshSummary,
targetSnippet,
@ -77,19 +78,27 @@ export function AgentSsh({ agent, g }: { agent: string; g: GrantState }) {
title={`whether ${agent} may get SSH certificates from the swarm`}
popoverLabel={`${agent} SSH access`}
>
<p class="ui-badge-popover-help" role="note">
<strong>Warning:</strong> SSH access with no constraints on the
machine makes {agent} unconstrained there. Whatever {agent}'s own
sandbox stops it doing, it can do on a machine it can log in to. Only
allow this for machines you would let {agent} control completely.
</p>
<p class="ui-badge-popover-help">
Lets {agent} log in over SSH to machines that trust the swarm's SSH
certificate authority. With this on, {agent} gets a certificate that
names it and lasts one hour, and renews it every 20 minutes; its key
lives in memory only. The change reaches {agent} within about 25
minutes. A certificate already signed stays valid until it expires, at
most three hours. A machine has to trust the authority before anything
works: see "on the target" below.
certificate authority. With this on, {agent} signs its own key for a
certificate that names it, on its first <code>ssh</code> after the
swarm has updated its login, which takes seconds. The certificate
lasts one hour, and the key lives in memory only. Switching off stops
new certificates within seconds; one already signed stays valid until
it expires, an hour after signing. A machine has to trust the
authority before anything works: see "on the target" below.
</p>
<p class="ui-badge-popover-help" role="note">
{agent} accepts the key a machine shows on the first connection and
refuses a changed one afterwards (trust on first use). It does not
check the key against anything you registered.
refuses a changed one afterwards (trust on first use). If a host's key
really changed, check the new key out of band, then forget the old one
with <code>{forgetHostKeyCommand(agent)}</code> on the hive host.
</p>
<CheckboxField
id={`ssh-${agent}`}

View file

@ -73,3 +73,12 @@ export function targetSnippet(
return `# in ~<user>/.ssh/authorized_keys on the target\n${authorizedKeysLine(agent, ca)}`;
}
}
/**
* The command that makes the agent forget a host's recorded key, run as root on
* the hive host after the operator has checked the new key out of band. The
* agent's pins live in its state directory.
*/
export function forgetHostKeyCommand(agent: string): string {
return `ssh-keygen -R <host> -f /agents/${agent}/state/ssh_known_hosts`;
}

View file

@ -4,6 +4,7 @@ import assert from "node:assert/strict";
import { test } from "node:test";
import {
authorizedKeysLine,
forgetHostKeyCommand,
sshBody,
sshSummary,
targetSnippet,
@ -37,6 +38,14 @@ test("the authorized_keys line trusts the CA for one principal only", () => {
);
});
test("the forget command names the agent's pin file and leaves the host to the operator", () => {
assert.equal(
forgetHostKeyCommand("atlas"),
"ssh-keygen -R <host> -f /agents/atlas/state/ssh_known_hosts",
);
assert.ok(forgetHostKeyCommand("maia").includes("/agents/maia/"));
});
test("each target form names the agent and the CA key, and keeps <user> for the operator", () => {
for (const form of ["nixos", "home-manager", "ssh"]) {
const text = targetSnippet(form, "atlas", ca);

View file

@ -3,18 +3,26 @@
#
# On every agent that has a store address (the same switch as ./bao.nix: with no
# store there is nothing to sign with). The agent has no standing SSH credential:
# `${unitName}.service` generates an ed25519 key in its own `/run` directory
# (tmpfs, never on disk), logs in to the swarm secret store with the identity
# ./bao.nix keeps current, and has the public half signed by the store's SSH
# engine (`swarm_secret_client::policy::SSH_MOUNT`). The role it signs through
# issues a user certificate with this agent's name as its only principal, valid
# for an hour; a timer signs again well before that runs out.
# the helper below generates an ed25519 key in `/run/${unitName}` (tmpfs, never
# on disk), logs in to the swarm secret store with the identity ./bao.nix keeps
# current, and has the public half signed by the store's SSH engine
# (`swarm_secret_client::policy::SSH_MOUNT`). The role it signs through issues a
# user certificate with this agent's name as its only principal, valid for an
# hour.
#
# Whether the store signs is the operator's per-agent switch in the swarm UI
# (the `ssh` permission, `swarm-controller`'s `agent_ssh`). With it off the store
# refuses, the unit logs one line and ends successfully, and the next tick asks
# again; with it on the agent has a certificate within one tick. The unit
# needs no change to be switched.
# (the `ssh` permission, `swarm-controller`'s `agent_ssh`). Every signing makes a
# fresh login, so a token never outlives the policies the role had when it was
# issued by more than the helper's own run. Two things run the helper:
#
# - `${unitName}.service`, from a timer, renews well before the certificate
# runs out. With the switch off the store refuses: the unit logs one line, ends
# successfully and asks again at the next tick.
# - `ssh` itself, through a `Match exec` line in the system config: when the
# certificate is missing or close to its end, the helper signs before the
# connection starts. That is what makes the first `ssh` after the operator
# switches SSH on work at once. It is silent and its exit status is whether a
# usable certificate exists; nothing in the config depends on the answer.
#
# A target machine trusts the CA, not this key: ../target-modules/agent-ssh.nix.
{
@ -38,28 +46,57 @@ let
unitName = "hive-agent-ssh-cert";
runtimeDir = unitName;
keyFile = "/run/${runtimeDir}/key";
dir = "/run/${runtimeDir}";
keyFile = "${dir}/key";
pubFile = "${keyFile}.pub";
# `ssh` reads `<IdentityFile>-cert.pub` by itself, but the config names it so
# that the file a Host block uses is visible in the config.
certFile = "${keyFile}-cert.pub";
stagingFile = "/run/${runtimeDir}/cert.new";
errFile = "/run/${runtimeDir}/bao.err";
stagingFile = "${dir}/cert.new";
errFile = "${dir}/bao.err";
# The store's CA, copied out of the unit's credentials so that the helper can
# reach the store from an `ssh` run, which has no credentials directory.
serverCaCopy = "${dir}/server-ca.pem";
# Written when the store refuses, so that a loop of `ssh` calls does not log in
# to the store each time the switch is off.
refusedFile = "${dir}/refused";
# In the state dir, which is bound from the host, so a pin survives a restart.
knownHostsFile = "/agents/${agentName}/state/ssh_known_hosts";
signPath = "${ca.mountPath}/sign/${ca.roleName}";
# A certificate with less than this left is signed again by `ssh`. The timer
# fires every `ca.renewEveryMinutes`, so a healthy agent never gets here.
ensureMarginSec = 600;
# How long a refusal is remembered. The delay the operator sees after
# switching on is at most this, then one login and one signing.
refusedForSec = 10;
storeRetry = import ../host-modules/lib/store-retry.nix { };
binPath = lib.makeBinPath [
pkgs.openbao
pkgs.openssh
pkgs.coreutils
pkgs.gnused
pkgs.util-linux
];
# The helper, and the unit's script: one text. `HIVE_SSH_CERT_ENSURE=1` is the
# `ssh` entry; without it this is the timer's. The tools go after the caller's
# `PATH`, so a caller that has its own wins.
helper = pkgs.writeShellScript unitName config.systemd.services.${unitName}.script;
in
{
config = lib.mkIf (baoCfg.addr != null) {
# Every host: the certificate, and the pins in the state dir, which outlives
# the container root. `accept-new` records a host's key at first contact and
# refuses a changed one after that.
# refuses a changed one after that. The `Match` line has no settings: ssh
# runs its command while it reads the config, before it loads any identity,
# and the answer is not used.
programs.ssh.extraConfig = ''
Match exec "HIVE_SSH_CERT_ENSURE=1 ${helper} </dev/null"
Host *
IdentityFile ${keyFile}
CertificateFile ${certFile}
@ -77,11 +114,6 @@ in
"hive-agent-bao-identity.service"
];
wantedBy = [ "multi-user.target" ];
path = [
pkgs.openbao
pkgs.openssh
pkgs.coreutils
];
# ../host-modules/lib/store-retry.nix.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
@ -98,19 +130,77 @@ in
UMask = "0377";
LoadCredential = [ serverCaCredential ];
};
environment = {
BAO_ADDR = toString baoCfg.addr;
BAO_CLIENT_CERT = baoCfg.clientCertFile;
BAO_CLIENT_KEY = baoCfg.clientKeyFile;
};
script = ''
set -euo pipefail
# `UMask=0377` in the unit makes every file this script creates `0400`,
# so each one is removed before the command that creates it. `ssh` runs
# the same text, which therefore sets it too.
umask 0377
export PATH="$PATH:${binPath}"
export BAO_ADDR=${lib.escapeShellArg (toString baoCfg.addr)}
export BAO_CLIENT_CERT=${lib.escapeShellArg baoCfg.clientCertFile}
export BAO_CLIENT_KEY=${lib.escapeShellArg baoCfg.clientKeyFile}
dir=${lib.escapeShellArg dir}
key=${lib.escapeShellArg keyFile}
pub=${lib.escapeShellArg pubFile}
cert=${lib.escapeShellArg certFile}
staged=${lib.escapeShellArg stagingFile}
err=${lib.escapeShellArg errFile}
server_ca=${lib.escapeShellArg serverCaCopy}
refused=${lib.escapeShellArg refusedFile}
ensure=
if [ -n "''${HIVE_SSH_CERT_ENSURE-}" ]; then ensure=1; fi
# The `ssh` entry is silent and says yes or no by its exit status; the
# unit's output is its log.
if [ -n "$ensure" ]; then exec >/dev/null 2>&1; fi
# `ssh` can run before the unit's first start has made the directory.
if [ ! -d "$dir" ]; then mkdir -m 0700 "$dir"; fi
# One run at a time, the unit's or an `ssh`'s: two would race on the key
# and the staged file.
exec {lock}<"$dir"
if ! flock -w 20 "$lock"; then
echo "another run holds $dir and did not finish in 20 seconds." >&2
exit 1
fi
# Epoch second the certificate ends, 0 when there is none.
cert_end() {
local end
end=$(ssh-keygen -L -f "$cert" 2>/dev/null | sed -n 's/^ *Valid: from .* to //p') || end=
if [ -n "$end" ]; then date -d "$end" +%s 2>/dev/null || echo 0; else echo 0; fi
}
if [ -n "$ensure" ]; then
now=$(date +%s)
# Enough left: nothing to do, and nothing asked of the store.
if [ "$(cert_end)" -gt $((now + ${toString ensureMarginSec})) ]; then exit 0; fi
# The store refused a moment ago: the switch is off.
if [ -e "$refused" ] && [ $((now - $(stat -c %Y "$refused"))) -lt ${toString refusedForSec} ]; then
exit 1
fi
fi
# The unit has the credentials directory and keeps the store's CA where
# an `ssh` run can find it.
if [ -n "''${CREDENTIALS_DIRECTORY-}" ] && [ -s "$CREDENTIALS_DIRECTORY/${serverCaCredential}" ]; then
rm -f "$server_ca"
cp "$CREDENTIALS_DIRECTORY/${serverCaCredential}" "$server_ca"
fi
if [ -s "$server_ca" ]; then export BAO_CACERT="$server_ca"; fi
trap 'rm -f "$err" "$staged"' EXIT
# No identity delivered: ./bao.nix's check reports that; saying it
# twice adds nothing.
for f in "$BAO_CLIENT_CERT" "$BAO_CLIENT_KEY"; do
if [ ! -s "$f" ]; then
echo "this agent has no store identity, so it cannot get an SSH certificate." >&2
if [ -n "$ensure" ]; then exit 1; fi
exit 0
fi
if [ ! -r "$f" ]; then
@ -119,19 +209,6 @@ in
fi
done
if [ -s "$CREDENTIALS_DIRECTORY/${serverCaCredential}" ]; then
export BAO_CACERT="$CREDENTIALS_DIRECTORY/${serverCaCredential}"
fi
# `UMask=0377` makes every file this script creates `0400`, so each one
# is removed before the command that creates it.
key=${lib.escapeShellArg keyFile}
pub=${lib.escapeShellArg pubFile}
cert=${lib.escapeShellArg certFile}
staged=${lib.escapeShellArg stagingFile}
err=${lib.escapeShellArg errFile}
trap 'rm -f "$err" "$staged"' EXIT
# The key lives as long as /run does, which is as long as this
# container: a restart is a new key. Only a pair with both halves is
# kept.
@ -141,6 +218,8 @@ in
chmod 0400 "$key"
fi
# A fresh login every time: a token carries the policies the role had
# when it was issued, so only a new one sees the operator's switch.
rm -f "$err"
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "the swarm secret store at $BAO_ADDR refused or could not answer this agent's certificate login:" >&2
@ -157,7 +236,10 @@ in
# so no failed unit. Anything else (a sealed or unreachable store) is
# one, and retried.
if [[ "$(<"$err")" == *"Code: 403"* ]]; then
rm -f "$refused"
: >"$refused"
echo "SSH certificates are not enabled for this agent (the swarm store refused to sign); asking again next tick."
if [ -n "$ensure" ]; then exit 1; fi
exit 0
fi
echo "the swarm secret store would not sign this agent's SSH key at ${signPath}:" >&2
@ -173,6 +255,7 @@ in
# One rename: ssh never reads a half-written certificate.
mv -f "$staged" "$cert"
rm -f "$refused"
echo "signed this agent's SSH key as ${agentName}, valid for ${toString ca.certTtlMinutes} minutes."
'';
};

View file

@ -236,6 +236,12 @@ in
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
# Runs ./agent-modules/ssh.nix's helper, the timer's entry and the one `ssh`
# runs, against a stub `bao` and a real `ssh-keygen`.
script-test-agent-ssh-cert = import ./script-tests/agent-ssh-cert.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
# Runs ./agent-modules/backend-env.nix's fetch-and-seed unit against a stub
# `bao` that keeps its leaves in files.
script-test-agent-backend-env = import ./script-tests/agent-backend-env.nix {

View file

@ -562,7 +562,7 @@ let
&& !(lib.hasInfix "valid_principals" s)
&& !(lib.hasInfix ".ssh" s)
&& lib.elem "hive-agent-bao-identity.service" u.after
&& u.environment.BAO_CLIENT_CERT == agentForgeBao.services.hyperhive.agent.bao.clientCertFile;
&& lib.hasInfix "export BAO_CLIENT_CERT=${agentForgeBao.services.hyperhive.agent.bao.clientCertFile}" s;
}
{
# The certificate lives an hour and the timer fires every third of
@ -599,17 +599,35 @@ let
&& !(lib.hasInfix "hive-agent-ssh-cert" agentForgeNoBao.environment.etc."ssh/ssh_config".text);
}
{
# A refused signing is the switch being off, not a fault: one line and a
# clean exit. Any other failure still fails the unit so it retries.
name = "the ssh certificate unit ends cleanly on a 403 and fails on any other signing error";
# `ssh` runs the helper before it loads an identity, so the first `ssh`
# after the operator's switch has a certificate. The line comes before the
# `Host *` block it serves, runs the `ssh` entry, and is on every agent
# with a store and no other.
name = "the ssh config signs on demand: a Match exec line runs the helper's ssh entry ahead of the Host block";
ok =
let
conf = agentForgeBao.environment.etc."ssh/ssh_config".text;
# What comes before the first `Host *`.
before = lib.head (lib.splitString "Host *" conf);
in
lib.hasInfix "Match exec \"HIVE_SSH_CERT_ENSURE=1 /nix/store/" before
&& lib.hasInfix "</dev/null\"" conf
&& !(lib.hasInfix "Match exec" agentForgeNoBao.environment.etc."ssh/ssh_config".text);
}
{
# The helper is the unit's script and the one `ssh` runs: the ssh entry
# is silent, signs from a fresh login, and remembers a refusal.
name = "the ssh helper is silent in the ssh entry, logs in afresh to sign, and keeps the store's CA for the ssh run";
ok =
let
s = (sshUnit agentForgeBao).script;
at = lib.splitString "Code: 403" s;
in
lib.length at == 2
&& lib.hasInfix "exit 0" (lib.head (lib.splitString "exit 1" (lib.elemAt at 1)))
&& lib.hasInfix "exit 1" (lib.elemAt at 1);
lib.hasInfix "HIVE_SSH_CERT_ENSURE" s
&& lib.hasInfix "exec >/dev/null 2>&1" s
&& lib.hasInfix "bao login -method=cert -token-only" s
&& lib.hasInfix "export BAO_TOKEN" s
&& lib.hasInfix "server-ca.pem" s
&& lib.hasInfix "Code: 403" s;
}
];
in

View file

@ -0,0 +1,92 @@
# `checks.script-test-agent-ssh-cert` — runs ../agent-modules/ssh.nix's helper,
# the text of the unit `hive-agent-ssh-cert`, against a stub `bao` and a real
# `ssh-keygen`, in both entries: the timer's and the `ssh` one. The cases are in
# ./agent-ssh-cert.sh.
#
# The one edit to the text is its `/run/` prefix, moved under the build
# directory because the sandbox has no writable `/run`. The helper adds its own
# tools after the caller's `PATH`, so the stub in front wins.
{
pkgs,
lib,
self,
nixosSystem,
}:
let
inherit
(import ../module-eval/lib.nix {
inherit
pkgs
lib
self
nixosSystem
;
})
agentWith
;
machine = agentWith { services.hyperhive.agent.bao.addr = "https://bao.t.local:8200"; };
# `login` prints a token; `write` signs `public_key=@<file>` with the test CA
# for `FAKE_SIGN_VALID` (an `ssh-keygen -V` interval), or answers 403 when
# `FAKE_SIGN_MODE` is `refuse`, or fails otherwise when it is `fail`. A call
# without the token the last login printed fails, so a script that signs
# without a login of its own cannot pass.
fakeBao = pkgs.writeShellScriptBin "bao" ''
umask 022
echo "$1" >> "$FAKE_BAO_LOG"
case "$1" in
login)
printf 'tok-%s' "$(wc -l < "$FAKE_BAO_LOG")"
;;
write)
if [ "''${BAO_TOKEN-}" != "tok-$(( $(wc -l < "$FAKE_BAO_LOG") - 1 ))" ]; then
echo "fake bao: write without the token of the login just before it" >&2
exit 97
fi
case "$FAKE_SIGN_MODE" in
refuse)
printf 'Error writing data: Code: 403. Errors:\n* permission denied\n' >&2
exit 2
;;
fail)
printf 'Error writing data: Code: 503. Errors:\n* sealed\n' >&2
exit 2
;;
esac
pub=
for a in "$@"; do
case "$a" in public_key=@*) pub=''${a#public_key=@} ;; esac
done
tmp=$(mktemp -d)
cp "$pub" "$tmp/k.pub"
${pkgs.openssh}/bin/ssh-keygen -q -s "$FAKE_CA_KEY" -I fake -n a1 -V "$FAKE_SIGN_VALID" "$tmp/k.pub"
cat "$tmp/k-cert.pub"
;;
*)
echo "fake bao: unexpected call: $*" >&2
exit 98
;;
esac
'';
script = pkgs.writeText "agent-ssh-cert-script" machine.systemd.services.hive-agent-ssh-cert.script;
in
pkgs.runCommand "hyperhive-script-test-agent-ssh-cert"
{
nativeBuildInputs = [
pkgs.openssh
pkgs.coreutils
pkgs.util-linux
];
SSH_CERT_SCRIPT = script;
FAKE_BAO_BIN = "${fakeBao}/bin";
SSH_CERT_PATH = lib.makeBinPath [
pkgs.coreutils
pkgs.bash
];
}
''
${pkgs.bash}/bin/bash ${./agent-ssh-cert.sh}
touch "$out"
''

View file

@ -0,0 +1,181 @@
# Cases for ./agent-ssh-cert.nix. Each case gets a fresh runtime directory and
# runs the helper in one entry (the timer's, or `ssh`'s with
# HIVE_SSH_CERT_ENSURE=1) with a clean environment, and asserts on the exit
# code, the output, the files left behind and the `bao` calls made.
set -uo pipefail
failures=0
count=0
case=
fail() {
echo "FAILED: $case: $*" >&2
failures=$((failures + 1))
}
top=$(mktemp -d)
ssh-keygen -q -t ed25519 -N "" -f "$top/ca"
# setup <description>
setup() {
case="$1"
dir=$(mktemp -d)
rt=$dir/run/hive-agent-ssh-cert
mkdir -p "$dir/run/hive-agent-bao-identity/current" "$dir/creds"
mkdir -m 0700 "$rt"
printf cert >"$dir/run/hive-agent-bao-identity/current/cert"
printf key >"$dir/run/hive-agent-bao-identity/current/key"
printf ca >"$dir/creds/hive-agent-bao-server-ca"
log=$dir/bao.log
: >"$log"
sed -e "s|/run/|$dir/run/|g" "$SSH_CERT_SCRIPT" >"$dir/script"
chmod +x "$dir/script"
if grep -q ' /run/\|"/run/\|=/run/' "$dir/script"; then fail "the script still names /run after the rewrite"; fi
sign_mode=ok
valid=+1h
creds=$dir/creds
}
# go [ensure]
go() {
count=$((count + 1))
local ensure=
if [ "${1-}" = ensure ]; then ensure=1; fi
(
cd "$dir" || exit 99
exec env -i \
PATH="$FAKE_BAO_BIN:$SSH_CERT_PATH" \
HOME="$dir" \
CREDENTIALS_DIRECTORY="$creds" \
HIVE_SSH_CERT_ENSURE="$ensure" \
FAKE_BAO_LOG="$log" \
FAKE_CA_KEY="$top/ca" \
FAKE_SIGN_MODE="$sign_mode" \
FAKE_SIGN_VALID="$valid" \
"$dir/script"
) >"$dir/out" 2>"$dir/err"
rc=$?
}
expect_rc() {
if [ "$rc" != "$1" ]; then fail "exit $rc, expected $1; stderr: $(<"$dir/err")"; fi
}
expect_silent() {
if [ -s "$dir/out" ] || [ -s "$dir/err" ]; then fail "output where none was wanted: $(<"$dir/out") $(<"$dir/err")"; fi
}
expect_calls() {
local want
want=$(printf '%s\n' "$@")
if [ "$(<"$log")" != "${want%$'\n'}" ]; then fail "bao calls were '$(<"$log" tr '\n' ' ')', expected '$*'"; fi
}
expect_cert() {
if ! ssh-keygen -L -f "$rt/key-cert.pub" >/dev/null 2>&1; then fail "no readable certificate at $rt/key-cert.pub"; fi
}
# The seconds the certificate at $rt has left; 0 when there is none.
left() {
local end
end=$(ssh-keygen -L -f "$rt/key-cert.pub" 2>/dev/null | sed -n 's/^ *Valid: from .* to //p')
if [ -z "$end" ]; then echo 0; else echo $(($(date -d "$end" +%s) - $(date +%s))); fi
}
# A certificate for the key the helper will keep, valid for `$1`, put in place
# as a first run would have left it.
seed_cert() {
mkdir -p "$rt"
rm -f "$rt/key" "$rt/key.pub" "$rt/key-cert.pub"
(umask 0377 && ssh-keygen -q -t ed25519 -N "" -f "$rt/key" && chmod 0400 "$rt/key")
ssh-keygen -q -s "$top/ca" -I seed -n a1 -V "$1" "$rt/key.pub"
}
setup "ssh finds no certificate: it logs in, signs, and is silent and true"
go ensure
expect_rc 0
expect_silent
expect_calls login write
expect_cert
if [ "$(left)" -lt 3000 ]; then fail "the new certificate has only $(left)s left"; fi
setup "ssh finds a certificate with time left: it asks the store nothing"
seed_cert +1h
go ensure
expect_rc 0
expect_silent
expect_calls
setup "ssh finds a certificate close to its end: it signs again with a fresh login"
seed_cert +5m
before=$(left)
go ensure
expect_rc 0
expect_silent
expect_calls login write
if [ "$(left)" -le "$before" ]; then fail "the certificate was not replaced ($before then $(left))"; fi
setup "ssh finds an expired certificate: it signs again"
seed_cert -2h:-1h
go ensure
expect_rc 0
expect_calls login write
if [ "$(left)" -lt 3000 ]; then fail "an expired certificate was kept ($(left)s)"; fi
setup "the store refuses (switch off): ssh gets false, quietly, and no certificate"
sign_mode=refuse
go ensure
expect_rc 1
expect_silent
expect_calls login write
if [ -e "$rt/key-cert.pub" ]; then fail "a certificate appeared although the store refused"; fi
setup "right after a refusal ssh does not ask again, and does once the memory has aged"
sign_mode=refuse
go ensure
go ensure
expect_rc 1
expect_calls login write
touch -d '1 minute ago' "$rt/refused"
sign_mode=ok
go ensure
expect_rc 0
expect_calls login write login write
expect_cert
setup "the timer's entry signs even with time left, and a refusal is one line and a clean exit"
seed_cert +1h
go
expect_rc 0
expect_calls login write
sign_mode=refuse
go
expect_rc 0
if ! grep -q 'not enabled for this agent' "$dir/out"; then fail "no line saying signing is off: $(<"$dir/out")"; fi
if [ -s "$dir/err" ]; then fail "stderr on a refusal: $(<"$dir/err")"; fi
setup "any other signing failure fails both entries, and ssh stays quiet"
sign_mode=fail
go
expect_rc 1
go ensure
expect_rc 1
expect_silent
setup "a store identity that is missing is true for the timer and false for ssh"
rm "$dir/run/hive-agent-bao-identity/current/cert"
go
expect_rc 0
go ensure
expect_rc 1
expect_calls
setup "the unit keeps the store's CA where an ssh run finds it, and the key stays unreadable to others"
go
expect_rc 0
if [ "$(<"$rt/server-ca.pem")" != ca ]; then fail "the store's CA was not copied"; fi
if [ "$(stat -c %a "$rt/key")" != 400 ]; then fail "the key is mode $(stat -c %a "$rt/key")"; fi
if [ "$(stat -c %a "$rt/key-cert.pub")" != 400 ]; then fail "the certificate is mode $(stat -c %a "$rt/key-cert.pub")"; fi
echo "$count runs, $failures failures"
if [ "$failures" -ne 0 ]; then exit 1; fi