diff --git a/docs/scheduler/ci.md b/docs/scheduler/ci.md index bc9909b6..689f73fd 100644 --- a/docs/scheduler/ci.md +++ b/docs/scheduler/ci.md @@ -130,7 +130,7 @@ can't hold the runner's single slot indefinitely). - **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. - **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after. - **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`. -- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only the register step reads the registration token; the daemon and the jobs it runs never have it in their environment. +- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only `swarm-ci-runner-register.service` loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials. ## Registration @@ -146,13 +146,18 @@ can't hold the runner's single slot indefinitely). writes the token to a root-only file inside the container, and restarts the runner when the value changes. It runs at boot and on deploy, not on a timer. -3. **The runner unit** (`gitea-runner-swarm` in the container) loads that - file with `LoadCredential=`, not as an environment variable: for a `host` - label, act_runner hands its whole environment to every job step. A - register step before the daemon reads the token from - `$CREDENTIALS_DIRECTORY` and runs `act_runner register` only when `.runner` - is missing; with a `.runner` in place the daemon starts without the token. - A pre-start step ahead of it removes `.runner` in three cases: +3. **`swarm-ci-runner-register.service`** in the container is the only unit + that loads that file, with `LoadCredential=`. systemd hands a unit's + credentials to all of its processes, and for a `host` label act_runner + runs every job step as a process of the daemon with the daemon's whole + environment, so the daemon unit carries the token in neither. The + register unit is a oneshot under the daemon's dynamic user and state + directory. It reads the token from `$CREDENTIALS_DIRECTORY` and runs + `act_runner register` only when `.runner` is missing, so the `.runner` it + writes is the daemon's own. **The runner unit** (`gitea-runner-swarm`) + starts it in a pre-start step and runs the daemon once it succeeds. + A root step of the register unit, ahead of registering, removes `.runner` + in three cases: - The forge has no record of the runner. Forgejo answers every call from an unknown runner with `unregistered runner`, and the daemon exits 1 when its startup `Declare` gets that answer. On the next start the step finds the @@ -161,11 +166,14 @@ can't hold the runner's single slot indefinitely). and logging the error without exiting; the next restart of the unit or container registers it again. - `.runner` names another forge address than the configured one. - - `.runner` names other labels than the configured ones. + - `.runner` names other labels than the configured ones, compared in the + form the daemon writes back: `:`, with `host` for a + label that names no scheme. A changed token alone doesn't register again: the daemon authenticates with the per-runner token in `.runner`. Without `.runner` and with no token yet - the register step fails, and `Restart=on-failure` retries every 15 seconds + the register unit fails and the runner unit's pre-start step with it; + `Restart=on-failure` on the runner unit starts both again every 15 seconds, with no start limit. Host jobs run as the runner's own user, so they can still read `.runner` @@ -231,7 +239,7 @@ A malicious `default.nix` or build script in a PR can therefore: - **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic). - **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials. -No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only the runner's register step reads, so a job step doesn't inherit it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls. +No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only `swarm-ci-runner-register.service` loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls. Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path. diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 95820eff..26b46724 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -55,22 +55,22 @@ of the cell says how. -| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | -| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `swarm/agents//matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | -| `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | -| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | -| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | -| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | -| `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | -| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | -| `swarm/agents//forge/