swarm-ci-runner: run the forge CI runner as a swarm service
The CI runner was a hive deploy module: hive-ci.nix ran it beside the forge, and hive-c0re registered it through the forge's admin API with the `core` token, writing the registration token through hive-priv. It could only run on the forge's host, and it kept an admin-scoped forge token on a hive. Now the runner is a swarm service: - swarm-ci-runner.nix runs it in a `swarm-ci-runner` container on the host that sets deploy.ci-runner.enable, reaching the forge by name. It needs neither the forge nor a hive there. - swarm-controller reads the forge's runner registration token with its own admin token and writes it to swarm/services/ci-runner/ registration-token whenever it differs from what it last wrote. Its grant on that path is create/update only. - The runner's own leaf (CN swarm-ci-runner, role and policy of the same name) reads that one path. swarm-bao-ci-runner.service fetches it into the runner's EnvironmentFile and restarts the runner on a change. - A pre-start step removes .runner when the daemon's journal since its last start holds Forgejo's "unregistered runner" refusal (the daemon exits 1 when its startup Declare gets it), or when .runner names another forge address; upstream's pre-start step then registers again. Removed from the hive side: hive-ci.nix and deploy.forgejo.ci.*, forge/ci_runner.rs, PrivRequest::RegisterCiRunner and its handler and ReadWritePaths entry, InfraContainer::Ci with hivectl's --ci and the dashboard's hive-ci journal entry, HYPERHIVE_FORGE_CI_ENABLED, and read:admin,write:admin from CORE_TOKEN_SCOPES: nothing else in hive-c0re calls /api/v1/admin/*. The old ci options, under both swarm.forge.ci and deploy.forgejo.ci, now fail evaluation naming deploy.ci-runner. The forge's half of the old ci.enable, the actions/checkout mirror and DEFAULT_ACTIONS_URL = self, is deploy.forgejo.actionsFromSelf. Closes #4954
This commit is contained in:
parent
3d43e9588c
commit
e2da5509ae
60 changed files with 1088 additions and 1201 deletions
|
|
@ -4,7 +4,7 @@ name: coverage
|
||||||
# than printed into the run log, so a scheduled run leaves something to fetch.
|
# than printed into the run log, so a scheduled run leaves something to fetch.
|
||||||
#
|
#
|
||||||
# ⚠️ The hour is deliberate: at the default job capacity of 1
|
# ⚠️ The hour is deliberate: at the default job capacity of 1
|
||||||
# (`services.hyperhive.deploy.forgejo.ci.concurrency`) this job holds the
|
# (`services.hyperhive.deploy.ci-runner.concurrency`) this job holds the
|
||||||
# runner for its whole timeout and everything else queues behind it.
|
# runner for its whole timeout and everything else queues behind it.
|
||||||
#
|
#
|
||||||
# Separate from ci.yml because a `workflow_dispatch` trigger there fires every
|
# Separate from ci.yml because a `workflow_dispatch` trigger there fires every
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ name: flake update
|
||||||
# apply. Stale PRs just stack; newest lock wins.
|
# apply. Stale PRs just stack; newest lock wins.
|
||||||
#
|
#
|
||||||
# `PR_BOT_TOKEN` (repo secret) is the `flake-bot` account's write token.
|
# `PR_BOT_TOKEN` (repo secret) is the `flake-bot` account's write token.
|
||||||
# Self-signed forge cert: not worked around with `curl -k` - hive-ci.nix
|
# Self-signed forge cert: not worked around with `curl -k` - swarm-ci-runner.nix
|
||||||
# already sets SSL_CERT_FILE to a bundle with the hive CA (wiring
|
# already sets SSL_CERT_FILE to a bundle with the hive CA (wiring
|
||||||
# confirmed; anchor coverage for the swarm forge specifically isn't).
|
# confirmed; anchor coverage for the swarm forge specifically isn't).
|
||||||
on:
|
on:
|
||||||
|
|
|
||||||
|
|
@ -55,7 +55,7 @@ the swarm — one of each, each on whichever host you choose
|
||||||
├── swarm-grafana dashboards over both stores
|
├── swarm-grafana dashboards over both stores
|
||||||
│
|
│
|
||||||
└── opt-in, not part of singleHostSwarm
|
└── opt-in, not part of singleHostSwarm
|
||||||
├── hive-ci Forgejo Actions runner (deploy.forgejo.ci)
|
├── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner)
|
||||||
├── swarm-snapshot-store btrfs-receive endpoint for agent snapshots
|
├── swarm-snapshot-store btrfs-receive endpoint for agent snapshots
|
||||||
└── wg-hive WireGuard mesh between hives (deploy.wireguard)
|
└── wg-hive WireGuard mesh between hives (deploy.wireguard)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -91,7 +91,7 @@ declarations.
|
||||||
[`scheduler/jobq.md`](scheduler/jobq.md) — operator-facing, no implementation detail.
|
[`scheduler/jobq.md`](scheduler/jobq.md) — operator-facing, no implementation detail.
|
||||||
- **How does the rebuild queue work? What are the concrete step kinds,
|
- **How does the rebuild queue work? What are the concrete step kinds,
|
||||||
queue sources, scheduler internals?** → [`scheduler/coordinator.md`](scheduler/coordinator.md).
|
queue sources, scheduler internals?** → [`scheduler/coordinator.md`](scheduler/coordinator.md).
|
||||||
- **How does the CI runner work? What's the autoregistration flow?** →
|
- **How does the CI runner work? How does it register?** →
|
||||||
[`scheduler/ci.md`](scheduler/ci.md).
|
[`scheduler/ci.md`](scheduler/ci.md).
|
||||||
- **How do I export Claude Code metrics (tokens, cost, tool calls) to
|
- **How do I export Claude Code metrics (tokens, cost, tool calls) to
|
||||||
Prometheus/Grafana?** → [`scheduler/observability.md`](scheduler/observability.md).
|
Prometheus/Grafana?** → [`scheduler/observability.md`](scheduler/observability.md).
|
||||||
|
|
|
||||||
|
|
@ -31,12 +31,14 @@ by a test). swarm-controller mints every agent token with it:
|
||||||
| `read:notification` | Poll `GET /notifications` for unread events. |
|
| `read:notification` | Poll `GET /notifications` for unread events. |
|
||||||
| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. |
|
| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. |
|
||||||
|
|
||||||
**`CORE_TOKEN_SCOPES`** (`hive-c0re::forge`, for its own `core` user):
|
**`CORE_TOKEN_SCOPES`** (`hive-c0re::forge`, for its own `core` user): the
|
||||||
everything in `AGENT_TOKEN_SCOPES` plus `read:admin` and `write:admin`.
|
same set as `AGENT_TOKEN_SCOPES`, with no admin scope. Forgejo's token scope
|
||||||
Site-admin membership alone isn't sufficient — Forgejo's token scope gate runs
|
gate runs before the user-permission check, so this token gets
|
||||||
before the user-permission check, so `/api/v1/admin/*` returns
|
`403 Forbidden` on `/api/v1/admin/*` even though `core` is a site admin.
|
||||||
`403 Forbidden` for any token without the admin scope bits, even when
|
hive-c0re only mints the token when the file at
|
||||||
the bearer is a site admin.
|
`/var/lib/hyperhive/forge-core-token` is missing or the forge rejects it, so
|
||||||
|
a token minted with admin scopes keeps them until an operator deletes that
|
||||||
|
file and restarts hive-c0re.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -34,20 +34,20 @@ that touches it.
|
||||||
│ hive-br0 10.42.0.1/24 │
|
│ hive-br0 10.42.0.1/24 │
|
||||||
│ ┌──────────┼──────────────┐ │
|
│ ┌──────────┼──────────────┐ │
|
||||||
└──────────────┼──────────┼──────────────┼───────────────────────────┘
|
└──────────────┼──────────┼──────────────┼───────────────────────────┘
|
||||||
vb-h-<a> vb-h-<b> vb-hive-ci veth pairs
|
vb-h-<a> vb-h-<b> vb-<runner> veth pairs
|
||||||
│ │ │
|
│ │ │
|
||||||
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
|
┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐ one private netns
|
||||||
│ agent a │ │ agent b │ │ hive-ci │ each; eth0 leases
|
│ agent a │ │ agent b │ │ runner │ each; eth0 leases
|
||||||
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
|
│ eth0 │ │ eth0 │ │ eth0 │ from the DHCP pool
|
||||||
└─────────┘ └─────────┘ └─────────┘
|
└─────────┘ └─────────┘ └─────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
| container | netns | IPv4 | listens / reached via |
|
| container | netns | IPv4 | listens / reached via |
|
||||||
| -------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
| ----------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
||||||
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||||||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||||||
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
|
| `hive-matrix` | host (shared) | host addresses | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost |
|
||||||
| `hive-ci` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||||||
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
| `h-<agent>` | private, veth on bridge | DHCP pool | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
|
||||||
|
|
||||||
The flows, end to end:
|
The flows, end to end:
|
||||||
|
|
@ -172,7 +172,7 @@ boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
|
||||||
from the container's (empty) port list, and deletes the host's
|
from the container's (empty) port list, and deletes the host's
|
||||||
`nixos-nat-*` chains without recreating them, silently wiping the
|
`nixos-nat-*` chains without recreating them, silently wiping the
|
||||||
bridge holes above plus the agents' NAT. Private-netns containers
|
bridge holes above plus the agents' NAT. Private-netns containers
|
||||||
(agents, hive-ci) may keep their own firewall — it's scoped to their
|
(agents, swarm-ci-runner) may keep their own firewall — it's scoped to their
|
||||||
namespace.
|
namespace.
|
||||||
|
|
||||||
### Reaching host services (`exposeHostPorts`)
|
### Reaching host services (`exposeHostPorts`)
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,21 @@
|
||||||
# hive-ci: Forgejo Actions Runner
|
# swarm-ci-runner: Forgejo Actions Runner
|
||||||
|
|
||||||
The `hive-ci` module runs a Forgejo Actions runner in a `hive-ci` nixos-container,
|
The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
|
||||||
executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.
|
`swarm-ci-runner` nixos-container, executing CI jobs from
|
||||||
|
`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
|
||||||
|
whichever host sets it, and needs neither the forge nor a hive on that host.
|
||||||
|
|
||||||
## For operators
|
## For operators
|
||||||
|
|
||||||
**Enabling it's one line**: `services.hyperhive.deploy.forgejo.ci.enable = true`
|
**Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
|
||||||
in the host NixOS config. No manual token provisioning — hive-c0re
|
in the NixOS config of the host that should run it. No manual token
|
||||||
registers the runner with the forge automatically.
|
provisioning: swarm-controller publishes the forge's runner registration token
|
||||||
|
to the secret store, and the runner reads it with a store identity of its own.
|
||||||
|
On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
|
||||||
|
`ci-runner.pem` and `ci-runner-key.pem` from the store host's
|
||||||
|
`/var/lib/swarm-bao-pki/` and name them in
|
||||||
|
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
|
||||||
|
`ciRunnerClientKeyFile`.
|
||||||
|
|
||||||
Two things worth knowing before you rely on it:
|
Two things worth knowing before you rely on it:
|
||||||
|
|
||||||
|
|
@ -25,9 +33,8 @@ Two things worth knowing before you rely on it:
|
||||||
disk-pressure GC config from _Host store maintenance_ below to your
|
disk-pressure GC config from _Host store maintenance_ below to your
|
||||||
host's NixOS config (not optional if you plan to actually use this).
|
host's NixOS config (not optional if you plan to actually use this).
|
||||||
|
|
||||||
Everything below this point is implementation detail: exact
|
Everything below this point is implementation detail: registration
|
||||||
autoregistration internals, container design, and the full security
|
internals, container design, and the full security writeup.
|
||||||
writeup.
|
|
||||||
|
|
||||||
## CI checks
|
## CI checks
|
||||||
|
|
||||||
|
|
@ -105,62 +112,68 @@ slow); run those manually before pushing Rust changes.
|
||||||
|
|
||||||
## Configuration reference
|
## Configuration reference
|
||||||
|
|
||||||
The runner registers against the forge on its own host, so enable it on the
|
`services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
|
||||||
swarm's forge host (`services.hyperhive.deploy.forgejo.enable`, see
|
(see _For operators_ above). The runner reaches the forge at
|
||||||
[`../swarm/services.md`](../swarm/services.md)). Evaluation fails on any other
|
`https://<swarm.forge.domain>`, so that name has to resolve from the host: the
|
||||||
host. There, nothing extra to enable beyond
|
gateway's dnsmasq answers it locally where the forge runs, and forwards it to
|
||||||
`services.hyperhive.deploy.forgejo.ci.enable = true` (see _For operators_ above).
|
the operator's DNS everywhere else.
|
||||||
|
|
||||||
Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
|
Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
|
||||||
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
|
name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
|
||||||
targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration
|
(workflow targeting; the default `hive-ci:host` is what this repo's workflows
|
||||||
string for example `"3h"` — a job that exceeds it's killed so a hung or runaway
|
name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
|
||||||
build can't hold the runner's single slot indefinitely).
|
for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
|
||||||
|
can't hold the runner's single slot indefinitely).
|
||||||
|
|
||||||
## Container design
|
## Container design
|
||||||
|
|
||||||
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
|
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
|
||||||
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
|
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
|
||||||
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
|
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`.
|
||||||
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only.
|
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path.
|
||||||
|
|
||||||
## Autoregistration flow
|
## Registration
|
||||||
|
|
||||||
Registration is **off the container's boot-critical path** — hive-c0re owns
|
1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
|
||||||
it and runs it out of band, so a slow forge or core-token never delays the
|
the forge's instance-wide token from
|
||||||
container's start. Gotcha: don't gate the container's own start on a forge
|
`GET /api/v1/admin/runners/registration-token` at start and every five
|
||||||
round-trip (a host-side unit that did this could exceed the nspawn start
|
minutes, and writes it to the secret store at
|
||||||
timeout and trip a restart loop) — registration must stay something
|
`swarm/services/ci-runner/registration-token` when it differs from the
|
||||||
hive-c0re drives after the container is already up. hive-c0re holds the
|
value it last wrote. Its store grant on that path is write-only.
|
||||||
core admin token only on the host; only the runner registration token
|
2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
|
||||||
reaches the container.
|
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
|
||||||
|
and nothing else, retrying every 30 seconds until the token is there. It
|
||||||
|
writes `TOKEN=<value>` to the runner's `EnvironmentFile` inside the
|
||||||
|
container, and restarts the runner when the value changes. It runs at boot
|
||||||
|
and on deploy, not on a timer.
|
||||||
|
3. **The runner unit** (`gitea-runner-swarm` in the container) registers in
|
||||||
|
the upstream pre-start step whenever `.runner` is missing or the token or
|
||||||
|
labels changed. A pre-start step of this module's, ahead of it, removes
|
||||||
|
`.runner` in two cases upstream doesn't check:
|
||||||
|
- The forge has no record of the runner. Forgejo answers every call from an
|
||||||
|
unknown runner with `unregistered runner`, and the daemon exits 1 when its
|
||||||
|
startup `Declare` gets that answer. On the next start the step finds the
|
||||||
|
message in the daemon's journal since its last start, and registers again.
|
||||||
|
A daemon that's already running when an admin deletes its runner keeps polling
|
||||||
|
and logging the error without exiting; the next restart of the unit or
|
||||||
|
container registers it again.
|
||||||
|
- `.runner` names another forge address than the configured one.
|
||||||
|
|
||||||
### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)
|
Until a token arrives the step fails, and `Restart=on-failure` retries every
|
||||||
|
15 seconds with no start limit.
|
||||||
|
|
||||||
Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — hive-c0re logs failures and never aborts the sweep; a healthy runner is never restarted.
|
A new registration is a new runner on the forge: the previous entry stays in
|
||||||
|
the forge admin panel as offline until an admin deletes it.
|
||||||
1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
|
|
||||||
- **200**: still registered — done, no restart.
|
|
||||||
- **404 / other non-200 / malformed**: stale — re-register (below).
|
|
||||||
- **transport error** (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner.
|
|
||||||
2. If absent or stale: mint a fresh token from `GET /api/v1/admin/runners/registration-token`, then hand it to hive-priv's `RegisterCiRunner`, which (as root) writes `TOKEN=<real>` **in place** to the host env-file `/run/hive-ci/runner-token` (preserving the inode nspawn pinned into the container at start) and restarts `gitea-runner-hive.service` inside the container so it picks up the credential and registers.
|
|
||||||
|
|
||||||
### Container side
|
|
||||||
|
|
||||||
- The container boots immediately — nothing gates its start on registration.
|
|
||||||
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
|
|
||||||
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it's already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
|
|
||||||
- **Convergence**: because the token write targets the _host_ file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
|
|
||||||
|
|
||||||
## Actions checkout mirror
|
## Actions checkout mirror
|
||||||
|
|
||||||
<!-- vale write-good.Passive = NO -->
|
<!-- vale write-good.Passive = NO -->
|
||||||
|
|
||||||
When `deploy.forgejo.ci.enable` is set, the swarm controller autoseeds an
|
With `services.hyperhive.deploy.forgejo.actionsFromSelf` set on the forge's
|
||||||
`actions/checkout` pull-mirror on the local forge and sets Forgejo's
|
host, the swarm controller seeds an `actions/checkout` pull-mirror on the forge
|
||||||
`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI
|
and Forgejo's `DEFAULT_ACTIONS_URL` points at the forge itself. CI
|
||||||
`uses: actions/checkout@vN` steps resolve entirely on loopback — no
|
`uses: actions/checkout@vN` steps then resolve on the forge, with no external
|
||||||
external DNS on the CI critical path.
|
DNS on the CI critical path.
|
||||||
|
|
||||||
<!-- vale write-good.Passive = YES -->
|
<!-- vale write-good.Passive = YES -->
|
||||||
|
|
||||||
|
|
@ -192,7 +205,7 @@ to avoid provisioning collisions.
|
||||||
|
|
||||||
## Security: unsandboxed builds and trusted contributors
|
## Security: unsandboxed builds and trusted contributors
|
||||||
|
|
||||||
**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks:
|
**The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:
|
||||||
|
|
||||||
### What unsandboxed builds mean
|
### What unsandboxed builds mean
|
||||||
|
|
||||||
|
|
@ -200,10 +213,10 @@ nspawn containers can't create user-namespaces, so the module sets `nix.settings
|
||||||
|
|
||||||
A malicious `default.nix` or build script in a PR can therefore:
|
A malicious `default.nix` or build script in a PR can therefore:
|
||||||
|
|
||||||
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
|
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
|
||||||
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
|
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
|
||||||
|
|
||||||
The core admin token (`forge-core-token`) **isn't** bind-mounted into the container. it's held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but can't use the admin token to issue privileged API calls.
|
No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which reaches the container's env-file. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
|
||||||
|
|
||||||
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
|
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
|
||||||
|
|
||||||
|
|
@ -221,7 +234,7 @@ The current design is appropriate for a trusted-team hive where all contributors
|
||||||
## Host store maintenance (recommended)
|
## Host store maintenance (recommended)
|
||||||
|
|
||||||
The CI runner builds derivations through the **host** nix-daemon — the
|
The CI runner builds derivations through the **host** nix-daemon — the
|
||||||
hive-ci container shares the host store and has no daemon of its own. Build
|
`swarm-ci-runner` container shares the host store and has no daemon of its own. Build
|
||||||
outputs accumulate in `/nix/store` with no automatic collection, and a busy
|
outputs accumulate in `/nix/store` with no automatic collection, and a busy
|
||||||
CI day can fill the disk until every job fails fast with `ENOSPC`.
|
CI day can fill the disk until every job fails fast with `ENOSPC`.
|
||||||
|
|
||||||
|
|
@ -278,6 +291,7 @@ builder — disk-pressure collection at least runs while nothing is mid-copy.
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
- `nix/host-modules/hive-ci.nix`: runner configuration, autoregistration script, container setup.
|
- `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
|
||||||
|
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
|
||||||
- `.forgejo/workflows/ci.yml`: workflow definition.
|
- `.forgejo/workflows/ci.yml`: workflow definition.
|
||||||
- `docs/process/gotchas.md`: nix sandboxing limitations in containers.
|
- `docs/process/gotchas.md`: nix sandboxing limitations in containers.
|
||||||
|
|
|
||||||
|
|
@ -474,8 +474,8 @@ Two things to know about the weights:
|
||||||
same value and the weight does _not_ rank agents against each other.
|
same value and the weight does _not_ rank agents against each other.
|
||||||
What `80` buys is that agents yield to everything **not** on this
|
What `80` buys is that agents yield to everything **not** on this
|
||||||
drop-in path: host services (nginx and dnsmasq among them) and the
|
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||||
infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,
|
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
|
||||||
`hive-matrix`), which stay at the kernel default of `100`.
|
stay at the kernel default of `100`.
|
||||||
- `IOWeight=` is only honoured when the backing device runs the BFQ
|
- `IOWeight=` is only honoured when the backing device runs the BFQ
|
||||||
scheduler or has blk-iocost QoS enabled. On a host using
|
scheduler or has blk-iocost QoS enabled. On a host using
|
||||||
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
|
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
|
||||||
|
|
|
||||||
|
|
@ -265,7 +265,7 @@ one per hive. It ensures them at start and every five minutes after
|
||||||
whitelists = the `operators` team, and no user. The controller leaves a
|
whitelists = the `operators` team, and no user. The controller leaves a
|
||||||
repo with no `main` rule alone;
|
repo with no `main` rule alone;
|
||||||
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
|
- the pull-mirrors from `deploy.forgejo.mirrors` on the controller's
|
||||||
host (with the `actions/checkout` one `deploy.forgejo.ci.enable` adds);
|
host (with the `actions/checkout` one `deploy.forgejo.actionsFromSelf` adds);
|
||||||
- `internal/docs` (private) and `internal/knowledge` (public, with a
|
- `internal/docs` (private) and `internal/knowledge` (public, with a
|
||||||
seed `README.md` while empty);
|
seed `README.md` while empty);
|
||||||
- the `agent-configs` org avatar
|
- the `agent-configs` org avatar
|
||||||
|
|
|
||||||
|
|
@ -213,7 +213,7 @@ Two consumers, and only one of them is fine:
|
||||||
next to every agent's flake. The bundle is the runtime-to-build-time
|
next to every agent's flake. The bundle is the runtime-to-build-time
|
||||||
bridge.
|
bridge.
|
||||||
- **The Matrix container crosses the same bridge**, via the shared
|
- **The Matrix container crosses the same bridge**, via the shared
|
||||||
`lib/hive-ca-trust.nix` helper that `hive-ci` and `hive-forge` already
|
`lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
|
||||||
use: the bundle is bind-mounted read-only into the container, and a
|
use: the bundle is bind-mounted read-only into the container, and a
|
||||||
oneshot concatenates it with the system CAs before tuwunel starts.
|
oneshot concatenates it with the system CAs before tuwunel starts.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -69,6 +69,7 @@ of the cell says how.
|
||||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
||||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||||
|
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into the runner's `EnvironmentFile`; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed |
|
||||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||||
|
|
||||||
<!-- vale write-good.Passive = YES -->
|
<!-- vale write-good.Passive = YES -->
|
||||||
|
|
|
||||||
|
|
@ -302,7 +302,7 @@ next controller restart.
|
||||||
### Per-principal identities
|
### Per-principal identities
|
||||||
|
|
||||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||||
identity and sharing one merges the identities. These four units read one path
|
identity and sharing one merges the identities. These five units read one path
|
||||||
each and each holds a leaf, a role and a policy of its own:
|
each and each holds a leaf, a role and a policy of its own:
|
||||||
|
|
||||||
| unit | option pair under `deploy.bao.` | reads |
|
| unit | option pair under `deploy.bao.` | reads |
|
||||||
|
|
@ -311,20 +311,23 @@ each and each holds a leaf, a role and a policy of its own:
|
||||||
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
|
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
|
||||||
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
||||||
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
|
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
|
||||||
|
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` |
|
||||||
|
|
||||||
The first exists **per hive**, because the path it reads carries a hive name
|
The first exists **per hive**, because the path it reads carries a hive name
|
||||||
and every hive runs its own reader. Its subject is
|
and every hive runs its own reader. Its subject is
|
||||||
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
||||||
composed spelling as a hive name, so nobody can name a hive into another hive's
|
composed spelling as a hive name, so nobody can name a hive into another hive's
|
||||||
role. The other three read a path that names a swarm service rather than a hive,
|
role. The other four read a path that names a swarm service rather than a hive,
|
||||||
so one role each is enough and their subjects are the flat
|
so one role each is enough and their subjects are the flat
|
||||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName` and
|
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||||
`deploy.bao.forwarderOidcCommonName`.
|
`deploy.bao.forwarderOidcCommonName` and `deploy.bao.ciRunnerCommonName`.
|
||||||
|
|
||||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults
|
On a host that mints its own PKI, `glue-bao-tls.nix` signs all five and defaults
|
||||||
all eight options, and there is nothing to do. Elsewhere you issue each leaf from
|
all ten options, and there is nothing to do. Elsewhere you issue each leaf from
|
||||||
that CA out of band and name it here — one file per principal rather than one file
|
that CA out of band and name it here — one file per principal rather than one file
|
||||||
shared by four, which is the whole of what this buys.
|
shared by five, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||||
|
refuses the build wherever the runner runs without its pair, as
|
||||||
|
`swarm-grafana.nix` does for Grafana.
|
||||||
|
|
||||||
On a host without the store, where the forwarder never runs, forgetting one of
|
On a host without the store, where the forwarder never runs, forgetting one of
|
||||||
the other six isn't a quiet degrade. A hand-configured remote-store hive that
|
the other six isn't a quiet degrade. A hand-configured remote-store hive that
|
||||||
|
|
|
||||||
|
|
@ -221,3 +221,16 @@ has one either way, reached by its swarm name through the gateway.
|
||||||
|
|
||||||
Agent-side configuration, and what a hive's own collector does, are in
|
Agent-side configuration, and what a hive's own collector does, are in
|
||||||
[`../scheduler/observability.md`](../scheduler/observability.md).
|
[`../scheduler/observability.md`](../scheduler/observability.md).
|
||||||
|
|
||||||
|
### CI runner
|
||||||
|
|
||||||
|
One Forgejo Actions runner, in a `swarm-ci-runner` container on the host that
|
||||||
|
sets `deploy.ci-runner.enable`. It doesn't derive from `allSwarmServices`: a
|
||||||
|
runner executes pull-request code, so running one is a decision of its own.
|
||||||
|
|
||||||
|
It needs neither the forge nor a hive on its host. It reaches the forge at
|
||||||
|
`https://<swarm.forge.domain>`, and registers with the token swarm-controller
|
||||||
|
publishes to the secret store, which it reads under its own `ci-runner` leaf.
|
||||||
|
On a host without the store, that leaf is the one file to copy. Registration,
|
||||||
|
the container and the threat model are in
|
||||||
|
[`../scheduler/ci.md`](../scheduler/ci.md).
|
||||||
|
|
|
||||||
|
|
@ -600,7 +600,6 @@ Bare `hivectl stop` stops everything; scope flags narrow it to specific sub-agen
|
||||||
|
|
||||||
* `--agents` — All sub-agent containers
|
* `--agents` — All sub-agent containers
|
||||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||||
* `--ci` — The CI runner container (`hive-ci`)
|
|
||||||
* `--forge` — The forge container (`hive-forge`)
|
* `--forge` — The forge container (`hive-forge`)
|
||||||
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||||
* `--matrix` — The matrix container (`hive-matrix`)
|
* `--matrix` — The matrix container (`hive-matrix`)
|
||||||
|
|
@ -621,7 +620,6 @@ Bare `hivectl start` restores the agents stopped by the last broad-scope `stop`
|
||||||
|
|
||||||
* `--agents` — All sub-agent containers
|
* `--agents` — All sub-agent containers
|
||||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||||
* `--ci` — The CI runner container (`hive-ci`)
|
|
||||||
* `--forge` — The forge container (`hive-forge`)
|
* `--forge` — The forge container (`hive-forge`)
|
||||||
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||||
* `--matrix` — The matrix container (`hive-matrix`)
|
* `--matrix` — The matrix container (`hive-matrix`)
|
||||||
|
|
@ -641,7 +639,6 @@ Bare `hivectl restart` restarts everything; scope flags narrow it.
|
||||||
|
|
||||||
* `--agents` — All sub-agent containers
|
* `--agents` — All sub-agent containers
|
||||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||||
* `--ci` — The CI runner container (`hive-ci`)
|
|
||||||
* `--forge` — The forge container (`hive-forge`)
|
* `--forge` — The forge container (`hive-forge`)
|
||||||
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||||
* `--matrix` — The matrix container (`hive-matrix`)
|
* `--matrix` — The matrix container (`hive-matrix`)
|
||||||
|
|
|
||||||
|
|
@ -56,13 +56,13 @@ Concretely, the core daemon's dashboard `/api` carries **no
|
||||||
application-layer authentication** — the core daemon serves operator-authority
|
application-layer authentication** — the core daemon serves operator-authority
|
||||||
routes unauthenticated at the HTTP layer. Their protection is entirely (a) the
|
routes unauthenticated at the HTTP layer. Their protection is entirely (a) the
|
||||||
gateway, which fronts all operator traffic and is where operator auth lives,
|
gateway, which fronts all operator traffic and is where operator auth lives,
|
||||||
and (b) network isolation, which keeps agents — and `hive-ci`'s untrusted PR
|
and (b) network isolation, which keeps agents — and the CI runner's untrusted
|
||||||
builds — off host-loopback so nothing can reach `127.0.0.1:<dashboard_port>`
|
PR builds — off host-loopback so nothing can reach `127.0.0.1:<dashboard_port>`
|
||||||
directly. This is deliberate given the load-bearing role of network isolation
|
directly. This is deliberate given the load-bearing role of network isolation
|
||||||
above, but it's a standing invariant: the `/api` must never bind to a
|
above, but it's a standing invariant: the `/api` must never bind to a
|
||||||
non-loopback address or get exposed outside the gateway, and every new
|
non-loopback address or get exposed outside the gateway, and every new
|
||||||
operator-authority route inherits that assumption. hyperhive treats `hive-ci`
|
operator-authority route inherits that assumption. hyperhive treats
|
||||||
like an agent for this purpose — it runs untrusted PR code and is
|
`swarm-ci-runner` like an agent for this purpose — it runs untrusted PR code and is
|
||||||
netns-isolated for the same reason.
|
netns-isolated for the same reason.
|
||||||
|
|
||||||
The boundary rests on three layers:
|
The boundary rests on three layers:
|
||||||
|
|
|
||||||
|
|
@ -205,9 +205,9 @@ Unsandboxed nix builds run as `nixbld` users (non-root, typically UIDs 30001-300
|
||||||
Without sandbox isolation, a build derivation's builder script has read access to
|
Without sandbox isolation, a build derivation's builder script has read access to
|
||||||
any file in the container that the nixbld user can read.
|
any file in the container that the nixbld user can read.
|
||||||
|
|
||||||
The blast radius also has a **network** dimension. hive-ci runs its unsandboxed
|
The blast radius also has a **network** dimension. `swarm-ci-runner` runs its
|
||||||
builds of untrusted PR code in its own private netns behind the hive bridge: a
|
unsandboxed builds of untrusted PR code in its own private netns behind the
|
||||||
build reaches the forge only through the gateway and can't reach host-loopback
|
bridge: a build reaches the forge by name and can't reach host-loopback
|
||||||
services — including the core dashboard at `127.0.0.1:<dashboard_port>`, which
|
services — including the core dashboard at `127.0.0.1:<dashboard_port>`, which
|
||||||
has no application-layer auth of its own (see [`docs/scheduler/ci.md`](../scheduler/ci.md)). The `0600`
|
has no application-layer auth of its own (see [`docs/scheduler/ci.md`](../scheduler/ci.md)). The `0600`
|
||||||
token policy bounds file reads; network isolation bounds network reach.
|
token policy bounds file reads; network isolation bounds network reach.
|
||||||
|
|
@ -286,7 +286,6 @@ known operations; there is no arbitrary command pass-through:
|
||||||
| `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) |
|
| `ControlInfraContainer` | `systemctl <action> container@<container>.service` — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re` has no variant, so no request can name it) |
|
||||||
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |
|
||||||
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
| `SetAgentPaused` | create / remove the `<state>/<name>/harness/paused` marker that parks an agent's turn loop |
|
||||||
| `RegisterCiRunner` | write `/run/hive-ci/runner-token` (host path, root-owned) then `systemctl --machine=hive-ci restart gitea-runner-hive.service`. Only the registration token crosses; the forge admin token never enters the container |
|
|
||||||
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
| `EnsureAgentSubvolume` | `btrfs subvolume create <state>/<name>` for a new agent — no-op when the path exists or the filesystem isn't btrfs |
|
||||||
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
| `UpgradeAgentSubvolume` | migrate an existing plain state dir into a subvolume: create, `cp -a --reflink=auto`, atomic swap. Operator opt-in, and the caller stops the agent first |
|
||||||
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
| `DeleteAgentSubvolume` | `btrfs subvolume delete <state>/<name>` — purge path only, never a plain destroy |
|
||||||
|
|
@ -301,7 +300,7 @@ known operations; there is no arbitrary command pass-through:
|
||||||
**Container allowlist** — `hive-priv` validates every request against
|
**Container allowlist** — `hive-priv` validates every request against
|
||||||
an allowlist before any operation: the allowlist accepts only names
|
an allowlist before any operation: the allowlist accepts only names
|
||||||
matching the agent-name convention (char-validated) or the known
|
matching the agent-name convention (char-validated) or the known
|
||||||
sibling service containers (`hive-forge`, `hive-matrix`, `hive-ci`),
|
sibling service containers (`hive-forge`, `hive-matrix`),
|
||||||
and rejects arbitrary container names. `hive-gateway` is a host unit,
|
and rejects arbitrary container names. `hive-gateway` is a host unit,
|
||||||
not a container, so it's not in this list — see `ReloadGatewayNginx`
|
not a container, so it's not in this list — see `ReloadGatewayNginx`
|
||||||
above for how the hard-coded unit name scopes its access instead.
|
above for how the hard-coded unit name scopes its access instead.
|
||||||
|
|
|
||||||
|
|
@ -515,15 +515,15 @@ deep-link directly to a specific agent's journal. A "fetched N ago"
|
||||||
chip appears after the `↻ refresh` button following each successful
|
chip appears after the `↻ refresh` button following each successful
|
||||||
fetch and ticks every 30 s.
|
fetch and ticks every 30 s.
|
||||||
|
|
||||||
**INFRA sub-tab** — journald viewer for the four hive infrastructure
|
**INFRA sub-tab** — journald viewer for the three hive infrastructure
|
||||||
containers (`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`), a
|
containers (`hive-forge`, `hive-gateway`, `hive-matrix`), a
|
||||||
fixed client-side list (`INFRA_NAMES` in `logs.js` — no dashboard API
|
fixed client-side list (`INFRA_NAMES` in `logs.js` — no dashboard API
|
||||||
exposes just the name list). No unit filter (infra containers don't run
|
exposes just the name list). No unit filter (infra containers don't run
|
||||||
the per-agent hive daemons) — always the full machine journal (or, for
|
the per-agent hive daemons) — always the full machine journal (or, for
|
||||||
the gateway, the host journal filtered to its own unit). Fetches
|
the gateway, the host journal filtered to its own unit). Fetches
|
||||||
`GET /api/journal/{name}?lines=500`, same "fetched N ago" ticker as
|
`GET /api/journal/{name}?lines=500`, same "fetched N ago" ticker as
|
||||||
AGENT. A `?agent=<name>` deep-link routes here instead of AGENT when
|
AGENT. A `?agent=<name>` deep-link routes here instead of AGENT when
|
||||||
the name is one of the four infra containers.
|
the name is one of the three infra containers.
|
||||||
|
|
||||||
**SYSTEM sub-tab** — host-side service logs. Unit selector
|
**SYSTEM sub-tab** — host-side service logs. Unit selector
|
||||||
(`hive-c0re.service` / `hive-priv.service`). Fetches
|
(`hive-c0re.service` / `hive-priv.service`). Fetches
|
||||||
|
|
|
||||||
|
|
@ -167,7 +167,6 @@
|
||||||
lib.mkDefault
|
lib.mkDefault
|
||||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swagger-ui-theme;
|
self.packages.${pkgs.stdenv.hostPlatform.system}.swagger-ui-theme;
|
||||||
};
|
};
|
||||||
hive-ci = ./nix/host-modules/hive-ci.nix;
|
|
||||||
hive-forge = ./nix/host-modules/hive-forge;
|
hive-forge = ./nix/host-modules/hive-forge;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,7 @@
|
||||||
/builds.html (the build lifecycle hub). Default tab: K3PT ST4T3.
|
/builds.html (the build lifecycle hub). Default tab: K3PT ST4T3.
|
||||||
The section <div> ids (tombstones-section, container-load-section)
|
The section <div> ids (tombstones-section, container-load-section)
|
||||||
match what core.js's renderers target. Hive infra containers
|
match what core.js's renderers target. Hive infra containers
|
||||||
(`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`) have no
|
(`hive-forge`, `hive-gateway`, `hive-matrix`) have no
|
||||||
dashboard panel — `hivectl stop`/`start`/`restart` is the only
|
dashboard panel — `hivectl stop`/`start`/`restart` is the only
|
||||||
control surface. -->
|
control surface. -->
|
||||||
<header class="page-header">
|
<header class="page-header">
|
||||||
|
|
|
||||||
|
|
@ -60,11 +60,11 @@ select an agent above</pre
|
||||||
>
|
>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<!-- INFRA: journald viewer for infrastructure containers (hive-ci,
|
<!-- INFRA: journald viewer for infrastructure containers (hive-forge,
|
||||||
hive-forge, hive-gateway, hive-matrix). Always fetches the full
|
hive-gateway, hive-matrix). Always fetches the full
|
||||||
machine journal — no unit filter (infra containers don't run the
|
machine journal — no unit filter (infra containers don't run the
|
||||||
per-agent hive daemons). Backed by GET /api/journal/{name}?lines=N.
|
per-agent hive daemons). Backed by GET /api/journal/{name}?lines=N.
|
||||||
Deep-link: ?agent=hive-ci routes here instead of the AGENT tab. -->
|
Deep-link: ?agent=hive-forge routes here instead of the AGENT tab. -->
|
||||||
<section
|
<section
|
||||||
class="logs-pane"
|
class="logs-pane"
|
||||||
id="logs-pane-infra"
|
id="logs-pane-infra"
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
// /logs.html entry point: log viewer with three sub-tabs (AGENT, INFRA,
|
// /logs.html entry point: log viewer with three sub-tabs (AGENT, INFRA,
|
||||||
// SYSTEM). Build log history lives at /builds.html, not here.
|
// SYSTEM). Build log history lives at /builds.html, not here.
|
||||||
// AGENT — per-container journald viewer, backed by GET /api/journal/{name}
|
// AGENT — per-container journald viewer, backed by GET /api/journal/{name}
|
||||||
// INFRA — infra-container journald viewer (hive-ci, hive-forge, …),
|
// INFRA — infra-container journald viewer (hive-forge, hive-gateway, …),
|
||||||
// same API but full machine journal only (no unit filter)
|
// same API but full machine journal only (no unit filter)
|
||||||
// SYSTEM — host service logs, backed by GET /api/journal-host
|
// SYSTEM — host service logs, backed by GET /api/journal-host
|
||||||
//
|
//
|
||||||
|
|
@ -89,7 +89,7 @@ import "@hive/shared/hive-tab-strip.js";
|
||||||
if (agentRefresh) agentRefresh.addEventListener("click", fetchAgent);
|
if (agentRefresh) agentRefresh.addEventListener("click", fetchAgent);
|
||||||
|
|
||||||
// ─── INFRA tab ────────────────────────────────────────────────────────
|
// ─── INFRA tab ────────────────────────────────────────────────────────
|
||||||
// Infra containers (hive-ci, hive-forge, hive-gateway, hive-matrix) don't
|
// Infra containers (hive-forge, hive-gateway, hive-matrix) don't
|
||||||
// run the per-agent hive daemons, so the unit filter is inapplicable. We
|
// run the per-agent hive daemons, so the unit filter is inapplicable. We
|
||||||
// always fetch the full machine journal for them.
|
// always fetch the full machine journal for them.
|
||||||
|
|
||||||
|
|
@ -138,11 +138,11 @@ import "@hive/shared/hive-tab-strip.js";
|
||||||
if (infraSelect) infraSelect.addEventListener("change", fetchInfra);
|
if (infraSelect) infraSelect.addEventListener("change", fetchInfra);
|
||||||
if (infraRefresh) infraRefresh.addEventListener("click", fetchInfra);
|
if (infraRefresh) infraRefresh.addEventListener("click", fetchInfra);
|
||||||
|
|
||||||
// Fixed allowlist — the four hive infra services never change at
|
// Fixed allowlist — the three hive infra services never change at
|
||||||
// runtime, and there's no dashboard API exposing just the name list
|
// runtime, and there's no dashboard API exposing just the name list
|
||||||
// (the one that used to, `/api/state`'s `infra_containers` field, was
|
// (the one that used to, `/api/state`'s `infra_containers` field, was
|
||||||
// start/stop-panel-only and is gone). Mirrors `hive_priv_sock::InfraContainer::ALL`.
|
// start/stop-panel-only and is gone). Mirrors `hive_priv_sock::InfraContainer::ALL`.
|
||||||
const INFRA_NAMES = ["hive-ci", "hive-forge", "hive-gateway", "hive-matrix"];
|
const INFRA_NAMES = ["hive-forge", "hive-gateway", "hive-matrix"];
|
||||||
|
|
||||||
// ─── container list init ──────────────────────────────────────────────
|
// ─── container list init ──────────────────────────────────────────────
|
||||||
// Fetch /api/state once to populate the AGENT selector (agents only);
|
// Fetch /api/state once to populate the AGENT selector (agents only);
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
//! Journal-read endpoints for the dashboard.
|
//! Journal-read endpoints for the dashboard.
|
||||||
//!
|
//!
|
||||||
//! `GET /api/journal/{name}` reads a managed agent container's journal, OR
|
//! `GET /api/journal/{name}` reads a managed agent container's journal, OR
|
||||||
//! one of the four hive infra services (`hive-ci`, `hive-forge`,
|
//! one of the three hive infra services (`hive-forge`, `hive-gateway`,
|
||||||
//! `hive-gateway`, `hive-matrix` — [`hive_priv_sock::InfraContainer`] is the
|
//! `hive-matrix` — [`hive_priv_sock::InfraContainer`] is the
|
||||||
//! allowlist). A container's journal is a `journalctl -M` read, delegated
|
//! allowlist). A container's journal is a `journalctl -M` read, delegated
|
||||||
//! to the root helper since entering a machine needs privileges hive-c0re
|
//! to the root helper since entering a machine needs privileges hive-c0re
|
||||||
//! doesn't have; `hive-gateway` is nginx on the host, so it reads host
|
//! doesn't have; `hive-gateway` is nginx on the host, so it reads host
|
||||||
|
|
@ -42,8 +42,8 @@ pub(super) struct JournalQuery {
|
||||||
/// container namespace and needs root — is delegated to hive-priv.
|
/// container namespace and needs root — is delegated to hive-priv.
|
||||||
///
|
///
|
||||||
/// `name` is either a managed agent name (`iris`, optionally already
|
/// `name` is either a managed agent name (`iris`, optionally already
|
||||||
/// carrying the `h-` prefix) or one of the four infra names (`hive-ci` /
|
/// carrying the `h-` prefix) or one of the three infra names (`hive-forge` /
|
||||||
/// `hive-forge` / `hive-gateway` / `hive-matrix` — see
|
/// `hive-gateway` / `hive-matrix` — see
|
||||||
/// [`hive_priv_sock::InfraContainer`]). Infra targets don't run the
|
/// [`hive_priv_sock::InfraContainer`]). Infra targets don't run the
|
||||||
/// per-agent hive daemons, so `unit` is ignored for them — the whole
|
/// per-agent hive daemons, so `unit` is ignored for them — the whole
|
||||||
/// machine journal, or for the gateway the host journal filtered to its
|
/// machine journal, or for the gateway the host journal filtered to its
|
||||||
|
|
@ -52,7 +52,7 @@ pub(super) struct JournalQuery {
|
||||||
get,
|
get,
|
||||||
path = "/api/journal/{name}",
|
path = "/api/journal/{name}",
|
||||||
params(
|
params(
|
||||||
("name" = String, Path, description = "agent name, or one of the four infra container names"),
|
("name" = String, Path, description = "agent name, or one of the three infra container names"),
|
||||||
JournalQuery,
|
JournalQuery,
|
||||||
),
|
),
|
||||||
responses(
|
responses(
|
||||||
|
|
|
||||||
|
|
@ -1,266 +0,0 @@
|
||||||
//! hive-ci Forgejo Actions runner registration, driven from hive-c0re.
|
|
||||||
//!
|
|
||||||
//! Moves the runner-registration forge round-trip OFF the container's
|
|
||||||
//! boot-critical path (it was a host-side `hive-ci-prefetch` oneshot that
|
|
||||||
//! gated `container@hive-ci` start — see `nix/host-modules/hive-ci.nix`).
|
|
||||||
//! hive-c0re holds the forge admin token; here it validates the runner's
|
|
||||||
//! existing credentials and, when they're absent or stale, mints a fresh
|
|
||||||
//! registration token and hands it to hive-priv, which writes it to the host
|
|
||||||
//! env-file the container bind-mounts read-only and restarts the in-container
|
|
||||||
//! runner. The admin token never enters the container — only the registration
|
|
||||||
//! token does, exactly as the prefetch did.
|
|
||||||
|
|
||||||
use anyhow::{Context as _, Result};
|
|
||||||
use serde_json::Value;
|
|
||||||
use url::Url;
|
|
||||||
|
|
||||||
use super::forge_http_base;
|
|
||||||
|
|
||||||
/// Host path to the hive-ci runner's persisted credentials. The container is
|
|
||||||
/// non-ephemeral, so `.runner` survives restarts; a present file with a sane
|
|
||||||
/// `id` means the runner is already registered.
|
|
||||||
const RUNNER_FILE: &str = "/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner";
|
|
||||||
|
|
||||||
/// Whether the operator enabled the CI runner. The nix module sets
|
|
||||||
/// `HYPERHIVE_FORGE_CI_ENABLED=1` on `hive-c0re.service` when
|
|
||||||
/// `services.hyperhive.deploy.forgejo.ci.enable` is on; absent means CI is off and
|
|
||||||
/// there is no hive-ci container to register a runner for.
|
|
||||||
fn ci_enabled() -> bool {
|
|
||||||
std::env::var("HYPERHIVE_FORGE_CI_ENABLED")
|
|
||||||
.is_ok_and(|v| v == "1" || v.eq_ignore_ascii_case("true"))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Ensure the hive-ci runner is registered against the forge. Best-effort:
|
|
||||||
/// every failure is logged and swallowed so it never aborts the startup
|
|
||||||
/// sweep ([`super::ensure_all`]). No-op when CI is disabled or the runner
|
|
||||||
/// already holds valid credentials (so a healthy runner is never restarted).
|
|
||||||
pub(super) async fn ensure_ci_runner_registered(core_token: &str) {
|
|
||||||
if !ci_enabled() {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
// Existing creds still valid on the forge, AND pointed at the forge
|
|
||||||
// address we are configured for → nothing to do.
|
|
||||||
if let Some(id) = existing_runner_id()
|
|
||||||
&& runner_address_current()
|
|
||||||
&& runner_valid(core_token, id).await
|
|
||||||
{
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
// Absent or stale → mint a fresh registration token and hand it to
|
|
||||||
// hive-priv (root) to write the env-file + restart the runner.
|
|
||||||
match fetch_registration_token(core_token).await {
|
|
||||||
Ok(token) => {
|
|
||||||
if let Err(e) = crate::priv_client::register_ci_runner(&token).await {
|
|
||||||
tracing::warn!(error = ?e, "ci runner: hive-priv register_ci_runner failed");
|
|
||||||
crate::warnings::set_boot_warning(
|
|
||||||
"ci_runner_register",
|
|
||||||
"crit",
|
|
||||||
format!("ci runner: hive-priv register_ci_runner failed: {e}"),
|
|
||||||
);
|
|
||||||
} else {
|
|
||||||
tracing::info!("ci runner: registered hive-ci runner with a fresh token");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(error = ?e, "ci runner: fetch registration token failed");
|
|
||||||
crate::warnings::set_boot_warning(
|
|
||||||
"ci_runner_fetch_token",
|
|
||||||
"crit",
|
|
||||||
format!("ci runner: fetch registration token failed: {e}"),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Parse the runner id from the persisted `.runner` JSON, if present and sane
|
|
||||||
/// (a `0` id is the gitea-runner "unregistered" sentinel).
|
|
||||||
fn existing_runner_id() -> Option<u64> {
|
|
||||||
let raw = std::fs::read_to_string(RUNNER_FILE).ok()?;
|
|
||||||
let json: Value = serde_json::from_str(&raw).ok()?;
|
|
||||||
let id = json.get("id")?.as_u64()?;
|
|
||||||
(id != 0).then_some(id)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Whether the persisted `.runner` still names the forge host we are
|
|
||||||
/// configured for.
|
|
||||||
///
|
|
||||||
/// `act_runner` records the `--instance` URL it was registered with and the
|
|
||||||
/// daemon reads the address from there and nowhere else — the nix option only
|
|
||||||
/// reaches `register`, and upstream re-registers on a changed token or labels,
|
|
||||||
/// never on a changed URL. So a forge rename leaves a runner dialling a name
|
|
||||||
/// its container's `extraHosts` no longer resolves, with every existing check
|
|
||||||
/// still green: the runner id is untouched, so `runner_valid` says yes.
|
|
||||||
///
|
|
||||||
/// Fail-safe in the same direction as [`runner_valid`]: anything we cannot
|
|
||||||
/// read or parse returns `true` (keep the existing creds). Only a *positive*
|
|
||||||
/// disagreement between two parseable hosts is treated as stale, because the
|
|
||||||
/// cost of a false stale is re-registering on every boot.
|
|
||||||
fn runner_address_current() -> bool {
|
|
||||||
let Ok(raw) = std::fs::read_to_string(RUNNER_FILE) else {
|
|
||||||
return true;
|
|
||||||
};
|
|
||||||
runner_address_matches(&raw, forge_http_base())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Host-only comparison of `.runner`'s `address` against the configured base.
|
|
||||||
///
|
|
||||||
/// Split from [`runner_address_current`] so it is testable without the
|
|
||||||
/// filesystem. Compares **hosts**, not whole URLs: both strings come from the
|
|
||||||
/// same nix expression (`http://${swarm.forge.domain}`, reaching the runner as
|
|
||||||
/// `instances.hive.url` and this daemon as `HIVE_FORGE_URL`), but scheme,
|
|
||||||
/// port and trailing slash are exactly the kind of cosmetic difference that
|
|
||||||
/// would otherwise re-register the runner on every boot.
|
|
||||||
fn runner_address_matches(raw: &str, configured: &str) -> bool {
|
|
||||||
let Some(address) = serde_json::from_str::<Value>(raw)
|
|
||||||
.ok()
|
|
||||||
.and_then(|j| j.get("address")?.as_str().map(str::to_owned))
|
|
||||||
else {
|
|
||||||
return true;
|
|
||||||
};
|
|
||||||
match (Url::parse(&address), Url::parse(configured)) {
|
|
||||||
(Ok(a), Ok(b)) => match (a.host_str(), b.host_str()) {
|
|
||||||
(Some(a), Some(b)) => a.eq_ignore_ascii_case(b),
|
|
||||||
_ => true,
|
|
||||||
},
|
|
||||||
_ => true,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Bound on reaching the forge.
|
|
||||||
const HTTP_CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
|
|
||||||
/// Bound on one whole forge admin call, body included.
|
|
||||||
const HTTP_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(15);
|
|
||||||
|
|
||||||
/// A client with both forge bounds applied.
|
|
||||||
fn http_client() -> Result<reqwest::Client> {
|
|
||||||
reqwest::Client::builder()
|
|
||||||
.connect_timeout(HTTP_CONNECT_TIMEOUT)
|
|
||||||
.timeout(HTTP_TIMEOUT)
|
|
||||||
.build()
|
|
||||||
.context("build HTTP client")
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Context for `what` failing with `e`; a timeout names the bound that fired.
|
|
||||||
fn http_context(what: &str, e: &reqwest::Error) -> String {
|
|
||||||
if e.is_connect() && e.is_timeout() {
|
|
||||||
format!("{what}: connect timed out after {HTTP_CONNECT_TIMEOUT:?}")
|
|
||||||
} else if e.is_timeout() {
|
|
||||||
format!("{what}: timed out after {HTTP_TIMEOUT:?}")
|
|
||||||
} else {
|
|
||||||
what.to_owned()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `GET /admin/runners/{id}` — `true` iff the runner still exists on the forge
|
|
||||||
/// (HTTP 200). A 404 (deleted from the admin panel) or any other status means
|
|
||||||
/// re-registration is needed. A transport error (forge unreachable) is treated
|
|
||||||
/// as "keep the existing creds" so a network blip never wipes a valid runner.
|
|
||||||
async fn runner_valid(core_token: &str, id: u64) -> bool {
|
|
||||||
let url = format!("{}/api/v1/admin/runners/{id}", forge_http_base());
|
|
||||||
let client = match http_client() {
|
|
||||||
Ok(c) => c,
|
|
||||||
Err(e) => {
|
|
||||||
tracing::warn!(error = ?e, "ci runner: build HTTP client failed; keeping existing creds");
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
match client
|
|
||||||
.get(&url)
|
|
||||||
.header("Authorization", format!("token {core_token}"))
|
|
||||||
.send()
|
|
||||||
.await
|
|
||||||
{
|
|
||||||
Ok(resp) => resp.status().is_success(),
|
|
||||||
Err(e) => {
|
|
||||||
let what = http_context("validation request failed", &e);
|
|
||||||
tracing::warn!(error = ?e, "ci runner: {what}; keeping existing creds");
|
|
||||||
true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `GET /admin/runners/registration-token` — mint a fresh registration token.
|
|
||||||
/// forgejo-api (0.11) doesn't wrap this endpoint, so this is a raw request
|
|
||||||
/// against the local HTTP forge ([`forge_http_base`]), mirroring the prefetch's
|
|
||||||
/// former curl; Forgejo accepts `Authorization: token <admin>`.
|
|
||||||
async fn fetch_registration_token(core_token: &str) -> Result<String> {
|
|
||||||
let url = format!(
|
|
||||||
"{}/api/v1/admin/runners/registration-token",
|
|
||||||
forge_http_base()
|
|
||||||
);
|
|
||||||
let resp = http_client()?
|
|
||||||
.get(&url)
|
|
||||||
.header("Authorization", format!("token {core_token}"))
|
|
||||||
.send()
|
|
||||||
.await
|
|
||||||
.map_err(|e| {
|
|
||||||
let what = http_context("GET admin/runners/registration-token", &e);
|
|
||||||
anyhow::Error::new(e).context(what)
|
|
||||||
})?;
|
|
||||||
let status = resp.status();
|
|
||||||
let json: Value = resp.json().await.map_err(|e| {
|
|
||||||
let what = http_context("parse registration-token response", &e);
|
|
||||||
anyhow::Error::new(e).context(what)
|
|
||||||
})?;
|
|
||||||
if !status.is_success() {
|
|
||||||
anyhow::bail!("registration-token HTTP {status}: {json}");
|
|
||||||
}
|
|
||||||
json.get("token")
|
|
||||||
.and_then(Value::as_str)
|
|
||||||
.map(str::to_owned)
|
|
||||||
.context("registration-token response missing 'token' field")
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::runner_address_matches;
|
|
||||||
|
|
||||||
fn runner(address: &str) -> String {
|
|
||||||
format!(r#"{{"id":7,"uuid":"u","name":"hive","token":"t","address":"{address}"}}"#)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The case the bug is about: a forge rename leaves the old host behind.
|
|
||||||
#[test]
|
|
||||||
fn differing_host_is_stale() {
|
|
||||||
assert!(!runner_address_matches(
|
|
||||||
&runner("http://forge.pr1ma.darkest.space"),
|
|
||||||
"http://forge.constellation.darkest.space",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Control for the arm above — without it, a helper that always returned
|
|
||||||
/// `false` would pass `differing_host_is_stale` and re-register forever.
|
|
||||||
#[test]
|
|
||||||
fn same_host_is_current() {
|
|
||||||
assert!(runner_address_matches(
|
|
||||||
&runner("http://forge.example"),
|
|
||||||
"http://forge.example",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Cosmetic differences must NOT count: both sides are rendered from one
|
|
||||||
/// nix expression, but a port or trailing slash appearing on one side is
|
|
||||||
/// not a rename.
|
|
||||||
#[test]
|
|
||||||
fn port_scheme_and_trailing_slash_do_not_count() {
|
|
||||||
assert!(runner_address_matches(
|
|
||||||
&runner("http://forge.example:80/"),
|
|
||||||
"https://FORGE.example",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Fail-safe: anything unreadable keeps the existing credentials, so a
|
|
||||||
/// malformed or older `.runner` never costs a working runner.
|
|
||||||
#[test]
|
|
||||||
fn unparseable_input_keeps_creds() {
|
|
||||||
assert!(runner_address_matches("not json", "http://forge.example"));
|
|
||||||
assert!(runner_address_matches(
|
|
||||||
r#"{"id":7}"#,
|
|
||||||
"http://forge.example"
|
|
||||||
));
|
|
||||||
assert!(runner_address_matches(
|
|
||||||
&runner("://nonsense"),
|
|
||||||
"http://forge.example",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -4,7 +4,6 @@
|
||||||
//! teams and repos).
|
//! teams and repos).
|
||||||
//! No-op when `hive-forge` isn't running. Full design: `docs/integrations/forge.md`.
|
//! No-op when `hive-forge` isn't running. Full design: `docs/integrations/forge.md`.
|
||||||
|
|
||||||
mod ci_runner;
|
|
||||||
mod pr_comment;
|
mod pr_comment;
|
||||||
mod reconcile;
|
mod reconcile;
|
||||||
mod repos;
|
mod repos;
|
||||||
|
|
@ -233,8 +232,8 @@ pub async fn sync_agent(name: &str, core_token: Option<&str>) -> bool {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The `core_token.is_some()` half of [`ensure_all`]: the meta repo, the
|
/// The `core_token.is_some()` half of [`ensure_all`]: the meta repo, the
|
||||||
/// local knowledge clone, the core avatar, CI runner registration, and
|
/// local knowledge clone, the core avatar, and removing this hive's own
|
||||||
/// removing this hive's own config-PR hook —
|
/// config-PR hook —
|
||||||
/// every step that needs an authenticated forge client. The swarm-wide
|
/// every step that needs an authenticated forge client. The swarm-wide
|
||||||
/// objects (orgs, the `operators` team, mirrors, `internal/docs`,
|
/// objects (orgs, the `operators` team, mirrors, `internal/docs`,
|
||||||
/// `internal/knowledge`, the `agent-configs` avatar) are the
|
/// `internal/knowledge`, the `agent-configs` avatar) are the
|
||||||
|
|
@ -271,9 +270,6 @@ async fn ensure_all_orgs_and_repos(token: &str) {
|
||||||
format!("forge: core avatar upload failed: {e}"),
|
format!("forge: core avatar upload failed: {e}"),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
// Register the hive-ci Actions runner (off the container's boot path;
|
|
||||||
// no-op when CI is disabled or the runner already holds valid creds).
|
|
||||||
ci_runner::ensure_ci_runner_registered(token).await;
|
|
||||||
if let Err(e) = remove_config_pr_webhook(token).await {
|
if let Err(e) = remove_config_pr_webhook(token).await {
|
||||||
tracing::warn!(error = ?e, "forge: remove_config_pr_webhook failed");
|
tracing::warn!(error = ?e, "forge: remove_config_pr_webhook failed");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -38,13 +38,10 @@ fn core_avatar_png_path() -> std::path::PathBuf {
|
||||||
std::path::PathBuf::from(dir).join("branding/hyperhive.png")
|
std::path::PathBuf::from(dir).join("branding/hyperhive.png")
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Bootstrap `core` token scopes — adds `read:admin,write:admin` on
|
/// Bootstrap `core` token scopes — the same set as swarm-controller's
|
||||||
/// top of the agent scopes (swarm-controller's `AGENT_TOKEN_SCOPES`)
|
/// `AGENT_TOKEN_SCOPES`, with no admin scope: nothing in hive-c0re calls
|
||||||
/// so the host daemon can drive `/api/v1/admin/*`. Site-admin
|
/// `/api/v1/admin/*`. See `docs/integrations/forge.md::Token scopes`.
|
||||||
/// membership alone isn't enough: the token's own scope gate runs
|
const CORE_TOKEN_SCOPES: &str = "read:user,write:user,read:notification,write:notification,write:repository,write:issue,write:organization,write:misc";
|
||||||
/// before the user-permission check.
|
|
||||||
/// See `docs/integrations/forge.md::Token scopes`.
|
|
||||||
const CORE_TOKEN_SCOPES: &str = "read:admin,write:admin,read:user,write:user,read:notification,write:notification,write:repository,write:issue,write:organization,write:misc";
|
|
||||||
|
|
||||||
/// Pull the access token out of forgejo's success message. Format
|
/// Pull the access token out of forgejo's success message. Format
|
||||||
/// has shifted across versions (table form vs. "Access token was
|
/// has shifted across versions (table form vs. "Access token was
|
||||||
|
|
@ -197,9 +194,7 @@ pub(super) async fn ensure_core_avatar(token: &str) -> Result<()> {
|
||||||
/// Outcome of probing whether the persisted core token still works
|
/// Outcome of probing whether the persisted core token still works
|
||||||
/// against the *current* forge. Existence on disk is not validity: a
|
/// against the *current* forge. Existence on disk is not validity: a
|
||||||
/// token minted before a forge rebuild / re-provision is unknown to the
|
/// token minted before a forge rebuild / re-provision is unknown to the
|
||||||
/// new forge's DB and 401s on every call — which silently breaks the
|
/// new forge's DB and 401s on every call.
|
||||||
/// hive-ci runner-registration prefetch (it reads this same token to
|
|
||||||
/// fetch a runner registration token).
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
enum CoreTokenCheck {
|
enum CoreTokenCheck {
|
||||||
/// Token authenticated successfully — keep using it.
|
/// Token authenticated successfully — keep using it.
|
||||||
|
|
@ -260,8 +255,8 @@ async fn check_core_token(token: &str) -> CoreTokenCheck {
|
||||||
|
|
||||||
/// Ensure the bootstrap `core` admin user + a token at
|
/// Ensure the bootstrap `core` admin user + a token at
|
||||||
/// `CORE_TOKEN_PATH`. The token is what hive-c0re uses for forgejo
|
/// `CORE_TOKEN_PATH`. The token is what hive-c0re uses for forgejo
|
||||||
/// API calls (org creation, meta-repo push, and the hive-ci
|
/// API calls (meta-repo push, collaborator grants, PR comments).
|
||||||
/// runner-registration prefetch). Returns the token.
|
/// Returns the token.
|
||||||
///
|
///
|
||||||
/// Idempotent, but validity-aware: when a token file is already present
|
/// Idempotent, but validity-aware: when a token file is already present
|
||||||
/// it is **probed against the current forge** before being trusted. A
|
/// it is **probed against the current forge** before being trusted. A
|
||||||
|
|
|
||||||
|
|
@ -242,7 +242,7 @@ pub fn gateway_agents_conf() -> PathBuf {
|
||||||
/// (used in `Path::new` + user-facing `format!` messages), so it stays a
|
/// (used in `Path::new` + user-facing `format!` messages), so it stays a
|
||||||
/// const rather than a `PathBuf` fn.
|
/// const rather than a `PathBuf` fn.
|
||||||
// nix: read at provisioning; the same literal is in
|
// nix: read at provisioning; the same literal is in
|
||||||
// `nix/host-modules/hive-c0re/default.nix` and `nix/host-modules/hive-ci.nix` — must match.
|
// `nix/host-modules/hive-c0re/default.nix` — must match.
|
||||||
pub const FORGE_CORE_TOKEN: &str = "/var/lib/hyperhive/forge-core-token";
|
pub const FORGE_CORE_TOKEN: &str = "/var/lib/hyperhive/forge-core-token";
|
||||||
|
|
||||||
/// `/run/hyperhive` — the runtime root (host admin socket + per-agent dirs).
|
/// `/run/hyperhive` — the runtime root (host admin socket + per-agent dirs).
|
||||||
|
|
|
||||||
|
|
@ -369,25 +369,8 @@ pub async fn set_agent_paused(agent_name: &str, paused: bool) -> Result<()> {
|
||||||
.await?)
|
.await?)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Register the hive-ci Forgejo Actions runner: hand the freshly-minted
|
/// Start / stop a hive infrastructure service (`hive-gateway`, `hive-forge`,
|
||||||
/// registration token to hive-priv, which writes it to the host-side
|
/// `hive-matrix`) on the host via `systemctl <action> <unit>`,
|
||||||
/// `/run/hive-ci/runner-token` env-file and restarts the in-container runner.
|
|
||||||
/// The forge admin token stays in hive-c0re; only the registration token
|
|
||||||
/// crosses to the (host-path) env-file the container bind-mounts read-only.
|
|
||||||
///
|
|
||||||
/// # Errors
|
|
||||||
/// Propagates the hive-priv call failure: the socket / IPC error, or the
|
|
||||||
/// root-side error when the token is rejected (empty or control characters),
|
|
||||||
/// the env-file write fails, or the runner restart exits non-zero.
|
|
||||||
pub async fn register_ci_runner(token: &str) -> Result<()> {
|
|
||||||
ok(call(&PrivRequest::RegisterCiRunner {
|
|
||||||
token: token.to_owned(),
|
|
||||||
})
|
|
||||||
.await?)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Start / stop a hive infrastructure service (`hive-ci`, `hive-gateway`,
|
|
||||||
/// `hive-forge`, `hive-matrix`) on the host via `systemctl <action> <unit>`,
|
|
||||||
/// where the unit is derived root-side from the variant
|
/// where the unit is derived root-side from the variant
|
||||||
/// (`container@<name>.service`, or `nginx.service` for the gateway). The
|
/// (`container@<name>.service`, or `nginx.service` for the gateway). The
|
||||||
/// [`InfraContainer`] enum is the allowlist — hive-priv needs no name
|
/// [`InfraContainer`] enum is the allowlist — hive-priv needs no name
|
||||||
|
|
|
||||||
|
|
@ -883,9 +883,6 @@ async fn scoped_agents(scope: &LifecycleScope) -> Result<Vec<String>> {
|
||||||
fn scoped_infra(scope: &LifecycleScope) -> Vec<InfraContainer> {
|
fn scoped_infra(scope: &LifecycleScope) -> Vec<InfraContainer> {
|
||||||
let everything = scope.is_everything();
|
let everything = scope.is_everything();
|
||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
if everything || scope.ci {
|
|
||||||
out.push(InfraContainer::Ci);
|
|
||||||
}
|
|
||||||
if everything || scope.forge {
|
if everything || scope.forge {
|
||||||
out.push(InfraContainer::Forge);
|
out.push(InfraContainer::Forge);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -387,12 +387,12 @@ pub struct QuotaRow {
|
||||||
/// (the bare `hivectl stop` / `start`); set individual fields to restrict
|
/// (the bare `hivectl stop` / `start`); set individual fields to restrict
|
||||||
/// (e.g. only `agents` → just the sub-agent containers). `agents` covers
|
/// (e.g. only `agents` → just the sub-agent containers). `agents` covers
|
||||||
/// every managed sub-agent container; the rest are the named infra
|
/// every managed sub-agent container; the rest are the named infra
|
||||||
/// services (`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`) —
|
/// services (`hive-forge`, `hive-gateway`, `hive-matrix`) —
|
||||||
/// containers, bar the gateway, which is the host's nginx unit.
|
/// containers, bar the gateway, which is the host's nginx unit.
|
||||||
//
|
//
|
||||||
// A flat bag of independent flag toggles — one bool per selectable
|
// A flat bag of independent flag toggles — one bool per selectable
|
||||||
// container class — is exactly the right shape here: each maps 1:1 to a
|
// container class — is exactly the right shape here: each maps 1:1 to a
|
||||||
// `hivectl` `--ci` / `--forge` / `--gateway` / `--matrix` flag, and they're
|
// `hivectl` `--forge` / `--gateway` / `--matrix` flag, and they're
|
||||||
// orthogonal (any subset is valid), so a state machine or two-variant enums
|
// orthogonal (any subset is valid), so a state machine or two-variant enums
|
||||||
// would only obscure the mapping. Hence the `struct_excessive_bools` allow.
|
// would only obscure the mapping. Hence the `struct_excessive_bools` allow.
|
||||||
#[allow(clippy::struct_excessive_bools)]
|
#[allow(clippy::struct_excessive_bools)]
|
||||||
|
|
@ -407,8 +407,6 @@ pub struct LifecycleScope {
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub agent_names: Vec<String>,
|
pub agent_names: Vec<String>,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub ci: bool,
|
|
||||||
#[serde(default)]
|
|
||||||
pub forge: bool,
|
pub forge: bool,
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub gateway: bool,
|
pub gateway: bool,
|
||||||
|
|
@ -421,8 +419,7 @@ impl LifecycleScope {
|
||||||
/// classes" (the bare `hivectl stop` / `start` with no scope flags).
|
/// classes" (the bare `hivectl stop` / `start` with no scope flags).
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn is_everything(&self) -> bool {
|
pub fn is_everything(&self) -> bool {
|
||||||
!(self.agents || self.ci || self.forge || self.gateway || self.matrix)
|
!(self.agents || self.forge || self.gateway || self.matrix) && self.agent_names.is_empty()
|
||||||
&& self.agent_names.is_empty()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,7 @@ pub const AGENT_PREFIX: &str = "h-";
|
||||||
/// [`PrivRequest::ControlInfraContainer`], and it has one variant more than
|
/// [`PrivRequest::ControlInfraContainer`], and it has one variant more than
|
||||||
/// this list (the gateway). Keep the distinction — a name that belongs to
|
/// this list (the gateway). Keep the distinction — a name that belongs to
|
||||||
/// no container has no business reaching a `-M` / `nixos-container` call.
|
/// no container has no business reaching a `-M` / `nixos-container` call.
|
||||||
pub const SIBLING_CONTAINERS: &[&str] = &["hive-forge", "hive-matrix", "hive-ci"];
|
pub const SIBLING_CONTAINERS: &[&str] = &["hive-forge", "hive-matrix"];
|
||||||
|
|
||||||
/// Lifecycle verb for [`PrivRequest::ControlInfraContainer`]. Maps directly
|
/// Lifecycle verb for [`PrivRequest::ControlInfraContainer`]. Maps directly
|
||||||
/// to `systemctl <verb> <unit>`, where the unit comes from
|
/// to `systemctl <verb> <unit>`, where the unit comes from
|
||||||
|
|
@ -75,14 +75,13 @@ impl InfraAction {
|
||||||
/// unknown or unsafe target — notably `hive-c0re`, which has no variant and
|
/// unknown or unsafe target — notably `hive-c0re`, which has no variant and
|
||||||
/// would sever the daemon socket — is *unrepresentable* rather than caught
|
/// would sever the daemon socket — is *unrepresentable* rather than caught
|
||||||
/// by a runtime check. The c0re↔hive-priv wire form uses serde's default
|
/// by a runtime check. The c0re↔hive-priv wire form uses serde's default
|
||||||
/// variant naming (`"Ci"`, `"Forge"`, …); it's an internal protocol (both
|
/// variant naming (`"Forge"`, `"Gateway"`, …); it's an internal protocol (both
|
||||||
/// ends rebuild together) so it needn't match the container name.
|
/// ends rebuild together) so it needn't match the container name.
|
||||||
/// [`name`](Self::name) is the separate stable identity string
|
/// [`name`](Self::name) is the separate stable identity string
|
||||||
/// (`hive-ci`), and [`service_unit`](Self::service_unit) the systemd unit
|
/// (`hive-forge`), and [`service_unit`](Self::service_unit) the systemd unit
|
||||||
/// it actually resolves to.
|
/// it actually resolves to.
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
pub enum InfraContainer {
|
pub enum InfraContainer {
|
||||||
Ci,
|
|
||||||
Forge,
|
Forge,
|
||||||
Gateway,
|
Gateway,
|
||||||
Matrix,
|
Matrix,
|
||||||
|
|
@ -108,23 +107,21 @@ impl InfraContainer {
|
||||||
/// Every controllable infra target. A superset of
|
/// Every controllable infra target. A superset of
|
||||||
/// [`SIBLING_CONTAINERS`] — not all of these are containers (see the
|
/// [`SIBLING_CONTAINERS`] — not all of these are containers (see the
|
||||||
/// test).
|
/// test).
|
||||||
pub const ALL: [InfraContainer; 4] = [
|
pub const ALL: [InfraContainer; 3] = [
|
||||||
InfraContainer::Ci,
|
|
||||||
InfraContainer::Forge,
|
InfraContainer::Forge,
|
||||||
InfraContainer::Gateway,
|
InfraContainer::Gateway,
|
||||||
InfraContainer::Matrix,
|
InfraContainer::Matrix,
|
||||||
];
|
];
|
||||||
|
|
||||||
/// Stable identity string, e.g. `hive-ci`. This is what the operator
|
/// Stable identity string, e.g. `hive-forge`. This is what the operator
|
||||||
/// types, what the dashboard displays, and what [`FromStr`](std::str::FromStr)
|
/// types, what the dashboard displays, and what [`FromStr`](std::str::FromStr)
|
||||||
/// parses — it stays `hive-gateway` even though the gateway is no
|
/// parses — it stays `hive-gateway` even though the gateway is no
|
||||||
/// longer a container, because it names the *service*, not its
|
/// longer a container, because it names the *service*, not its
|
||||||
/// implementation. (Distinct from the serde wire form, which is the
|
/// implementation. (Distinct from the serde wire form, which is the
|
||||||
/// default variant name `"Ci"`.)
|
/// default variant name `"Forge"`.)
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn name(self) -> &'static str {
|
pub fn name(self) -> &'static str {
|
||||||
match self {
|
match self {
|
||||||
InfraContainer::Ci => "hive-ci",
|
|
||||||
InfraContainer::Forge => "hive-forge",
|
InfraContainer::Forge => "hive-forge",
|
||||||
InfraContainer::Gateway => "hive-gateway",
|
InfraContainer::Gateway => "hive-gateway",
|
||||||
InfraContainer::Matrix => "hive-matrix",
|
InfraContainer::Matrix => "hive-matrix",
|
||||||
|
|
@ -153,7 +150,7 @@ impl InfraContainer {
|
||||||
impl std::str::FromStr for InfraContainer {
|
impl std::str::FromStr for InfraContainer {
|
||||||
type Err = ();
|
type Err = ();
|
||||||
|
|
||||||
/// Parse an infra name (`hive-ci`, …) into a variant. `Err(())` for
|
/// Parse an infra name (`hive-forge`, …) into a variant. `Err(())` for
|
||||||
/// anything that isn't one.
|
/// anything that isn't one.
|
||||||
fn from_str(s: &str) -> Result<Self, ()> {
|
fn from_str(s: &str) -> Result<Self, ()> {
|
||||||
Self::ALL.into_iter().find(|c| c.name() == s).ok_or(())
|
Self::ALL.into_iter().find(|c| c.name() == s).ok_or(())
|
||||||
|
|
@ -478,19 +475,6 @@ pub enum PrivRequest {
|
||||||
paused: bool,
|
paused: bool,
|
||||||
},
|
},
|
||||||
|
|
||||||
/// Register the hive-ci Forgejo Actions runner: write the registration
|
|
||||||
/// token to the host-side `/run/hive-ci/runner-token` env-file (root-owned,
|
|
||||||
/// bind-mounted read-only into the container) as `TOKEN=<token>`, then
|
|
||||||
/// `systemctl --machine=hive-ci restart gitea-runner-hive.service` so the
|
|
||||||
/// runner picks up the credential. hive-c0re holds the forge admin token
|
|
||||||
/// and mints the registration token; only the registration token is written
|
|
||||||
/// here, and only to a host path — the admin token never enters the
|
|
||||||
/// container. The token is validated single-line + non-empty root-side.
|
|
||||||
RegisterCiRunner {
|
|
||||||
/// Forge runner registration token minted by hive-c0re.
|
|
||||||
token: String,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// Start / stop a hive infrastructure container on the host via
|
/// Start / stop a hive infrastructure container on the host via
|
||||||
/// `systemctl <action> container@<container>.service`. The
|
/// `systemctl <action> container@<container>.service`. The
|
||||||
/// [`InfraContainer`] enum is the allowlist — serde rejects unknown /
|
/// [`InfraContainer`] enum is the allowlist — serde rejects unknown /
|
||||||
|
|
@ -810,8 +794,8 @@ mod tests {
|
||||||
#[test]
|
#[test]
|
||||||
fn service_unit_wraps_containers_but_not_host_units() {
|
fn service_unit_wraps_containers_but_not_host_units() {
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
InfraContainer::Ci.service_unit(),
|
InfraContainer::Forge.service_unit(),
|
||||||
"container@hive-ci.service"
|
"container@hive-forge.service"
|
||||||
);
|
);
|
||||||
assert_eq!(InfraContainer::Gateway.service_unit(), "nginx.service");
|
assert_eq!(InfraContainer::Gateway.service_unit(), "nginx.service");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -407,8 +407,6 @@ async fn exec(
|
||||||
paused,
|
paused,
|
||||||
} => exec_set_agent_paused(agent_name, paused),
|
} => exec_set_agent_paused(agent_name, paused),
|
||||||
|
|
||||||
PrivRequest::RegisterCiRunner { ref token } => register_ci_runner(token).await,
|
|
||||||
|
|
||||||
PrivRequest::ControlInfraContainer { container, action } => {
|
PrivRequest::ControlInfraContainer { container, action } => {
|
||||||
control_infra_container(container, action).await
|
control_infra_container(container, action).await
|
||||||
}
|
}
|
||||||
|
|
@ -1167,107 +1165,6 @@ async fn daemon_reload() -> Result<(String, String)> {
|
||||||
Ok((String::new(), String::new()))
|
Ok((String::new(), String::new()))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Host path to the hive-ci runner's persisted registration credentials.
|
|
||||||
///
|
|
||||||
/// Paired with `hive-c0re`'s `forge::ci_runner::RUNNER_FILE`, which reads the
|
|
||||||
/// same file to decide whether a runner is registered and whether it still
|
|
||||||
/// names the configured forge host. Deliberately duplicated rather than shared:
|
|
||||||
/// `hive-priv` is the minimal root helper and does not depend on `hive-c0re`.
|
|
||||||
const RUNNER_CREDENTIALS: &str =
|
|
||||||
"/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner";
|
|
||||||
|
|
||||||
/// Delete the runner's persisted credentials so upstream's `ExecStartPre` takes
|
|
||||||
/// its **absence** branch on the next start.
|
|
||||||
///
|
|
||||||
/// Absence is the state we want, so `NotFound` is success. Anything else — a
|
|
||||||
/// permission error above all — is NOT swallowed: it means the file is still
|
|
||||||
/// there, the restart will take upstream's already-registered branch, and the
|
|
||||||
/// caller would return `Ok` for a registration that never happened. That is the
|
|
||||||
/// same shape as a precondition that "passes" because it could not read the file
|
|
||||||
/// it was checking, and it is worth failing loudly to avoid.
|
|
||||||
///
|
|
||||||
/// Split from [`register_ci_runner`] purely so this rule is testable without a
|
|
||||||
/// container or a `systemctl`.
|
|
||||||
fn clear_runner_credentials(path: &str) -> Result<()> {
|
|
||||||
match std::fs::remove_file(path) {
|
|
||||||
Ok(()) => Ok(()),
|
|
||||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
|
|
||||||
Err(e) => {
|
|
||||||
Err(anyhow::Error::new(e).context(format!("remove stale runner credentials {path}")))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `RegisterCiRunner` — write the runner registration token to the host-side
|
|
||||||
/// `/run/hive-ci/runner-token` env-file, then restart the in-container runner
|
|
||||||
/// so it re-registers. The forge admin token never enters the container; only
|
|
||||||
/// the registration token c0re passes here is written, and it lands on a host
|
|
||||||
/// path bind-mounted read-only into hive-ci.
|
|
||||||
async fn register_ci_runner(token: &str) -> Result<(String, String)> {
|
|
||||||
use std::io::Write as _;
|
|
||||||
use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _};
|
|
||||||
// Reject anything that could corrupt the `KEY=VALUE` env-file or smuggle a
|
|
||||||
// second line — a forge registration token is an opaque single-line string.
|
|
||||||
if token.is_empty() || token.contains(['\n', '\r', '\0']) {
|
|
||||||
bail!("ci runner registration token empty or contains control characters");
|
|
||||||
}
|
|
||||||
let token_path = "/run/hive-ci/runner-token";
|
|
||||||
// In-place truncate+write of the existing inode (mirrors the prefetch's
|
|
||||||
// `echo > $FILE`), NOT a temp+rename: nspawn pins this file's inode into
|
|
||||||
// hive-ci at container start, so a rename would leave the running runner
|
|
||||||
// reading the old content. Format + perms match the tmpfiles seed and the
|
|
||||||
// prefetch: `TOKEN=<tok>`, mode 0600, root-owned. Created 0600 when the
|
|
||||||
// seed is missing, so the token is never on disk at a wider mode.
|
|
||||||
std::fs::OpenOptions::new()
|
|
||||||
.write(true)
|
|
||||||
.create(true)
|
|
||||||
.truncate(true)
|
|
||||||
.mode(0o600)
|
|
||||||
.open(token_path)
|
|
||||||
.and_then(|mut f| f.write_all(format!("TOKEN={token}\n").as_bytes()))
|
|
||||||
.with_context(|| format!("write {token_path}"))?;
|
|
||||||
std::fs::set_permissions(token_path, std::fs::Permissions::from_mode(0o600))
|
|
||||||
.with_context(|| format!("chmod {token_path}"))?;
|
|
||||||
// Remove the persisted credentials BEFORE restarting, or the restart is a
|
|
||||||
// no-op as far as registration goes.
|
|
||||||
//
|
|
||||||
// Upstream's `ExecStartPre` only re-registers when `.runner` is absent, the
|
|
||||||
// labels changed, or the *registration token hash* changed — never when the
|
|
||||||
// instance URL changed. c0re only calls this helper once it has already
|
|
||||||
// decided the existing credentials are absent or stale
|
|
||||||
// (`forge::ci_runner::ensure_ci_runner_registered` returns early otherwise),
|
|
||||||
// so by the time we are here a re-registration is exactly what is wanted and
|
|
||||||
// deleting the file is the narrow way to guarantee it happens.
|
|
||||||
//
|
|
||||||
// Writing a fresh token is NOT sufficient on its own: whether the hash
|
|
||||||
// changes depends on whether the forge mints a new registration token per
|
|
||||||
// request or hands back a stable one, which is Forgejo's behaviour to
|
|
||||||
// choose and change. Gating our remediation on the absence branch — the one
|
|
||||||
// upstream evaluates unconditionally — makes that question moot instead of
|
|
||||||
// load-bearing.
|
|
||||||
//
|
|
||||||
// See [`clear_runner_credentials`] for why absence is the branch we aim at
|
|
||||||
// and why only `NotFound` counts as success.
|
|
||||||
clear_runner_credentials(RUNNER_CREDENTIALS)?;
|
|
||||||
// Restart the in-container runner so it reads the new token and registers.
|
|
||||||
let out = Command::new("systemctl")
|
|
||||||
.args(["--machine=hive-ci", "restart", "gitea-runner-hive.service"])
|
|
||||||
.output()
|
|
||||||
.await
|
|
||||||
.context("systemctl restart gitea-runner-hive.service in hive-ci")?;
|
|
||||||
if !out.status.success() {
|
|
||||||
bail!(
|
|
||||||
"systemctl restart gitea-runner-hive.service in hive-ci exited {}: {}",
|
|
||||||
out.status,
|
|
||||||
String::from_utf8_lossy(&out.stderr).trim()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
Ok((
|
|
||||||
String::from_utf8_lossy(&out.stdout).into_owned(),
|
|
||||||
String::from_utf8_lossy(&out.stderr).into_owned(),
|
|
||||||
))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `ControlInfraContainer` — start/stop a hive infrastructure service via
|
/// `ControlInfraContainer` — start/stop a hive infrastructure service via
|
||||||
/// `systemctl <verb> <unit>`. The [`InfraContainer`] enum is the allowlist:
|
/// `systemctl <verb> <unit>`. The [`InfraContainer`] enum is the allowlist:
|
||||||
/// serde already rejected any unknown / unsafe name (hive-c0re has no
|
/// serde already rejected any unknown / unsafe name (hive-c0re has no
|
||||||
|
|
@ -3196,12 +3093,11 @@ fn ensure_socket_dir_in(root: &Path, name: &str) -> Result<()> {
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{
|
use super::{
|
||||||
BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile,
|
BindMount, BoundedRun, OwnedFd, PAUSED_MARKER_FILE, PrivRequest, StagedFile,
|
||||||
check_fd_agreement, clear_runner_credentials, contains_secret_shaped_run,
|
check_fd_agreement, contains_secret_shaped_run, describe_forge_admin,
|
||||||
describe_forge_admin, ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags,
|
ensure_plain_filename, ensure_socket_dir_in, git_overlay_flags, limits_dropin_body,
|
||||||
limits_dropin_body, open_dir, open_export_dest, partial_name, publish_file,
|
open_dir, open_export_dest, partial_name, publish_file, redact_secret_line,
|
||||||
redact_secret_line, remove_marker_in, run_bounded, single_output_path, toplevel_attr,
|
remove_marker_in, run_bounded, single_output_path, toplevel_attr, validate_credential_name,
|
||||||
validate_credential_name, validate_snapshot_name, write_agent_dir_file,
|
validate_snapshot_name, write_agent_dir_file, write_bridge_dns_marker_in,
|
||||||
write_bridge_dns_marker_in,
|
|
||||||
};
|
};
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use std::sync::atomic::{AtomicU32, Ordering};
|
use std::sync::atomic::{AtomicU32, Ordering};
|
||||||
|
|
@ -4035,60 +3931,6 @@ mod tests {
|
||||||
assert!(serde_json::from_str::<PrivRequest>(r#"{"op":"sync_agent_tmpfilez"}"#).is_err());
|
assert!(serde_json::from_str::<PrivRequest>(r#"{"op":"sync_agent_tmpfilez"}"#).is_err());
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The runner-credential clear, in all three states that matter. The
|
|
||||||
/// PRESENCE arm is the load-bearing one: an implementation that did nothing
|
|
||||||
/// at all would pass the "absent is fine" arm perfectly, and the whole point
|
|
||||||
/// of the call is that the file is *gone* afterwards — upstream re-registers
|
|
||||||
/// on absence and on nothing else we control.
|
|
||||||
#[test]
|
|
||||||
fn clearing_runner_credentials_removes_it_and_tolerates_absence() {
|
|
||||||
let dir = scratch();
|
|
||||||
let path = dir.join(".runner");
|
|
||||||
let path_str = path.to_str().unwrap();
|
|
||||||
|
|
||||||
// Absent → success (this is the state we are aiming for).
|
|
||||||
assert!(!path.exists());
|
|
||||||
clear_runner_credentials(path_str).unwrap();
|
|
||||||
|
|
||||||
// Present → success AND actually gone. Without this arm a no-op passes.
|
|
||||||
std::fs::write(&path, r#"{"id":7,"address":"http://old.invalid"}"#).unwrap();
|
|
||||||
assert!(
|
|
||||||
path.exists(),
|
|
||||||
"control: the file must exist before the clear"
|
|
||||||
);
|
|
||||||
clear_runner_credentials(path_str).unwrap();
|
|
||||||
assert!(
|
|
||||||
!path.exists(),
|
|
||||||
"stale credentials must be GONE, or the restart takes upstream's \
|
|
||||||
already-registered branch and registration silently never happens"
|
|
||||||
);
|
|
||||||
|
|
||||||
std::fs::remove_dir_all(&dir).ok();
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A failure that is not `NotFound` must propagate, never read as success.
|
|
||||||
/// A directory in the file's place makes `remove_file` fail with a non-
|
|
||||||
/// `NotFound` error without needing to drop privileges in a test.
|
|
||||||
#[test]
|
|
||||||
fn clearing_runner_credentials_propagates_a_real_failure() {
|
|
||||||
let dir = scratch();
|
|
||||||
let path = dir.join(".runner");
|
|
||||||
std::fs::create_dir(&path).unwrap();
|
|
||||||
|
|
||||||
let err = clear_runner_credentials(path.to_str().unwrap())
|
|
||||||
.expect_err("a non-NotFound failure must NOT be reported as success");
|
|
||||||
assert!(
|
|
||||||
format!("{err:#}").contains("remove stale runner credentials"),
|
|
||||||
"error must name what it failed to do, got: {err:#}"
|
|
||||||
);
|
|
||||||
assert!(
|
|
||||||
path.exists(),
|
|
||||||
"nothing was removed, and the caller must know"
|
|
||||||
);
|
|
||||||
|
|
||||||
std::fs::remove_dir_all(&dir).ok();
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The pause marker round-trips through the same root-only path the
|
/// The pause marker round-trips through the same root-only path the
|
||||||
/// credential writes use, and BOTH directions are idempotent — the
|
/// credential writes use, and BOTH directions are idempotent — the
|
||||||
/// dashboard toggle and `hivectl pause|resume` fire blind, without
|
/// dashboard toggle and `hivectl pause|resume` fire blind, without
|
||||||
|
|
|
||||||
|
|
@ -202,9 +202,6 @@ pub struct ScopeArgs {
|
||||||
/// A specific sub-agent by name. Repeatable: `--agent a --agent b`.
|
/// A specific sub-agent by name. Repeatable: `--agent a --agent b`.
|
||||||
#[arg(long = "agent", value_name = "NAME")]
|
#[arg(long = "agent", value_name = "NAME")]
|
||||||
agent: Vec<String>,
|
agent: Vec<String>,
|
||||||
/// The CI runner container (`hive-ci`).
|
|
||||||
#[arg(long)]
|
|
||||||
ci: bool,
|
|
||||||
/// The forge container (`hive-forge`).
|
/// The forge container (`hive-forge`).
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
forge: bool,
|
forge: bool,
|
||||||
|
|
@ -221,7 +218,6 @@ impl ScopeArgs {
|
||||||
hive_host_sock::LifecycleScope {
|
hive_host_sock::LifecycleScope {
|
||||||
agents: self.agents,
|
agents: self.agents,
|
||||||
agent_names: self.agent.clone(),
|
agent_names: self.agent.clone(),
|
||||||
ci: self.ci,
|
|
||||||
forge: self.forge,
|
forge: self.forge,
|
||||||
gateway: self.gateway,
|
gateway: self.gateway,
|
||||||
matrix: self.matrix,
|
matrix: self.matrix,
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,6 @@ fn single_agent_scope(name: &str) -> hive_host_sock::LifecycleScope {
|
||||||
hive_host_sock::LifecycleScope {
|
hive_host_sock::LifecycleScope {
|
||||||
agents: false,
|
agents: false,
|
||||||
agent_names: vec![name.to_owned()],
|
agent_names: vec![name.to_owned()],
|
||||||
ci: false,
|
|
||||||
forge: false,
|
forge: false,
|
||||||
gateway: false,
|
gateway: false,
|
||||||
matrix: false,
|
matrix: false,
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,6 @@
|
||||||
./deploy.nix
|
./deploy.nix
|
||||||
./local-defaults.nix
|
./local-defaults.nix
|
||||||
./hive-c0re
|
./hive-c0re
|
||||||
./hive-ci.nix
|
|
||||||
./hive-forge
|
./hive-forge
|
||||||
./hive-gateway
|
./hive-gateway
|
||||||
./hive-matrix-service.nix
|
./hive-matrix-service.nix
|
||||||
|
|
@ -49,6 +48,7 @@
|
||||||
./swarm-bao-service.nix
|
./swarm-bao-service.nix
|
||||||
./swarm-bao.nix
|
./swarm-bao.nix
|
||||||
./swarm-ca.nix
|
./swarm-ca.nix
|
||||||
|
./swarm-ci-runner.nix
|
||||||
./swarm-secret-publisher.nix
|
./swarm-secret-publisher.nix
|
||||||
./swarm-nats-service.nix
|
./swarm-nats-service.nix
|
||||||
./swarm-nats.nix
|
./swarm-nats.nix
|
||||||
|
|
|
||||||
|
|
@ -20,9 +20,9 @@
|
||||||
#
|
#
|
||||||
# ⚠️ Each entry is an attrset with an `enable`, not a bare bool, so a
|
# ⚠️ Each entry is an attrset with an `enable`, not a bare bool, so a
|
||||||
# service that grows a second *deployment* decision has somewhere to put
|
# service that grows a second *deployment* decision has somewhere to put
|
||||||
# it — `deploy.forgejo = { enable; ci; }` is then an ordinary addition
|
# it — `deploy.nats = { enable; autoGenerateCallout; }` is then an ordinary
|
||||||
# rather than a migration. `ci` ("does this host run the runner too") is
|
# addition rather than a migration, where a bare bool leaves it
|
||||||
# exactly that shape, and a bare bool leaves it unrepresentable.
|
# unrepresentable.
|
||||||
#
|
#
|
||||||
# ⚠️ The renames below are deliberately in this one file rather than
|
# ⚠️ The renames below are deliberately in this one file rather than
|
||||||
# spread across the service modules, so all of them have a single home
|
# spread across the service modules, so all of them have a single home
|
||||||
|
|
@ -31,6 +31,32 @@
|
||||||
{ lib, config, ... }:
|
{ lib, config, ... }:
|
||||||
let
|
let
|
||||||
deployCfg = config.services.hyperhive.deploy;
|
deployCfg = config.services.hyperhive.deploy;
|
||||||
|
|
||||||
|
# A removed CI-runner option. Its replacement has the same name under
|
||||||
|
# `deploy.ci-runner`.
|
||||||
|
ciRunnerRemoved =
|
||||||
|
from: opt:
|
||||||
|
lib.mkRemovedOptionModule
|
||||||
|
(
|
||||||
|
[
|
||||||
|
"services"
|
||||||
|
"hyperhive"
|
||||||
|
]
|
||||||
|
++ from
|
||||||
|
++ [ opt ]
|
||||||
|
)
|
||||||
|
''
|
||||||
|
The CI runner is the swarm service in swarm-ci-runner.nix. It runs on
|
||||||
|
any host, needs neither the forge nor a hive there, and registers with
|
||||||
|
the token swarm-controller publishes to the secret store. Set
|
||||||
|
services.hyperhive.deploy.ci-runner.${opt} instead.
|
||||||
|
|
||||||
|
The runner registers as a new runner on the forge, so delete the old
|
||||||
|
one in the forge's admin panel (Site administration, Actions, Runners).
|
||||||
|
To keep resolving `uses: actions/checkout` from the forge, set
|
||||||
|
services.hyperhive.deploy.forgejo.actionsFromSelf = true on its host.
|
||||||
|
See docs/scheduler/ci.md.
|
||||||
|
'';
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
|
|
@ -146,33 +172,20 @@ in
|
||||||
[ "services" "hyperhive" "deploy" "swarm-otel" "clientSecretFile" ]
|
[ "services" "hyperhive" "deploy" "swarm-otel" "clientSecretFile" ]
|
||||||
)
|
)
|
||||||
|
|
||||||
# The CI runner, and the only entry here that renames more than an
|
# Removed rather than renamed: the runner is a different container with a
|
||||||
# `enable`: every knob under it describes the runner THIS host would run,
|
# registration of its own, so an old definition is a deployment to re-read
|
||||||
# so leaving `name`/`concurrency`/`labels`/`package` in the namespace that
|
# rather than a path to forward.
|
||||||
# must be identical swarm-wide would keep the original defect for four
|
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "enable")
|
||||||
# more options. Renamed one by one because `ci` is a plain attrset of
|
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "name")
|
||||||
# options rather than a submodule type, so there is no parent path to
|
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "concurrency")
|
||||||
# rename in a single entry.
|
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "labels")
|
||||||
(lib.mkRenamedOptionModule
|
(ciRunnerRemoved [ "swarm" "forge" "ci" ] "package")
|
||||||
[ "services" "hyperhive" "swarm" "forge" "ci" "enable" ]
|
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "enable")
|
||||||
[ "services" "hyperhive" "deploy" "forgejo" "ci" "enable" ]
|
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "name")
|
||||||
)
|
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "concurrency")
|
||||||
(lib.mkRenamedOptionModule
|
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "labels")
|
||||||
[ "services" "hyperhive" "swarm" "forge" "ci" "name" ]
|
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "package")
|
||||||
[ "services" "hyperhive" "deploy" "forgejo" "ci" "name" ]
|
(ciRunnerRemoved [ "deploy" "forgejo" "ci" ] "jobTimeout")
|
||||||
)
|
|
||||||
(lib.mkRenamedOptionModule
|
|
||||||
[ "services" "hyperhive" "swarm" "forge" "ci" "concurrency" ]
|
|
||||||
[ "services" "hyperhive" "deploy" "forgejo" "ci" "concurrency" ]
|
|
||||||
)
|
|
||||||
(lib.mkRenamedOptionModule
|
|
||||||
[ "services" "hyperhive" "swarm" "forge" "ci" "labels" ]
|
|
||||||
[ "services" "hyperhive" "deploy" "forgejo" "ci" "labels" ]
|
|
||||||
)
|
|
||||||
(lib.mkRenamedOptionModule
|
|
||||||
[ "services" "hyperhive" "swarm" "forge" "ci" "package" ]
|
|
||||||
[ "services" "hyperhive" "deploy" "forgejo" "ci" "package" ]
|
|
||||||
)
|
|
||||||
|
|
||||||
# The queue's callout identity. What stays under `swarm.nats` is what the
|
# The queue's callout identity. What stays under `swarm.nats` is what the
|
||||||
# queue IS to every hive — the ports it answers on and the client id it is
|
# queue IS to every hive — the ports it answers on and the client id it is
|
||||||
|
|
@ -216,8 +229,7 @@ in
|
||||||
#
|
#
|
||||||
# `mirrors` renames in one entry rather than one per field: it is a single
|
# `mirrors` renames in one entry rather than one per field: it is a single
|
||||||
# option of a list-of-submodule type, so the rename carries its whole
|
# option of a list-of-submodule type, so the rename carries its whole
|
||||||
# value. The `ci` block above needs five because it is a plain attrset of
|
# value.
|
||||||
# separate options, which is the case with no parent path to rename.
|
|
||||||
(lib.mkRenamedOptionModule
|
(lib.mkRenamedOptionModule
|
||||||
[ "services" "hyperhive" "swarm" "forge" "openFirewall" ]
|
[ "services" "hyperhive" "swarm" "forge" "openFirewall" ]
|
||||||
[ "services" "hyperhive" "deploy" "forgejo" "openFirewall" ]
|
[ "services" "hyperhive" "deploy" "forgejo" "openFirewall" ]
|
||||||
|
|
@ -451,9 +463,8 @@ in
|
||||||
];
|
];
|
||||||
|
|
||||||
# ⚠️ Only `deploy.forgejo.enable` is declared here. The rest of
|
# ⚠️ Only `deploy.forgejo.enable` is declared here. The rest of
|
||||||
# `deploy.forgejo` is in ./hive-forge/default.nix and ./hive-ci.nix: the
|
# `deploy.forgejo` is in ./hive-forge/default.nix: the knobs live with the
|
||||||
# knobs live with the module that reads them; this file stays the registry
|
# module that reads them; this file stays the registry of toggles.
|
||||||
# of toggles.
|
|
||||||
options.services.hyperhive.deploy = {
|
options.services.hyperhive.deploy = {
|
||||||
forgejo.enable = lib.mkOption {
|
forgejo.enable = lib.mkOption {
|
||||||
type = lib.types.bool;
|
type = lib.types.bool;
|
||||||
|
|
@ -602,6 +613,23 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
ci-runner.enable = lib.mkOption {
|
||||||
|
type = lib.types.bool;
|
||||||
|
default = false;
|
||||||
|
example = true;
|
||||||
|
description = ''
|
||||||
|
Run the swarm's Forgejo Actions runner in a `swarm-ci-runner`
|
||||||
|
container on this host. It reaches the forge at
|
||||||
|
{option}`services.hyperhive.swarm.forge.domain` and registers with
|
||||||
|
the token swarm-controller publishes to the secret store, so it
|
||||||
|
needs neither the forge nor a hive on this host.
|
||||||
|
|
||||||
|
Not derived from
|
||||||
|
{option}`services.hyperhive.deploy.allSwarmServices`: a runner
|
||||||
|
executes pull-request code, so running one is a decision of its own.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
hive-controller.enable = lib.mkOption {
|
hive-controller.enable = lib.mkOption {
|
||||||
type = lib.types.bool;
|
type = lib.types.bool;
|
||||||
default = false;
|
default = false;
|
||||||
|
|
|
||||||
|
|
@ -45,6 +45,8 @@ let
|
||||||
swarm-bao-forwarder-oidc = true;
|
swarm-bao-forwarder-oidc = true;
|
||||||
# ./swarm-nats.nix: the queue's TLS leaf, not a secret, but the same wait.
|
# ./swarm-nats.nix: the queue's TLS leaf, not a secret, but the same wait.
|
||||||
swarm-bao-nats-tls = deployCfg.nats.enable;
|
swarm-bao-nats-tls = deployCfg.nats.enable;
|
||||||
|
# ./swarm-ci-runner.nix
|
||||||
|
swarm-bao-ci-runner = deployCfg.ci-runner.enable;
|
||||||
};
|
};
|
||||||
|
|
||||||
# Readers that log in under another unit's principal: reader unit → the
|
# Readers that log in under another unit's principal: reader unit → the
|
||||||
|
|
|
||||||
|
|
@ -105,6 +105,8 @@ in
|
||||||
# The store forwarder's.
|
# The store forwarder's.
|
||||||
forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem";
|
forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem";
|
||||||
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
|
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
|
||||||
|
ciRunnerClientCertFile = lib.mkDefault "${pkiDir}/ci-runner.pem";
|
||||||
|
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
|
||||||
|
|
||||||
# The granter: every `swarm-bao-*-policy` unit on this host logs in with
|
# The granter: every `swarm-bao-*-policy` unit on this host logs in with
|
||||||
# it. ⚠️ Unlike every other leaf here, never the file an operator copies:
|
# it. ⚠️ Unlike every other leaf here, never the file an operator copies:
|
||||||
|
|
@ -256,6 +258,12 @@ in
|
||||||
[ -s ${pkiDir}/nats-auth.pem ] || ${signLeaf} ${pkiDir} nats-auth \
|
[ -s ${pkiDir}/nats-auth.pem ] || ${signLeaf} ${pkiDir} nats-auth \
|
||||||
${lib.escapeShellArg deployCfg.bao.natsAuthCommonName} "" clientAuth
|
${lib.escapeShellArg deployCfg.bao.natsAuthCommonName} "" clientAuth
|
||||||
|
|
||||||
|
# The CI runner's token fetch. Minted here for the matrix-ctl leaf's
|
||||||
|
# reason: the runner is a swarm singleton, so on a runner host without
|
||||||
|
# the store this is the file an operator copies.
|
||||||
|
[ -s ${pkiDir}/ci-runner.pem ] || ${signLeaf} ${pkiDir} ci-runner \
|
||||||
|
${lib.escapeShellArg deployCfg.bao.ciRunnerCommonName} "" clientAuth
|
||||||
|
|
||||||
${lib.optionalString deployCfg.hive-controller.enable ''
|
${lib.optionalString deployCfg.hive-controller.enable ''
|
||||||
# This hive's gateway-leaf identity, which opens
|
# This hive's gateway-leaf identity, which opens
|
||||||
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
|
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
|
||||||
|
|
|
||||||
|
|
@ -349,8 +349,8 @@
|
||||||
The kernel default is `100`. The hyperhive default of `80` means
|
The kernel default is `100`. The hyperhive default of `80` means
|
||||||
agent containers yield slightly to everything that is *not* on
|
agent containers yield slightly to everything that is *not* on
|
||||||
this drop-in path — host services and the infrastructure
|
this drop-in path — host services and the infrastructure
|
||||||
containers (`hive-ci`, `hive-forge`, `hive-gateway`,
|
containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
|
||||||
`hive-matrix`), which stay at `100`. Note this is a hive-wide
|
stay at `100`. Note this is a hive-wide
|
||||||
value, so it does not rank agents against *each other*: they all
|
value, so it does not rank agents against *each other*: they all
|
||||||
share one weight.
|
share one weight.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,470 +0,0 @@
|
||||||
{
|
|
||||||
pkgs,
|
|
||||||
lib,
|
|
||||||
config,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
cfg = config.services.hyperhive.deploy.forgejo.ci;
|
|
||||||
forgeCfg = config.services.hyperhive.swarm.forge;
|
|
||||||
forgeDeployCfg = config.services.hyperhive.deploy.forgejo;
|
|
||||||
gatewayCfg = config.services.hyperhive.gateway;
|
|
||||||
networkCfg = config.services.hyperhive.network;
|
|
||||||
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
|
|
||||||
|
|
||||||
# Self-signed TLS is the gateway default (no operator cert / ACME). When
|
|
||||||
# active, forgejo's ROOT_URL is `https://forge.<domain>` and the leaf is
|
|
||||||
# signed by the host hive CA — so the runner's Node-based actions (e.g.
|
|
||||||
# `upload-artifact`, which POSTs to the ROOT_URL-derived artifact endpoint)
|
|
||||||
# reject the chain, since Node trusts only its bundled CA bundle, not the
|
|
||||||
# system store. Trust the hive CA explicitly via NODE_EXTRA_CA_CERTS below.
|
|
||||||
# The bind-mount + `container@` ordering that makes the CA reachable are
|
|
||||||
# shared with hive-forge via the `hive-ca-trust` helper; only the Node
|
|
||||||
# `NODE_EXTRA_CA_CERTS` consumption is hive-ci-specific.
|
|
||||||
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
|
|
||||||
useSelfSigned = caTrust.useSelfSigned;
|
|
||||||
caContainerPath = caTrust.caContainerPath;
|
|
||||||
|
|
||||||
# The runner daemon needs the hive CA too, and `NODE_EXTRA_CA_CERTS` above
|
|
||||||
# does not give it to them: that variable is Node's, and it is ADDITIVE.
|
|
||||||
# `gitea-runner` is Go, whose trust store is REPLACING -- it reads
|
|
||||||
# `SSL_CERT_FILE` and then trusts only what that file contains. So the two
|
|
||||||
# consumers need different treatment from the same CA, which is exactly the
|
|
||||||
# split the helper documents at its head.
|
|
||||||
#
|
|
||||||
# Without this the daemon fails every startup call with
|
|
||||||
# `x509: certificate signed by unknown authority` the moment it reaches a
|
|
||||||
# TLS endpoint -- and it reaches one whenever the address it is registered
|
|
||||||
# at redirects to the gateway's https vhost. The unit then crash-loops on
|
|
||||||
# `Restart=on-failure`, no job is ever picked up, and nothing reports it as
|
|
||||||
# broken: every pull request simply sits at "Waiting to run".
|
|
||||||
#
|
|
||||||
# ⚠️ This is deliberately NOT claimed as the whole story of that outage --
|
|
||||||
# what address the daemon holds is a separate question. It is here because
|
|
||||||
# the trust gap is real under every explanation of it, and adding a CA to
|
|
||||||
# one unit cannot break a path that already works.
|
|
||||||
caBundleModule = caTrust.trustBundle {
|
|
||||||
inherit pkgs;
|
|
||||||
name = "hive-ci";
|
|
||||||
consumers = [ "gitea-runner-hive" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
# Private network namespace, attached to the hive bridge so the
|
|
||||||
# runner reaches the forge via the gateway — and cannot reach
|
|
||||||
# host-loopback (127.0.0.1:7000 dashboard, raw forge port, etc.).
|
|
||||||
# See docs/networking/network.md.
|
|
||||||
privateNetwork = true;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
# Forgejo Actions runner in a `hive-ci` nixos-container.
|
|
||||||
#
|
|
||||||
# ⚠️ Two properties here are load-bearing and neither is visible from the
|
|
||||||
# config that implements it:
|
|
||||||
#
|
|
||||||
# - The container gets its OWN network namespace, not the host's, so a
|
|
||||||
# PR's nix build cannot reach host-loopback services. That is a
|
|
||||||
# prompt-injection boundary; moving this to host netns for convenience
|
|
||||||
# removes it. Reachability of the forge from inside:
|
|
||||||
# `docs/networking/network.md::Container isolation`.
|
|
||||||
# - The forge ADMIN token never enters this container — hive-c0re holds
|
|
||||||
# it and hands in only a registration token. Full flow:
|
|
||||||
# `docs/scheduler/ci.md::Container design`.
|
|
||||||
#
|
|
||||||
# Non-ephemeral on purpose: gitea-actions-runner writes its registered
|
|
||||||
# credentials to its stateDir on first registration and reuses them, so an
|
|
||||||
# ephemeral container would re-register on every restart.
|
|
||||||
#
|
|
||||||
# Nix builds here use the shared /nix/store with `sandbox-fallback = true`
|
|
||||||
# — nspawn containers cannot create the user namespaces nix sandboxing
|
|
||||||
# wants: `docs/process/gotchas.md`.
|
|
||||||
|
|
||||||
options.services.hyperhive.deploy.forgejo.ci = {
|
|
||||||
enable = lib.mkOption {
|
|
||||||
type = lib.types.bool;
|
|
||||||
default = false;
|
|
||||||
example = true;
|
|
||||||
description = ''
|
|
||||||
Run a Forgejo Actions runner in a `hive-ci` nixos-container.
|
|
||||||
Grouped under `services.hyperhive.deploy.forgejo` because the runner
|
|
||||||
is tightly coupled to the forge instance it registers against.
|
|
||||||
Disabled by default. It registers against the forge on this same
|
|
||||||
host, so it requires
|
|
||||||
{option}`services.hyperhive.deploy.forgejo.enable` here.
|
|
||||||
|
|
||||||
On first start the container auto-registers against hive-forge using
|
|
||||||
hive-c0re's admin token — no manual token provisioning needed.
|
|
||||||
Runner credentials are persisted in the container's state dir and
|
|
||||||
reused on every subsequent boot.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
name = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "hive-ci";
|
|
||||||
example = "prod-hive";
|
|
||||||
description = ''
|
|
||||||
Runner name as shown in the Forgejo admin panel. Defaults to
|
|
||||||
"hive-ci"; override when multiple hives share a Forgejo instance.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
concurrency = lib.mkOption {
|
|
||||||
type = lib.types.ints.positive;
|
|
||||||
default = 1;
|
|
||||||
example = 4;
|
|
||||||
description = ''
|
|
||||||
Maximum number of workflow jobs the runner executes in parallel.
|
|
||||||
Each job gets its own temporary working directory; multiple parallel
|
|
||||||
jobs share the container's nix store and cargo registry cache.
|
|
||||||
Higher values trade memory + CPU headroom for throughput.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
labels = lib.mkOption {
|
|
||||||
type = lib.types.listOf lib.types.str;
|
|
||||||
default = [ "hive-ci:host" ];
|
|
||||||
example = [
|
|
||||||
"hive-ci:host"
|
|
||||||
"nix:host"
|
|
||||||
];
|
|
||||||
description = ''
|
|
||||||
Runner labels in `<name>:<scheme>` format. The `host` scheme runs
|
|
||||||
commands directly in the container (no docker/podman). Workflow
|
|
||||||
files target this runner with `runs-on: [hive-ci]`.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
package = lib.mkOption {
|
|
||||||
type = lib.types.package;
|
|
||||||
default = pkgs.gitea-actions-runner;
|
|
||||||
defaultText = lib.literalExpression "pkgs.gitea-actions-runner";
|
|
||||||
description = "gitea-actions-runner package.";
|
|
||||||
};
|
|
||||||
|
|
||||||
jobTimeout = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "1h";
|
|
||||||
example = "3h";
|
|
||||||
description = ''
|
|
||||||
Per-job wall-clock timeout the runner enforces (act_runner's
|
|
||||||
`runner.timeout`). A job that exceeds it is killed, so a hung or
|
|
||||||
runaway build is bounded instead of holding the runner's single
|
|
||||||
slot indefinitely. Default `1h` comfortably covers a cold-cache
|
|
||||||
nix build while still bounding a stuck job; raise it (e.g.
|
|
||||||
`"3h"`) if you legitimately run jobs longer than that. Accepts a
|
|
||||||
Go duration string (`30m`, `1h`, `2h30m`). Note: this is
|
|
||||||
enforced by the runner process, so it only fires while that
|
|
||||||
process is itself healthy.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkIf cfg.enable {
|
|
||||||
assertions = [
|
|
||||||
{
|
|
||||||
# The runner reaches the forge through THIS host's gateway, and
|
|
||||||
# hive-c0re registers it through the local forge container; neither
|
|
||||||
# exists on a host that does not run the forge. A runner on another
|
|
||||||
# host, registered by the swarm instead, is a separate design.
|
|
||||||
assertion = forgeDeployCfg.enable;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.deploy.forgejo.ci.enable requires
|
|
||||||
services.hyperhive.deploy.forgejo.enable on the same host.
|
|
||||||
The CI runner reaches the forge through this host's gateway and
|
|
||||||
is registered through the local forge container, so it can only
|
|
||||||
run where the forge does. Enable CI on the swarm's forge host.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# The runner container has a private netns of its own, so it is
|
|
||||||
# attached to the bridge and reaches the forge by name through the
|
|
||||||
# gateway vhost the assertion above already insists on.
|
|
||||||
services.hyperhive.network.enable = lib.mkDefault true;
|
|
||||||
services.hyperhive.gateway.enable = lib.mkDefault true;
|
|
||||||
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
|
|
||||||
|
|
||||||
# Create /run/hive-ci/ on the host and seed runner-token with a
|
|
||||||
# placeholder. The container bind-mounts this file read-only; hive-c0re
|
|
||||||
# (via hive-priv's RegisterCiRunner) overwrites it with the real
|
|
||||||
# registration token when it registers the runner out of band. The
|
|
||||||
# placeholder keeps the runner's EnvironmentFile present from first boot,
|
|
||||||
# before c0re has registered — the runner's ExecStartPre precond (below)
|
|
||||||
# distinguishes the placeholder from a real token.
|
|
||||||
systemd.tmpfiles.rules = [
|
|
||||||
"d /run/hive-ci 0700 root root -"
|
|
||||||
"f /run/hive-ci/runner-token 0600 root root - TOKEN=placeholder"
|
|
||||||
];
|
|
||||||
|
|
||||||
# Tell hive-c0re that CI is enabled so its startup sweep registers the
|
|
||||||
# runner. Registration is OFF the container's boot-critical path:
|
|
||||||
# hive-c0re (`forge/ci_runner.rs`) validates the persisted `.runner`
|
|
||||||
# against the forge and, when absent/stale, mints a registration token and
|
|
||||||
# hands it to hive-priv to write `/run/hive-ci/runner-token` + restart the
|
|
||||||
# runner. The forge admin token stays in hive-c0re; only the registration
|
|
||||||
# token reaches the host env-file the container mounts read-only.
|
|
||||||
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_CI_ENABLED = "1";
|
|
||||||
|
|
||||||
# `caTrust.containerOrdering` orders this unit after `hive-tls-ca.service`
|
|
||||||
# in self-signed mode (see the hive-ca-trust helper). Runner registration
|
|
||||||
# is not on the boot-critical path — hive-c0re owns it out of band — so
|
|
||||||
# the container boots immediately and needs no start-timeout bump.
|
|
||||||
systemd.services."container@hive-ci" = caTrust.containerOrdering;
|
|
||||||
|
|
||||||
containers.hive-ci = {
|
|
||||||
autoStart = true;
|
|
||||||
ephemeral = false;
|
|
||||||
# Journal files on the host, not inside the container: nixpkgs hardcodes
|
|
||||||
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
|
|
||||||
extraFlags = [ "--link-journal=host" ];
|
|
||||||
inherit privateNetwork;
|
|
||||||
hostBridge = networkCfg.bridgeName;
|
|
||||||
|
|
||||||
bindMounts = {
|
|
||||||
# Pre-filled by hive-c0re (via hive-priv) with the runner
|
|
||||||
# registration token; tmpfiles seeds a `TOKEN=placeholder` before
|
|
||||||
# that. Read-only: the container reads TOKEN= from here; the core
|
|
||||||
# admin token never enters the container.
|
|
||||||
"/run/hive-ci/runner-token" = {
|
|
||||||
hostPath = "/run/hive-ci/runner-token";
|
|
||||||
isReadOnly = true;
|
|
||||||
};
|
|
||||||
# Route the container's nix through the HOST nix-daemon (as the agent
|
|
||||||
# containers do) instead of an in-container daemon: with
|
|
||||||
# NIX_REMOTE=daemon below, builds run on the host daemon and inherit
|
|
||||||
# its buildMachines + max-jobs. Note that `fallback` is NOT inherited
|
|
||||||
# that way — it's a client-side option transmitted per connection, so
|
|
||||||
# the container sets it itself (see nix.settings.fallback below).
|
|
||||||
# Bind the *directory* (not the socket file) so the mount stays live
|
|
||||||
# across a host nix-daemon restart, which recreates the socket inode.
|
|
||||||
"/nix/var/nix/daemon-socket" = {
|
|
||||||
hostPath = "/nix/var/nix/daemon-socket";
|
|
||||||
isReadOnly = false;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
# Self-signed mode: bind the public hive CA cert read-only so the
|
|
||||||
# runner's Node actions trust the gateway/forge leaf (consumed via
|
|
||||||
# NODE_EXTRA_CA_CERTS in the container config). Shared bind-mount +
|
|
||||||
# ordering come from the hive-ca-trust helper.
|
|
||||||
// caTrust.bindMount;
|
|
||||||
|
|
||||||
config =
|
|
||||||
{ pkgs, lib, ... }:
|
|
||||||
{
|
|
||||||
# Assembles system CAs + the hive CA into one bundle and sets
|
|
||||||
# `SSL_CERT_FILE` on the runner unit. See `caBundleModule` above for
|
|
||||||
# why the Node variable beside it is not enough.
|
|
||||||
imports = [
|
|
||||||
../container-modules/swarm-container.nix
|
|
||||||
caBundleModule
|
|
||||||
];
|
|
||||||
|
|
||||||
# Its own netns, so it keeps its own firewall; resolvconf stays on
|
|
||||||
# and serves `networking.nameservers` below.
|
|
||||||
services.hyperhive.swarmContainer = {
|
|
||||||
inherit privateNetwork;
|
|
||||||
writesOwnResolvConf = false;
|
|
||||||
};
|
|
||||||
|
|
||||||
# Point the forge domain at the bridge IP so the runner can
|
|
||||||
# reach the forge through the gateway — both for registration /
|
|
||||||
# polling (runner URL below) and for artifact uploads (the
|
|
||||||
# Forgejo Actions artifact API uses ROOT_URL, i.e. the public
|
|
||||||
# forge domain, not a localhost URL). The gateway vhost for
|
|
||||||
# `forgeCfg.domain` proxies all `/` → forge; `addSSL = true`
|
|
||||||
# means HTTP:80 is served without redirect alongside HTTPS:443.
|
|
||||||
# Ports 80 and 443 are always open on the bridge firewall (see
|
|
||||||
# hive-network.nix). No DNS lookup needed — /etc/hosts wins.
|
|
||||||
networking.extraHosts = "${networkCfg.bridgeIp} ${forgeCfg.domain}";
|
|
||||||
# DNS: use the hive resolver on the bridge IP (dnsmasq in
|
|
||||||
# hive-gateway) for external lookups (git checkout, crate
|
|
||||||
# registries, etc.). The bridge→loopback DROP rule does not
|
|
||||||
# affect traffic destined for the bridge IP itself.
|
|
||||||
networking.nameservers = [ networkCfg.bridgeIp ];
|
|
||||||
# Bridge-attached via privateNetwork=true + hostBridge. The
|
|
||||||
# gateway's dnsmasq serves a DHCP pool covering all usable bridge
|
|
||||||
# addresses (see dhcp-range in hive-gateway.nix) — agents and
|
|
||||||
# service containers alike receive IPs dynamically.
|
|
||||||
networking.interfaces.eth0.useDHCP = true;
|
|
||||||
|
|
||||||
# nspawn containers can't create user-namespaces, so nix
|
|
||||||
# sandboxing always fails. Fall back to unsandboxed builds.
|
|
||||||
# Moot once every nix invocation in the container routes
|
|
||||||
# through the host daemon (the daemon governs sandboxing).
|
|
||||||
# See docs/process/gotchas.md and nix/agent-modules/default.nix.
|
|
||||||
nix.settings.sandbox-fallback = lib.mkForce true;
|
|
||||||
# Degrade to a local build when a remote builder is unreachable
|
|
||||||
# rather than failing the check. `fallback` is a client-side
|
|
||||||
# option (transmitted per daemon connection), so routing through
|
|
||||||
# the host daemon does NOT inherit the host's value — the
|
|
||||||
# container has to set it. Same reasoning as the agent
|
|
||||||
# containers; see nix/agent-modules/default.nix.
|
|
||||||
nix.settings.fallback = true;
|
|
||||||
nix.settings.experimental-features = [
|
|
||||||
"nix-command"
|
|
||||||
"flakes"
|
|
||||||
];
|
|
||||||
|
|
||||||
# Build through the HOST nix-daemon (bind-mounted socket above), not
|
|
||||||
# an in-container daemon. NIX_REMOTE=daemon makes root services (the
|
|
||||||
# runner) use the daemon socket rather than store=auto → local store.
|
|
||||||
# Disable BOTH the service and its .socket unit: the .socket owns the
|
|
||||||
# listen path /nix/var/nix/daemon-socket/socket and would otherwise
|
|
||||||
# race the bind mount over the same path.
|
|
||||||
systemd.globalEnvironment.NIX_REMOTE = "daemon";
|
|
||||||
systemd.services.nix-daemon.enable = false;
|
|
||||||
systemd.sockets.nix-daemon.enable = false;
|
|
||||||
|
|
||||||
# package is top-level on gitea-actions-runner, not per-instance.
|
|
||||||
services.gitea-actions-runner.package = cfg.package;
|
|
||||||
|
|
||||||
services.gitea-actions-runner.instances.hive = {
|
|
||||||
enable = true;
|
|
||||||
name = cfg.name;
|
|
||||||
# Route through the gateway (bridge IP, port 80) so the
|
|
||||||
# runner never touches host-loopback. The forge domain
|
|
||||||
# resolves to the bridge IP via networking.extraHosts above;
|
|
||||||
# the gateway vhost `forgeCfg.domain` proxies to the forge
|
|
||||||
# on HTTP:80 (addSSL=true, no HTTP→HTTPS redirect).
|
|
||||||
url = "http://${forgeCfg.domain}";
|
|
||||||
# EnvironmentFile providing TOKEN= — pre-filled by hive-c0re
|
|
||||||
# (via hive-priv) with the runner registration token;
|
|
||||||
# bind-mounted read-only from /run/hive-ci/runner-token on the
|
|
||||||
# host. The runner's ExecStartPre precond gates on this being a
|
|
||||||
# real (non-placeholder) token or an already-registered .runner.
|
|
||||||
tokenFile = "/run/hive-ci/runner-token";
|
|
||||||
labels = cfg.labels;
|
|
||||||
settings = {
|
|
||||||
runner.capacity = cfg.concurrency;
|
|
||||||
# Per-job wall-clock cap — see the `jobTimeout` option.
|
|
||||||
runner.timeout = cfg.jobTimeout;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# No tmpfiles rule: /run/hive-ci/runner-token is bind-mounted
|
|
||||||
# read-only from the host (pre-filled before container start).
|
|
||||||
# nspawn creates the /run/hive-ci/ mount-point directory
|
|
||||||
# automatically before launching the container's init.
|
|
||||||
|
|
||||||
# No hive-ci-register.service inside the container: all forge
|
|
||||||
# API calls (runner validation, token fetch) live in hive-c0re
|
|
||||||
# (forge/ci_runner.rs), out of band. The core admin token never
|
|
||||||
# enters this container.
|
|
||||||
|
|
||||||
# git is already in the gitea-actions-runner service PATH (the
|
|
||||||
# nixpkgs module builds it from the package's runtime deps).
|
|
||||||
# nix is required for `nix flake check` / `nix build` workflow
|
|
||||||
# steps — it's not included by the upstream module.
|
|
||||||
# Use the `path` service attribute (generates ExecSearchPath=)
|
|
||||||
# to prepend nix's bin dir to PATH without touching the
|
|
||||||
# environment.PATH the nixpkgs module sets — overriding that
|
|
||||||
# would lose git, curl, nodejs, and other runner deps.
|
|
||||||
environment.systemPackages = [
|
|
||||||
pkgs.git
|
|
||||||
pkgs.nix
|
|
||||||
];
|
|
||||||
|
|
||||||
systemd.services."gitea-runner-hive" = {
|
|
||||||
path = [ pkgs.nix ];
|
|
||||||
# A CI runner must be able to EXECUTE what it builds, and by
|
|
||||||
# default it cannot: the upstream unit uses `DynamicUser`, and
|
|
||||||
# systemd mounts a dynamic unit's state directory `noexec`.
|
|
||||||
# Measured on the live runner rather than inferred — from a job's
|
|
||||||
# own /proc/self/mountinfo:
|
|
||||||
#
|
|
||||||
# /var/lib/private ro,nosuid,nodev,noexec (tmpfs)
|
|
||||||
# /var/lib/private/gitea-runner rw,nosuid,nodev,noexec,idmapped
|
|
||||||
#
|
|
||||||
# with /tmp as the control at rw,nosuid,nodev — no noexec, and a
|
|
||||||
# script there executes fine. Nothing noticed for as long as every
|
|
||||||
# workflow built inside the nix sandbox and executed nothing in the
|
|
||||||
# workspace; the first job that compiled a build script into the
|
|
||||||
# runner's own target dir died with EACCES on execve, reported as
|
|
||||||
# "could not execute process ... (never executed)".
|
|
||||||
#
|
|
||||||
# Both spellings are listed because `ExecPaths=` resolves against
|
|
||||||
# the HOST root, where the dynamic-user layout makes
|
|
||||||
# /var/lib/gitea-runner a symlink into private/. The `-` prefix
|
|
||||||
# means "ignore if absent", so whichever layout the unit ends up
|
|
||||||
# with, the other is a no-op rather than a startup failure.
|
|
||||||
serviceConfig.ExecPaths = [
|
|
||||||
"-/var/lib/gitea-runner"
|
|
||||||
"-/var/lib/private/gitea-runner"
|
|
||||||
];
|
|
||||||
# Trust the hive CA in Node-based actions. With self-signed TLS,
|
|
||||||
# forgejo's ROOT_URL is `https://forge.<domain>` (CA-signed leaf),
|
|
||||||
# so actions like `upload-artifact` — whose Node HTTP client uses
|
|
||||||
# Node's *bundled* CA bundle, not the system store — reject the
|
|
||||||
# chain with "unable to verify the first certificate". Pointing
|
|
||||||
# NODE_EXTRA_CA_CERTS at the bind-mounted CA adds it to Node's
|
|
||||||
# roots for every action, hive-wide. Inherited by the job
|
|
||||||
# processes the runner spawns (host execution mode). Only set in
|
|
||||||
# self-signed mode; with an operator cert / ACME the public CA
|
|
||||||
# already validates and the bind-mount is absent.
|
|
||||||
environment = lib.mkIf useSelfSigned {
|
|
||||||
NODE_EXTRA_CA_CERTS = caContainerPath;
|
|
||||||
};
|
|
||||||
# Gate runner start on credentials being present. Registration is
|
|
||||||
# done OUT OF BAND by hive-c0re (forge/ci_runner.rs): it mints a
|
|
||||||
# token and, via hive-priv, writes /run/hive-ci/runner-token +
|
|
||||||
# restarts this unit. Pass iff already registered (.runner present)
|
|
||||||
# OR a real (non-placeholder) token is in place; else exit 1 so the
|
|
||||||
# runner stays down (Restart=on-failure retries) until c0re
|
|
||||||
# registers it, rather than blocking the container's start.
|
|
||||||
#
|
|
||||||
# No nix-daemon wait: builds route through the HOST daemon
|
|
||||||
# (bound socket above), which is up before the container — nix's
|
|
||||||
# own connect retries cover any brief window. `mkBefore` runs this
|
|
||||||
# ahead of any upstream pre-steps.
|
|
||||||
#
|
|
||||||
# The `+` prefix runs this step with FULL PRIVILEGES instead of as
|
|
||||||
# the unit's (dynamic) `gitea-runner` user. The token file is 0600
|
|
||||||
# root-owned, so an unprivileged check cannot read it — and it did
|
|
||||||
# not fail closed: `[ -s ]` succeeds on a stat alone, `grep` then
|
|
||||||
# died with EACCES, and `! grep` turned that error into `true`, so
|
|
||||||
# the gate exited 0 on an unreadable file. It reported "a real token
|
|
||||||
# is present" without ever having looked, for its entire existence.
|
|
||||||
#
|
|
||||||
# This grants the runner nothing new: `tokenFile` above becomes
|
|
||||||
# `EnvironmentFile=`, which systemd already reads AS ROOT before
|
|
||||||
# dropping privileges, so the payload never passes through an
|
|
||||||
# unprivileged reader either way. The `+` only lets the gate observe
|
|
||||||
# what systemd observes.
|
|
||||||
serviceConfig.ExecStartPre = lib.mkBefore [
|
|
||||||
"+${pkgs.writeShellScript "hive-ci-runner-precond" ''
|
|
||||||
RUNNER=/var/lib/gitea-runner/hive/.runner
|
|
||||||
TOKEN_FILE=/run/hive-ci/runner-token
|
|
||||||
if [ -f "$RUNNER" ]; then exit 0; fi
|
|
||||||
# Readability is asserted SEPARATELY and fails loudly. Folding
|
|
||||||
# it into the test below is what hid the bug: an unreadable file
|
|
||||||
# and a file holding a real token produced the same verdict, so
|
|
||||||
# "I could not look" was indistinguishable from "I looked and it
|
|
||||||
# is fine".
|
|
||||||
if [ -e "$TOKEN_FILE" ] && [ ! -r "$TOKEN_FILE" ]; then
|
|
||||||
echo "hive-ci runner: $TOKEN_FILE exists but is unreadable by this step; the credential gate cannot run" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
if [ -s "$TOKEN_FILE" ] && ! ${pkgs.gnugrep}/bin/grep -q '^TOKEN=placeholder$' "$TOKEN_FILE"; then
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
echo "hive-ci runner: not registered and no real token yet; waiting for hive-c0re to register" >&2
|
|
||||||
exit 1
|
|
||||||
''}"
|
|
||||||
];
|
|
||||||
# Registration is out of band: hive-c0re explicitly restarts this
|
|
||||||
# unit once it writes the real token, which is the primary path. As
|
|
||||||
# a safety net, if the precond above fires first (no creds yet at
|
|
||||||
# boot) retry until c0re registers, rather than staying down — with
|
|
||||||
# no start-limit rate cap so it keeps retrying however long forge /
|
|
||||||
# the core token take to settle.
|
|
||||||
serviceConfig.Restart = lib.mkForce "on-failure";
|
|
||||||
# `mkForce`: the upstream gitea-actions-runner module also sets
|
|
||||||
# RestartSec (2), so ours needs the higher priority or the two
|
|
||||||
# collide into an eval error — same as the sibling `Restart`.
|
|
||||||
serviceConfig.RestartSec = lib.mkForce 15;
|
|
||||||
startLimitIntervalSec = 0;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
@ -74,23 +74,18 @@ let
|
||||||
"https://${cfg.domain}${portSuffix}/";
|
"https://${cfg.domain}${portSuffix}/";
|
||||||
effectiveRootUrl = if cfg.rootUrl != null then cfg.rootUrl else defaultRootUrl;
|
effectiveRootUrl = if cfg.rootUrl != null then cfg.rootUrl else defaultRootUrl;
|
||||||
|
|
||||||
# When CI is enabled, the runner needs `actions/checkout` resolvable
|
# `actionsFromSelf` resolves `uses:` against this forge, which then has to
|
||||||
# without external DNS (hive-ci shares the host netns, so a host-resolver
|
# hold the action.
|
||||||
# blip otherwise reds every `actions/checkout@vN` fetch from
|
actionsFromSelf = deployCfg.forgejo.actionsFromSelf;
|
||||||
# data.forgejo.org). Auto-append a pull-mirror of it and point
|
|
||||||
# forgejo's DEFAULT_ACTIONS_URL at this instance so `uses:` resolves local.
|
|
||||||
ciEnabled = config.services.hyperhive.deploy.forgejo.ci.enable;
|
|
||||||
actionCheckoutMirror = {
|
actionCheckoutMirror = {
|
||||||
upstream = "https://github.com/actions/checkout";
|
upstream = "https://github.com/actions/checkout";
|
||||||
dest = "actions/checkout";
|
dest = "actions/checkout";
|
||||||
};
|
};
|
||||||
# Auto-append the actions/checkout mirror only when CI is on AND the
|
# Unless the operator already declared that dest, which would duplicate it.
|
||||||
# operator hasn't already declared that dest themselves (else CI-on +
|
|
||||||
# an explicit `actions/checkout` entry would duplicate it).
|
|
||||||
effectiveMirrors =
|
effectiveMirrors =
|
||||||
deployCfg.forgejo.mirrors
|
deployCfg.forgejo.mirrors
|
||||||
++ lib.optional (
|
++ lib.optional (
|
||||||
ciEnabled && !(lib.any (m: m.dest == actionCheckoutMirror.dest) deployCfg.forgejo.mirrors)
|
actionsFromSelf && !(lib.any (m: m.dest == actionCheckoutMirror.dest) deployCfg.forgejo.mirrors)
|
||||||
) actionCheckoutMirror;
|
) actionCheckoutMirror;
|
||||||
|
|
||||||
# Share host netns — forgejo's HTTP / SSH listeners then look
|
# Share host netns — forgejo's HTTP / SSH listeners then look
|
||||||
|
|
@ -179,12 +174,24 @@ in
|
||||||
so a host-resolver blip leaves a *stale* mirror, never a hard
|
so a host-resolver blip leaves a *stale* mirror, never a hard
|
||||||
failure on whatever reads it.
|
failure on whatever reads it.
|
||||||
|
|
||||||
When `services.hyperhive.deploy.forgejo.ci.enable` is set, an
|
With {option}`services.hyperhive.deploy.forgejo.actionsFromSelf`
|
||||||
`actions/checkout` mirror is auto-appended to this list and
|
set, an `actions/checkout` mirror is appended to this list.
|
||||||
forgejo's `DEFAULT_ACTIONS_URL` is pointed at this instance, so CI
|
'';
|
||||||
`uses: actions/checkout@vN` steps resolve entirely on loopback with
|
};
|
||||||
no external DNS on the critical path (the seed/re-sync needs
|
|
||||||
external DNS, but that's off the CI path).
|
actionsFromSelf = lib.mkOption {
|
||||||
|
type = lib.types.bool;
|
||||||
|
default = false;
|
||||||
|
example = true;
|
||||||
|
description = ''
|
||||||
|
Resolve a workflow's `uses: <org>/<action>@vN` against this forge
|
||||||
|
instead of `data.forgejo.org`, and add an `actions/checkout`
|
||||||
|
pull-mirror to {option}`services.hyperhive.deploy.forgejo.mirrors`
|
||||||
|
so the checkout step has something to resolve to. A CI job's
|
||||||
|
checkout then needs no external DNS; only the mirror's re-sync
|
||||||
|
does.
|
||||||
|
|
||||||
|
Any other action a workflow names has to be mirrored here too.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -641,16 +648,16 @@ in
|
||||||
# Enable Forgejo Actions so the runner registration token
|
# Enable Forgejo Actions so the runner registration token
|
||||||
# API endpoint is available. Without this the endpoint
|
# API endpoint is available. Without this the endpoint
|
||||||
# returns "runner registration token not found" regardless
|
# returns "runner registration token not found" regardless
|
||||||
# of token scopes. Required by `hive-ci-register.service`
|
# of token scopes. swarm-controller's `forge::runner_token`
|
||||||
# in the hive-ci container.
|
# reads it.
|
||||||
actions.ENABLED = true;
|
actions.ENABLED = true;
|
||||||
# When CI is enabled, resolve `uses: <org>/<action>@vN` from
|
# With `actionsFromSelf`, resolve `uses: <org>/<action>@vN` from
|
||||||
# THIS instance (the seeded `actions/checkout` pull-mirror)
|
# THIS instance (the seeded `actions/checkout` pull-mirror)
|
||||||
# instead of the upstream default `data.forgejo.org` — keeps
|
# instead of the upstream default `data.forgejo.org` — keeps
|
||||||
# the checkout step on loopback, immune to a host-resolver
|
# the checkout step on loopback, immune to a host-resolver
|
||||||
# blip. `self` = forgejo expands actions against its
|
# blip. `self` = forgejo expands actions against its
|
||||||
# own ROOT_URL.
|
# own ROOT_URL.
|
||||||
actions.DEFAULT_ACTIONS_URL = lib.mkIf ciEnabled "self";
|
actions.DEFAULT_ACTIONS_URL = lib.mkIf actionsFromSelf "self";
|
||||||
# F3 (federation) computes its data dir relative to the
|
# F3 (federation) computes its data dir relative to the
|
||||||
# forgejo binary, which lands in the read-only nix
|
# forgejo binary, which lands in the read-only nix
|
||||||
# store and crashes anything that touches the F3
|
# store and crashes anything that touches the F3
|
||||||
|
|
@ -1099,7 +1106,7 @@ in
|
||||||
lib.mkIf deployCfg.swarm-controller.enable (builtins.toJSON effectiveMirrors);
|
lib.mkIf deployCfg.swarm-controller.enable (builtins.toJSON effectiveMirrors);
|
||||||
warnings = lib.optional (effectiveMirrors != [ ] && !deployCfg.swarm-controller.enable) ''
|
warnings = lib.optional (effectiveMirrors != [ ] && !deployCfg.swarm-controller.enable) ''
|
||||||
services.hyperhive.deploy.forgejo.mirrors (or the actions/checkout mirror
|
services.hyperhive.deploy.forgejo.mirrors (or the actions/checkout mirror
|
||||||
deploy.forgejo.ci.enable adds) is set on a host that does not run the
|
deploy.forgejo.actionsFromSelf adds) is set on a host that does not run the
|
||||||
swarm-controller. The controller seeds mirrors from its own host's list,
|
swarm-controller. The controller seeds mirrors from its own host's list,
|
||||||
so these are not seeded: declare them on the swarm-controller's host.
|
so these are not seeded: declare them on the swarm-controller's host.
|
||||||
'';
|
'';
|
||||||
|
|
|
||||||
|
|
@ -68,7 +68,7 @@
|
||||||
# DHCP pool covering all usable host addresses on the bridge
|
# DHCP pool covering all usable host addresses on the bridge
|
||||||
# subnet — bounds computed by hive-network.nix from
|
# subnet — bounds computed by hive-network.nix from
|
||||||
# bridgeIp/bridgePrefixLength. All containers (agents and service
|
# bridgeIp/bridgePrefixLength. All containers (agents and service
|
||||||
# containers such as hive-ci) receive their IPs dynamically.
|
# containers such as swarm-ci-runner) receive their IPs dynamically.
|
||||||
dhcp-range = "${networkCfg.dhcpRangeStart},${networkCfg.dhcpRangeEnd},1h";
|
dhcp-range = "${networkCfg.dhcpRangeStart},${networkCfg.dhcpRangeEnd},1h";
|
||||||
dhcp-leasefile = "/var/lib/dnsmasq/dnsmasq.leases";
|
dhcp-leasefile = "/var/lib/dnsmasq/dnsmasq.leases";
|
||||||
# No explicit upstream: non-hive queries follow dnsmasq's
|
# No explicit upstream: non-hive queries follow dnsmasq's
|
||||||
|
|
|
||||||
|
|
@ -229,7 +229,7 @@ in
|
||||||
Read-only derived flag: `true` when the gateway serves the
|
Read-only derived flag: `true` when the gateway serves the
|
||||||
self-signed leaf the secret store issues — i.e. neither `tls.certDir` nor
|
self-signed leaf the secret store issues — i.e. neither `tls.certDir` nor
|
||||||
`tls.acme.enable` is configured. Single source of truth for the
|
`tls.acme.enable` is configured. Single source of truth for the
|
||||||
self-signed condition; consumed by the `hive-tls` and `hive-ci`
|
self-signed condition; consumed by the `hive-tls` and `hive-ca-trust`
|
||||||
modules so the derivation isn't duplicated. Internal — not meant to
|
modules so the derivation isn't duplicated. Internal — not meant to
|
||||||
be set by operators (use `tls.certDir` / `tls.acme` to override the
|
be set by operators (use `tls.certDir` / `tls.acme` to override the
|
||||||
self-signed default).
|
self-signed default).
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ let
|
||||||
gatewayCfg = config.services.hyperhive.gateway;
|
gatewayCfg = config.services.hyperhive.gateway;
|
||||||
hyperhiveDomain = config.services.hyperhive.domain;
|
hyperhiveDomain = config.services.hyperhive.domain;
|
||||||
|
|
||||||
# Same runtime→build-time bridge hive-ci and hive-forge already cross:
|
# Same runtime→build-time bridge hive-forge and swarm-ci-runner cross:
|
||||||
# binds the hive trust bundle (which folds in the swarm root) into the
|
# binds the hive trust bundle (which folds in the swarm root) into the
|
||||||
# container and orders the container after `hive-tls-ca.service`. The
|
# container and orders the container after `hive-tls-ca.service`. The
|
||||||
# assembled bundle itself comes from `caTrust.trustBundle`, imported in
|
# assembled bundle itself comes from `caTrust.trustBundle`, imported in
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,7 @@ let
|
||||||
d.swarm-controller.enable
|
d.swarm-controller.enable
|
||||||
d.swarm-ui.enable
|
d.swarm-ui.enable
|
||||||
d.forgejo.enable
|
d.forgejo.enable
|
||||||
d.forgejo.ci.enable
|
d.ci-runner.enable
|
||||||
d.matrix.enable
|
d.matrix.enable
|
||||||
d.authelia.enable
|
d.authelia.enable
|
||||||
d.bao.enable
|
d.bao.enable
|
||||||
|
|
|
||||||
|
|
@ -118,9 +118,6 @@ in
|
||||||
# /run/hive-agent — creates per-agent socket dirs
|
# /run/hive-agent — creates per-agent socket dirs
|
||||||
# /run/systemd — container@ drop-ins + machined state
|
# /run/systemd — container@ drop-ins + machined state
|
||||||
# /run/lock — nixos-container's create/destroy lock file
|
# /run/lock — nixos-container's create/destroy lock file
|
||||||
# /run/hive-ci — register_ci_runner's runner-token write
|
|
||||||
# (else token stays placeholder → runner
|
|
||||||
# crash-loops → no CI hive-wide)
|
|
||||||
# /var/lib/nixos-containers — container rootfs
|
# /var/lib/nixos-containers — container rootfs
|
||||||
# /var/lib/hyperhive — agent state (forge/matrix token files)
|
# /var/lib/hyperhive — agent state (forge/matrix token files)
|
||||||
# /nix — nix store + profile updates on create/update
|
# /nix — nix store + profile updates on create/update
|
||||||
|
|
@ -131,7 +128,6 @@ in
|
||||||
"/run/hive-agent"
|
"/run/hive-agent"
|
||||||
"/run/systemd"
|
"/run/systemd"
|
||||||
"/run/lock"
|
"/run/lock"
|
||||||
"/run/hive-ci"
|
|
||||||
"/var/lib/nixos-containers"
|
"/var/lib/nixos-containers"
|
||||||
"/var/lib/hyperhive"
|
"/var/lib/hyperhive"
|
||||||
"/nix"
|
"/nix"
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@
|
||||||
#
|
#
|
||||||
# `bindMount` + `containerOrdering` are the language-agnostic half. The
|
# `bindMount` + `containerOrdering` are the language-agnostic half. The
|
||||||
# *consumption* differs per runtime: an additive variable (Node's
|
# *consumption* differs per runtime: an additive variable (Node's
|
||||||
# `NODE_EXTRA_CA_CERTS`, hive-ci) points straight at `caContainerPath` from
|
# `NODE_EXTRA_CA_CERTS`, swarm-ci-runner) points straight at `caContainerPath` from
|
||||||
# the call site, while a *replacing* one (Go's `SSL_CERT_FILE`, rustls) needs
|
# the call site, while a *replacing* one (Go's `SSL_CERT_FILE`, rustls) needs
|
||||||
# the system-CAs+hive-CA concat that `trustBundle` below does for it.
|
# the system-CAs+hive-CA concat that `trustBundle` below does for it.
|
||||||
#
|
#
|
||||||
|
|
@ -19,7 +19,7 @@
|
||||||
# caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
|
# caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
|
||||||
# # then, in the container:
|
# # then, in the container:
|
||||||
# # bindMounts = { … } // caTrust.bindMount;
|
# # bindMounts = { … } // caTrust.bindMount;
|
||||||
# # systemd.services."container@hive-ci" = lib.mkMerge [ caTrust.containerOrdering … ];
|
# # systemd.services."container@swarm-ci-runner" = lib.mkMerge [ caTrust.containerOrdering … ];
|
||||||
# # environment.NODE_EXTRA_CA_CERTS = caTrust.caContainerPath; # consumption, per-caller
|
# # environment.NODE_EXTRA_CA_CERTS = caTrust.caContainerPath; # consumption, per-caller
|
||||||
#
|
#
|
||||||
# `tlsCfg` = config.services.hyperhive.deploy.hive-controller.tls
|
# `tlsCfg` = config.services.hyperhive.deploy.hive-controller.tls
|
||||||
|
|
@ -87,7 +87,7 @@ in
|
||||||
};
|
};
|
||||||
|
|
||||||
# Fold into the caller's `container@<name>` unit (via `lib.mkMerge` if the
|
# Fold into the caller's `container@<name>` unit (via `lib.mkMerge` if the
|
||||||
# caller adds its own keys, e.g. hive-ci's `TimeoutStartSec`). Orders the
|
# caller adds its own keys). Orders the
|
||||||
# container after the host `hive-tls-ca.service` so the bind source exists
|
# container after the host `hive-tls-ca.service` so the bind source exists
|
||||||
# before nspawn sets the mount up — a condition-skipped/late CA would
|
# before nspawn sets the mount up — a condition-skipped/late CA would
|
||||||
# otherwise fail the container start.
|
# otherwise fail the container start.
|
||||||
|
|
@ -99,7 +99,7 @@ in
|
||||||
# System CAs + hive CA in one bundle, with `SSL_CERT_FILE` set on each
|
# System CAs + hive CA in one bundle, with `SSL_CERT_FILE` set on each
|
||||||
# consumer — for runtimes whose trust variable *replaces* the store (Go,
|
# consumer — for runtimes whose trust variable *replaces* the store (Go,
|
||||||
# rustls-native-certs). An additive one (Node's `NODE_EXTRA_CA_CERTS`,
|
# rustls-native-certs). An additive one (Node's `NODE_EXTRA_CA_CERTS`,
|
||||||
# hive-ci) needs no bundle and should not use this.
|
# swarm-ci-runner) needs no bundle and should not use this.
|
||||||
#
|
#
|
||||||
# imports = [ (caTrust.trustBundle { inherit pkgs; name = "swarm-nats";
|
# imports = [ (caTrust.trustBundle { inherit pkgs; name = "swarm-nats";
|
||||||
# consumers = [ "swarm-nats-auth" ]; }) ];
|
# consumers = [ "swarm-nats-auth" ]; }) ];
|
||||||
|
|
|
||||||
|
|
@ -429,6 +429,10 @@ let
|
||||||
capabilities = ["create", "read", "update"]
|
capabilities = ["create", "read", "update"]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
path "${credentialMountPath}/data/${ciRunnerTokenLeaf}" {
|
||||||
|
capabilities = ["create", "update"]
|
||||||
|
}
|
||||||
|
|
||||||
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
||||||
capabilities = ["read"]
|
capabilities = ["read"]
|
||||||
}
|
}
|
||||||
|
|
@ -519,6 +523,12 @@ let
|
||||||
# it; the controller reads it and holds no other copy.
|
# it; the controller reads it and holds no other copy.
|
||||||
controllerClientLeaf = "swarm/controller/swarm-controller/oidc/client";
|
controllerClientLeaf = "swarm/controller/swarm-controller/oidc/client";
|
||||||
|
|
||||||
|
# The forge's runner registration token, the nix half of
|
||||||
|
# `swarm_secret_client::ci::registration_token_path`. The controller writes
|
||||||
|
# it and cannot read it back (`forge::runner_token` keeps the value it last
|
||||||
|
# wrote in memory); the CI runner's fetch unit is its one reader.
|
||||||
|
ciRunnerTokenLeaf = "swarm/services/ci-runner/registration-token";
|
||||||
|
|
||||||
# The UI's OIDC client secret. The publisher writes it; the granter reads it
|
# The UI's OIDC client secret. The publisher writes it; the granter reads it
|
||||||
# into the `oidc` auth method's config, which is the only copy bao uses.
|
# into the `oidc` auth method's config, which is the only copy bao uses.
|
||||||
baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client";
|
baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client";
|
||||||
|
|
@ -853,6 +863,16 @@ let
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
|
|
||||||
|
# The CI runner's token fetch. The one reader of the path, which the
|
||||||
|
# controller writes and cannot read back.
|
||||||
|
ciRunnerReaders = [
|
||||||
|
{
|
||||||
|
name = "swarm-ci-runner";
|
||||||
|
cn = baoDeploy.ciRunnerCommonName;
|
||||||
|
policyText = readStanza "${credentialMountPath}/data/${ciRunnerTokenLeaf}";
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
# The role name IS the policy name, as for the three service principals
|
# The role name IS the policy name, as for the three service principals
|
||||||
# above: the role attaches the policy by spelling it identically, and one
|
# above: the role attaches the policy by spelling it identically, and one
|
||||||
# string for both objects removes the way they drift apart.
|
# string for both objects removes the way they drift apart.
|
||||||
|
|
@ -1640,6 +1660,19 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
ciRunnerCommonName = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "swarm-ci-runner";
|
||||||
|
description = ''
|
||||||
|
Subject the store's `swarm-ci-runner` cert-auth role accepts: the
|
||||||
|
identity `swarm-bao-ci-runner`, the unit that fetches the forge's
|
||||||
|
runner registration token for the CI runner, presents. Its grant is one
|
||||||
|
path, `swarm/services/ci-runner/registration-token`, and read only.
|
||||||
|
|
||||||
|
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
queueAgentCommonNamePrefix = lib.mkOption {
|
queueAgentCommonNamePrefix = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "swarm-bao-queue-agent";
|
default = "swarm-bao-queue-agent";
|
||||||
|
|
@ -1847,6 +1880,32 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
ciRunnerClientCertFile = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.str;
|
||||||
|
default = null;
|
||||||
|
example = "/var/lib/swarm-bao-pki/ci-runner.pem";
|
||||||
|
description = ''
|
||||||
|
Certificate the unit that fetches the CI runner's registration token
|
||||||
|
presents to the store. Its subject must be
|
||||||
|
{option}`services.hyperhive.deploy.bao.ciRunnerCommonName`.
|
||||||
|
|
||||||
|
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
|
||||||
|
On a runner host without the store, it is a file the operator copies
|
||||||
|
from the store's host. A path, never a value.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
ciRunnerClientKeyFile = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.str;
|
||||||
|
default = null;
|
||||||
|
example = "/var/lib/swarm-bao-pki/ci-runner-key.pem";
|
||||||
|
description = ''
|
||||||
|
Private key for
|
||||||
|
{option}`services.hyperhive.deploy.bao.ciRunnerClientCertFile`.
|
||||||
|
A path, never a value.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
granterCommonName = lib.mkOption {
|
granterCommonName = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "bao-granter";
|
default = "bao-granter";
|
||||||
|
|
@ -2907,6 +2966,8 @@ in
|
||||||
|
|
||||||
systemd.services.swarm-bao-nats-auth-policy = readerPolicyUnit "write the queue responder's agent-credential-reader bao policy and cert-auth role" natsAuthReaders;
|
systemd.services.swarm-bao-nats-auth-policy = readerPolicyUnit "write the queue responder's agent-credential-reader bao policy and cert-auth role" natsAuthReaders;
|
||||||
|
|
||||||
|
systemd.services.swarm-bao-ci-runner-policy = readerPolicyUnit "write the CI runner's registration-token-reader bao policy and cert-auth role" ciRunnerReaders;
|
||||||
|
|
||||||
# A FOURTH sibling, same shape and same reasons as the two above. This
|
# A FOURTH sibling, same shape and same reasons as the two above. This
|
||||||
# one is what turns `swarm-services-issuer` from a declaration into a
|
# one is what turns `swarm-services-issuer` from a declaration into a
|
||||||
# grant: a bao policy reaches nothing until a login role hands it to a
|
# grant: a bao policy reaches nothing until a login role hands it to a
|
||||||
|
|
@ -3111,6 +3172,7 @@ in
|
||||||
"swarm-bao-services-issuer-policy.service"
|
"swarm-bao-services-issuer-policy.service"
|
||||||
"swarm-bao-nats-tls-policy.service"
|
"swarm-bao-nats-tls-policy.service"
|
||||||
"swarm-bao-hive-leaf-policy.service"
|
"swarm-bao-hive-leaf-policy.service"
|
||||||
|
"swarm-bao-ci-runner-policy.service"
|
||||||
]
|
]
|
||||||
++ granterAfter;
|
++ granterAfter;
|
||||||
requires = [ "swarm-bao-pki.service" ];
|
requires = [ "swarm-bao-pki.service" ];
|
||||||
|
|
|
||||||
421
nix/host-modules/swarm-ci-runner.nix
Normal file
421
nix/host-modules/swarm-ci-runner.nix
Normal file
|
|
@ -0,0 +1,421 @@
|
||||||
|
# The swarm's Forgejo Actions runner, in a `swarm-ci-runner` container on the
|
||||||
|
# host that sets `deploy.ci-runner.enable`. It reaches the forge by name and
|
||||||
|
# reads its registration token from the secret store under a certificate of
|
||||||
|
# its own, so nothing here asks whether this host runs the forge or a hive.
|
||||||
|
#
|
||||||
|
# ⚠️ Two properties here are load-bearing and neither is visible from the
|
||||||
|
# config that implements it:
|
||||||
|
#
|
||||||
|
# - The container gets its OWN network namespace, not the host's, so a PR's
|
||||||
|
# nix build cannot reach host-loopback services. That is a
|
||||||
|
# prompt-injection boundary; moving this to host netns for convenience
|
||||||
|
# removes it. `docs/networking/network.md::Container isolation`.
|
||||||
|
# - The forge ADMIN token never enters this container. swarm-controller holds
|
||||||
|
# it and publishes only the registration token, which the runner's own store
|
||||||
|
# identity reads. `docs/scheduler/ci.md::Registration`.
|
||||||
|
#
|
||||||
|
# Non-ephemeral on purpose: the runner writes its registered credentials
|
||||||
|
# (`.runner`) to its state dir and reuses them, so an ephemeral container
|
||||||
|
# would register a new runner on every restart.
|
||||||
|
{
|
||||||
|
pkgs,
|
||||||
|
lib,
|
||||||
|
config,
|
||||||
|
...
|
||||||
|
}:
|
||||||
|
let
|
||||||
|
cfg = config.services.hyperhive.deploy.ci-runner;
|
||||||
|
forgeCfg = config.services.hyperhive.swarm.forge;
|
||||||
|
baoCfg = config.services.hyperhive.swarm.bao;
|
||||||
|
baoDeploy = config.services.hyperhive.deploy.bao;
|
||||||
|
gatewayCfg = config.services.hyperhive.gateway;
|
||||||
|
networkCfg = config.services.hyperhive.network;
|
||||||
|
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
|
||||||
|
|
||||||
|
machine = "swarm-ci-runner";
|
||||||
|
# The upstream module names its unit `gitea-runner-<instance>`.
|
||||||
|
instance = "swarm";
|
||||||
|
runnerUnit = "gitea-runner-${instance}";
|
||||||
|
|
||||||
|
# What the runner registers against and records in `.runner`. The precond
|
||||||
|
# below compares the two, so both come from here.
|
||||||
|
forgeUrl = "https://${forgeCfg.domain}";
|
||||||
|
|
||||||
|
# The forge's gateway serves a leaf from the swarm CA in self-signed mode.
|
||||||
|
# The runner daemon is Go, whose `SSL_CERT_FILE` replaces the system store,
|
||||||
|
# so it gets the assembled bundle; the Node actions it runs get
|
||||||
|
# `NODE_EXTRA_CA_CERTS`, which is additive, further down.
|
||||||
|
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
|
||||||
|
caBundleModule = caTrust.trustBundle {
|
||||||
|
inherit pkgs;
|
||||||
|
name = machine;
|
||||||
|
consumers = [ runnerUnit ];
|
||||||
|
};
|
||||||
|
|
||||||
|
# The upstream unit is `DynamicUser`, so its state dir is
|
||||||
|
# `/var/lib/private/gitea-runner`; `/var/lib/gitea-runner` is a symlink to
|
||||||
|
# it, which is the spelling a step run as root (`+`) can follow.
|
||||||
|
runnerFile = "/var/lib/gitea-runner/${instance}/.runner";
|
||||||
|
|
||||||
|
# Inside the container, root-only. The token file is the runner's
|
||||||
|
# `EnvironmentFile` (`TOKEN=<value>`); the host unit below writes it through
|
||||||
|
# the container's root, at `hostTokenFile`.
|
||||||
|
stateDir = "/var/lib/swarm-ci-runner";
|
||||||
|
tokenFile = "${stateDir}/registration-token.env";
|
||||||
|
hostTokenFile = "/var/lib/nixos-containers/${machine}${tokenFile}";
|
||||||
|
# The epoch second the daemon last started. The precond reads the daemon's
|
||||||
|
# journal from there.
|
||||||
|
daemonStartedFile = "${stateDir}/daemon-started";
|
||||||
|
|
||||||
|
# The nix half of `swarm_secret_client::ci::registration_token_path`, in the
|
||||||
|
# form `bao kv get` takes (no KV v2 `data/`).
|
||||||
|
storePath = "secret/swarm/services/ci-runner/registration-token";
|
||||||
|
|
||||||
|
# What Forgejo answers a runner it has no record of — deleted in the admin
|
||||||
|
# panel, or a forge whose database was replaced — on every call but
|
||||||
|
# `Register` (`routers/api/actions/runner/interceptor.go`). The daemon logs
|
||||||
|
# it and exits 1 when its startup `Declare` gets it. ⚠️ Nothing this module
|
||||||
|
# prints may contain it: the precond would read its own line as a rejection.
|
||||||
|
rejectedMessage = "unregistered runner";
|
||||||
|
|
||||||
|
haveClientIdentity =
|
||||||
|
baoDeploy.ciRunnerClientCertFile != null && baoDeploy.ciRunnerClientKeyFile != null;
|
||||||
|
|
||||||
|
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
||||||
|
refreshConsumer = import ./lib/refresh-consumer.nix { };
|
||||||
|
storeRetry = import ./lib/store-retry.nix { };
|
||||||
|
|
||||||
|
privateNetwork = true;
|
||||||
|
in
|
||||||
|
{
|
||||||
|
options.services.hyperhive.deploy.ci-runner = {
|
||||||
|
# `enable` is in ./deploy.nix.
|
||||||
|
|
||||||
|
name = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "swarm-ci-runner";
|
||||||
|
description = ''
|
||||||
|
Runner name as shown in the Forgejo admin panel.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
concurrency = lib.mkOption {
|
||||||
|
type = lib.types.ints.positive;
|
||||||
|
default = 1;
|
||||||
|
example = 4;
|
||||||
|
description = ''
|
||||||
|
Maximum number of workflow jobs the runner executes in parallel.
|
||||||
|
Each job gets its own temporary working directory; parallel jobs
|
||||||
|
share the host's nix store and the container's cargo registry
|
||||||
|
cache. Higher values trade memory and CPU headroom for throughput.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
labels = lib.mkOption {
|
||||||
|
type = lib.types.listOf lib.types.str;
|
||||||
|
default = [ "hive-ci:host" ];
|
||||||
|
example = [
|
||||||
|
"hive-ci:host"
|
||||||
|
"nix:host"
|
||||||
|
];
|
||||||
|
description = ''
|
||||||
|
Runner labels in `<name>:<scheme>` format. The `host` scheme runs
|
||||||
|
commands directly in the container (no docker/podman). This repo's
|
||||||
|
workflows target the default with `runs-on: hive-ci`.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
package = lib.mkOption {
|
||||||
|
type = lib.types.package;
|
||||||
|
default = pkgs.gitea-actions-runner;
|
||||||
|
defaultText = lib.literalExpression "pkgs.gitea-actions-runner";
|
||||||
|
description = "gitea-actions-runner package.";
|
||||||
|
};
|
||||||
|
|
||||||
|
jobTimeout = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "1h";
|
||||||
|
example = "3h";
|
||||||
|
description = ''
|
||||||
|
Per-job wall-clock timeout the runner enforces (act_runner's
|
||||||
|
`runner.timeout`). A job that exceeds it is killed, so a hung build
|
||||||
|
does not hold a runner slot indefinitely. Accepts a Go duration
|
||||||
|
string (`30m`, `1h`, `2h30m`). Enforced by the runner process, so
|
||||||
|
it only fires while that process is healthy.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
config = lib.mkIf cfg.enable {
|
||||||
|
assertions = [
|
||||||
|
{
|
||||||
|
assertion = haveClientIdentity;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.deploy.ci-runner.enable requires the runner's
|
||||||
|
swarm-secret-store client identity: set both
|
||||||
|
|
||||||
|
services.hyperhive.deploy.bao.ciRunnerClientCertFile
|
||||||
|
services.hyperhive.deploy.bao.ciRunnerClientKeyFile
|
||||||
|
|
||||||
|
swarm-bao-ci-runner.service reads the forge's runner registration
|
||||||
|
token with it, and the runner cannot register without that token.
|
||||||
|
|
||||||
|
On the host that runs the store, glue-bao-tls.nix supplies both as
|
||||||
|
defaults. Elsewhere, copy ci-runner.pem and ci-runner-key.pem from
|
||||||
|
the store host's /var/lib/swarm-bao-pki/ and name them here.
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# The container hangs off the bridge, takes its address from the
|
||||||
|
# gateway's DHCP and resolves through its dnsmasq. The gateway also runs
|
||||||
|
# `hive-tls-ca`, which writes the trust bundle bound in below.
|
||||||
|
services.hyperhive.network.enable = lib.mkDefault true;
|
||||||
|
services.hyperhive.gateway.enable = lib.mkDefault true;
|
||||||
|
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
|
||||||
|
|
||||||
|
# Shaped after ./swarm-grafana.nix's `swarm-bao-grafana-oidc`, except that
|
||||||
|
# a missing token fails too: swarm-controller publishes it at its start and
|
||||||
|
# every five minutes, so a retry is what turns "no value there" into one.
|
||||||
|
systemd.services.swarm-bao-ci-runner = {
|
||||||
|
description = "fetch the CI runner's registration token from the swarm secret store";
|
||||||
|
after = lib.optionals baoDeploy.enable [
|
||||||
|
"swarm-bao-pki.service"
|
||||||
|
"container@${baoCfg.machine}.service"
|
||||||
|
];
|
||||||
|
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
|
||||||
|
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
|
||||||
|
before = [ "container@${machine}.service" ];
|
||||||
|
wantedBy = [
|
||||||
|
"multi-user.target"
|
||||||
|
"container@${machine}.service"
|
||||||
|
];
|
||||||
|
path = [
|
||||||
|
baoDeploy.package
|
||||||
|
pkgs.coreutils
|
||||||
|
pkgs.systemd
|
||||||
|
];
|
||||||
|
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
|
||||||
|
serviceConfig = storeRetry.serviceConfig // {
|
||||||
|
Type = "oneshot";
|
||||||
|
RemainAfterExit = true;
|
||||||
|
SyslogIdentifier = "swarm-bao-ci-runner";
|
||||||
|
TimeoutStartSec = 30;
|
||||||
|
};
|
||||||
|
environment = {
|
||||||
|
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||||
|
BAO_CLIENT_CERT = baoDeploy.ciRunnerClientCertFile;
|
||||||
|
BAO_CLIENT_KEY = baoDeploy.ciRunnerClientKeyFile;
|
||||||
|
}
|
||||||
|
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||||||
|
BAO_CACERT = baoDeploy.serverCaFile;
|
||||||
|
};
|
||||||
|
script = ''
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
${atomicWriteSecret}
|
||||||
|
${refreshConsumer}
|
||||||
|
|
||||||
|
err="$(mktemp)"
|
||||||
|
trap 'rm -f "$err"' EXIT
|
||||||
|
|
||||||
|
# `-token-only` answers on stdout and needs no token helper.
|
||||||
|
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||||||
|
echo "could not log in to swarm-bao with the CI runner's certificate; leaving its registration token as it is." >&2
|
||||||
|
if [ -s "$err" ]; then
|
||||||
|
cat "$err" >&2
|
||||||
|
else
|
||||||
|
echo "bao failed without writing a diagnostic." >&2
|
||||||
|
fi
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
export BAO_TOKEN
|
||||||
|
|
||||||
|
if ! token="$(bao kv get -field=value ${lib.escapeShellArg storePath} 2>"$err")"; then
|
||||||
|
echo "swarm-bao did not return ${storePath}; retrying until swarm-controller publishes it." >&2
|
||||||
|
if [ -s "$err" ]; then
|
||||||
|
cat "$err" >&2
|
||||||
|
else
|
||||||
|
echo "bao failed without writing a diagnostic." >&2
|
||||||
|
fi
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -z "$token" ]; then
|
||||||
|
echo "swarm-bao returned an empty ${storePath}; retrying." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Root-owned: systemd reads an `EnvironmentFile` as root, before the
|
||||||
|
# runner drops to its dynamic user.
|
||||||
|
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf hostTokenFile)}
|
||||||
|
if secret_differs ${lib.escapeShellArg hostTokenFile} "TOKEN=$token"; then
|
||||||
|
atomic_write_secret 0600 "" ${lib.escapeShellArg hostTokenFile} "TOKEN=$token"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# `EnvironmentFile=` is read once, at start. A changed token makes the
|
||||||
|
# upstream pre-start step register again.
|
||||||
|
refresh_consumer ${lib.escapeShellArg machine} ${runnerUnit}.service ${lib.escapeShellArg hostTokenFile}
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
systemd.services."container@${machine}" = caTrust.containerOrdering;
|
||||||
|
|
||||||
|
containers.${machine} = {
|
||||||
|
autoStart = true;
|
||||||
|
ephemeral = false;
|
||||||
|
# Journal files on the host, not inside the container: nixpkgs hardcodes
|
||||||
|
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
|
||||||
|
extraFlags = [ "--link-journal=host" ];
|
||||||
|
inherit privateNetwork;
|
||||||
|
hostBridge = networkCfg.bridgeName;
|
||||||
|
|
||||||
|
bindMounts = {
|
||||||
|
# Builds go through the HOST nix-daemon, so they inherit its
|
||||||
|
# buildMachines and max-jobs. The directory rather than the socket
|
||||||
|
# file, so the mount survives a daemon restart replacing the inode.
|
||||||
|
"/nix/var/nix/daemon-socket" = {
|
||||||
|
hostPath = "/nix/var/nix/daemon-socket";
|
||||||
|
isReadOnly = false;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
// caTrust.bindMount;
|
||||||
|
|
||||||
|
config =
|
||||||
|
{ pkgs, lib, ... }:
|
||||||
|
{
|
||||||
|
imports = [
|
||||||
|
../container-modules/swarm-container.nix
|
||||||
|
caBundleModule
|
||||||
|
];
|
||||||
|
|
||||||
|
# Its own netns, so it keeps its own firewall; resolvconf stays on
|
||||||
|
# and serves `networking.nameservers` below.
|
||||||
|
services.hyperhive.swarmContainer = {
|
||||||
|
inherit privateNetwork;
|
||||||
|
writesOwnResolvConf = false;
|
||||||
|
};
|
||||||
|
|
||||||
|
# The gateway's dnsmasq on the bridge answers the forge's name with
|
||||||
|
# the bridge IP where this host runs the forge, and forwards it
|
||||||
|
# upstream everywhere else.
|
||||||
|
networking.nameservers = [ networkCfg.bridgeIp ];
|
||||||
|
networking.interfaces.eth0.useDHCP = true;
|
||||||
|
|
||||||
|
# nspawn containers can't create user namespaces, so nix sandboxing
|
||||||
|
# fails; see docs/process/gotchas.md. `fallback` is a client-side
|
||||||
|
# option sent per daemon connection, so the host's value does not
|
||||||
|
# reach builds started here.
|
||||||
|
nix.settings.sandbox-fallback = lib.mkForce true;
|
||||||
|
nix.settings.fallback = true;
|
||||||
|
nix.settings.experimental-features = [
|
||||||
|
"nix-command"
|
||||||
|
"flakes"
|
||||||
|
];
|
||||||
|
|
||||||
|
# Both the service and its socket are off: the socket unit owns
|
||||||
|
# `/nix/var/nix/daemon-socket/socket` and would race the bind mount.
|
||||||
|
systemd.globalEnvironment.NIX_REMOTE = "daemon";
|
||||||
|
systemd.services.nix-daemon.enable = false;
|
||||||
|
systemd.sockets.nix-daemon.enable = false;
|
||||||
|
|
||||||
|
systemd.tmpfiles.rules = [
|
||||||
|
"d ${stateDir} 0700 root root -"
|
||||||
|
# `EnvironmentFile=` fails the unit when the file is missing; an
|
||||||
|
# empty one leaves `TOKEN` unset for the precond to report.
|
||||||
|
"f ${tokenFile} 0600 root root -"
|
||||||
|
];
|
||||||
|
|
||||||
|
services.gitea-actions-runner.package = cfg.package;
|
||||||
|
|
||||||
|
services.gitea-actions-runner.instances.${instance} = {
|
||||||
|
enable = true;
|
||||||
|
inherit (cfg) name labels;
|
||||||
|
url = forgeUrl;
|
||||||
|
inherit tokenFile;
|
||||||
|
settings = {
|
||||||
|
runner.capacity = cfg.concurrency;
|
||||||
|
runner.timeout = cfg.jobTimeout;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
environment.systemPackages = [
|
||||||
|
pkgs.git
|
||||||
|
pkgs.nix
|
||||||
|
];
|
||||||
|
|
||||||
|
systemd.services.${runnerUnit} = {
|
||||||
|
# `path` adds to the PATH the upstream module builds instead of
|
||||||
|
# replacing it.
|
||||||
|
path = [ pkgs.nix ];
|
||||||
|
# The upstream unit is `DynamicUser`, whose state dir systemd mounts
|
||||||
|
# `noexec`, so a job could not execute what it builds in its
|
||||||
|
# workspace. `ExecPaths=` resolves against the host root, where
|
||||||
|
# `/var/lib/gitea-runner` is a symlink into `private/`; `-` makes
|
||||||
|
# whichever spelling is absent a no-op.
|
||||||
|
serviceConfig.ExecPaths = [
|
||||||
|
"-/var/lib/gitea-runner"
|
||||||
|
"-/var/lib/private/gitea-runner"
|
||||||
|
];
|
||||||
|
environment = lib.mkIf caTrust.useSelfSigned {
|
||||||
|
NODE_EXTRA_CA_CERTS = caTrust.caContainerPath;
|
||||||
|
};
|
||||||
|
# Runs before upstream's pre-start step, which registers whenever
|
||||||
|
# `.runner` is absent or the token or labels changed. This one
|
||||||
|
# removes `.runner` in the two cases upstream cannot see, and holds
|
||||||
|
# the unit down until there is a token.
|
||||||
|
#
|
||||||
|
# `+`: root, because `.runner`, the token and the system journal
|
||||||
|
# are all out of the dynamic user's reach.
|
||||||
|
serviceConfig.ExecStartPre = lib.mkMerge [
|
||||||
|
(lib.mkBefore [
|
||||||
|
"+${pkgs.writeShellScript "swarm-ci-runner-precond" ''
|
||||||
|
set -eu
|
||||||
|
runner=${runnerFile}
|
||||||
|
started=${daemonStartedFile}
|
||||||
|
|
||||||
|
# The forge refused the registration the daemon last started
|
||||||
|
# with. Read on the next start rather than when the daemon
|
||||||
|
# exits, so journald has its last lines by then.
|
||||||
|
if [ -f "$started" ]; then
|
||||||
|
since="$(< "$started")"
|
||||||
|
${pkgs.coreutils}/bin/rm -f "$started"
|
||||||
|
if ${pkgs.systemd}/bin/journalctl -u ${runnerUnit}.service --since "@$since" -o cat --no-pager \
|
||||||
|
| ${pkgs.gnugrep}/bin/grep -qF ${lib.escapeShellArg rejectedMessage}; then
|
||||||
|
echo "the forge has no record of this runner's registration; registering again" >&2
|
||||||
|
${pkgs.coreutils}/bin/rm -f "$runner"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# `.runner` records the address it registered at, and the
|
||||||
|
# daemon dials that one, not the configured one.
|
||||||
|
if [ -f "$runner" ] \
|
||||||
|
&& [ "$(${pkgs.jq}/bin/jq -r .address "$runner")" != ${lib.escapeShellArg forgeUrl} ]; then
|
||||||
|
echo "$runner names another forge address; registering again at ${forgeUrl}" >&2
|
||||||
|
${pkgs.coreutils}/bin/rm -f "$runner"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Upstream registers on a changed token hash, an empty token
|
||||||
|
# included, and a registration with no token fails.
|
||||||
|
if [ -z "''${TOKEN:-}" ]; then
|
||||||
|
echo "no registration token yet; swarm-bao-ci-runner on the host delivers it" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
''}"
|
||||||
|
])
|
||||||
|
(lib.mkAfter [
|
||||||
|
"+${pkgs.writeShellScript "swarm-ci-runner-mark-start" ''
|
||||||
|
${pkgs.coreutils}/bin/date +%s > ${daemonStartedFile}
|
||||||
|
''}"
|
||||||
|
])
|
||||||
|
];
|
||||||
|
# Restarted until the token arrives, however long the store and
|
||||||
|
# the controller take. `mkForce`: upstream sets both.
|
||||||
|
serviceConfig.Restart = lib.mkForce "on-failure";
|
||||||
|
serviceConfig.RestartSec = lib.mkForce 15;
|
||||||
|
startLimitIntervalSec = 0;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
@ -54,6 +54,7 @@ let
|
||||||
deployCfg.bao.natsCommonName
|
deployCfg.bao.natsCommonName
|
||||||
deployCfg.bao.granterCommonName
|
deployCfg.bao.granterCommonName
|
||||||
deployCfg.bao.natsAuthCommonName
|
deployCfg.bao.natsAuthCommonName
|
||||||
|
deployCfg.bao.ciRunnerCommonName
|
||||||
]
|
]
|
||||||
# The per-hive subjects, spelled out per hive rather than as the prefix.
|
# The per-hive subjects, spelled out per hive rather than as the prefix.
|
||||||
# The prefix alone would reserve the wrong string: the role for hive
|
# The prefix alone would reserve the wrong string: the role for hive
|
||||||
|
|
|
||||||
|
|
@ -170,7 +170,7 @@ let
|
||||||
_: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile
|
_: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile
|
||||||
) baoGrantWithConsumers.systemd.services;
|
) baoGrantWithConsumers.systemd.services;
|
||||||
|
|
||||||
# The thirteen units that write a `swarm-*` grant, by name, for the discovery
|
# The fourteen units that write a `swarm-*` grant, by name, for the discovery
|
||||||
# control below.
|
# control below.
|
||||||
grantingUnitNames = [
|
grantingUnitNames = [
|
||||||
"swarm-bao-controller-policy"
|
"swarm-bao-controller-policy"
|
||||||
|
|
@ -186,6 +186,7 @@ let
|
||||||
"swarm-bao-nats-auth-policy"
|
"swarm-bao-nats-auth-policy"
|
||||||
"swarm-bao-operator-viewer-policy"
|
"swarm-bao-operator-viewer-policy"
|
||||||
"swarm-bao-hive-leaf-policy"
|
"swarm-bao-hive-leaf-policy"
|
||||||
|
"swarm-bao-ci-runner-policy"
|
||||||
];
|
];
|
||||||
|
|
||||||
# Comment lines dropped first: both the HCL and the scripts explain
|
# Comment lines dropped first: both the HCL and the scripts explain
|
||||||
|
|
@ -794,24 +795,24 @@ let
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# A store host without the granter's pair writes its grants some other
|
# A store host without the granter's pair writes its grants some other
|
||||||
# way, so none of the thirteen units may exist. Without this arm
|
# way, so none of the fourteen units may exist. Without this arm
|
||||||
# `lib.mkIf haveGranter` could be dropped from any of them and every other
|
# `lib.mkIf haveGranter` could be dropped from any of them and every other
|
||||||
# case here would still pass.
|
# case here would still pass.
|
||||||
name = "without the granter's pair none of the thirteen granting units render";
|
name = "without the granter's pair none of the fourteen granting units render";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGranterOptOut.systemd.services;
|
s = baoGranterOptOut.systemd.services;
|
||||||
in
|
in
|
||||||
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
|
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
|
||||||
# The control: the same store with the pair renders all thirteen.
|
# The control: the same store with the pair renders all fourteen.
|
||||||
&& lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames;
|
&& lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# 🩸 What replaced the silent skip. With no bootstrap token the thirteen still
|
# 🩸 What replaced the silent skip. With no bootstrap token the fourteen still
|
||||||
# render, and a refused granter fails them with the step that fixes it.
|
# render, and a refused granter fails them with the step that fixes it.
|
||||||
# A store host that never named a token is told to name one, since the
|
# A store host that never named a token is told to name one, since the
|
||||||
# unit that sets the granter up renders only where it has.
|
# unit that sets the granter up renders only where it has.
|
||||||
name = "a store host without a bootstrap token renders the thirteen, each failing loudly with the one-time step";
|
name = "a store host without a bootstrap token renders the fourteen, each failing loudly with the one-time step";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
s = baoGranterNoToken.systemd.services;
|
s = baoGranterNoToken.systemd.services;
|
||||||
|
|
@ -1254,6 +1255,62 @@ let
|
||||||
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
|
&& !(lib.hasInfix "secret/data/swarm/hives/*" s)
|
||||||
&& !(lib.hasInfix "secret/metadata/swarm/hives" s);
|
&& !(lib.hasInfix "secret/metadata/swarm/hives" s);
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
# `forge::runner_token` writes the runner's registration token and keeps
|
||||||
|
# the last value in memory, so its stanza has no `read`. Pinned as the
|
||||||
|
# whole stanza, so an added capability fails, and spelled to the leaf:
|
||||||
|
# `services/*` would reach every swarm service's OIDC client secret.
|
||||||
|
name = "the controller writes the CI runner's registration token and cannot read it back";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||||
|
in
|
||||||
|
lib.hasInfix "path \"secret/data/swarm/services/ci-runner/registration-token\" {\n capabilities = [\"create\", \"update\"]\n}" s
|
||||||
|
&& !(lib.hasInfix "secret/data/swarm/services/*" s);
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# The runner's role is the token's only reader. Its policy is the one
|
||||||
|
# leaf, read-only, under the runner's own subject.
|
||||||
|
name = "the CI runner's grant is its registration token and nothing else";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = baoGrantHere.systemd.services.swarm-bao-ci-runner-policy.script;
|
||||||
|
in
|
||||||
|
lib.hasInfix "path \"secret/data/swarm/services/ci-runner/registration-token\" {" s
|
||||||
|
&& lib.hasInfix "capabilities = [\"read\"]" s
|
||||||
|
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1
|
||||||
|
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
|
||||||
|
&& !(lib.hasInfix "secret/data/swarm/agents" s)
|
||||||
|
&& !(lib.hasInfix "secret/data/swarm/hives" s)
|
||||||
|
&& lib.hasInfix "auth/cert/certs/swarm-ci-runner" s
|
||||||
|
&& lib.hasInfix "allowed_common_names=swarm-ci-runner" s
|
||||||
|
&& lib.hasInfix "token_policies=swarm-ci-runner" s;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# Read access to the token is a grant on its path, so the policy units
|
||||||
|
# that name it are the whole list of who reaches it: the controller,
|
||||||
|
# which writes it, and the runner. The publisher's `services/*` write
|
||||||
|
# also covers it and holds no read.
|
||||||
|
name = "no granting unit but the controller's and the CI runner's names the runner's token";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
naming = lib.filter (
|
||||||
|
u: lib.hasInfix "ci-runner/registration-token" baoGrantHere.systemd.services.${u}.script
|
||||||
|
) grantingUnitNames;
|
||||||
|
in
|
||||||
|
lib.sort lib.lessThan naming == [
|
||||||
|
"swarm-bao-ci-runner-policy"
|
||||||
|
"swarm-bao-controller-policy"
|
||||||
|
];
|
||||||
|
}
|
||||||
|
{
|
||||||
|
name = "the PKI unit signs the CI runner's leaf under its own subject";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
|
||||||
|
in
|
||||||
|
lib.hasInfix "/ci-runner.pem ]" s && lib.hasInfix "swarm-ci-runner \"\" clientAuth" s;
|
||||||
|
}
|
||||||
{
|
{
|
||||||
# Revocation, and the reason it is a stanza of its own: `delete` on the
|
# Revocation, and the reason it is a stanza of its own: `delete` on the
|
||||||
# `data/` path soft-deletes the newest version and leaves earlier ones
|
# `data/` path soft-deletes the newest version and leaves earlier ones
|
||||||
|
|
@ -1386,8 +1443,8 @@ let
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# What makes the case above mean something: discovery by the granter's
|
# What makes the case above mean something: discovery by the granter's
|
||||||
# certificate reaches all thirteen units, and each yields calls.
|
# certificate reaches all fourteen units, and each yields calls.
|
||||||
name = "the granter-policy check sees all thirteen granting units, and parses calls from each";
|
name = "the granter-policy check sees all fourteen granting units, and parses calls from each";
|
||||||
ok =
|
ok =
|
||||||
lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames
|
lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames
|
||||||
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits)
|
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits)
|
||||||
|
|
|
||||||
|
|
@ -47,10 +47,10 @@ let
|
||||||
# only hive without one is a hive told to have none.
|
# only hive without one is a hive told to have none.
|
||||||
noAgentQueue = hiveWithController { deploy.hive-controller.queue.agentNatsUrl = null; };
|
noAgentQueue = hiveWithController { deploy.hive-controller.queue.agentNatsUrl = null; };
|
||||||
|
|
||||||
# On the forge's host: the runner is refused anywhere else.
|
# On the store's host, which mints the runner's store identity.
|
||||||
withCi = hive {
|
withCi = hive {
|
||||||
deploy.forgejo.enable = true;
|
deploy.bao.enable = true;
|
||||||
deploy.forgejo.ci.enable = true;
|
deploy.ci-runner.enable = true;
|
||||||
};
|
};
|
||||||
|
|
||||||
# A priority collision is a property of the *option*, not
|
# A priority collision is a property of the *option*, not
|
||||||
|
|
@ -63,7 +63,7 @@ let
|
||||||
# stack (measured — this is not a hypothetical).
|
# stack (measured — this is not a hypothetical).
|
||||||
forceCiServiceConfigs =
|
forceCiServiceConfigs =
|
||||||
let
|
let
|
||||||
svcs = withCi.containers.hive-ci.config.systemd.services;
|
svcs = withCi.containers.swarm-ci-runner.config.systemd.services;
|
||||||
vals = lib.concatMap (s: builtins.attrValues (s.serviceConfig or { })) (builtins.attrValues svcs);
|
vals = lib.concatMap (s: builtins.attrValues (s.serviceConfig or { })) (builtins.attrValues svcs);
|
||||||
in
|
in
|
||||||
builtins.foldl' (acc: v: builtins.seq v acc) true vals;
|
builtins.foldl' (acc: v: builtins.seq v acc) true vals;
|
||||||
|
|
@ -245,7 +245,7 @@ let
|
||||||
# `gitea-actions-runner.nix` adds a plain `serviceConfig.*`
|
# `gitea-actions-runner.nix` adds a plain `serviceConfig.*`
|
||||||
# definition that collides with one of ours, and nix refuses to
|
# definition that collides with one of ours, and nix refuses to
|
||||||
# merge two plain definitions at *host* eval. No other check
|
# merge two plain definitions at *host* eval. No other check
|
||||||
# instantiates a host with `containers.hive-ci` actually enabled, so
|
# instantiates a host with `containers.swarm-ci-runner` actually enabled, so
|
||||||
# the collision only surfaces on operator deploy, not in CI.
|
# the collision only surfaces on operator deploy, not in CI.
|
||||||
name = "the CI container's unit definitions merge without a priority collision";
|
name = "the CI container's unit definitions merge without a priority collision";
|
||||||
ok = forceCiServiceConfigs;
|
ok = forceCiServiceConfigs;
|
||||||
|
|
|
||||||
|
|
@ -46,11 +46,21 @@ let
|
||||||
deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
|
deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
|
||||||
};
|
};
|
||||||
|
|
||||||
ciNoForge = hive { deploy.forgejo.ci.enable = true; };
|
# The runner alone: no hive, no forge, no store, its store identity placed
|
||||||
ciWithForge = hive {
|
# by hand.
|
||||||
deploy.forgejo.enable = true;
|
ciAlone = hive {
|
||||||
deploy.forgejo.ci.enable = true;
|
deploy.hive-controller.enable = false;
|
||||||
|
deploy.ci-runner.enable = true;
|
||||||
|
deploy.bao.ciRunnerClientCertFile = "/etc/pki/bao-ci-runner.pem";
|
||||||
|
deploy.bao.ciRunnerClientKeyFile = "/etc/pki/bao-ci-runner-key.pem";
|
||||||
};
|
};
|
||||||
|
ciNoIdentity = hive {
|
||||||
|
deploy.hive-controller.enable = false;
|
||||||
|
deploy.ci-runner.enable = true;
|
||||||
|
};
|
||||||
|
|
||||||
|
# A host config that still names the runner under the forge.
|
||||||
|
ciUnderForge = hive { deploy.forgejo.ci.enable = true; };
|
||||||
|
|
||||||
runsForge = m: m.services.hyperhive.deploy.forgejo.enable && m.containers ? hive-forge;
|
runsForge = m: m.services.hyperhive.deploy.forgejo.enable && m.containers ? hive-forge;
|
||||||
|
|
||||||
|
|
@ -122,13 +132,28 @@ let
|
||||||
== "https://forge.t.local/user/oauth2/authelia/callback";
|
== "https://forge.t.local/user/oauth2/authelia/callback";
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The runner reaches the forge through this host's gateway and is
|
# The runner reaches the forge by name and reads its token with its own
|
||||||
# registered through the local container. The second arm is the
|
# certificate, so nothing about the host it runs on is asked.
|
||||||
# control: a refusal that fires everywhere is not a check.
|
name = "the CI runner runs on a host with neither the forge nor a hive";
|
||||||
name = "CI is refused on a host that does not run the forge";
|
|
||||||
ok =
|
ok =
|
||||||
refusedOver ciNoForge "deploy.forgejo.enable on the same host"
|
ciAlone.containers ? swarm-ci-runner
|
||||||
&& !(refusedOver ciWithForge "deploy.forgejo.enable on the same host");
|
&& !(ciAlone.containers ? hive-forge)
|
||||||
|
&& !ciAlone.services.hyperhive.deploy.hive-controller.enable
|
||||||
|
&& !(refusedOver ciAlone "ci-runner")
|
||||||
|
&&
|
||||||
|
ciAlone.systemd.services.swarm-bao-ci-runner.environment.BAO_CLIENT_CERT
|
||||||
|
== "/etc/pki/bao-ci-runner.pem";
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# Without the identity the token never arrives; the control is the
|
||||||
|
# placed pair above.
|
||||||
|
name = "the CI runner is refused without its store identity";
|
||||||
|
ok = refusedOver ciNoIdentity "deploy.bao.ciRunnerClientCertFile";
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# A removed option evaluates to an assertion naming the replacement.
|
||||||
|
name = "a CI option under deploy.forgejo names its replacement under deploy.ci-runner";
|
||||||
|
ok = refusedOver ciUnderForge "services.hyperhive.deploy.ci-runner.enable instead";
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# Nothing writes the delivery path away from the forge, and a
|
# Nothing writes the delivery path away from the forge, and a
|
||||||
|
|
|
||||||
|
|
@ -116,6 +116,12 @@ let
|
||||||
deploy.forgejo.enable = true;
|
deploy.forgejo.enable = true;
|
||||||
deploy.forgejo.mirrors = [ aMirror ];
|
deploy.forgejo.mirrors = [ aMirror ];
|
||||||
};
|
};
|
||||||
|
controllerWithSelfActions = hive {
|
||||||
|
deploy.swarm-controller.enable = true;
|
||||||
|
deploy.forgejo.enable = true;
|
||||||
|
deploy.forgejo.mirrors = [ aMirror ];
|
||||||
|
deploy.forgejo.actionsFromSelf = true;
|
||||||
|
};
|
||||||
|
|
||||||
# The controller with every shared service on another host, and the same
|
# The controller with every shared service on another host, and the same
|
||||||
# controller hosting them all: the swarm UI's links must not tell the two
|
# controller hosting them all: the swarm UI's links must not tell the two
|
||||||
|
|
@ -150,7 +156,7 @@ let
|
||||||
serviceContainersWithCi = hive {
|
serviceContainersWithCi = hive {
|
||||||
deploy.hive-controller.enable = false;
|
deploy.hive-controller.enable = false;
|
||||||
deploy.allSwarmServices = true;
|
deploy.allSwarmServices = true;
|
||||||
deploy.forgejo.ci.enable = true;
|
deploy.ci-runner.enable = true;
|
||||||
};
|
};
|
||||||
serviceContainers = serviceContainersWithCi.containers;
|
serviceContainers = serviceContainersWithCi.containers;
|
||||||
cases = [
|
cases = [
|
||||||
|
|
@ -412,7 +418,7 @@ let
|
||||||
name = "a service container on a netns of its own keeps its firewall and resolvconf";
|
name = "a service container on a netns of its own keeps its firewall and resolvconf";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
c = serviceContainers.hive-ci.config;
|
c = serviceContainers.swarm-ci-runner.config;
|
||||||
in
|
in
|
||||||
c.networking.firewall.enable && c.networking.resolvconf.enable;
|
c.networking.firewall.enable && c.networking.resolvconf.enable;
|
||||||
}
|
}
|
||||||
|
|
@ -446,6 +452,22 @@ let
|
||||||
in
|
in
|
||||||
warns mirrorsNoController && !(warns controllerWithMirrors);
|
warns mirrorsNoController && !(warns controllerWithMirrors);
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
# The mirror is what `self` resolves `actions/checkout` to, so the two
|
||||||
|
# come together. The control is the same forge without the option.
|
||||||
|
name = "actionsFromSelf adds the checkout mirror and resolves actions from the forge";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
mirrorsOf =
|
||||||
|
m: builtins.fromJSON m.systemd.services.swarm-controller.environment.SWARM_CONTROLLER_FORGE_MIRRORS;
|
||||||
|
actionsUrlOf =
|
||||||
|
m: m.containers.hive-forge.config.services.forgejo.settings.actions.DEFAULT_ACTIONS_URL or null;
|
||||||
|
in
|
||||||
|
lib.elem "actions/checkout" (map (m: m.dest) (mirrorsOf controllerWithSelfActions))
|
||||||
|
&& actionsUrlOf controllerWithSelfActions == "self"
|
||||||
|
&& mirrorsOf controllerWithMirrors == [ aMirror ]
|
||||||
|
&& actionsUrlOf controllerWithMirrors == null;
|
||||||
|
}
|
||||||
{
|
{
|
||||||
name = "the old c0re org-avatar option lands on the controller's unit";
|
name = "the old c0re org-avatar option lands on the controller's unit";
|
||||||
ok =
|
ok =
|
||||||
|
|
|
||||||
|
|
@ -36,6 +36,7 @@ use crate::webhook::DeliveryKind;
|
||||||
pub mod agent_token;
|
pub mod agent_token;
|
||||||
pub mod legacy_tokens;
|
pub mod legacy_tokens;
|
||||||
pub mod objects;
|
pub mod objects;
|
||||||
|
pub mod runner_token;
|
||||||
pub mod site_admin;
|
pub mod site_admin;
|
||||||
|
|
||||||
/// An agent's open config-PR, as [`Client::list_open_config_prs`] reports it
|
/// An agent's open config-PR, as [`Client::list_open_config_prs`] reports it
|
||||||
|
|
|
||||||
|
|
@ -84,9 +84,8 @@ const OPERATORS_TEAM_UNITS: [&str; 7] = [
|
||||||
];
|
];
|
||||||
|
|
||||||
/// Periodic sync interval for pull-mirrors. Forgejo syncs mirrors on-access by
|
/// Periodic sync interval for pull-mirrors. Forgejo syncs mirrors on-access by
|
||||||
/// default, which re-introduces external DNS latency on every `git clone` (the
|
/// default, which re-introduces external DNS latency on every `git clone`. A
|
||||||
/// hive-ci runner shares the host netns and is therefore affected by host
|
/// fixed periodic interval isolates CI from transient DNS
|
||||||
/// resolver blips). A fixed periodic interval isolates CI from transient DNS
|
|
||||||
/// failures — a stale mirror is acceptable; a broken clone because of a
|
/// failures — a stale mirror is acceptable; a broken clone because of a
|
||||||
/// momentary DNS blip is not. Written in the form Forgejo echoes back, so an
|
/// momentary DNS blip is not. Written in the form Forgejo echoes back, so an
|
||||||
/// unchanged mirror compares equal and is not re-patched.
|
/// unchanged mirror compares equal and is not re-patched.
|
||||||
|
|
@ -118,7 +117,7 @@ Do **not** push directly to `main` — agents have read-only access.
|
||||||
";
|
";
|
||||||
|
|
||||||
/// The operator-declared pull-mirrors (nix `deploy.forgejo.mirrors`, plus the
|
/// The operator-declared pull-mirrors (nix `deploy.forgejo.mirrors`, plus the
|
||||||
/// CI-auto `actions/checkout` entry), JSON-encoded by `hive-forge/default.nix`.
|
/// `actions/checkout` entry `deploy.forgejo.actionsFromSelf` adds), JSON-encoded by `hive-forge/default.nix`.
|
||||||
/// Absent or empty means no mirrors.
|
/// Absent or empty means no mirrors.
|
||||||
const MIRRORS_ENV: &str = "SWARM_CONTROLLER_FORGE_MIRRORS";
|
const MIRRORS_ENV: &str = "SWARM_CONTROLLER_FORGE_MIRRORS";
|
||||||
|
|
||||||
|
|
|
||||||
116
swarm-controller/src/forge/runner_token.rs
Normal file
116
swarm-controller/src/forge/runner_token.rs
Normal file
|
|
@ -0,0 +1,116 @@
|
||||||
|
//! The forge's runner registration token, published to the swarm secret store
|
||||||
|
//! at [`ci::registration_token_path`], where the CI runner's fetch unit
|
||||||
|
//! (`nix/host-modules/swarm-ci-runner.nix`) reads it under the runner's own
|
||||||
|
//! certificate.
|
||||||
|
//!
|
||||||
|
//! Forgejo answers `GET /admin/runners/registration-token` with the same
|
||||||
|
//! instance-wide token until someone resets it. The runner re-registers on
|
||||||
|
//! every new value it reads, which leaves its previous entry on the forge, so
|
||||||
|
//! a pass writes only when the forge's token differs from the one this process
|
||||||
|
//! last wrote. The controller's store grant on the path is write-only, so that
|
||||||
|
//! last write is held in memory: a restarted controller writes the current
|
||||||
|
//! token once.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use swarm_secret_client::ci;
|
||||||
|
|
||||||
|
use super::Client;
|
||||||
|
|
||||||
|
/// How often [`spawn`] re-reads the forge's token.
|
||||||
|
const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
|
||||||
|
|
||||||
|
/// Whether `current`, the forge's token now, has to be written, given the one
|
||||||
|
/// this process last wrote (`None` before its first write).
|
||||||
|
fn needs_write(last_written: Option<&str>, current: &str) -> bool {
|
||||||
|
last_written != Some(current)
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Client {
|
||||||
|
/// The forge's instance-wide runner registration token.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// When the forge refuses the call, or answers without a token.
|
||||||
|
pub async fn runner_registration_token(&self) -> Result<String> {
|
||||||
|
let answer = self
|
||||||
|
.api
|
||||||
|
.admin_get_registration_token()
|
||||||
|
.await
|
||||||
|
.context("GET /admin/runners/registration-token")?;
|
||||||
|
answer
|
||||||
|
.token
|
||||||
|
.filter(|t| !t.is_empty())
|
||||||
|
.context("the forge answered without a runner registration token")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One pass: read the forge's token and write it when it is not the one
|
||||||
|
/// `last_written` holds. Returns the token it wrote, if it wrote one.
|
||||||
|
async fn pass(client: &Client, last_written: Option<&str>) -> Result<Option<String>> {
|
||||||
|
let token = client.runner_registration_token().await?;
|
||||||
|
if !needs_write(last_written, &token) {
|
||||||
|
return Ok(None);
|
||||||
|
}
|
||||||
|
let path = ci::registration_token_path()?;
|
||||||
|
let store = crate::store::connect()
|
||||||
|
.await
|
||||||
|
.context("logging in to the swarm secret store")?;
|
||||||
|
store
|
||||||
|
.write(
|
||||||
|
&path,
|
||||||
|
&ci::Credential {
|
||||||
|
value: token.clone(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("writing {path}"))?;
|
||||||
|
Ok(Some(token))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publish the forge's runner registration token now and every
|
||||||
|
/// [`RECONCILE_INTERVAL`] after. A pass that fails is logged and retried on
|
||||||
|
/// the next tick; it never stops the daemon.
|
||||||
|
pub fn spawn(client: Arc<Client>) {
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
|
||||||
|
let mut last_written: Option<String> = None;
|
||||||
|
loop {
|
||||||
|
ticker.tick().await;
|
||||||
|
match pass(&client, last_written.as_deref()).await {
|
||||||
|
Ok(Some(written)) => {
|
||||||
|
tracing::info!("runner registration token: written to the store");
|
||||||
|
last_written = Some(written);
|
||||||
|
}
|
||||||
|
Ok(None) => tracing::debug!("runner registration token: unchanged"),
|
||||||
|
Err(e) => tracing::warn!(
|
||||||
|
error = %format!("{e:#}"),
|
||||||
|
retry_in_s = RECONCILE_INTERVAL.as_secs(),
|
||||||
|
"runner registration token: pass failed; retrying next tick"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::needs_write;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_first_pass_writes() {
|
||||||
|
assert!(needs_write(None, "t1"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_unchanged_token_is_not_rewritten() {
|
||||||
|
// A rewrite of the same value is harmless to the runner, but it is a
|
||||||
|
// new store version every five minutes for nothing.
|
||||||
|
assert!(!needs_write(Some("t1"), "t1"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_reset_token_is_written() {
|
||||||
|
assert!(needs_write(Some("t1"), "t2"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -2314,8 +2314,9 @@ fn spawn_matrix_account_backfill(
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Start the forge's periodic passes — the swarm-wide objects, and agents'
|
/// Start the forge's periodic passes — the swarm-wide objects, the runner
|
||||||
/// tokens (which also sweeps legacy tokens; see `forge::legacy_tokens`) —
|
/// registration token, and agents' tokens (which also sweeps legacy tokens;
|
||||||
|
/// see `forge::legacy_tokens`) —
|
||||||
/// when a forge is configured. Lifted out of `main` for
|
/// when a forge is configured. Lifted out of `main` for
|
||||||
/// `clippy::too_many_lines`.
|
/// `clippy::too_many_lines`.
|
||||||
fn spawn_forge_workers(
|
fn spawn_forge_workers(
|
||||||
|
|
@ -2326,6 +2327,7 @@ fn spawn_forge_workers(
|
||||||
return;
|
return;
|
||||||
};
|
};
|
||||||
forge::objects::spawn(Arc::clone(&client));
|
forge::objects::spawn(Arc::clone(&client));
|
||||||
|
forge::runner_token::spawn(Arc::clone(&client));
|
||||||
let sched = Arc::clone(jobq);
|
let sched = Arc::clone(jobq);
|
||||||
forge::agent_token::spawn(client, move |agents| {
|
forge::agent_token::spawn(client, move |agents| {
|
||||||
if let Err(e) = queue_forge_token_mints(&sched, agents) {
|
if let Err(e) = queue_forge_token_mints(&sched, agents) {
|
||||||
|
|
|
||||||
54
swarm-secret-client/src/ci.rs
Normal file
54
swarm-secret-client/src/ci.rs
Normal file
|
|
@ -0,0 +1,54 @@
|
||||||
|
//! The CI runner agreement: where the forge's runner registration token lives,
|
||||||
|
//! and what the object at that path holds.
|
||||||
|
//!
|
||||||
|
//! `swarm-controller` fetches the token with the forge's admin API and writes
|
||||||
|
//! it here; `nix/host-modules/swarm-ci-runner.nix` reads it under the runner's
|
||||||
|
//! own certificate. That runner's read stanza is the only one in the store that
|
||||||
|
//! covers this path.
|
||||||
|
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
Error,
|
||||||
|
path::{Kind, principal_prefix},
|
||||||
|
};
|
||||||
|
|
||||||
|
/// The runner's name segment under [`Kind::Service`].
|
||||||
|
const SERVICE: &str = "ci-runner";
|
||||||
|
|
||||||
|
/// The path holding the forge's runner registration token.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
/// Never in practice: the name segment is a constant. The `Result` is
|
||||||
|
/// [`principal_prefix`]'s.
|
||||||
|
pub fn registration_token_path() -> Result<String, Error> {
|
||||||
|
let prefix = principal_prefix(Kind::Service, SERVICE)?;
|
||||||
|
Ok(format!("{prefix}/registration-token"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What the registration-token path holds.
|
||||||
|
///
|
||||||
|
/// No `Debug` derive: `value` registers a runner that receives jobs from every
|
||||||
|
/// repo on the forge, and a derived `Debug` is one `{:?}` away from a log line.
|
||||||
|
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
pub struct Credential {
|
||||||
|
/// The token itself. `swarm-ci-runner.nix` reads the store with
|
||||||
|
/// `bao kv get -field=value`, so this name is load-bearing for a reader
|
||||||
|
/// this crate does not control, the same as [`crate::matrix::Credential`]'s.
|
||||||
|
pub value: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_registration_token_is_under_the_runners_service_prefix() {
|
||||||
|
// The literal is what `swarm-bao.nix` grants the runner's role, and
|
||||||
|
// what the controller's write stanza names.
|
||||||
|
assert_eq!(
|
||||||
|
registration_token_path().expect("a constant segment is legal"),
|
||||||
|
"swarm/services/ci-runner/registration-token"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -5,7 +5,7 @@
|
||||||
//! the rules every path obeys ([`path`]), the translation from this
|
//! the rules every path obeys ([`path`]), the translation from this
|
||||||
//! deployment's environment into a logged-in client ([`client`]), and, per kind
|
//! deployment's environment into a logged-in client ([`client`]), and, per kind
|
||||||
//! of secret, the path it lives at together with the fields it holds
|
//! of secret, the path it lives at together with the fields it holds
|
||||||
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`]). Each of those is a thing the controller
|
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`], [`ci`]). Each of those is a thing the controller
|
||||||
//! and a hive must say identically, so it is said once here.
|
//! and a hive must say identically, so it is said once here.
|
||||||
//!
|
//!
|
||||||
//! [`policy`] is the same kind of agreement seen from the other side: which of
|
//! [`policy`] is the same kind of agreement seen from the other side: which of
|
||||||
|
|
@ -23,6 +23,7 @@
|
||||||
//! value inside it, and its doc explains why that is not circular.
|
//! value inside it, and its doc explains why that is not circular.
|
||||||
|
|
||||||
pub mod acp;
|
pub mod acp;
|
||||||
|
pub mod ci;
|
||||||
pub mod client;
|
pub mod client;
|
||||||
pub mod forge;
|
pub mod forge;
|
||||||
pub mod github;
|
pub mod github;
|
||||||
|
|
|
||||||
|
|
@ -40,6 +40,9 @@ pub enum Kind {
|
||||||
/// is what uses it.
|
/// is what uses it.
|
||||||
#[strum(serialize = "hives")]
|
#[strum(serialize = "hives")]
|
||||||
Hive,
|
Hive,
|
||||||
|
/// One swarm service — the things a swarm runs beside the controller.
|
||||||
|
#[strum(serialize = "services")]
|
||||||
|
Service,
|
||||||
/// The controller itself. There is one per swarm, so the name segment
|
/// The controller itself. There is one per swarm, so the name segment
|
||||||
/// below it does not vary; the shape stays uniform anyway, because one
|
/// below it does not vary; the shape stays uniform anyway, because one
|
||||||
/// grant pattern over `<kind>/<name>` is cheaper than a special case.
|
/// grant pattern over `<kind>/<name>` is cheaper than a special case.
|
||||||
|
|
@ -50,7 +53,7 @@ pub enum Kind {
|
||||||
impl Kind {
|
impl Kind {
|
||||||
/// Every kind, so callers that must cover the whole set can iterate rather
|
/// Every kind, so callers that must cover the whole set can iterate rather
|
||||||
/// than restate it — a second list is a list that drifts.
|
/// than restate it — a second list is a list that drifts.
|
||||||
pub const ALL: [Kind; 3] = [Kind::Agent, Kind::Hive, Kind::Controller];
|
pub const ALL: [Kind; 4] = [Kind::Agent, Kind::Hive, Kind::Service, Kind::Controller];
|
||||||
|
|
||||||
/// What to call the name in an error — singular, because the message reads
|
/// What to call the name in an error — singular, because the message reads
|
||||||
/// "hive name ... is not a single path segment".
|
/// "hive name ... is not a single path segment".
|
||||||
|
|
@ -59,6 +62,7 @@ impl Kind {
|
||||||
match self {
|
match self {
|
||||||
Kind::Agent => "agent",
|
Kind::Agent => "agent",
|
||||||
Kind::Hive => "hive",
|
Kind::Hive => "hive",
|
||||||
|
Kind::Service => "service",
|
||||||
Kind::Controller => "controller",
|
Kind::Controller => "controller",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -171,7 +175,7 @@ mod tests {
|
||||||
// decision. Spelling each one out is what makes adding a kind a
|
// decision. Spelling each one out is what makes adding a kind a
|
||||||
// deliberate edit.
|
// deliberate edit.
|
||||||
let segments: Vec<&str> = Kind::ALL.iter().map(|k| (*k).into()).collect();
|
let segments: Vec<&str> = Kind::ALL.iter().map(|k| (*k).into()).collect();
|
||||||
assert_eq!(segments, ["agents", "hives", "controller"]);
|
assert_eq!(segments, ["agents", "hives", "services", "controller"]);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue