Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
98d895cf9e | ||
|
|
07852cabc1 |
34 changed files with 634 additions and 617 deletions
|
|
@ -121,8 +121,18 @@ The mode is load-bearing, not cosmetic. Write permission on a
|
||||||
them, and the sticky bit is the only thing that would restrain that (it
|
them, and the sticky bit is the only thing that would restrain that (it
|
||||||
is not set here). A world-writable socket dir therefore lets anything
|
is not set here). A world-writable socket dir therefore lets anything
|
||||||
able to reach the path delete an agent's socket and bind its own — and
|
able to reach the path delete an agent's socket and bind its own — and
|
||||||
the gateway container has all of `/run/hive-agent` bind-mounted in.
|
nginx reaches all of `/run/hive-agent` as a plain host path. Dropping
|
||||||
Dropping `o=w` removes that permission rather than qualifying it.
|
`o=w` removes that permission rather than qualifying it.
|
||||||
|
|
||||||
|
⚠️ **The gateway's nginx and dnsmasq are host services, next to
|
||||||
|
`hive-c0re`** (see `docs/gateway.md`) — there is no namespace between
|
||||||
|
them and the rest of the host. That costs no network isolation: nginx
|
||||||
|
binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a
|
||||||
|
netns would have to be opened up for anyway.
|
||||||
|
🔑 It does mean nothing *implicitly* scopes the privileged reload verb,
|
||||||
|
so the scope is explicit: the unit name is hard-coded in `hive-priv` —
|
||||||
|
see `PrivRequest::ReloadGatewayNginx`. **A caller cannot name the unit,
|
||||||
|
so the verb cannot be steered at another service.**
|
||||||
|
|
||||||
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
|
⚠️ Contrast `/shared`, which *is* sticky world-writable (`1777`): it has
|
||||||
many legitimate writers, so sticky is the best available answer there.
|
many legitimate writers, so sticky is the best available answer there.
|
||||||
|
|
|
||||||
|
|
@ -381,7 +381,7 @@ that allows the underlying resource access.
|
||||||
| `manage_root_agent` | may lifecycle-manage the root/manager agent via `kill`/`start`/`restart` |
|
| `manage_root_agent` | may lifecycle-manage the root/manager agent via `kill`/`start`/`restart` |
|
||||||
| `read_host_journal` | `get_host_journal` MCP tool is registered + `GET /journal-host` requests are served |
|
| `read_host_journal` | `get_host_journal` MCP tool is registered + `GET /journal-host` requests are served |
|
||||||
| `query_agent_state` | may call `get_loose_ends` / `CountPendingReminders` targeting non-child agents |
|
| `query_agent_state` | may call `get_loose_ends` / `CountPendingReminders` targeting non-child agents |
|
||||||
| `infra_admin` | may call `restart(name)` on hive infrastructure containers (`hive-ci`, `hive-gateway`, `hive-forge`); each restart is logged to the dashboard AUDIT trail |
|
| `infra_admin` | may call `restart(name)` on hive infrastructure containers (`hive-ci`, `hive-forge`, `hive-matrix` — **not** `hive-gateway`, which is the host's nginx and is operator-only); each restart is logged to the dashboard AUDIT trail |
|
||||||
|
|
||||||
**Config storage** — per-agent capabilities live in
|
**Config storage** — per-agent capabilities live in
|
||||||
`/var/lib/hyperhive/meta/capabilities.json` alongside `tool-groups.json`.
|
`/var/lib/hyperhive/meta/capabilities.json` alongside `tool-groups.json`.
|
||||||
|
|
|
||||||
|
|
@ -500,9 +500,9 @@ Two things to know about the weights:
|
||||||
override in `meta/resource-limits.json`, so every agent carries the
|
override in `meta/resource-limits.json`, so every agent carries the
|
||||||
same value and the weight does *not* rank agents against each other.
|
same value and the weight does *not* rank agents against each other.
|
||||||
What `80` buys is that agents yield to everything **not** on this
|
What `80` buys is that agents yield to everything **not** on this
|
||||||
drop-in path: host services and the infra containers (`hive-ci`,
|
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||||
`hive-forge`, `hive-gateway`, `hive-matrix`), which stay at the
|
infra containers (`hive-ci`, `hive-forge`, `hive-matrix`), which stay
|
||||||
kernel default of `100`.
|
at the kernel default of `100`.
|
||||||
- `IOWeight=` is only honoured when the backing device runs the BFQ
|
- `IOWeight=` is only honoured when the backing device runs the BFQ
|
||||||
scheduler or has blk-iocost QoS enabled. On a host using
|
scheduler or has blk-iocost QoS enabled. On a host using
|
||||||
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
|
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# hive-gateway
|
# hive-gateway
|
||||||
|
|
||||||
Single nginx in front of every hyperhive web surface. Container `hive-gateway`, shared host netns, system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.
|
Single nginx in front of every hyperhive web surface. Runs on the **host**, next to hive-c0re; system-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`. (It lived in a `hive-gateway` container until #3088 — one that shared the host netns anyway, so the boundary gave no network isolation while costing a resolv.conf sync, a machine-bus reload and three bind mounts.)
|
||||||
|
|
||||||
## Vhost map
|
## Vhost map
|
||||||
|
|
||||||
|
|
@ -112,18 +112,19 @@ now set unconditionally for every agent. The mechanism:
|
||||||
a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
|
a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
|
||||||
if the socket is not yet bound, nginx returns 502 caught by the
|
if the socket is not yet bound, nginx returns 502 caught by the
|
||||||
`error_page 502 503 504 = /__hive_agent_unreachable` directive.
|
`error_page 502 503 504 = /__hive_agent_unreachable` directive.
|
||||||
The gateway container bind-mounts `/var/lib/hyperhive/gateway/` at
|
nginx includes `/var/lib/hyperhive/gateway/agents.conf` — the same
|
||||||
`/run/hive-state/`; nginx includes `/run/hive-state/agents.conf`.
|
path c0re writes, since both run on the host.
|
||||||
After each write, c0re triggers the appropriate nginx action inside
|
After each write, c0re triggers the appropriate nginx action via
|
||||||
the gateway container via `hive-priv` (which runs as root and has
|
`hive-priv` (which is root; hive-c0re runs as the unprivileged
|
||||||
`--machine=hive-gateway` transport rights that hive-c0re lacks).
|
`hive-core` user and cannot act on a system unit).
|
||||||
`hive-priv` queries `ActiveState` and dispatches:
|
`hive-priv` queries `ActiveState` and dispatches:
|
||||||
- active → `systemctl reload nginx` (SIGHUP, zero-downtime)
|
- active → `systemctl reload nginx` (SIGHUP, zero-downtime)
|
||||||
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
|
- failed → `systemctl reset-failed nginx` + `systemctl start nginx`
|
||||||
- otherwise → `systemctl start nginx`
|
- otherwise → `systemctl start nginx`
|
||||||
This is intentionally host-side: `IN_MOVED_TO` from an atomic rename
|
This is an explicit trigger rather than a path unit watching the
|
||||||
does not propagate across the nspawn mount-namespace boundary, so a
|
file: the write and the reload belong in one causal chain c0re can
|
||||||
path unit inside the container would never fire.
|
retry and report on (`RELOAD_PENDING`), not two units racing on an
|
||||||
|
inotify event.
|
||||||
|
|
||||||
c0re regenerates `agents.conf` (and triggers a reload) on two
|
c0re regenerates `agents.conf` (and triggers a reload) on two
|
||||||
triggers: every topology change (new/removed agents) and every 10s
|
triggers: every topology change (new/removed agents) and every 10s
|
||||||
|
|
@ -160,8 +161,8 @@ selected by which (if any) external TLS source is set:
|
||||||
| mode | config | cert source | `.well-known` scheme |
|
| mode | config | cert source | `.well-known` scheme |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | host hive-CA signs a gateway leaf (RSA-4096) | `https` |
|
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | host hive-CA signs a gateway leaf (RSA-4096) | `https` |
|
||||||
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx inside container via HTTP-01 | `https` |
|
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
|
||||||
| operator cert | `tls.certDir` set | bind-mounted from host | `https` |
|
| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
|
||||||
|
|
||||||
The `gateway.selfSignedTls` option is **deprecated and ignored** — self-signed
|
The `gateway.selfSignedTls` option is **deprecated and ignored** — self-signed
|
||||||
is now derived from the absence of `tls.certDir` / `tls.acme`. Setting it to
|
is now derived from the absence of `tls.certDir` / `tls.acme`. Setting it to
|
||||||
|
|
@ -182,7 +183,7 @@ services.hyperhive.gateway = {
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
nginx inside the gateway container obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). The gateway container shares the host network namespace (`privateNetwork = false`) so outbound ACME requests work without any extra routing. Certs are stored inside the container's persistent state dir (`/var/lib/acme/` inside `hive-gateway`; survives restarts because `ephemeral = false`).
|
nginx obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). Certs land in `/var/lib/acme/` on the host, managed by nixpkgs's `security.acme` in the ordinary way.
|
||||||
|
|
||||||
**Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.<swarm-domain>`, `chat.<swarm-domain>`) gets its own cert via separate ACME challenges — the swarm services default to names under `services.hyperhive.swarm.domain`, so **every one of those names must resolve to this host too**, not just the hive's own.
|
**Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.<swarm-domain>`, `chat.<swarm-domain>`) gets its own cert via separate ACME challenges — the swarm services default to names under `services.hyperhive.swarm.domain`, so **every one of those names must resolve to this host too**, not just the hive's own.
|
||||||
|
|
||||||
|
|
@ -194,9 +195,11 @@ Mutual exclusion: `tls.certDir` set together with `tls.acme.enable = true` fails
|
||||||
|
|
||||||
On by default, and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).
|
On by default, and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).
|
||||||
|
|
||||||
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. The leaf dir is bind-mounted read-only into the gateway container at `/run/hive-ca`; an in-container import unit copies the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`) with the owner/mode nginx needs.
|
The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.tls.caValidityDays`, default ~20y) under `services.hyperhive.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
|
||||||
|
|
||||||
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated, in-container leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
|
⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.** It does two jobs. It re-modes the leaf (`hive-tls-ca` writes the key `0600 root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a `0600` key fails the config test and blocks the unit), and it guarantees **every cert path the nginx config names exists** — which is what the swarm-services fallback below is for. Removing it re-creates the #3097 outage.
|
||||||
|
|
||||||
|
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
|
||||||
|
|
||||||
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](swarm/ca.md) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
|
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](swarm/ca.md) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
|
||||||
|
|
||||||
|
|
@ -220,11 +223,11 @@ services.hyperhive.gateway = {
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
The directory is bind-mounted read-only into the gateway container at `/run/hive-tls/`. nginx uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses.
|
nginx reads the directory directly and uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses.
|
||||||
|
|
||||||
`tls.certDir` and `tls.acme.enable` set together is an assertion error.
|
`tls.certDir` and `tls.acme.enable` set together is an assertion error.
|
||||||
|
|
||||||
**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx inside the gateway container runs as the `nginx` user and cannot read a key with that ownership. Fix with:
|
**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx runs as the `nginx` user and cannot read a key with that ownership. Fix with:
|
||||||
|
|
||||||
```nix
|
```nix
|
||||||
security.acme.certs."example.com".group = "nginx";
|
security.acme.certs."example.com".group = "nginx";
|
||||||
|
|
@ -424,11 +427,10 @@ JS/CSS assets skip `try_files` entirely and get the immutable cache
|
||||||
headers. Nix store paths are content-addressed — the hash changes on
|
headers. Nix store paths are content-addressed — the hash changes on
|
||||||
any content change — so `max-age=31536000` is safe.
|
any content change — so `max-age=31536000` is safe.
|
||||||
|
|
||||||
**Why nix store is reachable from the gateway container**: nspawn
|
**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
|
||||||
containers bind-mount `/nix/store` read-only by default. The
|
store path baked in at hive-c0re build time, and c0re (writing
|
||||||
`HIVE_AGENT_FRONTEND_DIR` path is a nix store path baked in at
|
`agents.conf`) and nginx (serving files from it) are on the same machine,
|
||||||
hive-c0re build time — the same path is visible to both c0re (writing
|
so they see the same store.
|
||||||
`agents.conf`) and the gateway nginx (serving files from it).
|
|
||||||
|
|
||||||
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
|
**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
|
||||||
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
|
unset (e.g. a build that predates `cfg.frontend`), each agent gets the
|
||||||
|
|
@ -493,9 +495,7 @@ services.hyperhive.gateway.auth = {
|
||||||
The credential store lives at the fixed path
|
The credential store lives at the fixed path
|
||||||
`/var/lib/hyperhive/gateway/gateway.htpasswd` on the host. A tmpfiles
|
`/var/lib/hyperhive/gateway/gateway.htpasswd` on the host. A tmpfiles
|
||||||
rule pre-creates the file on first boot; no manual path configuration
|
rule pre-creates the file on first boot; no manual path configuration
|
||||||
is required. The file is exposed inside the gateway container at
|
is required. nginx reads it at that path directly.
|
||||||
`/run/hive-state/gateway.htpasswd` via the existing gateway state
|
|
||||||
bind-mount.
|
|
||||||
|
|
||||||
Manage users with `hivectl gateway`. `hivectl` sends the request over the
|
Manage users with `hivectl gateway`. `hivectl` sends the request over the
|
||||||
host admin socket and the `hive-c0re` daemon performs the write at its
|
host admin socket and the `hive-c0re` daemon performs the write at its
|
||||||
|
|
|
||||||
|
|
@ -115,49 +115,19 @@ schemes pick their own.
|
||||||
dnsmasq is **authoritative** for the hive's own zones — answers
|
dnsmasq is **authoritative** for the hive's own zones — answers
|
||||||
`<hive-domain>`, `forge.<hive-domain>`, `matrix.<hive-domain>`
|
`<hive-domain>`, `forge.<hive-domain>`, `matrix.<hive-domain>`
|
||||||
queries with the bridge IP (where nginx is reachable). Everything
|
queries with the bridge IP (where nginx is reachable). Everything
|
||||||
else is forwarded to the host's own resolvers: dnsmasq reads the
|
else is forwarded to the host's own resolvers: dnsmasq runs on the host
|
||||||
gateway container's `/etc/resolv.conf`, the host copy nixos-container
|
and reads the host's `/etc/resolv.conf` directly. Containers don't need
|
||||||
makes at each container start. Containers don't need to know the
|
to know the upstream — they query the bridge IP and dnsmasq does the
|
||||||
upstream — they query the bridge IP and dnsmasq does the right thing
|
right thing per-name.
|
||||||
per-name.
|
|
||||||
|
|
||||||
That copy is one-shot — systemd-nspawn(1) is explicit that nothing
|
There is deliberately no fallback `server=`: dnsmasq queries all known
|
||||||
propagates into it after early init, because resolv.conf is normally
|
upstreams in parallel, so a hardcoded public resolver would take a share
|
||||||
updated by rename rather than in place. Left alone, a host network
|
of normal traffic, not just cover the gap.
|
||||||
change (new router, new lease, laptop moving networks) would strand
|
|
||||||
dnsmasq on a resolver that no longer answers, and every non-hive
|
|
||||||
lookup from every agent would hang until someone restarted the
|
|
||||||
gateway. The host-side **`hive-gateway-resolv`** path unit closes
|
|
||||||
that: it watches `/etc/resolv.conf`, `machinectl copy-to`s it into
|
|
||||||
the container, and reloads dnsmasq (`SIGHUP` — re-read upstreams +
|
|
||||||
flush cache, nothing dropped). The watch is armed before
|
|
||||||
`network-pre.target` so the boot's first DHCP write is caught as well,
|
|
||||||
and the sync also runs once per gateway start to pick up a resolver
|
|
||||||
change that happened while the container was down. A host file with no
|
|
||||||
`nameserver` line is
|
|
||||||
skipped rather than pushed, so a mid-rewrite snapshot can't blank the
|
|
||||||
hive's DNS. There is deliberately no fallback `server=`: dnsmasq
|
|
||||||
queries all known upstreams in parallel, so a hardcoded public
|
|
||||||
resolver would take a share of normal traffic, not just cover the gap.
|
|
||||||
|
|
||||||
Two alternatives that look simpler and aren't:
|
dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
|
||||||
|
directly, so a network change (new router, new lease, laptop moving
|
||||||
- **A path unit inside the container.** The host replaces
|
networks) reaches it the moment openresolv rewrites the file. There is
|
||||||
`/etc/resolv.conf` by rename, and that `IN_MOVED_TO` doesn't cross
|
nothing to synchronise and no unit watching for it.
|
||||||
the nspawn mount namespace — the same reason hive-c0re reloads nginx
|
|
||||||
from the host side after each `agents.conf` write.
|
|
||||||
- **Bind-mounting the host's `/etc/resolv.conf` into the container.**
|
|
||||||
openresolv writes a temp file and renames over the target, so the
|
|
||||||
bind mount would pin the *first* inode for the container's whole
|
|
||||||
lifetime — strictly worse than the copy, which at least a restart
|
|
||||||
clears. (Reachability is not the problem here: the gateway runs with
|
|
||||||
`privateNetwork = false`, so it shares the host's netns and can reach
|
|
||||||
anything the host can.)
|
|
||||||
|
|
||||||
`machinectl copy-to` is used rather than writing to the container's
|
|
||||||
rootfs from the host, so the push goes through the container's own
|
|
||||||
mount namespace and stays correct if `/etc` is ever assembled
|
|
||||||
differently (e.g. `system.etc.overlay`).
|
|
||||||
|
|
||||||
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
|
||||||
listener only accepts queries from the bridge interface (plus lo for
|
listener only accepts queries from the bridge interface (plus lo for
|
||||||
|
|
@ -289,4 +259,4 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
|
||||||
|
|
||||||
## Cross-references
|
## Cross-references
|
||||||
|
|
||||||
- `docs/gateway.md` — vhost map + the gateway container's other duties
|
- `docs/gateway.md` — vhost map + the gateway's other duties
|
||||||
|
|
|
||||||
|
|
@ -187,8 +187,16 @@ bind-mount compatibility with user namespace UID mapping and is tracked as a TOD
|
||||||
|
|
||||||
`hive-c0re` runs as the unprivileged system user `hive-core`
|
`hive-c0re` runs as the unprivileged system user `hive-core`
|
||||||
(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It cannot
|
(`/var/lib/hyperhive` owned by `hive-core:hive-core`). It cannot
|
||||||
directly invoke `nixos-container`, `journalctl -M`, or `systemctl
|
directly invoke `nixos-container`, `journalctl -M`, or act on a system
|
||||||
-M hive-gateway` — those require root. `hive-priv` fills this gap.
|
unit (`systemctl reload nginx`) — those require root. `hive-priv` fills
|
||||||
|
this gap.
|
||||||
|
|
||||||
|
⚠️ **`ReloadGatewayNginx` acts on a host unit, so nothing implicitly
|
||||||
|
scopes it.** Its containment is the unit name hard-coded in `hive-priv`:
|
||||||
|
a caller cannot name the unit, so the verb cannot be steered at another
|
||||||
|
service. **A privileged verb needs something bounding what it can act
|
||||||
|
on; when that isn't a namespace, it has to be a constant the caller
|
||||||
|
can't supply.**
|
||||||
|
|
||||||
### hive-priv
|
### hive-priv
|
||||||
|
|
||||||
|
|
@ -207,7 +215,7 @@ known operations; there is no arbitrary command pass-through:
|
||||||
| `DestroyContainer` | `nixos-container destroy <name>` |
|
| `DestroyContainer` | `nixos-container destroy <name>` |
|
||||||
| `ListContainers` | `nixos-container list` |
|
| `ListContainers` | `nixos-container list` |
|
||||||
| `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` |
|
| `ReadContainerJournal` | `journalctl -M <container> -n <n> [filters...]` |
|
||||||
| `ReloadGatewayNginx` | `systemctl -M hive-gateway reload/start/reset-failed nginx` |
|
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
|
||||||
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
|
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list + network isolation vars) |
|
||||||
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
|
||||||
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
|
||||||
|
|
|
||||||
|
|
@ -808,7 +808,7 @@ Bare `hivectl stop` stops everything; scope flags narrow it to specific sub-agen
|
||||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||||
* `--ci` — The CI runner container (`hive-ci`)
|
* `--ci` — The CI runner container (`hive-ci`)
|
||||||
* `--forge` — The forge container (`hive-forge`)
|
* `--forge` — The forge container (`hive-forge`)
|
||||||
* `--gateway` — The gateway container (`hive-gateway`)
|
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||||
* `--matrix` — The matrix container (`hive-matrix`)
|
* `--matrix` — The matrix container (`hive-matrix`)
|
||||||
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only
|
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only
|
||||||
* `--no-wait` — Return immediately after the stop DAGs are queued instead of waiting for them with live per-node progress
|
* `--no-wait` — Return immediately after the stop DAGs are queued instead of waiting for them with live per-node progress
|
||||||
|
|
@ -829,7 +829,7 @@ Bare `hivectl start` restores the agents stopped by the last broad-scope `stop`
|
||||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||||
* `--ci` — The CI runner container (`hive-ci`)
|
* `--ci` — The CI runner container (`hive-ci`)
|
||||||
* `--forge` — The forge container (`hive-forge`)
|
* `--forge` — The forge container (`hive-forge`)
|
||||||
* `--gateway` — The gateway container (`hive-gateway`)
|
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||||
* `--matrix` — The matrix container (`hive-matrix`)
|
* `--matrix` — The matrix container (`hive-matrix`)
|
||||||
* `--no-wait` — Return immediately after the start DAGs are queued instead of waiting for them with live per-node progress
|
* `--no-wait` — Return immediately after the start DAGs are queued instead of waiting for them with live per-node progress
|
||||||
|
|
||||||
|
|
@ -849,7 +849,7 @@ Bare `hivectl restart` restarts everything; scope flags narrow it.
|
||||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||||
* `--ci` — The CI runner container (`hive-ci`)
|
* `--ci` — The CI runner container (`hive-ci`)
|
||||||
* `--forge` — The forge container (`hive-forge`)
|
* `--forge` — The forge container (`hive-forge`)
|
||||||
* `--gateway` — The gateway container (`hive-gateway`)
|
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||||
* `--matrix` — The matrix container (`hive-matrix`)
|
* `--matrix` — The matrix container (`hive-matrix`)
|
||||||
* `--graceful` — Gracefully quiesce each agent on the stop half (see `stop --graceful`). Applies to agents only
|
* `--graceful` — Gracefully quiesce each agent on the stop half (see `stop --graceful`). Applies to agents only
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -79,8 +79,10 @@ lifecycle events, or another container's boot log.
|
||||||
- `unit` — filter to a systemd unit (e.g. `hive-c0re.service`).
|
- `unit` — filter to a systemd unit (e.g. `hive-c0re.service`).
|
||||||
- `container` — nspawn machine name verbatim. Agent containers use
|
- `container` — nspawn machine name verbatim. Agent containers use
|
||||||
the `h-<name>` prefix (e.g. `h-iris`); infrastructure containers
|
the `h-<name>` prefix (e.g. `h-iris`); infrastructure containers
|
||||||
use their full name (e.g. `hive-ci`, `hive-forge`, `hive-matrix`,
|
use their full name (e.g. `hive-ci`, `hive-forge`, `hive-matrix`).
|
||||||
`hive-gateway`). Omit for the host journal.
|
Omit for the host journal. The gateway has no machine — its nginx
|
||||||
|
runs on the host, so read it with `unit: nginx.service` and no
|
||||||
|
`container`.
|
||||||
- `lines` — how many lines to return (default 30, max 100).
|
- `lines` — how many lines to return (default 30, max 100).
|
||||||
- `priority` — minimum syslog level (`emerg` … `debug`).
|
- `priority` — minimum syslog level (`emerg` … `debug`).
|
||||||
- `grep` — regex matched against log message fields (`journalctl --grep`).
|
- `grep` — regex matched against log message fields (`journalctl --grep`).
|
||||||
|
|
|
||||||
|
|
@ -180,10 +180,12 @@ omitted — agents share the host netns, so there is no per-container net
|
||||||
counter (per-agent network needs the netns-isolation roadmap in
|
counter (per-agent network needs the netns-isolation roadmap in
|
||||||
`docs/network.md`).
|
`docs/network.md`).
|
||||||
|
|
||||||
**1NFR4** — start / stop / restart the four hive infrastructure
|
**1NFR4** — start / stop / restart the four hive infrastructure services
|
||||||
containers (`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`)
|
(`hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`) directly from the
|
||||||
directly from the dashboard, without needing an `infra_admin` agent's
|
dashboard, without needing an `infra_admin` agent's `restart` MCP tool.
|
||||||
`restart` MCP tool. One row per container: name, a `badge-ok`/`badge-fail`
|
Three are containers; `hive-gateway` is the host's `nginx.service`, and is
|
||||||
|
the one an agent may **not** restart — this panel is the way it gets
|
||||||
|
bounced. One row per service: name, a `badge-ok`/`badge-fail`
|
||||||
running/stopped dot, and `↺ R3ST4RT` + `■ ST0P` (running) or `▶ ST4RT`
|
running/stopped dot, and `↺ R3ST4RT` + `■ ST0P` (running) or `▶ ST4RT`
|
||||||
(stopped) buttons, same themed-confirm pattern as the K3PT ST4T3
|
(stopped) buttons, same themed-confirm pattern as the K3PT ST4T3
|
||||||
tombstone actions. Backed by
|
tombstone actions. Backed by
|
||||||
|
|
@ -460,7 +462,7 @@ The current capabilities are:
|
||||||
| `manage_root_agent` | allows the `set_status` / lifecycle tools on the root agent |
|
| `manage_root_agent` | allows the `set_status` / lifecycle tools on the root agent |
|
||||||
| `read_host_journal` | unlocks `get_host_journal` to read journald from inside a container |
|
| `read_host_journal` | unlocks `get_host_journal` to read journald from inside a container |
|
||||||
| `query_agent_state` | allows `get_loose_ends(agent: "<name>")` calls targeting other agents |
|
| `query_agent_state` | allows `get_loose_ends(agent: "<name>")` calls targeting other agents |
|
||||||
| `infra_admin` | allows `restart` on hive infrastructure containers (`hive-ci`, `hive-gateway`, `hive-forge`); each restart is logged to the AUDIT trail |
|
| `infra_admin` | allows `restart` on hive infrastructure containers (`hive-ci`, `hive-forge`, `hive-matrix`; the gateway is operator-only); each restart is logged to the AUDIT trail |
|
||||||
|
|
||||||
Each row is one agent. Columns are the capability names returned by
|
Each row is one agent. Columns are the capability names returned by
|
||||||
`GET /api/capabilities` as `caps: Vec<String>`. Checking or unchecking
|
`GET /api/capabilities` as `caps: Vec<String>`. Checking or unchecking
|
||||||
|
|
@ -1340,9 +1342,11 @@ below — some endpoints aren't in it yet.
|
||||||
rootfs every ~5 min, `-x` excluding the shared read-only nix store.
|
rootfs every ~5 min, `-x` excluding the shared read-only nix store.
|
||||||
`null` until the first sample lands.
|
`null` until the first sample lands.
|
||||||
- `POST /api/infra-container/{name}/{action}` — start / stop / restart a
|
- `POST /api/infra-container/{name}/{action}` — start / stop / restart a
|
||||||
hive infra container (C0R3 › 1NFR4 panel). `name` parses into the
|
hive infra service (C0R3 › 1NFR4 panel). `name` parses into the
|
||||||
`InfraContainer` allowlist (`hive-ci`/`hive-forge`/`hive-gateway`/
|
`InfraContainer` allowlist (`hive-ci`/`hive-forge`/`hive-gateway`/
|
||||||
`hive-matrix`, 400 on unknown), `action` ∈ `start|stop|restart`. Calls
|
`hive-matrix`, 400 on unknown), and the variant decides the unit —
|
||||||
|
`container@<name>.service`, or `nginx.service` for the gateway.
|
||||||
|
`action` ∈ `start|stop|restart`. Calls
|
||||||
the same `priv_client::control_infra_container` helper the
|
the same `priv_client::control_infra_container` helper the
|
||||||
`infra_admin` agent path uses; records an `audit_log` entry
|
`infra_admin` agent path uses; records an `audit_log` entry
|
||||||
(`start_infra`/`stop_infra`/`restart_infra`, actor `"operator"`) either
|
(`start_infra`/`stop_infra`/`restart_infra`, actor `"operator"`) either
|
||||||
|
|
|
||||||
|
|
@ -326,7 +326,8 @@ pub struct GetHostJournalArgs {
|
||||||
/// nspawn machine name verbatim (e.g. `h-iris`). Omit for host journal.
|
/// nspawn machine name verbatim (e.g. `h-iris`). Omit for host journal.
|
||||||
/// Agent containers use the `h-` prefix (e.g. `h-iris`); infrastructure
|
/// Agent containers use the `h-` prefix (e.g. `h-iris`); infrastructure
|
||||||
/// containers use their full name (e.g. `hive-ci`, `hive-forge`,
|
/// containers use their full name (e.g. `hive-ci`, `hive-forge`,
|
||||||
/// `hive-matrix`, `hive-gateway`).
|
/// `hive-matrix`). The gateway has no machine — its nginx runs on the
|
||||||
|
/// host, so read it with `unit: nginx.service` and no `container`.
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub container: Option<String>,
|
pub container: Option<String>,
|
||||||
/// Number of lines to return (default 30, max 100).
|
/// Number of lines to return (default 30, max 100).
|
||||||
|
|
|
||||||
|
|
@ -681,8 +681,9 @@ impl AgentServer {
|
||||||
Only succeeds if `name` is a direct child of this agent in the topology \
|
Only succeeds if `name` is a direct child of this agent in the topology \
|
||||||
tree — the server enforces this. No approval required. \
|
tree — the server enforces this. No approval required. \
|
||||||
Agents holding the `infra_admin` capability may also pass a hive \
|
Agents holding the `infra_admin` capability may also pass a hive \
|
||||||
infrastructure container name (`hive-ci`, `hive-gateway`, `hive-forge`) \
|
infrastructure container name (`hive-ci`, `hive-forge`, `hive-matrix`) \
|
||||||
to restart it directly via the privileged helper."
|
to restart it directly via the privileged helper. The gateway is \
|
||||||
|
not restartable by an agent — ask the operator."
|
||||||
)]
|
)]
|
||||||
async fn restart(&self, Parameters(args): Parameters<RestartArgs>) -> String {
|
async fn restart(&self, Parameters(args): Parameters<RestartArgs>) -> String {
|
||||||
let log = format!("{args:?}");
|
let log = format!("{args:?}");
|
||||||
|
|
@ -790,7 +791,8 @@ impl AgentServer {
|
||||||
`container`: nspawn machine name verbatim (e.g. `h-iris`). Omit for host journal. \
|
`container`: nspawn machine name verbatim (e.g. `h-iris`). Omit for host journal. \
|
||||||
Agent containers use the `h-` prefix (e.g. `h-iris`, `h-atlas`); \
|
Agent containers use the `h-` prefix (e.g. `h-iris`, `h-atlas`); \
|
||||||
infrastructure containers use their full name (e.g. `hive-ci`, `hive-forge`, \
|
infrastructure containers use their full name (e.g. `hive-ci`, `hive-forge`, \
|
||||||
`hive-matrix`, `hive-gateway`). \
|
`hive-matrix`). The gateway is not a container: its nginx logs are in \
|
||||||
|
the host journal, so omit `container` and pass `unit: nginx.service`. \
|
||||||
`lines`: how many lines (default 30, max 100). \
|
`lines`: how many lines (default 30, max 100). \
|
||||||
`priority`: minimum syslog level enum. \
|
`priority`: minimum syslog level enum. \
|
||||||
`grep`: regex matched against log message fields (journalctl --grep). \
|
`grep`: regex matched against log message fields (journalctl --grep). \
|
||||||
|
|
|
||||||
|
|
@ -60,8 +60,8 @@ fn allowed_capability_tools() -> Vec<String> {
|
||||||
match t.as_str() {
|
match t.as_str() {
|
||||||
"read_host_journal" => tools.push("get_host_journal".to_owned()),
|
"read_host_journal" => tools.push("get_host_journal".to_owned()),
|
||||||
// infra_admin lets an agent restart hive infrastructure
|
// infra_admin lets an agent restart hive infrastructure
|
||||||
// containers (hive-ci / hive-gateway / hive-forge) through the
|
// containers (hive-ci / hive-forge / hive-matrix — not the
|
||||||
// existing `restart` tool. Unlock it here so agents that hold
|
// gateway) through the existing `restart` tool. Unlock it here so agents that hold
|
||||||
// the capability without the full `lifecycle` group can still
|
// the capability without the full `lifecycle` group can still
|
||||||
// call it; c0re re-checks the capability server-side and only
|
// call it; c0re re-checks the capability server-side and only
|
||||||
// honours infra-container names via this path.
|
// honours infra-container names via this path.
|
||||||
|
|
|
||||||
|
|
@ -169,11 +169,11 @@ pub async fn serve(
|
||||||
/// Best-effort unlinks any stale socket left from a crashed previous
|
/// Best-effort unlinks any stale socket left from a crashed previous
|
||||||
/// harness (clean exit removes it, but `bind(2)` refuses to overwrite
|
/// harness (clean exit removes it, but `bind(2)` refuses to overwrite
|
||||||
/// an existing file) and `mkdir -p`s the parent for first-boot. Mode
|
/// an existing file) and `mkdir -p`s the parent for first-boot. Mode
|
||||||
/// `0o666` — world-accessible so the gateway container's nginx process
|
/// `0o666` — world-accessible so the gateway's nginx process can
|
||||||
/// can `connect(2)` without sharing a group with the agent user.
|
/// `connect(2)` without sharing a group with the agent user. What bounds
|
||||||
/// The per-agent subdir (`/run/hive-agent/<name>/`) is only accessible
|
/// that is the per-agent subdir (`/run/hive-agent/<name>/`): the socket
|
||||||
/// to containers that have it bind-mounted, so world-accessible sockets
|
/// mode grants everyone, so the directory's permissions decide who gets
|
||||||
/// are not a material risk.
|
/// to ask.
|
||||||
///
|
///
|
||||||
/// Marker-gating + the gateway-side consumer: see
|
/// Marker-gating + the gateway-side consumer: see
|
||||||
/// [`docs/gateway.md::Per-agent unix-socket upstream`](../../../docs/gateway.md).
|
/// [`docs/gateway.md::Per-agent unix-socket upstream`](../../../docs/gateway.md).
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,13 @@
|
||||||
//! Dashboard endpoint for operator-driven infra-container lifecycle
|
//! Dashboard endpoint for operator-driven infra lifecycle (start / stop /
|
||||||
//! (start / stop / restart on `hive-ci`, `hive-forge`, `hive-gateway`,
|
//! restart on `hive-ci`, `hive-forge`, `hive-gateway`, `hive-matrix`).
|
||||||
//! `hive-matrix`). Parallels the `infra_admin`-gated agent path in
|
//! Parallels the `infra_admin`-gated agent path in
|
||||||
//! `socket_server/lifecycle_handlers.rs::handle_restart_infra`, but this one
|
//! `socket_server/lifecycle_handlers.rs::handle_restart_infra`, but this one
|
||||||
//! is reached from the dashboard — already fully operator-authenticated —
|
//! is reached from the dashboard — already fully operator-authenticated —
|
||||||
//! so no capability check is needed here, just the same audit trail.
|
//! so no capability check is needed here, just the same audit trail.
|
||||||
|
//!
|
||||||
|
//! The two surfaces cover different sets: this endpoint takes all four,
|
||||||
|
//! while the agent path refuses the gateway — nginx on the host fronts
|
||||||
|
//! every hive service, so bouncing it is the operator's call.
|
||||||
|
|
||||||
use axum::{
|
use axum::{
|
||||||
extract::{Path as AxumPath, State},
|
extract::{Path as AxumPath, State},
|
||||||
|
|
@ -27,7 +31,7 @@ use super::{AppState, error_response};
|
||||||
post,
|
post,
|
||||||
path = "/api/infra-container/{name}/{action}",
|
path = "/api/infra-container/{name}/{action}",
|
||||||
params(
|
params(
|
||||||
("name" = String, Path, description = "infra container name (hive-ci/hive-forge/hive-gateway/hive-matrix)"),
|
("name" = String, Path, description = "infra service name (hive-ci/hive-forge/hive-gateway/hive-matrix)"),
|
||||||
("action" = String, Path, description = "start | stop | restart"),
|
("action" = String, Path, description = "start | stop | restart"),
|
||||||
),
|
),
|
||||||
responses(
|
responses(
|
||||||
|
|
@ -53,8 +57,8 @@ pub(super) async fn post_infra_container(
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
let unit = container.unit_name();
|
let target = container.name();
|
||||||
tracing::info!(%unit, %action, "dashboard: infra container action");
|
tracing::info!(%target, %action, "dashboard: infra container action");
|
||||||
let result = crate::priv_client::control_infra_container(container, infra_action).await;
|
let result = crate::priv_client::control_infra_container(container, infra_action).await;
|
||||||
let outcome = if result.is_ok() {
|
let outcome = if result.is_ok() {
|
||||||
crate::audit_log::AuditOutcome::Ok
|
crate::audit_log::AuditOutcome::Ok
|
||||||
|
|
@ -66,12 +70,12 @@ pub(super) async fn post_infra_container(
|
||||||
state
|
state
|
||||||
.coord
|
.coord
|
||||||
.audit_log
|
.audit_log
|
||||||
.record("operator", action_label, unit, outcome, detail.as_deref())
|
.record("operator", action_label, target, outcome, detail.as_deref())
|
||||||
{
|
{
|
||||||
state.coord.emit_audit_entry(entry);
|
state.coord.emit_audit_entry(entry);
|
||||||
}
|
}
|
||||||
match result {
|
match result {
|
||||||
Ok(()) => (StatusCode::OK, "ok").into_response(),
|
Ok(()) => (StatusCode::OK, "ok").into_response(),
|
||||||
Err(e) => error_response(&format!("{unit}: {e:#}")),
|
Err(e) => error_response(&format!("{target}: {e:#}")),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,15 @@
|
||||||
//! Journal-read endpoints for the dashboard.
|
//! Journal-read endpoints for the dashboard.
|
||||||
//!
|
//!
|
||||||
//! `GET /api/journal/{name}` reads a managed agent container's journal, OR
|
//! `GET /api/journal/{name}` reads a managed agent container's journal, OR
|
||||||
//! one of the four hive infra containers (`hive-ci`, `hive-forge`,
|
//! one of the four hive infra services (`hive-ci`, `hive-forge`,
|
||||||
//! `hive-gateway`, `hive-matrix` — [`hive_priv_sock::InfraContainer`] is the
|
//! `hive-gateway`, `hive-matrix` — [`hive_priv_sock::InfraContainer`] is the
|
||||||
//! allowlist), via the root helper (`journalctl -M`, delegated to hive-priv
|
//! allowlist). A container's journal is a `journalctl -M` read, delegated
|
||||||
//! since hive-c0re is unprivileged). `GET /api/journal-host` reads
|
//! to the root helper since entering a machine needs privileges hive-c0re
|
||||||
//! host-side journald, both gated by an allow-list of known units so
|
//! doesn't have; `hive-gateway` is nginx on the host, so it reads host
|
||||||
//! arbitrary unit names can't be probed. Operator-only by virtue of the
|
//! journald filtered to that unit and needs no helper at all.
|
||||||
//! dashboard binding host-only.
|
//! `GET /api/journal-host` reads host-side journald, both gated by an
|
||||||
|
//! allow-list of known units so arbitrary unit names can't be probed.
|
||||||
|
//! Operator-only by virtue of the dashboard binding host-only.
|
||||||
|
|
||||||
use axum::{
|
use axum::{
|
||||||
extract::Path as AxumPath,
|
extract::Path as AxumPath,
|
||||||
|
|
@ -40,11 +42,12 @@ pub(super) struct JournalQuery {
|
||||||
/// container namespace and needs root — is delegated to hive-priv.
|
/// container namespace and needs root — is delegated to hive-priv.
|
||||||
///
|
///
|
||||||
/// `name` is either a managed agent name (`iris`, optionally already
|
/// `name` is either a managed agent name (`iris`, optionally already
|
||||||
/// carrying the `h-` prefix) or one of the four infra container names
|
/// carrying the `h-` prefix) or one of the four infra names (`hive-ci` /
|
||||||
/// (`hive-ci` / `hive-forge` / `hive-gateway` / `hive-matrix` — see
|
/// `hive-forge` / `hive-gateway` / `hive-matrix` — see
|
||||||
/// [`hive_priv_sock::InfraContainer`]). Infra containers don't run the
|
/// [`hive_priv_sock::InfraContainer`]). Infra targets don't run the
|
||||||
/// per-agent hive daemons, so `unit` is ignored for them — always the
|
/// per-agent hive daemons, so `unit` is ignored for them — the whole
|
||||||
/// full machine journal.
|
/// machine journal, or for the gateway the host journal filtered to its
|
||||||
|
/// own unit.
|
||||||
#[utoipa::path(
|
#[utoipa::path(
|
||||||
get,
|
get,
|
||||||
path = "/api/journal/{name}",
|
path = "/api/journal/{name}",
|
||||||
|
|
@ -66,7 +69,17 @@ pub(super) async fn get_journal(
|
||||||
let lines = q.lines.unwrap_or(500).min(5000);
|
let lines = q.lines.unwrap_or(500).min(5000);
|
||||||
|
|
||||||
if let Ok(infra) = name.parse::<hive_priv_sock::InfraContainer>() {
|
if let Ok(infra) = name.parse::<hive_priv_sock::InfraContainer>() {
|
||||||
return read_journal_response(infra.unit_name(), None, lines).await;
|
return match infra.target() {
|
||||||
|
hive_priv_sock::InfraTarget::Container(machine) => {
|
||||||
|
read_journal_response(machine, None, lines).await
|
||||||
|
}
|
||||||
|
// No machine to enter — the gateway's nginx is a host unit, so
|
||||||
|
// this is a plain host-journal read filtered to it. `-M` is
|
||||||
|
// what needed root here, not journalctl itself.
|
||||||
|
hive_priv_sock::InfraTarget::HostUnit(unit) => {
|
||||||
|
read_host_journal_response(Some(unit), lines).await
|
||||||
|
}
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Defense-in-depth format check so weird chars never reach the
|
// Defense-in-depth format check so weird chars never reach the
|
||||||
|
|
@ -180,21 +193,42 @@ pub(super) async fn get_journal_host(
|
||||||
axum::extract::Query(q): axum::extract::Query<JournalHostQuery>,
|
axum::extract::Query(q): axum::extract::Query<JournalHostQuery>,
|
||||||
) -> Result<Response, ProblemDetails> {
|
) -> Result<Response, ProblemDetails> {
|
||||||
let lines = q.lines.unwrap_or(500).min(5000);
|
let lines = q.lines.unwrap_or(500).min(5000);
|
||||||
let allowed = ["hive-c0re.service", "hive-priv.service"];
|
// `nginx.service` is the gateway — its logs are host-side.
|
||||||
|
let allowed = ["hive-c0re.service", "hive-priv.service", "nginx.service"];
|
||||||
|
let unit = match q.unit.as_deref().filter(|s| !s.is_empty()) {
|
||||||
|
Some(u) => {
|
||||||
|
let unit = if u.ends_with(".service") {
|
||||||
|
u.to_owned()
|
||||||
|
} else {
|
||||||
|
format!("{u}.service")
|
||||||
|
};
|
||||||
|
if !allowed.contains(&unit.as_str()) {
|
||||||
|
return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST)
|
||||||
|
.with_detail(format!("journal-host: unknown unit {unit:?}")));
|
||||||
|
}
|
||||||
|
Some(unit)
|
||||||
|
}
|
||||||
|
None => None,
|
||||||
|
};
|
||||||
|
read_host_journal_response(unit.as_deref(), lines).await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `journalctl [-u <unit>]` on the host + response formatting. No `-M`, so
|
||||||
|
/// no root and no priv-client hop — hive-c0re reads host journald directly.
|
||||||
|
///
|
||||||
|
/// ⚠️ `unit` is trusted by the time it gets here: [`get_journal_host`]
|
||||||
|
/// allow-lists an operator-supplied one, and [`get_journal`] passes a unit
|
||||||
|
/// that came from the [`hive_priv_sock::InfraContainer`] enum. Don't hand
|
||||||
|
/// this a raw query parameter.
|
||||||
|
async fn read_host_journal_response(
|
||||||
|
unit: Option<&str>,
|
||||||
|
lines: u32,
|
||||||
|
) -> Result<Response, ProblemDetails> {
|
||||||
let mut cmd = tokio::process::Command::new("journalctl");
|
let mut cmd = tokio::process::Command::new("journalctl");
|
||||||
cmd.args(["--no-pager", "--output=short-iso", "--lines"])
|
cmd.args(["--no-pager", "--output=short-iso", "--lines"])
|
||||||
.arg(lines.to_string());
|
.arg(lines.to_string());
|
||||||
if let Some(u) = q.unit.as_deref().filter(|s| !s.is_empty()) {
|
if let Some(u) = unit {
|
||||||
let unit = if u.ends_with(".service") {
|
cmd.args(["-u", u]);
|
||||||
u.to_owned()
|
|
||||||
} else {
|
|
||||||
format!("{u}.service")
|
|
||||||
};
|
|
||||||
if !allowed.contains(&unit.as_str()) {
|
|
||||||
return Err(ProblemDetails::from_status_code(StatusCode::BAD_REQUEST)
|
|
||||||
.with_detail(format!("journal-host: unknown unit {unit:?}")));
|
|
||||||
}
|
|
||||||
cmd.args(["-u", &unit]);
|
|
||||||
}
|
}
|
||||||
match cmd.output().await {
|
match cmd.output().await {
|
||||||
Ok(out) => {
|
Ok(out) => {
|
||||||
|
|
|
||||||
|
|
@ -154,7 +154,7 @@ async fn infra_container_views() -> Vec<InfraContainerView> {
|
||||||
let mut infra_containers = Vec::with_capacity(hive_priv_sock::InfraContainer::ALL.len());
|
let mut infra_containers = Vec::with_capacity(hive_priv_sock::InfraContainer::ALL.len());
|
||||||
for container in hive_priv_sock::InfraContainer::ALL {
|
for container in hive_priv_sock::InfraContainer::ALL {
|
||||||
infra_containers.push(InfraContainerView {
|
infra_containers.push(InfraContainerView {
|
||||||
name: container.unit_name(),
|
name: container.name(),
|
||||||
running: crate::lifecycle::infra_is_running(container).await,
|
running: crate::lifecycle::infra_is_running(container).await,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,7 @@
|
||||||
//! Runtime nginx include-file generator for the gateway's per-agent
|
//! Runtime nginx include-file generator for the gateway's per-agent
|
||||||
//! `/agent/<name>/` location blocks. Writes
|
//! `/agent/<name>/` location blocks. Writes
|
||||||
//! `/var/lib/hyperhive/gateway/agents.conf` on every topology change.
|
//! `/var/lib/hyperhive/gateway/agents.conf` on every topology change.
|
||||||
//! UDS upstream selection, reload trigger (`systemd-run
|
//! UDS upstream selection, the reload trigger, and idempotency:
|
||||||
//! --machine=hive-gateway`), and idempotency:
|
|
||||||
//! `docs/gateway.md::Per-agent unix-socket upstream`.
|
//! `docs/gateway.md::Per-agent unix-socket upstream`.
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
|
|
@ -19,13 +18,13 @@ use crate::agent_sockets;
|
||||||
/// Set when `write` publishes a new agents.conf; cleared when
|
/// Set when `write` publishes a new agents.conf; cleared when
|
||||||
/// `reload_gateway_nginx` submits the reload command successfully.
|
/// `reload_gateway_nginx` submits the reload command successfully.
|
||||||
/// Lets `spawn_poll` retry the reload on subsequent ticks when the
|
/// Lets `spawn_poll` retry the reload on subsequent ticks when the
|
||||||
/// previous attempt failed (e.g. gateway container temporarily down,
|
/// previous attempt failed (e.g. gateway nginx temporarily down, priv
|
||||||
/// systemd-run not found) without re-writing the already-correct file.
|
/// helper unreachable) without re-writing the already-correct file.
|
||||||
static RELOAD_PENDING: AtomicBool = AtomicBool::new(false);
|
static RELOAD_PENDING: AtomicBool = AtomicBool::new(false);
|
||||||
|
|
||||||
/// Unix timestamp (seconds) of the last failed reload attempt.
|
/// Unix timestamp (seconds) of the last failed reload attempt.
|
||||||
/// `reload_if_pending` backs off to once per `RELOAD_RETRY_SECS` after a
|
/// `reload_if_pending` backs off to once per `RELOAD_RETRY_SECS` after a
|
||||||
/// failure so a permanently broken gateway (bad config, container down)
|
/// failure so a permanently broken gateway (bad config, nginx down)
|
||||||
/// doesn't hammer `systemctl` on every 10-second `spawn_poll` tick.
|
/// doesn't hammer `systemctl` on every 10-second `spawn_poll` tick.
|
||||||
static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
|
static LAST_FAILED_RELOAD: AtomicU64 = AtomicU64::new(0);
|
||||||
|
|
||||||
|
|
@ -79,7 +78,7 @@ fn render(names: &[String], frontend_dir: Option<&str>) -> String {
|
||||||
let mut out = String::from(
|
let mut out = String::from(
|
||||||
"# Generated by hive-c0re \u{2014} do not edit.\
|
"# Generated by hive-c0re \u{2014} do not edit.\
|
||||||
\n# Refreshed on every topology change + when agents bind/drop their unix sockets.\
|
\n# Refreshed on every topology change + when agents bind/drop their unix sockets.\
|
||||||
\n# Reload triggered by hive-c0re via systemd-run --machine=hive-gateway.\n",
|
\n# Reload triggered by hive-c0re via hive-priv (systemctl reload nginx).\n",
|
||||||
);
|
);
|
||||||
for name in names {
|
for name in names {
|
||||||
// Two upstream forms because named locations (split mode's
|
// Two upstream forms because named locations (split mode's
|
||||||
|
|
@ -165,15 +164,15 @@ fn render(names: &[String], frontend_dir: Option<&str>) -> String {
|
||||||
/// body matches what's already on disk (idempotent; avoids spurious
|
/// body matches what's already on disk (idempotent; avoids spurious
|
||||||
/// gateway reloads on a quiet tick).
|
/// gateway reloads on a quiet tick).
|
||||||
///
|
///
|
||||||
/// After a successful write, triggers the appropriate nginx action inside
|
/// After a successful write, triggers the appropriate nginx action via
|
||||||
/// the gateway container via `hive-priv` (which has the
|
/// `hive-priv` (hive-c0re runs unprivileged and cannot act on a system
|
||||||
/// `--machine=hive-gateway` transport rights hive-c0re lacks):
|
/// unit): reload when nginx is active, reset-failed+start when in a
|
||||||
/// reload when nginx is active, reset-failed+start when in a failed
|
/// failed state, plain start otherwise. Writer and nginx are on the same
|
||||||
/// state, plain start otherwise. This is intentionally host-side rather
|
/// machine, so it is a plain unit action with no machine-bus hop.
|
||||||
/// than relying on a systemd path unit inside the container watching the
|
/// It stays an explicit trigger rather than a systemd path
|
||||||
/// bind-mounted file: `IN_MOVED_TO` (fired by the atomic rename) does
|
/// unit watching the file: the write already knows it changed something,
|
||||||
/// not reliably propagate across the nspawn mount-namespace boundary, so
|
/// and a watcher would turn one causal edge into a race with the writer's
|
||||||
/// the path-unit approach was silently broken (see `docs/gateway.md`).
|
/// own rename (see `docs/gateway.md`).
|
||||||
///
|
///
|
||||||
/// The priv call is best-effort — a failed sync is logged but not fatal.
|
/// The priv call is best-effort — a failed sync is logged but not fatal.
|
||||||
/// `reload_if_pending` retries on the next `spawn_poll` tick so a
|
/// `reload_if_pending` retries on the next `spawn_poll` tick so a
|
||||||
|
|
@ -212,7 +211,7 @@ pub async fn write(names: &[String]) -> Result<()> {
|
||||||
|
|
||||||
/// Retry a pending nginx reload if a previous attempt failed.
|
/// Retry a pending nginx reload if a previous attempt failed.
|
||||||
/// Called by `spawn_poll` on each tick so a transient failure
|
/// Called by `spawn_poll` on each tick so a transient failure
|
||||||
/// (gateway container temporarily down, systemd-run error) is
|
/// (gateway nginx temporarily down, priv-helper error) is
|
||||||
/// recovered automatically without requiring a new file write.
|
/// recovered automatically without requiring a new file write.
|
||||||
///
|
///
|
||||||
/// Backs off to one retry per `RELOAD_RETRY_SECS` after a failure so a
|
/// Backs off to one retry per `RELOAD_RETRY_SECS` after a failure so a
|
||||||
|
|
@ -240,8 +239,8 @@ pub async fn reload_if_pending() {
|
||||||
/// Synchronise the gateway nginx unit with the current agents.conf via
|
/// Synchronise the gateway nginx unit with the current agents.conf via
|
||||||
/// `hive-priv` (privileged helper). The state-aware logic (active →
|
/// `hive-priv` (privileged helper). The state-aware logic (active →
|
||||||
/// reload; failed → reset-failed + start; inactive/unknown → start)
|
/// reload; failed → reset-failed + start; inactive/unknown → start)
|
||||||
/// runs inside hive-priv where it has the `--machine=hive-gateway`
|
/// runs inside hive-priv, which is root; hive-c0re runs as the
|
||||||
/// transport rights that hive-c0re (unprivileged) lacks.
|
/// unprivileged `hive-core` user and cannot act on a system unit.
|
||||||
///
|
///
|
||||||
/// `RELOAD_PENDING` is cleared only after a successful operation so
|
/// `RELOAD_PENDING` is cleared only after a successful operation so
|
||||||
/// `reload_if_pending` keeps retrying on failure.
|
/// `reload_if_pending` keeps retrying on failure.
|
||||||
|
|
|
||||||
|
|
@ -604,14 +604,14 @@ pub async fn is_running(name: &str) -> bool {
|
||||||
.is_ok_and(|s| s.success())
|
.is_ok_and(|s| s.success())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// True when a hive infrastructure container's systemd unit is active.
|
/// True when a hive infrastructure service's systemd unit is active.
|
||||||
/// Sibling of [`is_running`] for sub-agents, but infra container/unit names
|
/// Sibling of [`is_running`] for sub-agents, but infra names (`hive-ci`, …)
|
||||||
/// (`hive-ci`, …) already have no `h-` prefix to strip, so this queries
|
/// have no `h-` prefix to strip and are not all containers, so the unit
|
||||||
/// `container@<unit_name>.service` directly rather than going through
|
/// comes from the variant itself rather than from [`container_name`]. Used
|
||||||
/// [`container_name`]. Used by the dashboard C0R3 page's 1NFR4 sub-tab to
|
/// by the dashboard C0R3 page's 1NFR4 sub-tab to show each one's live
|
||||||
/// show each infra container's live status dot.
|
/// status dot.
|
||||||
pub async fn infra_is_running(container: hive_priv_sock::InfraContainer) -> bool {
|
pub async fn infra_is_running(container: hive_priv_sock::InfraContainer) -> bool {
|
||||||
let unit = format!("container@{}.service", container.unit_name());
|
let unit = container.service_unit();
|
||||||
Command::new("systemctl")
|
Command::new("systemctl")
|
||||||
.args(["is-active", "--quiet", &unit])
|
.args(["is-active", "--quiet", &unit])
|
||||||
.status()
|
.status()
|
||||||
|
|
|
||||||
|
|
@ -168,9 +168,9 @@ pub async fn sync_agents(hive: &HiveEnv, agents: &[AgentSpec]) -> Result<()> {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Refresh /var/lib/hyperhive/gateway/agents.conf — the nginx include
|
// Refresh /var/lib/hyperhive/gateway/agents.conf — the nginx include
|
||||||
// file the gateway container bind-mounts and nginx reads at runtime.
|
// file the gateway reads at runtime. c0re then triggers a reload (or
|
||||||
// c0re triggers a reload (or start) inside hive-gateway via hive-priv
|
// start) of the host's nginx via hive-priv, since c0re is unprivileged.
|
||||||
// after writing the file. Same best-effort + non-fatal shape.
|
// Same best-effort + non-fatal shape.
|
||||||
if let Err(e) = crate::gateway_nginx::write(&agent_names).await {
|
if let Err(e) = crate::gateway_nginx::write(&agent_names).await {
|
||||||
tracing::warn!(error = ?e, "gateway_nginx::write failed (non-fatal)");
|
tracing::warn!(error = ?e, "gateway_nginx::write failed (non-fatal)");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -219,11 +219,12 @@ pub fn shared_root() -> PathBuf {
|
||||||
pub const KNOWLEDGE_DIR: &str = "/var/lib/hyperhive/knowledge";
|
pub const KNOWLEDGE_DIR: &str = "/var/lib/hyperhive/knowledge";
|
||||||
|
|
||||||
/// `gateway/` — generated nginx include fragments for the gateway vhost.
|
/// `gateway/` — generated nginx include fragments for the gateway vhost.
|
||||||
/// The gateway container bind-mounts *this subdir only* (not the whole
|
/// nginx runs on the host and reads this path directly. ⚠️ Nothing about
|
||||||
/// state root) at `/run/hive-state/`, so nginx can read `agents.conf`
|
/// this path confines it: what keeps nginx away from the rest of
|
||||||
/// without the rest of `/var/lib/hyperhive/` (forge/matrix tokens, etc.)
|
/// `/var/lib/hyperhive/` (forge/matrix tokens, etc.) is the unit's own
|
||||||
/// being exposed to the gateway container.
|
/// sandbox, so widening that sandbox widens what a gateway compromise
|
||||||
// nix: bind-mounted into the gateway container (hive-gateway.nix) — must match.
|
/// reaches.
|
||||||
|
// nix: named by the gateway's nginx config (hive-gateway/vhosts.nix) — must match.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn gateway_dir() -> PathBuf {
|
pub fn gateway_dir() -> PathBuf {
|
||||||
state_root().join("gateway")
|
state_root().join("gateway")
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue