diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml index 9adac1ec..90ede59b 100644 --- a/.forgejo/workflows/ci.yml +++ b/.forgejo/workflows/ci.yml @@ -53,3 +53,17 @@ jobs: # protection) — a hit blocks merge. See # scripts/check-comment-blocks.sh. run: sh scripts/check-comment-blocks.sh + + doc-refs: + name: doc-pointer lint + runs-on: [hive-ci] + # Pure git+grep — seconds. + timeout-minutes: 5 + steps: + - uses: actions/checkout@v3 + - name: lint + # Flags `docs/…md` paths and relative markdown links that no + # longer resolve. Pointing at a doc is what the comment-block + # lint above pushes people toward, so the pointers need a gate of + # their own. See scripts/check-doc-refs.sh. + run: sh scripts/check-doc-refs.sh diff --git a/.prettierignore b/.prettierignore index 8c27f02e..35f58a1f 100644 --- a/.prettierignore +++ b/.prettierignore @@ -8,12 +8,12 @@ docs/tools/hivectl-cli.md # indentation normalisation, so these files need manual re-wrap before # adding them back to the formatter scope. CLAUDE.md -docs/approvals.md -docs/conventions.md -docs/gateway.md -docs/matrix.md -docs/persistence.md -docs/terminal-rendering.md +docs/agent-lifecycle/approvals.md +docs/process/conventions.md +docs/networking/gateway.md +docs/integrations/matrix.md +docs/agent-lifecycle/persistence.md +docs/web-ui/terminal-rendering.md docs/tools/forge.md docs/tools/matrix.md docs/turn-loop/README.md diff --git a/docs/agent-lifecycle/approvals.md b/docs/agent-lifecycle/approvals.md index 5ed06c2c..cb44e3ce 100644 --- a/docs/agent-lifecycle/approvals.md +++ b/docs/agent-lifecycle/approvals.md @@ -209,7 +209,7 @@ One-shot rows fire once (if past due, on the next worker pass) and are deleted b ### Scheduled prompt delivery: todo, not a broker message An agent target's delivery is `push_todo` (`Coordinator::push_todo`, -`docs/coordinator.md` covers the mechanism generally), not a broker +`docs/scheduler/coordinator.md` covers the mechanism generally), not a broker `Message` — a scheduled prompt wakes its target with a todo instead of driving an immediate turn, by design. `key = "schedule:"` per target drives `push_todo`'s own upsert-by-key dedup: a re-fire of the @@ -419,7 +419,7 @@ submitter pushes again (or closes it) to retry. Long-running approval work — `MergeConfigPr`, `UpdateMetaInputs`, `Spawn` — runs as a DAG on the global job queue -(`docs/coordinator.md::Job queue`), submitted by the approval handler +(`docs/scheduler/coordinator.md::Job queue`), submitted by the approval handler rather than run inline: | `ApprovalKind` | DAG submitted | source | diff --git a/docs/agent-lifecycle/persistence.md b/docs/agent-lifecycle/persistence.md index 04bf7672..5bb14834 100644 --- a/docs/agent-lifecycle/persistence.md +++ b/docs/agent-lifecycle/persistence.md @@ -466,7 +466,7 @@ subvolume creation/deletion above. This is the same subvolume `hivectl agent subvol snapshot push` sends to the swarm's snapshot store — see -[`docs/snapshot-store.md`](../networking/snapshot-store.md) for what a pushed +[`docs/networking/snapshot-store.md`](../networking/snapshot-store.md) for what a pushed snapshot contains and how the store authenticates a sender. ## `/var/lib/swarm-controller/` (swarm-controller host only) diff --git a/docs/networking/network.md b/docs/networking/network.md index c6b34978..653e264b 100644 --- a/docs/networking/network.md +++ b/docs/networking/network.md @@ -184,7 +184,7 @@ host-side service on another port — e.g. a dev OTLP collector you want agents to reach directly — is unreachable. (hyperhive's own telemetry needs none of this: `otel.enable` opens its collector's port itself, and `otel.endpoint` is the *upstream*, which no agent ever dials. See -`docs/observability.md`.) +`docs/scheduler/observability.md`.) `services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an @@ -274,4 +274,4 @@ loopback — the hive-c0re admin socket is a UDS, not TCP. ## Cross-references -- `docs/gateway.md` — vhost map + the gateway's other duties +- `docs/networking/gateway.md` — vhost map + the gateway's other duties diff --git a/docs/networking/snapshot-store.md b/docs/networking/snapshot-store.md index d3c3e2a6..f15d6e55 100644 --- a/docs/networking/snapshot-store.md +++ b/docs/networking/snapshot-store.md @@ -184,7 +184,7 @@ the access-review list. The snapshot covers an agent's **state subvolume**, which is the parent of `state/`, `claude/` and `harness/` (see -[`docs/persistence.md`'s btrfs subvolume +[`docs/agent-lifecycle/persistence.md`'s btrfs subvolume section](../agent-lifecycle/persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname) for how and when that subvolume is created). Consequences: diff --git a/docs/process/conventions.md b/docs/process/conventions.md index 3282cffd..2a456c53 100644 --- a/docs/process/conventions.md +++ b/docs/process/conventions.md @@ -195,7 +195,7 @@ Per-variant fields: — `agent` is the affected agent (target of the spawn / config commit), not the asker. `description` is the manager's free-text blurb shown on the dashboard card. `commit_ref` is the - kind-specific payload (see `docs/approvals.md::Approval kinds + kind-specific payload (see `docs/agent-lifecycle/approvals.md::Approval kinds (wire shapes)`). - `Reminder { id, owner, message, due_at, age_seconds }` — `due_at` is the absolute time the scheduler is targeting (RFC @@ -424,7 +424,7 @@ start runs across the `StopForUpdate → Swap → RebuildBookkeeping` brace and the tail `Reconcile` node. `flake.nix` itself is no longer regenerated host-side on rebuild — it's tracked in the agent's proposed/applied repos and rides along on every fetch (see -`docs/approvals.md::Two repos per agent`). +`docs/agent-lifecycle/approvals.md::Two repos per agent`). Anything that changes per-container state on the host should be re-applied here so a manual `↻ R3BU1LD` from the dashboard is @@ -528,7 +528,7 @@ the state dir at provisioning time: (The matrix profile avatar is **not** a oneshot — `hive-matrix-daemon` sets it over its live authenticated Client; see -`docs/persistence.md::matrix avatar`.) +`docs/agent-lifecycle/persistence.md::matrix avatar`.) Shape contract — every one of these: @@ -549,7 +549,7 @@ Shape contract — every one of these: 5. **Re-runnable**: a second invocation produces the same final state (idempotent uploads, idempotent config rewrites). Used by the `.path` watchers that re-fire on token appearance (see - `docs/persistence.md::Matrix per-agent daemon`). + `docs/agent-lifecycle/persistence.md::Matrix per-agent daemon`). The artefact lives under the agent user's home where applicable (`~/.config/tea/config.yml`) and is chown'd to that user, but the diff --git a/docs/process/gotchas.md b/docs/process/gotchas.md index 50638da7..0182a9f3 100644 --- a/docs/process/gotchas.md +++ b/docs/process/gotchas.md @@ -289,7 +289,7 @@ and fail outright if the host daemon's `nix.settings.sandbox-fallback` is `false` (nixpkgs default). `nix/agent-modules/default.nix` does `lib.mkForce true` so builds fall back to unsandboxed local builds rather than failing. Security -implications: `docs/security.md`. +implications: `docs/trust-boundary/security.md`. ### Linking workspace binaries locally needs `nix develop` @@ -377,7 +377,7 @@ connects to the compositor at `127.0.0.1:`. - **Port allocation**: a **fixed** port (`hyperhive.gui.vncPort`, default 5900). No per-agent hashing: network isolation is unconditional (each agent has its own netns — see - `docs/network.md#container-isolation`), so the VNC port is + `docs/networking/network.md#container-isolation`), so the VNC port is container-local and can't collide across agents. The harness learns the port from the `HIVE_GUI_VNC_PORT` env var (set on the harness service when `gui.enable`) — no marker file, no runtime hash. (Unlike diff --git a/docs/scheduler/ci.md b/docs/scheduler/ci.md index 7c85b73c..86ecacd3 100644 --- a/docs/scheduler/ci.md +++ b/docs/scheduler/ci.md @@ -102,7 +102,7 @@ build can't hold the runner's single slot indefinitely). - **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://` (resolved to the bridge IP via `networking.extraHosts`). It cannot reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `forge.behindGateway = true`. - **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter). -- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`. +- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/process/gotchas.md`. - **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only. ## Auto-registration flow @@ -235,4 +235,4 @@ GC on the coordinator host won't reclaim space on the builder. - `nix/host-modules/hive-ci.nix`: runner configuration, auto-registration script, container setup. - `.forgejo/workflows/ci.yml`: workflow definition. -- `docs/gotchas.md`: nix sandboxing limitations in containers. +- `docs/process/gotchas.md`: nix sandboxing limitations in containers. diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md index 8d753aba..fe6dbdda 100644 --- a/docs/scheduler/coordinator.md +++ b/docs/scheduler/coordinator.md @@ -1,8 +1,8 @@ # hive-c0re coordinator internals Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems. -For the public API surface (dashboard, socket protocol) see `docs/conventions.md` -and `docs/persistence.md`. +For the public API surface (dashboard, socket protocol) see `docs/process/conventions.md` +and `docs/agent-lifecycle/persistence.md`. --- @@ -571,7 +571,7 @@ which is almost never desired. Leave off on non-x86 hosts. ## See also -- `docs/approvals.md` — approval flow + scheduled prompts -- `docs/persistence.md` — SQLite schema, state-dir layout -- `docs/conventions.md` — wire protocol, recipient sentinels -- `docs/agent-hierarchy.md` — topology and parent/child relations +- `docs/agent-lifecycle/approvals.md` — approval flow + scheduled prompts +- `docs/agent-lifecycle/persistence.md` — SQLite schema, state-dir layout +- `docs/process/conventions.md` — wire protocol, recipient sentinels +- `docs/agent-lifecycle/agent-hierarchy.md` — topology and parent/child relations diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index d2e60d23..ff231f10 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -58,7 +58,7 @@ That is a **different risk from the one the collector fixes**, and strictly smaller than what preceded it: before, every agent held the upstream credential itself, so it could do all of the above *and* use the token anywhere else. The collector removes the token and keeps the pipe. Agents are inside the trust -boundary (`docs/security.md`: capability = accepted risk), so an agent being +boundary (`docs/trust-boundary/security.md`: capability = accepted risk), so an agent being able to *send* is an accepted extension of that boundary — but it is not closed by this design, and nothing here should be read as closing it. @@ -179,7 +179,7 @@ and point whatever consumes it at `10.42.0.1:5432` rather than loopback: inside a container, loopback is the *container*. The bridge IP is the host's address on the `hive-br0` bridge. The service must also bind an address the bridge can reach — a `127.0.0.1`-only listener stays unreachable no matter what the -firewall allows. See `docs/network.md::Reaching host services` for details. +firewall allows. See `docs/networking/network.md::Reaching host services` for details. ⚠️ **None of this is needed for hyperhive's own telemetry** — `otel.enable` contributes the collector's port and derives the agent-facing endpoint itself. diff --git a/docs/trust-boundary/boundary.md b/docs/trust-boundary/boundary.md index f4ec9fce..3832c001 100644 --- a/docs/trust-boundary/boundary.md +++ b/docs/trust-boundary/boundary.md @@ -19,7 +19,7 @@ always on), the gateway proxies all operator-facing traffic, and reachable via the gateway. - **Agent** — speaks only for itself, only over its per-agent unix socket. The socket's identity _is_ the agent (see - `docs/conventions.md`, "identity = socket"). An agent must not + `docs/process/conventions.md`, "identity = socket"). An agent must not be able to reach the core daemon's HTTP surface, another agent's socket, or another agent's web UI. @@ -46,7 +46,7 @@ defence-in-depth layered on top. Network isolation is complete and always on: every agent container runs in a private netns behind the hive bridge, and there is no -shared-netns mode. See `docs/network.md`. +shared-netns mode. See `docs/networking/network.md`. Concretely, the core daemon's dashboard `/api` carries **no application-layer authentication** — operator-authority routes are served @@ -125,7 +125,7 @@ nginx reaches all of `/run/hive-agent` as a plain host path. Dropping `o=w` removes that permission rather than qualifying it. ⚠️ **The gateway's nginx and dnsmasq are host services, next to -`hive-c0re`** (see `docs/gateway.md`) — there is no namespace between +`hive-c0re`** (see `docs/networking/gateway.md`) — there is no namespace between them and the rest of the host. That costs no network isolation: nginx binds the host's `:80`/`:443` and reaches `localhost` upstreams, which a netns would have to be opened up for anyway. diff --git a/frontend/packages/dashboard/src/core.html b/frontend/packages/dashboard/src/core.html index f416a8b8..0c4342bc 100644 --- a/frontend/packages/dashboard/src/core.html +++ b/frontend/packages/dashboard/src/core.html @@ -51,7 +51,7 @@ cgroup v2 on the host. Polled only while this sub-tab is active. -->
-

live cpu + memory per agent container, from cgroup v2 on the host. cpu is % of total host capacity (all cores), sampled over ~200ms each refresh; polled every 5s while this tab is open. network is omitted on purpose — agents share the host netns, so there is no per-container counter (see docs/network.md).

+

live cpu + memory per agent container, from cgroup v2 on the host. cpu is % of total host capacity (all cores), sampled over ~200ms each refresh; polled every 5s while this tab is open. network is omitted on purpose — agents share the host netns, so there is no per-container counter (see docs/networking/network.md).

loading…

diff --git a/frontend/packages/dashboard/src/swarm.js b/frontend/packages/dashboard/src/swarm.js index eafefb05..000f69c8 100644 --- a/frontend/packages/dashboard/src/swarm.js +++ b/frontend/packages/dashboard/src/swarm.js @@ -619,7 +619,7 @@ export function renderContainers(s) { // TCP fallback — the gateway proxies the prefix to the per-agent // harness (unix-domain via `agent-sockets.json`, or a computed TCP // loopback port while the socket marker is absent). See - // `docs/web-ui/dashboard.md::Container row` + `docs/gateway.md::Vhost map`. + // `docs/web-ui/dashboard.md::Container row` + `docs/networking/gateway.md::Vhost map`. const gatewayLinks = !!(s && s.gateway_enabled); // Forge public URL: state.forge_public_url (set by the NixOS module // from services.hyperhive.forge.publicUrl) or null — never guessed diff --git a/frontend/packages/shared/src/terminal/terminal.css b/frontend/packages/shared/src/terminal/terminal.css index c35f5e12..7f552e19 100644 --- a/frontend/packages/shared/src/terminal/terminal.css +++ b/frontend/packages/shared/src/terminal/terminal.css @@ -14,7 +14,7 @@ No `.term-input` here — composers are a separate concern, owned by each page's own CSS. Row taxonomy + layout contract documented in `docs/web-ui/shape.md::Shared terminal pane` and - `docs/terminal-rendering.md`. */ + `docs/web-ui/terminal-rendering.md`. */ .terminal-wrap { position: relative; @@ -106,7 +106,7 @@ is now four: the four severities every row already carries. Structural identity (this is a turn boundary, this is a tool call) is carried by the row's icon (`◆`, `🔧`, `✅`, …) and summary text instead of colour — - see docs/terminal-rendering.md. */ + see docs/web-ui/terminal-rendering.md. */ .live .level-debug { color: var(--muted); } .live .level-info { color: var(--fg); } .live .level-warn { color: var(--amber); border-left-color: var(--amber); } diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index d1e16055..992811b7 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -1,7 +1,7 @@ # The swarm's telemetry collector: one per swarm, in a `swarm-otel` # nixos-container beside the swarm's other shared services. This is the # **swarm** tier; `otel.nix` is the hive tier that forwards into it. -# `docs/observability.md` owns the tier model and why the two stay separate +# `docs/scheduler/observability.md` owns the tier model and why the two stay separate # when co-located. # # A container rather than a second host unit, for the same reason every diff --git a/nix/packages/reference-docs.nix b/nix/packages/reference-docs.nix index d44169ed..b40aabb2 100644 --- a/nix/packages/reference-docs.nix +++ b/nix/packages/reference-docs.nix @@ -17,6 +17,12 @@ # `docs.nix` (website repo) needs no changes to pick either up — it # already walks every subdirectory generically. # +# For crates that is `$out/crates/.md`, one per workspace crate +# with a README.md (mara's ask: "behaves as if hive-core/README.md lives +# at docs/crates/hive-core.md" — lint:allow, a path this derivation +# creates in $out, so the repo has no such file). The crate's own README +# stays the single source of truth. +# # Output layout: # $out/ — the repo docs/ tree verbatim # $out/crates/.md — one per workspace crate README @@ -74,7 +80,7 @@ stdenv.mkDerivation { # Rewrite relative links that were correct from the crate's own # position in the repo tree but aren't once the file is # virtually one level under docs/ instead: - # ../docs/x.md (crate root -> repo docs/) -> ../x.md + # ../docs/x.md (crate root -> repo docs/) -> ../x.md lint:allow (stand-in name) # (crates/ -> docs/ is one directory shallower than # / -> docs/ was, so the leading docs/ segment drops) # ../ (that crate's own directory, no file) diff --git a/scripts/check-doc-refs.sh b/scripts/check-doc-refs.sh new file mode 100755 index 00000000..b4ead89e --- /dev/null +++ b/scripts/check-doc-refs.sh @@ -0,0 +1,98 @@ +#!/bin/sh +# CI lint: flags documentation pointers that no longer resolve. +# +# A pointer at a doc is the hive's preferred alternative to duplicating that +# doc's prose next to the code. That trade only pays while the pointer +# resolves, and nothing was checking. A `docs/` reorganisation left 15 distinct +# dead paths across nix, css, html, js, markdown and .prettierignore. +# +# Emits a CI error annotation per hit and exits 1 if any pointer is dead, 0 +# otherwise. Runs as its own job; seconds, pure git+grep. +# +# Two arms, because a pointer is written two ways: +# root — a repo-root-relative path in prose or a comment, `docs//.md` +# rel — a markdown link resolved against the linking file, [x](..//.md) +# (spelled with placeholders on purpose: a literal example path here would be +# a dead pointer this script then reports against itself) +# +# Scope is every tracked file, deliberately unrestricted by type: prose +# describes a path in words, and a stale path in a config file (an ignore +# rule, a build input) is a live defect rather than a stale comment. +# +# Escape hatch: a line containing `lint:allow` is exempt. Reserve it for a +# path that is correct while absent — a file another derivation synthesizes, +# or a literal example — and keep the reason next to the marker. +set -eu + +root_pattern='docs/[A-Za-z0-9._/-]*\.md' +rel_pattern='\]\(\.\.*/[A-Za-z0-9._/-]*\.md' + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +# Lines carrying the marker, as `file:line`, to skip below. +git ls-files -z | xargs -0 -r grep -n 'lint:allow' /dev/null 2>/dev/null \ + | cut -d: -f1,2 > "$tmp/allow" || true + +exempt() { grep -qxF "$1:$2" "$tmp/allow" 2>/dev/null; } + +git ls-files -z | xargs -0 -r grep -nEo "$root_pattern" /dev/null 2>/dev/null \ + > "$tmp/root" || true +git ls-files -z | xargs -0 -r grep -nEo "$rel_pattern" /dev/null 2>/dev/null \ + > "$tmp/rel" || true + +dead=0 +alive=0 + +while IFS=: read -r file lineno ref; do + [ -n "${ref:-}" ] || continue + exempt "$file" "$lineno" && continue + if [ -f "$ref" ]; then + alive=$((alive + 1)) + else + dead=$((dead + 1)) + printf '::error file=%s,line=%s::doc pointer does not resolve: %s\n' \ + "$file" "$lineno" "$ref" + fi +done < "$tmp/root" + +while IFS=: read -r file lineno match; do + [ -n "${match:-}" ] || continue + exempt "$file" "$lineno" && continue + link="${match#](}" + # Resolved from the linking file's directory, so `../` counts the same way + # the renderer counts it. `-m` keeps a link that escapes the repo root + # resolvable enough to report rather than erroring out here. + target="$(cd "$(dirname "$file")" && realpath -m "$link")" + if [ -f "$target" ]; then + alive=$((alive + 1)) + else + dead=$((dead + 1)) + printf '::error file=%s,line=%s::relative doc link does not resolve: %s\n' \ + "$file" "$lineno" "$link" + fi +done < "$tmp/rel" + +# A lint whose extractor matches nothing passes forever and reports success +# while checking not one thing. Both arms therefore have to have found +# something that resolves before a clean run means anything: if a rename, a +# pattern edit or a tree move silences the search, that is a failure of this +# script, not a property of the tree. +if [ "$(wc -l < "$tmp/root")" -eq 0 ]; then + echo 'check-doc-refs: root-pointer search matched nothing — pattern or tree changed' >&2 + exit 1 +fi +if [ "$(wc -l < "$tmp/rel")" -eq 0 ]; then + echo 'check-doc-refs: relative-link search matched nothing — pattern or tree changed' >&2 + exit 1 +fi +if [ "$alive" -eq 0 ]; then + echo 'check-doc-refs: not one pointer resolved — the resolver is broken, not the tree' >&2 + exit 1 +fi + +if [ "$dead" -gt 0 ]; then + printf 'check-doc-refs: %s dead doc pointer(s); %s resolved\n' "$dead" "$alive" >&2 + exit 1 +fi +exit 0 diff --git a/scripts/pre-push b/scripts/pre-push index 3b6ea478..62634252 100755 --- a/scripts/pre-push +++ b/scripts/pre-push @@ -1,6 +1,6 @@ #!/bin/sh -# Git pre-push hook: runs the tracker-tag lint and the comment-block -# lint against the working tree before any push lands on the remote. +# Git pre-push hook: runs the tracker-tag, comment-block and doc-pointer +# lints against the working tree before any push lands on the remote. # Catches issues that would fail CI and require a follow-up commit # (common failure mode: a nix comment containing a hash-issue-number # tag added mid-session). @@ -28,5 +28,11 @@ if ! sh "$repo_root/scripts/check-comment-blocks.sh"; then exit 1 fi +echo "pre-push: running doc-pointer lint..." >&2 +if ! sh "$repo_root/scripts/check-doc-refs.sh"; then + echo "pre-push: doc-pointer lint FAILED — fix before pushing" >&2 + exit 1 +fi + echo "pre-push: lints passed ✓" >&2 exit 0