- nix/checks.nix: 'hyperhive#3685' in a source comment trips the
tracker-tag lint (prose-only, per hive-rules.md) — argus caught it on
review. Reworded without the ref.
- frontend/npm-deps-hash: the earlier scaffold commit
(be18f460) added preact/typescript to agent/package.json, which
regenerated package-lock.json but never re-derived this hash in
lockstep — nix flake check's frontend derivation was failing on a
hash mismatch. Regenerated via
'nix run nixpkgs#prefetch-npm-deps -- frontend/package-lock.json'.
- StatusChips.tsx: drop the stale 'useAgentState lands in a later
commit' comment now that Root.tsx already wires it.
- nix/checks.nix: add agent-typecheck, same shape as swarm-ui-typecheck,
so a real TS error in the agent package fails CI instead of just
esbuild-stripping silently.
Ports the row taxonomy in docs/terminal-rendering.md (app.js's
renderStream/renderRichToolUse/renderToolResult/renderTaskEvent) to
real Preact components + hooks:
- lib/streamRow.ts: plain-data StreamRow model (one row = one line/
panel), lib/classifyEvent.ts: raw stream-json event -> StreamRow[],
almost entirely dispatching on the backend's precomputed _icon/
_summary/_category/_body/_body_type fields, same as the old client.
- lib/markdown.ts, lib/linkify.tsx: sanitized-markdown + auto-link
helpers, same behavior as app.js's mdNode/terminal.js's linkify.
- hooks/useLiveStream.ts: backfill + SSE + seq-dedupe + coalescing,
reduced to a plain StreamRow[] — deliberately has no opinion on
scroll position, only on what rows exist.
- components/Row.tsx: renders one StreamRow (flat or details).
- components/LiveStream.tsx: owns the scrollable DOM node + a from-
scratch sticky-bottom implementation — not the old MutationObserver
+ rAF snap-animation + smoothScrollingUntil gate. stickToBottom is
plain state driven by the scroll handler; snapping is an instant
scrollTop write in a useLayoutEffect that runs after Preact has
already committed the new rows, so it always sees the real
scrollHeight. A load-older prepend is told apart from a normal
append via an explicit ref set right before calling loadMore(),
rather than inferred from DOM mutation shape after the fact.
Wired into Root.tsx below Header/StatusChips; turn_start/turn_end
also nudge useAgentState's refresh() for a snappier badge update than
the plain poll interval.
Reuses @hive/shared/terminal.css's existing row-kind classes as-is —
the taxonomy's visual language isn't what changed, the component
model underneath it is.
- useAgentState hook: polls GET /api/state (4s interval for now — see
its file comment for why this isn't yet the SSE-triggered + login-
only-timer cadence the old page used; that lands with the live
stream + term-input commit, which is when clobbering the operator's
in-progress input actually becomes a risk).
- format.ts: fmtTokens/fmtAge, same output shapes as app.js's.
- modelEffort.ts: POST /api/model + /api/effort (same-origin, plain
fetch).
- dashboardBase.ts + pauseAction.ts: pause/resume POST to the
*dashboard's* origin via a real <form> submit, kept unchanged from
app.js — a cross-origin fetch needs CORS headers hive-c0re doesn't
send, a form submit sidesteps that same as it already did.
- Root.tsx: wires it all together, including app.js's "any non-online
status forces the turn-state badge to offline" behavior.
Screenshot-verified against a mock GET /api/state (real fetch, not
hardcoded props) + the real agent.css/theme.css/colors.css.
Builds + tsc --noEmit clean.
Adds a new Preact/TSX build alongside the existing app.js (esbuild
entry `main.tsx` → dist/static/main.{js,css}, same jsx/tsconfig shape
swarm-ui already uses) and the first real page slice: Header +
StatusChips, composing the Badge/Dropdown components from the prior
commit's PR. Not wired into index.html yet — app.js keeps rendering
the live page untouched while this fills in component by component
(state polling, the live SSE stream, login flow, inbox/todos, term
input) in follow-up commits on this branch.
StatusChips folds the model/effort pickers and pause into the badge
row itself (each badge IS its own control), replacing the old
overflow-menu-only pickers — the concrete fix the design guide already
names this page as needing. Presentational only for now (props, not
live data) so it's reviewable against sample data before being wired
to /api/state.
Screenshot-verified against the real agent.css/theme.css/colors.css
(headless chromium, sample data) — renders correctly.
Builds + tsc --noEmit clean.
Two findings on one panel, both from mara.
Form: 24 label series drawn as overlapping timeseries lines is
unreadable by construction. The panel is a current-count ranking, not a
time evolution, so it becomes a horizontal bar gauge on an instant
sort_desc query -- identity on the axis, magnitude in the bar, and a
fixed colour because colour was carrying no information that the axis
was not already carrying.
Title: the metric counts open AND closed issues. Measured, not assumed
-- the largest single label reads 1619 while gitea_issues_open reads
117, and no label can outnumber the set it is a subset of. The panel had
been asserting the opposite since it was written.
mara on the PR: "many hint texts are incorrect, most hives dont work on
hyperhive itself". Four panel descriptions asserted local practice as if
it were general:
Releases "expected to sit at zero on a hive that ships from
main" -- a claim about how THIS project releases
Automation surface "every hive registering the same webhook has bitten
this forge before" -- a war story from this
deployment, meaningless on any other
Organizations "near-constant by design" -- assumes our org layout
Issues open/closed "usually a bulk triage, not a productive afternoon"
-- a claim about our working rhythm
Each now describes what the metric IS and what a CHANGE means, and says
outright where the panel cannot know: whether zero releases is normal
depends on how a hive's projects ship, and a vertical step in closed is
some batch operation that this panel cannot distinguish from a burst of
real work.
The board title is NOT an instance of this and is unchanged -- all seven
dashboards use the 'hyperhive - <thing>' product namespace, checked
rather than assumed.
The forge restarted (new forgejo PID at 12:36), so it re-read [metrics]
and gitea_issues_by_label now exists -- 24 series carrying real label
names, read back out of the store rather than inferred from the metric
name appearing in a label-values list.
The panel shipped deliberately empty with EMPTY UNTIL THE FORGE RESTARTS
in its title and a matching noValue, because an empty graph and an
undeployed metric look identical. That warning is now false, and a stale
warning is worse than none: it tells a reader the panel cannot have data
at the moment it does.
Keeps the operational half rather than deleting it -- the restart
requirement is a fact about the NEXT metric someone enables, so it moves
into the description as a note and into noValue, where it fires only when
the series really is missing.
Adds the fourth per-service board. Unlike its three siblings it answers
"what does the forge CONTAIN", not "is the forge healthy" -- forgejo's
exporter publishes a census of stored objects and no request, error or
latency series at all.
That distinction is the board's own description rather than something a
reader has to infer, because a fourth per-service board that silently
answers a different question than the other three is worse than no board:
someone checks it during an incident and reads a flat line as calm.
Panels: repositories, users, orgs, issues open/closed, releases, the
open-vs-closed pair over time, repos-and-users together (they move in
lockstep when an agent is created, so divergence is the signal),
discussion volume, and the webhook/hooktask pair.
One panel ships deliberately EMPTY: issues-by-label. The metric is
enabled in the forge's config but forgejo reads its [metrics] section at
startup, so the series does not exist until that process restarts -- and
a config merge alone does not restart it. The panel title says so, the
description says how to tell "not deployed" from "no labelled issues",
and `noValue` says it again in the graph itself. An empty panel with no
explanation is the failure this board is trying not to be.
Gated: every panel's metric confirmed to exist in the store, with a
bogus name checked absent in the same run so the check can fail, plus a
known-positive control against the label-values query itself -- a
download that silently returned nothing would otherwise report every
metric missing. Gate kept as verify-3494-forge-board.sh.
mara: non-interactive Badge and StatusChip render identically (same
padding/radius/font, same tone-to-color mapping) and StatusChip's
4-tone/single-label shape is a strict subset of Badge's — no real
reason to keep both. Deletes StatusChip.tsx/.css, migrates AgentsPage's
config-PR chip, HivesPage's freshness chip, and ComponentsPage's own
table-status sample to Badge (tone/value, no onClick). Also drops the
now-redundant standalone StatusChip demo section on /components (the
Badge section already covers all 4 former chip tones plus accent).
Updated the two stale StatusChip references outside swarm-ui too:
design-guide.md's component-list example and colors.css's WCAG-
contrast-rationale comment.
New shared Preact primitives for the per-agent terminal redesign:
- Badge (@hive/shared/badge.js): a labelled status pill. Plain <span>
when static (e.g. "alive"), a real <button> with a disclosure caret
when given onClick (e.g. opens a Dropdown, or toggles itself in
place). Same component either way so a status row reads as one
consistent set of badges regardless of which are interactive.
- Dropdown (@hive/shared/dropdown.js): a small option list anchored
directly under whatever opened it (no portal, no native <dialog> —
see the file comment for why). Closes on outside click or Escape.
This is the fix for the design guide's own named anti-example: the
agent page's model/effort pickers live in the overflow menu while the
current model/effort only show as a disconnected chip. Badge+Dropdown
composed together is that control moved inline, next to what it shows.
Demoed on swarm-ui's /components page: all Badge tones, a label-prefixed
badge, an interactive badge that opens a Dropdown (model-picker shape),
and an interactive badge that toggles itself (pause/resume shape).
Both packages build + tsc --noEmit clean.
The comment-block lint (added in 79dc8ca6) now trips on this block —
genuinely pre-existing, unrelated to that change, just newly caught.
Per the lint's own suggested remedy: relocated the full per-directive
reasoning plus both footguns (session-cache keying, the Host-header
clobber that can recurse a subrequest into itself) to a new
"Dialing another vhost by name" section in docs/gateway.md, and left
a short why + pointer comment in the source. No behavior change.
verifiedProxyTo (43ae164d) verified TLS but left Host to nixpkgs'
recommendedProxySettings, which sets Host to the CALLING vhost, not
the target. Since every consumer resolves back to this same gateway,
nginx picks the vhost to answer by Host header (not by the SNI
proxy_ssl_name already sends) — so every auth subrequest looped back
into its own vhost's auth_request, recursing until nginx's subrequest
depth limit turned it into a 500, on every domain gated by SSO.
Pin Host (and the rest of the header set nixpkgs' recommended include
would otherwise still be the one to set) inside verifiedProxyTo, and
set recommendedProxySettings = false on each of the four call sites so
nixpkgs' own copy — appended after a location's extraConfig — can't
clobber it back.
One list, in nix/reserved-names.nix, handed to everything that needs it
as HIVE_RESERVED_NAMES. Keeping it current becomes a config change
rather than a rebuild, and hive names and agent names -- one namespace
going forward -- are checked against the same file: swarm-otel.nix's
hand-written reservedOwners is gone.
Whitespace-separated rather than JSON, deliberately, unlike the
structured env vars beside it. Every entry is an Ident ([a-z0-9-]), so
whitespace cannot occur inside a name and the encoding is provably
lossless; JSON would mean either a parser dependency in a crate whose
purpose is to have none, or a copy of the parse in every consumer.
An UNSET variable is not "nothing is reserved". Both creation sites log
an error and return a warning saying the check did not run, so a
misconfigured deployment says so instead of silently accepting every
name. A blank value folds into unset: nix always renders a non-empty
list, so present-but-empty is a rendering fault, not a declaration.
Two guards whose subject moved out of their own file now assert their
own case is still in it, because a guard that can be retired by an edit
elsewhere is not a guard:
- swarm-otel.nix asserts reserved-names.nix still contains its
swarmTierName.
- hive-sh4re's sentinel drift test PANICS when the variable is missing
rather than skipping -- a drift test that quietly does nothing still
reports green. checks.nix and devshell.nix both export it so CI and a
local cargo test agree. Verified as a pair: with the variable set, 8
tests pass; with it unset, exactly the 4 drift tests fail and the
unrelated ones still pass.
The reserved-name check landed only in `swarm-controller::create_agent`.
That daemon is opt-in and off on most hives, so the ordinary per-hive
flow -- `request_init_config` -> `handle_request_init_config` ->
`require_new_child` -> `submit_init_config` -- stayed exactly as
unguarded as before: an agent named `operator`, `forge` or `system` was
still creatable through the path every hive uses, with no warning.
Caught in review by argus. The issue named `create_agent` as the
existing shape to copy, so the shape got copied and the question of
which OTHER sites create an agent never got asked -- an issue naming one
call site is describing an exemplar, not an inventory.
Same treatment as the other path: warn, do not refuse.
The warning needs somewhere to go. `Response` had `Ok` (carries nothing)
and `Err` (refuses), so a check that warns had no way to reach its
caller. Adds `Response::OkWarn { warnings }` -- additive, every existing
`Ok` site is untouched -- rendered by `format_ack` *after* the success
line rather than instead of it: the approval really was queued, and a
warning shown as a failure invites a retry that queues a second one.
Mutation-verified: dropping the warnings and unconditionally appending a
marker each turn a different test red.
An agent's name was checked for shape and never for meaning:
`Ident::parse` is 1-63 chars of [a-z0-9-] and there was no reserved-name
list anywhere in the tree. So an agent could be called `operator`,
`forge` or `todo` -- names the message layer already produces as a
sender -- and a wake from that component became indistinguishable, at the
broker, from a message sent by the agent.
Adds `RESERVED_NAMES` + `is_reserved_name` to `hive-types`, the zero-dep
leaf both `hive-c0re` and `swarm-controller` already depend on, so
neither grows a dependency to use it.
Every entry is a value some component actually produces as a message
`from`/`to`, taken from `hive-sh4re`'s own sentinel constants rather than
guessed: operator, system, reminder, forge, scheduled, todo, compact,
graceful-stop. Two sentinels are deliberately absent -- `<parent>` and
`<children>` are unreachable as agent names because the charset rejects
them, and `ruth` is a real agent, so wanting that name is a name being
*taken*, which the roster answers.
Deliberately not enforced inside `Ident::parse`: parsing runs on every
read of an already-created name, so rejecting there would make existing
agents unreadable rather than un-creatable -- and it would be a refusal,
which is a stronger action than the warning this is used for today.
`create_agent` now warns on both halves -- a reserved name, and a name
that is also a hive in the roster -- and does not refuse. The warnings
ride on `CreateAgentResponse` rather than only the daemon's log, because
the person who can still fix the name in one keystroke is holding the
response, not reading the journal. `skip_serializing_if` keeps the
no-warning JSON byte-identical to before, so this is a non-breaking first
step toward refusing later.
`hive-sh4re` gains a drift test tying its sentinel constants to the list:
two crates that cannot import each other's intent now fail loudly if a
sentinel is added without being reserved. Mutation-verified -- forcing
the predicate false, forcing it true, and dropping a single entry each
turn a different test red.
Consumers reached authelia at `127.0.0.1:<port>`, which encoded a
co-location nobody agreed to: the gateway and authelia are not required
to share a host, so the literal is a requirement stated only by being
unwriteable any other way. Moving them to the name is the point of the
issue.
But a name over https is only half of "https and auth". nginx's
`proxy_ssl_verify` is OFF by default and there was no `proxy_ssl_*`
anywhere in the tree, so the obvious repoint would have produced an
encrypted, unauthenticated hop -- which works, and keeps working,
against any certificate at all.
Adds `gateway.lib.verifiedProxyTo <name>` next to the rest of the vhost
kit, so the convention has one definition rather than a copy in each
consuming module, and repoints the four call sites through it.
Each directive was checked against a real nginx with the opposite arm
run as a control:
- the CA *bundle* (root + intermediate) is accepted -- worth checking,
since `hive-ca-trust.nix` warns off consumers that read only one
certificate, and nginx is not one of those
- verification checks the chain: an unrelated CA fails
- and the HOSTNAME: a wrong `proxy_ssl_name` fails even with a good
chain. Chain-only would accept any cert this CA ever signed, which
for an internal CA is every service on the hive
- with verify off, the wrong CA passes -- so the failures above come
from verification, not from the connection
Bind addresses are untouched. This changes what consumers dial, not what
anything listens on.
The upstream gitea-actions-runner unit uses DynamicUser, and systemd
mounts a dynamic unit's state directory noexec. Measured from a job's
own /proc/self/mountinfo on the live runner, with /tmp as the control:
the state dir carries noexec, /tmp does not, and a script written and
chmod +x'd in the workspace fails execve with EACCES while the same
script in /tmp runs.
This was invisible for as long as every workflow compiled inside the
nix sandbox and executed nothing from the workspace. The first job that
built a binary into the runner's own target dir -- an instrumented
coverage run -- died on its first build script.
ExecPaths= re-mounts the state dir executable. Both spellings are
listed with the ignore-if-absent prefix because ExecPaths resolves
against the host root, where the dynamic-user layout makes
/var/lib/gitea-runner a symlink into private/.
/etc/gitconfig shipped `helper = git-credential-hive-forge`. git prepends
`git-credential-` to any helper value that is not an absolute path, so
that resolves to `git-credential-git-credential-hive-forge`, which does
not exist -- no helper runs at all. The sibling github.nix has always used
the short form.
Measured rather than read off the docs, with the arms isolated from the
personal ~/.gitconfig:
helper = git-credential-hive-forge -> 0 credentials, and git prints
"'credential-git-credential-hive-forge' is not a git command"
helper = hive-forge -> 1 credential, clean stderr
The reason this survived: every long-lived agent has a personal
~/.gitconfig naming the helper by ABSOLUTE path, which git accepts, so
pushes keep working and the stderr line reads as noise. The system config
is masked exactly where someone would notice it and bites a fresh agent
that has no such file.
The authelia vhost intercepts upstream errors and serves a friendly
"SSO unavailable" page. The `=` form of `error_page` takes its status
from the redirected location, and that location serves a file -- so the
page is returned as **200**.
That is right for a human typing the URL and wrong for every machine
caller, all of which reach authelia through this same vhost by name:
- `/api/authz/auth-request` -- nginx `auth_request` treats any 2xx as
success, so a down authelia means access GRANTED
- `/api/oidc/introspection` -- a token check answering 200
- `/api/oidc/token`, `/.well-known/openid-configuration` -- clients
parsing an HTML error page as their JSON document
Routes `/api/` and `/.well-known/` without the interception. A longer
prefix wins over `/`, and the intercept directives live inside the `/`
location rather than at server level, so they do not reach the new ones.
Split by AUDIENCE rather than by an enumerated path list: a human still
gets the page, and every machine caller -- including the login page's own
XHR, and any endpoint added later -- gets the real status.
Measured against a real nginx with a dead upstream, both arms: machine
paths return 502 where they returned 200+HTML, a subrequest through the
new prefix denies (matching a direct port dial) where through `/` it
served the protected content, and the browser control confirms the
friendly page survives. URI preservation checked separately against a
live echo upstream -- `proxy_pass` with no URI part passes the full
original path.
Adds `cargo llvm-cov` to the devshell and a coverage workflow that runs
only when someone asks for it.
Manual dispatch rather than nightly or per-PR, per the discussion on the
issue: a coverage run needs its own instrumented build and cannot reuse
the normal test artifacts, so it roughly doubles a test job. "Do we have
glaring holes" is a question someone asks occasionally, not a gate every
PR pays for, and not a number worth spending farm time on every night
whether or not anyone reads it. Manual dispatch costs nothing until the
answer is wanted.
Its own workflow file rather than a job in ci.yml: that file already has
a `workflow_dispatch` trigger so `hive-forge ci-rerun` can retrigger
without an empty commit, and a trigger there fires EVERY job in the
file -- a coverage job added there would run on every pull request.
No threshold and no --fail-under-lines. A coverage gate mostly teaches
people to write assertion-free tests that execute lines; the report is
the deliverable and the number is for a human to read.
The devshell needs LLVM_COV / LLVM_PROFDATA set explicitly: cargo
llvm-cov expects rustup's `llvm-tools-preview` beside the toolchain and
nixpkgs has no such component, so without them it aborts with "failed to
find llvm-tools-preview" -- which reads like a missing install rather
than a path the shell has to name.
Verified by running it: `cargo llvm-cov --package hive-types` produces a
real report (3 tests, 85.44% regions). Both the variable names and the
package were wrong on the first attempt and only running it said so --
the names take no `_PATH` suffix, and the binaries are in
`llvmPackages.llvm`, not `llvmPackages.bintools`, which is the linker
wrapper and ships neither.
Host metrics arrived carrying `hive` and no host attribute, so "which
host is out of memory" was answerable only as "which hive" -- true only
while a hive is one machine, which is the framing the swarm work exists
to end.
Adds the collector's own `resourcedetection` processor to the metrics
pipeline, emitting both `host.name` and `host.id`.
`resourcedetection` rather than a value picked in nix: this is the
canonical OTel mechanism for host identity, and choosing our own would
answer a question the tool already answers, differently from every other
deployment. A resource attribute is captured into the stored series, so
a private naming scheme is expensive in the way that lasts.
Both attributes, deliberately: `host.name` is readable and what a panel
groups by, but splits the series silently on a rename with nothing
linking old to new; `host.id` survives that and is unreadable alone.
Hosts get renamed and repurposed exactly when services move off one box.
The processor sits in the pipeline, so it applies to every receiver --
agent-pushed OTLP and scraped targets included, not just `hostmetrics`.
Keys verified against otelcol-contrib 0.151.0 with two deliberate
mutations rejected in the same run (a bogus resource attribute and a
bogus detector key), so "accepted" is distinguishable from a validator
that ignores what it does not recognise.
Turns on gitea_issues_by_label and gitea_issues_by_repository, gated on
behindGateway exactly like the metrics endpoint itself -- they are only
reachable through the authenticated location that condition creates.
These are off by default upstream because they are the only metrics in
this section whose series count grows with the CONTENT of the forge
rather than with its feature set, so the question is cardinality.
Measured rather than assumed: ~19 distinct labels and ~29 repos on this
instance, well under 100 series, against a store that took +2234 from
nine host scrapers.
Still no TOKEN. Forgejo can guard the endpoint with its own static
bearer, but the swarm authenticates the scraper at the gateway, and a
second per-service credential would be the one that stops getting
rotated -- the reasoning already recorded above ENABLED applies
unchanged to the breakdowns.
Forgejo's issue-list endpoint answers 'was this merged?' only inside the
nested pull_request object, while state says closed for a merged PR and
for one closed without merging alike. So the obvious top-level query is
null or ambiguous for every row, and with a // default it renders as a
confident 'nothing merged' that cannot ever be right -- a wrong answer
shaped exactly like a clean one.
Copy merged and merged_at up to the top level of each PR row so the
obvious query is the correct one. Additive: the nested object is left
untouched so an existing consumer keeps working, and issue rows have no
pull_request and pass through unchanged.
The head branch is deliberately not hoisted: this endpoint does not carry
it at all -- the row's ref is an empty string, not the branch -- so there
is nothing to lift. pr show has head_branch.
mara: "only the abstract jobq part was asked for in the first place" —
the previous revision still carried the full hive-c0re step-kind table
under a "core-specific nodes" heading; that catalogue belongs in
coordinator.md (where it already lived) alongside the rest of
hive-c0re's job-queue internals, not duplicated here.
jobq.md is now just the domain-agnostic model (graph of steps + shared
resource slots) plus the generic row/step/glyph framing for watching it
in the dashboard — no hive-c0re-specific step names anywhere.
Coordinator.md's job-queue section, docs/README.md, and the root
CLAUDE.md reading-paths index are updated to match.
mara: "the issue wanted jobq the abstract thing docs, but you
documented the hive nodes" — the original PR only had the
core-specific node table (the issue's second ask); it was missing the
first ask, an operator-facing explanation of the jobq engine itself
(graph of steps + shared resource slots) independent of what hive-c0re
specifically builds on top of it.
Adds that as its own section up front, and reframes the existing node
table under an explicit "core-specific nodes" heading so the two asks
read as clearly separate pieces.
hyperhive/hyperhive#2999 asked for an operator-facing page explaining
the job queue's node kinds in the abstract, without the coordinator's
implementation detail (leases, resource windows, module layout).
Add docs/jobq.md: what a queue row/step means, a plain-language table
of every node label an operator actually sees in the dashboard
(BU1LDS/R3BU1LD QU3U3 and swarm-ui's /jobs page both render the same
graph), and a rough shape for a typical rebuild.
Wire it into docs/README.md and the root CLAUDE.md reading-paths index
alongside coordinator.md. Add a pointer from coordinator.md's job-queue
section to the new page so the two don't try to explain the same thing
twice at different levels.
The token file is 0600 root-owned and the precondition ran as the unit's
dynamic gitea-runner user, so it could not read it -- and it did not fail
closed. `[ -s ]` succeeds on a stat alone, grep then died with EACCES,
and `! grep` turned that error into true, so the gate exited 0 on a file
it had never read. It has reported 'a real token is present' without
looking for its entire existence, which is why the runner kept retrying
instead of holding still and saying why.
Prefix the step with + so it runs with full privileges. This grants the
runner nothing new: tokenFile becomes EnvironmentFile=, which systemd
already reads as root before dropping privileges, so the payload never
passes through an unprivileged reader either way -- the + only lets the
gate observe what systemd observes.
Also assert readability separately and loudly, rather than folding it
into the placeholder test: an unreadable file and a real token produced
the same verdict, so 'I could not look' was indistinguishable from 'I
looked and it is fine'.
Upstream's ExecStartPre re-registers only when .runner is absent, the
labels changed, or the registration token hash changed -- never when the
instance URL changed. So writing a fresh token and restarting the runner
registers only if the hash happens to differ, which is Forgejo's choice
to make: whether the admin registration-token endpoint mints a new token
per request or hands back a stable one is not ours to depend on.
hive-c0re already decides correctly -- ensure_ci_runner_registered only
reaches this helper once it has concluded the credentials are absent or
stale -- but the remediation was a no-op, so re-registration was
requested every boot and never happened.
Remove .runner before the restart so upstream takes its absence branch,
the one it evaluates unconditionally. NotFound is success; any other
error propagates rather than reporting Ok for a registration that never
ran.
Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
Nothing in the tree rendered a `[credential "<forge>"]` entry for an
agent, so every agent's `~/.gitconfig` accumulated one by hand, per
generation of forge address. Append-only, none ever removed, and after
the domain move the live one absent entirely:
[credential "http://forge.<old-hive>"]
[credential "http://localhost:3000"]
The absence of a writer is the defect. A value interpolated at eval time
follows a rename; a value captured into a mutable home file does not.
hive-c0re's own gitconfig already derives its scope from
`swarm.forge.domain` and moved correctly for exactly that reason.
What made it expensive to diagnose is that it does not present as a
credential problem. `git fetch` against a stale remote still succeeds --
the old name redirects and a public read needs no auth -- so the break
surfaces only at the first authenticated push, long after the move, as
`could not read Username for '<new host>'`. That names a host the agent
was never configured for, which reads like DNS or TLS.
Same class as the CI runner keeping its registered address, one tier
down.
The shape is `github.nix`'s, unchanged: a small credential helper that
reads the token from the agent's state file at invocation, with the
token PATH baked in rather than the value, because claude's Bash tool
runs in a minimal env that never sources /etc/set-environment.
`environment.etc."gitconfig"` is already bound by github.nix; the two
merge rather than collide because the option is `lines`. Verified by
eval with both modules defining it before this was written -- a silent
last-wins there would drop one integration's credentials and look
exactly like this bug again.
The pending-bytes panel described itself as the leading indicator for
the slow-consumer stat 'beside it', and then sat alone in its own row
with eighteen grid units of empty space. A description that tells the
reader to look beside it, where nothing is, is worse than no
description.
Slow consumers and pending bytes are now adjacent in the top row, and
the two JetStream panels share the second. No gaps.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
They differ by one letter and a grep for the shorter name matches both.
The summary is what ships; the per-process scraper is deliberately absent,
and a reader (human or grep) cannot tell from the diff alone.
Nothing measured the host itself. hive-c0re emits per-container metrics
and each service exports its own, so the store could answer 'is this
container using more memory than last week' and could not answer 'is the
machine under them out of memory' -- the question every one of those
readings is implicitly relative to.
Hive tier rather than swarm: a host is owned by the hive running on it,
and the swarm tier runs one collector for the whole swarm, so it would
have to reach other machines to see theirs.
Every scraper the pinned receiver offers except process, which adds a
series set per running process -- unbounded on a hive host, where the
nine shipped are a fixed handful. Coverage and cardinality are different
axes and only the second can hurt the store.
Refs #3649
The collector's client is registered for authelia.bearer.authz, but the
prometheus scrape asked for no scopes, so every token came back carrying
none and authelia refused it at introspection with 'the requested scope
is invalid, unknown, or malformed'. Forgejo metrics were never scraped.
The rule was already stated one field below, for the audience: registered
is not requested. The two travel together, and a config read cannot see
the one that is missing -- every inspection of the rendered config came
back correct, because what was present was correct.
NODE_EXTRA_CA_CERTS beside it is Node's and additive. gitea-runner is Go,
whose trust store is replacing: it reads SSL_CERT_FILE and trusts only
what that names. So the container trusted the hive CA for its Node
actions and never for the daemon itself.
The daemon then fails startup with x509 unknown authority the moment it
reaches a TLS endpoint, crash-loops on Restart=on-failure, and no job is
picked up -- every pull request just sits at Waiting to run.
Not claimed as the whole cause of that outage: which address the daemon
holds is a separate question. The trust gap is real under every
explanation, and adding a CA to one unit cannot break a working path.
