Fetches an opaque 32-byte value and writes it where hive-matrix.nix
already looks, so the homeserver never learns the store exists.
Chosen over authelia's OIDC secret deliberately: that one needs a
.secret AND a matching .digest, so shipping it first would debug 'can a
reader authenticate and get bytes back' and 'did we write authelia's
file format right' at once, with an SSO outage as the failure mode. Here
the failure is narrow -- new agent accounts cannot be provisioned,
existing ones untouched.
Every failure path keeps the local token: no such key, sealed store,
unreachable store, empty value. The activation script's mint-if-absent
is untouched, so a hive with no store behaves exactly as it does today.
matrixMachine is a literal because hive-matrix.nix declares no `machine`
option -- `matrixCfg.machine` parses cleanly and fails at module-system
resolution, which is the kind of error only reading the target module
catches.
A CA that signs exactly two things -- the store's server certificate and
the client certificate of whoever reads from it -- and distributes
nothing. Not the hive CA, not the swarm CA: the store will distribute
both, and an authority you must already hold a certificate from cannot
be one the store hands out. Not the gateway's HTTPS material either,
self-signed or ACME; that is a different trust domain with a different
audience.
The minting lives here rather than in swarm-bao.nix because it is an
opinion about where the store's identity comes from -- the most
consequential one available. The service serves what it is handed. A
deployment with a real internal CA drops this file and names its own
paths in serverCertFile / clientCaFile, and nothing in the store
changes. Ordering simplifies too: with the unit and its consumer in one
module, before/requiredBy is internal rather than a cross-module fact.
Idempotent on ABSENCE only. Re-issuing the CA invalidates every client
certificate already trusting it, so a rebuild that refreshed it would
lock out every reader in the swarm at once.
OpenBao in a swarm-bao nixos-container, unsealed by the host TPM through
the built-in pkcs11 seal, with shamir as the documented opt-out for
hardware that has no TPM. The seal guard asks the package whether it was
built with an HSM rather than reading its version, and falls back to the
version cliff only when a package advertises no tags -- it fails closed.
Every listener serves TLS: loopback unconditionally, because the host
running the store is always one of its readers, plus whatever
extraListenAddresses names, because which network the other hives share
is a deployment fact. Client-certificate verification turns on only when
clientCaFile says what to trust.
The store's identity is an input, and nothing in THIS module fills it
in. Service-to-store mTLS is a separate trust domain from the gateway's
HTTPS certificates and from both CAs in this tree: a store must not take
its identity from an authority it will itself distribute, or it cannot
come up before the thing that issues it. The module asserts when the
paths are unset rather than reaching for whichever CA happens to be
wired; what supplies them on a self-contained deploy is the next commit.
The leaf is copied to the container's state dir by a host unit rather
than bind-mounted: nixos-container refuses to start when a bind source
is missing, and a directory holding a leaf usually holds the CA's key
beside it.
A swarm has one store, so running it is a fact about this machine and
belongs beside the other deploy.* toggles rather than in the namespace
every host agrees on. Attrset with an enable, not a bare bool, so a
second deployment decision has somewhere to go.
Asserted from swarm.enableRequiredServices alongside the other
once-per-swarm services, and asserted as a mkDefault -- which is what
keeps *where* the store runs a separate question from *that* it runs.
Set it directly to put the store on a host of its own; a hive that does
not run it is a client either way, reaching it by name.
Two descriptions in the file still narrated the old swarm.* layout as
something that used to be true, and the queue's claimed it belongs on
the shared-services host. Both now say what holds.
Both retentionPeriods move to deploy.<store>.retentionPeriod. Measured
before moving: each is read exactly once, in the container definition of
the module that deploys the store, and no client hive consults either --
so they are options on the auto-deployed service rather than something
every host has to agree on.
What stays in swarm.* is what a client needs to reach the store: the
package, the name it answers on, the port. Both modules said "what stays
here is its package, domain, retention and wiring" in their options-block
comments; retention no longer does, so those say so.
Renames go in deploy.nix with the rest, so the whole move keeps one home
and one file to delete when the deprecation window closes.
Last of the swarm.* -> deploy.* moves for the enable-shaped options.
Whether this host also runs the CI runner is a per-machine decision, so
it was sitting in the namespace that has to be identical swarm-wide.
Renames all five options, not just enable: name, concurrency, labels and
package all describe the runner THIS host would run, and leaving them
behind would keep the original defect for four more options. One rename
entry each, because ci is a plain attrset rather than a submodule type,
so there is no parent path to rename in one go.
deploy.forgejo is the name deploy.nix's own header already specifies,
including this exact case -- "ci (does this host run the runner too) is
exactly that shape". It is the only entry with no enable: the forge is
not optional, it deploys with hyperhive itself, so running the runner is
the only deployment decision it has.
Also fixes hive-c0re/src/forge/ci_runner.rs, whose doc comment named
services.hyperhive.forge.ci.enable -- missing the swarm. segment, so it
had never resolved to a real option.
HIVE_MATRIX_URL was conditioned on this host deploying the homeserver,
so a hive that is a client of the swarm's matrix forwarded nothing and
every agent's hive-matrix-daemon no-opped. hive-c0re already provisions
those agents' accounts against a remote homeserver through
HIVE_MATRIX_API_URL three lines below, whose gate is `apiUrl != null` —
it registered the users and then never told them where to log in.
gatewayHost is swarm-scoped and defaults to chat.<swarm-domain>, so it
names the swarm's one homeserver from any member hive. Its being set is
the real question, and a null swarm domain is already caught by the
assertion in hive-network.nix rather than by this expression.
swarm.matrix.enable was the last toggle still spelled swarm.*, which
is the namespace every host in the swarm agrees on - and "does this
host run the homeserver" is exactly what differs between them. It
moves to deploy.matrix.enable with the rest; the genuinely swarm-wide
matrix settings stay where they are.
mkRenamedOptionModule keeps existing configs evaluating with one
warning naming both paths.
The sweep for readers had to be unanchored and cross-language: three
of them were outside nix, including a hive-c0re error message telling
the operator to set the old name. A dotted grep also cannot see the
rename module's own list form, so that was checked separately.
services.hyperhive.swarm.* used to carry both swarm-wide facts (name,
domain, hives, ca) and the 'does THIS host run it' toggles for every
swarm service. deploy.nix already split those toggles out into their
own services.hyperhive.deploy.* namespace; this is the docs-side
follow-up now that swarm.* is genuinely swarm-wide-only.
nix/docs/default.nix: host.md drops the swarm/deploy subtrees
(builtins.removeAttrs on the already-picked host tree, not
lib.recursiveUpdate -- that merges rather than deletes, which would
have silently kept them on host.md); swarm.md and deploy.md are new
markdown pages, each its own pickSubtrees root. nix/packages/default.nix
exposes docs-swarm/docs-deploy as flake outputs alongside the existing
docs-host/docs-agent. docs/gotchas.md's Nix options reference section
and docs/swarm/README.md get pointers to the new split.
Per review: `deploy.otel` does not imply swarm level, and there is a
hive-tier collector too -- `services.hyperhive.otel.enable`, which every
hive runs. The tier is the entire distinction between the two, so the
name has to carry it, matching `deploy.swarm-controller` and
`deploy.swarm-ui`.
15 swarm-tier references renamed across 8 files. The hive-tier
collector's 16 references are deliberately untouched, verified as a
control on the same command.
Three spellings needed three different patterns, all inside this one
rename: the dotted path; the `mkRenamedOptionModule` target written as a
nix LIST (`[ ... "deploy" "otel" "enable" ]`), which no dotted grep can
match; and prose in docs/observability.md spelled `deploy.otel` with
neither `.enable` nor a leading dot. Unanchored `deploy\.otel\b` is the
only pattern that finds all three.
Round-3 review catch: the all-local mode still set
`services.hyperhive.swarm.controller.enable` -- the exact path this
branch's own `mkRenamedOptionModule` deprecates. It forwards correctly,
so nothing broke, but every eval with `enableAllLocalDefaults = true`
printed a deprecation warning and the tree depended on the shim it is
retiring.
It survived the rewrite because the path is SPLIT ACROSS NESTING:
`swarm = { controller.enable = ...; }`. No grep for a dotted path can
match text that is not contiguous -- not the prefix-anchored pattern the
rewrite used, and not the suffix-anchored one added after that missed
hive-tls.nix. What finds it is the bare tail with no prefix at all
(`^\s*controller\.enable\s*=`), which reports exactly one hit outside
deploy.nix's declarations.
The assignment moves out of the `swarm` attrset rather than staying in
it: `deploy.*` is a different top-level path, so the collision the
neighbouring warning describes -- two definitions of `swarm`, the nested
one silently lost -- does not apply.
Two corrections from review, applied forward on this branch rather than
by rewriting it.
`deploy.<service>` was a bare bool, which makes
`deploy.forgejo = { enable; ci; }` unrepresentable -- the nested
CI-runner sub-option this namespace was designed around. Every entry is
now an attrset with an `enable`, so a second per-host deployment
decision becomes an ordinary addition rather than a migration.
`deploy.controller` is now `deploy.swarm-controller`, consistent with
`deploy.swarm-ui`, which was introduced in the same commit.
89 references rewritten across 24 files -- nix, Rust, docs, and the
repo's own CLAUDE.md.
The prefix-anchored sweep missed exactly one, and it was live code:
hive-tls.nix spells it `hyperhiveCfg.deploy.controller` -- the only
`hyperhiveCfg` prefix among 45 references. A suffix grep
(`\.deploy\.<name>`) finds it; a path-anchored one cannot, because the
head of a reference is whatever alias the reading file happens to bind.
The one move where the suffix grep is the wrong instrument. There are two
otel options a word apart — `swarm.otel.enable` (one per swarm) and
`otel.enable` (one per hive, every hive runs it) — so `\.otel\.enable`
matches twenty-five references of which most must not change. The module
already carries a comment warning about exactly this, on a line that
names `swarm.otel` in full rather than through a binding.
Triaged by hand and confirmed the other way round: after the move, a grep
for the per-hive option still finds it in the files that should keep it.
Also worth recording what the alias sweep cannot do. `swarm.nix` reads
this as `swarmCfg.otel.enable`, where `swarmCfg = cfg.swarm` and `cfg =
config.services.hyperhive` — an alias bound to an alias, two hops from
the option path. No syntactic sweep resolves that chain, which is why the
module system's own evaluation is the only complete check here and the
static sweeps are a way to narrow the work, not to finish it.
Same shape as the others. Three of the eight references were prose
rather than code: two assertion messages naming the option to an
operator, and a `defaultText` in the authelia module, whose
`oidc.hiveIdentities` derives from whether the queue runs here.
Also corrects a comment in swarm.nix that this commit's own hunk sits
directly beneath. It said `enableRequiredServices` turns on matrix and
authelia "but NOT nats (nats has no mode that enables it)", which stopped
being true when the queue was added to that switch — before this change,
not by it. The reasoning it introduces is still right and still worth
keeping; only the premise had expired.
The largest of these moves: sixteen references spelled through `let`
aliases across eight modules, plus eight more spelled as a path, plus
five documentation pages.
authelia is also the clearest case for why the two namespaces exist.
`swarm.authelia.url` is needed by *every* hive in the swarm — it says
where to send a browser to authenticate — while running the container is
the business of exactly one host. The client half and the server half
were sharing a namespace whose whole contract is "identical everywhere",
and only one of them could honour it.
`swarm.authelia.oidc.clients` stays where it is for the same reason:
several modules register a client there, gated on authelia running here,
and the registry itself is what the service *is* rather than a decision
about this machine.
One sweep note worth recording: a grep for `swarm.authelia.enable` misses
`swarmCfg.authelia.enable`, because the prefix is whatever the reading
file bound. Grepping the suffix `.authelia.enable` finds both, and found
a reference in swarm.nix that the path-shaped pattern did not.
One commit rather than two because they are not independent: the UI's
`enable` had the controller's as its literal default, so moving the
controller alone would leave the UI's default naming an option that no
longer exists.
The UI keeps that derivation in its new home — it is a view onto the
controller's state and reaches it over that daemon's unix socket, so the
host running the controller is the host that can serve it.
Three spellings had to move together for the UI, not one: the `default`,
the `defaultText` shown in the options doc, and the description prose
that names the old path in words. A grep for the option path finds the
first two.
The sweep also reached outside nix: `swarm-controller`'s crate README and
its `//!` module doc both named the option, as did this repo's own
CLAUDE.md and four pages under docs/. An option's name is API, and its
documentation lives wherever someone thought to write it down.
Same move as grafana, and the three belong together: they derive from one
switch and a store with no UI is as useless as a UI with no store.
`victorialogs` is the case that shows why the option-path sweep is not
enough on its own. It has **zero** references spelled
`swarm.victorialogs.enable` anywhere in the tree, and four spelled
through `let` aliases (`vlCfg.enable` in the collector, `cfg.enable` in
its own module). A sweep for the path would have reported nothing to do
and left every reader broken.
Prose moved with the code rather than being left behind: the comments in
swarm-required-services.nix that explained why the pair derives together
now sit above the assignments that do it, instead of above the gap where
they used to be.
`ssoLocal` in the grafana module read `cfg.enable`, where `cfg` is the
local binding for `services.hyperhive.swarm.grafana` — the option set the
previous commit removed `enable` from. Evaluation fails on the missing
attribute.
The sweep that missed it looked for the option *path*
(`grafana.enable`). This reference is spelled through a `let` alias, so
no grep for the path can find it, and a parse check cannot either: it is
syntactically valid and only wrong once the module system resolves the
attribute.
The check that does find it: for each file, extract the local bindings
that point at the option set being changed, then grep for
`<alias>.<removed-attr>`. Run against this tree it reports nothing for
grafana and sixteen references for authelia, which is the next module to
move.
`services.hyperhive.swarm.*` is meant to be identical on every host in a
swarm — it describes the swarm, and every hive needs all of it to be a
client. But it also carried the `enable` toggles, which are precisely the
values that must differ per machine. The namespace that should be the
same everywhere held the one thing that cannot be.
Adds `services.hyperhive.deploy.*` for a host's deployment decisions, and
moves the first of them (`swarm.grafana.enable` -> `deploy.grafana`) as
the pattern for the rest. Flat and named for the thing deployed rather
than grouped under a "swarm services" attribute: from the deploy side it
does not matter what kind of thing each one is, and a grouping by service
kind would re-encode the service-side taxonomy into a layer that does not
care about it.
Behaviour is unchanged. The move is a rename in the strict sense — same
type, same meaning, new path — so `mkRenamedOptionModule` carries it and
existing configs keep evaluating with one warning naming both paths. The
renames live in the new module rather than the service modules, so the
whole migration has a single home and a single file to delete when the
deprecation window closes.
harness-base.nix has never existed in this tree. Four comments named it,
or a `harness-base` module, as the place to look:
- weston-vnc.nix: the agent user is declared and home-chowned by
nix/agent-modules/user.nix
- hive-ci.nix: the sandbox-fallback reasoning lives in
nix/agent-modules/default.nix -- which the very next comment block in
the same file already cites correctly
- packages/default.nix: the per-bin consumer is
nix/agent-modules/packages.nix
- hive-sh4re/src/assets.rs: HIVE_ASSETS_DIR is set by
hive-c0re/environment.nix and agent-modules/default.nix +
agent-service.nix, and the package is built by nix/packages/assets.nix
-- not the equally nonexistent nix/assets.nix
assets.rs was twice declared out of scope on the sibling PR because it
names a module rather than a file. That distinction was real and
irrelevant: neither the module nor the nix/assets.nix path it points at
exists. Reading the wording is not checking the reference.
Every replacement path was verified to exist, with a deliberately bogus
path as a control.
Per mara on #3725: the on/off toggle is removed, and required env vars
unset lead to a crash. HIVE_NETWORK_ISOLATION is gone from
hive-network.nix -- it was the toggle.
Validation happens once at daemon startup rather than per container.
The variables are process-global, so a bad value breaks every container
rather than one: failing at boot gives a single diagnostic naming the
bad value, and cannot reach a state where some containers were
configured before it was noticed.
Option<NetworkIsolation> collapses to NetworkIsolation through the wire
type, client and helper, which deletes the branch instead of leaving it
unreachable. serde(default) is dropped on that field deliberately: a
request omitting isolation is now rejected rather than defaulting to a
container sharing the host's network namespace.
What this replaces was a silent security downgrade. Of the four ways
into the old fallback, two logged nothing at all -- a container came up
without isolation and the journal agreed it was fine.
Doc comments that still described the removed branch are updated
(argus's note on #3723 scoped that to this issue). The hive-priv one is
a minimal edit inside the block #3723 rewrites; de-splicing is that
PR's job.
- nix/checks.nix: 'hyperhive#3685' in a source comment trips the
tracker-tag lint (prose-only, per hive-rules.md) — argus caught it on
review. Reworded without the ref.
- frontend/npm-deps-hash: the earlier scaffold commit
(be18f460) added preact/typescript to agent/package.json, which
regenerated package-lock.json but never re-derived this hash in
lockstep — nix flake check's frontend derivation was failing on a
hash mismatch. Regenerated via
'nix run nixpkgs#prefetch-npm-deps -- frontend/package-lock.json'.
- StatusChips.tsx: drop the stale 'useAgentState lands in a later
commit' comment now that Root.tsx already wires it.
- nix/checks.nix: add agent-typecheck, same shape as swarm-ui-typecheck,
so a real TS error in the agent package fails CI instead of just
esbuild-stripping silently.
Two findings on one panel, both from mara.
Form: 24 label series drawn as overlapping timeseries lines is
unreadable by construction. The panel is a current-count ranking, not a
time evolution, so it becomes a horizontal bar gauge on an instant
sort_desc query -- identity on the axis, magnitude in the bar, and a
fixed colour because colour was carrying no information that the axis
was not already carrying.
Title: the metric counts open AND closed issues. Measured, not assumed
-- the largest single label reads 1619 while gitea_issues_open reads
117, and no label can outnumber the set it is a subset of. The panel had
been asserting the opposite since it was written.
mara on the PR: "many hint texts are incorrect, most hives dont work on
hyperhive itself". Four panel descriptions asserted local practice as if
it were general:
Releases "expected to sit at zero on a hive that ships from
main" -- a claim about how THIS project releases
Automation surface "every hive registering the same webhook has bitten
this forge before" -- a war story from this
deployment, meaningless on any other
Organizations "near-constant by design" -- assumes our org layout
Issues open/closed "usually a bulk triage, not a productive afternoon"
-- a claim about our working rhythm
Each now describes what the metric IS and what a CHANGE means, and says
outright where the panel cannot know: whether zero releases is normal
depends on how a hive's projects ship, and a vertical step in closed is
some batch operation that this panel cannot distinguish from a burst of
real work.
The board title is NOT an instance of this and is unchanged -- all seven
dashboards use the 'hyperhive - <thing>' product namespace, checked
rather than assumed.
The forge restarted (new forgejo PID at 12:36), so it re-read [metrics]
and gitea_issues_by_label now exists -- 24 series carrying real label
names, read back out of the store rather than inferred from the metric
name appearing in a label-values list.
The panel shipped deliberately empty with EMPTY UNTIL THE FORGE RESTARTS
in its title and a matching noValue, because an empty graph and an
undeployed metric look identical. That warning is now false, and a stale
warning is worse than none: it tells a reader the panel cannot have data
at the moment it does.
Keeps the operational half rather than deleting it -- the restart
requirement is a fact about the NEXT metric someone enables, so it moves
into the description as a note and into noValue, where it fires only when
the series really is missing.
Adds the fourth per-service board. Unlike its three siblings it answers
"what does the forge CONTAIN", not "is the forge healthy" -- forgejo's
exporter publishes a census of stored objects and no request, error or
latency series at all.
That distinction is the board's own description rather than something a
reader has to infer, because a fourth per-service board that silently
answers a different question than the other three is worse than no board:
someone checks it during an incident and reads a flat line as calm.
Panels: repositories, users, orgs, issues open/closed, releases, the
open-vs-closed pair over time, repos-and-users together (they move in
lockstep when an agent is created, so divergence is the signal),
discussion volume, and the webhook/hooktask pair.
One panel ships deliberately EMPTY: issues-by-label. The metric is
enabled in the forge's config but forgejo reads its [metrics] section at
startup, so the series does not exist until that process restarts -- and
a config merge alone does not restart it. The panel title says so, the
description says how to tell "not deployed" from "no labelled issues",
and `noValue` says it again in the graph itself. An empty panel with no
explanation is the failure this board is trying not to be.
Gated: every panel's metric confirmed to exist in the store, with a
bogus name checked absent in the same run so the check can fail, plus a
known-positive control against the label-values query itself -- a
download that silently returned nothing would otherwise report every
metric missing. Gate kept as verify-3494-forge-board.sh.
The comment-block lint (added in 79dc8ca6) now trips on this block —
genuinely pre-existing, unrelated to that change, just newly caught.
Per the lint's own suggested remedy: relocated the full per-directive
reasoning plus both footguns (session-cache keying, the Host-header
clobber that can recurse a subrequest into itself) to a new
"Dialing another vhost by name" section in docs/gateway.md, and left
a short why + pointer comment in the source. No behavior change.
verifiedProxyTo (43ae164d) verified TLS but left Host to nixpkgs'
recommendedProxySettings, which sets Host to the CALLING vhost, not
the target. Since every consumer resolves back to this same gateway,
nginx picks the vhost to answer by Host header (not by the SNI
proxy_ssl_name already sends) — so every auth subrequest looped back
into its own vhost's auth_request, recursing until nginx's subrequest
depth limit turned it into a 500, on every domain gated by SSO.
Pin Host (and the rest of the header set nixpkgs' recommended include
would otherwise still be the one to set) inside verifiedProxyTo, and
set recommendedProxySettings = false on each of the four call sites so
nixpkgs' own copy — appended after a location's extraConfig — can't
clobber it back.
One list, in nix/reserved-names.nix, handed to everything that needs it
as HIVE_RESERVED_NAMES. Keeping it current becomes a config change
rather than a rebuild, and hive names and agent names -- one namespace
going forward -- are checked against the same file: swarm-otel.nix's
hand-written reservedOwners is gone.
Whitespace-separated rather than JSON, deliberately, unlike the
structured env vars beside it. Every entry is an Ident ([a-z0-9-]), so
whitespace cannot occur inside a name and the encoding is provably
lossless; JSON would mean either a parser dependency in a crate whose
purpose is to have none, or a copy of the parse in every consumer.
An UNSET variable is not "nothing is reserved". Both creation sites log
an error and return a warning saying the check did not run, so a
misconfigured deployment says so instead of silently accepting every
name. A blank value folds into unset: nix always renders a non-empty
list, so present-but-empty is a rendering fault, not a declaration.
Two guards whose subject moved out of their own file now assert their
own case is still in it, because a guard that can be retired by an edit
elsewhere is not a guard:
- swarm-otel.nix asserts reserved-names.nix still contains its
swarmTierName.
- hive-sh4re's sentinel drift test PANICS when the variable is missing
rather than skipping -- a drift test that quietly does nothing still
reports green. checks.nix and devshell.nix both export it so CI and a
local cargo test agree. Verified as a pair: with the variable set, 8
tests pass; with it unset, exactly the 4 drift tests fail and the
unrelated ones still pass.
Consumers reached authelia at `127.0.0.1:<port>`, which encoded a
co-location nobody agreed to: the gateway and authelia are not required
to share a host, so the literal is a requirement stated only by being
unwriteable any other way. Moving them to the name is the point of the
issue.
But a name over https is only half of "https and auth". nginx's
`proxy_ssl_verify` is OFF by default and there was no `proxy_ssl_*`
anywhere in the tree, so the obvious repoint would have produced an
encrypted, unauthenticated hop -- which works, and keeps working,
against any certificate at all.
Adds `gateway.lib.verifiedProxyTo <name>` next to the rest of the vhost
kit, so the convention has one definition rather than a copy in each
consuming module, and repoints the four call sites through it.
Each directive was checked against a real nginx with the opposite arm
run as a control:
- the CA *bundle* (root + intermediate) is accepted -- worth checking,
since `hive-ca-trust.nix` warns off consumers that read only one
certificate, and nginx is not one of those
- verification checks the chain: an unrelated CA fails
- and the HOSTNAME: a wrong `proxy_ssl_name` fails even with a good
chain. Chain-only would accept any cert this CA ever signed, which
for an internal CA is every service on the hive
- with verify off, the wrong CA passes -- so the failures above come
from verification, not from the connection
Bind addresses are untouched. This changes what consumers dial, not what
anything listens on.
The upstream gitea-actions-runner unit uses DynamicUser, and systemd
mounts a dynamic unit's state directory noexec. Measured from a job's
own /proc/self/mountinfo on the live runner, with /tmp as the control:
the state dir carries noexec, /tmp does not, and a script written and
chmod +x'd in the workspace fails execve with EACCES while the same
script in /tmp runs.
This was invisible for as long as every workflow compiled inside the
nix sandbox and executed nothing from the workspace. The first job that
built a binary into the runner's own target dir -- an instrumented
coverage run -- died on its first build script.
ExecPaths= re-mounts the state dir executable. Both spellings are
listed with the ignore-if-absent prefix because ExecPaths resolves
against the host root, where the dynamic-user layout makes
/var/lib/gitea-runner a symlink into private/.
/etc/gitconfig shipped `helper = git-credential-hive-forge`. git prepends
`git-credential-` to any helper value that is not an absolute path, so
that resolves to `git-credential-git-credential-hive-forge`, which does
not exist -- no helper runs at all. The sibling github.nix has always used
the short form.
Measured rather than read off the docs, with the arms isolated from the
personal ~/.gitconfig:
helper = git-credential-hive-forge -> 0 credentials, and git prints
"'credential-git-credential-hive-forge' is not a git command"
helper = hive-forge -> 1 credential, clean stderr
The reason this survived: every long-lived agent has a personal
~/.gitconfig naming the helper by ABSOLUTE path, which git accepts, so
pushes keep working and the stderr line reads as noise. The system config
is masked exactly where someone would notice it and bites a fresh agent
that has no such file.
The authelia vhost intercepts upstream errors and serves a friendly
"SSO unavailable" page. The `=` form of `error_page` takes its status
from the redirected location, and that location serves a file -- so the
page is returned as **200**.
That is right for a human typing the URL and wrong for every machine
caller, all of which reach authelia through this same vhost by name:
- `/api/authz/auth-request` -- nginx `auth_request` treats any 2xx as
success, so a down authelia means access GRANTED
- `/api/oidc/introspection` -- a token check answering 200
- `/api/oidc/token`, `/.well-known/openid-configuration` -- clients
parsing an HTML error page as their JSON document
Routes `/api/` and `/.well-known/` without the interception. A longer
prefix wins over `/`, and the intercept directives live inside the `/`
location rather than at server level, so they do not reach the new ones.
Split by AUDIENCE rather than by an enumerated path list: a human still
gets the page, and every machine caller -- including the login page's own
XHR, and any endpoint added later -- gets the real status.
Measured against a real nginx with a dead upstream, both arms: machine
paths return 502 where they returned 200+HTML, a subrequest through the
new prefix denies (matching a direct port dial) where through `/` it
served the protected content, and the browser control confirms the
friendly page survives. URI preservation checked separately against a
live echo upstream -- `proxy_pass` with no URI part passes the full
original path.
Adds `cargo llvm-cov` to the devshell and a coverage workflow that runs
only when someone asks for it.
Manual dispatch rather than nightly or per-PR, per the discussion on the
issue: a coverage run needs its own instrumented build and cannot reuse
the normal test artifacts, so it roughly doubles a test job. "Do we have
glaring holes" is a question someone asks occasionally, not a gate every
PR pays for, and not a number worth spending farm time on every night
whether or not anyone reads it. Manual dispatch costs nothing until the
answer is wanted.
Its own workflow file rather than a job in ci.yml: that file already has
a `workflow_dispatch` trigger so `hive-forge ci-rerun` can retrigger
without an empty commit, and a trigger there fires EVERY job in the
file -- a coverage job added there would run on every pull request.
No threshold and no --fail-under-lines. A coverage gate mostly teaches
people to write assertion-free tests that execute lines; the report is
the deliverable and the number is for a human to read.
The devshell needs LLVM_COV / LLVM_PROFDATA set explicitly: cargo
llvm-cov expects rustup's `llvm-tools-preview` beside the toolchain and
nixpkgs has no such component, so without them it aborts with "failed to
find llvm-tools-preview" -- which reads like a missing install rather
than a path the shell has to name.
Verified by running it: `cargo llvm-cov --package hive-types` produces a
real report (3 tests, 85.44% regions). Both the variable names and the
package were wrong on the first attempt and only running it said so --
the names take no `_PATH` suffix, and the binaries are in
`llvmPackages.llvm`, not `llvmPackages.bintools`, which is the linker
wrapper and ships neither.
Host metrics arrived carrying `hive` and no host attribute, so "which
host is out of memory" was answerable only as "which hive" -- true only
while a hive is one machine, which is the framing the swarm work exists
to end.
Adds the collector's own `resourcedetection` processor to the metrics
pipeline, emitting both `host.name` and `host.id`.
`resourcedetection` rather than a value picked in nix: this is the
canonical OTel mechanism for host identity, and choosing our own would
answer a question the tool already answers, differently from every other
deployment. A resource attribute is captured into the stored series, so
a private naming scheme is expensive in the way that lasts.
Both attributes, deliberately: `host.name` is readable and what a panel
groups by, but splits the series silently on a rename with nothing
linking old to new; `host.id` survives that and is unreadable alone.
Hosts get renamed and repurposed exactly when services move off one box.
The processor sits in the pipeline, so it applies to every receiver --
agent-pushed OTLP and scraped targets included, not just `hostmetrics`.
Keys verified against otelcol-contrib 0.151.0 with two deliberate
mutations rejected in the same run (a bogus resource attribute and a
bogus detector key), so "accepted" is distinguishable from a validator
that ignores what it does not recognise.
Turns on gitea_issues_by_label and gitea_issues_by_repository, gated on
behindGateway exactly like the metrics endpoint itself -- they are only
reachable through the authenticated location that condition creates.
These are off by default upstream because they are the only metrics in
this section whose series count grows with the CONTENT of the forge
rather than with its feature set, so the question is cardinality.
Measured rather than assumed: ~19 distinct labels and ~29 repos on this
instance, well under 100 series, against a store that took +2234 from
nine host scrapers.
Still no TOKEN. Forgejo can guard the endpoint with its own static
bearer, but the swarm authenticates the scraper at the gateway, and a
second per-service credential would be the one that stops getting
rotated -- the reasoning already recorded above ENABLED applies
unchanged to the breakdowns.
The token file is 0600 root-owned and the precondition ran as the unit's
dynamic gitea-runner user, so it could not read it -- and it did not fail
closed. `[ -s ]` succeeds on a stat alone, grep then died with EACCES,
and `! grep` turned that error into true, so the gate exited 0 on a file
it had never read. It has reported 'a real token is present' without
looking for its entire existence, which is why the runner kept retrying
instead of holding still and saying why.
Prefix the step with + so it runs with full privileges. This grants the
runner nothing new: tokenFile becomes EnvironmentFile=, which systemd
already reads as root before dropping privileges, so the payload never
passes through an unprivileged reader either way -- the + only lets the
gate observe what systemd observes.
Also assert readability separately and loudly, rather than folding it
into the placeholder test: an unreadable file and a real token produced
the same verdict, so 'I could not look' was indistinguishable from 'I
looked and it is fine'.
Two more per-service boards, cut from the withdrawn combined services
board: the log store (4 panels) and the metrics store (5).
The combined board was one page carrying panels for four different
services; the seams were already there, so this is a re-cut rather than
new authoring. Each board's subject is one service, and the services
page that remains will carry only panels whose subject is the *set*.
Both were imported by the operator before being committed here, so the
queries are known to draw against the live stores rather than only
parsing.
Nothing in the tree rendered a `[credential "<forge>"]` entry for an
agent, so every agent's `~/.gitconfig` accumulated one by hand, per
generation of forge address. Append-only, none ever removed, and after
the domain move the live one absent entirely:
[credential "http://forge.<old-hive>"]
[credential "http://localhost:3000"]
The absence of a writer is the defect. A value interpolated at eval time
follows a rename; a value captured into a mutable home file does not.
hive-c0re's own gitconfig already derives its scope from
`swarm.forge.domain` and moved correctly for exactly that reason.
What made it expensive to diagnose is that it does not present as a
credential problem. `git fetch` against a stale remote still succeeds --
the old name redirects and a public read needs no auth -- so the break
surfaces only at the first authenticated push, long after the move, as
`could not read Username for '<new host>'`. That names a host the agent
was never configured for, which reads like DNS or TLS.
Same class as the CI runner keeping its registered address, one tier
down.
The shape is `github.nix`'s, unchanged: a small credential helper that
reads the token from the agent's state file at invocation, with the
token PATH baked in rather than the value, because claude's Bash tool
runs in a minimal env that never sources /etc/set-environment.
`environment.etc."gitconfig"` is already bound by github.nix; the two
merge rather than collide because the option is `lines`. Verified by
eval with both modules defining it before this was written -- a silent
last-wins there would drop one integration's credentials and look
exactly like this bug again.
The pending-bytes panel described itself as the leading indicator for
the slow-consumer stat 'beside it', and then sat alone in its own row
with eighteen grid units of empty space. A description that tells the
reader to look beside it, where nothing is, is worse than no
description.
Slow consumers and pending bytes are now adjacent in the top row, and
the two JetStream panels share the second. No gaps.
Split out of the combined swarm-services board, which mara declined as
multiple dashboards in one trench coat. Its 17 panels partition exactly
by subject -- 3 services / 5 metrics store / 5 queue / 4 log store, with
no panel belonging to two -- so this is a re-cut of verified panels, not
new authoring.
Seven panels: connections, subscriptions, slow consumers and JetStream
API errors as stats, messages over the range, plus pending bytes (the
leading indicator for a slow consumer) and JetStream store size against
its configured limit (the only capacity question the queue has).
Connections is red at zero, unlike every other count here: three things
hold a connection at all times, so an empty count means the queue is
refusing everyone -- and a callout denial reaches a client as a timeout
rather than an error, which makes this often the first visible sign.
They differ by one letter and a grep for the shorter name matches both.
The summary is what ships; the per-process scraper is deliberately absent,
and a reader (human or grep) cannot tell from the diff alone.
Nothing measured the host itself. hive-c0re emits per-container metrics
and each service exports its own, so the store could answer 'is this
container using more memory than last week' and could not answer 'is the
machine under them out of memory' -- the question every one of those
readings is implicitly relative to.
Hive tier rather than swarm: a host is owned by the hive running on it,
and the swarm tier runs one collector for the whole swarm, so it would
have to reach other machines to see theirs.
Every scraper the pinned receiver offers except process, which adds a
series set per running process -- unbounded on a hive host, where the
nine shipped are a fixed handful. Coverage and cardinality are different
axes and only the second can hurt the store.
Refs #3649
The collector's client is registered for authelia.bearer.authz, but the
prometheus scrape asked for no scopes, so every token came back carrying
none and authelia refused it at introspection with 'the requested scope
is invalid, unknown, or malformed'. Forgejo metrics were never scraped.
The rule was already stated one field below, for the audience: registered
is not requested. The two travel together, and a config read cannot see
the one that is missing -- every inspection of the rendered config came
back correct, because what was present was correct.
NODE_EXTRA_CA_CERTS beside it is Node's and additive. gitea-runner is Go,
whose trust store is replacing: it reads SSL_CERT_FILE and trusts only
what that names. So the container trusted the hive CA for its Node
actions and never for the daemon itself.
The daemon then fails startup with x509 unknown authority the moment it
reaches a TLS endpoint, crash-loops on Restart=on-failure, and no job is
picked up -- every pull request just sits at Waiting to run.
Not claimed as the whole cause of that outage: which address the daemon
holds is a separate question. The trust gap is real under every
explanation, and adding a CA to one unit cannot break a working path.
