Rust half mirrors hive-c0re/src/dashboard/mod.rs's utoipa pattern
exactly: an ApiDoc root, #[utoipa::path(...)] on /health (the one
existing route), and a raw JSON route at /api/openapi.json served
via OpenApiRouter::split_for_parts(). Only annotated routes appear
in the spec.
Gateway wiring extends the swarm-UI vhost (the only vhost swarm-
controller is reachable from) with:
- /api/ — proxied to the controller's unix socket untouched (no URI
segment after the socket path), so a route swarm-controller
registers is the path nginx forwards, no prefix-stripping to keep
in sync by hand.
- /api/docs/ (+ the bare /api/docs redirect) — the same
swagger-ui-theme dist the per-hive dashboard already serves at its
own /api/docs/, reused as-is since it's generic.
Both new locations reuse the same auth_request block the vhost's own
'/' already applies, factored into a shared swarmAuthRequest string —
auth_request does not inherit across sibling nginx locations, so
without this the page itself would be gated while its own API and
API docs sat open.
cargo test -p swarm-controller + cargo clippy --all-targets both
clean. Verified the new nginx wiring evaluates correctly with a
throwaway nixosSystem eval (services.hyperhive.swarm.{controller,ui}
enabled): /api/ proxies to the socket, /api/docs redirects, and both
require auth_request the same as the vhost root.
Fixes hyperhive#3212
Per mara: a general fix, not one name in one container.
Every container inherits a COPY of the host's /etc/resolv.conf at start
(nixos-containers.nix: cp --remove-destination, one shot, not a
bind-mount), so the address written there is the address every container
tries - in its own netns. That makes the value load-bearing:
value host host-netns containers bridged containers
127.0.0.1 ok ok THEIR OWN loopback
bridge IP ok ok ok
dnsmasq binds lo and the bridge, so the bridge IP works for the host
too. It is the only value correct on both sides of a netns boundary.
resolveLocalQueries goes ON for its plumbing, not its address: it points
dnsmasq's own upstreams at a separate resolv-file, without which
dnsmasq reads /etc/resolv.conf and every non-hive query loops the moment
the host is pointed at dnsmasq. Its two loopback-publishing effects
(networking.nameservers and resolvconf.useLocalResolver) are overridden.
Cost: the host's DNS now depends on dnsmasq being up. Every container
already did.
The forge container keeps its hosts entry from the previous commit -
not redundancy, a fallback in a different failure domain: it works with
no DNS at all, so SSO does not ride on a host-wide resolver change.
authelia refuses to authorize a non-https target: the auth subrequest for
`X-Original-URL: http://…` answers 400, and nginx's auth_request only
understands 2xx/401/403, so a plain-http visit died as "auth request
unexpected status: 400" — a bare 400 in the browser with no hint that a
login exists.
The shared vhostListen binds :80 as well as :443 and every vhost used
addSSL, so the door was open on a port the lock cannot work on. That is
harmless for forge and matrix, where http is merely insecure rather than
broken, so the asymmetry stays local to the one vhost whose correctness
depends on the scheme.
removeAttrs because nixos asserts on a vhost declaring both addSSL and
forceSSL.
Measured against the pinned binary rather than reasoned about:
X-Original-URL: http://constellation…/ -> 400 Bad Request
X-Original-URL: https://constellation…/ -> 401 + Location: auth…?rd=…
Four wiring sites, same as forge/matrix/authelia: vhost, cert name, DNS,
local-dev hosts. This commit is the last two.
The apex is a SIBLING of forge./chat./auth., not a child of anything
dnsmasq already answers for, so the /<hive domain>/ rule does not cover
it - without an explicit record the name is unresolvable from every
agent netns while working fine from the host, which is the shape that
nearly shipped when authelia's name moved.
Published to agents deliberately (mara: publishing it is fine).
Reachability is not the access control: the vhost's auth_request plus
authelia's group:operators rule are, and an agent that resolves the name
still cannot open the page.
Serves the static bundle on the swarm apex and gates it with
auth_request - the first one in this gateway, everything else being
auth_basic + htpasswd.
Header set measured against the pinned authelia (4.39.20) rather than
copied from an example: X-Original-URL and X-Original-Method are present
as literals and are what the auth-request implementation reads, while
X-Forwarded-Uri does not appear in that binary at all - sending it would
look like configuration and be dead weight. The endpoint is
/api/authz/auth-request; /api/verify is the legacy path older examples
show.
auth_request_set captures the return URL BEFORE the error_page jump: in
the 401 handler $request_uri is the internal one, so building the link
there sends the operator back to the auth subrequest rather than the
page they asked for.
Authorisation is the access_control rule from the previous commit, not
this subrequest: auth_request answers 'is there a session'.
Per mara on the PR: how a thing used to be deployed stops being
relevant the moment this is merged and the hives are rebuilt, so the
comments state the current constraint and nothing about the container
that used to hold it.
Removes the two ⚠️ blocks this PR added (hive-tls-ca's ordering, the
resign propagation), the matrix ordering's, dnsmasq's resolv.conf
paragraph, and the pre-existing 'REMOVED WITH THE CONTAINER' inventory
in the gateway module. The 🚨 do-not-simplify warning on the cert-copy
unit keeps both its reasons and loses the container framing.
Kept deliberately: the two operator-facing 'this used to default to
true' option notes (migration information for someone upgrading), and
the SupplementaryGroups block, which documents why an absence is
load-bearing rather than how a deployment used to look.
Prose-only sweep of every remaining claim that nginx or dnsmasq lives
in a container: the port comment (root in a container -> root on the
host), upstreamHost's netns rationale, the ACME state dir, the store
path reachability note, the vhost tree header, dnsmasq's resolv.conf
paragraph (there is no copy and no path unit watching it any more),
the two hive-network bridge comments, and swarm-controller's socket
access-control note, which described a bind-mount that no longer
exists.
No behaviour change; all of it was describing a mechanism that was
deleted.
`agents.conf` and `gateway.htpasswd` move from /var/lib/hyperhive/gateway
to /var/lib/hive-gateway/conf, alongside the `tls/` the gateway already
kept there.
nginx reads both as an unprivileged user. Under c0re's state dir it could
only reach them by traversing a directory systemd re-declares `0750
hive-core` on every c0re start — so nginx was given `SupplementaryGroups
= [ "hive-core" ]`, which also handed it read access to everything else
group-readable in that tree. The tokens are individually 0600, but the
broker sqlite carries no explicit mode: every message between every agent
was readable by the process whose job is parsing untrusted network input.
Moving the files removes the need and the exposure together. The group is
gone, and its absence is now commented as load-bearing so it doesn't come
back as a fix for a symptom it would recreate.
Also drops this module's `/var/lib/hyperhive` tmpfiles rule. It declared
`0755 root root` and could never win against `StateDirectoryMode`, and a
losing declaration still reads as a guarantee — that is what sent the
first diagnosis of the outage looking for who had changed the mode.
Ordering is unchanged and still the thing that makes a fresh boot work:
tmpfiles runs before services and seeds both files empty-but-valid, nginx
names them (an `include` of a missing file is fatal, not empty), and
content arrives when c0re writes and reloads — which it does on every
topology change, so a boot against the empty seed resolves itself.
Folds in the mode fix: `write` now sets 0644 on the tmp file before the
rename, because a rename carries the source's mode and discards the
destination's, and the tmpfiles rule that declares 0644 is
create-if-absent so it never re-applies.
Two claims this branch falsified and left standing, both caught in
review:
`vhosts.nix`'s `errorPages` param comment enumerated the set
(`{ notFound, unreachable, unauthorized }`) and adding a fourth member
made the enumeration wrong at the point a reader consults it.
`gateway.md` said extending custom error pages beyond the per-agent
routes was "a separate follow-up" — while this branch is that follow-up,
so the doc contradicted the code sitting next to it.
Rewrites the scope rule as the criterion rather than a list, since a
list is what went stale: a route earns a page when the default status
would point at the wrong component. That covers the per-agent routes and
the sso vhost, and explains why forge/matrix/fluffychat still don't
qualify — their upstreams being down means what the code says.
A dead authelia upstream almost always means "no users yet" — authelia
treats an empty user store as a fatal startup error, so an enabled but
unbootstrapped swarm crash-loops behind a vhost that is working
perfectly. nginx's default 502 says the opposite: it points at the
proxy, which is the one component that is fine.
Adds `ssoUnavailable` to the shared error-page set and wires it on the
authelia vhost the same way the per-agent blocks wire
`__hive_agent_unreachable`: `proxy_intercept_errors on` plus an internal
location serving the static page.
The page leads with the bootstrap command rather than burying it under
an explanation, and names the container journal as the fallback for the
cases where users are not the problem. Same Catppuccin template as its
siblings, so this costs no new styling.
authelia has listened on 127.0.0.1:9091 since it was stood up, with
nothing proxying to it — so `auth.<swarm.domain>` resolved and then
refused the connection. This is the vhost that was never written.
Follows forge and matrix exactly: one `optionalAttrs` attrset merged into
`virtualHosts`, TLS chosen by `vhostTlsFor` (the swarm-services leaf
already names it, since `swarm.serviceDomains` includes
`authelia.domain`), and the same four wiring sites those two occupy —
vhost, dnsmasq address, local-dev `/etc/hosts`, and the arg lists that
feed both files.
Gated on this host running the container, not on authelia being
configured: every hive knows the swarm's `authelia.url`, but only the one
serving it may claim the name. A client hive declaring this vhost would
answer for a service it does not run.
Two things that are deliberate rather than incidental:
`X-Forwarded-{Proto,Host,Uri,For}` are set because authelia decides by
the *original* request — the login redirect and the session cookie's
domain both derive from them. Without them every request looks like it
arrived at 127.0.0.1 over plain http.
And no `auth_basic`. Applying the gateway's basic-auth block to the SSO
provider would put the login page behind the login mechanism it exists to
replace.
localHostsEntry's 127.0.0.1 rows leak into the address= answers dnsmasq
gives agent containers now that it runs on the host, sending every
agent back to itself instead of the bridge IP.
Per review: docs represent current state. Every "used to" / "no longer"
clause this branch introduced is gone — including the History section in
network.md, which was a whole subsection about a sync mechanism that
doesn't exist.
Where the removed clause was carrying a real constraint, the constraint
stays and is stated in the present tense instead of as a delta: nothing
narrows what the gateway's nginx can reach except the directory
permissions in front of a socket, and nothing bounds `ReloadGatewayNginx`
except the hard-coded unit name. Those read as rules now rather than as
the story of how they came to be rules.
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.
The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.
Four things made it more than a deletion, none of them visible in the nix
diff:
- The self-signed cert service also imports the hive CA leaf, so removing it
with the container would have left nginx naming a missing cert file, which
it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
for the reason its doc gave, and `--machine=` was both transport and
scope — so the unit name is now hard-coded in the helper as the
containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.
Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.
Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.
Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
main is already unborked -- the constraint-syntax fix landed directly in
31e47632 and the hive-tls-ca ordering race in b5066ad0. What is left is
the reason the syntax bug became an OUTAGE rather than a warning, and it
is independent of both.
nginx refuses to load a config naming a cert file that does not exist:
`cannot load certificate ... no such file` fails the pre-start test, so
the vhost does not degrade -- the ENTIRE proxy dies and restart-loops
into start-limit-hit, taking the forge, the dashboard and matrix with
it. Signing tolerated the leaf's absence and the container import
tolerated it; the vhost reference did not, and that is the one that
decides whether anything serves.
Absence stays a normal state here: the leaf only exists where the swarm
CA is autoconfigured. So the import now installs the HIVE leaf at the
services path instead of removing the file. That is a name mismatch on
those vhosts -- browsers warn, strict clients refuse -- and every other
name keeps working. A bad certificate is a bad afternoon; a dead gateway
is a dead hive.
Also keeps the comment explaining why every nameConstraints entry needs
its own `permitted;` prefix, so the next reader doesn't re-derive it
from an outage.
nginx already carried TLS per vhost, so this is a cert choice rather
than a restructure: a vhost whose name this hive's CA cannot sign
gets the swarm-services pair, and every other vhost keeps the hive
pair.
Which vhosts those are is not decided here. Both the sub-CA's name
constraints and this choice read swarm.serviceDomains, so "a swarm
service" means one thing in one place -- a vhost served a cert its
issuer is constrained out of would fail at TLS, and the two lists
drifting is the only way to get there.
Only in self-signed mode. With ACME or an operator cert there is a
single issuer that already covers every name, and a second pair would
be a cert nobody asked for.
The container import copies the pair only when the host issued one,
and removes a stale copy otherwise: the leaf exists only where the
swarm CA is autoconfigured, so absent is a normal state and a
leftover from a host that stopped issuing it is not.
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
Second slice of the swarm-global service consolidation, same shape as
the forge move: the operator-facing host options become
services.hyperhive.swarm.matrix.*, and one mkRenamedOptionModule on the
namespace carries the whole subtree (nested gui.* included), so there is
no leaf list to forget an entry from.
The rename lives in hive-matrix.nix, the module that declares the
options, so each service's migration stays independent of its siblings.
The per-agent hyperhive.matrix.{enable,url} and hyperhive.matrixAccounts
are a different namespace -- a client pointer at the service, not the
service -- and deliberately do not move.
Forgejo is a swarm-global service, so its operator-facing host options
move to services.hyperhive.swarm.forge (and .swarm.forge.ci) as the
first of the namespace consolidation.
Existing hive configs keep evaluating: swarm-renames.nix maps every
moved leaf with mkRenamedOptionModule, which also emits a deprecation
warning naming both the old and new path, so an operator is told what to
rename rather than discovering it from a failed eval.
The per-agent hyperhive.forge.url does NOT move. It is a client pointer
at whatever forge an agent talks to - it shares a word with the service
and nothing else, and the two are already documented as separate option
surfaces.
Verified by evaluating the host module, since no Rust gate evaluates
nix: setting the old paths and reading the new ones yields the values
(httpPort 3999, ci.concurrency 7), and config.warnings carries the
rename notice.
/run/hive-agent/<name> was 0777 root root in steady state, not just during
first spawn. A directory without the sticky bit lets any user unlink files
in it, and the gateway container has all of /run/hive-agent bind-mounted
in, so anything that could reach the path could delete an agent's
agent.sock, bind its own, and receive that agent's todos from hive-c0re.
Two mechanisms were writing the dir and undoing each other: the tmpfiles.d
entry wrote 0777 root root, then hive-c0re round-tripped through hive-priv's
ChownSocketDir to narrow it. `d` re-asserts mode and owner on every apply
and the file is regenerated on any agent's spawn or destroy, so every such
event reset every agent's dir back to world-writable.
SyncAgentTmpfiles now carries each agent's container uid/gid and the entry
declares the answer: 0751 <uid> <gid>. Three principals need the dir and no
two share a group -- the harness binds its sockets (owner rwx), hive-c0re
dials agent.sock and the gateway's nginx dials web.sock (both only need
traverse, and both sockets are already 0666).
Deletes ChownSocketDir and ChmodSocketDir, both priv_client wrappers, the
either/or in host_config with its two swallowed warn!s, and the now-dead
socket_dir_path -- two verbs off the privileged helper's surface and one
round-trip off every agent spawn.
Also makes the two tmpfiles rules for /run/hive-agent itself agree: the
gateway module said hive-core, the generated file said root, and which won
depended on the order systemd read them in.
services.hyperhive.c0re.swaggerUiTheme -> services.hyperhive.gateway.swaggerUiTheme.
hive-c0re has no relationship to this option (it's consumed entirely
by the gateway's nginx config) since the previous two commits moved
the theme files out of hive-c0re/ and gateway to hosting the whole
dist itself — the option shouldn't live in c0re's namespace either.
Pure option move: default wiring in flake.nix now sets
services.hyperhive.gateway.swaggerUiTheme instead of the c0re one;
hive-gateway/default.nix reads it via the module's own already-bound
cfg (services.hyperhive.gateway) instead of reaching into c0re's.
Verified: nix build .#swagger-ui-theme unaffected (same store path,
pure option-plumbing change, no content change); a full nixosSystem
eval of nixosModules.default resolves the option at its new path and
confirms the gateway's /api/docs/ location still aliases the correct
store path.
Extends the theme-only alias into the full shape mara asked for on the
PR thread:
1. nix/packages/swagger-ui-dist.nix — plain vendored Swagger UI 5.17.14
dist, sourced directly from the swagger-ui-dist npm package (same
release the Rust utoipa-swagger-ui-vendored crate ships, verified
via matching gitHead commit) rather than through Cargo.lock/cargo.
2. nix/packages/swagger-ui-theme.nix — overlays our 3 override files
(index.html, hyperhive-theme.css, and now swagger-initializer.js)
onto (1).
3. vhosts.nix's swaggerUiLocations now prefix-matches the whole
/api/docs/ tree (not just 2 exact-match files) straight from (2),
plus a `= /api/docs` redirect shim since hive-c0re's own redirect
is going away too. /api/openapi.json (outside this prefix) keeps
proxying to c0re unchanged — that's the one thing that stays
dynamic.
New file swagger-initializer.js needed hand-verification: the plain
vendored copy hardcodes the swagger.io petstore demo URL.
utoipa-swagger-ui normally rewrites it per-request from a {{config}}
placeholder its own build.rs injects — since hive-c0re won't be
serving this file at all once its SwaggerUi mount is removed, that
rewrite has to be baked in statically here instead. Derived by
actually running build.rs's own two transforms (strip the default
layout: line, splice the Config JSON in place of the url/deepLinking
block) against the real vendored file, not typed from scratch —
verified byte-for-byte against what format_config() would produce for
hive-c0re's actual single-URL config, and checked with node --check.
Coordinated with damocles: he's taking the corresponding hive-c0re
side (drop the utoipa-swagger-ui dependency + SwaggerUi::new(...)
mount, keep only the plain /api/openapi.json route) once this lands.
Verified: nix fmt clean; nix build .#swagger-ui-theme succeeds, output
byte-matches the checked-in override files and node --check passes on
swagger-initializer.js; a full nixosSystem eval of nixosModules.default
resolves both new locations (/api/docs/ aliased to the right store
path, = /api/docs redirecting) with auth threaded through.
The gateway container's /etc/resolv.conf is a one-shot copy: nixos-container
cps it in from the host in its start script, and nspawn's --resolv-conf=auto
copies (not binds) for a writable host-netns container. systemd-nspawn(1)
states the consequence outright — "no further propagation of configuration is
generally done after the one-time early initialization (this is because the
file is usually updated through copying and renaming)".
dnsmasq has no explicit upstream and follows that file, so a host network
change strands it on a resolver that no longer answers and every non-hive
lookup from every agent hangs. Agents' own resolvers point at the static
bridge IP and never go stale, which is why the symptom presents as "the
gateway needs a kick".
Add a host-side hive-gateway-resolv path unit watching /etc/resolv.conf.
On change it machinectl copy-to's the file into the container and reloads
dnsmasq — ExecReload is kill -HUP, so upstreams are re-read and the cache
flushed without dropping anything; nginx never notices.
- watched from the HOST: a rename on the host doesn't cross the nspawn mount
namespace, so an in-container path unit can't see it (same reason c0re
reloads nginx from the host side)
- copy, not a file bind-mount: openresolv renames over the file, so a bind
would pin the first inode forever — strictly worse than today
- machinectl copy-to writes through the container's own mount namespace, so
this holds regardless of how the container assembles /etc
- armed Before=network-pre.target so the boot's first DHCP write is caught,
and re-run on gateway start for changes made while it was down
- a host file with no nameserver line is skipped, not pushed, so a
mid-rewrite snapshot can't blank hive DNS
- deliberately no fallback server=: dnsmasq queries all known upstreams in
parallel, so a hardcoded public resolver would take a share of normal
traffic rather than only covering the gap
