The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.
The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.
An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
The `_` vhost was serving the hive's own surface, so every dashboard and
agent-UI request matched the default server rather than a named vhost --
and so did a request for any name at all, including a raw IP.
Split it: `_` keeps only `return 444`, and the hive surface moves to a
vhost named for the hive domain. `_` is `mkDefault` so an operator can
claim default_server themselves, plus an assertion for the case where
they add one without turning ours off -- nginx refuses to start on a
duplicate default_server and nixpkgs asserts nothing, so that would
otherwise surface as a gateway outage at rebuild time.
Last of the four. The vhost, its `auth_request` block and the swarm
apex's dns record move into swarm-ui.nix; vhosts.nix drops `uiCfg`,
`controllerCfg` and `autheliaCfg` and is now 259 lines of hive surface
with no swarm service in it.
Also collapses a THIRD copy of the per-service list. `networking.hosts`
restated every service's name with its own copy of that service's guard,
after the vhosts and the dnsmasq records had each done the same. It asks
the same question — which names does this host answer for — so it now
reads the same answer: a service added later lands in /etc/hosts with no
edit, and cannot land there under a different condition than it used for
DNS.
The `forceSSL`-not-`addSSL` comment travels intact: it records that
authelia answers an http auth subrequest with 400 and nginx's
auth_request only understands 2xx/401/403, so the scheme is load-bearing
for this vhost and no other.
Moves the matrix sub-domain vhost out of the gateway's vhosts.nix, its
`address=` rule out of dnsmasq.nix, and the Accept-header
`$matrix_spa_target` map out of the gateway's appendHttpConfig — all
three into hive-matrix.nix.
The map is the one that had no business being where it was: it exists
solely for the SPA fallback in the vhost's `/` location, and
`appendHttpConfig` is a `lines` option, so a module can contribute to
it without the gateway assembling it.
The `.well-known/matrix/*` delegation deliberately stays on the hive's
own vhost. The spec requires it at the SERVER NAME, which is the hive
domain: that is the hive answering "where is my homeserver", not the
homeserver answering for itself. Moving it would have been the obvious
symmetric thing and it would have been wrong.
Moves the authelia vhost out of the gateway's vhosts.nix and its
`address=` rule out of dnsmasq.nix, into swarm-authelia.nix.
Both land inside that module's existing `cfg.enable` guard, which is
the load-bearing part: every hive in a swarm knows `authelia.url`, but
only the host that RUNS the container may claim the name. A client hive
declaring the vhost would answer for a service it does not run, and
publishing the DNS record would point every agent on its bridge at that
wrong answer.
The kit grows a fourth member, `errorPages`, because the vhost aims its
502/503/504 at the gateway's styled sso-unavailable page. Republished
rather than imported per module: a service rendering its own would drift
from the rest of the gateway the first time the theme changed.
Moves `forgeVhost` out of the gateway's vhosts.nix and the forge's
`address=` rule out of dnsmasq.nix, into nix/host-modules/hive-forge —
the module that already owns everything else about the forge.
The gateway keeps what is gateway knowledge (the listen set, which
issuer covers a name, the header block) and loses the last reason it
had to read `swarm.forge` at all: `forgeCfg` is gone from both files
and from the module's `let`.
Both halves stay gated on `behindGateway` — with it off the operator
fronts forgejo themselves, so this hive must neither claim the vhost nor
answer DNS for the name.
Adds `services.hyperhive.gateway.localNames` (internal): hostnames the
hive resolver answers with the bridge IP, contributed by the modules
that own them. The service says which name, the gateway says where it
points — the same split `lib.tlsFor` already makes.
No behaviour change yet: the list is empty until the service modules
contribute in the following commits.
The assertion is not defensive padding. Duplicate `address=` rules do
not make dnsmasq complain; it resolves them by precedence, so a name
claimed twice silently stops being served by one of its claimants. That
failure mode only becomes reachable because contribution is now open, so
it gets closed in the same commit that opens it.
Slice 1 of #3202. The listen set, the per-name TLS attrs and the
security headers move out of vhosts.nix into ./vhost-lib.nix and are
published as `services.hyperhive.gateway.lib` (internal, readOnly).
No behaviour change: vhosts.nix consumes the published value, so the
rendered vhost tree is identical.
The point is the next slice. Today a swarm service's vhost lives in
the gateway because only the gateway knows the port pair, the issuer
for a name, and the header block. Publishing those three is what lets
a service module declare its own vhost without the gateway having to
know that service by name.
Rust half mirrors hive-c0re/src/dashboard/mod.rs's utoipa pattern
exactly: an ApiDoc root, #[utoipa::path(...)] on /health (the one
existing route), and a raw JSON route at /api/openapi.json served
via OpenApiRouter::split_for_parts(). Only annotated routes appear
in the spec.
Gateway wiring extends the swarm-UI vhost (the only vhost swarm-
controller is reachable from) with:
- /api/ — proxied to the controller's unix socket untouched (no URI
segment after the socket path), so a route swarm-controller
registers is the path nginx forwards, no prefix-stripping to keep
in sync by hand.
- /api/docs/ (+ the bare /api/docs redirect) — the same
swagger-ui-theme dist the per-hive dashboard already serves at its
own /api/docs/, reused as-is since it's generic.
Both new locations reuse the same auth_request block the vhost's own
'/' already applies, factored into a shared swarmAuthRequest string —
auth_request does not inherit across sibling nginx locations, so
without this the page itself would be gated while its own API and
API docs sat open.
cargo test -p swarm-controller + cargo clippy --all-targets both
clean. Verified the new nginx wiring evaluates correctly with a
throwaway nixosSystem eval (services.hyperhive.swarm.{controller,ui}
enabled): /api/ proxies to the socket, /api/docs redirects, and both
require auth_request the same as the vhost root.
Fixes hyperhive#3212
Per mara: a general fix, not one name in one container.
Every container inherits a COPY of the host's /etc/resolv.conf at start
(nixos-containers.nix: cp --remove-destination, one shot, not a
bind-mount), so the address written there is the address every container
tries - in its own netns. That makes the value load-bearing:
value host host-netns containers bridged containers
127.0.0.1 ok ok THEIR OWN loopback
bridge IP ok ok ok
dnsmasq binds lo and the bridge, so the bridge IP works for the host
too. It is the only value correct on both sides of a netns boundary.
resolveLocalQueries goes ON for its plumbing, not its address: it points
dnsmasq's own upstreams at a separate resolv-file, without which
dnsmasq reads /etc/resolv.conf and every non-hive query loops the moment
the host is pointed at dnsmasq. Its two loopback-publishing effects
(networking.nameservers and resolvconf.useLocalResolver) are overridden.
Cost: the host's DNS now depends on dnsmasq being up. Every container
already did.
The forge container keeps its hosts entry from the previous commit -
not redundancy, a fallback in a different failure domain: it works with
no DNS at all, so SSO does not ride on a host-wide resolver change.
Four wiring sites, same as forge/matrix/authelia: vhost, cert name, DNS,
local-dev hosts. This commit is the last two.
The apex is a SIBLING of forge./chat./auth., not a child of anything
dnsmasq already answers for, so the /<hive domain>/ rule does not cover
it - without an explicit record the name is unresolvable from every
agent netns while working fine from the host, which is the shape that
nearly shipped when authelia's name moved.
Published to agents deliberately (mara: publishing it is fine).
Reachability is not the access control: the vhost's auth_request plus
authelia's group:operators rule are, and an agent that resolves the name
still cannot open the page.
Serves the static bundle on the swarm apex and gates it with
auth_request - the first one in this gateway, everything else being
auth_basic + htpasswd.
Header set measured against the pinned authelia (4.39.20) rather than
copied from an example: X-Original-URL and X-Original-Method are present
as literals and are what the auth-request implementation reads, while
X-Forwarded-Uri does not appear in that binary at all - sending it would
look like configuration and be dead weight. The endpoint is
/api/authz/auth-request; /api/verify is the legacy path older examples
show.
auth_request_set captures the return URL BEFORE the error_page jump: in
the 401 handler $request_uri is the internal one, so building the link
there sends the operator back to the auth subrequest rather than the
page they asked for.
Authorisation is the access_control rule from the previous commit, not
this subrequest: auth_request answers 'is there a session'.
Per mara on the PR: how a thing used to be deployed stops being
relevant the moment this is merged and the hives are rebuilt, so the
comments state the current constraint and nothing about the container
that used to hold it.
Removes the two ⚠️ blocks this PR added (hive-tls-ca's ordering, the
resign propagation), the matrix ordering's, dnsmasq's resolv.conf
paragraph, and the pre-existing 'REMOVED WITH THE CONTAINER' inventory
in the gateway module. The 🚨 do-not-simplify warning on the cert-copy
unit keeps both its reasons and loses the container framing.
Kept deliberately: the two operator-facing 'this used to default to
true' option notes (migration information for someone upgrading), and
the SupplementaryGroups block, which documents why an absence is
load-bearing rather than how a deployment used to look.
Prose-only sweep of every remaining claim that nginx or dnsmasq lives
in a container: the port comment (root in a container -> root on the
host), upstreamHost's netns rationale, the ACME state dir, the store
path reachability note, the vhost tree header, dnsmasq's resolv.conf
paragraph (there is no copy and no path unit watching it any more),
the two hive-network bridge comments, and swarm-controller's socket
access-control note, which described a bind-mount that no longer
exists.
No behaviour change; all of it was describing a mechanism that was
deleted.
`agents.conf` and `gateway.htpasswd` move from /var/lib/hyperhive/gateway
to /var/lib/hive-gateway/conf, alongside the `tls/` the gateway already
kept there.
nginx reads both as an unprivileged user. Under c0re's state dir it could
only reach them by traversing a directory systemd re-declares `0750
hive-core` on every c0re start — so nginx was given `SupplementaryGroups
= [ "hive-core" ]`, which also handed it read access to everything else
group-readable in that tree. The tokens are individually 0600, but the
broker sqlite carries no explicit mode: every message between every agent
was readable by the process whose job is parsing untrusted network input.
Moving the files removes the need and the exposure together. The group is
gone, and its absence is now commented as load-bearing so it doesn't come
back as a fix for a symptom it would recreate.
Also drops this module's `/var/lib/hyperhive` tmpfiles rule. It declared
`0755 root root` and could never win against `StateDirectoryMode`, and a
losing declaration still reads as a guarantee — that is what sent the
first diagnosis of the outage looking for who had changed the mode.
Ordering is unchanged and still the thing that makes a fresh boot work:
tmpfiles runs before services and seeds both files empty-but-valid, nginx
names them (an `include` of a missing file is fatal, not empty), and
content arrives when c0re writes and reloads — which it does on every
topology change, so a boot against the empty seed resolves itself.
Folds in the mode fix: `write` now sets 0644 on the tmp file before the
rename, because a rename carries the source's mode and discards the
destination's, and the tmpfiles rule that declares 0644 is
create-if-absent so it never re-applies.
authelia has listened on 127.0.0.1:9091 since it was stood up, with
nothing proxying to it — so `auth.<swarm.domain>` resolved and then
refused the connection. This is the vhost that was never written.
Follows forge and matrix exactly: one `optionalAttrs` attrset merged into
`virtualHosts`, TLS chosen by `vhostTlsFor` (the swarm-services leaf
already names it, since `swarm.serviceDomains` includes
`authelia.domain`), and the same four wiring sites those two occupy —
vhost, dnsmasq address, local-dev `/etc/hosts`, and the arg lists that
feed both files.
Gated on this host running the container, not on authelia being
configured: every hive knows the swarm's `authelia.url`, but only the one
serving it may claim the name. A client hive declaring this vhost would
answer for a service it does not run.
Two things that are deliberate rather than incidental:
`X-Forwarded-{Proto,Host,Uri,For}` are set because authelia decides by
the *original* request — the login redirect and the session cookie's
domain both derive from them. Without them every request looks like it
arrived at 127.0.0.1 over plain http.
And no `auth_basic`. Applying the gateway's basic-auth block to the SSO
provider would put the login page behind the login mechanism it exists to
replace.
Per review: docs represent current state. Every "used to" / "no longer"
clause this branch introduced is gone — including the History section in
network.md, which was a whole subsection about a sync mechanism that
doesn't exist.
Where the removed clause was carrying a real constraint, the constraint
stays and is stated in the present tense instead of as a delta: nothing
narrows what the gateway's nginx can reach except the directory
permissions in front of a socket, and nothing bounds `ReloadGatewayNginx`
except the hard-coded unit name. Those read as rules now rather than as
the story of how they came to be rules.
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.
The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.
Four things made it more than a deletion, none of them visible in the nix
diff:
- The self-signed cert service also imports the hive CA leaf, so removing it
with the container would have left nginx naming a missing cert file, which
it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
for the reason its doc gave, and `--machine=` was both transport and
scope — so the unit name is now hard-coded in the helper as the
containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.
Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.
Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.
Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
main is already unborked -- the constraint-syntax fix landed directly in
31e47632 and the hive-tls-ca ordering race in b5066ad0. What is left is
the reason the syntax bug became an OUTAGE rather than a warning, and it
is independent of both.
nginx refuses to load a config naming a cert file that does not exist:
`cannot load certificate ... no such file` fails the pre-start test, so
the vhost does not degrade -- the ENTIRE proxy dies and restart-loops
into start-limit-hit, taking the forge, the dashboard and matrix with
it. Signing tolerated the leaf's absence and the container import
tolerated it; the vhost reference did not, and that is the one that
decides whether anything serves.
Absence stays a normal state here: the leaf only exists where the swarm
CA is autoconfigured. So the import now installs the HIVE leaf at the
services path instead of removing the file. That is a name mismatch on
those vhosts -- browsers warn, strict clients refuse -- and every other
name keeps working. A bad certificate is a bad afternoon; a dead gateway
is a dead hive.
Also keeps the comment explaining why every nameConstraints entry needs
its own `permitted;` prefix, so the next reader doesn't re-derive it
from an outage.
nginx already carried TLS per vhost, so this is a cert choice rather
than a restructure: a vhost whose name this hive's CA cannot sign
gets the swarm-services pair, and every other vhost keeps the hive
pair.
Which vhosts those are is not decided here. Both the sub-CA's name
constraints and this choice read swarm.serviceDomains, so "a swarm
service" means one thing in one place -- a vhost served a cert its
issuer is constrained out of would fail at TLS, and the two lists
drifting is the only way to get there.
Only in self-signed mode. With ACME or an operator cert there is a
single issuer that already covers every name, and a second pair would
be a cert nobody asked for.
The container import copies the pair only when the host issued one,
and removes a stale copy otherwise: the leaf exists only where the
swarm CA is autoconfigured, so absent is a normal state and a
leftover from a host that stopped issuing it is not.
Second slice of the swarm-global service consolidation, same shape as
the forge move: the operator-facing host options become
services.hyperhive.swarm.matrix.*, and one mkRenamedOptionModule on the
namespace carries the whole subtree (nested gui.* included), so there is
no leaf list to forget an entry from.
The rename lives in hive-matrix.nix, the module that declares the
options, so each service's migration stays independent of its siblings.
The per-agent hyperhive.matrix.{enable,url} and hyperhive.matrixAccounts
are a different namespace -- a client pointer at the service, not the
service -- and deliberately do not move.
Forgejo is a swarm-global service, so its operator-facing host options
move to services.hyperhive.swarm.forge (and .swarm.forge.ci) as the
first of the namespace consolidation.
Existing hive configs keep evaluating: swarm-renames.nix maps every
moved leaf with mkRenamedOptionModule, which also emits a deprecation
warning naming both the old and new path, so an operator is told what to
rename rather than discovering it from a failed eval.
The per-agent hyperhive.forge.url does NOT move. It is a client pointer
at whatever forge an agent talks to - it shares a word with the service
and nothing else, and the two are already documented as separate option
surfaces.
Verified by evaluating the host module, since no Rust gate evaluates
nix: setting the old paths and reading the new ones yields the values
(httpPort 3999, ci.concurrency 7), and config.warnings carries the
rename notice.
/run/hive-agent/<name> was 0777 root root in steady state, not just during
first spawn. A directory without the sticky bit lets any user unlink files
in it, and the gateway container has all of /run/hive-agent bind-mounted
in, so anything that could reach the path could delete an agent's
agent.sock, bind its own, and receive that agent's todos from hive-c0re.
Two mechanisms were writing the dir and undoing each other: the tmpfiles.d
entry wrote 0777 root root, then hive-c0re round-tripped through hive-priv's
ChownSocketDir to narrow it. `d` re-asserts mode and owner on every apply
and the file is regenerated on any agent's spawn or destroy, so every such
event reset every agent's dir back to world-writable.
SyncAgentTmpfiles now carries each agent's container uid/gid and the entry
declares the answer: 0751 <uid> <gid>. Three principals need the dir and no
two share a group -- the harness binds its sockets (owner rwx), hive-c0re
dials agent.sock and the gateway's nginx dials web.sock (both only need
traverse, and both sockets are already 0666).
Deletes ChownSocketDir and ChmodSocketDir, both priv_client wrappers, the
either/or in host_config with its two swallowed warn!s, and the now-dead
socket_dir_path -- two verbs off the privileged helper's surface and one
round-trip off every agent spawn.
Also makes the two tmpfiles rules for /run/hive-agent itself agree: the
gateway module said hive-core, the generated file said root, and which won
depended on the order systemd read them in.
services.hyperhive.c0re.swaggerUiTheme -> services.hyperhive.gateway.swaggerUiTheme.
hive-c0re has no relationship to this option (it's consumed entirely
by the gateway's nginx config) since the previous two commits moved
the theme files out of hive-c0re/ and gateway to hosting the whole
dist itself — the option shouldn't live in c0re's namespace either.
Pure option move: default wiring in flake.nix now sets
services.hyperhive.gateway.swaggerUiTheme instead of the c0re one;
hive-gateway/default.nix reads it via the module's own already-bound
cfg (services.hyperhive.gateway) instead of reaching into c0re's.
Verified: nix build .#swagger-ui-theme unaffected (same store path,
pure option-plumbing change, no content change); a full nixosSystem
eval of nixosModules.default resolves the option at its new path and
confirms the gateway's /api/docs/ location still aliases the correct
store path.
Extends the theme-only alias into the full shape mara asked for on the
PR thread:
1. nix/packages/swagger-ui-dist.nix — plain vendored Swagger UI 5.17.14
dist, sourced directly from the swagger-ui-dist npm package (same
release the Rust utoipa-swagger-ui-vendored crate ships, verified
via matching gitHead commit) rather than through Cargo.lock/cargo.
2. nix/packages/swagger-ui-theme.nix — overlays our 3 override files
(index.html, hyperhive-theme.css, and now swagger-initializer.js)
onto (1).
3. vhosts.nix's swaggerUiLocations now prefix-matches the whole
/api/docs/ tree (not just 2 exact-match files) straight from (2),
plus a `= /api/docs` redirect shim since hive-c0re's own redirect
is going away too. /api/openapi.json (outside this prefix) keeps
proxying to c0re unchanged — that's the one thing that stays
dynamic.
New file swagger-initializer.js needed hand-verification: the plain
vendored copy hardcodes the swagger.io petstore demo URL.
utoipa-swagger-ui normally rewrites it per-request from a {{config}}
placeholder its own build.rs injects — since hive-c0re won't be
serving this file at all once its SwaggerUi mount is removed, that
rewrite has to be baked in statically here instead. Derived by
actually running build.rs's own two transforms (strip the default
layout: line, splice the Config JSON in place of the url/deepLinking
block) against the real vendored file, not typed from scratch —
verified byte-for-byte against what format_config() would produce for
hive-c0re's actual single-URL config, and checked with node --check.
Coordinated with damocles: he's taking the corresponding hive-c0re
side (drop the utoipa-swagger-ui dependency + SwaggerUi::new(...)
mount, keep only the plain /api/openapi.json route) once this lands.
Verified: nix fmt clean; nix build .#swagger-ui-theme succeeds, output
byte-matches the checked-in override files and node --check passes on
swagger-initializer.js; a full nixosSystem eval of nixosModules.default
resolves both new locations (/api/docs/ aliased to the right store
path, = /api/docs redirecting) with auth threaded through.
The gateway container's /etc/resolv.conf is a one-shot copy: nixos-container
cps it in from the host in its start script, and nspawn's --resolv-conf=auto
copies (not binds) for a writable host-netns container. systemd-nspawn(1)
states the consequence outright — "no further propagation of configuration is
generally done after the one-time early initialization (this is because the
file is usually updated through copying and renaming)".
dnsmasq has no explicit upstream and follows that file, so a host network
change strands it on a resolver that no longer answers and every non-hive
lookup from every agent hangs. Agents' own resolvers point at the static
bridge IP and never go stale, which is why the symptom presents as "the
gateway needs a kick".
Add a host-side hive-gateway-resolv path unit watching /etc/resolv.conf.
On change it machinectl copy-to's the file into the container and reloads
dnsmasq — ExecReload is kill -HUP, so upstreams are re-read and the cache
flushed without dropping anything; nginx never notices.
- watched from the HOST: a rename on the host doesn't cross the nspawn mount
namespace, so an in-container path unit can't see it (same reason c0re
reloads nginx from the host side)
- copy, not a file bind-mount: openresolv renames over the file, so a bind
would pin the first inode forever — strictly worse than today
- machinectl copy-to writes through the container's own mount namespace, so
this holds regardless of how the container assembles /etc
- armed Before=network-pre.target so the boot's first DHCP write is caught,
and re-run on gateway start for changes made while it was down
- a host file with no nameserver line is skipped, not pushed, so a
mid-rewrite snapshot can't blank hive DNS
- deliberately no fallback server=: dnsmasq queries all known upstreams in
parallel, so a hardcoded public resolver would take a share of normal
traffic rather than only covering the gap
