The runner's prefetch 401/403 diagnostic baked a literal issue tag into
its runtime journal output. Per the hive-wide rule (no issue/PR #NNN tags
in code), strip it — the message already explains the stale-core-token
condition and the operator remedy in full prose, so nothing is lost.
Two bugs that together kept the runner registration token from
refreshing (#1475):
1. Unit name: the prefetch's before/wantedBy/partOf targeted
nixos-container@hive-ci.service, but a declarative containers.<n> is
the host unit container@<n>.service (confirmed against the live
container@hive-matrix.service during the #1465 incident). The wrong
name made all three silent no-ops, so the partOf never bound — the
RemainAfterExit oneshot stayed 'active (exited)' and never re-ran on
nixos-container restart, leaving the stale token in place. Corrected
to container@hive-ci.service.
2. 401-hardening: the registration-token fetch used a bare curl -sf | jq,
so a forge-core-token that is stale/invalid for the current forge
(e.g. after a forge rebuild) 401s and fails silently every attempt for
the full 60s loop, then exits with a misleading 'core token absent or
forge unreachable'. Now capture the HTTP status and fail fast + loudly
on 401/403 with a clear message pointing at re-minting the core token.
The host systemd unit for a declarative nixos-container is
container@<name>.service (confirmed from the live
container@hive-matrix.service status), not nixos-container@. The prior
commit's nixos-container@ ordering targeted a non-existent unit (a
silent no-op). Use container@hive-{matrix,gateway}.service so the
after-ordering actually takes effect.
Per operator review on the PR: when the network module is on, the
matrix container's resolver is the dnsmasq in the gateway container, so
order the matrix container start after the gateway container. This is
robustness for tuwunel's lazy federation lookups, not a boot
requirement — the boot fix is the resolv.conf nameserver line (the
failure was a parse error on an empty resolv.conf, not connectivity).
Soft 'after' (not 'requires') keeps lifecycles decoupled; network.enable
asserts gateway.enable so the gateway container unit always exists.
tuwunel hard-fails to start when /etc/resolv.conf has no nameserver
line (Failed to configure DNS resolver: no nameservers found in
config -> exit 1 -> systemd start-limit). The declarative
containers.hive-matrix generates its own resolv.conf via resolvconf
and, unlike agent containers whose resolv.conf is written by
hive-c0re's lifecycle, has no nameserver source -> it comes up empty
(just 'options edns0'). Defaulting network.enable on surfaced this:
the host DNS moved to the bridge dnsmasq but the container was never
pointed at it, so the homeserver could not boot, taking down matrix
for all agents.
Point the container at the hive resolver (the dnsmasq the network
module runs at bridgeIp) when the network module is enabled; the
container always shares the host netns (privateNetwork = false) so it
reaches bridgeIp whether or not isolateContainers is set. With the
network module off, inherit the host resolv.conf.
The matrix server-server spec resolves a portless m.server delegated
host to the federation default port 8448 (not 443 — the URL convention
that https implies 443 does not apply to the m.server field). The
gateway reused publicPortSuffix, which intentionally drops :443 for the
client base_url, so .well-known/matrix/server advertised the bare
gateway host. Peers then federated to <gatewayHost>:8448, which nothing
listens on, while the federation endpoint actually lives on the gateway
443 vhost (/_matrix/ proxied to tuwunel). Always emit the explicit port
in the federation delegation so peers reach the live 443 endpoint.
The hive-ci-prefetch oneshot has RemainAfterExit=true and is wired to the
container only via wantedBy + before. Once it runs successfully it stays
'active (exited)' indefinitely, so systemd skips it on subsequent container
restarts. The runner-token file it wrote is never refreshed.
This breaks the runner after its first registration: the token written on
the first successful boot is either a placeholder (forge-core-token wasn't
ready yet) or a registration token that has since been consumed/rotated.
On the next container restart prefetch does not re-run, the stale token
persists, and the in-container register service fails with
'invalid_argument: runner registration token not found' — exactly the
symptom in the field (worked briefly, two orphan runners registered, then
permanently offline).
Add partOf = [ nixos-container@hive-ci.service ] so a container stop/restart
propagates to the prefetch unit, forcing it to re-run and fetch a fresh
registration token before the container comes back up. before= still orders
it ahead of the container start within the same transaction.
WriteAgentForgeToken and WriteAgentMatrixToken write agent state files
under /var/lib/hyperhive/agents/<name>/state/. The hive-priv service
has ProtectSystem=strict which makes all of /var read-only by default,
but /var/lib/hyperhive was missing from ReadWritePaths, causing EROFS
on every token write.
Closes#1274.
Addresses mara's review: add a NixOS assertion that rejects
hsts.enable = true when no TLS mode is configured. HSTS over
plain HTTP is silently ignored by browsers; the assertion turns
a silent misconfiguration into a build-time error.
HSTS was unconditionally tied to hasTls. This is risky: enabling it on a
deployment that later loses TLS locks browsers out until max-age expires.
Add three options under services.hyperhive.gateway.hsts:
enable — bool, default false
maxAge — seconds, default 31536000 (1 year)
includeSubDomains — bool, default true
HSTS header is now only emitted when hsts.enable = true.
X-Frame-Options, X-Content-Type-Options, Referrer-Policy at server
scope on _, forge, and matrix vhosts. HSTS added when TLS is active.
nginx inheritance rule: locations with their own add_header (CORS API
endpoints like /.well-known/matrix/client, /_matrix/) are unaffected —
they already carry the headers they need. HTML-serving and proxy
locations pick the security headers up automatically.
On first boot (or after a wipe) hive-c0re writes forge-core-token only
after the forge container starts and the admin is provisioned. This
lags hive-c0re.service becoming active. The previous code bailed
immediately with TOKEN=placeholder if the token file was absent,
causing the runner to fail registration with 'token not found'.
Fix: merge both waits (core-token file appearance + forge API ready)
into a single 60s retry loop. The early-bail path is removed; the
script only exits cleanly if .runner is valid (writes placeholder)
or a fresh registration token is obtained.
When .runner exists but core-token is absent after 60s, we keep the
existing credentials (safe — the runner holds valid creds; next boot
will validate properly).
Closes#1224 (which tracks #1221).
The gitea-actions-runner's host-scheme job processes use the
service's environment PATH, not the NixOS login-shell PATH.
Without this, 'nix flake check' and similar steps fail with
'nix: command not found'.
- add pkgs.nix to environment.systemPackages (ensures the binary
is in /run/current-system/sw/bin)
- set explicit PATH on gitea-runner-hive service covering both
/run/current-system/sw/bin and /nix/var/nix/profiles/default/bin
With ProtectSystem=strict (added in the previous commit), / is read-only
inside hive-priv. nix creates a temp result symlink in its cwd; without
WorkingDirectory that cwd is / (systemd default), causing:
error: creating symlink /.tmp.tmp-...: Read-only file system
Point WorkingDirectory at the StateDirectory (/var/lib/hive-priv) which is
always writable under strict. Nix drops its temp symlink there instead.
add services.hyperhive.gateway.tls.certDir option: operators with a
CA-signed cert (Let's Encrypt, corporate CA) point at the ACME output
dir instead of using the auto-generated self-signed cert.
- tls.certDir: host path bind-mounted r/o at /run/hive-tls/ in gateway
- tls.certName / tls.keyName: filenames within certDir (default: cert.pem / key.pem, matches nixpkgs security.acme layout)
- hasTls = selfSignedTls || certDir != null: publicScheme=https in both cases
- assertion: selfSignedTls=true + certDir set together is an error
- openFirewall: httpsPort opened in both TLS modes
- docs/gateway.md: TLS modes table + operator-cert section
- docs updated in swarm.md peer config reference in the cert TLS section
when using operator cert, swarm peers can omit certFingerprint —
standard CA bundle handles trust automatically.
isolated agents reach forge via http://forge.<domain> → nginx.
without the gateway there is nothing on port 80 to serve that
hostname. assert early rather than fail silently at runtime.
addresses argus yellow note on PR #1150.
when isolateContainers=true, isolated agents have dnsmasq as their
resolver — forge.<domain> resolves to bridgeIp. route HIVE_FORGE_URL
through nginx on port 80 instead of exposing the raw forge port.
- HIVE_FORGE_URL: http://<forge.domain> when isolated (nginx proxies)
- bridge firewall: open 80+443 for agents to reach nginx (gateway)
- remove forge-specific httpPort rule (no longer needed)
- update docs/gateway.md + docs/network.md
per mara's review comment on PR #1150.
When containers run in private netns (isolateContainers=true), host
loopback is unreachable so HIVE_FORGE_URL=http://127.0.0.1:3000 breaks.
- nix/modules/hive-network.nix: when isolateContainers is on + forge
is enabled, open forge.httpPort on the bridge interface so agents
can reach forgejo at bridgeIp:httpPort (forgejo binds 0.0.0.0)
- nix/modules/hive-c0re.nix: HIVE_FORGE_URL switches to bridge IP
when network.enable && isolateContainers; loopback path retained
when isolateContainers=false
- docs/network.md: add Forge access + Forge URL rows to effects table
- docs/gateway.md: rewrite HIVE_FORGE_URL section for both modes
Add opt-in WireGuard mesh support to services.hyperhive.swarm:
- swarm.peers.<domain>.wireguardPublicKey — peer's wg public key
- swarm.peers.<domain>.wireguardEndpoint — peer's UDP endpoint (optional)
- swarm.peers.<domain>.wireguardAddress — peer's mesh IP with prefix
- swarm.wireguard.enable — bring up wg-hive interface
- swarm.wireguard.privateKeyFile — path to host's wg private key
- swarm.wireguard.address — this host's mesh IP/prefix
- swarm.wireguard.listenPort — UDP listen port (default 51820)
- swarm.wireguard.persistentKeepalive — keepalive seconds (default 25)
When enabled, generates networking.wireguard.interfaces.wg-hive with
one peer entry per mesh-enabled swarm.peers entry. Opens listenPort
UDP on the host firewall. Adds wireguard_address to HYPERHIVE_PEERS
JSON so hive-c0re can use mesh IPs for intra-swarm routing.
Assertions guard against enable=true without privateKeyFile or address.
Also refactors networking.firewall.allowedTCPPortRanges from the
nested attrset form (which conflicted with the new allowedUDPPorts
line) to the per-attribute form.
docs/swarm.md: adds WireGuard setup section with key generation
commands, two-hive config example, NAT/keepalive notes.
Two argus review fixups:
1. Move isolateContainers assertion to an unconditional `lib.mkIf
cfg.isolateContainers` arm. The prior placement inside
`mkIf (enable && isolateContainers)` was dead code — the assertion
could never fire because both flags were already true by the time the
block activated. Now `isolateContainers=true; enable=false;` raises
a NixOS assertion error at eval time.
2. Add comment on HIVE_NETWORK_SUBNET noting it carries the host-side
bridge IP (e.g. "10.42.0.1/24"), not the canonical network address
("10.42.0.0/24"). Rust consumer must normalize via bitwise AND before
subnet membership checks or address arithmetic.
Adds `services.hyperhive.network.isolateContainers` (bool, default
false). When enabled alongside `network.enable`, activates:
- IP forwarding + NAT masquerade so isolated agents reach the internet
- nftables DROP rule blocking bridge-subnet → loopback (defence-in-depth
against compromised agent reaching the c0re dashboard)
- `HIVE_NETWORK_ISOLATION`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET`
injected into the hive-c0re service env; the Rust lifecycle reads these
to set `PRIVATE_NETWORK`, `LOCAL_ADDRESS`, and `HOST_BRIDGE` in each
agent container's conf
Config block rewritten as `lib.mkMerge [...]` — the prior `lib.mkIf //
lib.mkIf` pattern was invalid nix (mkIf returns a tagged value, not an
attrset; // on it is a type error). See docs/network.md for full design.
Privsep splits repo ownership: hive-priv (root) fetches the hive-core-owned
meta/applied repos via nix, and hive-c0re (hive-core) fetches the agent-owned
proposed-config repos. git/libgit2's dubious-ownership guard rejects both
('repository path is not owned by current user'), failing every rebuild.
Install a root-trusted gitconfig with safe.directory=* in each service's
HOME; both already have read access — this only satisfies the guard.
Two privsep follow-ups, both in the hive-c0re service:
1. The migration 'chown -R hive-core /var/lib/hyperhive' (which only
started running once ExecStartPre stopped failing 203/EXEC) stomped every
agent's bind-mounted creds — agents/<name>/{claude,state,harness,config}
are owned by the per-agent/manager users — logging all agents out with no
way back in. Scope it to everything *except* agents/, plus the agents/
dir node itself so c0re can still create new per-agent subdirs. Each
container's hive-agent-user-migrate activation chowns the contents back.
2. nix (prebuild 'nix build', flake-check, meta eval in c0re; nixos-container
update->nix in priv) writes its cache under $HOME/.cache. Both services
run as users with no home -> HOME=/var/empty (unwritable) -> Lix cache
init fails, rebuilds error out. Set HOME to each service's StateDirectory
(adding one for hive-priv).
hive-priv had no PATH, so the nixos-container it runs for every container
op (incl. lifecycle::list) failed ENOENT. build_all swallows that into an
empty list — the 'no managed containers' symptom. Give the helper the
minimal set nixos-container shells out to (nixos-container, nix, util-linux,
e2fsprogs) on top of the systemd/coreutils/findutils already in the unit
PATH, rather than the whole system profile.
Also route forge/matrix is_present() through priv_client::list_containers
instead of spawning nixos-container directly from unprivileged hive-c0re.
The state-migration chown invoked ${pkgs.coreutils}/bin/sh, which does not
exist (coreutils ships chown, not sh), so ExecStartPre exited 203/EXEC on
every boot. hive-c0re hit its start limit and never came up, so the gateway
returned 502 Bad Gateway. Call chown directly with the +- prefix (run as
root, tolerate failure) instead of going through a shell for '|| true'.
Tuwunel 1.6.1 appends " 💕" to every new user's display name via the
new_user_displayname_suffix config option. Set it to empty string so
agent and operator matrix accounts get clean names.
Existing users (atlas, mara, root) need a one-time display name reset.
Atlas's was already corrected via the client API.
hive-ci-register.service now runs unconditionally on every boot (not
just when .runner is absent). Before fetching a registration token it
validates existing .runner credentials via the forge admin API:
- 200: runner still registered, write dummy token and exit
- 404: runner deleted from forge, purge .runner and re-register
- 000: forge unreachable, keep credentials (runner surfaces the error)
- other non-200 or malformed .runner: purge and re-register
Removes ConditionPathExists so stale credentials from a wiped forge
no longer block the runner indefinitely. Updates docs/ci.md to match.
Remove the custom htpasswdFile option and bind-mount. The htpasswd file
now lives at the fixed path /var/lib/hyperhive/gateway/gateway.htpasswd
on the host, which is already exposed inside the container at
/run/hive-state/gateway.htpasswd via the existing gateway state
bind-mount — no extra bind-mount needed.
A tmpfiles rule pre-creates the file so nginx can open it even before
any users exist (empty file → all requests return 401, which is correct).
hivectl gateway commands default --file to the standard path so
`hivectl gateway create-user alice` just works without any flags.
Per argus review: the hardcoded /etc/hyperhive/gateway.htpasswd
example was wrong for operators with a custom htpasswdFile path.
Move the unauthorized.html from the static agentErrorPagesDir derivation
into a pkgs.writeText inside the lib.optionalAttrs guard where
cfg.auth.htpasswdFile is in scope and statically known non-null.
The rendered page now shows the operator's actual configured path.
