Agents in private netns can't reach host loopback directly; they
use forge.<domain> and matrix.<domain> via the gateway. Update four
locations that still claimed direct loopback access:
- hive-forge.nix: module header + openFirewall description
- hive-matrix.nix: openFirewall description
- docs/matrix.md: container design bullet + openFirewall section
Agents in private netns reach the forge at forge.<domain> via the
gateway, not via localhost directly. The conclusion (firewall only
matters for external access) is correct; the mechanism description
was stale from the pre-isolation era.
Per mara: the mirror seeding belongs in hive-c0re's forge provisioning
sweep, where the core admin token + org-ensure already live — not a parallel
host-side nix oneshot.
- forge.rs: ensure_mirrors() reads HYPERHIVE_FORGE_MIRRORS (JSON list of
{upstream,dest}), ensures each dest org (reuse ensure_org) + creates the
pull-mirror via the migrate API (reuse forge_http, serde_json::json! body,
409/existing = success). Called in ensure_all() right after the SEEDED_ORGS
loop (token in scope, warn-and-continue like the other ensure_* steps).
- hive-forge.nix: forward effectiveMirrors to c0re via
systemd.services.hive-c0re.environment.HYPERHIVE_FORGE_MIRRORS; drop the
forgejo-seed-mirrors.service + its script + the host-side core-token read.
Keep the forge.mirrors option, DEFAULT_ACTIONS_URL=self (CI-gated), and the
dest-shape / no-c0re-namespace-collision assertions.
Verified locally: nix parse + treefmt (incl rustfmt) clean; serde/serde_json
patterns mirror dashboard.rs. cargo build runs in CI (no cc in my container).
nixfmt wanted the ${...} on a single line (verified locally:
nix build .#checks.x86_64-linux.formatting passes). The earlier CI 'docs'
failure was a transient muede-pc2 build hiccup — the docs check builds clean
locally on the same drv.
- drop the three cross-ref tracker tags from comments/description (prose only)
- build the orgs + migrate JSON bodies with jq -n --arg (an upstream URL
containing a quote no longer corrupts the request)
- don't auto-append the actions/checkout mirror when the operator already
declared that dest (avoids a duplicate effectiveMirrors entry when CI is on)
General-purpose mirror mechanism for the internal forge, per mara's call on
#2074 (real Forgejo pull-mirrors, nix-configured — not a pushed clone).
- services.hyperhive.forge.mirrors: list of { upstream, dest } pull-mirrors,
any repo. Each is created as a real Forgejo pull-mirror (re-syncs from
upstream), dest = <owner>/<repo> in its own org.
- When forge.ci.enable is set: an actions/checkout mirror is auto-appended +
forgejo DEFAULT_ACTIONS_URL is pointed at this instance, so CI
'uses: actions/checkout@vN' resolves on loopback — immune to a host-resolver
blip that previously reded every checkout (the seed/re-sync needs external
DNS, but that's off the CI critical path).
- forgejo-seed-mirrors.service: host-side oneshot (the core admin token never
enters a container), modelled on hive-ci-prefetch — waits <=60s for the core
token, then idempotently ensures each dest org + creates the pull-mirror via
the migrate API. partOf the forge container so it re-ensures on restart.
- assertions: dest must be <owner>/<repo>; mirror orgs can't shadow the
c0re-managed namespaces (config/shared/agents/core) so the seed never races
hive-c0re's own provisioning.
Supersedes #2074 (the raw-clone stopgap) as the durable #2072 fix.
General-purpose Forgejo pull-mirror config (services.hyperhive.forge.mirrors:
list of {upstream, dest}). When CI is enabled, auto-append an actions/checkout
mirror + point forgejo DEFAULT_ACTIONS_URL at this instance so CI's
actions/checkout@vN resolves on loopback (immune to host-resolver blips, #2072).
Seed oneshot (creates the dest orgs + pull-mirrors via the migrate API) is the
next commit.
Follow-up to the harness fix: any unit whose ExecStart is a generated
wrapper (a NixOS `script =` block or a writeShellScript) gets its journal
SyslogIdentifier derived from the wrapper's store-path basename — an
opaque, unstable name. Pin an explicit stable identifier on every unit
hyperhive declares.
script/wrapper units (leaked store paths): forgejo-gpg-init,
hive-gateway-self-signed-cert, hive-tls-ca, hive-ci-prefetch,
hyperhive-isolated-dns, tea-login, forge-avatar-sync,
matrix-avatar-sync, hive-dashboard-links.
direct-binary units (basename already sane; explicit for tidiness +
future-proofing): hive-c0re, hive-priv, hive-matrix-daemon,
hive-bash-daemon, weston.
Only changes the displayed line identifier; journalctl -u <unit>
filtering (keys on _SYSTEMD_UNIT) is unaffected.
The internal forge is the canonical store for the meta flake, every
agent's config repo, and the internal/* repos, so it can no longer be
optional. Remove the services.hyperhive.forge.enable toggle:
- hive-forge.nix: drop the `enable` option; the forge config now
deploys gated on `services.hyperhive.enable` (it ships with hyperhive).
- hive-c0re.nix: HIVE_FORGE_URL env unconditional; forge-public-URL gate
drops the enable check (keeps behindGateway).
- hive-gateway.nix: local /etc/hosts forge entry keyed on behindGateway.
- hive-ci.nix: drop the now-moot `forge.ci.enable requires forge.enable`
assertion (forge is always present); reword the option doc.
- nix/docs/default.nix: drop the `forge.enable = mkForce false` stub
(option gone); the options-doc eval stays light via hyperhive.enable.
- hive-c0re forge.rs / hivectl.rs: reword 'forge.enable = true' error
text to 'wait for hive-c0re to start the container' (the runtime
token-absent path is unchanged — it's a bootstrap-timing check, not
the opt-out).
- docs/approvals.md, docs/ci.md: drop stale forge.enable references.
Migration: configs that set `services.hyperhive.forge.enable = false`
must drop the line — the forge is now mandatory.
Prereq/companion to #1838 (PR-based config flow, which assumes the forge
is always present).
Make self-signed TLS the gateway's default whenever no external TLS source
is configured, and deprecate the explicit selfSignedTls toggle. Self-signed
is now derived as `tls.certDir == null && !tls.acme.enable`, so an operator
selects a TLS mode by setting tls.certDir or tls.acme — or neither, for the
self-signed default. There is no http-only mode: matrix discovery hardcodes
https, so the gateway always terminates TLS.
The selfSignedTls option is kept as a deprecated no-op (warns when set to
false) so existing configs still eval. The two selfSignedTls mutual-
exclusion assertions and the HSTS-requires-TLS assertion are dropped — they
are impossible or vacuous now that self-signed is the floor. The hive-tls
module and the forge ROOT_URL scheme consume the derived value: the gateway
always terminates TLS, so behind the gateway the forge is always advertised
over https.
Updates docs/gateway.md (TLS-modes table, self-signed section, the removed
http-only section, firewall + discovery notes). Eval-proven: default →
self-signed (hive CA active, https ROOT_URL); tls.certDir → CA inactive;
selfSignedTls=false → deprecation warning fires.
Fixes#1724. The forgejo ROOT_URL was hardcoded `http://` even when the
gateway terminates TLS, producing broken clone links and mixed-content
redirects for a TLS deployment.
Derive the scheme + port from gateway state: `https` (on httpsPort) when
the gateway terminates TLS (`gateway.selfSignedTls` or `gateway.tls.certDir`
set), `http` (on gateway.port) otherwise — dropping the port suffix on the
canonical port for the scheme (80 / 443). The per-agent in-cluster
HIVE_FORGE_URL is unaffected (it stays http on the gateway's :80 path);
this only changes the public ROOT_URL forgejo advertises.
Refreshes the `rootUrl` option description (the TLS scheme is auto-derived
now, so the manual-override-for-TLS note is gone).
Eval-proven: gateway.selfSignedTls = true → ROOT_URL =
https://forge.<domain>/; false → http://forge.<domain>/.
Forgejo's local Actions-artifact storage defaults to
/var/lib/forgejo/data/actions_artifacts but Forgejo does not pre-create it.
The artifact endpoint ingests the chunked upload, then the merge-chunks step
fails with 'lstat .../actions_artifacts/tmpNNN: no such file or directory',
so every upload-artifact step dies after a successful build. Pre-create the
dir (forgejo-owned) via tmpfiles. actions.ENABLED already registers the
endpoints; this gives them somewhere to write.
Forgejo reported "does not have a signing key" on PRs despite the
forgejo-gpg-init service generating one. Two causes, both fixed:
- SIGNING_KEY = "default" resolves through the forgejo process's git
config (user.signingkey), not by scanning GNUPGHOME — and the keygen
never set it, so "default" found no key. forgejo-gpg-init now sets the
forgejo user's git user.signingkey to the generated key (+ commit and
tag gpgsign), with HOME pinned to the state dir so the global config
lands where forgejo reads it.
- The keygen guard was stamp-file based, so a partial state wipe that
lost the key while keeping the stamp would never regenerate. It's now
keyed on the actual secret key (regenerate iff gpg shows none) and
runs idempotently before forgejo on each start.
Also drops the heredoc for the gpg params in favour of printf (avoids
nix-string indentation fragility) and corrects the stale comment that
claimed "default" scans GNUPGHOME.
The gateway container starts alongside every hyperhive deployment, so
gating it behind a separate enable flag was a footgun: an operator who
set it false lost the only thing exposed to the outside while the
agent containers kept running. Re-gate the gateway config on the
top-level services.hyperhive.enable instead.
- hive-gateway.nix: drop the gateway.enable mkOption; gate the config
block on config.services.hyperhive.enable.
- hive-forge.nix: behindGateway now defaults to services.hyperhive.enable;
remove the behindGateway-requires-gateway assertion (now vacuous).
- hive-network.nix: remove both gateway.enable assertions (vacuous).
- hive-c0re.nix: drop the firewall.allowedTCPPortRanges 8100-8999
fallback that opened agent ports when the gateway was off (the
gateway is now the sole entry point); HIVE_GATEWAY_ENABLED is always
set since the gateway always runs.
- nix/docs/default.nix: remove the gateway.enable = mkForce false stub
(would be an eval error against the removed option; the gateway is
already re-gated on hyperhive.enable, which docs force false).
- hive-matrix.nix, dashboard.rs: comment/prose updates only.
BREAKING: operators relying on services.hyperhive.gateway.enable = false
to suppress the gateway must instead point their own reverse proxy at
the gateway's port. NixOS errors clearly on the now-unknown option.
Without [actions] ENABLED = true in forgejo's config, the
GET /api/v1/admin/runners/registration-token endpoint returns
"runner registration token not found" regardless of token scopes,
causing hive-ci-register.service to fail on first boot.
Part of the docs-migration chore (issue #708). Remove GitHub issue
numbers from inline comments, option descriptions, and rustdoc —
these are contextless noise for anyone reading the code without
access to the original discussions. Replace with prose that captures
the same rationale directly.
No functional change. Build still clean (cargo check passes).
Continues #718 docs-extraction. Three more blocks moved to
`docs/gateway.md` (which already houses the gateway architecture
story from #775):
1. **Firewall posture (gateway on vs off)** — was a 22-line block
above `networking.firewall = lib.mkIf ...` in hive-c0re.nix.
Trimmed to 3-line ref. New `docs/gateway.md::Firewall posture
(host-level)` section covers the gateway-on / gateway-off
trade-off + why dashboard port stays loopback-only.
2. **`HIVE_FORGE_URL` loopback rationale** — was a 14-line block
above the env-var assignment. Trimmed to 5-line ref. New
`docs/gateway.md::HIVE_FORGE_URL: loopback for in-cluster,
sub-domain for the operator` section covers the in-cluster vs
external split + why agent containers can't use the sub-domain.
3. **hive-forge container shape** — was a 15-line top-of-`config`
block in hive-forge.nix explaining the nixos-container + host
netns choices. Trimmed to 4-line ref. New
`docs/gateway.md::hive-forge container shape` section captures
the same content with state-dir + wipe-via-destroy notes.
Net: hive-c0re.nix -29 lines, hive-forge.nix -11 lines, gateway.md
+44 lines. Same pattern as #782 (first pass) per iris's #10114
guidance — substantive WHY moves to docs as named sub-paragraphs,
in-code shrinks to `// see docs/<file>::<section>` refs.
Verified: `nix eval` on agent-base toplevel still resolves
cleanly; firewall posture unchanged (still 0 ports opened in the
gateway-on case + the same 8100..8999 range in the gateway-off
case).
Continues #718. Follow-up batches: remaining harness-base.nix
blocks, nix/docs/default.nix, nix/assets.nix, nix/templates/weston-vnc.nix.
mara on PR #754: "would it be better to specify full forge domain in
options instead?"
Drops the awkward `cfg.subdomain` label option. Now `cfg.domain` is
the single source of truth for both the forgejo `DOMAIN` setting
(existing semantics) AND the gateway vhost server-name (new).
## Before / after
```nix
# before: separate label + cfg.domain juggling
services.hyperhive.forge.subdomain = "forge"; # → forge.<hive>
services.hyperhive.forge.domain = "localhost"; # unused for vhost
# after: full domain, single option
services.hyperhive.forge.domain = "forge.darkest.space"; # ← used for ROOT_URL + vhost
```
## Default
`cfg.domain` default auto-derives:
- `forge.<services.hyperhive.domain>` when hive-domain is set
- `"localhost"` otherwise (pre-#749 direct-on-port shape)
So the common case (hive-domain set) gets `forge.<hive>` for free,
operators with a bespoke shape (`git.example.com`) set the full
hostname directly.
## Assertions
- `cfg.domain != ""` — empty would render `.<hive>` shaped garbage
in both server_name + /etc/hosts.
- `cfg.behindGateway → gateway.enable` — can't route through a
gateway that isn't running.
(The previous "subdomain = empty" assertion is dropped — that
edge case is gone with the rename.)
## Verified
- default with `hyperhive.domain = "test.local"` → `forge.test.local`,
`ROOT_URL = http://forge.test.local/`, vhost present
- `forge.domain = "git.example.com"` → `git.example.com`,
`ROOT_URL = http://git.example.com/`, vhost = `["_", "git.example.com"]`
- `gateway.enable = false` → `forge.domain` falls back to `localhost`,
`ROOT_URL = http://localhost:3000/`, no gateway vhost
(`behindGateway = false`)
- `/etc/hosts` (when `localHostsEntry = true`) → unique entries for
hive-domain + forge.domain (de-duped via `lib.unique` for the
edge case where forge.domain = hive-domain)
- full container toplevel builds clean
## PR title
(Will fix the PR title separately — still says "/forge/" which is
wrong since the rewrite to sub-domain shape.)
argus on PR #754 v2 review:
> `subdomain = ""` edge case: when `cfg.subdomain = ""`, the
> `localHostsEntry` appends ".${domain}" (invalid hostname; bare
> domain is already covered) and the virtualHosts key becomes
> ".${domain}" (nginx treats this as a wildcard catch-all, not a
> bare-domain server block). docs call this "advanced: collides
> with dashboard server block" — the actual nginx behavior is
> more surprising than that.
Fix: reject `""` at assertion time rather than ship the surprising
behaviour. Bare-domain landing is what the dashboard already
serves; there's no use case for `""` that null doesn't already
cover. Updated option description + dropped the now-dead branch
from the `subdomain` let-binding.
Verified: `services.hyperhive.forge.subdomain = ""` triggers the
new assertion at toplevel build with a clear message pointing at
`null` as the right opt-out. Default + `null` paths still build
clean.
mara on #749:9609: "we will go with sub domains for forge and matrix
(redirected in well known in the latter case, not user visible). close /
fix PRs you have open that dont match this."
Reshapes the v1 sub-path (`<host>/forge/`) approach into a sub-domain
vhost (`forge.<host>/`) per the mara verdict. matrix gets the same
treatment in damocles's #751 follow-up.
## Why sub-domain
- forgejo's default `ROOT_URL = http://<host>/` works without any
`X-Forwarded-Prefix` gymnastics — sub-domain hosting is the
canonical Forgejo deploy shape, matches every upstream-doc example.
- Cookie / storage isolation between the dashboard and forge (XSS blast
radius shrinks; a future forge XSS can't reach dashboard session).
- matches the matrix-spec pattern that #751 wires up for the
homeserver.
## Mechanics
**forge options:**
- `services.hyperhive.forge.subdomain` — nullable str, default `"forge"`
→ rendered sub-domain is `forge.<hive-domain>`. Set to `null` to opt
out (forge stays direct on `httpPort`); set to `""` for bare-domain
landing (advanced, collides with dashboard).
- `services.hyperhive.forge.rootUrl` — nullable str override. When
null, auto-derived: `http://<subdomain>.<hive>/` when gateway is on
+ subdomain set, else `http://<domain>:<httpPort>/` (direct).
- **Asserts** rootUrl ends with `/` (argus 🟡 on #754: forgejo's
ROOT_URL contract requires trailing slash, else emits
`https://forge.example.com.user.id` shaped garbage). Asserts
`subdomain != null` requires `hyperhive.domain` set.
**gateway:**
- New `virtualHosts."<subdomain>.<hive-domain>"` server block —
separate from the `"_"` catch-all. Proxies all `/` →
`http://127.0.0.1:<forge.httpPort>/` so forgejo handles requests at
root (no prefix translation needed; matches the upstream-default
ROOT_URL shape).
- Git-tuned: `client_max_body_size 1G`, `proxy_read_timeout 1h`,
`proxy_send_timeout 1h`, `proxy_buffering off`,
`proxyWebsockets = true`. SSH stays direct on `cfg.sshPort`.
- `networking.hosts` (when `localHostsEntry = true`) now also adds
`forge.<hive-domain> -> 127.0.0.1` for the dev loop.
## Verified
- `nix eval ROOT_URL` → `http://forge.test.local/` (default with
gateway on)
- `nix eval ROOT_URL` with `gateway.enable = false` → `http://localhost:3000/`
(current direct shape preserved)
- `nix eval virtualHosts attrs` → `["_", "forge.test.local"]`
- `nix eval networking.hosts` with `localHostsEntry = true` →
`{"127.0.0.1": ["test.local", "forge.test.local"], ...}`
- bad rootUrl (no trailing /) triggers assertion at toplevel build
with the spelled-out forgejo failure mode
- full container toplevel builds clean
(`nixos-system-hive-gateway-26.05pre-git`)
## Migration
ROOT_URL change is a one-way migration on rebuild:
- Existing agent `git remote origin` URLs (`http://localhost:3000/...`)
**keep working** — forgejo accepts any inbound URL; the URL on the
agent side is unchanged.
- New clone-link copy-paste from forge UI uses `forge.<hive>/...` —
operators copying clones after this lands need to go through the
new sub-domain.
- Direct browsing on `:3000` shows pages with `forge.<hive>` links →
works if hosts entry / DNS resolves, broken otherwise. Operators
should switch to `http://forge.<hive>/`.
## Out of scope
- TLS termination (mara explicit on #15: no TLS v0)
- SSH-over-HTTPS / wildcard cert provisioning
- matrix sub-domain (damocles's #751, sibling work)
Closes#749. Addresses argus 🟡 on #754.
argus picked option (a) on #653: put the upgrade note in each option's
`description` so it shows up in `nix flake show` + the rendered
options docs, right next to the option itself. cheapest option, no
eval-time noise (a `warnings` block would fire on every new
deployment that wants false — the normal case now).
Appended a `**Breaking change as of #651**` paragraph to each of the
three `openFirewall` descriptions, naming the exact option string the
operator needs to set to restore the old behaviour.
Gateway's note specifically calls out that external reach is the
common case (operator's primary entry point), so the upgrade hint
is most likely needed there.
mara on #651: "Dont default openFirewall to true."
Flip the `openFirewall` default from `true` to `false` for all three
modules that expose host-side ports:
- `services.hyperhive.forge.openFirewall` (httpPort 3000 + sshPort 2222)
- `services.hyperhive.gateway.openFirewall` (port 80)
- `services.hyperhive.matrix.openFirewall` (httpPort 8008)
Rationale: secure-by-default. With shared host netns, the host +
every agent container reach these services via `localhost` regardless
of the firewall — the open only matters for access from outside the
host. Operators who want external reach now flip the bool explicitly:
services.hyperhive.gateway.openFirewall = true;
Each description updated to explain the new default + when to flip
it (operator's browser, external git clients, federation announcement,
etc.). Behind a host-level reverse proxy that handles TLS, leave off.
Verified via `nix eval` on a clean stub config:
- forge openFirewall = false
- gateway openFirewall = false
- matrix openFirewall = false
- networking.firewall.allowedTCPPorts = [] (was: [80 2222 3000 8008])
Note: c0re's direct ports (7000/8000/8100-8999) are gated separately
via #621 on `gateway.enable` — that gate stays; this PR only touches
the per-module `openFirewall` knobs.
Closes#651.
Per [mara on PR #615 comment 7349](http://localhost:3000/hyperhive/hyperhive/pulls/615#issuecomment-7349):
> follow nix conventions, services.hyperhive it is. the earlier we
> change this, the less breakage.
Renames the entire host-side option tree under `services.hyperhive.*`:
- `services.hive-c0re.*` → `services.hyperhive.c0re.*`
- `hyperhive.enable` → `services.hyperhive.enable`
- `hyperhive.domain` → `services.hyperhive.domain`
- `hyperhive.forge.*` → `services.hyperhive.forge.*`
- `hyperhive.matrix.*` → `services.hyperhive.matrix.*`
Per mara's "earlier = less breakage", the previous `services.hive-c0re.enable`
deprecation alias is dropped. Operators get a clear eval error on the
old paths pointing at the rename. Single migration moment.
Per-agent options in `nix/templates/harness-base.nix` (`hyperhive.model`,
`hyperhive.allowedRecipients`, etc.) stay at `hyperhive.*` — they're
container-level config, not services in the NixOS sense.
Verified via `nix flake check --no-build` + an end-to-end NixOS eval
exercising every renamed path.
Follow-up needed: rust source comments referencing the old NixOS
option names (`hive-c0re/src/{meta,coordinator,main,dashboard}.rs`)
should be updated in a separate pure-rust PR to keep this one
strictly nix-only.
- forge nix option moves to hyperhive.forge.enable, defaults true;
hive-c0re imports the forge module so it's on by default.
- drop the agent.nix container-row viewer + /api/agent-config; link
to the agent-configs forge repo instead.
- restructure pending approvals into a card (identity header /
what-changed body / decision actions) with a link to the proposal
commit on the forge.
- diff opens in the side panel with a 3-way base toggle: vs applied
(running) / vs last-approved / vs previous proposal, served by the
new /api/approval-diff/{id}?base= endpoint.
nixpkgs's services.forgejo defaults to forgejo-lts (11.0.13 today);
LTS lags far enough behind that any prior non-LTS run against the
same state dir leaves the DB at a migration the LTS binary can't
read ('database newer than binary, refusing to start'). default to
the latest release line and let operators opt down to LTS by
overriding services.hive-forge.package.
forgejo's F3 init resolves data-dir before checking ENABLED, so
`forgejo admin user create` still fataled on the RO nix-store
default. set [F3] PATH = /var/lib/forgejo/data/f3 alongside the
disable.
forgejo's F3 federation subsystem resolves its data dir relative to
the binary, which under nixos lands at /run/current-system/sw/bin/data/f3
(read-only nix store) and fatals the daemon at boot. we don't
federate; turn it off.
avoids fighting an operator-side `services.forgejo` over the
singleton module options. container shares host netns
(`privateNetwork = false`) so agents still dial the forge via
plain `localhost:<httpPort>` and the host firewall is the only
layer that matters. container name is `hive-forge` (no `h-`
prefix) so hive-c0re's lifecycle scanner ignores it — operator
manages it with the standard `nixos-container` CLI. state lives
at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/` and
survives restarts.
new `services.hive-forge.enable` (off by default) wraps
`services.forgejo` with hyperhive-friendly defaults: sqlite (no
extra db service), built-in ssh on 2222 so it doesn't fight the
host's openssh, http on 3000 (outside hyperhive's 7000/8000/8100-8999
ranges), registration off (operator seeds agent users), private
repos by default. exported as `nixosModules.hive-forge` — operator
imports it on the host alongside hive-c0re. container-side wiring
(MCP tools or a bind-mounted token) is deferred; containers already
share the host netns so they can reach http://localhost:3000 today.
