fix: use forge domain URL + open 80/443 for isolated agents
when isolateContainers=true, isolated agents have dnsmasq as their resolver — forge.<domain> resolves to bridgeIp. route HIVE_FORGE_URL through nginx on port 80 instead of exposing the raw forge port. - HIVE_FORGE_URL: http://<forge.domain> when isolated (nginx proxies) - bridge firewall: open 80+443 for agents to reach nginx (gateway) - remove forge-specific httpPort rule (no longer needed) - update docs/gateway.md + docs/network.md per mara's review comment on PR #1150.
This commit is contained in:
parent
c97120f016
commit
806d0e4a61
4 changed files with 28 additions and 31 deletions
|
|
@ -200,27 +200,22 @@ dashboard reach by design — the surface is privileged (approve /
|
||||||
deny / destroy) and must not be exposed without a real reverse
|
deny / destroy) and must not be exposed without a real reverse
|
||||||
proxy in front.
|
proxy in front.
|
||||||
|
|
||||||
## `HIVE_FORGE_URL`: bridge gateway for isolated agents, loopback for shared-netns
|
## `HIVE_FORGE_URL`: domain via gateway for isolated agents, loopback for shared-netns
|
||||||
|
|
||||||
Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
|
Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
|
||||||
`hive-forge` calls against it. `hive-c0re.nix` sets this based on the
|
`hive-forge` calls against it. `hive-c0re.nix` sets this based on the
|
||||||
network isolation mode:
|
network isolation mode:
|
||||||
|
|
||||||
- **`network.isolateContainers = true`**: agents run in private netns,
|
- **`network.isolateContainers = true`**: agents run in private netns and
|
||||||
so host loopback is unreachable. `HIVE_FORGE_URL` is set to
|
get the bridge dnsmasq as their resolver. `HIVE_FORGE_URL` is set to
|
||||||
`http://<bridgeIp>:<forge.httpPort>`. Forgejo binds `0.0.0.0` so it's
|
`http://<forge.domain>` (default `forge.<hive-domain>`). Agents resolve
|
||||||
reachable at the bridge gateway IP. `hive-network.nix` opens
|
the hostname via dnsmasq → bridge IP, then reach nginx on port 80 (bridge
|
||||||
`forge.httpPort` on the bridge interface automatically.
|
firewall opens 80+443 when isolation is on). nginx proxies to forgejo — the
|
||||||
|
same path an operator browser takes, no raw port exposure needed.
|
||||||
- **`network.isolateContainers = false`** (default): agents share the host's
|
- **`network.isolateContainers = false`** (default): agents share the host's
|
||||||
network namespace, so loopback reaches forgejo directly. `HIVE_FORGE_URL`
|
network namespace, so loopback reaches forgejo directly. `HIVE_FORGE_URL`
|
||||||
is `http://127.0.0.1:<forge.httpPort>`.
|
is `http://127.0.0.1:<forge.httpPort>`.
|
||||||
|
|
||||||
The sub-domain default (`forge.<hive-domain>`) is for **operator
|
|
||||||
browsers + cross-host clients**, not in-cluster traffic. Using the
|
|
||||||
sub-domain URL inside agent containers would fail every `hive-forge`
|
|
||||||
invocation with "Name or service not known" — the agent's nspawn
|
|
||||||
doesn't have DNS for the external hostname.
|
|
||||||
|
|
||||||
## hive-forge container shape
|
## hive-forge container shape
|
||||||
|
|
||||||
Private Forgejo wrapped in a nixos-container (`hive-forge`, not
|
Private Forgejo wrapped in a nixos-container (`hive-forge`, not
|
||||||
|
|
|
||||||
|
|
@ -95,6 +95,11 @@ agent containers.
|
||||||
interface only. Other interfaces stay closed. The hive resolver
|
interface only. Other interfaces stay closed. The hive resolver
|
||||||
isn't an external-facing service.
|
isn't an external-facing service.
|
||||||
|
|
||||||
|
When `isolateContainers = true`, `allowedTCPPorts` is extended with
|
||||||
|
`[ 80 443 ]` so isolated agents can reach nginx (gateway container,
|
||||||
|
shared host netns) for the forge sub-domain, per-agent UI proxies,
|
||||||
|
and any other HTTP services.
|
||||||
|
|
||||||
## Container isolation
|
## Container isolation
|
||||||
|
|
||||||
`services.hyperhive.network.isolateContainers` (default `false`) flips
|
`services.hyperhive.network.isolateContainers` (default `false`) flips
|
||||||
|
|
@ -108,8 +113,8 @@ agent containers from shared host netns to private netns. Set only after
|
||||||
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
|
| IP forwarding | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1` |
|
||||||
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
| Internet NAT | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC |
|
||||||
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
|
| Loopback DROP | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks |
|
||||||
| Forge access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts` — opens `forge.httpPort` on the bridge interface so isolated agents can reach forgejo at `<bridgeIp>:<httpPort>` |
|
| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents reach nginx on the host (shared netns) |
|
||||||
| Forge URL | `HIVE_FORGE_URL` flips from `http://127.0.0.1:3000` to `http://<bridgeIp>:3000` — forwarded to containers via meta flake |
|
| Forge URL | `HIVE_FORGE_URL` flips from `http://127.0.0.1:3000` to `http://forge.<domain>` — agents resolve via dnsmasq, nginx proxies to forgejo |
|
||||||
| c0re signal | `HIVE_NETWORK_ISOLATION=1`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` |
|
| c0re signal | `HIVE_NETWORK_ISOLATION=1`, `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` |
|
||||||
|
|
||||||
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
|
`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
|
||||||
|
|
|
||||||
|
|
@ -557,17 +557,17 @@ in
|
||||||
HYPERHIVE_SWARM_NAME = config.services.hyperhive.swarmName;
|
HYPERHIVE_SWARM_NAME = config.services.hyperhive.swarmName;
|
||||||
}
|
}
|
||||||
// lib.optionalAttrs config.services.hyperhive.forge.enable {
|
// lib.optionalAttrs config.services.hyperhive.forge.enable {
|
||||||
# In-cluster forge URL. When containers are isolated (private netns),
|
# In-cluster forge URL.
|
||||||
# 127.0.0.1 is the container's own loopback — unreachable for host
|
# - Isolated (private netns): containers resolve `forge.<domain>` via
|
||||||
# services. Use the bridge gateway IP instead; forgejo binds 0.0.0.0
|
# the bridge dnsmasq and reach nginx on port 80. No raw forge port
|
||||||
# so it's reachable there. Shared-netns mode keeps loopback path.
|
# needed — nginx proxies to forgejo as it does for the operator.
|
||||||
# External `forge.<hive>` sub-domain isn't DNS-resolvable from inside
|
# - Shared netns: host loopback is reachable, use direct port.
|
||||||
# nspawn either way. See `docs/gateway.md::HIVE_FORGE_URL`.
|
# See `docs/gateway.md::HIVE_FORGE_URL`.
|
||||||
HIVE_FORGE_URL =
|
HIVE_FORGE_URL =
|
||||||
if
|
if
|
||||||
config.services.hyperhive.network.enable && config.services.hyperhive.network.isolateContainers
|
config.services.hyperhive.network.enable && config.services.hyperhive.network.isolateContainers
|
||||||
then
|
then
|
||||||
"http://${config.services.hyperhive.network.bridgeIp}:${toString config.services.hyperhive.forge.httpPort}"
|
"http://${config.services.hyperhive.forge.domain}"
|
||||||
else
|
else
|
||||||
"http://127.0.0.1:${toString config.services.hyperhive.forge.httpPort}";
|
"http://127.0.0.1:${toString config.services.hyperhive.forge.httpPort}";
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -215,16 +215,13 @@ in
|
||||||
ip saddr ${cfg.bridgeIp}/${toString cfg.bridgePrefixLength} ip daddr 127.0.0.0/8 drop
|
ip saddr ${cfg.bridgeIp}/${toString cfg.bridgePrefixLength} ip daddr 127.0.0.0/8 drop
|
||||||
'';
|
'';
|
||||||
|
|
||||||
# Allow isolated agents to reach the forge via the bridge gateway IP.
|
# Allow isolated agents to reach the gateway (nginx on the host, shared
|
||||||
# Forgejo binds 0.0.0.0 so it's reachable at `bridgeIp:httpPort` from
|
# netns). Port 80 covers `http://forge.<domain>`, per-agent UI proxies,
|
||||||
# inside agent containers; without this rule the default INPUT policy
|
# and any other HTTP services the gateway fronts. Port 443 for HTTPS.
|
||||||
# drops the connection before it reaches forgejo. Only added when forge
|
networking.firewall.interfaces.${cfg.bridgeName}.allowedTCPPorts = [
|
||||||
# is enabled — no-op otherwise.
|
80
|
||||||
networking.firewall.interfaces.${cfg.bridgeName}.allowedTCPPorts =
|
443
|
||||||
lib.optionals config.services.hyperhive.forge.enable
|
];
|
||||||
[
|
|
||||||
config.services.hyperhive.forge.httpPort
|
|
||||||
];
|
|
||||||
|
|
||||||
# Tells hive-c0re to pass PRIVATE_NETWORK + bridge settings to each
|
# Tells hive-c0re to pass PRIVATE_NETWORK + bridge settings to each
|
||||||
# container. HIVE_NETWORK_SUBNET is host-bridge IP/prefix, not canonical
|
# container. HIVE_NETWORK_SUBNET is host-bridge IP/prefix, not canonical
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue