deploy squid on every hive host, with exporter, journal forwarding and self-check
This commit is contained in:
atlas
mara
parent
f66b169158
commit
a2cf6fe8bb
2 changed files with 589 additions and 9 deletions
|
|
@ -26,6 +26,7 @@ that touches it.
|
|||
│ │
|
||||
│ nginx :80/:443 [hive-gateway] │
|
||||
│ dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP) [hive-gateway] │
|
||||
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
|
||||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||||
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
|
||||
|
|
@ -44,6 +45,7 @@ that touches it.
|
|||
| container | netns | IPv4 | listens / reached via |
|
||||
| ------------------------- | ----------------------- | -------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| gateway (host) | host | host addresses | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge |
|
||||
| egress proxy (host) | host | `bridgeIp` | squid `bridgeIp:3128`; no agent uses it yet |
|
||||
| `hive-forge` | host (shared) | host addresses | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost |
|
||||
| `hive-matrix` | private, veth on bridge | DHCP pool | tuwunel on UDS `/run/hive-matrix/tuwunel.sock` → the matrix vhost; out to store, SSO, peers |
|
||||
| `swarm-ci-runner` | private, veth on bridge | DHCP pool | outbound only (runner → forge); no inbound surface |
|
||||
|
|
@ -206,6 +208,42 @@ service bound to `127.0.0.1` only stays unreachable; rebind it to
|
|||
The port is reachable by **every** agent on the bridge subnet (like
|
||||
DNS/gateway), so only expose services safe for any agent to reach.
|
||||
|
||||
## Egress proxy (squid)
|
||||
|
||||
Every hive host runs squid in the host netns, listening on `<bridgeIp>:3128`
|
||||
(`nix/host-modules/hive-squid.nix`). The port is open on the bridge interface
|
||||
only. **No agent uses it yet**: agent traffic still goes straight out through
|
||||
the NAT. The proxy goes first so you can check its metrics and logs before
|
||||
anything reroutes.
|
||||
|
||||
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
|
||||
host. It never decrypts TLS, so it holds no signing key. It caches nothing.
|
||||
|
||||
- **Allow-list.** `services.hyperhive.squid.allowedDomains` defaults to the
|
||||
swarm's own service names (`services.hyperhive.swarm.serviceDomains`). Each
|
||||
entry matches that exact name. Everything else gets a 403.
|
||||
- **Denied before any allow.** Destinations that resolve to loopback,
|
||||
link-local or the bridge subnet, and IP-literal hosts. Squid resolves names
|
||||
on the host, so without these a listed name pointing at `127.0.0.1` would
|
||||
walk around the bridge→loopback DROP rule. Only `CONNECT` to port 443 and
|
||||
plain HTTP to ports 80 and 443 pass.
|
||||
- **Metrics.** `squid-exporter` serves `prometheus-squid-exporter` on
|
||||
`127.0.0.1:9301`. The hive collector scrapes it as job `squid` through
|
||||
`services.hyperhive.otel.scrapeTargets`. `squid_up` and
|
||||
`squid_client_http_requests_total` are the two to watch.
|
||||
- **Logs.** One syslog line per request reaches the journal as
|
||||
`squid.service`, and `services.hyperhive.otel.journalUnits` forwards that
|
||||
unit and the self-check's to the swarm's log store. A refusal shows as `TCP_DENIED/403` with the
|
||||
client's source address.
|
||||
- **Self-check.** `squid-selfcheck.timer` runs `squid-selfcheck.service`
|
||||
every minute from the bridge IP: `CONNECT` to `services.hyperhive.squid.selfcheck.canary`
|
||||
(default: the forge's name) must answer 200, and to `deny-canary.invalid`
|
||||
must answer 403. It writes one `squid-selfcheck ok=… allow_status=…
|
||||
deny_status=…` line per run and fails the unit when either is wrong. The
|
||||
host's resolver looks up the canary, so a host whose resolver maps
|
||||
that name to a loopback or bridge address (a single-host swarm with
|
||||
`gateway.localHostsEntry`) fails the check.
|
||||
|
||||
## Container isolation
|
||||
|
||||
Each agent container runs in a private network namespace with a dedicated
|
||||
|
|
|
|||
|
|
@ -373,11 +373,13 @@ a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
|
|||
no broker event, no per-hive dashboard banner. Both of those used to exist;
|
||||
this change removed them in favour of the log line.
|
||||
|
||||
The hive tier's own collector runs a `journald`
|
||||
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
|
||||
agent's journal, this collector already shares a filesystem with the unit it
|
||||
reads, being a host systemd service rather than a container) and forwards
|
||||
that one log line alongside the rest of the hive's telemetry. A
|
||||
The hive tier's own collector runs a `journald` receiver scoped to the units
|
||||
in `services.hyperhive.otel.journalUnits`: `hive-c0re.service`, plus
|
||||
`squid.service` and `squid-selfcheck.service` (see [Egress proxy](../networking/network.md#egress-proxy-squid)).
|
||||
No bind mount is needed. Unlike an agent's journal, this collector already
|
||||
shares a filesystem with the units it reads, because it's a host systemd
|
||||
service rather than a container. It forwards the crash log line alongside the
|
||||
rest of the hive's telemetry. A
|
||||
swarm-level grafana alert, `hh-container-crashed` in
|
||||
[`nix/host-modules/swarm-grafana.nix`](../../nix/host-modules/swarm-grafana.nix),
|
||||
matches it directly and pages the operator — no agent, and no per-hive
|
||||
|
|
|
|||
Loading…
Reference in a new issue