vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

bao: serve the browser UI to admins via a loopback-only listener

Browse source 
openbao gains a second listener, `ui`, on 127.0.0.1:<deploy.bao.uiPort>
(default 8204) with TLS off and no client-certificate requirement, and
`ui = true`. The existing listeners are unchanged. An nginx inside the
store's container, on 127.0.0.1:<deploy.bao.uiProxyPort> (default 8206),
forwards only /ui/ and /v1/ to it, redirects / to /ui/, answers 403 on
sys/unseal, sys/seal, sys/step-down, sys/rekey* and sys/generate-root*,
and 404 on everything else.

The gateway on the store's host serves `swarm.bao.ui.domain` (default
bao-ui.<swarm>) behind the authelia auth_request subrequest, proxying to
that nginx; the name joins serviceDomains and localNames like every
other gateway-published swarm service. authelia gets an access_control
rule restricting that name to group:admins, rendered wherever authelia
runs, since the default policy admits any session.

Trade-off, ruled by the operator on the parent issue: the UI listener
asks for no client certificate, so on that door a bao token alone is the
credential.

Three comments and a doc line claimed every API listener demands a
client certificate; they now except the loopback UI listener. The
module-eval case counting declared listeners excludes `ui` by name, as
it already did `metrics`.

On a self-signed gateway, the UI's name is a swarm service name, so its
host requests the services leaf from the store. `swarm-services-cert`
sits Before= and RequiredBy= the gateway's cert import, which nginx
Requires=. On a host whose only swarm name is the UI, that would hold
nginx, and with it the stream passthrough every reader dials, on a login
to a store that may be sealed. hive-tls drops those two edges exactly
when the UI is the only local swarm name: nginx starts on the existing
hive-leaf fallback, and the script's existing re-import reloads nginx
once the leaf issues. Every other host keeps both edges.
This commit is contained in:
atlas 2026-09-28 17:28:59 +02:00 • committed by mara
commit 3898ca33c7
10 changed files with 376 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

13
nix/host-modules/hive-tls.nix
View file

@ -35,6 +35,15 @@ let
builtins.head localServiceDomains
);
# Whether the gateway's start waits for the services leaf. Not when the
# store's own browser UI is the only swarm name this host fronts: that
# host's nginx also carries the stream passthrough every reader dials to
# reach the store, and waiting on a login to that store would take that
# path down whenever it is sealed. There the gateway starts on the hive
# leaf fallback, and the issuing script re-imports the services leaf into
# the running nginx once it lands.
gatewayWaitsForServicesLeaf = localServiceDomains != [ baoCfg.ui.domain ];
# The host-managed hive CA is the trust anchor for self-signed mode.
# It is only stood up when the gateway actually serves a self-signed
# cert: the gateway must run here and be in self-signed mode. The
@ -723,8 +732,8 @@ in
# bundle reaches the root the other way round, at the end of the script
# below: it restarts `hive-tls-ca` when the root CHANGED, which is the
# same path that already covered a store coming up hours late.
before = [ "hive-gateway-self-signed-cert.service" ];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
before = lib.optional gatewayWaitsForServicesLeaf "hive-gateway-self-signed-cert.service";
requiredBy = lib.optional gatewayWaitsForServicesLeaf "hive-gateway-self-signed-cert.service";
# The store's container, where it runs here. On a hive that reads a
# store hosted elsewhere no such unit exists and systemd ignores
# the name, which is the correct behaviour rather than a gap: what

8
nix/host-modules/swarm-authelia.nix
View file

@ -1508,6 +1508,14 @@ in
domain = swarmDomain;
subject = [ "group:${operatorGroup}" ];
policy = "one_factor";
}
# The store's browser UI. Not keyed to a deploy flag: its
# vhost is on the store's host, which need not be this
# one, and without this rule any session passes it.
++ lib.optional (swarmDomain != null) {
domain = hyperhiveCfg.swarm.bao.ui.domain;
subject = [ "group:${operatorGroup}" ];
policy = "one_factor";
};
};

236
nix/host-modules/swarm-bao.nix
View file

@ -19,6 +19,7 @@
# `deploy.bao.extraListenAddresses` names, and — for readers inside an agent
# container — the nginx *stream* passthrough below, which routes on the SNI
# without decrypting and so leaves the client certificate intact.
# The browser UI's vhost fronts a different listener; see `uiListener`.
#
# ⚠️ THIS MODULE HAS NO OPINION ABOUT WHERE THE STORE'S IDENTITY COMES FROM.
# A store must not take its certificates from an authority it will itself
@ -942,7 +943,8 @@ let
// listenerTls;
}
// extraListeners
// metricsListener;
// metricsListener
// uiListener;
# Metrics get their own listener rather than a flag on the one above, and
# that follows from what a scraper can express: a prometheus scrape config
@ -979,6 +981,63 @@ let
# above.
metricsScrapeTarget = "127.0.0.1:${toString baoDeploy.metricsPort}";
# The browser UI's listener. No client certificate, because a browser has
# none to present, so anything that can dial this address is one bao token
# away from the API. That is why it is loopback only, and why the only
# thing proxying to it is the nginx below, which browsers reach only
# through the gateway, past authelia's `admins` rule. Plain HTTP for the
# metrics listener's reason: the hop never leaves this netns.
uiListener = {
ui = {
type = "tcp";
address = "127.0.0.1:${toString baoDeploy.uiPort}";
tls_disable = true;
};
};
# Every port this container binds in the host's netns, each listener's
# derived cluster port included.
containerPorts = [
cfg.port
(cfg.port + 1)
baoDeploy.metricsPort
(baoDeploy.metricsPort + 1)
baoDeploy.uiPort
(baoDeploy.uiPort + 1)
baoDeploy.uiProxyPort
cfg.otel.telemetryPort
];
# Paths a browser session has no business sending to the store: they unseal,
# seal, rekey or mint a root token, and the unauthenticated ones among them
# can start or cancel an attempt with no token at all. `seal-status` must
# stay reachable, since the UI polls it, which is why the first pattern is
# anchored at the end.
uiDeniedPaths = [
"~* ^/v1/sys/(unseal|seal|step-down)/?$"
"~* ^/v1/sys/(rekey|generate-root)"
];
# Passes on what the gateway already set for the browser's request. bao
# reads none of these unless `x_forwarded_for_authorized_addrs` is set on
# the listener, which it is not, so its audit log names 127.0.0.1.
uiProxyHeaders = ''
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header X-Forwarded-Host $http_x_forwarded_host;
'';
# The gateway vhost's gate, copied from ./swarm-victorialogs.nix for that
# file's reason: one string does not justify importing ./swarm-ui.nix.
# `auth_request` does not inherit across sibling locations, so every
# location but the subrequest itself repeats it.
swarmAuthRequest = ''
auth_request /__hive_authelia;
auth_request_set $target_url $scheme://$http_host$request_uri;
error_page 401 =302 https://${autheliaCfg.domain}/?rd=$target_url;
'';
# ⚠️ Neither the listener above nor the forwarder below has a condition of
# its own, and they lost theirs for the same reason at different times.
#
@ -1812,6 +1871,33 @@ in
'';
};
uiPort = lib.mkOption {
type = lib.types.port;
default = 8204;
description = ''
Loopback port of the listener that serves the store's browser UI.
It requires **no client certificate**, so only the nginx in the
store's container proxies to it; see
{option}`services.hyperhive.deploy.bao.uiProxyPort`.
⚠️ openbao also binds this port **+ 1** as the listener's cluster
address, so neither may coincide with another port the store or its
container binds. An assertion checks this.
'';
};
uiProxyPort = lib.mkOption {
type = lib.types.port;
default = 8206;
description = ''
Loopback port of the nginx inside the store's container that fronts
{option}`services.hyperhive.deploy.bao.uiPort`. The gateway's
{option}`services.hyperhive.swarm.bao.ui.domain` vhost proxies here.
This nginx forwards only `/ui/` and `/v1/`, and refuses the
unseal, seal, step-down, rekey and generate-root endpoints.
'';
};
extraListenAddresses = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ ];
@ -1858,6 +1944,21 @@ in
'';
};
ui.domain = lib.mkOption {
type = lib.types.str;
default = "bao-ui.${domainBase}";
defaultText = lib.literalExpression ''"bao-ui.''${services.hyperhive.swarm.domain}"'';
description = ''
Name the gateway serves the store's browser UI on, to members of
authelia's `admins` group only. Swarm-wide because authelia's host
writes the access rule for it and the store's host serves it.
Must differ from {option}`services.hyperhive.swarm.bao.domain`: that
name is the mutual-TLS endpoint every reader dials, and it has no
vhost.
'';
};
port = lib.mkOption {
type = lib.types.port;
default = 8200;
@ -1953,6 +2054,26 @@ in
about it. Pick any other free port.
'';
}
{
assertion = lib.length (lib.unique containerPorts) == lib.length containerPorts;
message = ''
Two ports the store's container binds in the host's netns coincide:
${lib.concatMapStringsSep ", " toString containerPorts}. They are
services.hyperhive.swarm.bao.port and +1, deploy.bao.metricsPort and
+1, deploy.bao.uiPort and +1 (openbao binds each listener's port + 1
as its cluster address), deploy.bao.uiProxyPort, and
swarm.bao.otel.telemetryPort. Pick free ports.
'';
}
{
assertion = cfg.ui.domain != cfg.domain;
message = ''
services.hyperhive.swarm.bao.ui.domain equals
services.hyperhive.swarm.bao.domain (${cfg.domain}). That name is the
store's mutual-TLS endpoint; a vhost on it would terminate the TLS
every reader authenticates with. Give the UI a name of its own.
'';
}
{
assertion = haveServerTls;
message = ''
@ -2024,16 +2145,18 @@ in
# to varies, and a multi-host swarm is the operator's upstream DNS. This
# covers the case that has no upstream record to configure.
#
# ⚠️ DNS only. Bao is deliberately absent from `swarm.serviceDomains`
# and gets no vhost: nginx terminating TLS would strip the client
# certificate, which is how the store authenticates every hive — see
# this file's header. `localNames` is the one half of the sibling
# pattern that applies.
services.hyperhive.gateway.localNames = [ cfg.domain ];
# ⚠️ DNS only for `cfg.domain`. It is deliberately absent from
# `swarm.serviceDomains` and gets no vhost: nginx terminating TLS would
# strip the client certificate, which is how the store authenticates
# every hive — see this file's header. The UI's name is an ordinary
# sibling service name and gets the whole pattern.
services.hyperhive.gateway.localNames = [
cfg.domain
cfg.ui.domain
];
# All three, and no vhost among them: the `ssl_preread` stream
# server below needs the nginx process without asking it to serve
# anything, and it listens on the bridge address.
# The `ssl_preread` stream server below needs the nginx process and
# listens on the bridge address; the UI's vhost is the one vhost.
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
services.hyperhive.network.enable = lib.mkDefault true;
@ -2125,6 +2248,46 @@ in
# certificate its CA signed.
services.hyperhive.network.exposeHostPorts = [ cfg.port ];
# The operators' door to the UI, shaped like ./swarm-victorialogs.nix's
# vhost. It proxies to the container's nginx over loopback (shared
# netns), so no port opens anywhere. `auth_request` only proves a
# session; restricting it to `admins` is ./swarm-authelia.nix's rule
# for this name. `forceSSL` because authelia answers a plain-http
# subrequest with 400, and `removeAttrs` because nixos refuses
# `addSSL` beside it.
services.nginx.virtualHosts.${cfg.ui.domain} =
(builtins.removeAttrs (hyperhiveCfg.gateway.lib.tlsFor cfg.ui.domain) [ "addSSL" ])
// {
forceSSL = true;
listen = hyperhiveCfg.gateway.lib.listen;
extraConfig = hyperhiveCfg.gateway.lib.securityHeaders;
locations = {
"/" = {
proxyPass = "http://127.0.0.1:${toString baoDeploy.uiProxyPort}";
extraConfig = swarmAuthRequest;
};
"= /__hive_authelia" = {
proxyPass = "https://${autheliaCfg.domain}/api/authz/auth-request";
# nixpkgs appends its OWN `Host $host` after extraConfig,
# which would override verifiedProxyTo's — see the comment
# on verifiedProxyTo in hive-gateway/vhost-lib.nix.
recommendedProxySettings = false;
extraConfig = ''
internal;
${hyperhiveCfg.gateway.lib.verifiedProxyTo autheliaCfg.domain}
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-Method $request_method;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
'';
};
};
};
# THE delivery unit for the forwarder's own OIDC client secret, copied
# in shape from ./swarm-otel.nix's `swarm-bao-otel-oidc`: a cert login
# that fails LOUDLY, since every state it fails on is one a retry fixes,
@ -2450,9 +2613,10 @@ in
'';
};
# The swarm's first grant, written from the HOST. Every API listener sets
# `tls_require_and_verify_client_cert`, so a client needs an identity
# wherever it runs — and only the host has one: the granter's leaf.
# The swarm's first grant, written from the HOST. Every API listener but
# the loopback UI one sets `tls_require_and_verify_client_cert`, so a
# client needs an identity wherever it runs — and only the host has one:
# the granter's leaf.
systemd.services.swarm-bao-controller-policy = lib.mkIf haveGranter {
description = "write the swarm controller's bao policy and cert-auth role";
after = [ "container@${cfg.machine}.service" ] ++ granterAfter;
@ -3281,12 +3445,52 @@ in
settings = {
listener = listeners;
storage.raft.path = stateDir;
# Served on every listener, but a browser can only use the
# one that asks for no client certificate: `ui`, via the
# nginx below.
ui = true;
}
// advertise
// telemetry
// sealSettings;
};
# The only client of the `ui` listener. Loopback in the host's
# netns, where the gateway's vhost for `swarm.bao.ui.domain` dials
# it. Everything but the UI and its API is 404, and the key-share
# and seal endpoints are 403 whatever token is presented.
# `absolute_redirect off` keeps the `/` redirect relative;
# otherwise its Location names this listener, not the gateway.
services.nginx = {
enable = true;
virtualHosts.bao-ui = {
listen = [
{
addr = "127.0.0.1";
port = baoDeploy.uiProxyPort;
}
];
extraConfig = ''
absolute_redirect off;
'';
locations = {
"= /".return = "302 /ui/";
"/".return = "404";
"/ui/" = {
proxyPass = "http://127.0.0.1:${toString baoDeploy.uiPort}";
extraConfig = uiProxyHeaders;
};
"/v1/" = {
proxyPass = "http://127.0.0.1:${toString baoDeploy.uiPort}";
extraConfig = uiProxyHeaders;
};
}
// lib.genAttrs uiDeniedPaths (_: {
return = "403";
});
};
};
# The private key crosses the user boundary here, not on the mount.
# `swarm-bao-certs` installs it 0600 root-owned, and the unit runs
# as a `DynamicUser`, so the bind-mounted file is unreadable to it —
@ -3348,9 +3552,9 @@ in
# ⚠️ NO `units` allowlist here, and that is the point rather than a
# simplification. A shared collector needs one because the journal
# it reads holds six containers' units plus the host's own; this
# one reads only openbao and the two oneshots beside it, so there
# is nothing foreign to separate out — and a list of unit names is
# a thing to get wrong, which ships nothing while looking healthy.
# one reads only openbao, the UI's nginx and the two oneshots, so
# there is nothing foreign to separate out — and a list of unit names
# is a thing to get wrong, which ships nothing while looking healthy.
assertions = [
{
# Sibling of the agent forwarder's identical assertion, and it

5
nix/host-modules/swarm.nix
View file

@ -109,7 +109,10 @@ let
# this list, `gateway.lib.tlsFor` falls back to the hive leaf, which
# cannot cover a name under a different apex — see the ⚠️ above this
# list for what that looked like the last time a name was missed here.
++ lib.optional deployCfg.victorialogs.enable swarmCfg.victorialogs.domain;
++ lib.optional deployCfg.victorialogs.enable swarmCfg.victorialogs.domain
# The store's browser UI, not the store: `swarm.bao.domain` is mTLS and
# never gets a gateway certificate.
++ lib.optional deployCfg.bao.enable swarmCfg.bao.ui.domain;
# Hives whose entry still carries the removed `certFingerprint`. Scanned
# here, at top level, because that is the only place an assertion about a

96
nix/module-eval/bao-basics.nix
View file

@ -41,6 +41,13 @@ let
deploy.bao.serverKeyFile = "/etc/pki/bao-key.pem";
};
autheliaOnly = hive { deploy.authelia.enable = true; };
baoWithForge = hive {
deploy.bao.enable = true;
deploy.forgejo.enable = true;
};
# The store's units live inside its container, so the gates below have to
# look there rather than at the host's service set.
baoUnits = machine: machine.containers.swarm-bao.config.systemd.services;
@ -281,6 +288,95 @@ let
&& !(lib.hasInfix restart sh)
&& lib.hasInfix "# then unseal" sh;
}
{
# The browser UI's listener must not loosen the one every reader dials.
# `loopback` is named so an empty filter cannot pass.
name = "every listener but metrics and the UI still requires a client certificate";
ok =
let
l = (baoSettings baoPkcs11).listener;
api = builtins.removeAttrs l [
"metrics"
"ui"
];
in
api ? loopback
&& lib.all (x: (x.tls_require_and_verify_client_cert or false) == true) (lib.attrValues api);
}
{
# No client certificate on this one, so loopback is all that keeps it
# away from everything but this netns.
name = "the UI listener is loopback-only and the UI is served";
ok =
let
s = baoSettings baoPkcs11;
in
(s.ui or false) == true
&& lib.hasPrefix "127.0.0.1:" (s.listener.ui.address or "")
&& !(s.listener.ui ? tls_require_and_verify_client_cert);
}
{
# The nginx in front of it is the listener's only client and must bind
# loopback too, and it refuses the endpoints a browser never needs.
name = "the UI's nginx binds loopback and refuses the unseal and root-generation paths";
ok =
let
v = baoPkcs11.containers.swarm-bao.config.services.nginx.virtualHosts.bao-ui;
in
lib.all (x: x.addr == "127.0.0.1") v.listen
&& (v.locations."~* ^/v1/sys/(unseal|seal|step-down)/?$".return or null) == "403"
&& (v.locations."~* ^/v1/sys/(rekey|generate-root)".return or null) == "403";
}
{
# A new vhost gets no group gate from `auth_request`; authelia's
# default policy admits any session. Evaluated on a host that runs
# authelia and no store, because that is where the rule has to render.
name = "authelia restricts the store's UI name to admins wherever authelia runs";
ok =
let
rules =
autheliaOnly.containers.swarm-authelia.config.services.authelia.instances.swarm.settings.access_control.rules;
in
!autheliaOnly.services.hyperhive.deploy.bao.enable
&& builtins.elem {
domain = "bao-ui.t.local";
subject = [ "group:admins" ];
policy = "one_factor";
} rules;
}
{
# The other half of that rule: the vhost exists, and asks authelia.
name = "the store's host serves the UI name behind the authelia subrequest";
ok =
let
v = baoPkcs11.services.nginx.virtualHosts."bao-ui.t.local";
in
lib.hasInfix "auth_request /__hive_authelia;" (v.locations."/".extraConfig or "")
&& (v.locations."/".proxyPass or "") == "http://127.0.0.1:8206";
}
{
# The stream passthrough every reader dials rides the same nginx as the
# UI's vhost. On a host whose only swarm name is the UI, that nginx must
# not wait on the services leaf, which needs a login to this very store.
# The control is a store host that also fronts forge: its gateway waits,
# exactly as it did before the UI existed.
name = "a store host fronting only the UI does not hold nginx on a store login";
ok =
let
waits =
m:
builtins.elem "hive-gateway-self-signed-cert.service" m.systemd.services.swarm-services-cert.requiredBy;
orders =
m:
builtins.elem "hive-gateway-self-signed-cert.service" m.systemd.services.swarm-services-cert.before;
in
baoPkcs11.services.hyperhive.swarm.localServiceDomains == [ "bao-ui.t.local" ]
&& baoPkcs11.systemd.services ? hive-gateway-self-signed-cert
&& !(waits baoPkcs11)
&& !(orders baoPkcs11)
&& waits baoWithForge
&& orders baoWithForge;
}
];
in
runGroup "bao-basics" cases

4
nix/module-eval/bao-grants.nix
View file

@ -297,8 +297,8 @@ let
cases = [
{
# Reads the rendered unit on the HOST, which is where the write happens:
# every API listener demands a client certificate, and the host is the
# side that has one.
# every API listener but the loopback UI one demands a client
# certificate, and the host is the side that has one.
name = "a store host renders the granting unit on the host, logging in as the granter";
ok =
let

14
nix/module-eval/bao-otel-collector.nix
View file

@ -115,13 +115,19 @@ let
}
{
# Control for the case above: these settings are rendered per deployment,
# not constants a passing case could be indifferent to. The metrics
# listener is excluded by name — it answers on a port of its own and is
# not one of the API addresses this case is counting.
# not constants a passing case could be indifferent to. The metrics and
# UI listeners are excluded by name — each answers on a loopback port of
# its own and is not one of the declared addresses this case is counting.
name = "a declared extra address renders a second listener beside loopback";
ok =
builtins.length (
builtins.filter (n: n != "metrics") (builtins.attrNames (baoSettings baoTwoAddresses).listener)
builtins.filter (
n:
!(builtins.elem n [
"metrics"
"ui"
])
) (builtins.attrNames (baoSettings baoTwoAddresses).listener)
) == 2;
}
{