vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

grafana: fix the sso.md admin claim, deny the publisher the admin password

Browse source 
docs/swarm/sso.md still said Grafana ships admin/admin and that a dead
provider locks out the built-in admin. The password is minted into the
store and the API still takes basic auth for it, so the paragraph now says
where an operator reads it in an emergency. secrets.md says how to rotate
the password and to deploy the store's host first, and no longer calls SSO
the only way in.

The secret publisher's `services/*` write covered the admin-password
leaf, so a file-copier could set the password Grafana applies at its next
start. The leaf gets the same exact-path deny as the callout seeds, with
an arm in module-eval-bao-grants.
This commit is contained in:
atlas 2026-10-12 12:16:08 +02:00 • committed by mara
commit eecd2b62e8
4 changed files with 71 additions and 30 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/swarm/secrets.md
View file

@ -42,7 +42,7 @@ neither is a renaming of the other.
## Swarm-level — one of each per swarm ## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
@ -51,7 +51,7 @@ neither is a renaming of the other.
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value | `/var/lib/grafana-admin/password` inside the `swarm-grafana` container: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. Grafana doesn't start until the file exists, so it stays down rather than falling back to the upstream default `admin` | | Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, delete the leaf in the store: the next run of the unit mints a new one, and Grafana's `preStart` applies it | `/var/lib/grafana-admin/password` inside the `swarm-grafana` container: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. Grafana doesn't start until the file exists, so it stays down rather than falling back to the upstream default `admin` |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
@ -299,7 +299,7 @@ a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile` only way into the UI), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent: / `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so either one left a Grafana with no SSO and no password box — nothing failed, so
@ -317,6 +317,13 @@ the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source homeserver starts without its secret, and the forge registers no login source
without its. without its.
Grafana's admin password rides the same leaf and the same store. Where Grafana
and the store run on different hosts, deploy the store's host first:
`swarm-bao-grafana-oidc-policy` there writes the grant that lets Grafana create
the password, and until it has, the store refuses `swarm-bao-grafana-admin` on
the Grafana host, which retries, while Grafana stays down because its config
reads that file.
The **third reader** is the swarm collector's own OIDC client secret, the The **third reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that `swarm/services/<client-id>/oidc/client` out of the store on every host that

18
docs/swarm/sso.md
View file

@ -300,12 +300,18 @@ gateway refuses its token too. See [`swarmctl/README.md`](../../swarmctl/README.
with two ways in, and making authelia the only path there is a separate, with two ways in, and making authelia the only path there is a separate,
reversible switch (tuwunel's `login_with_password`, forgejo's own setting). reversible switch (tuwunel's `login_with_password`, forgejo's own setting).
⚠️ **Grafana is the exception, and not a reversible one.** Its module sets ⚠️ **Grafana is the exception, and not a reversible one.** Its module sets
`auth.disable_login_form` unconditionally, because Grafana ships an `auth.disable_login_form` unconditionally: its vhost is on the public
`admin`/`admin` account and its vhost is on the public gateway — so a gateway, so a password box there would be a way in whatever the
password box there is a way in whatever the deployment. SSO is the only deployment. The form is all this hides. Grafana's API still takes HTTP
door, and a dead provider locks everyone out, the built-in admin basic auth for the built-in `admin` account, whose password is a random
included. That's why `swarm.grafana.oidc.role` defaults to `Admin`; see one minted into the swarm secret store, not the upstream `admin`. A
[`services.md`](services.md#metrics-victoriametrics--grafana). dead provider locks out the SSO accounts but not that one: an operator
reads the password from `swarm/services/<grafana client id>/admin/password`
in the store (or, as root on the Grafana host, from
`/var/lib/grafana-admin/password` inside the `swarm-grafana` container)
and calls the API with it. See [`services.md`](services.md#metrics-victoriametrics--grafana)
and [`secrets.md`](secrets.md). SSO logins are the everyday way in, which
is why `swarm.grafana.oidc.role` defaults to `Admin`.
- **It doesn't provision users.** `swarmctl` creates and destroys agents - **It doesn't provision users.** `swarmctl` creates and destroys agents
continuously today, so the subject set belongs to a program rather continuously today, so the subject set belongs to a program rather
than to a config file. than to a config file.

13
nix/host-modules/swarm-bao.nix
View file

@ -482,6 +482,9 @@ let
# prefix, are denied. A write to the xkey's would replace the key every # prefix, are denied. A write to the xkey's would replace the key every
# queue request is sealed to; the issuer's is the authority to admit anyone # queue request is sealed to; the issuer's is the authority to admit anyone
# to the queue, and the user's is the one identity exempt from callout. # to the queue, and the user's is the one identity exempt from callout.
# Grafana's admin password is denied the same way: a write to it sets the
# password `grafana.service` applies to the server-admin account at its next
# start.
secretPublisherPolicyText = '' secretPublisherPolicyText = ''
path "${credentialMountPath}/data/swarm/hives/*" { path "${credentialMountPath}/data/swarm/hives/*" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
@ -503,6 +506,10 @@ let
capabilities = ["deny"] capabilities = ["deny"]
} }
path "${credentialMountPath}/data/${grafanaAdminLeaf}" {
capabilities = ["deny"]
}
path "${credentialMountPath}/data/${controllerClientLeaf}" { path "${credentialMountPath}/data/${controllerClientLeaf}" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
@ -860,7 +867,7 @@ let
cn = baoDeploy.grafanaOidcCommonName; cn = baoDeploy.grafanaOidcCommonName;
policyText = policyText =
readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client" readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"
+ createStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/admin/password"; + createStanza "${credentialMountPath}/data/${grafanaAdminLeaf}";
} }
]; ];
@ -922,6 +929,10 @@ let
userSeedLeaf = "swarm/services/swarm-nats/callout/user-seed"; userSeedLeaf = "swarm/services/swarm-nats/callout/user-seed";
userPublicLeaf = "swarm/public/swarm-nats/callout-user"; userPublicLeaf = "swarm/public/swarm-nats/callout-user";
# Grafana's admin password, which only Grafana's leaf creates. The nix half
# of `adminPasswordStorePath` in ./swarm-grafana.nix.
grafanaAdminLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/admin/password";
createStanza = path: '' createStanza = path: ''
path "${path}" { path "${path}" {
capabilities = ["create", "read"] capabilities = ["create", "read"]

17
nix/module-eval/bao-grants.nix
View file

@ -462,6 +462,23 @@ let
"update" "update"
]; ];
} }
{
# Grafana's admin password sits under the publisher's `services/*` write
# too, and a planted value is the password the next Grafana start applies
# to the server-admin account. The OIDC leaf beside it is the control:
# still the prefix's.
name = "the publisher may not write Grafana's admin password";
ok =
let
grants = grantsIn baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy.script;
in
(grantFor grants "secret/data/swarm/services/swarm-grafana/admin/password").caps == [ "deny" ]
&&
(grantFor grants "secret/data/swarm/services/swarm-grafana/oidc/client").caps == [
"create"
"update"
];
}
{ {
# The ordering is load-bearing and invisible at runtime: the controller's # The ordering is load-bearing and invisible at runtime: the controller's
# unit creates the KV and cert-auth mounts this one writes into, so # unit creates the KV and cert-auth mounts this one writes into, so