Watch
0
0
Fork
You've already forked hyperhive
0

trust the swarm services ca in swarm-ci-metrics

The poller's forge, authelia and collector calls go to leaves issued by
the swarm's own CA, and its reqwest/rustls roots come from SSL_CERT_FILE,
which the unit never set. Every pass since it was deployed fails
"invalid peer certificate: UnknownIssuer" before pushing, so the CI runs
dashboard has no hyperhive.ci.run.* samples (#5438).

- nix/host-modules/swarm-ci-metrics.nix: import lib/hive-ca-trust.nix and
  register a host trustBundle for the unit, as swarm-controller does; it
  adds swarm-ci-metrics-ca-bundle.service and sets SSL_CERT_FILE.
- nix/module-eval/ci-metrics.nix: a case asserting the unit's
  SSL_CERT_FILE is its bundle path when self-signed, with the bundle
  unit's ordering, and absent where the poller is off.
- docs/swarm/ci-metrics.md: how the poller trusts the swarm's CA.
This commit is contained in:
atlas 2026-10-17 22:51:16 +02:00 • committed by mara
commit 6630a51022
3 changed files with 51 additions and 3 deletions

View file

@ -18,6 +18,13 @@ none of them on its host. On a host without the store, its leaf
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one (`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
file to copy, as for the CI runner. file to copy, as for the CI runner.
With a self-signed gateway, those names serve leaves the swarm's own CA issued.
The poller reads them through `swarm-ci-metrics-ca-bundle.service`, which joins
the system CAs and the host's `trust-bundle.pem` into
`/run/swarm-ci-metrics-ca/trust-bundle.pem` and sets that file as the poller's
`SSL_CERT_FILE` ([CA](ca.md#the-trust-bundle)). Without it, every pass fails
with `invalid peer certificate: UnknownIssuer`.
## What it reads, and as whom ## What it reads, and as whom
| what | where | identity | | what | where | identity |

View file

@ -14,6 +14,7 @@
# its store leaf, handed in by `LoadCredential=`; the forge token and its OIDC # its store leaf, handed in by `LoadCredential=`; the forge token and its OIDC
# client secret it reads from the store under that leaf on every pass. # client secret it reads from the store under that leaf on every pass.
{ {
pkgs,
lib, lib,
config, config,
... ...
@ -40,6 +41,16 @@ let
haveClientIdentity = haveClientIdentity =
baoDeploy.ciMetricsClientCertFile != null && baoDeploy.ciMetricsClientKeyFile != null; baoDeploy.ciMetricsClientCertFile != null && baoDeploy.ciMetricsClientKeyFile != null;
# The forge, authelia and the collector serve leaves issued by the swarm's
# own CA, which the system store lacks, and the poller's `reqwest`/`rustls`
# roots come from `SSL_CERT_FILE`, which replaces that store. `trustBundle`
# below sets it to the system CAs plus the hive anchor; without it every
# pass fails `invalid peer certificate: UnknownIssuer`.
caTrust = import ./lib/hive-ca-trust.nix {
inherit lib gatewayCfg;
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
};
credential = id: "/run/credentials/${unit}.service/${id}"; credential = id: "/run/credentials/${unit}.service/${id}";
# Credential id → host path of the poller's store identity. # Credential id → host path of the poller's store identity.
storeFiles = { storeFiles = {
@ -51,6 +62,20 @@ let
}; };
in in
{ {
# `hostUnit`: a host service, so the bundle reads the host CA path and waits
# on `hive-tls-ca.service`. `enable`: `imports` is unconditional, so without
# it a host with the poller off gets the bundle unit and an `ExecStart`-less
# `swarm-ci-metrics` service.
imports = [
(caTrust.trustBundle {
inherit pkgs;
name = unit;
consumers = [ unit ];
hostUnit = true;
inherit (cfg) enable;
})
];
options.services.hyperhive.swarm.ci-metrics.clientId = lib.mkOption { options.services.hyperhive.swarm.ci-metrics.clientId = lib.mkOption {
type = lib.types.str; type = lib.types.str;
readOnly = true; readOnly = true;

View file

@ -1,8 +1,8 @@
# `checks.module-eval-ci-metrics` — see ./lib.nix for the shared rationale # `checks.module-eval-ci-metrics` — see ./lib.nix for the shared rationale
# (why this suite exists, naming convention, "evaluates not executes"). The CI # (why this suite exists, naming convention, "evaluates not executes"). The CI
# run poller's unit and timer, its default, its collector route and its IdP # run poller's unit and timer, its default, its CA trust, its collector route
# registration; its store grants are in ./bao-grants.nix beside every other # and its IdP registration; its store grants are in ./bao-grants.nix beside
# principal's, and its dashboard in ./grafana.nix. # every other principal's, and its dashboard in ./grafana.nix.
# #
# Every fixture is a shared one, so this cluster evaluates no host of its own. # Every fixture is a shared one, so this cluster evaluates no host of its own.
{ {
@ -78,6 +78,22 @@ let
&& lib.elem "otlp/swarm-ci-metrics" s.service.pipelines."metrics/${otel.producerName}".receivers && lib.elem "otlp/swarm-ci-metrics" s.service.pipelines."metrics/${otel.producerName}".receivers
&& h.services.nginx.virtualHosts.${otel.domain}.locations ? "/swarm-ci-metrics/"; && h.services.nginx.virtualHosts.${otel.domain}.locations ? "/swarm-ci-metrics/";
} }
{
# The forge, authelia and the collector chain to the swarm's own CA;
# without the bundle every pass fails `UnknownIssuer`. hive-c0re's
# bundle on the bare host is the control that the absence arm looks up
# a unit name the module would write.
name = "the poller trusts the swarm's CA through its own bundle, which exists only where the poller does";
ok =
allLocal.services.hyperhive.gateway.useSelfSigned
&& (env.SSL_CERT_FILE or null) == "/run/swarm-ci-metrics-ca/trust-bundle.pem"
&& lib.elem "swarm-ci-metrics-ca-bundle.service" svc.requires
&& lib.elem "hive-tls-ca.service" (
allLocal.systemd.services.swarm-ci-metrics-ca-bundle.requires or [ ]
)
&& bare.systemd.services ? hive-c0re-ca-bundle
&& !(bare.systemd.services ? swarm-ci-metrics-ca-bundle);
}
{ {
name = "the poller's client is registered as a machine, and the publisher carries its secret"; name = "the poller's client is registered as a machine, and the publisher carries its secret";
ok = ok =