Job-level timeout-minutes is parsed but not enforced by this runner; only
the runner's jobTimeout bounds a job. Also fixes the devshell coverage
comment, the shellcheck job comment, an option description citing a script
that is not in the repo, and comment file pointers that resolve to nothing.
The queue's metrics were scraped over loopback from the exporter, declared
only where the queue and the collector share a host, so a collector on any
other host had no job for it. Its target is now declared on every host as
`https://<swarm.nats.domain>/metrics`, and the queue's host gets a gateway
vhost on that name serving `= /metrics` from the exporter behind the
collector's auth subrequest (`/__nats_metrics_authz`, the forge's shape).
Every other path there is 403; the monitoring endpoint is never proxied.
The rules derived from `publishedScrapeTargets` admit the collector's
client and deny every other subject. The loopback entry, and with it
`glue-otel-scrape-targets.nix`, is gone.
In self-signed mode the vhost serves the queue's own `swarm-nats` leaf
(issued for exactly that name, `server_flag` set), copied 0640
root:nginx by `swarm-nats-gateway-cert`; until the leaf exists that unit
writes a throwaway pair, so nginx never starts on a missing file. The
name stays out of `swarm.serviceDomains`, so a host whose only swarm
service is the queue fronts no services name and its gateway neither
waits for nor needs the swarm-services leaf: no services-issuer pair has
to be placed on it. With ACME or an operator cert the vhost uses the
gateway's own TLS, as every other name does.
The queue's host now enables the gateway (`mkDefault`), as every other
swarm service's host does.
Fixes#5205
deploy.builder.enable defaults on under deploy.allSwarmServices (mkDefault),
where the host has the builder's store identity pair, as for the review
classifier. An explicit false still wins.
Grafana's own metrics were scraped over loopback through the
`grafana-metrics` nginx vhost, declared only where grafana and the
collector share a host, so a collector on any other host had no job for
it. Its target is now declared on every host as
`https://<swarm.grafana.domain>/metrics`, next to the other published
targets, and grafana's vhost serves `= /metrics` behind the collector's
auth subrequest (`/__grafana_metrics_authz`, the forge's shape). The
rules derived from `publishedScrapeTargets` admit the collector's client
there and deny every other subject.
The loopback scrape is gone with the vhost that served it and its
`swarm.grafana.metricsPort` option, so a single host still scrapes
grafana once, now by name. A config that sets that option now fails
evaluation with "option does not exist".
On the published vhost `/metrics` was answered with 403 for everyone.
It now answers the collector's token, and a plain session gets authelia's
`deny`. The design note that ruled the published vhost out for scraping
predates the collector's credential; it is replaced by the location's
own comment and a line on `swarm.grafana.domain`.
Part of #5205
Two comments said authelia's rule requires `group:operators`. The group
is `admins` (`operatorGroup` in swarm-authelia.nix), and the rule is now
an `operatorsOnly` pair.
The apex's `operatorsOnly` pair was guarded on
`deploy.swarm-controller.enable`, which is true only on the host that
runs the controller and with it the swarm UI's vhost. When authelia runs
on another host, that vhost still asks it, but the authelia host
rendered no rule for the apex at all, so every session reached the
swarm UI and its `/api/` through `default_policy = one_factor`.
The pair now renders whenever the swarm domain is set, like the bao UI's
pair. No vhost on a host that doesn't serve the apex asks about it, so
the extra rules there are inert. authelia 4.39.20's `validate-config`
accepts them on an authelia-only host's rendered access_control.
module-eval-swarm-otel-identity gains a case on the existing
`autheliaOnly` fixture (authelia without the controller): the apex has
exactly the allow+deny pair, the rule list is closed, and an account in
no group is refused `/api/agents` while `group:admins` passes.
Fixes#5120
authelia takes the first matching rule, and a rule with a `subject` does
not match a subject outside it. The swarm domain's and the bao UI's
`group:admins` rules had nothing after them, so an account in no group
skipped them and landed on `default_policy = one_factor`: any
authenticated account reached the swarm UI, its `/api/` (by cookie or by
bearer) and the bao UI.
Each of the two group rules is now an `operatorsOnly` pair: the group
allow, then a subject-less `deny` for the same domain, the shape
`collectorOnly` already gives the collector's metrics rules.
`default_policy` stays `one_factor`, which the forge and chat rely on.
module-eval-swarm-otel-identity gains two cases on `allLocal`: an
account in no group is refused the swarm domain (four paths) and the bao
UI while `group:admins` passes and the forge stays open; and a
predicate, `autheliaSubjectRulesClosed` in lib.nix, holds over the
rendered list: every subject-scoped allow is closed by a later
subject-less `deny` on its domain with no subject-less allow in between,
and nothing follows a domain-wide `deny` on its domain.
Fixes#5120
The socket scheme needs VictoriaMetrics 1.153.0 or later: older builds
create the socket 0600 whatever the umask, so nginx could not connect
and every consumer would get 502 with the module evaluating clean.
deploy.victoriametrics.package is an operator option, so an assertion
now refuses an older one, naming the option and the reason.
Turning DynamicUser off moves the state directory from
/var/lib/private/victoriametrics (behind a /var/lib/victoriametrics
symlink) to a plain /var/lib/victoriametrics. A oneshot ordered before
the store moves the old directory there and chowns it to the static
user when the plain path is the old symlink, empty, or missing, and does
nothing once the private directory is gone. Without it the persistent
container's history depends on systemd's own DynamicUser-off migration.
Refs #4815
The metrics store's container gets privateNetwork = true with no
interface but loopback. VictoriaMetrics listens on
unix:/run/swarm-victoriametrics/victoriametrics.sock, a host directory
(root:nginx 0770) bind-mounted into the container, and every vhost
location that forwarded to 127.0.0.1:8428 now forwards to that socket.
The store runs as a static user whose primary group is nginx's gid,
with UMask 0007, so the socket it creates is 0770 group nginx. This
relies on VictoriaMetrics >= 1.153.0 honouring the umask (the locked
nixpkgs ships 1.153.0). nixpkgs' readiness loop curls a TCP URL, so
postStart is replaced with the same loop over --unix-socket.
swarm.victoriametrics.port is removed (nothing binds it any more), with
a mkRemovedOptionModule entry, and dropped from swarm-otel's host
port-collision assertion. The container's resolver unit is dropped:
there is no interface to reach the bridge resolver through, and the
store resolves no names.
The collector scrape, the collector push and Grafana's datasource
already reach the store by its published domain, so they are unchanged.
Refs #4815
Refs #3560
DynamicUser implies PrivateTmp=disconnected (systemd.exec(5)), a tmpfs
of the unit's own rather than a private directory reachable from the
host namespace. The explicit setting now matches what the unit ran
with before it moved to a static user.
Refs #5235
With DynamicUser=true and RuntimeDirectoryPreserve=yes, systemd creates
the unit's runtime directory under /run/private/ (0700 root:root) and
symlinks /run/swarm-logs-auth to it (systemd.exec(5),
RuntimeDirectoryPreserve=). nginx runs as the unprivileged nginx user, so
its connect() to /run/swarm-logs-auth/auth.sock failed with EACCES, the
auth subrequest ended in a 502, and auth_request answered every agent
bearer on /select/logsql/ with a 500.
The unit now runs as a static swarm-logs-auth system user and group, the
shape swarm-controller already uses for its 0751 + Preserve socket dir.
RestrictSUIDSGID and RemoveIPC, which DynamicUser implied and the unit
did not set itself, are set explicitly.
module-eval-core-toggle gains a case that fails when any unit creating
the runtime directory of a unix: upstream of nginx combines DynamicUser
with RuntimeDirectoryPreserve. docs/process/gotchas.md records the trap.
Fixes#5235
VictoriaLogs' own metrics were scraped over loopback, declared only on
the host running the store, so a collector on any other host had no job
for it. Its target is now declared on every host as
`https://<swarm.victorialogs.domain>/metrics`, next to forgejo's and
authelia's, and the store's vhost serves `= /metrics` behind a bare
`auth_request` to authelia. The rules derived from
`publishedScrapeTargets` admit the collector's client there and deny every
other subject. The loopback entry is gone, so a single host still scrapes
the store once, now by name.
On that vhost `/metrics` was reachable through the `/` location by any
authelia session. The exact-match location now takes it, so a session
gets a `deny` on `/metrics` and nothing else on the name changes.
Part of #5205
No access rule covered the push route on `metrics.<swarm>`, so a request
there fell to `default_policy = one_factor`: any logged-in session cookie
for the swarm could write samples into VictoriaMetrics.
The name now gets two rules with no `resources`: the collector's client
at `one_factor`, then `deny`. Only the push route and `/metrics` on that
vhost send an authelia subrequest (the read API, Grafana's datasource
included, is served with no check), and the collector is the one caller
of both, so a domain-wide pair covers them with no path or query regex
to slip past. It replaces the per-path `/metrics` pair the published
target would otherwise derive for that name. Both rules sit ahead of the
apex and store-UI rules, and authelia matches `domain` exactly, so they
shadow no other name.
Authelia 4.39.20, read from source: a rule with no `domain` wildcard
matches with `strings.EqualFold` (access_control_domain.go), an empty
`resources` or `subject` list matches everything (access_control_rule.go),
and the first matching rule wins (authorizer.go).
Fixes#5218
The store's own /metrics was scraped over loopback, declared under
deploy.victoriametrics.enable, so a collector on another host never had
a job for it. Declare it as a published target on every host instead,
keyed on swarm.victoriametrics.domain, and serve it on the metrics vhost
behind the existing authz subrequest. Authelia's per-target rules (one
collector rule, then deny) now cover it too.
The loopback entry goes, so a single-host swarm scrapes the store once.
`swarmctl login` (and any `--swarm` verb) on a controller host failed TLS
with `UnknownIssuer` against the swarm's own gateway: its leaf chains to
the store's `pki` root, which no system store holds, and the only way
through was `--ca-file`/`SWARMCTL_CA_FILE` by hand.
The host's swarmctl wrapper now sets `SSL_CERT_FILE` to the bundle the
swarm-controller daemon already trusts (`/run/swarm-controller-ca/
trust-bundle.pem`: system CAs plus the swarm's root), when the gateway is
self-signed. swarmctl's reqwest/rustls resolves roots the same way the
daemon's does, so this is the path the daemon uses to reach the same
gateway. `--set` rather than a default, since a nix shell exports its own
`SSL_CERT_FILE`.
Pointing `SWARMCTL_CA_FILE` at the bundle instead would not work (#5243):
`--ca-file` adds each certificate as a root and fails the whole client on
one it can't parse.
Hosts without the controller (an operator laptop) still need `--ca-file`.
Refs #5239Fixes#5243
#5215 moved authelia's scrape to `publishedScrapeTargets` and removed
its loopback entry from the authelia module. It was rebased at merge
onto c06aa78b6, which had already moved that loopback entry into
glue-otel-scrape-targets.nix, so the entry survived there. Every host
running authelia and the collector, `singleHostSwarm` included, then
failed swarm-otel.nix's job-name assertion.
The glue now declares only grafana and nats. forge-placement's case
that pinned authelia on loopback now pins its absence, and
swarm-otel-identity gains a case reading the assertion itself on the
single-host fixture, since the option-level case cannot fail a
rebuild the way the assertion does.
Fixes#5247
Authelia 4.39.20 matches `resources` against the cleaned path plus
`?<query>` (authorization/types.go:94 NewObject, utils/url.go:11-34
URLPathFullClean), while nginx's `= /metrics` serves the request whatever
its query. `^/metrics$` therefore missed `/metrics?x=1`: both the collector
rule and the trailing `deny` were skipped and a plain session reached
`default_policy = one_factor`. The pattern is now `^<path>([?].*)?$` for
every published target.
Forgejo strips trailing slashes before routing (routers/common/
middleware.go stripSlashesMiddleware) and serves metrics without a token
when none is configured, so `forge.<swarm>/metrics/` reached its metrics
through the ungated `/` location. The forge vhost now answers
`= /metrics/` with 404; nginx merges slashes and decodes `%2F` before
matching, so that also covers `/metrics//` and `/metrics%2F`.
Part of #5205
Authelia's metrics were scraped over loopback, declared only on a host
running both authelia and the collector, so a split deploy collected none.
Its target is now declared on every host as
`https://<swarm.authelia.domain>/metrics`, next to forgejo's, and authelia's
vhost serves `= /metrics` behind the same collector auth subrequest as the
forge's. The loopback entry is gone, so a single host still scrapes
authelia once, now by name.
The forge-only `metricsRule` becomes `metricsRules`, derived from every
`publishedScrapeTargets` entry. Each target gets the collector-scoped
`one_factor` rule followed by an unconditional `deny`. The `deny` is
needed because authelia 4.39 skips a rule whose `subject` does not
match and moves on. With nothing after the collector rule, any other
authenticated session reached `default_policy = one_factor` and was
admitted. That was already true of the forge's `/metrics`, and this
change closes it there too.
Part of #5205
The wg-hive allowedTCPPorts rule gated on deploy.wireguard.enable was a
cross-service toggle read in swarm-nats.nix; it now lives in
glue-nats-wireguard.nix under the same condition and value.
`swarmctl login --swarm X` failed with "unexpected argument", and
`login --help` listed no `--swarm`: the flag was only accepted before the
verb. `--swarm` and `--ca-file` are now global, so they parse on either side
of the subcommand and show in every subcommand's help.
`--swarm` / `SWARMCTL_SWARM` accept a bare domain and prepend `https://`.
A value with a scheme is parsed as before, so `http://` is still refused.
On a swarm-controller host, the swarmctl wrapper sets
`SWARMCTL_LOGIN_DEFAULT_SWARM=https://<swarm.domain>` (`--set-default`).
Only `login` reads it, after `--swarm` and `SWARMCTL_SWARM`. The wrapper
does not set `SWARMCTL_SWARM`: that would send every socket verb (agent
create, schedule, inference, forge) through the gateway and make each need
a login. Other verbs resolve exactly as before.
No CA file is wired. The only CA material the module names is the
controller's assembled trust bundle (system CAs plus the hive anchor).
`--ca-file` hands every certificate in it to rustls-platform-verifier as
an extra root, and that fails the whole client on any one it cannot
parse, so pointing it at the system CAs is a risk nothing here can test.
The controller daemon trusts the same CA through `SSL_CERT_FILE`, which
skips unparsable certificates; using that for swarmctl is left open.
Machines that are not swarm-controller hosts (an operator laptop) get no
default; that case waits on an answer in the issue.
Fixes#5239
The swarm settings gain `runtime` (`api-key`, the default, or `acp`) and
an `acp` provider record: an https base URL, the provider's model, and
its prices per million tokens. A PUT now changes only the fields its
body names, so the settings form that sends model, effort and limit
leaves the runtime alone; `null` clears `effort` or `acp`. The body stays
strict, and the stored document keeps #5192's lenient read.
The controller sends the runtime in each run request. On `acp` the
classifier runs one prompt on a new `opencode acp` session per
classification through hive-runtime's AcpRuntime, in a scratch
directory, with a config rendered from the settings. opencode runs
`read` and other tools without asking, so OPENCODE_PERMISSION
{"*":"deny"} takes every tool away from the model, and a deny-all policy
refuses any request on top. The title call, which resends the prompt,
is turned off. The deadline cancels the turn, and a turn that does not
end within 15s of that is dropped, which also covers an agent stuck
before its prompt. The answer is the final assistant text, as ACP has
no result event.
The provider key is read per run from
swarm/services/review-classifier/acp-provider, which the bao UI's
admins login may create and update, and reaches opencode through
AcpRuntime::with_provider_key and the trampoline's allowlist.
Each ACP classification's cost is computed from the configured prices
and the reported tokens, and added to
hyperhive.review_classifier.cost.usage labelled repo, pr, model and
currency.
Part of #5096
The hive collector's `settings` was `{ extensions.file_storage = …; … }
// lib.optionalAttrs senderAuth { extensions.oauth2client/swarm = …; }`.
`//` is shallow, so whenever senderAuth held (every valid hive config)
the right-hand `extensions` replaced the left one and `file_storage`
vanished, while `service.extensions` still listed it. `otelcol validate`
then failed the config.yaml build:
Error: service::extensions: references extension "file_storage" which is not configured
Broken by 02831fb5e (PR #5178), which added `extensions.file_storage`
to the left-hand attrset; the `//` block for the auth extension predates
it (cb787997bd). Both extensions now live in one `extensions` attrset.
module-eval-hive-otel gains a case that every name in
`service.extensions` is defined under `extensions`; it evaluates false
on the parent commit and true here.
Fixes#5240
argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.
Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.
`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.
The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
All agents on a host reach the gateway from one address and share one
limit_req budget; a host boot is one cache miss per agent at once.
Defaults go from 5r/s burst 20 to 50r/s burst 500. The module-eval case
reads the option values.
Any request on logs.<swarm> carrying a well-formed swarm-agent bearer
made swarm-logs-auth do a bao cert login and KV read before it could
refuse, with no cap on the rate.
nginx now applies limit_req (zone swarm_logs_agent_auth, keyed by
client address, 5r/s burst 20 nodelay by default) on the internal
location that forwards to swarm-logs-auth, answering 403 when over,
since auth_request passes through only 401/403 and swarm-logs retries
a 401. The authelia path and the query location carry no limit.
swarm-logs-auth remembers bearers it has granted, keyed by SHA-256 of
the bearer, for --verdict-ttl seconds (deploy.victorialogs.authVerdictTtl,
60 by default), holding at most 4096 and evicting the oldest. Denials
are never remembered. A replaced or deleted queue secret therefore
keeps working at the log store for up to the TTL after its last check.
Closes#5190
The publisher's `enable` default read `deploy.authelia.enable` from the
publisher's own module (swarm-secret-publisher.nix). It now defaults to
`false`, and glue-secret-publisher-authelia.nix sets it `mkDefault true`
wherever authelia runs, so the effective value is unchanged. The option
loses its `defaultText`, and its description names the glue.
The `deploy.authelia.enable` read in `active` stays: every file the unit
copies is under authelia's `hostClientSecretDir`, so it is a requirement
of the unit, not a co-location toggle.
Part of #3742.
The forgejo entry in `swarm.otel.publishedScrapeTargets` was declared
under `deploy.forgejo.enable`, so it existed only in the forge host's
evaluation. Its two readers evaluate elsewhere: the collector renders
its prometheus jobs from its own host's list, and authelia registers
the collector client's audience from its own host's list. With the
forge, the collector and authelia on different hosts, the collector
had no forgejo job and authelia would have refused its token.
Declare the entry unconditionally in swarm-otel-service.nix, keyed on
`swarm.forge.domain`, which every host evaluates identically. A swarm
with no forge scrapes a name that answers nothing and shows a failing
job, the same trade the collector and grafana OIDC clients already make.
Single-host render is unchanged: the entry already existed there.
Part of #5205
Each store reader (forge's OIDC fetch and token publisher, matrix's OIDC
fetch, the hive collector's fetch, Grafana's OIDC fetch and admin password,
the queue's callout-xkey unit, the CI runner's token fetch, the swarm
collector's OIDC fetch) used to read `deploy.bao.enable` in its own module
to add `after`/`wants` on `container@<store machine>` and
`after`/`requires` on `swarm-bao-pki.service`. Per the #3742 ruling, that
cross-service read moves into glue-bao-readers-store-order.nix, which sets
the same edges when the store and the reader are both on this host and
restates each reader's own gate so no stub unit is rendered.
Rendered `after`/`wants`/`requires` are identical as sets to main on 16
fixtures, store here and elsewhere. New module-eval cases in existing
fixtures pin both arms.
Part of #3742.
The hive collector's secret-wait unit read deploy.authelia.enable to
order itself after authelia's container. That ordering is now set by
glue-hive-otel-authelia.nix, which restates the unit's own gate (otel's
`waits`). The rendered unit is unchanged.
The hive-otel module-eval fixture `hiveOtel` gains authelia, and a case
pins the edge with authelia here and its absence without.
Part of #3742.
Grafana runs with metrics.enabled and no basic_auth_*, so its Prometheus
exposition was served to any anonymous client of grafana.<swarm> through
the published vhost's `location "/"`. Add an exact-match `= /metrics`
location returning 403 there. The loopback `grafana-metrics` vhost and
the collector's scrape of it are unchanged.
Comments on the metricsPort option and the loopback vhost now state that
the published vhost refuses /metrics; the gateway doc says the same.
A module-eval case pins the refusing location and that `/` still proxies.
deploy.swarm-ui.enable and swarm.ui.enable are declared again as
nullOr bool, default null, and do nothing. Setting either adds an eval
warning that swarm-ui now runs wherever swarm-controller runs.
Remove deploy.swarm-ui.enable. Every reader (the swarm-ui vhost, the
authelia operator rule for the swarm apex, hive-network's
anyServiceEnabled) now reads deploy.swarm-controller.enable, and the
controller's webhook public URL is gated on the swarm domain alone,
since the vhost it names exists wherever the daemon does.
Both spellings of the old switch (deploy.swarm-ui.enable and the
already-renamed swarm.ui.enable) become mkRemovedOptionModule entries,
so a host that still sets either fails evaluation with a message
naming the controller toggle instead of silently changing.
Ruling (mara, #3742): "swarm-ui always runs wherever controller runs".
Part of #3742
`swarm-bao-agent-pki` carried its own copy of the mtime-vs-start compare,
with the same `stat -c %.6Y | tr -d .` read: on a comma-locale host the
`-le` errored and, under `seal = "pkcs11"`, openbao was restarted on
every run of the unit.
The compare moves out of `refresh_unit` into `written_after_start`
(0 newer, 1 not newer, 2 unreadable and logged at `err`), and both
`refresh_unit` and the agent-pki unit call it. The agent-pki unit cannot
call `refresh_unit` itself: it restarts with `restart`, not
`try-restart`, runs a fail-closed check of the bundle between the
compare and the restart, and under shamir prints the step instead of
restarting. It acts only on 0, so an unreadable time skips the restart
as `refresh_unit` does.
`script-test-refresh-consumer` covers the three return codes in the
agent-pki call shape; `module-eval-bao-basics` asserts both seal arms
decide on that call and that no `%.6Y` read is left in the unit.
Refs #5230
`refresh_unit` read the credential file's mtime with
`stat -c %.6Y | tr -d .`. coreutils prints that fraction with the
locale's decimal separator, so on a host whose locale uses a comma
(muede-lpt2: `1791232888,193159`) `tr` stripped nothing, the `-le`
against the unit's start errored, the "unchanged" early return was
skipped, and every run try-restarted the consumer. Every unit on a
store-refresh timer was restarted every ~10 minutes; for
`swarm-bao-ci-runner` that is `gitea-runner-swarm`, which killed every
CI job running at the time.
The mtime is now read with `date -r … +%s%6N`, the same digit-only
format the start time already used (`date -d … +%s%6N` on systemd's
fixed English timestamp), so neither side has a separator for a locale
to change.
A value that still does not read as digits — an unparsable start time,
a missing file — now restarts nothing and logs at `err`. Skipping on
doubt rather than restarting on doubt: the file's mtime is kept, so a
genuinely changed credential still restarts its consumer on the first
run that can read both times, whereas restarting on doubt restarts it
on every run, which is this bug. Both reads also fall back to empty
instead of aborting the caller's `set -e` script.
`script-test-refresh-consumer` runs the rendered helper under
`de_DE.UTF-8` against a stub `systemctl`, and checks first that `stat`
really prints a comma there. The helper on main fails 14 of its
assertions.
Fixes#5230
swarm-logs reads `swarm/agents/<agent>/queue` from the swarm secret
store with the agent's own store certificate (the read hive-agent makes
for its queue connection) and presents it as
`Bearer swarm-agent.<agent>.<secret>`, which the gateway's
swarm-logs-auth checks. On a 401 it reads the secret again and retries
once, so a secret re-minted between the read and the check does not
fail the query.
The OIDC mint against the hive's shared `hive-<h>-agent` client is gone
from swarm-logs, along with the unused `HIVE_AGENT_OIDC_CA_FILE` anchor.
logs.nix now gates on the store address instead of the queue's token
endpoint and wraps the binary with the paths queue-identity.nix gives
the harness; the host forwards the query URL wherever it forwards a
store identity.
The shared client is still delivered into containers; removing that is
Part of #4459.
A new `auth_request` responder on the log store's host. nginx sends it
every `/select/logsql/` request whose `Authorization` is
`Bearer swarm-agent.<agent>.<secret>`; it answers 200 when the secret is
the one stored at `swarm/agents/<agent>/queue` (`swarm-agent-verify`),
401 otherwise. Every other bearer still goes to authelia, so agents
still presenting the hive's shared OIDC client keep working.
`auth_request` takes a fixed URI, so the split is a `map` on
`$http_authorization` plus an internal location that rewrites to the
map's pick. Checked against nginx 1.30.5 with stub upstreams: an agent
token only the stub validator accepts gets 200, a token only the stub
authelia accepts gets 200, every wrong one 401, and the internal
locations 404 from outside.
The responder serves a unix socket in its own traverse-only runtime
directory and reads the store under its own leaf
(`swarm-logs-auth`), whose policy is `swarm/agents/+/queue`, read-only.
glue-bao-tls mints the leaf where the store runs; elsewhere the
operator places it, and eval refuses a log store host without one.
Part of #4459.
The metrics vhost's `/` proxied every path to VictoriaMetrics with no
auth_request, so anything that could reach metrics.<swarm> could write
(`/api/v1/import`, `/api/v1/write`, ...) and delete
(`/api/v1/admin/tsdb/delete_series`) series. Only the OTLP push route
was gated.
`/` now answers 403. The read routes Grafana's Prometheus datasource
and vmui use are forwarded by exact or `^~` prefix location, each with
the matched path as its proxy_pass URI. The bare domain, which the
swarm UI's Metrics quick link opens, redirects to vmui instead of the
store's index page. The OTLP push location is unchanged.
Closes#5186
The metrics datasource defaulted to http://127.0.0.1:<port>, which only
reached the store when Grafana and VictoriaMetrics ran on the same host.
Swarm-tier services may not assume colocation (#4499), so the default is
now https://<swarm.victoriametrics.domain>, the gateway vhost the store's
host already serves. That vhost's query routes carry no auth_request, and
the Grafana container already trusts the hive CA for its authelia token
exchange, so no credential is added.
The logs datasource stays on loopback: the log store's query route needs
a bearer token Grafana does not hold yet.
Part of #3742 (item 4, metrics).
Seven host oneshots fetch a secret from the swarm secret store into a file
their consumer loads at start: swarm-bao-grafana-oidc, swarm-bao-otel-oidc,
swarm-bao-matrix-oidc, swarm-bao-ci-runner, swarm-bao-forge-oidc,
swarm-bao-hive-otel-oidc and swarm-bao-forwarder-oidc. Each was
RemainAfterExit with no timer, so it ran at boot or deploy and never again:
a secret re-minted in the store reached its consumer only through an
operator restarting the fetch.
lib/store-refresh-timer.nix gives each one a timer (OnUnitInactiveSec 10min,
RandomizedDelaySec 1min, the agent's own store re-read interval), and each
drops RemainAfterExit so a timer start is not a no-op. Each script already
writes the file only when secret_differs, and refresh_consumer/refresh_unit
restart the consumer only when the file is newer than its last start, so a
run reading an unchanged value restarts nothing.
Two docs said these units do not run on a timer; credentials.md's
ci-runner row said the runner token is not re-pulled.
Part of #5017 (slice T).
A hive key becomes a certificate subject and a directory name, but only the
reserved-name and fragment guards looked at it, so `a/b` or `../x` evaluated
and misbehaved later. Assert at eval time that every key is 1-63 characters
of [a-z0-9-], the rule `Ident` applies in hive-types, and name the offending
keys in the message.
The store host's `swarm-bao-pki` now also signs, for every hive in
`swarm.hives` other than its own, the reader (`<hive>`), queue-agent
(`<queueAgentCommonNamePrefix>-<hive>`) and gateway-leaf
(`<hiveLeafCommonNamePrefix>-<hive>`) leaves, into
/var/lib/swarm-bao-pki/hives/<hive>/ with the basenames the own hive's
leaves use. Same CA, same `signLeaf`, idempotent on absence like every
other leaf in the unit. `signLeaf` now takes the CA from the top of the
PKI directory rather than from the leaf's directory; every existing
caller passed that directory, so their output is unchanged.
The operator still copies the files to the other hive's host and names
them in config; only the openssl step against the CA goes away
(mara on #5016, c95685: "ok mint").
Closes#5016
Follow-ups from the review of #5142.
A `Done` answer with fewer or more rows than the classifications sent, or
none, now posts the no-verdict comment instead of the classifier's overall
verdict, so a short answer can't approve.
Requests are accepted per repo, PR and head into a set bounded at the two
running plus eight waiting. A request for a key already running or waiting
starts nothing and logs at info; a request with every place taken starts no
run and posts a no-verdict comment saying so. Before, every delivery spawned
a task waiting on the semaphore, so toggling the review request queued one
paid run per toggle.
Table cells get a zero-width space after `#`, `!`, `://` and `www.`, so a
model's reason can't render an issue or PR reference or a bare URL as a link.
The secret publisher's policy denies the classifier's forge token leaf, the
way #5139 denied Grafana's admin password, with a module-eval-bao-grants arm
pinning it. The controller posts reviews with whatever token is there, so a
publisher write picked the account reviews come from.
Closes#5143, part of #5096.
The docs tell the operator to write
`secret/swarm/services/review-classifier/anthropic-api-key` by hand in
the bao UI, but the UI's OIDC login for authelia's `admins` group lands
on `swarm-operator-viewer`, which is list+read on `secret/metadata/*`
and nothing under `secret/data/`. bao refuses the write, so an admin
can't set the key and every classifier run fails with "no API key".
The viewer policy gains `create` and `update` on that one leaf, no
`read`: a UI session still never sees a secret value. The leaf is a
named binding, shared with the classifier's reader policy.
module-eval pins the viewer's two stanzas and that no `data/` grant
carries `read`. docs/swarm/secrets.md and services.md describe the
write.
Closes#5148
Requesting the forge account `review-classifier` as a reviewer now runs the
repo's rules against the PR and posts the result as that account's review,
on the head commit the run read. A push starts no run; only a fresh review
request does.
- An instance-wide `pull_request_review_request` system hook delivers to
`/webhook/forge/review-request`. A request for any other reviewer, or a
removed request, does nothing.
- The controller reads the PR, loads the rules with the same reader as
`GET /api/review-classifier/rules`, sends one request on the try-out
subject, and posts APPROVED or REQUEST_CHANGES with the table as the body.
- With no clean verdict (PR or rules don't load, the run fails, or any
classification has no answer) it posts a COMMENT saying why, never an
approval. The event is one function, `no_clean_verdict_event`.
- Table cells are flattened, markup-escaped, and every `@` gets a
zero-width space, so a model's reason can't break the table or mention
anyone.
- At most two requested reviews run at once; the rest wait in memory.
- The account is minted the way hive accounts are: a five-minute pass
queues a `MintClassifierForgeToken` node when the token at
`swarm/services/review-classifier/forge/token` is missing or stale. The
token is `write:repository`. The controller's bao policy gains that leaf
(create/read/update), pinned in module-eval bao-grants. The classifier
never sees the token.
The account gets no repo access automatically; a repo admin adds it by
hand. A re-request retires the account's earlier review: Forgejo clears
`official` on a reviewer's earlier reviews when it is requested again
(models/issues/review.go AddReviewRequest, v16.0.5), and approvals and
rejection blocks count official reviews only.
Refs #5137
docs/swarm/sso.md still said Grafana ships admin/admin and that a dead
provider locks out the built-in admin. The password is minted into the
store and the API still takes basic auth for it, so the paragraph now says
where an operator reads it in an emergency. secrets.md says how to rotate
the password and to deploy the store's host first, and no longer calls SSO
the only way in.
The secret publisher's `services/*` write covered the admin-password
leaf, so a file-copier could set the password Grafana applies at its next
start. The leaf gets the same exact-path deny as the callout seeds, with
an arm in module-eval-bao-grants.
