Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
19c9f0c815 | ||
|
|
4573865745 | ||
|
|
a8bc084262 | ||
|
|
645fd0d56c | ||
|
|
b2596097d8 |
10 changed files with 924 additions and 7 deletions
2
Cargo.lock
generated
2
Cargo.lock
generated
|
|
@ -4605,10 +4605,12 @@ dependencies = [
|
||||||
"hive-jobq",
|
"hive-jobq",
|
||||||
"hive-jobq-wire",
|
"hive-jobq-wire",
|
||||||
"hive-types",
|
"hive-types",
|
||||||
|
"hmac 0.13.0",
|
||||||
"problem_details",
|
"problem_details",
|
||||||
"reqwest 0.13.1",
|
"reqwest 0.13.1",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
|
"sha2 0.11.0",
|
||||||
"swarm-authelia-bridge-sock",
|
"swarm-authelia-bridge-sock",
|
||||||
"swarm-queue-client",
|
"swarm-queue-client",
|
||||||
"tokio",
|
"tokio",
|
||||||
|
|
|
||||||
|
|
@ -47,6 +47,10 @@ hive-c0re maintains the local clone at
|
||||||
The endpoint is protected by an auto-generated HMAC secret that
|
The endpoint is protected by an auto-generated HMAC secret that
|
||||||
hive-c0re verifies on every delivery.
|
hive-c0re verifies on every delivery.
|
||||||
|
|
||||||
|
On a swarm, a **second** hook on the same repo points at the
|
||||||
|
swarm controller — see `docs/swarm/README.md` § Swarm-wide forge
|
||||||
|
webhooks. Both are expected; neither should be deleted.
|
||||||
|
|
||||||
2. **Periodic pull** — a background task in `hive-c0re::main`
|
2. **Periodic pull** — a background task in `hive-c0re::main`
|
||||||
pulls on a fixed cadence as a fallback (webhook missed, c0re
|
pulls on a fixed cadence as a fallback (webhook missed, c0re
|
||||||
restarted between pushes). The pull is best-effort — a failure
|
restarted between pushes). The pull is best-effort — a failure
|
||||||
|
|
|
||||||
|
|
@ -468,6 +468,23 @@ sends to the swarm's snapshot store — see
|
||||||
[`docs/snapshot-store.md`](snapshot-store.md) for what a pushed
|
[`docs/snapshot-store.md`](snapshot-store.md) for what a pushed
|
||||||
snapshot contains and how the store authenticates a sender.
|
snapshot contains and how the store authenticates a sender.
|
||||||
|
|
||||||
|
## `/var/lib/swarm-controller/` (swarm-controller host only)
|
||||||
|
|
||||||
|
Only present on the one host running
|
||||||
|
`services.hyperhive.swarm.controller.enable`. systemd `StateDirectory=`,
|
||||||
|
so it survives restarts and redeploys.
|
||||||
|
|
||||||
|
- `webhook-secret` — the HMAC key the swarm's forge webhooks are signed
|
||||||
|
with. **Keep it.** It is handed to Forgejo when a hook is registered,
|
||||||
|
so replacing the file means every subsequent delivery fails
|
||||||
|
verification until the hook is re-registered with the new value. It is
|
||||||
|
generated automatically on first start; there is nothing to configure.
|
||||||
|
|
||||||
|
If the file is unreadable at startup the daemon still starts and logs
|
||||||
|
`webhook secret unavailable`; the webhook endpoint then answers 503
|
||||||
|
rather than accepting deliveries it cannot verify. Everything else the
|
||||||
|
controller serves is unaffected.
|
||||||
|
|
||||||
## Run-time dirs
|
## Run-time dirs
|
||||||
|
|
||||||
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
||||||
|
|
|
||||||
|
|
@ -349,6 +349,30 @@ list, which would look like a silent swarm rather than a controller that
|
||||||
cannot see. The body says which. Status survives a controller restart:
|
cannot see. The body says which. Status survives a controller restart:
|
||||||
it is stored in the queue, not in the daemon.
|
it is stored in the queue, not in the daemon.
|
||||||
|
|
||||||
|
### Swarm-wide forge webhooks
|
||||||
|
|
||||||
|
At startup the controller registers two Forgejo hooks pointing at
|
||||||
|
itself — a `push` hook on `internal/knowledge` and a `pull_request` hook
|
||||||
|
on the `agent-configs` org, both under
|
||||||
|
`https://<swarm.ui.domain>/webhook/forge/`.
|
||||||
|
|
||||||
|
**You will see two hooks where there used to be one**, and that is the
|
||||||
|
intended state for now: each hive still registers and receives its own,
|
||||||
|
and the controller's is an additional copy that it currently logs and
|
||||||
|
nothing more. **Do not delete either.** Removing a hive's hook stops that
|
||||||
|
hive acting on knowledge pushes and config PRs; removing the
|
||||||
|
controller's just gets recreated on its next start.
|
||||||
|
|
||||||
|
Nothing to configure. The hooks are registered only when this host also
|
||||||
|
serves the swarm UI vhost — that is what publishes the endpoint, and a
|
||||||
|
hook the forge cannot reach would collect failed deliveries while
|
||||||
|
looking healthy. The HMAC secret is generated on first start and kept
|
||||||
|
(see [`docs/persistence.md`](../persistence.md)).
|
||||||
|
|
||||||
|
To check it is working, push to `internal/knowledge` and look for
|
||||||
|
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
|
||||||
|
refused delivery logs `webhook: refused delivery` with the reason.
|
||||||
|
|
||||||
## Cross-references
|
## Cross-references
|
||||||
|
|
||||||
- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
|
- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,7 @@ let
|
||||||
|
|
||||||
natsCfg = config.services.hyperhive.swarm.nats;
|
natsCfg = config.services.hyperhive.swarm.nats;
|
||||||
forgeCfg = config.services.hyperhive.swarm.forge;
|
forgeCfg = config.services.hyperhive.swarm.forge;
|
||||||
|
uiCfg = config.services.hyperhive.swarm.ui;
|
||||||
|
|
||||||
# The controller's own OAuth2 client. It is NOT a hive: the per-hive
|
# The controller's own OAuth2 client. It is NOT a hive: the per-hive
|
||||||
# clients the roster issues belong to hives, and the responder's client
|
# clients the roster issues belong to hives, and the responder's client
|
||||||
|
|
@ -85,6 +86,33 @@ let
|
||||||
SWARM_CONTROLLER_FORGE_TOKEN_FILE = "%d/forge-token";
|
SWARM_CONTROLLER_FORGE_TOKEN_FILE = "%d/forge-token";
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# How the forge must address this controller to deliver a swarm-wide
|
||||||
|
# webhook. Gated on the swarm UI being served *here*, because that module
|
||||||
|
# is what declares the vhost and the `/webhook/forge/` location inside it:
|
||||||
|
# absent, nothing outside this host can reach the endpoint.
|
||||||
|
#
|
||||||
|
# 🔑 The gate is the point, not a detail. The daemon registers hooks only
|
||||||
|
# when this is set, and a hook whose `target_url` nothing answers is worse
|
||||||
|
# than no hook at all — forgejo keeps the registration, marks every
|
||||||
|
# delivery failed, and the hook still reads as configured.
|
||||||
|
#
|
||||||
|
# ⚠️ Correct ONLY while the vhost and this daemon share a host, and they do
|
||||||
|
# by construction: the UI's `/api/` location proxies
|
||||||
|
# `http://unix:<socketPath>`, a path that resolves nowhere else, and this
|
||||||
|
# daemon binds no TCP address at all (see the header). `swarm.ui.enable`
|
||||||
|
# therefore is not a guess about *some* host publishing the endpoint — it is
|
||||||
|
# the flag that declares that vhost, on the box holding the socket.
|
||||||
|
#
|
||||||
|
# ⇒ The day this daemon grows a TCP listener so another host can front it,
|
||||||
|
# THIS LINE SILENTLY STOPS BEING RIGHT: the env goes unset on the controller
|
||||||
|
# host, registration quietly does not happen, and nothing errors. That is
|
||||||
|
# the moment to add an explicit `publicUrl` option — not before, because
|
||||||
|
# until then there is exactly one derivable answer and an option would only
|
||||||
|
# be a second place to get it wrong.
|
||||||
|
webhookEnv = lib.optionalAttrs uiCfg.enable {
|
||||||
|
SWARM_CONTROLLER_PUBLIC_URL = "https://${uiCfg.domain}";
|
||||||
|
};
|
||||||
|
|
||||||
# Not a secret to deliver — `swarm-authelia-bridge`'s own bearer check
|
# Not a secret to deliver — `swarm-authelia-bridge`'s own bearer check
|
||||||
# is satisfied by THIS daemon's existing queue OIDC identity
|
# is satisfied by THIS daemon's existing queue OIDC identity
|
||||||
# (`queueEnv` above): "one identity per principal" already covers this,
|
# (`queueEnv` above): "one identity per principal" already covers this,
|
||||||
|
|
@ -594,6 +622,7 @@ in
|
||||||
}
|
}
|
||||||
// queueEnv
|
// queueEnv
|
||||||
// forgeEnv
|
// forgeEnv
|
||||||
|
// webhookEnv
|
||||||
// authBridgeEnv;
|
// authBridgeEnv;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -188,6 +188,27 @@ in
|
||||||
proxyPass = "http://unix:${controllerCfg.socketPath}:";
|
proxyPass = "http://unix:${controllerCfg.socketPath}:";
|
||||||
extraConfig = swarmAuthRequest;
|
extraConfig = swarmAuthRequest;
|
||||||
};
|
};
|
||||||
|
# ⚠️ THE ONE LOCATION ON THIS VHOST WITH NO `swarmAuthRequest`,
|
||||||
|
# and that is deliberate rather than an omission.
|
||||||
|
#
|
||||||
|
# A forge webhook is a machine POST carrying an HMAC signature
|
||||||
|
# and no session cookie. An authelia auth-request subrequest
|
||||||
|
# authenticates a *browser session*; there is nothing here for it
|
||||||
|
# to check, so guarding this location would not make it safer, it
|
||||||
|
# would make it permanently unreachable.
|
||||||
|
#
|
||||||
|
# What replaces it: swarm-controller verifies the
|
||||||
|
# `X-Hub-Signature-256` HMAC over the raw body before looking at
|
||||||
|
# anything else, and answers 401 on any mismatch. That check is
|
||||||
|
# the access control for this path — see the `webhook` module.
|
||||||
|
#
|
||||||
|
# Scoped to `/webhook/forge/` rather than `/webhook/` so the
|
||||||
|
# carve-out is exactly as wide as the endpoint that justifies it:
|
||||||
|
# a future `/webhook/<something-else>` does not inherit the
|
||||||
|
# bypass by living under a shared prefix.
|
||||||
|
"/webhook/forge/" = {
|
||||||
|
proxyPass = "http://unix:${controllerCfg.socketPath}:";
|
||||||
|
};
|
||||||
# Swagger UI: same "nginx hosts the themed dist straight from
|
# Swagger UI: same "nginx hosts the themed dist straight from
|
||||||
# the store, only /api/openapi.json is dynamic" shape as the
|
# the store, only /api/openapi.json is dynamic" shape as the
|
||||||
# per-hive gateway's `swaggerUiLocations`.
|
# per-hive gateway's `swaggerUiLocations`.
|
||||||
|
|
|
||||||
|
|
@ -38,6 +38,12 @@ problem_details = { version = "0.9.0", features = ["axum"] }
|
||||||
# same shape `hive-c0re/src/job_queue/scheduler.rs` uses over its own graph.
|
# same shape `hive-c0re/src/job_queue/scheduler.rs` uses over its own graph.
|
||||||
hive-jobq.workspace = true
|
hive-jobq.workspace = true
|
||||||
hive-jobq-wire.workspace = true
|
hive-jobq-wire.workspace = true
|
||||||
|
# The forge webhook HMAC (`webhook.rs`). Kept in this crate rather than
|
||||||
|
# shared with hive-c0re's equivalent: c0re's copy is scheduled to be deleted
|
||||||
|
# with its webhook routes once registration moves here, so the second holder
|
||||||
|
# is departing, not arriving — see that module's docs.
|
||||||
|
hmac.workspace = true
|
||||||
|
sha2.workspace = true
|
||||||
# Validates `POST /api/agents`' `name` before it becomes `agent`/`repo`
|
# Validates `POST /api/agents`' `name` before it becomes `agent`/`repo`
|
||||||
# everywhere downstream — see `create_agent`'s doc comment for why this is
|
# everywhere downstream — see `create_agent`'s doc comment for why this is
|
||||||
# defense-in-depth, not the only gate (per an argus review finding).
|
# defense-in-depth, not the only gate (per an argus review finding).
|
||||||
|
|
|
||||||
|
|
@ -19,10 +19,14 @@ use anyhow::{Context, Result};
|
||||||
use forgejo_api::structs::{
|
use forgejo_api::structs::{
|
||||||
AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
|
AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
|
||||||
ChangeFileOperationOperation, ChangeFilesOptions, CreateBranchProtectionOption,
|
ChangeFileOperationOperation, ChangeFilesOptions, CreateBranchProtectionOption,
|
||||||
CreateRepoOption, RepoGetContentsQuery,
|
CreateHookOption, CreateHookOptionConfig, CreateHookOptionType, CreateRepoOption,
|
||||||
|
RepoGetContentsQuery,
|
||||||
};
|
};
|
||||||
use forgejo_api::{ApiErrorKind, Auth, Forgejo, ForgejoError};
|
use forgejo_api::{ApiErrorKind, Auth, Forgejo, ForgejoError};
|
||||||
use reqwest::StatusCode;
|
use reqwest::StatusCode;
|
||||||
|
use std::collections::BTreeMap;
|
||||||
|
|
||||||
|
use crate::webhook::DeliveryKind;
|
||||||
|
|
||||||
/// The forge org that owns agent repos. Same org `hive-c0re::forge`
|
/// The forge org that owns agent repos. Same org `hive-c0re::forge`
|
||||||
/// already uses for its own single-hive `CreateRepo` path — this is
|
/// already uses for its own single-hive `CreateRepo` path — this is
|
||||||
|
|
@ -38,6 +42,20 @@ pub const AGENTS_ORG: &str = "agents";
|
||||||
/// [`Client::create_repo`]'s doc comment.
|
/// [`Client::create_repo`]'s doc comment.
|
||||||
const OPERATORS_TEAM: &str = "operators";
|
const OPERATORS_TEAM: &str = "operators";
|
||||||
|
|
||||||
|
/// The org owning per-agent config repos, and where the `pull_request` hook
|
||||||
|
/// lives. Same value as `hive-c0re::forge::CONFIG_ORG` — one forge, one org.
|
||||||
|
///
|
||||||
|
/// ⚠️ Duplicated across the crate boundary (this crate deliberately does not
|
||||||
|
/// depend on `hive-c0re`), so nothing makes the two fail together. The
|
||||||
|
/// failure mode if they drift is quiet: the hook is created on an org that
|
||||||
|
/// exists, forgejo reports it healthy, and it simply never fires.
|
||||||
|
const CONFIG_ORG: &str = "agent-configs";
|
||||||
|
|
||||||
|
/// The hive-wide knowledge repo, where the `push` hook lives. Same values as
|
||||||
|
/// `hive-c0re::workers::knowledge::{ORG, REPO}`.
|
||||||
|
const KNOWLEDGE_ORG: &str = "internal";
|
||||||
|
const KNOWLEDGE_REPO: &str = "knowledge";
|
||||||
|
|
||||||
/// The env vars `swarm-controller.nix`'s `forgeEnv` sets. Named here
|
/// The env vars `swarm-controller.nix`'s `forgeEnv` sets. Named here
|
||||||
/// once so [`Client::from_env`] and the nix module can't drift silently
|
/// once so [`Client::from_env`] and the nix module can't drift silently
|
||||||
/// — a rename on one side without the other fails loudly (env var
|
/// — a rename on one side without the other fails loudly (env var
|
||||||
|
|
@ -325,6 +343,136 @@ impl Client {
|
||||||
Err(e) => Err(e).with_context(|| format!("check for {path} in {AGENTS_ORG}/{repo}")),
|
Err(e) => Err(e).with_context(|| format!("check for {path} in {AGENTS_ORG}/{repo}")),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Register the swarm-wide hooks against this controller, so a real
|
||||||
|
/// forge event reaches [`crate::webhook`] instead of the endpoint only
|
||||||
|
/// being reachable by hand.
|
||||||
|
///
|
||||||
|
/// **These are registered ALONGSIDE the per-hive hooks, not instead of
|
||||||
|
/// them.** Every hive keeps receiving and acting on its own deliveries
|
||||||
|
/// exactly as today; the controller receives a copy and (for now) logs
|
||||||
|
/// it. Taking the hive-side registration away is a later step, and it
|
||||||
|
/// has to be later: fan-out swarm→hive does not exist yet, so a hook
|
||||||
|
/// moved now would point at a receiver that forwards nowhere — silent on
|
||||||
|
/// both sides, indistinguishable from no activity.
|
||||||
|
///
|
||||||
|
/// ⛔ **No stale-hook deletion arm, unlike the two per-hive registrars
|
||||||
|
/// this otherwise mirrors.** Their arm deletes hooks matching their own
|
||||||
|
/// path with a foreign base; copying it here would delete the hives'
|
||||||
|
/// live hooks, which are not stale — they are the path still in
|
||||||
|
/// production. The controller only ever adds its own.
|
||||||
|
///
|
||||||
|
/// Idempotent: an existing hook with the same `target_url` is left
|
||||||
|
/// alone, so this is safe on every boot.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
///
|
||||||
|
/// Returns the first failure. A listing failure is not fatal — it falls
|
||||||
|
/// through to the create attempt, which is idempotent server-side by way
|
||||||
|
/// of the already-exists fold, the same best-effort shape `hive-c0re`
|
||||||
|
/// uses.
|
||||||
|
pub async fn ensure_swarm_webhooks(&self, public_base: &str, secret: &str) -> Result<()> {
|
||||||
|
for kind in DeliveryKind::ALL {
|
||||||
|
let target_url = kind.target_url(public_base);
|
||||||
|
let (event, scope) = match kind {
|
||||||
|
DeliveryKind::Knowledge => (
|
||||||
|
"push",
|
||||||
|
HookScope::Repo {
|
||||||
|
org: KNOWLEDGE_ORG,
|
||||||
|
repo: KNOWLEDGE_REPO,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
DeliveryKind::ConfigPr => ("pull_request", HookScope::Org { org: CONFIG_ORG }),
|
||||||
|
};
|
||||||
|
self.ensure_hook(&scope, &target_url, event, secret)
|
||||||
|
.await
|
||||||
|
.with_context(|| format!("register swarm webhook {target_url}"))?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One hook, idempotently. Split out of [`Self::ensure_swarm_webhooks`]
|
||||||
|
/// so the org-scoped and repo-scoped calls do not each grow their own
|
||||||
|
/// copy of the list-then-create logic.
|
||||||
|
async fn ensure_hook(
|
||||||
|
&self,
|
||||||
|
scope: &HookScope<'_>,
|
||||||
|
target_url: &str,
|
||||||
|
event: &str,
|
||||||
|
secret: &str,
|
||||||
|
) -> Result<()> {
|
||||||
|
match scope.list_hook_urls(&self.api).await {
|
||||||
|
Ok(urls) => {
|
||||||
|
if urls.iter().any(|u| u == target_url) {
|
||||||
|
tracing::debug!(%target_url, "swarm forge: webhook already registered");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
// Best-effort, same as the per-hive registrars: a forge that
|
||||||
|
// cannot be listed may still accept a create, and a
|
||||||
|
// duplicate create is folded into success below.
|
||||||
|
tracing::debug!(error = %e, %target_url, "swarm forge: listing hooks failed; attempting create");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut additional = BTreeMap::new();
|
||||||
|
additional.insert("secret".to_owned(), secret.to_owned());
|
||||||
|
let hook = CreateHookOption {
|
||||||
|
active: Some(true),
|
||||||
|
authorization_header: None,
|
||||||
|
branch_filter: None,
|
||||||
|
config: CreateHookOptionConfig {
|
||||||
|
content_type: "json".to_owned(),
|
||||||
|
url: url::Url::parse(target_url)
|
||||||
|
.with_context(|| format!("parse webhook target url {target_url}"))?,
|
||||||
|
additional,
|
||||||
|
},
|
||||||
|
events: Some(vec![event.to_owned()]),
|
||||||
|
r#type: CreateHookOptionType::Forgejo,
|
||||||
|
};
|
||||||
|
match scope.create_hook(&self.api, hook).await {
|
||||||
|
Ok(()) => {
|
||||||
|
tracing::info!(%target_url, %event, "swarm forge: webhook registered");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
Err(e) if is_already_exists(&e) => {
|
||||||
|
tracing::debug!(%target_url, "swarm forge: webhook already present");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
Err(e) => Err(e.into()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Where a hook lives. The knowledge hook is repo-scoped and the config-PR
|
||||||
|
/// hook is org-scoped, mirroring exactly where the per-hive registrars put
|
||||||
|
/// theirs — a hook on the wrong scope would never fire, and forgejo would
|
||||||
|
/// report that as a perfectly healthy hook with no deliveries.
|
||||||
|
enum HookScope<'a> {
|
||||||
|
Repo { org: &'a str, repo: &'a str },
|
||||||
|
Org { org: &'a str },
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HookScope<'_> {
|
||||||
|
/// The `url` config value of every hook currently on this scope.
|
||||||
|
async fn list_hook_urls(&self, api: &Forgejo) -> Result<Vec<String>, ForgejoError> {
|
||||||
|
let hooks = match self {
|
||||||
|
Self::Repo { org, repo } => api.repo_list_hooks(org, repo).all().await?,
|
||||||
|
Self::Org { org } => api.org_list_hooks(org).send().await?,
|
||||||
|
};
|
||||||
|
Ok(hooks
|
||||||
|
.iter()
|
||||||
|
.filter_map(|h| h.config.as_ref()?.get("url").cloned())
|
||||||
|
.collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn create_hook(&self, api: &Forgejo, hook: CreateHookOption) -> Result<(), ForgejoError> {
|
||||||
|
match self {
|
||||||
|
Self::Repo { org, repo } => api.repo_create_hook(org, repo, hook).await.map(drop),
|
||||||
|
Self::Org { org } => api.org_create_hook(org, hook).await.map(drop),
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Base64-encode `content` for a forgejo content-API call — every file
|
/// Base64-encode `content` for a forgejo content-API call — every file
|
||||||
|
|
|
||||||
|
|
@ -3,11 +3,14 @@
|
||||||
//! `services.hyperhive.swarm.controller.enable` on, and serves HTTP over a
|
//! `services.hyperhive.swarm.controller.enable` on, and serves HTTP over a
|
||||||
//! unix socket that the hive-gateway's nginx proxies to.
|
//! unix socket that the hive-gateway's nginx proxies to.
|
||||||
//!
|
//!
|
||||||
//! Holds one piece of read-only state: the swarm's hive directory, loaded
|
//! Configuration is read-only and loaded once at startup from env vars the
|
||||||
//! once at startup from an env var the NixOS module sets
|
//! NixOS module sets (`services.hyperhive.swarm.controller`) — see
|
||||||
//! (`services.hyperhive.swarm.controller`) — see `load_hives`. Still no
|
//! `load_hives`. A config change means a redeploy, same as every other
|
||||||
//! persistence and no writes; a config change means a redeploy, same as
|
//! option this process reads.
|
||||||
//! every other option this process reads.
|
//!
|
||||||
|
//! The one thing it does persist is `webhook-secret` under its
|
||||||
|
//! `StateDirectory` (see `webhook`), because that key is handed to Forgejo
|
||||||
|
//! at registration and so cannot be regenerated per boot.
|
||||||
//!
|
//!
|
||||||
//! Distinct from `hive-c0re`, which is per-hive: c0re owns the agents on
|
//! Distinct from `hive-c0re`, which is per-hive: c0re owns the agents on
|
||||||
//! one host, this owns what is true across hives.
|
//! one host, this owns what is true across hives.
|
||||||
|
|
@ -35,6 +38,7 @@ use utoipa_axum::{router::OpenApiRouter, routes};
|
||||||
mod auth;
|
mod auth;
|
||||||
mod forge;
|
mod forge;
|
||||||
mod status;
|
mod status;
|
||||||
|
mod webhook;
|
||||||
|
|
||||||
/// Node payload for the swarm-level job graph. Named `Swarm*` rather than
|
/// Node payload for the swarm-level job graph. Named `Swarm*` rather than
|
||||||
/// the bare `NodeKind`/`Resource` `hive-c0re::job_queue::model` already
|
/// the bare `NodeKind`/`Resource` `hive-c0re::job_queue::model` already
|
||||||
|
|
@ -273,6 +277,7 @@ fn socket_path() -> PathBuf {
|
||||||
(name = "links", description = "swarm service quick links"),
|
(name = "links", description = "swarm service quick links"),
|
||||||
(name = "jobq", description = "the swarm-level job graph"),
|
(name = "jobq", description = "the swarm-level job graph"),
|
||||||
(name = "agents", description = "creating agent identities at swarm level"),
|
(name = "agents", description = "creating agent identities at swarm level"),
|
||||||
|
(name = "webhook", description = "swarm-wide forge webhook receipt"),
|
||||||
)
|
)
|
||||||
)]
|
)]
|
||||||
struct ApiDoc;
|
struct ApiDoc;
|
||||||
|
|
@ -329,6 +334,17 @@ struct AppState {
|
||||||
/// claimed — same "queue now, fail per-job" shape as an unreachable
|
/// claimed — same "queue now, fail per-job" shape as an unreachable
|
||||||
/// swarm queue.
|
/// swarm queue.
|
||||||
jobq: Arc<Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>>,
|
jobq: Arc<Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>>,
|
||||||
|
/// HMAC secret for swarm-wide forge webhooks, loaded once at startup.
|
||||||
|
/// `None` when it could not be read or created — the webhook endpoint
|
||||||
|
/// then refuses every delivery with 503 rather than admitting one it
|
||||||
|
/// cannot verify. Deliberately not fatal to startup: nothing is
|
||||||
|
/// registered against that endpoint yet, and the rest of this daemon's
|
||||||
|
/// surface is unaffected. See [`webhook::load_or_generate_secret`].
|
||||||
|
///
|
||||||
|
/// `Arc<str>`, not `Arc<String>`: the value is never mutated after
|
||||||
|
/// startup, and this way `as_deref()` yields the `&str` the verifier
|
||||||
|
/// takes without a second hop through `String`.
|
||||||
|
webhook_secret: Option<Arc<str>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Env var the controller's NixOS module sets from
|
/// Env var the controller's NixOS module sets from
|
||||||
|
|
@ -645,6 +661,47 @@ async fn get_jobq_rollup(State(state): State<AppState>) -> Json<Vec<hive_jobq_wi
|
||||||
Json(hive_jobq_wire::state_rollup(graph, roots))
|
Json(hive_jobq_wire::state_rollup(graph, roots))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The swarm's own public base URL, as the forge must address it.
|
||||||
|
///
|
||||||
|
/// Set by `swarm-controller.nix` **only when this host actually serves the
|
||||||
|
/// swarm vhost** that carries the `/webhook/forge/` location. Absent means
|
||||||
|
/// "the endpoint is not reachable from outside", and the right response to
|
||||||
|
/// that is to register nothing: a hook pointing at a URL nothing answers is
|
||||||
|
/// worse than no hook, because forgejo records failed deliveries against a
|
||||||
|
/// registration that looks configured.
|
||||||
|
const PUBLIC_URL_ENV: &str = "SWARM_CONTROLLER_PUBLIC_URL";
|
||||||
|
|
||||||
|
/// Register the swarm-wide forge hooks against this controller, in the
|
||||||
|
/// background.
|
||||||
|
///
|
||||||
|
/// Detached rather than awaited, and never fatal: the forge may be slow or
|
||||||
|
/// briefly down at boot, and none of the daemon's other routes depend on a
|
||||||
|
/// hook existing. Registration is idempotent, so the next restart retries.
|
||||||
|
///
|
||||||
|
/// Silently does nothing when any of the three preconditions is missing —
|
||||||
|
/// each is a legitimate deployment shape (no forge here, no state directory
|
||||||
|
/// to hold a secret, no public vhost), and each is already logged where it
|
||||||
|
/// is discovered.
|
||||||
|
fn register_swarm_webhooks(forge: Option<Arc<forge::Client>>, secret: Option<Arc<str>>) {
|
||||||
|
let Some(forge) = forge else { return };
|
||||||
|
let Some(secret) = secret else { return };
|
||||||
|
let Ok(public_url) = std::env::var(PUBLIC_URL_ENV) else {
|
||||||
|
tracing::info!(
|
||||||
|
"{PUBLIC_URL_ENV} unset; not registering swarm-wide forge webhooks (the \
|
||||||
|
endpoint is not published on this host)"
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
tokio::spawn(async move {
|
||||||
|
if let Err(e) = forge.ensure_swarm_webhooks(&public_url, &secret).await {
|
||||||
|
tracing::warn!(
|
||||||
|
error = %format!("{e:#}"),
|
||||||
|
"registering swarm-wide forge webhooks failed; retrying on next start"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
async fn main() -> Result<()> {
|
async fn main() -> Result<()> {
|
||||||
tracing_subscriber::fmt()
|
tracing_subscriber::fmt()
|
||||||
|
|
@ -747,7 +804,7 @@ async fn main() -> Result<()> {
|
||||||
};
|
};
|
||||||
let deps = WorkerDeps {
|
let deps = WorkerDeps {
|
||||||
auth,
|
auth,
|
||||||
forge: forge_client,
|
forge: forge_client.clone(),
|
||||||
};
|
};
|
||||||
|
|
||||||
let jobq = Arc::new(Mutex::new(hive_jobq::scheduler::Scheduler::new(
|
let jobq = Arc::new(Mutex::new(hive_jobq::scheduler::Scheduler::new(
|
||||||
|
|
@ -756,11 +813,29 @@ async fn main() -> Result<()> {
|
||||||
)));
|
)));
|
||||||
spawn_jobq_worker(Arc::clone(&jobq), deps);
|
spawn_jobq_worker(Arc::clone(&jobq), deps);
|
||||||
|
|
||||||
|
// Same "log and carry on" shape as the queue/bridge/forge wiring above.
|
||||||
|
// A controller that cannot hold a webhook secret still serves every
|
||||||
|
// other route; the webhook endpoint answers 503, which is the honest
|
||||||
|
// answer rather than a silent accept.
|
||||||
|
let webhook_secret = match webhook::load_or_generate_secret() {
|
||||||
|
Ok(secret) => Some(Arc::from(secret)),
|
||||||
|
Err(e) => {
|
||||||
|
tracing::warn!(
|
||||||
|
error = %format!("{e:#}"),
|
||||||
|
"webhook secret unavailable; swarm-wide forge webhooks are off"
|
||||||
|
);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
register_swarm_webhooks(forge_client, webhook_secret.clone());
|
||||||
|
|
||||||
let state = AppState {
|
let state = AppState {
|
||||||
hives: Arc::new(load_hives()),
|
hives: Arc::new(load_hives()),
|
||||||
links: Arc::new(load_links()),
|
links: Arc::new(load_links()),
|
||||||
status,
|
status,
|
||||||
jobq,
|
jobq,
|
||||||
|
webhook_secret,
|
||||||
};
|
};
|
||||||
|
|
||||||
let (router, api) = OpenApiRouter::<AppState>::with_openapi(ApiDoc::openapi())
|
let (router, api) = OpenApiRouter::<AppState>::with_openapi(ApiDoc::openapi())
|
||||||
|
|
@ -771,6 +846,7 @@ async fn main() -> Result<()> {
|
||||||
.routes(routes!(get_jobq_graph))
|
.routes(routes!(get_jobq_graph))
|
||||||
.routes(routes!(get_jobq_rollup))
|
.routes(routes!(get_jobq_rollup))
|
||||||
.routes(routes!(create_agent))
|
.routes(routes!(create_agent))
|
||||||
|
.routes(routes!(webhook::post_webhook_forge))
|
||||||
.split_for_parts();
|
.split_for_parts();
|
||||||
// Just the JSON, not the UI — Swagger UI itself is nginx-hosted from
|
// Just the JSON, not the UI — Swagger UI itself is nginx-hosted from
|
||||||
// the nix store (see the module doc comment above). `api` is
|
// the nix store (see the module doc comment above). `api` is
|
||||||
|
|
|
||||||
590
swarm-controller/src/webhook.rs
Normal file
590
swarm-controller/src/webhook.rs
Normal file
|
|
@ -0,0 +1,590 @@
|
||||||
|
//! Forgejo webhook receipt at the swarm level.
|
||||||
|
//!
|
||||||
|
//! A Forgejo webhook has exactly **one** `target_url`, so every hive
|
||||||
|
//! registering the same swarm-wide hooks is last-writer-wins rather than
|
||||||
|
//! idempotent, and every hive but the most recent silently stops receiving
|
||||||
|
//! deliveries. One owner is the only non-racing shape, and the only
|
||||||
|
//! swarm-wide thing in the deployment is this daemon.
|
||||||
|
//!
|
||||||
|
//! **This relays; it does not interpret.** The payload stays opaque bytes,
|
||||||
|
//! keyed by a [`DeliveryKind`] from the URL path. The hives already know
|
||||||
|
//! what a delivery *means* — which repo is the knowledge repo, which config
|
||||||
|
//! PR action queues an approval — so parsing it here would be a second place
|
||||||
|
//! deciding that, and the two would drift.
|
||||||
|
//!
|
||||||
|
//! **The HMAC code is not shared with `hive-c0re` because c0re's copy is
|
||||||
|
//! leaving, not staying.** Once registration moves here, hives stop
|
||||||
|
//! registering *and* receiving, and c0re's routes and secret go with it. A
|
||||||
|
//! shared crate is right when a second consumer *arrives*; here it is
|
||||||
|
//! departing. The two verifiers meanwhile check different hooks with
|
||||||
|
//! different secrets and never need to agree.
|
||||||
|
//!
|
||||||
|
//! **These hooks are registered ALONGSIDE the per-hive ones** — see
|
||||||
|
//! [`crate::forge::Client::ensure_swarm_webhooks`], which carries why, and
|
||||||
|
//! why moving them cannot come before fan-out.
|
||||||
|
|
||||||
|
use anyhow::{Context as _, Result};
|
||||||
|
use axum::{
|
||||||
|
body::Bytes,
|
||||||
|
extract::{Path, State},
|
||||||
|
http::{HeaderMap, StatusCode},
|
||||||
|
response::{IntoResponse as _, Response},
|
||||||
|
};
|
||||||
|
use hmac::{Hmac, KeyInit as _, Mac as _};
|
||||||
|
use sha2::Sha256;
|
||||||
|
|
||||||
|
use super::AppState;
|
||||||
|
|
||||||
|
/// Where the HMAC secret is kept, relative to the daemon's state directory.
|
||||||
|
///
|
||||||
|
/// The secret is generated on first start and handed to Forgejo when the
|
||||||
|
/// hook is registered, so it must survive a restart — a rotated secret
|
||||||
|
/// would make every subsequent delivery fail verification with a correctly
|
||||||
|
/// configured hook on both sides.
|
||||||
|
const SECRET_FILE: &str = "webhook-secret";
|
||||||
|
|
||||||
|
/// Fallback state directory, used only when `STATE_DIRECTORY` is unset (a
|
||||||
|
/// dev run outside systemd). Under the unit it is always set, because
|
||||||
|
/// `StateDirectory = "swarm-controller"` is declared there.
|
||||||
|
const DEFAULT_STATE_DIR: &str = "/var/lib/swarm-controller";
|
||||||
|
|
||||||
|
/// Path to the HMAC secret file.
|
||||||
|
///
|
||||||
|
/// Reads systemd's `STATE_DIRECTORY` rather than introducing a
|
||||||
|
/// `SWARM_CONTROLLER_*` env var for it: the unit already declares
|
||||||
|
/// `StateDirectory=`, and a second env var would be a second declaration of
|
||||||
|
/// the same fact, free to drift from the first.
|
||||||
|
///
|
||||||
|
/// ⚠️ `StateDirectory=` accepts a *list*, in which case systemd exports the
|
||||||
|
/// paths colon-separated. Only the first is taken — a bare `PathBuf::from`
|
||||||
|
/// of the whole variable would silently produce a path containing a colon
|
||||||
|
/// the day someone adds a second directory to the unit.
|
||||||
|
fn secret_path() -> std::path::PathBuf {
|
||||||
|
secret_path_from(std::env::var("STATE_DIRECTORY").ok().as_deref())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The pure half of [`secret_path`], split out so the colon-list handling is
|
||||||
|
/// testable **without touching the environment**.
|
||||||
|
///
|
||||||
|
/// Not a stylistic split: `std::env::set_var` mutates process-global state,
|
||||||
|
/// and cargo runs a crate's tests on parallel threads in one process, so two
|
||||||
|
/// tests setting it race — which is not hypothetical here, it is how the
|
||||||
|
/// first version of this module's tests failed.
|
||||||
|
fn secret_path_from(raw: Option<&str>) -> std::path::PathBuf {
|
||||||
|
let dir = raw
|
||||||
|
.and_then(|raw| raw.split(':').next())
|
||||||
|
.filter(|first| !first.is_empty())
|
||||||
|
.unwrap_or(DEFAULT_STATE_DIR);
|
||||||
|
std::path::PathBuf::from(dir).join(SECRET_FILE)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Load the swarm's webhook HMAC secret, generating and persisting it if the
|
||||||
|
/// file is absent or malformed.
|
||||||
|
///
|
||||||
|
/// Returns a hex-encoded 32-byte secret (64 hex chars).
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
///
|
||||||
|
/// Returns an error when the secret cannot be generated (no `/dev/urandom`)
|
||||||
|
/// or cannot be persisted. A malformed existing file is **not** an error: it
|
||||||
|
/// is regenerated, because a secret that cannot be parsed is
|
||||||
|
/// indistinguishable from one that was never written, and failing startup
|
||||||
|
/// over it would strand the daemon with no path forward.
|
||||||
|
pub(super) fn load_or_generate_secret() -> Result<String> {
|
||||||
|
load_or_generate_at(&secret_path())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The path-taking half of [`load_or_generate_secret`], split out for the
|
||||||
|
/// same reason as [`secret_path_from`]: a test can point it at a scratch
|
||||||
|
/// file without mutating a process-global env var.
|
||||||
|
fn load_or_generate_at(path: &std::path::Path) -> Result<String> {
|
||||||
|
if let Ok(raw) = std::fs::read_to_string(path) {
|
||||||
|
let trimmed = raw.trim().to_owned();
|
||||||
|
if trimmed.len() == 64 && trimmed.chars().all(|c| c.is_ascii_hexdigit()) {
|
||||||
|
return Ok(trimmed);
|
||||||
|
}
|
||||||
|
tracing::warn!(
|
||||||
|
path = %path.display(),
|
||||||
|
"webhook-secret file malformed (wrong length/chars); regenerating"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let secret = generate_hex_secret()?;
|
||||||
|
std::fs::create_dir_all(path.parent().unwrap_or(path))
|
||||||
|
.with_context(|| format!("create dir for {}", path.display()))?;
|
||||||
|
std::fs::write(path, format!("{secret}\n"))
|
||||||
|
.with_context(|| format!("write webhook secret to {}", path.display()))?;
|
||||||
|
tracing::info!(path = %path.display(), "webhook secret generated and persisted");
|
||||||
|
Ok(secret)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read 32 random bytes from `/dev/urandom` and hex-encode them.
|
||||||
|
fn generate_hex_secret() -> Result<String> {
|
||||||
|
use std::io::Read as _;
|
||||||
|
|
||||||
|
let mut buf = [0u8; 32];
|
||||||
|
let mut f =
|
||||||
|
std::fs::File::open("/dev/urandom").context("open /dev/urandom for secret generation")?;
|
||||||
|
f.read_exact(&mut buf)
|
||||||
|
.context("read 32 bytes from /dev/urandom")?;
|
||||||
|
Ok(hex_encode(&buf))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hex-encode `bytes` as a lowercase string.
|
||||||
|
fn hex_encode(bytes: &[u8]) -> String {
|
||||||
|
let mut out = String::with_capacity(bytes.len() * 2);
|
||||||
|
for b in bytes {
|
||||||
|
out.push(char::from_digit(u32::from(b >> 4), 16).unwrap_or('0'));
|
||||||
|
out.push(char::from_digit(u32::from(b & 0xf), 16).unwrap_or('0'));
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Decode a lowercase hex string into bytes; returns `None` on invalid input.
|
||||||
|
fn hex_decode(s: &str) -> Option<Vec<u8>> {
|
||||||
|
if !s.len().is_multiple_of(2) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let mut out = Vec::with_capacity(s.len() / 2);
|
||||||
|
let mut chars = s.chars();
|
||||||
|
while let (Some(hi), Some(lo)) = (chars.next(), chars.next()) {
|
||||||
|
let hi = u8::try_from(hi.to_digit(16)?).ok()?;
|
||||||
|
let lo = u8::try_from(lo.to_digit(16)?).ok()?;
|
||||||
|
out.push((hi << 4) | lo);
|
||||||
|
}
|
||||||
|
Some(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Which swarm-wide hook a delivery arrived on.
|
||||||
|
///
|
||||||
|
/// The kind comes from the URL path rather than from Forgejo's
|
||||||
|
/// `X-Forgejo-Event` header, so the routing key is decided by *the
|
||||||
|
/// registration we made* instead of by a field the sender chooses. Each
|
||||||
|
/// registered hook gets its own `target_url`, exactly as the per-hive hooks
|
||||||
|
/// do today.
|
||||||
|
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||||
|
pub(super) enum DeliveryKind {
|
||||||
|
/// Push events on the hive-wide knowledge repo.
|
||||||
|
Knowledge,
|
||||||
|
/// `pull_request` events on the agent-config repos.
|
||||||
|
ConfigPr,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The route prefix a registered `target_url` must point at.
|
||||||
|
///
|
||||||
|
/// ⚠️ Deliberately **not** `/webhook/knowledge` or `/webhook/config-pr`, the
|
||||||
|
/// paths the per-hive receivers use. Both hive-side registrars delete any
|
||||||
|
/// hook whose URL ends with *their* path but has a different base — see
|
||||||
|
/// `hive-c0re`'s `forge::ensure_config_pr_webhook` and
|
||||||
|
/// `workers::knowledge::ensure_webhook`. A swarm-level hook under those
|
||||||
|
/// paths would therefore be deleted by every hive on every boot, and the
|
||||||
|
/// symptom is a hook that silently stops existing. `webhook_urls_survive_the_hive_side_reapers`
|
||||||
|
/// pins that.
|
||||||
|
const ROUTE_PREFIX: &str = "/webhook/forge/";
|
||||||
|
|
||||||
|
impl DeliveryKind {
|
||||||
|
/// Every kind, for the registration sweep. An array rather than a
|
||||||
|
/// hand-written list at the call site, so adding a kind cannot leave one
|
||||||
|
/// hook unregistered.
|
||||||
|
pub(super) const ALL: [Self; 2] = [Self::Knowledge, Self::ConfigPr];
|
||||||
|
|
||||||
|
/// The `target_url` to register with Forgejo for this kind, given the
|
||||||
|
/// swarm's public base URL.
|
||||||
|
///
|
||||||
|
/// Built here rather than at the registration call site so the URL that
|
||||||
|
/// is *registered* and the route that *serves* it are the same fact in
|
||||||
|
/// one place. A trailing slash on the base is tolerated — it arrives from
|
||||||
|
/// config, and `https://swarm//webhook/...` would be a hook that 404s.
|
||||||
|
pub(super) fn target_url(self, public_base: &str) -> String {
|
||||||
|
format!(
|
||||||
|
"{}{ROUTE_PREFIX}{}",
|
||||||
|
public_base.trim_end_matches('/'),
|
||||||
|
self.as_str()
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse the `{kind}` path segment. Unknown values are rejected rather
|
||||||
|
/// than accepted-and-ignored: a typo in a registered `target_url` must
|
||||||
|
/// be *observable*, and a 200 for an unrecognised path is exactly the
|
||||||
|
/// silence this issue exists to remove.
|
||||||
|
fn parse(segment: &str) -> Option<Self> {
|
||||||
|
match segment {
|
||||||
|
"knowledge" => Some(Self::Knowledge),
|
||||||
|
"config-pr" => Some(Self::ConfigPr),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Stable string form, used for logging and (once fan-out lands) as the
|
||||||
|
/// event's routing key. Deliberately the same spelling as the path
|
||||||
|
/// segment so a journal line can be matched against a registered URL.
|
||||||
|
fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::Knowledge => "knowledge",
|
||||||
|
Self::ConfigPr => "config-pr",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Why a delivery was refused.
|
||||||
|
///
|
||||||
|
/// A value rather than a string, so the status code is chosen by matching on
|
||||||
|
/// the reason instead of on message text — `hive-c0re`'s equivalent does
|
||||||
|
/// `e.contains("unavailable")`, which couples an HTTP response to the exact
|
||||||
|
/// wording of a log message.
|
||||||
|
#[derive(Debug, PartialEq, Eq)]
|
||||||
|
pub(super) enum Refusal {
|
||||||
|
/// No secret was loaded at startup: the endpoint cannot verify anything,
|
||||||
|
/// which is this daemon's fault and not the caller's. 503, so a retry is
|
||||||
|
/// meaningful.
|
||||||
|
SecretUnavailable,
|
||||||
|
/// The signature was missing, malformed, or did not match. 401.
|
||||||
|
BadSignature(String),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Refusal {
|
||||||
|
fn status(&self) -> StatusCode {
|
||||||
|
match self {
|
||||||
|
Self::SecretUnavailable => StatusCode::SERVICE_UNAVAILABLE,
|
||||||
|
Self::BadSignature(_) => StatusCode::UNAUTHORIZED,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn message(&self) -> String {
|
||||||
|
match self {
|
||||||
|
Self::SecretUnavailable => {
|
||||||
|
"webhook HMAC secret unavailable; endpoint disabled".to_owned()
|
||||||
|
}
|
||||||
|
Self::BadSignature(detail) => detail.clone(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Verify the `X-Hub-Signature-256` header Forgejo attaches to a delivery
|
||||||
|
/// (`sha256=<hex>`).
|
||||||
|
///
|
||||||
|
/// The body is the raw wire bytes, not a re-serialised structure: an HMAC is
|
||||||
|
/// over exactly what was sent, and any parse-then-reserialise step would
|
||||||
|
/// change whitespace or key order and fail every signature.
|
||||||
|
///
|
||||||
|
/// The refusal messages are safe to log — they never contain the secret or
|
||||||
|
/// the expected digest, so a caller can surface *why* a delivery was refused
|
||||||
|
/// without handing an attacker the answer.
|
||||||
|
pub(super) fn verify(
|
||||||
|
secret: Option<&str>,
|
||||||
|
headers: &HeaderMap,
|
||||||
|
body: &Bytes,
|
||||||
|
) -> Result<(), Refusal> {
|
||||||
|
let secret = secret.ok_or(Refusal::SecretUnavailable)?;
|
||||||
|
|
||||||
|
let sig = headers
|
||||||
|
.get("x-hub-signature-256")
|
||||||
|
.and_then(|v| v.to_str().ok())
|
||||||
|
.unwrap_or("");
|
||||||
|
if sig.is_empty() {
|
||||||
|
return Err(Refusal::BadSignature(
|
||||||
|
"missing X-Hub-Signature-256 header".to_owned(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let Some(sig_hex) = sig.strip_prefix("sha256=") else {
|
||||||
|
return Err(Refusal::BadSignature(
|
||||||
|
"X-Hub-Signature-256 missing 'sha256=' prefix".to_owned(),
|
||||||
|
));
|
||||||
|
};
|
||||||
|
let Some(expected) = hex_decode(sig_hex) else {
|
||||||
|
return Err(Refusal::BadSignature(
|
||||||
|
"X-Hub-Signature-256 contains non-hex chars".to_owned(),
|
||||||
|
));
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut mac = Hmac::<Sha256>::new_from_slice(secret.as_bytes())
|
||||||
|
.map_err(|e| Refusal::BadSignature(format!("HMAC key error: {e}")))?;
|
||||||
|
mac.update(body);
|
||||||
|
// `verify_slice` is constant-time and also rejects a wrong-length digest,
|
||||||
|
// which is why the comparison is not written by hand here.
|
||||||
|
mac.verify_slice(&expected)
|
||||||
|
.map_err(|_| Refusal::BadSignature("X-Hub-Signature-256 mismatch".to_owned()))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// POST `/webhook/forge/{kind}` — a swarm-wide Forgejo delivery.
|
||||||
|
///
|
||||||
|
/// Verifies the HMAC over the raw body, then (once fan-out lands) relays the
|
||||||
|
/// delivery to every hive over the swarm queue. The payload is never parsed
|
||||||
|
/// here — see the module docs.
|
||||||
|
///
|
||||||
|
/// Returns 200 on an accepted delivery so Forgejo does not retry. A refused
|
||||||
|
/// one answers 401 (bad signature) or 503 (this daemon has no secret), and
|
||||||
|
/// an unknown `{kind}` answers 404 rather than a cheerful 200.
|
||||||
|
#[utoipa::path(
|
||||||
|
post,
|
||||||
|
path = "/webhook/forge/{kind}",
|
||||||
|
params(
|
||||||
|
("kind" = String, Path, description = "which swarm-wide hook this delivery arrived on: \
|
||||||
|
`knowledge` or `config-pr`")
|
||||||
|
),
|
||||||
|
request_body(
|
||||||
|
content = String,
|
||||||
|
content_type = "application/json",
|
||||||
|
description = "Forgejo webhook payload, taken as raw bytes so HMAC \
|
||||||
|
verification runs over the exact wire bytes and the \
|
||||||
|
delivery can be relayed unmodified"
|
||||||
|
),
|
||||||
|
responses(
|
||||||
|
(status = 200, description = "delivery accepted", body = String),
|
||||||
|
(status = 401, description = "bad or missing HMAC signature"),
|
||||||
|
(status = 404, description = "unknown hook kind in the path"),
|
||||||
|
(status = 503, description = "HMAC secret unavailable at startup"),
|
||||||
|
),
|
||||||
|
tag = "webhook"
|
||||||
|
)]
|
||||||
|
pub(super) async fn post_webhook_forge(
|
||||||
|
State(state): State<AppState>,
|
||||||
|
Path(kind): Path<String>,
|
||||||
|
headers: HeaderMap,
|
||||||
|
body: Bytes,
|
||||||
|
) -> Response {
|
||||||
|
// Order matters: verify BEFORE looking at the path. An unauthenticated
|
||||||
|
// caller learning which `{kind}` values exist, from the difference
|
||||||
|
// between 404 and 401, is a small leak — but it is free to close.
|
||||||
|
if let Err(refusal) = verify(state.webhook_secret.as_deref(), &headers, &body) {
|
||||||
|
let message = refusal.message();
|
||||||
|
tracing::warn!(%kind, %message, "webhook: refused delivery");
|
||||||
|
return (refusal.status(), message).into_response();
|
||||||
|
}
|
||||||
|
|
||||||
|
let Some(kind) = DeliveryKind::parse(&kind) else {
|
||||||
|
tracing::warn!(%kind, "webhook: delivery on an unknown hook kind");
|
||||||
|
return (StatusCode::NOT_FOUND, "unknown hook kind").into_response();
|
||||||
|
};
|
||||||
|
|
||||||
|
tracing::info!(
|
||||||
|
kind = kind.as_str(),
|
||||||
|
bytes = body.len(),
|
||||||
|
"webhook: verified delivery"
|
||||||
|
);
|
||||||
|
(StatusCode::OK, "ok").into_response()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::{DeliveryKind, Refusal, load_or_generate_at, secret_path_from, verify};
|
||||||
|
use axum::{body::Bytes, http::HeaderMap, http::StatusCode};
|
||||||
|
use hmac::{Hmac, KeyInit as _, Mac as _};
|
||||||
|
use sha2::Sha256;
|
||||||
|
use std::fmt::Write as _;
|
||||||
|
|
||||||
|
/// Mint the header Forgejo would send. Written out rather than reusing
|
||||||
|
/// the verifier's internals so a test cannot pass by sharing a bug with
|
||||||
|
/// the code under test.
|
||||||
|
fn sign(secret: &str, body: &[u8]) -> String {
|
||||||
|
let mut mac = Hmac::<Sha256>::new_from_slice(secret.as_bytes()).expect("key");
|
||||||
|
mac.update(body);
|
||||||
|
let mut hex = String::new();
|
||||||
|
for byte in mac.finalize().into_bytes() {
|
||||||
|
write!(hex, "{byte:02x}").expect("writing to a String cannot fail");
|
||||||
|
}
|
||||||
|
format!("sha256={hex}")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn headers_with(sig: &str) -> HeaderMap {
|
||||||
|
let mut h = HeaderMap::new();
|
||||||
|
h.insert("x-hub-signature-256", sig.parse().expect("header value"));
|
||||||
|
h
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Each refusal reached separately, so a change that collapses them into
|
||||||
|
/// one always-refuse path is visible — and the accept case alongside
|
||||||
|
/// them, so a verifier that refuses *everything* cannot pass either.
|
||||||
|
#[test]
|
||||||
|
fn a_delivery_is_admitted_only_with_a_matching_signature() {
|
||||||
|
let body = Bytes::from_static(b"{\"action\":\"opened\"}");
|
||||||
|
let good = sign("s3cret", &body);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
verify(Some("s3cret"), &headers_with(&good), &body),
|
||||||
|
Ok(()),
|
||||||
|
"a correctly signed delivery must be admitted"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
verify(Some("other"), &headers_with(&good), &body),
|
||||||
|
Err(Refusal::BadSignature(_))
|
||||||
|
),
|
||||||
|
"a signature minted with a different secret must be refused"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
verify(
|
||||||
|
Some("s3cret"),
|
||||||
|
&headers_with(&good),
|
||||||
|
&Bytes::from_static(b"tampered")
|
||||||
|
),
|
||||||
|
Err(Refusal::BadSignature(_))
|
||||||
|
),
|
||||||
|
"a body that does not match the digest must be refused"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
verify(
|
||||||
|
Some("s3cret"),
|
||||||
|
&headers_with(good.trim_start_matches("sha256=")),
|
||||||
|
&body
|
||||||
|
),
|
||||||
|
Err(Refusal::BadSignature(_))
|
||||||
|
),
|
||||||
|
"a bare hex digest with no 'sha256=' prefix is not what Forgejo sends"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
verify(Some("s3cret"), &HeaderMap::new(), &body),
|
||||||
|
Err(Refusal::BadSignature(_))
|
||||||
|
),
|
||||||
|
"a delivery with no signature header at all must be refused"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
verify(None, &headers_with(&good), &body),
|
||||||
|
Err(Refusal::SecretUnavailable),
|
||||||
|
"with no secret loaded the endpoint must refuse, not admit"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The status codes, asserted on the value rather than on message text.
|
||||||
|
#[test]
|
||||||
|
fn a_missing_secret_is_this_daemons_fault_and_a_bad_signature_is_the_callers() {
|
||||||
|
assert_eq!(
|
||||||
|
Refusal::SecretUnavailable.status(),
|
||||||
|
StatusCode::SERVICE_UNAVAILABLE,
|
||||||
|
"no secret is a server-side condition: a retry is meaningful"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
Refusal::BadSignature(String::new()).status(),
|
||||||
|
StatusCode::UNAUTHORIZED,
|
||||||
|
"a bad signature is the caller's, and a retry will not help"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// An unknown segment must not parse. The whole point of the 404 is that
|
||||||
|
/// a mistyped `target_url` is observable rather than silently accepted.
|
||||||
|
#[test]
|
||||||
|
fn only_the_registered_hook_kinds_parse() {
|
||||||
|
assert_eq!(
|
||||||
|
DeliveryKind::parse("knowledge"),
|
||||||
|
Some(DeliveryKind::Knowledge)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
DeliveryKind::parse("config-pr"),
|
||||||
|
Some(DeliveryKind::ConfigPr)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
DeliveryKind::parse("config_pr"),
|
||||||
|
None,
|
||||||
|
"underscore is not the registered spelling; accepting both would make a journal line ambiguous"
|
||||||
|
);
|
||||||
|
assert_eq!(DeliveryKind::parse(""), None);
|
||||||
|
assert_eq!(DeliveryKind::parse("../knowledge"), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A malformed stored secret is replaced rather than fatal — and the
|
||||||
|
/// replacement is a *usable* secret that survives a reload, because a
|
||||||
|
/// version rotating on every call would pass a naive length check while
|
||||||
|
/// silently breaking every registration made with the previous value.
|
||||||
|
///
|
||||||
|
#[test]
|
||||||
|
fn a_malformed_secret_file_is_regenerated_and_then_stable() {
|
||||||
|
let dir = std::env::temp_dir().join(format!("swarm-hook-{}", std::process::id()));
|
||||||
|
std::fs::create_dir_all(&dir).expect("mkdir");
|
||||||
|
let path = dir.join("webhook-secret");
|
||||||
|
std::fs::write(&path, "not-a-hex-secret\n").expect("seed");
|
||||||
|
|
||||||
|
let secret = load_or_generate_at(&path).expect("regenerates");
|
||||||
|
let again = load_or_generate_at(&path).expect("second load");
|
||||||
|
std::fs::remove_dir_all(&dir).ok();
|
||||||
|
|
||||||
|
assert_eq!(secret.len(), 64, "hex-encoded 32 bytes");
|
||||||
|
assert!(secret.chars().all(|c| c.is_ascii_hexdigit()));
|
||||||
|
assert_eq!(
|
||||||
|
again, secret,
|
||||||
|
"a valid secret must be read back, not rotated"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `StateDirectory=` may name several directories, in which case systemd
|
||||||
|
/// exports them colon-separated — take the first, never the raw value.
|
||||||
|
///
|
||||||
|
/// Asserted against the pure half, so this never touches the process
|
||||||
|
/// environment. The first version of these tests *did* use
|
||||||
|
/// `std::env::set_var`, and it failed: cargo runs a crate's tests on
|
||||||
|
/// parallel threads within **one** process, so the sibling env-setting
|
||||||
|
/// test removed the variable mid-run and the other silently fell back to
|
||||||
|
/// `/var/lib/swarm-controller`.
|
||||||
|
#[test]
|
||||||
|
fn the_secret_path_survives_a_multi_valued_state_directory() {
|
||||||
|
assert_eq!(
|
||||||
|
secret_path_from(Some("/var/lib/a:/var/lib/b")),
|
||||||
|
std::path::Path::new("/var/lib/a/webhook-secret")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
secret_path_from(Some("/var/lib/only")),
|
||||||
|
std::path::Path::new("/var/lib/only/webhook-secret")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
secret_path_from(None),
|
||||||
|
std::path::Path::new("/var/lib/swarm-controller/webhook-secret"),
|
||||||
|
"unset falls back to the directory the unit declares"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
secret_path_from(Some("")),
|
||||||
|
std::path::Path::new("/var/lib/swarm-controller/webhook-secret"),
|
||||||
|
"an empty value must not resolve to a relative path"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The URL that gets registered must match the route that serves it, and
|
||||||
|
/// a trailing slash on the configured base must not produce `//webhook`.
|
||||||
|
#[test]
|
||||||
|
fn a_target_url_is_the_route_this_module_serves() {
|
||||||
|
assert_eq!(
|
||||||
|
DeliveryKind::Knowledge.target_url("https://swarm.example"),
|
||||||
|
"https://swarm.example/webhook/forge/knowledge"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
DeliveryKind::ConfigPr.target_url("https://swarm.example/"),
|
||||||
|
"https://swarm.example/webhook/forge/config-pr",
|
||||||
|
"a trailing slash on the base must not double up"
|
||||||
|
);
|
||||||
|
for kind in DeliveryKind::ALL {
|
||||||
|
let url = kind.target_url("https://swarm.example");
|
||||||
|
let segment = url.rsplit('/').next().expect("a last segment");
|
||||||
|
assert_eq!(
|
||||||
|
DeliveryKind::parse(segment),
|
||||||
|
Some(kind),
|
||||||
|
"the last path segment of a registered URL must parse back \
|
||||||
|
to the kind that built it"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A cross-daemon invariant with nothing else to enforce it: both
|
||||||
|
/// per-hive registrars in `hive-c0re` **delete** hooks whose URL ends
|
||||||
|
/// with their own path but carries a different base — see
|
||||||
|
/// `forge::ensure_config_pr_webhook` and `knowledge::ensure_webhook`.
|
||||||
|
///
|
||||||
|
/// While the swarm-level hooks live alongside the per-hive ones, a
|
||||||
|
/// controller URL matching either suffix would be deleted by every hive
|
||||||
|
/// on every boot: the swarm hook would simply cease to exist, with the
|
||||||
|
/// cause in a different daemon's startup sweep. Serving these under
|
||||||
|
/// `/webhook/forge/` is what avoids it, and this is the only place that
|
||||||
|
/// says so in a form that fails.
|
||||||
|
#[test]
|
||||||
|
fn webhook_urls_survive_the_hive_side_reapers() {
|
||||||
|
for suffix in ["/webhook/knowledge", "/webhook/config-pr"] {
|
||||||
|
for kind in DeliveryKind::ALL {
|
||||||
|
let url = kind.target_url("https://swarm.example");
|
||||||
|
assert!(
|
||||||
|
!url.ends_with(suffix),
|
||||||
|
"{url} ends with {suffix}, which every hive's startup \
|
||||||
|
sweep deletes as a stale hook"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Loading…
Reference in a new issue