drop the hive-side matrix code: sweep, hivectl matrix and the apiUrl option
This commit is contained in:
damocles
mara
parent
a00bcb8c05
commit
6dbb18b6d4
26 changed files with 49 additions and 1007 deletions
|
|
@ -109,16 +109,7 @@ On a hive that already has a ruth container from an earlier release, create
|
|||
her once the same way, so that her config repo and store identity exist. The
|
||||
container keeps starting with the hive meanwhile.
|
||||
|
||||
## 5 · Matrix
|
||||
|
||||
Your matrix account comes from SSO. Invite it to the rooms you want it in:
|
||||
|
||||
```bash
|
||||
hivectl matrix invite mara --room '#some-room:yourserver'
|
||||
hivectl matrix invite @mara:yourserver --room '!abc:yourserver'
|
||||
```
|
||||
|
||||
## 6 · Your first agent
|
||||
## 5 · Your first agent
|
||||
|
||||
Create it from the swarm UI, or:
|
||||
|
||||
|
|
|
|||
|
|
@ -96,11 +96,8 @@ and federation peers all reach the homeserver at `chat.<swarm-domain>`
|
|||
through the gateway; from outside the host that takes
|
||||
`gateway.openFirewall`.
|
||||
|
||||
hive-c0re talks to matrix (its own sender login, invites) through `swarm.matrix.apiUrl`, which defaults to
|
||||
`https://<swarm.matrix.gatewayHost>` on every hive. That login carries
|
||||
the hive's matrix access token, and on a hive that doesn't run the
|
||||
homeserver it crosses the network, so the default is TLS; hive-c0re
|
||||
verifies the gateway's certificate against the hive trust bundle.
|
||||
hive-c0re does not talk to the homeserver: matrix is swarm-level, and
|
||||
the hive has no matrix API URL to set.
|
||||
|
||||
Federation peers find the homeserver through `.well-known/matrix/server`
|
||||
at the server name, which names `<gatewayHost>:443`
|
||||
|
|
@ -143,8 +140,8 @@ a token.
|
|||
reads only `.yaml`/`.yml` entries from there, so the sibling
|
||||
credentials are invisible to it. The `.yaml` suffix on the credential
|
||||
id is what makes this work.
|
||||
4. **hive-c0re** holds no appservice token and creates no account. Its
|
||||
`@hive-<hive>:` token comes from the swarm store (below).
|
||||
4. **hive-c0re** holds no appservice token, creates no account and does
|
||||
not talk to matrix.
|
||||
5. **`swarm-controller` creates every account**: each agent's, with its
|
||||
token stored at `swarm/agents/<agent>/matrix/main` for the agent's
|
||||
`hive-matrix-daemon` to read as the agent itself, and each hive's
|
||||
|
|
@ -246,21 +243,14 @@ whose hives share a homeserver gives each of them its own identity: the
|
|||
homeserver attributes an action to the hive that took it, and revoking one
|
||||
hive's standing leaves the others alone.
|
||||
|
||||
Its access token is the **sender token**, and it's the credential
|
||||
hive-c0re presents for every homeserver call it makes on the hive's
|
||||
behalf. It's per hive for the same reason the account is:
|
||||
`swarm-controller`, its only minter, mints it for every hive with the swarm's
|
||||
appservice token and publishes it to `swarm/hives/<hive>/matrix/sender-token`, and the hive
|
||||
reads it from there under its own certificate. That path sits inside the
|
||||
hive's own read grant (`swarm/hives/<hive>/*`), so a hive fetches its own
|
||||
token and gets a refusal on any other hive's. The name says what it
|
||||
authenticates as — an account the swarm's appservice creates — rather
|
||||
than any privilege level, because it carries none.
|
||||
|
||||
It needs no promotion for what the hive does with it. Inviting a user into a
|
||||
room it has the power in is an ordinary client call — there is no homeserver
|
||||
admin in any of it, and no Synapse admin API to reach for either, since
|
||||
tuwunel has none.
|
||||
Its access token is the **sender token**. `swarm-controller`, its only
|
||||
minter, mints it for every hive with the swarm's appservice token and
|
||||
publishes it to `swarm/hives/<hive>/matrix/sender-token`, inside the hive's own
|
||||
read grant (`swarm/hives/<hive>/*`). Nothing on the hive uses it any more:
|
||||
hive-c0re used to present it for its Space and invite calls and no longer
|
||||
makes any. The name says what it authenticates as — an account the
|
||||
swarm's appservice creates — rather than any privilege level, because it
|
||||
carries none.
|
||||
|
||||
Promoting a user to homeserver admin and resetting a password both need an
|
||||
admin **sender**: `!admin …` messages into `#admins:<server_name>`, and
|
||||
|
|
@ -270,17 +260,6 @@ admin. Of the swarm's own accounts only `@swarm` is a homeserver admin
|
|||
[Operators are homeserver admins](#operators-are-homeserver-admins)), so
|
||||
both are swarm-level operations.
|
||||
|
||||
<details><summary>Store precedence for a hive's sender account</summary>
|
||||
|
||||
`ensure_hive_user` reads the per-hive path
|
||||
`swarm/hives/<hive>/matrix/sender-token` **first, on every sweep**, not
|
||||
just when that file is missing (`sender_source`'s decision).
|
||||
`swarm-controller` mints a per-hive token within five minutes of a hive
|
||||
appearing; the sweep then overwrites the per-hive file, no boot
|
||||
required.
|
||||
|
||||
</details>
|
||||
|
||||
<details><summary>Appservice identity</summary>
|
||||
|
||||
`swarm-matrix-ctl appservice render` renders the registration from the
|
||||
|
|
@ -344,17 +323,14 @@ Initial rollout settings:
|
|||
stored yet the account comes up without backup and picks it up on the
|
||||
next daemon start. A failure is a `WARN` and never stops the daemon.
|
||||
|
||||
## Hive sender token
|
||||
## Retired hive-side matrix code
|
||||
|
||||
hive-c0re keeps the hive's `@hive-<hive>:` sender token on disk for
|
||||
`hivectl matrix invite`, refreshed from the swarm secret store by the
|
||||
matrix sweep (at boot, then every 30 minutes).
|
||||
|
||||
Earlier versions also created a private "hive" Matrix Space and a
|
||||
`hive-chat` room, invited every agent to both, and kept their ids in
|
||||
`/var/lib/hyperhive/matrix/space-room-id` and `chat-room-id`. The sweep no
|
||||
longer touches any of them. The rooms and files are harmless leftovers, so
|
||||
delete them by hand if you like.
|
||||
Earlier versions of hive-c0re created a private "hive" Matrix Space and a
|
||||
`hive-chat` room, invited every agent to both, kept the hive's sender token
|
||||
on disk and offered `hivectl matrix invite`. All of that is gone: matrix is
|
||||
swarm-level, so the hive no longer talks to the homeserver. The Space, the
|
||||
room and the files `/var/lib/hyperhive/matrix/{access-token,space-room-id,chat-room-id}`
|
||||
are harmless leftovers, so delete them by hand if you like.
|
||||
|
||||
## Configuration tuning
|
||||
|
||||
|
|
|
|||
|
|
@ -7,8 +7,6 @@ This document contains the help content for the `hivectl` command-line program.
|
|||
* [`hivectl`↴](#hivectl)
|
||||
* [`hivectl forge`↴](#hivectl-forge)
|
||||
* [`hivectl forge reconcile-config`↴](#hivectl-forge-reconcile-config)
|
||||
* [`hivectl matrix`↴](#hivectl-matrix)
|
||||
* [`hivectl matrix invite`↴](#hivectl-matrix-invite)
|
||||
* [`hivectl agent`↴](#hivectl-agent)
|
||||
* [`hivectl agent restart`↴](#hivectl-agent-restart)
|
||||
* [`hivectl agent pause`↴](#hivectl-agent-pause)
|
||||
|
|
@ -45,14 +43,13 @@ This document contains the help content for the `hivectl` command-line program.
|
|||
|
||||
## `hivectl`
|
||||
|
||||
Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that don't go through the broker — manual user provisioning on the bundled matrix container, plus future recovery / debugging verbs.
|
||||
Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that don't go through the broker, plus future recovery / debugging verbs.
|
||||
|
||||
**Usage:** `hivectl [OPTIONS] <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `forge` — Reconcile an agent's config between this hive and the forge
|
||||
* `matrix` — matrix-tuwunel invites
|
||||
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
|
||||
* `list-agents` — Show all managed agents with their status and technical state
|
||||
* `quota-enable` — Enable btrfs qgroup accounting on the agent-state filesystem
|
||||
|
|
@ -112,36 +109,6 @@ Always prints the diff first. `--from forge` resets the local checkout to forge
|
|||
|
||||
|
||||
|
||||
## `hivectl matrix`
|
||||
|
||||
matrix-tuwunel invites.
|
||||
|
||||
Manual invites into a room by id or alias.
|
||||
|
||||
**Usage:** `hivectl matrix <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `invite` — Invite a matrix user to a room. Idempotent
|
||||
|
||||
|
||||
|
||||
## `hivectl matrix invite`
|
||||
|
||||
Invite a matrix user to a room. Idempotent
|
||||
|
||||
**Usage:** `hivectl matrix invite --room <ROOM> <USER>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<USER>` — User to invite: a full id (`@mara:server`) or a bare localpart (qualified with the homeserver's `server_name`)
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--room <ROOM>` — Target room id (`!abc:server`) or alias (`#name:server`)
|
||||
|
||||
|
||||
|
||||
## `hivectl agent`
|
||||
|
||||
Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running.
|
||||
|
|
|
|||
|
|
@ -10,8 +10,8 @@ login user in `services.hyperhive.c0re.adminUsers` (adds you to the
|
|||
Available via the `hive-c0re` package in the host NixOS config.
|
||||
|
||||
Unlike the `hive-c0re` daemon subcommands (which go through the broker),
|
||||
`hivectl` covers direct host-side administration: manual provisioning of
|
||||
matrix accounts, container lifecycle shortcuts, and interactive agent shell access.
|
||||
`hivectl` covers direct host-side administration: container lifecycle
|
||||
shortcuts and interactive agent shell access.
|
||||
|
||||
This page is the curated guide. For the exhaustive flag-by-flag
|
||||
reference autogenerated from the binary's own command tree, see
|
||||
|
|
@ -42,26 +42,6 @@ hivectl forge reconcile-config iris --verbose # include the full diff, not
|
|||
supported yet (forge `main` is branch-protected; resolve via a config
|
||||
PR). With no `--from` it prompts for the direction after the diff.
|
||||
|
||||
## Matrix
|
||||
|
||||
Manual invites into a room. Requires the `hive-matrix` container to be
|
||||
running (`services.hyperhive.deploy.matrix.enable = true`).
|
||||
|
||||
```bash
|
||||
hivectl matrix invite mara --room '#some-room:server' # a room alias
|
||||
hivectl matrix invite @mara:server --room '!abc:server' # ...or a room id
|
||||
```
|
||||
|
||||
Human matrix accounts come from SSO, not `hivectl`; matrix homeserver
|
||||
admin should eventually come from membership in authelia's `admins`
|
||||
group; nobody has built that sync yet.
|
||||
|
||||
- `invite`: invites a matrix user (full `@user:server` or a bare
|
||||
localpart, qualified with the homeserver's `server_name`) to the
|
||||
`--room` id / `#alias`. Uses the sender token; the account must be a
|
||||
member of the target room with invite power.
|
||||
Idempotent — already-member / already-invited is a no-op.
|
||||
|
||||
## Agents
|
||||
|
||||
Container lifecycle shortcuts that go through the host admin socket.
|
||||
|
|
|
|||
|
|
@ -90,8 +90,7 @@ pub async fn publish_nix_credential() -> bool {
|
|||
/// This hive's token from the store, under the hive's own identity.
|
||||
///
|
||||
/// `None`, never an error, for every way it can come up empty — no hive name,
|
||||
/// no `BAO_*` identity, an unreachable store, nothing at the path — the same
|
||||
/// as `matrix::stored_sender_token`.
|
||||
/// no `BAO_*` identity, an unreachable store, nothing at the path.
|
||||
async fn stored_reader_token() -> Option<ReaderToken> {
|
||||
let hive = std::env::var("HYPERHIVE_HIVE_NAME")
|
||||
.ok()
|
||||
|
|
|
|||
|
|
@ -106,8 +106,6 @@ pub(super) trait Host: Sync {
|
|||
fn purge_dirs(&self, name: &str) -> Vec<PathBuf>;
|
||||
/// Publish the hive's forge credential for nix.
|
||||
async fn forge_ensure_all(&self);
|
||||
/// One matrix sender-token pass; whether the token is in place.
|
||||
async fn matrix_ensure_all(&self) -> bool;
|
||||
/// One `/knowledge` pull.
|
||||
async fn knowledge_pull(&self, coord: &Arc<Coordinator>) -> Result<()>;
|
||||
/// One swarm wanted-state pull.
|
||||
|
|
@ -245,10 +243,6 @@ impl Host for RealHost {
|
|||
crate::forge::ensure_all().await;
|
||||
}
|
||||
|
||||
async fn matrix_ensure_all(&self) -> bool {
|
||||
crate::matrix::ensure_all().await
|
||||
}
|
||||
|
||||
async fn knowledge_pull(&self, coord: &Arc<Coordinator>) -> Result<()> {
|
||||
crate::workers::knowledge::pull(coord).await
|
||||
}
|
||||
|
|
@ -350,7 +344,6 @@ async fn dispatch(
|
|||
// until their whole subtree settles.
|
||||
NodeKind::AgentWindow { .. } => Ok(()),
|
||||
NodeKind::ForgeSweep => run_forge_sweep(host).await,
|
||||
NodeKind::MatrixSweep => run_matrix_sweep(host, MATRIX_SWEEP_DEADLINE).await,
|
||||
NodeKind::KnowledgePull => run_knowledge_pull(host, coord, KNOWLEDGE_PULL_DEADLINE).await,
|
||||
NodeKind::WantedPull => run_wanted_pull(host, coord).await,
|
||||
};
|
||||
|
|
@ -367,56 +360,6 @@ async fn run_forge_sweep(host: &impl Host) -> Result<()> {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Matrix sender-token sweep as a DAG node — see [`NodeKind::MatrixSweep`].
|
||||
/// Every pass reports here, whoever submitted it: as the node's own error on
|
||||
/// the dashboard, and into the debounced warning.
|
||||
async fn run_matrix_sweep(host: &impl Host, deadline: std::time::Duration) -> Result<()> {
|
||||
let ok = tokio::time::timeout(deadline, host.matrix_ensure_all())
|
||||
.await
|
||||
.unwrap_or_else(|_| {
|
||||
tracing::warn!(?deadline, "matrix: sweep timed out");
|
||||
false
|
||||
});
|
||||
let mut health = matrix_sweep_health()
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if ok {
|
||||
health.record_ok();
|
||||
Ok(())
|
||||
} else {
|
||||
health.record_err(matrix_sweep_warning);
|
||||
anyhow::bail!("matrix ensure_all: one or more agents failed sync (see logs)")
|
||||
}
|
||||
}
|
||||
|
||||
/// A pass still running holds [`Resource::MatrixSweep`] against every later
|
||||
/// one. Each HTTP call has its own 10 s timeout, but the call count grows with
|
||||
/// the agent count and `lifecycle::list()` has none. A pass dropped here leaves
|
||||
/// at worst a created but unpersisted room, which the next pass adopts by name.
|
||||
///
|
||||
/// [`Resource::MatrixSweep`]: super::resource::Resource::MatrixSweep
|
||||
const MATRIX_SWEEP_DEADLINE: std::time::Duration = std::time::Duration::from_mins(10);
|
||||
|
||||
/// Warning for a matrix sweep that keeps failing. Two misses in a row, so one
|
||||
/// bad pass (homeserver mid-restart, a transient HTTP blip) doesn't flap the
|
||||
/// dashboard; cleared by the next clean pass.
|
||||
fn matrix_sweep_health() -> &'static Mutex<SweepHealth> {
|
||||
static HEALTH: OnceLock<Mutex<SweepHealth>> = OnceLock::new();
|
||||
HEALTH.get_or_init(|| Mutex::new(SweepHealth::new("matrix_ensure_all", "warn", 2)))
|
||||
}
|
||||
|
||||
fn matrix_sweep_warning(ctx: sweep_health::SweepFailure) -> String {
|
||||
let age = ctx.since_last_ok.map_or_else(
|
||||
|| "no success this session".to_owned(),
|
||||
|d| format!("last ok {} ago", sweep_health::fmt_age(d)),
|
||||
);
|
||||
format!(
|
||||
"matrix sender-token sweep failing ({} consecutive, {age}) \
|
||||
— `hivectl matrix invite` may not work",
|
||||
ctx.consecutive
|
||||
)
|
||||
}
|
||||
|
||||
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
|
||||
/// pass reports here, whoever submitted it: as the node's own error on the
|
||||
/// dashboard, and into the debounced warning.
|
||||
|
|
|
|||
|
|
@ -11,12 +11,11 @@ use hive_jobq::NodeId;
|
|||
use hive_jobq::scheduler::Scheduler;
|
||||
|
||||
use super::{
|
||||
Host, dispatch, drift, matrix_sweep_warning, run_create, run_destroy_bookkeeping,
|
||||
run_destroy_container, run_drain, run_emit_rebuilt, run_forge_sweep, run_knowledge_pull,
|
||||
run_matrix_sweep, run_meta_sync, run_pause_drain, run_pause_signal, run_prebuild,
|
||||
run_provision, run_purge_state, run_rebuild_bookkeeping, run_reconcile, run_set_wanted,
|
||||
run_signal, run_start, run_stop, run_stop_for_update, run_swap, run_wanted_pull,
|
||||
run_write_dropin, substitute,
|
||||
Host, dispatch, drift, run_create, run_destroy_bookkeeping, run_destroy_container, run_drain,
|
||||
run_emit_rebuilt, run_forge_sweep, run_knowledge_pull, run_meta_sync, run_pause_drain,
|
||||
run_pause_signal, run_prebuild, run_provision, run_purge_state, run_rebuild_bookkeeping,
|
||||
run_reconcile, run_set_wanted, run_signal, run_start, run_stop, run_stop_for_update, run_swap,
|
||||
run_wanted_pull, run_write_dropin, substitute,
|
||||
};
|
||||
use crate::coordinator::{AgentPaths, Coordinator, HiveEnv};
|
||||
use crate::job_queue::model::NodeKind;
|
||||
|
|
@ -43,8 +42,7 @@ struct FakeHost {
|
|||
/// Store paths a prebuild had to build because the store lacked them.
|
||||
builds: Mutex<Vec<String>>,
|
||||
/// Names of the calls that fail (`kill`, `realise`, `prebuild`, `swap_update`,
|
||||
/// `set_paused`, ...). `recorded_build` fails by finding nothing and
|
||||
/// `matrix_ensure_all` by reporting false.
|
||||
/// `set_paused`, ...). `recorded_build` fails by finding nothing.
|
||||
failing: Vec<&'static str>,
|
||||
/// The sweeps and pulls never finish, so only a deadline ends them.
|
||||
hang: bool,
|
||||
|
|
@ -277,14 +275,6 @@ impl Host for FakeHost {
|
|||
self.record("forge_ensure_all".to_owned());
|
||||
}
|
||||
|
||||
async fn matrix_ensure_all(&self) -> bool {
|
||||
self.record("matrix_ensure_all".to_owned());
|
||||
if self.hang {
|
||||
std::future::pending::<()>().await;
|
||||
}
|
||||
!self.fails("matrix_ensure_all")
|
||||
}
|
||||
|
||||
async fn knowledge_pull(&self, _coord: &Arc<Coordinator>) -> anyhow::Result<()> {
|
||||
if self.hang {
|
||||
std::future::pending::<()>().await;
|
||||
|
|
@ -1088,31 +1078,6 @@ async fn forge_sweep_runs_once_and_never_fails() {
|
|||
assert_eq!(host.calls(), ["forge_ensure_all"]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn matrix_sweep_fails_the_node_when_an_agent_failed_sync() {
|
||||
run_matrix_sweep(&FakeHost::up(), Duration::from_secs(5))
|
||||
.await
|
||||
.expect("ok");
|
||||
let err = run_matrix_sweep(
|
||||
&FakeHost::up().failing("matrix_ensure_all"),
|
||||
Duration::from_secs(5),
|
||||
)
|
||||
.await
|
||||
.expect_err("failed sync");
|
||||
assert!(format!("{err:#}").contains("failed sync"), "{err:#}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn matrix_sweep_past_its_deadline_fails_instead_of_hanging() {
|
||||
let host = FakeHost {
|
||||
hang: true,
|
||||
..FakeHost::up()
|
||||
};
|
||||
run_matrix_sweep(&host, Duration::from_millis(50))
|
||||
.await
|
||||
.expect_err("deadline");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn knowledge_pull_passes_the_result_through_and_times_out() {
|
||||
let (_dir, coord) = coordinator();
|
||||
|
|
@ -1274,7 +1239,6 @@ async fn dispatch_routes_the_agentless_and_hostless_kinds() {
|
|||
NodeKind::ForgeSweep,
|
||||
NodeKind::WantedPull,
|
||||
NodeKind::KnowledgePull,
|
||||
NodeKind::MatrixSweep,
|
||||
alice(|agent| NodeKind::AgentWindow { agent }),
|
||||
alice(|agent| NodeKind::Signal { agent }),
|
||||
alice(|agent| NodeKind::Drain { agent }),
|
||||
|
|
@ -1292,12 +1256,7 @@ async fn dispatch_routes_the_agentless_and_hostless_kinds() {
|
|||
);
|
||||
}
|
||||
let calls = host.calls();
|
||||
for want in [
|
||||
"forge_ensure_all",
|
||||
"wanted_pull",
|
||||
"knowledge_pull",
|
||||
"matrix_ensure_all",
|
||||
] {
|
||||
for want in ["forge_ensure_all", "wanted_pull", "knowledge_pull"] {
|
||||
assert!(calls.iter().any(|c| c == want), "{want} in {calls:?}");
|
||||
}
|
||||
}
|
||||
|
|
@ -1511,22 +1470,3 @@ async fn purge_keeps_going_past_a_dir_it_cannot_remove() {
|
|||
assert!(stuck.exists(), "remove_dir_all refuses a plain file");
|
||||
assert!(!applied.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn matrix_sweep_warning_says_how_long_it_has_been_failing() {
|
||||
use crate::stats::sweep_health::SweepFailure;
|
||||
let never = matrix_sweep_warning(SweepFailure {
|
||||
consecutive: 3,
|
||||
since_last_ok: None,
|
||||
});
|
||||
assert!(
|
||||
never.contains("3 consecutive, no success this session"),
|
||||
"{never}"
|
||||
);
|
||||
let after_ok = matrix_sweep_warning(SweepFailure {
|
||||
consecutive: 2,
|
||||
since_last_ok: Some(Duration::from_secs(90)),
|
||||
});
|
||||
assert!(after_ok.contains("2 consecutive, last ok "), "{after_ok}");
|
||||
assert!(!after_ok.contains("no success"), "{after_ok}");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -99,9 +99,6 @@ pub enum NodeKind {
|
|||
/// One-shot boot-time publish of the hive's forge read token for nix
|
||||
/// (`forge::ensure_all`). Agentless.
|
||||
ForgeSweep,
|
||||
/// Matrix sender-token sweep (`matrix::ensure_all`): at boot and every 30
|
||||
/// minutes. Agentless.
|
||||
MatrixSweep,
|
||||
/// `/knowledge` pull (`knowledge::pull`): at boot, on the swarm
|
||||
/// knowledge-changed event, and hourly. Agentless.
|
||||
KnowledgePull,
|
||||
|
|
@ -168,10 +165,7 @@ impl NodeKind {
|
|||
| NodeKind::AgentWindow { agent }
|
||||
| NodeKind::EmitRebuilt { agent, .. }
|
||||
| NodeKind::SetWanted { agent, .. } => agent,
|
||||
NodeKind::ForgeSweep
|
||||
| NodeKind::MatrixSweep
|
||||
| NodeKind::KnowledgePull
|
||||
| NodeKind::WantedPull => "",
|
||||
NodeKind::ForgeSweep | NodeKind::KnowledgePull | NodeKind::WantedPull => "",
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -31,9 +31,6 @@ pub enum Resource {
|
|||
/// overlapping `reset` / `clean` / `pull` passes there fail on
|
||||
/// `.git/index.lock` and the remote-tracking ref lock.
|
||||
KnowledgeTree,
|
||||
/// The hive's sender-token file, held by every `MatrixSweep`. Two
|
||||
/// overlapping passes would race on writing it.
|
||||
MatrixSweep,
|
||||
}
|
||||
|
||||
/// This resource's name on the generic graph wire.
|
||||
|
|
@ -49,7 +46,6 @@ impl hive_jobq_wire::WireResource for Resource {
|
|||
Resource::Agent(agent) => format!("agent:{agent}"),
|
||||
|
||||
Resource::KnowledgeTree => "knowledge-tree".to_owned(),
|
||||
Resource::MatrixSweep => "matrix-sweep".to_owned(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -245,31 +245,6 @@ mod tests {
|
|||
.count()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn two_passes_of_one_sweep_never_run_together() {
|
||||
let q = JobQueue::new(1);
|
||||
for _ in 0..2 {
|
||||
q.insert_job(|b| vec![templates::matrix_sweep(b).guid()])
|
||||
.expect("insert");
|
||||
}
|
||||
q.insert_job(|b| vec![templates::knowledge_pull(b).guid()])
|
||||
.expect("insert");
|
||||
|
||||
let first = claim(&q).expect("a matrix pass is runnable");
|
||||
let other = claim(&q).expect("a different sweep runs alongside it");
|
||||
assert!(
|
||||
claim(&q).is_none(),
|
||||
"the second matrix pass waits for the first"
|
||||
);
|
||||
assert_eq!(running(&q), ["knowledge_pull", "matrix_sweep"]);
|
||||
|
||||
first.await;
|
||||
other.await;
|
||||
let second = claim(&q).expect("the second matrix pass runs once the first is done");
|
||||
assert_eq!(running(&q), ["matrix_sweep"]);
|
||||
second.await;
|
||||
}
|
||||
|
||||
/// A tick landing while a pass is already live queues a second pass; the
|
||||
/// shared `Resource::KnowledgeTree` dep holds it back until the first
|
||||
/// finishes, so the two never run together.
|
||||
|
|
|
|||
|
|
@ -370,14 +370,6 @@ pub fn destroy(builder: &JobBuilder, agent: &str, purge: bool) {
|
|||
// `Reconcile` root per drifted agent) — no anchor node and no per-agent child
|
||||
// DAGs.
|
||||
|
||||
/// One matrix sender-token sweep. Holding [`Resource::MatrixSweep`] is what keeps
|
||||
/// two passes from overlapping, whichever caller submitted each.
|
||||
pub fn matrix_sweep(builder: &JobBuilder) -> Handle<'_> {
|
||||
builder
|
||||
.node(NodeKind::MatrixSweep)
|
||||
.needs(Resource::MatrixSweep)
|
||||
}
|
||||
|
||||
/// One `/knowledge` pull, holding [`Resource::KnowledgeTree`] for its duration.
|
||||
pub fn knowledge_pull(builder: &JobBuilder) -> Handle<'_> {
|
||||
builder
|
||||
|
|
|
|||
|
|
@ -1507,19 +1507,13 @@ fn destroy_container_must_not_suppress_crash_watch() {
|
|||
|
||||
#[test]
|
||||
fn each_sweep_holds_its_own_resource() {
|
||||
// Same kind, same capacity-1 resource: two passes serialise. Different
|
||||
// kinds, disjoint resources and no edges: they run side by side. That the
|
||||
// Same kind, same capacity-1 resource: two passes serialise. That the
|
||||
// scheduler honours this is `hive_jobq`'s
|
||||
// `unrelated_nodes_needing_the_same_resource_are_serialized`.
|
||||
let q = JobQueue::new(1);
|
||||
insert(&q, |builder| {
|
||||
let _ = templates::matrix_sweep(builder);
|
||||
let _ = templates::knowledge_pull(builder);
|
||||
});
|
||||
assert_eq!(
|
||||
declared_resources_of_kind(&q, "matrix_sweep"),
|
||||
vec![vec![Resource::MatrixSweep]]
|
||||
);
|
||||
assert_eq!(
|
||||
declared_resources_of_kind(&q, "knowledge_pull"),
|
||||
vec![vec![Resource::KnowledgeTree]]
|
||||
|
|
|
|||
|
|
@ -26,7 +26,6 @@ mod gateway_nginx;
|
|||
mod job_queue;
|
||||
mod lifecycle;
|
||||
mod loose_ends;
|
||||
mod matrix;
|
||||
mod meta;
|
||||
mod meta_rev;
|
||||
mod migrate;
|
||||
|
|
@ -233,7 +232,7 @@ async fn cmd_serve(
|
|||
socket: &std::path::Path,
|
||||
) -> Result<()> {
|
||||
// Move any host-side state still at the legacy flat layout into its
|
||||
// subdir (`db/`, `forge/`, `matrix/`, `run/`) BEFORE opening the
|
||||
// subdir (`db/`, `forge/`, `run/`) BEFORE opening the
|
||||
// broker db — the broker + build-logs dbs are among the relocated
|
||||
// files. Idempotent; a no-op once migrated.
|
||||
crate::paths::relocate_legacy_state();
|
||||
|
|
@ -307,29 +306,6 @@ async fn cmd_serve(
|
|||
}
|
||||
}
|
||||
});
|
||||
// Matrix sweep (`matrix::ensure_all`): the hive's own sender token, kept
|
||||
// on disk for `hivectl matrix invite`. It
|
||||
// creates no accounts or tokens — swarm-controller mints those.
|
||||
// No-op when the hive-matrix container isn't running. Re-submitted every
|
||||
// 30 minutes; the startup pass is the boot `MatrixSweep` node.
|
||||
let mut matrix_shutdown = coord.shutdown_rx();
|
||||
let matrix_coord = coord.clone();
|
||||
tokio::spawn(async move {
|
||||
let interval = std::time::Duration::from_mins(30);
|
||||
loop {
|
||||
tokio::select! {
|
||||
() = tokio::time::sleep(interval) => submit_periodic_sweep(
|
||||
&matrix_coord,
|
||||
&job_queue::NodeKind::MatrixSweep,
|
||||
job_queue::templates::matrix_sweep,
|
||||
),
|
||||
_ = matrix_shutdown.changed() => {
|
||||
tracing::info!("matrix ensure_all: shutdown signal received");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
// Periodic broker vacuum: drop fully-acked messages older
|
||||
// than 30 days. Delivered-but-unacked rows (recoverable via
|
||||
// requeue_inflight) and undelivered rows are always kept.
|
||||
|
|
|
|||
|
|
@ -1,486 +0,0 @@
|
|||
//! Optional matrix-tuwunel wiring: the hive's `@hive-<hive>:` sender token,
|
||||
//! taken from the swarm secret store and kept on disk for `hivectl matrix
|
||||
//! invite`. No-op when no homeserver is configured, so operators who haven't
|
||||
//! flipped `services.hyperhive.deploy.matrix.enable = true` pay nothing.
|
||||
//!
|
||||
//! This module creates no accounts: `swarm-controller` mints every hive's
|
||||
//! sender account and every agent's account with the swarm's appservice token.
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
|
||||
/// Client-server API base this daemon provisions against, from
|
||||
/// `HIVE_MATRIX_API_URL` (set by the hyperhive NixOS module from
|
||||
/// `services.hyperhive.swarm.matrix.apiUrl`).
|
||||
///
|
||||
/// `None` means **this hive has no homeserver to provision against** and
|
||||
/// every matrix path no-ops — see [`is_present`]. There is deliberately no
|
||||
/// fallback: a loopback default is right only when the homeserver happens to
|
||||
/// share this daemon's netns, and an address baked into the binary is one
|
||||
/// that builds fine and then talks to the wrong machine. The nix module
|
||||
/// defaults it to `https://` on the homeserver's gateway vhost, on every
|
||||
/// hive whichever host runs tuwunel.
|
||||
///
|
||||
/// Also not an agent-facing address either way. An agent has its own netns;
|
||||
/// agents are handed the gateway vhost via `HIVE_MATRIX_URL`, and get
|
||||
/// nothing at all when the hive has no vhost to offer.
|
||||
fn matrix_http() -> Option<&'static str> {
|
||||
static BASE: std::sync::OnceLock<Option<String>> = std::sync::OnceLock::new();
|
||||
BASE.get_or_init(|| std::env::var("HIVE_MATRIX_API_URL").ok())
|
||||
.as_deref()
|
||||
}
|
||||
|
||||
/// [`matrix_http`] for the call sites that propagate with `?`.
|
||||
///
|
||||
/// # Errors
|
||||
/// When no homeserver is configured. Reaching one of these paths at all
|
||||
/// means an [`is_present`] gate was skipped, so the message names that
|
||||
/// rather than the missing variable.
|
||||
fn matrix_base() -> Result<&'static str> {
|
||||
matrix_http().context(
|
||||
"matrix: no homeserver configured \
|
||||
(services.hyperhive.swarm.matrix.apiUrl / HIVE_MATRIX_API_URL) — \
|
||||
this path should have been gated on matrix::is_present()",
|
||||
)
|
||||
}
|
||||
|
||||
/// Host path for the appservice sender account's matrix access token. Outside every
|
||||
/// purgeable path — not deleted by `destroy --purge` on any agent.
|
||||
#[must_use]
|
||||
pub fn sender_token_path() -> PathBuf {
|
||||
crate::paths::matrix_sender_token()
|
||||
}
|
||||
|
||||
/// Whether this hive has a homeserver to provision against.
|
||||
///
|
||||
/// **A configured API URL, not a local container**: whether the homeserver
|
||||
/// runs on this host is a different question, and a remote homeserver is
|
||||
/// provisioned through the same URL.
|
||||
#[must_use]
|
||||
pub fn is_present() -> bool {
|
||||
matrix_http().is_some()
|
||||
}
|
||||
|
||||
/// Bring the hive's `@hive-<hive>:` sender token at [`sender_token_path()`] in
|
||||
/// line with the swarm secret store.
|
||||
///
|
||||
/// `swarm-controller` is the only minter: it creates the account with the
|
||||
/// swarm's appservice token, keeps the stored token while it is live and
|
||||
/// replaces it when it is not. The file follows the store, and is kept as it
|
||||
/// is when the store has nothing or cannot be reached, so a store outage does
|
||||
/// not take the hive's matrix provisioning down with it.
|
||||
///
|
||||
/// # Errors
|
||||
/// When neither the store nor the file holds a token, or the file write fails.
|
||||
pub async fn ensure_hive_user() -> Result<()> {
|
||||
let path = sender_token_path();
|
||||
let on_disk = std::fs::read_to_string(&path).ok();
|
||||
let stored = stored_sender_token().await;
|
||||
match sender_source(stored.as_deref(), on_disk.as_deref()) {
|
||||
SenderSource::Keep => {
|
||||
tracing::debug!("matrix: the sender token is already present");
|
||||
Ok(())
|
||||
}
|
||||
SenderSource::Store(token) => persist_sender_token(&path, token),
|
||||
SenderSource::Unavailable => anyhow::bail!(
|
||||
"matrix: no sender token in the swarm store or at {}; swarm-controller \
|
||||
mints it into the store",
|
||||
path.display()
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// What [`ensure_hive_user`] does about the sender token this sweep.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
enum SenderSource<'a> {
|
||||
/// The file already holds the token to use.
|
||||
Keep,
|
||||
/// Write the store's token to the file.
|
||||
Store(&'a str),
|
||||
/// No token in the store or the file.
|
||||
Unavailable,
|
||||
}
|
||||
|
||||
/// Decide [`ensure_hive_user`]'s step from the store's token and the file's
|
||||
/// content, each `None` when absent. Blank counts as absent.
|
||||
fn sender_source<'a>(stored: Option<&'a str>, on_disk: Option<&str>) -> SenderSource<'a> {
|
||||
let present = |s: &&str| !s.trim().is_empty();
|
||||
let stored = stored.map(str::trim).filter(present);
|
||||
let on_disk = on_disk.map(str::trim).filter(present);
|
||||
match (stored, on_disk) {
|
||||
(Some(s), Some(d)) if s == d => SenderSource::Keep,
|
||||
(Some(s), _) => SenderSource::Store(s),
|
||||
(None, Some(_)) => SenderSource::Keep,
|
||||
(None, None) => SenderSource::Unavailable,
|
||||
}
|
||||
}
|
||||
|
||||
/// Write the appservice sender account's access token to `path`, 0600, creating the
|
||||
/// directory if it is not there. The file's mode is the only thing keeping an
|
||||
/// unprivileged reader off the hive's matrix credential.
|
||||
fn persist_sender_token(path: &std::path::Path, access_token: &str) -> Result<()> {
|
||||
crate::secret_file::write_secret_0600(path, format!("{access_token}\n").as_bytes())
|
||||
.with_context(|| format!("matrix: write the sender token to {}", path.display()))?;
|
||||
tracing::info!(path = %path.display(), "matrix: provisioned the sender token");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetch the sender token `swarm-controller` published, under this hive's own
|
||||
/// store identity.
|
||||
///
|
||||
/// The cert role is the hive's name, straight out of `HYPERHIVE_HIVE_NAME` —
|
||||
/// the same role string `lifecycle::agent_identity` logs in with, and already in
|
||||
/// this process's environment, so the store read costs no plumbing through
|
||||
/// [`ensure_all`]. That name is also the path's own hive segment, which is
|
||||
/// what makes this read reach **this** hive's token and 403 on any other's:
|
||||
/// the grant behind it is the hive's own `swarm/hives/<name>/*` stanza.
|
||||
///
|
||||
/// `None`, never an error, for every way this can come up empty — no hive
|
||||
/// name, no `BAO_*` identity, an unreachable store, nothing at the path. All
|
||||
/// four mean the same thing to the caller ("keep the file"), and three of
|
||||
/// them are the ordinary state of a swarm with no store token for this hive
|
||||
/// yet, so raising would turn a supported deployment into a warning every
|
||||
/// sweep.
|
||||
///
|
||||
/// 🩸 Logs the store **path** and never the value.
|
||||
async fn stored_sender_token() -> Option<String> {
|
||||
let hive = std::env::var("HYPERHIVE_HIVE_NAME")
|
||||
.ok()
|
||||
.filter(|h| !h.is_empty())?;
|
||||
let path = match swarm_secret_client::matrix::sender_token_path(&hive) {
|
||||
Ok(path) => path,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "matrix: this hive's name does not form a store path");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let store = match swarm_secret_client::SecretStore::from_env(&hive).await {
|
||||
Ok(store) => store,
|
||||
Err(e) => {
|
||||
tracing::debug!(error = %e, "matrix: no swarm secret store to read the sender token from");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
match store
|
||||
.read::<swarm_secret_client::matrix::Credential>(&path)
|
||||
.await
|
||||
{
|
||||
Ok(credential) if !credential.value.trim().is_empty() => {
|
||||
tracing::debug!(%path, "matrix: read the sender token from the swarm store");
|
||||
Some(credential.value)
|
||||
}
|
||||
Ok(_) => {
|
||||
tracing::warn!(%path, "matrix: the stored sender token is empty");
|
||||
None
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!(%path, error = %e, "matrix: no sender token in the store");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Discover the matrix `server_name` from the running homeserver via
|
||||
/// `GET /_matrix/key/v2/server` (unauthenticated federation key endpoint).
|
||||
/// The response JSON always includes `"server_name"` per the matrix spec.
|
||||
pub async fn discover_server_name(client: &reqwest::Client) -> Result<String> {
|
||||
let base = matrix_base()?;
|
||||
let url = format!("{base}/_matrix/key/v2/server");
|
||||
let resp = client
|
||||
.get(&url)
|
||||
.send()
|
||||
.await
|
||||
.context("matrix: GET /_matrix/key/v2/server")?;
|
||||
let status = resp.status();
|
||||
let body = resp
|
||||
.json::<serde_json::Value>()
|
||||
.await
|
||||
.context("matrix: parse /_matrix/key/v2/server response")?;
|
||||
if !status.is_success() {
|
||||
anyhow::bail!("matrix: /_matrix/key/v2/server returned HTTP {status}, body: {body}");
|
||||
}
|
||||
body["server_name"]
|
||||
.as_str()
|
||||
.map(str::to_owned)
|
||||
.with_context(|| {
|
||||
format!("matrix: /_matrix/key/v2/server response missing server_name field: {body}")
|
||||
})
|
||||
}
|
||||
|
||||
/// Read the sender token from disk. Returns an error if it
|
||||
/// is absent — callers should gate their homeserver calls on this.
|
||||
pub fn read_sender_token() -> Result<String> {
|
||||
let path = sender_token_path();
|
||||
std::fs::read_to_string(&path)
|
||||
.ok()
|
||||
.map(|s| s.trim().to_owned())
|
||||
.filter(|s| !s.is_empty())
|
||||
.with_context(|| {
|
||||
format!(
|
||||
"the matrix sender token was not found at {} — \
|
||||
ensure hive-c0re has started at least once with matrix enabled \
|
||||
(it provisions the hive's @hive-<hive>: account on boot)",
|
||||
path.display()
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
/// Fetch a user's current membership in a room via the sender token, or
|
||||
/// `None` if there is no membership event (never invited) or the lookup
|
||||
/// fails. Returns the raw membership string (`invite`, `join`, `leave`, …).
|
||||
async fn room_membership(
|
||||
client: &reqwest::Client,
|
||||
sender_token: &str,
|
||||
room_id: &str,
|
||||
user_id: &str,
|
||||
) -> Option<String> {
|
||||
let base = matrix_http()?;
|
||||
// A refused or failed read is "unknown": the caller falls through to the
|
||||
// invite attempt.
|
||||
let member = swarm_matrix_client::room_state(
|
||||
client,
|
||||
base,
|
||||
sender_token,
|
||||
room_id,
|
||||
"m.room.member",
|
||||
user_id,
|
||||
)
|
||||
.await
|
||||
.ok()??;
|
||||
member["membership"].as_str().map(ToOwned::to_owned)
|
||||
}
|
||||
|
||||
/// Invite a fully-qualified Matrix user id (`@user:server`) to `room_id`
|
||||
/// using the sender token. Idempotent: a user who is already a member or
|
||||
/// already has a pending invite is left untouched (no fresh invite is sent,
|
||||
/// so they are not re-notified). A refused invite is an error unless the
|
||||
/// user turns out to be invited or joined anyway — see [`forbidden_invite`].
|
||||
async fn invite_user_id(
|
||||
client: &reqwest::Client,
|
||||
sender_token: &str,
|
||||
room_id: &str,
|
||||
user_id: &str,
|
||||
) -> Result<()> {
|
||||
let base = matrix_base()?;
|
||||
|
||||
// Skip the invite entirely when the user is already invited or joined.
|
||||
// Re-POSTing an invite to a pending member re-sends the invite event,
|
||||
// which re-notifies the agent on every provisioning sweep.
|
||||
if let Some(membership) = room_membership(client, sender_token, room_id, user_id).await
|
||||
&& matches!(membership.as_str(), "invite" | "join")
|
||||
{
|
||||
tracing::debug!(%user_id, %room_id, %membership, "matrix: invite skipped (already a member/invited)");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
match swarm_matrix_client::invite(client, base, sender_token, room_id, user_id)
|
||||
.await
|
||||
.context("matrix: invite")?
|
||||
{
|
||||
swarm_matrix_client::Invited::Sent => {
|
||||
tracing::debug!(%user_id, %room_id, "matrix: invited to room");
|
||||
Ok(())
|
||||
}
|
||||
swarm_matrix_client::Invited::Forbidden => {
|
||||
// The pre-check misses a member when its read failed or a
|
||||
// concurrent invite landed after it, so a refusal re-reads the
|
||||
// membership.
|
||||
let membership = room_membership(client, sender_token, room_id, user_id).await;
|
||||
forbidden_invite(membership.as_deref(), user_id, room_id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Outcome of an invite the homeserver refused with 403. That is success only
|
||||
/// when `membership`, re-read after the refusal, shows the user invited or
|
||||
/// joined: `M_FORBIDDEN` covers both "already in the room" and real refusals
|
||||
/// (banned target, sender lacks power), so the errcode can't tell them apart.
|
||||
fn forbidden_invite(membership: Option<&str>, user_id: &str, room_id: &str) -> Result<()> {
|
||||
if matches!(membership, Some("invite" | "join")) {
|
||||
tracing::debug!(%user_id, %room_id, "matrix: invite refused, user already member/invited");
|
||||
return Ok(());
|
||||
}
|
||||
anyhow::bail!(
|
||||
"matrix: invite {user_id} to {room_id}: HTTP 403 and the user is not invited or joined"
|
||||
)
|
||||
}
|
||||
|
||||
/// Invite an arbitrary Matrix user to `room` (a room id or alias). `user`
|
||||
/// may be a fully-qualified id (`@name:server`) or a bare localpart, which
|
||||
/// is qualified with the homeserver's `server_name`. Returns the resolved
|
||||
/// room id the invite targeted. Used by `hivectl matrix invite`.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns an error if `room` is neither a room id nor an alias, or the
|
||||
/// invite POST fails for a reason other than the user already being a
|
||||
/// member / invited.
|
||||
pub async fn invite_user(
|
||||
client: &reqwest::Client,
|
||||
sender_token: &str,
|
||||
user: &str,
|
||||
room: &str,
|
||||
server_name: &str,
|
||||
) -> Result<String> {
|
||||
// Qualify a bare localpart to a full user id on the hive homeserver.
|
||||
let user_id = if user.starts_with('@') {
|
||||
user.to_owned()
|
||||
} else {
|
||||
format!("@{user}:{server_name}")
|
||||
};
|
||||
let room_id = if room.starts_with('#') {
|
||||
swarm_matrix_client::resolve_alias(client, matrix_base()?, sender_token, room).await?
|
||||
} else if room.starts_with('!') {
|
||||
room.to_owned()
|
||||
} else {
|
||||
anyhow::bail!(
|
||||
"matrix: --room {room:?} is neither a room id nor an alias; \
|
||||
prefix with '!' for a room id (!abc:server) or '#' for an \
|
||||
alias (#name:server)"
|
||||
)
|
||||
};
|
||||
invite_user_id(client, sender_token, &room_id, &user_id).await?;
|
||||
Ok(room_id)
|
||||
}
|
||||
|
||||
/// Make sure the hive's own `@hive-<hive>:` sender token is on disk.
|
||||
/// Accounts are not created here: `swarm-controller` mints them with the
|
||||
/// swarm's appservice token. Runs only as a `MatrixSweep` job node (see
|
||||
/// `job_queue::templates::matrix_sweep`). No-op when no homeserver is
|
||||
/// configured.
|
||||
///
|
||||
/// Returns `true` when the token is in place, `false` when it could not be
|
||||
/// fetched — the caller feeds this into a
|
||||
/// [`crate::stats::sweep_health::SweepHealth`] to raise a debounced
|
||||
/// hive warning on persistent failure (this sweep re-runs every 30
|
||||
/// minutes, so a one-off blip self-heals without ever warning).
|
||||
pub async fn ensure_all() -> bool {
|
||||
if !is_present() {
|
||||
tracing::debug!("matrix: hive-matrix container absent, skipping user sweep");
|
||||
return true;
|
||||
}
|
||||
if let Err(e) = ensure_hive_user().await {
|
||||
tracing::warn!(error = ?e, "matrix: ensure_hive_user failed");
|
||||
return false;
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn with_no_file_the_store_token_is_taken() {
|
||||
assert_eq!(sender_source(Some("tok"), None), SenderSource::Store("tok"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_store_token_replaces_a_different_file_token() {
|
||||
// The swarm re-minted a dead token; the file must follow it.
|
||||
assert_eq!(
|
||||
sender_source(Some("new\n"), Some("old\n")),
|
||||
SenderSource::Store("new")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_file_matching_the_store_is_kept() {
|
||||
// Trailing newline on disk is how `persist_sender_token` writes it.
|
||||
assert_eq!(
|
||||
sender_source(Some("tok"), Some("tok\n")),
|
||||
SenderSource::Keep
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn with_nothing_in_the_store_the_file_is_kept() {
|
||||
assert_eq!(sender_source(None, Some("tok\n")), SenderSource::Keep);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn with_no_token_anywhere_nothing_is_available() {
|
||||
assert_eq!(
|
||||
sender_source(Some(""), Some(" \n")),
|
||||
SenderSource::Unavailable
|
||||
);
|
||||
assert_eq!(sender_source(None, None), SenderSource::Unavailable);
|
||||
}
|
||||
|
||||
/// A banned target or a sender without power gets a 403; with no
|
||||
/// membership behind it, the invite did not happen.
|
||||
#[test]
|
||||
fn a_refused_invite_without_membership_is_an_error() {
|
||||
for membership in [None, Some("ban"), Some("leave")] {
|
||||
let outcome = forbidden_invite(membership, "@a:x", "!r:x");
|
||||
assert!(outcome.is_err(), "membership {membership:?} must not pass");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_refused_invite_for_a_member_is_success() {
|
||||
for membership in ["invite", "join"] {
|
||||
forbidden_invite(Some(membership), "@a:x", "!r:x")
|
||||
.expect("already invited/joined is the goal state");
|
||||
}
|
||||
}
|
||||
|
||||
/// The homeserver's reply body reaches no error: `invite_user_id` is
|
||||
/// driven against a stub whose 500 and 403 bodies carry a marker.
|
||||
#[tokio::test]
|
||||
async fn an_invite_refusal_error_does_not_quote_the_response_body() {
|
||||
const BODY: &str = r#"{"errcode":"M_UNKNOWN","error":"free-form-marker"}"#;
|
||||
for status in [500_u16, 403] {
|
||||
let app = axum::Router::new().fallback(move || async move {
|
||||
(
|
||||
axum::http::StatusCode::from_u16(status).expect("a status"),
|
||||
[(axum::http::header::CONTENT_TYPE, "application/json")],
|
||||
BODY,
|
||||
)
|
||||
});
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
|
||||
.await
|
||||
.expect("bind");
|
||||
let base = format!("http://{}", listener.local_addr().expect("local addr"));
|
||||
let server = tokio::spawn(async move {
|
||||
axum::serve(listener, app).await.expect("serve");
|
||||
});
|
||||
let client = swarm_matrix_client::client().expect("client");
|
||||
let outcome = swarm_matrix_client::invite(&client, &base, "tok", "!r:x", "@a:x").await;
|
||||
server.abort();
|
||||
let message = match (status, outcome) {
|
||||
(403, Ok(swarm_matrix_client::Invited::Forbidden)) => {
|
||||
format!("{:#}", forbidden_invite(None, "@a:x", "!r:x").unwrap_err())
|
||||
}
|
||||
(500, Err(e)) => format!("{e:#}"),
|
||||
(status, other) => panic!("HTTP {status}: unexpected {other:?}"),
|
||||
};
|
||||
assert!(message.contains(&status.to_string()), "{message}");
|
||||
assert!(!message.contains("free-form-marker"), "{message}");
|
||||
}
|
||||
}
|
||||
|
||||
/// A room that is neither an id nor an alias is refused before any
|
||||
/// request is made; with no default Space left, there is nothing to fall
|
||||
/// back to.
|
||||
#[tokio::test]
|
||||
async fn invite_user_refuses_a_room_that_is_neither_an_id_nor_an_alias() {
|
||||
let client = swarm_matrix_client::client().expect("client");
|
||||
let err = invite_user(&client, "tok", "mara", "general", "example.org")
|
||||
.await
|
||||
.expect_err("a bare name is not a room");
|
||||
assert!(
|
||||
format!("{err:#}").contains("neither a room id nor an alias"),
|
||||
"{err:#}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The sweep touches nothing, and reports success, on a hive with no
|
||||
/// homeserver configured (the test environment has none).
|
||||
#[tokio::test]
|
||||
async fn the_sweep_is_a_successful_no_op_without_a_homeserver() {
|
||||
assert!(!is_present());
|
||||
assert!(ensure_all().await);
|
||||
}
|
||||
}
|
||||
|
|
@ -2,10 +2,10 @@
|
|||
//! `/run/hyperhive` + `/run/hive-agent` runtime roots).
|
||||
//!
|
||||
//! Historically these were flat string literals scattered across many
|
||||
//! modules (`broker.sqlite`, `matrix-sender-token`, `agent-sockets.json`,
|
||||
//! modules (`broker.sqlite`, `agent-sockets.json`,
|
||||
//! …). This module is the **single Rust-side source** for every host
|
||||
//! path — the strictly host-side ones grouped into subdirs (`db/`,
|
||||
//! `matrix/`, `run/`), plus the **nix-coupled** roots
|
||||
//! `run/`), plus the **nix-coupled** roots
|
||||
//! (`agents/`, `applied/`, `meta/`, `gateway/`, `knowledge/`,
|
||||
//! the `/run` runtime dirs).
|
||||
//!
|
||||
|
|
@ -89,21 +89,6 @@ pub fn agent_identity_dir(name: &str) -> PathBuf {
|
|||
agent_identity_root().join(name)
|
||||
}
|
||||
|
||||
/// `matrix/` — host-side matrix provisioning state (the appservice sender token, hive
|
||||
/// Space room id, the hive sender account's password creds). The shared
|
||||
/// registration token is bind-mounted into the tuwunel container via nix
|
||||
/// and stays at its own path (tracked separately).
|
||||
#[must_use]
|
||||
pub fn matrix_dir() -> PathBuf {
|
||||
state_root().join("matrix")
|
||||
}
|
||||
|
||||
/// `matrix/access-token` — the matrix appservice sender account's access token.
|
||||
#[must_use]
|
||||
pub fn matrix_sender_token() -> PathBuf {
|
||||
matrix_dir().join("access-token")
|
||||
}
|
||||
|
||||
/// `run/` — runtime maps hive-c0re regenerates on every meta sync.
|
||||
#[must_use]
|
||||
pub fn run_dir() -> PathBuf {
|
||||
|
|
@ -252,10 +237,9 @@ pub fn agent_runtime_dir(name: &str) -> PathBuf {
|
|||
/// within the same filesystem is atomic.
|
||||
pub fn relocate_legacy_state() {
|
||||
let root = state_root();
|
||||
let moves: [(&str, PathBuf); 4] = [
|
||||
let moves: [(&str, PathBuf); 3] = [
|
||||
("broker.sqlite", db_dir().join("broker.sqlite")),
|
||||
("build_logs.sqlite", db_dir().join("build_logs.sqlite")),
|
||||
("matrix-sender-token", matrix_sender_token()),
|
||||
("agent-sockets.json", agent_sockets_file()),
|
||||
];
|
||||
for (old_rel, new) in &moves {
|
||||
|
|
|
|||
|
|
@ -203,7 +203,6 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
|
|||
// federation peer-config block) and open the web surfaces
|
||||
// (`hivectl open`).
|
||||
HostRequest::Urls => HostResponse::urls(hive_urls()),
|
||||
HostRequest::MatrixInvite { user, room } => handle_matrix_invite(user, room).await?,
|
||||
HostRequest::ReconcileConfigStatus { agent, verbose } => {
|
||||
crate::forge::reconcile_config_status(agent.as_str(), *verbose).await?
|
||||
}
|
||||
|
|
@ -343,25 +342,6 @@ async fn stream_agent_status(
|
|||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Matrix provisioning handlers
|
||||
//
|
||||
// The `hivectl matrix` subcommands used to run these in-process, which forced
|
||||
// the standalone CLI to link the whole daemon crate (matrix-sdk, reqwest, …).
|
||||
// They now run daemon-side over the host socket, where the daemon already holds
|
||||
// the sender token. Each op returns the operator-facing lines hivectl used to
|
||||
// `println!` in `HostResponse::messages` for the client to print verbatim.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Shared reqwest client for the matrix admin HTTP calls (30s timeout,
|
||||
/// mirroring the old in-CLI client).
|
||||
fn matrix_http_client() -> Result<reqwest::Client> {
|
||||
reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(30))
|
||||
.build()
|
||||
.context("build reqwest client")
|
||||
}
|
||||
|
||||
/// True when `name` has a state dir under the agents root, i.e. it's a
|
||||
/// managed agent rather than a bare (operator/human) account.
|
||||
///
|
||||
|
|
@ -375,21 +355,6 @@ fn agent_exists(name: &hive_types::Ident) -> Result<bool> {
|
|||
.with_context(|| format!("check agent state dir for {name}"))
|
||||
}
|
||||
|
||||
/// Guard: matrix provisioning needs a homeserver to provision against.
|
||||
///
|
||||
/// Answers "is one configured", not "is one running here": the address is
|
||||
/// the swarm's homeserver, wherever it runs.
|
||||
fn require_matrix_present() -> Result<()> {
|
||||
if crate::matrix::is_present() {
|
||||
return Ok(());
|
||||
}
|
||||
anyhow::bail!(
|
||||
"no matrix homeserver configured — set services.hyperhive.swarm.matrix.apiUrl (it defaults \
|
||||
to the swarm.matrix.gatewayHost vhost, and is null when that is) before provisioning \
|
||||
matrix users"
|
||||
)
|
||||
}
|
||||
|
||||
async fn handle_quota_enable() -> Result<HostResponse> {
|
||||
crate::priv_client::ensure_btrfs_quota()
|
||||
.await
|
||||
|
|
@ -498,26 +463,6 @@ async fn handle_push_snapshot(
|
|||
Ok(HostResponse::success())
|
||||
}
|
||||
|
||||
async fn handle_matrix_invite(user: &str, room: &str) -> Result<HostResponse> {
|
||||
require_matrix_present()?;
|
||||
let sender_token = crate::matrix::read_sender_token()?;
|
||||
let client = matrix_http_client()?;
|
||||
let server_name = crate::matrix::discover_server_name(&client)
|
||||
.await
|
||||
.context("discover matrix server_name")?;
|
||||
let room_id = crate::matrix::invite_user(&client, &sender_token, user, room, &server_name)
|
||||
.await
|
||||
.with_context(|| format!("matrix invite {user}"))?;
|
||||
let target = if user.starts_with('@') {
|
||||
user.to_owned()
|
||||
} else {
|
||||
format!("@{user}:{server_name}")
|
||||
};
|
||||
Ok(HostResponse::messages(vec![format!(
|
||||
"matrix: invited {target} to {room_id}"
|
||||
)]))
|
||||
}
|
||||
|
||||
/// Single-agent queue verbs the admin socket exposes. Each submits the
|
||||
/// matching DAG (persisting the `wanted` intent, serializing on the
|
||||
/// agent's lease, with the transient/crash-watch suppression the old
|
||||
|
|
|
|||
|
|
@ -212,7 +212,6 @@ fn submit_startup_sweep_nodes(coord: &Arc<Coordinator>) {
|
|||
|
||||
if let Err(e) = coord.job_queue.insert_job(|b| {
|
||||
let _ = b.node(NodeKind::ForgeSweep);
|
||||
let _ = crate::job_queue::templates::matrix_sweep(b);
|
||||
let _ = crate::job_queue::templates::knowledge_pull(b);
|
||||
let _ = b.node(NodeKind::WantedPull);
|
||||
Vec::new()
|
||||
|
|
|
|||
|
|
@ -223,9 +223,6 @@ pub enum HostRequest {
|
|||
#[serde(default)]
|
||||
scope: LifecycleScope,
|
||||
},
|
||||
/// Invite a matrix user to `room` (id or alias). Uses the daemon's
|
||||
/// admin token; idempotent (already-member / already-invited is a no-op).
|
||||
MatrixInvite { user: String, room: String },
|
||||
/// Report the divergence between agent `agent`'s local applied config
|
||||
/// checkout and its forge `agent-configs/<agent>` `main`. The daemon
|
||||
/// fetches forge `main` read-only and returns a human-readable report
|
||||
|
|
@ -452,7 +449,7 @@ pub struct HostResponse {
|
|||
pub nodes: Option<Vec<hive_jobq_wire::GraphNode>>,
|
||||
/// Free-form operator-facing output lines the client prints verbatim
|
||||
/// (one per line). Carries results a request produced daemon-side that
|
||||
/// have no structured home — e.g. the invited room id from `MatrixInvite`.
|
||||
/// have no structured home.
|
||||
/// Empty for requests that produce no such output.
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub messages: Vec<String>,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
The operator-facing host CLI. A thin client for the `hive-c0re`
|
||||
daemon — speaks the host admin socket protocol (`hive-host-sock`) and
|
||||
does not link the daemon crate. Container lifecycle and the
|
||||
matrix/gateway verbs all forward to the daemon and need it running; a few (`wg`/`peer-config`, `choom`) work off local
|
||||
gateway verbs all forward to the daemon and need it running; a few (`wg`/`peer-config`, `choom`) work off local
|
||||
host state instead.
|
||||
|
||||
## When to use it
|
||||
|
|
@ -27,7 +27,6 @@ dispatch:
|
|||
container.
|
||||
- **`forge.rs`** — reconciles an agent's config against its forge
|
||||
repo.
|
||||
- **`matrix.rs`** — invite a matrix user to a room.
|
||||
- **`wg.rs`** — WireGuard mesh helpers.
|
||||
- **`subvol.rs`** — btrfs state-subvolume ops.
|
||||
- **`quota.rs`**, **`util.rs`**, **`completions.rs`**, **`open.rs`** —
|
||||
|
|
|
|||
|
|
@ -10,8 +10,7 @@ use std::path::PathBuf;
|
|||
about = "hyperhive host CLI — operator-facing administration",
|
||||
long_about = "\
|
||||
Sibling to the `hive-c0re` daemon binary. Covers host-side admin \
|
||||
operations that don't go through the broker — manual user \
|
||||
provisioning on the bundled matrix container, plus future \
|
||||
operations that don't go through the broker, plus future \
|
||||
recovery / debugging verbs.\
|
||||
"
|
||||
)]
|
||||
|
|
@ -36,13 +35,6 @@ pub enum Cmd {
|
|||
#[command(subcommand)]
|
||||
cmd: ForgeCmd,
|
||||
},
|
||||
/// matrix-tuwunel invites.
|
||||
///
|
||||
/// Manual invites into a room by id or alias.
|
||||
Matrix {
|
||||
#[command(subcommand)]
|
||||
cmd: MatrixCmd,
|
||||
},
|
||||
/// Lifecycle actions on ONE managed agent container. Needs the
|
||||
/// hive-c0re daemon running.
|
||||
///
|
||||
|
|
@ -246,19 +238,6 @@ impl From<ReconcileFrom> for hive_host_sock::ReconcileDirection {
|
|||
}
|
||||
}
|
||||
|
||||
#[derive(Subcommand)]
|
||||
pub enum MatrixCmd {
|
||||
/// Invite a matrix user to a room. Idempotent.
|
||||
Invite {
|
||||
/// User to invite: a full id (`@mara:server`) or a bare
|
||||
/// localpart (qualified with the homeserver's `server_name`).
|
||||
user: String,
|
||||
/// Target room id (`!abc:server`) or alias (`#name:server`).
|
||||
#[arg(long)]
|
||||
room: String,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Subcommand)]
|
||||
pub enum WgCmd {
|
||||
/// Generate this hive's WireGuard key (if absent) and print its public
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue