swarm.matrix.enable was the last toggle still spelled swarm.*, which
is the namespace every host in the swarm agrees on - and "does this
host run the homeserver" is exactly what differs between them. It
moves to deploy.matrix.enable with the rest; the genuinely swarm-wide
matrix settings stay where they are.
mkRenamedOptionModule keeps existing configs evaluating with one
warning naming both paths.
The sweep for readers had to be unanchored and cross-language: three
of them were outside nix, including a hive-c0re error message telling
the operator to set the old name. A dotted grep also cannot see the
rename module's own list form, so that was checked separately.
Nine of the 38 .nix tokens mentioned anywhere in *.rs did not name
anything that exists. Twelve mentions, five distinct targets:
hive-c0re.nix, hive-gateway.nix and hive-forge.nix are all directories
now; nix/modules/ is not a directory we have; hive-forge-tools.nix was
a bash script the binary replaced and is gone.
Where the reference is load-bearing it is corrected rather than
deleted, because the reference is the point: a comment saying a
constant must match a nix literal is only useful if you can open the
file it names. Where the module member was unambiguous the path now
names it exactly.
paths.rs's STATE_ROOT marker was the worst of them: it claimed the
value came from services.hyperhive.c0re.statePath, in hive-c0re.nix.
Neither exists. The option is not declared anywhere and the file is a
directory, so a "must match" contract pointed at two things that
cannot be opened. /var/lib/hyperhive is hardcoded on both sides, which
is what the comment now says.
hive-forge-tools.nix keeps no replacement: naming a file that was
deliberately deleted helps nobody, and "replaces a prior bash script"
is complete without it.
Measured before and after with the same command: 9 unresolved of 38
before, 4 of 35 after. The remaining four are an example path in a doc
comment, an upstream nixpkgs path, and two from one synthetic test
fixture.
Takes the crate from 26 rustdoc warnings to 1, on top of the ten in the
previous commit.
argus's review findings:
- agent_sockets.rs: [`write`] was still ambiguous (function vs macro).
The previous change narrowed the qualifier and left the ambiguity;
[`write()`] is what resolves it.
- forge/users.rs <hex> and stats/container_stats.rs <name>: unclosed
HTML tags in prose, now backticked.
The rest of the crate, so the count actually reaches zero:
- job_queue/mod.rs: Queue::graph_snapshot -> JobQueue::graph_snapshot
(there is no Queue type), and super::scheduler -> scheduler (mod.rs
*is* job_queue, so super:: pointed outside it)
- job_queue/resource.rs: NodeKind -> super::model::NodeKind
- matrix.rs: password_path(name) -> password_path; and
forge::provision_user_token -> crate::forge::provision_user_token.
Note the path has no `users` segment: forge/mod.rs declares `mod
users` private and re-exports it, so the canonical path comes from the
re-export rather than the directory tree.
- socket_server/lifecycle_handlers.rs: InfraContainer ->
hive_priv_sock::InfraContainer
- stats/otel_metrics.rs: crate::meta::otel_config is a private fn no
path can name from another module, so it becomes prose
- main.rs: redundant explicit link target dropped
coordinator.rs:405 (CrashWatchGuard) is deliberately untouched: #3244
deletes that doc block, so fixing it here would conflict with an open PR
and repair a symbol that is about to stop existing.
argus's review note: current_room_state collapsed absent-state,
transport failure and an unparseable body into one None, so a
systematically failing GET was unobservable.
Only one of the three actually hides. A transport error takes the PUT
down with it one line later, and a 404 is the expected first-setup
case — both stay at debug. A non-404 HTTP failure is the silent one:
the read is broken while the write still succeeds, so the guard
switches off and the sweep resumes emitting with nothing to show for
it. That case, and only that case, warns.
Keeping the warn narrow is the point: one that also fired on every
expected 404 would train the reader to skip the line.
set_room_state PUT unconditionally and its doc called that idempotent.
It is, one level too high: a PUT of identical content is a no-op on the
room's STATE and the homeserver still appends an event to the TIMELINE.
Downstream an event is unread activity, then a todo, then a turn — for
every agent in the room. The provisioning sweep re-wiring the hive
Space's child link therefore woke the whole hive on the sweep's cadence,
forever, for a link that never changed (~1801s between events, measured
across eleven consecutive intervals).
Read the current content first and return early when it matches. The
lookup fails open — an unreadable state means write, because the
re-apply exists to repair a missing link and "don't know" must not be
treated as "fine". Only the steady state goes quiet.
Every prose and error-message reference to the moved options, including
two in hive-c0re/src/matrix.rs that omitted the services. prefix and so
read as the per-agent option. An error that names an option the operator
cannot find is worse than no hint.
`MATRIX_HTTP` was `http://localhost:8008`, compiled in, used at 18 call
sites. That address is right only while the homeserver happens to share
this daemon's netns, and its doc comment asserted exactly that as a
general fact. A hive whose homeserver lives anywhere else builds fine
and then talks to the wrong machine.
It now reads `HIVE_MATRIX_API_URL`, which `hive-c0re.nix` sets from
`hyperhive.matrix.apiUrl`. The matrix module fills that in with its own
loopback listener when it is the thing running tuwunel — there it is not
a guess but a fact about what it just started — and the operator sets it
by hand otherwise. There is no compiled-in fallback, for the same reason
`forge_http_base()` has none.
`is_present()` follows. It used to scan `nixos-container list` for
`hive-matrix`, which answers "is the homeserver a container on this
host" — a different question, and the reason a remote homeserver would
silently no-op no matter how it was addressed. It now asks whether a URL
is configured. A co-located hive is unaffected: the module supplies the
loopback URL whenever it runs tuwunel itself. It also stops being
`async`, since it no longer does IO, and `require_matrix_present`'s
message names both ways to have a homeserver rather than only the local
container.
Absent a URL, every matrix path no-ops exactly as it did with no
container, and the two accessors make that structural: `Option` for the
callers that fall back to `None`, a `Result` flavour naming the skipped
`is_present()` gate for the ones that propagate.
Third and last of #2860's agent-facing URL fallbacks. The operator's
ruling was "any special casing is done on the nix side - same binaries,
no hard coded fallback", so the default is deleted rather than replaced.
Every layer guessed the same wrong thing, and each guess was only ever
correct for a process sharing the host netns:
- nix/agent-modules/matrix.nix: matrixUrlDefault = localhost:8008, both
as the option's default and as a sentinel the daemon unit compared
against to decide whether to write HIVE_MATRIX_URL. Now nullOr str,
default null, the guard is != null, and the doc says what forge.url's
already says: null means "no matrix", not "guess one".
- nix/host-modules/hive-c0re/environment.nix: forwarded
http://127.0.0.1:<port> when no gatewayHost was set. hive-c0re shares
the host netns so it reads as harmless, but the value is handed to
agents, which do not -- there it names the agent itself. Now forwarded
only when there is a gateway vhost to name, matching the guard
HIVE_MATRIX_PUBLIC_URL already uses twelve lines below.
- hive-matrix-mcp: paths::DEFAULT_HOMESERVER was the same address
compiled in, so dropping the nix defaults alone would have left the
daemon dialling loopback inside the agent's own netns -- the very bug,
one layer down. homeserver_url() is now Option, and an account with no
homeserver is skipped with a log, exactly as one with no token is.
discover_token_accounts already refused to guess for the same reason.
Two comments taught the assumption back to the next reader ("shared host
netns means every agent container resolves localhost to the same
machine"); both now say which side of the netns boundary they describe.
MATRIX_HTTP keeps its value -- hive-c0re really does share the host
netns -- but no longer claims agents do.
Gated with nix eval against the extended agent-base config, as a pair:
with no url set the daemon unit carries no HIVE_MATRIX_URL, and with one
set it carries exactly that. Either check alone passes on a broken guard.
Per mara's standing calibration (#2463): extraction > silencing for
too_many_lines. Splits the space + chat-room provisioning tail into its
own fn, bringing ensure_all back under the 100-line threshold without
an #[allow].
Fast-follow for #2289 — matrix::ensure_all() (admin user, per-agent
sync, hive Space + chat-room provisioning/invites) ran periodically
every 30 minutes but only ever warn!'d to the journal on failure, so a
persistent problem (missing invites, broken admin token, etc.) was
invisible to the operator.
ensure_all() now returns bool (aggregate ok/fail across every
sub-step) instead of (), and both call sites in main.rs feed that into
a debounced SweepHealth("matrix_ensure_all", warn, threshold=2) —
matches the existing knowledge_pull pattern. A lone bad sweep
self-heals silently; two consecutive failures raise a banner that
clears on the next clean sweep.
forge::ensure_all()'s remaining independent steps are still open —
that sweep only runs once at startup (no periodic loop), so the
debounced pattern doesn't map as directly; left for a follow-up.
The restricted join rule needs room version >= 8. Pin it explicitly so a
future homeserver-default change can't silently invalidate the restricted
rule (which would fall back to invite-only and break the operator-joins-
from-the-Space path). Per review on the PR.
The hive Space was created empty — joining it surfaced no rooms because
Matrix doesn't auto-join a Space's children. Provision a default
"hive-chat" room on the matrix sweep, wire it bidirectionally to the
Space (m.space.child on the Space, m.space.parent on the room), and
invite @hive + every agent. The room uses a restricted join rule
allowing any Space member to join, so the operator (a Space member) can
join it from the Space hierarchy without an explicit invite.
Idempotent, mirroring ensure_hive_space: persisted chat-room-id wins,
else rediscover a non-space room named hive-chat, else createRoom. The
space-child link is re-applied each sweep (idempotent PUT) so a
recovered room reconverges its hierarchy link. Room id persisted to
matrix/chat-room-id (0600, survives destroy --purge).
Per mara's review: drop the #hive:<server> room alias (special chars) and
rediscover the canonical Space by its hardcoded plain name instead.
ensure_hive_space dedup is now:
1. room-id file present -> reuse it
2. else scan the admin's joined rooms for the m.space named HIVE_SPACE_NAME
('hive') and adopt the first match (re-persisting the file) -> recovers
the existing space after a state wipe instead of creating a duplicate
3. else createRoom (plain name, no alias)
find_space_by_name walks /joined_rooms and checks each room's m.room.create
type == m.space and m.room.name == 'hive'. No alias, no special-char anchor.
server_name is no longer needed by ensure_hive_space (dropped the param).
ensure_hive_space relied solely on the persisted room-id file. If that file
is ever lost (a full /var/lib/hyperhive wipe), the next sweep blind-creates a
new m.space — the homeserver keeps the old one, so duplicate hive spaces
accumulate (observed: multiple 'hive'/'pr1ma' rooms on the live instance).
Anchor the Space to a stable canonical alias #hive:<server>:
- fast path (room-id file present): reuse it and heal the alias mapping so
it keeps pointing at the canonical room
- no file: resolve #hive:<server> and adopt the existing room if present,
re-persisting the file — recovers the space after a wipe instead of
duplicating it
- only create (with the alias) when neither yields a room
server_name is now discovered before ensure_hive_space in ensure_all and
threaded through (the alias needs it). Existing deployments heal the alias
onto their current space on the next sweep; no new room is created when the
file is present.
Full `nix flake check` (CI) runs the treefmt formatting derivation. While the
hive-ci runner was offline (#1221), PRs merged without it, leaving 5 files
unformatted: hive-ag3nt/src/web_ui.rs, hive-c0re/src/bin/hivectl.rs,
hive-c0re/src/knowledge.rs, hive-c0re/src/matrix.rs,
hive-forge/src/verbs/attachment_get.rs. `nix fmt` output, pure formatting.
Per operator review on the PR: the prose marker list ('new password is:',
' to:', 'changed to:', etc.) was speculative — built from a misread
screenshot, not a real reply. The conduit admin bot always code-spans the
password, so the backtick anchor is the verified, complete format. Removing
the markers leaves a ~15-line function that's honest about what it parses.
If a non-code-span format ever appears, extract_new_password returns None and
the diagnostic logging in admin_room_send_and_poll records the raw body, so a
real format change is visible — far better than a speculative marker silently
mis-parsing it. Tests now cover the live format, symbol passwords, the
code-spanned-user-id error guard, and the no-codespan / non-password cases.
The live conduit admin-room reply (observed directly in #admins) is:
Successfully reset the password for user @x:server: `<password>`
The delimiter is ': ' after the user id, NOT ' to:' — so the ' to: ' marker
added earlier never matched the real format, and extract_new_password returned
None for every reply, timing out auto-recovery for every agent.
conduit always renders the password as a backtick code span, so anchor on
that directly: take the content of the first backtick pair when the message
is a password-reset success. Guard against a code-spanned matrix user id in
an error message (a real password has no whitespace and isn't @localpart:server).
Prose markers stay as a fallback for hypothetical non-code-span builds.
Tests cover the exact live format and the code-spanned-user-id error case.
mara observed the tuwunel reply renders the new password as a code span,
so the plain message body carries literal backticks:
Successfully reset password for @user:server to: `<password>`
The previous extraction stopped at the first whitespace, capturing the
surrounding backticks ("`<password>`") and producing a login string that
doesn't match the password the bot actually set — recovery would still
fail after parsing.
Strip a leading backtick after the marker and stop the token at the first
whitespace OR closing backtick. Generated passwords contain neither, so a
real password is never truncated mid-token. Two regression tests cover the
backtick-wrapped form, including trailing prose after the closing backtick.
extract_new_password had markers for 'is:', 'changed to:', 'reset to:',
'set to:' etc. but none match tuwunel's actual reset-password reply:
Successfully reset password for @user:server to: <password>
Here the verb ('reset') is not adjacent to 'to:', so every marker missed
and the function returned None for every polled event. The admin-room poll
then ran its full 15s loop without a match and the auto-recovery timed out
on every agent — even though the bot replied correctly and the backward
poll found the message. This is why matrix password auto-recovery kept
looping despite the poll-strategy fix.
Add a generic ' to: ' marker (with surrounding spaces) placed after the
specific verb markers and before the bare 'password:' last resort. Matrix
user ids and server names can't contain ' to: ', so it only ever anchors
on the prose delimiter. Two regression tests cover the exact production
wording.
The forward-pagination approach (dir=b anchor → dir=f poll) fails in
production: commands sent as @hive time out consistently even though
tuwunel responds. Root cause is likely a pagination-token direction
incompatibility in some tuwunel builds where 'end' from dir=b cannot
be used as 'from' for dir=f.
New strategy: send the command, capture the event_id from the PUT
response, then poll dir=b&limit=20 each tick. Events come back
newest-first; walk until we hit our own event_id, then stop — anything
before that marker arrived after our command. Simpler, avoids stored
tokens entirely.
Also:
- check formatted_body in addition to body (some admin bots put content
only in HTML formatted_body)
- add more extract_new_password patterns: 'changed to:', 'reset to:',
'set to:', 'new password:', 'password:' to handle different tuwunel
version response formats
- add unit tests for new patterns
Fixes#1283.
tuwunel can take longer than 5 seconds to process admin-room commands
during startup when the homeserver is under load. Bump both the poll
count (5→15) and the timeout message strings to match.
tuwunel can take longer than 5 seconds to process admin-room commands
during startup when the homeserver is under load. Bump poll count 5→15
and update timeout message strings to match.
tuwunel 1.6.x does not implement the Synapse admin REST API.
reset_user_password() now falls back to the Matrix admin room
(#admins:<server>) when PUT /_synapse/admin/v2/users returns 404:
1. Discover admin room ID via #admins:<server> alias
2. Get current messages end-token (pagination anchor)
3. Send 'reset-password @<localpart>:<server>' as @hive admin user
4. Poll for bot response up to 5 x 1s; extract password from message
5. Persist the new password and return it
The function signature changes from Result<()> to Result<String> so the
caller can use the effective password (which may be server-generated on
the admin-room path) for subsequent login calls.
Closes#1267.
Addresses mara's review: each credential type gets its own PrivRequest
variant, making the exact priv surface visible in the wire protocol.
No runtime filename dispatch — the operation name is the gate.
- WriteAgentForgeToken { agent_name, token } → state/forge-token
- WriteAgentMatrixToken { agent_name, token } → state/matrix-token
- priv_client: two typed fns (write_agent_forge_token, write_agent_matrix_token)
- forge.rs: split mint_and_persist_token into mint_and_persist_agent_token
(priv) + mint_and_persist_core_token (direct write); drop dead token_path fn
- matrix.rs: call write_agent_matrix_token directly
hive-c0re runs as the unprivileged hive-core user (privsep from #702)
and cannot write to agent-owned state directories. forge-token and
matrix-token writes were failing with EACCES on every startup sweep.
Add WriteAgentStateFile to PrivRequest: hive-priv (root) writes the
file 0600 and chowns it to the agent user so the agent can read it.
- hive-sh4re: add AGENT_STATE_ROOT constant + WriteAgentStateFile variant
- hive-priv: validate agent name + filename (no traversal), write via root
- priv_client: add write_agent_state_file helper
- forge: mint_and_persist_token routes agent paths through priv
- matrix: ensure_user_for routes matrix-token through priv
Closes#1257
forge_admin() spawned nixos-container run hive-forge directly from the
hive-core process. nixos-container run uses nsenter to enter the container
namespaces, which requires root. hive-core is unprivileged, so every call
failed with: nsenter: stat of /proc/<pid>/ns/user failed: Permission denied
Fix: add RunForgeAdmin { args } to PrivRequest. hive-priv (root) handles
it by spawning nixos-container run hive-forge -- runuser -u forgejo --
forgejo --work-path /var/lib/forgejo admin <args>. forge_admin() now calls
priv_client::run_forge_admin().
matrix: ensure_user_for hit M_USER_IN_USE then failed when the stored
password file was missing (state dirs wiped but homeserver kept accounts).
Previously required manual hivectl matrix reset-password <name>.
Fix: add auto_reset_password() — calls the admin API (PUT
/_synapse/admin/v2/users/@<name>:<server> with the hive admin token) to
set a new random password, then proceeds with login. Falls back to the
existing manual-recovery error if the admin token is unavailable.
Closes#1234
