The comment said "It parses; it does not prove a sample arrives",
which understates the gap in the direction that matters: it reads as
if a green build proves the collector *starts* and only runtime
delivery is unverified.
Measured while probing ingest-auth options for #3283: `otelcol
validate` ACCEPTS a receiver naming an auth extension that is absent
from the build, and the collector then dies at startup with `Failed
to start component`. So the check does not prove this config starts
at all.
Comment-only; no evaluated config changes.
Refs #3283.
Retention so a reader can ask what a hive last said without anyone
keeping a second copy. The container is ephemeral = false, so the store
survives a restart with no bind mount, and losing it degrades to the
true answer (nothing known) rather than a stale one.
The per-account grant is the part that is easy to miss: the server
option gives the SERVER JetStream, and an account gets it only from its
own. Measured against a running 2.14.1 with this exact two-account
shape - global-only makes kv operations fail 10039 while the server
starts cleanly and logs Starting JetStream, so no rendering check can
see it. Deliberately not granted to the callout account: the responder
mints credentials and has no business holding stream state.
argus on #3280: the receiver has no auth extension - the nixpkgs module
passes settings straight through and nothing wires one on - so
'unauthenticated to a bridge address' means reachable from an agent
container, not presents a credential. The firewall is the whole access
control.
Consequence, stated because it is a choice rather than an oversight:
any agent can push arbitrary OTLP and the collector forwards it under
the operator's credential, including data smuggled out in resource
attributes. That is a different risk from the one the collector fixes,
and strictly smaller than what preceded it - before, every agent held
the credential itself and could do all of that plus use the token
anywhere else. The collector removes the token and keeps the pipe.
Same principle this PR already applies to the availability trade: state
it where the reader is, rather than let it be discovered.
The headersCredential section still said hive-c0re forwards the file
into each agent container - that is exactly the behaviour this change
removes, so the doc contradicted the code rather than merely lagging
it. Same for the collector section, which presented the two paths the
change collapses.
Also drops the dangling collector.enable reference in the network
section.
mara: 'there should only be the one via otel-collector'. Collapses
collector.enable away - enabling otel runs the collector, and agents
always export to it.
The argument for it is the same one the collector exists for: as long
as the direct path stays selectable, the credential-in-settings.json
hole stays selectable. An option that can reintroduce a hole is a hole
with extra steps. HYPERHIVE_OTEL_HEADERS_CREDENTIAL is now never
forwarded to containers at all rather than conditionally.
Two consequences, both deliberate:
- the enable option's own promise changes. It said each harness exports
directly so telemetry survives anything host-side being down; that is
now false for every hive rather than for opted-in ones. Stated in the
option text rather than left for a reader to discover.
- the collector-requires-headersCredential assertion is gone. It was
correct only under the shape it was written for: when the collector
was opt-in FOR the credential, opting in without one was pure
indirection. With one path it isn't - an upstream needing no auth
header is legitimate, and the assertion would have rejected it. The
upstream header block is now omitted entirely in that case, rather
than rendering an ${env:...} reference nothing sets.
argus's non-blocking note on #3278: docs/observability.md is what this
repo's reading-paths index names as 'what OTEL options are available',
and it did not mention collector.enable/port/upstreamHeaderName at all.
The nix docstrings covered it, but not where a reader following the
established path would look.
Carries the two things a docstring is a poor home for: that endpoint
keeps meaning 'where telemetry ultimately goes' (the agent-facing value
is derived, so an existing deployment is unaffected), and the
availability trade the collector makes against the direct-export
property this page already promises.
The module hardcoded otlphttp, so a hive with protocol = "grpc" would
have had its agents' protocol respected end to end before the collector
and silently rewritten to HTTP after it. Splitting the path in two makes
the upstream half the one that has to keep honouring the option; the
agent half is pinned to the receiver's protocol and derived, not
configured.
The agent-facing endpoint becomes derived rather than `otel.endpoint`
re-defined: with the collector on, agents and hive-c0re's own exporter
send to the bridge address; with it off the expression is the old value
character for character. `otel.endpoint` keeps meaning "where telemetry
ultimately goes" on every hive that already sets it.
Suppressing HYPERHIVE_OTEL_HEADERS_CREDENTIAL is the half that actually
closes the hole this issue is about. The credential path host -> container
is sound (/run/credentials is dr-x------ root), but the hive-otel-header
oneshot then writes the value into the agent's own settings.json, mode
0600 and owned by the agent — so today every agent can read the hive's
upstream telemetry credential. With a collector there is nothing to
forward.
The protocol is derived for the same reason the endpoint is: the first
hop is now the collector's OTLP/HTTP receiver, while otel.protocol
describes the upstream link.
Renders services.opentelemetry-collector on the host: OTLP/HTTP receiver
on the bridge address, otlphttp exporter to otel.endpoint, and the
upstream credential delivered as EnvironmentFile so the collector
interpolates it at runtime and nix never sees the value.
Three things worth knowing, each measured rather than assumed:
- network.exposeHostPorts already exists and is wired (it opens the port
on the bridge interface only), so bridge reachability costs nothing.
- validateConfigFile defaults to isStorePath configFile, which is null on
the settings path - so upstream's default is OFF for exactly the way
this module configures it. Set true: it runs otelcol validate at build
time. It parses, it does not prove delivery.
- headersCredential's file is already NAME=value, i.e. EnvironmentFile
format, verified against a real settings.json rather than the doc.
An assertion refuses collector.enable with no headersCredential: the
collector exists to be the only holder of that token, and without one it
is indirection that reads as security.
Adds services.hyperhive.otel.collector.{enable,port}. No implementation
yet - this commit only declares the surface, so the option docs and the
default-off invariant can be reviewed on their own.
The reason the collector exists is the credential: without it every agent
needs headersCredential to reach the upstream, and the harness delivers
that token into the agent's own settings.json where the agent can read
it. The option text states the cost too - the collector becomes a
dependency in an export path that is currently direct.
Hit both today doing a live QA pass on the gateway: plain
nixpkgs#chromium has no color-emoji font (tofu boxes read exactly
like a missing-icon bug), and a page whose content loads async after
the initial paint (SSE, fetch-on-mount) screenshots as a bare
"loading..." placeholder without --virtual-time-budget. Neither is a
real product bug, but both need a documented fix for when an accurate
render actually matters, not just "know to discount it."
rustdoc builds without the test cfg, so `tests::hand_built_matches_the_reference`
resolves to nothing and broken_intra_doc_links denies it. Plain backticks
rather than making the item visible - a lint is not a reason to change an
item's visibility.
Earned, not lost: this is the first completed rustdoc run on a brand-new
crate, and the link was wrong from the first commit.
Per mara on the PR: all deps go into workspace level so versions stay
consistent. async-nats, data-encoding, nkeys and nats-jwt are new
[workspace.dependencies] entries; sha2 and futures-util now come from
there.
The crate had asked for sha2 0.10 while the workspace standard is 0.11,
and for the futures facade where the workspace carries futures-util. Both
resolved without adding a Cargo.lock entry - each was already in the tree
via something else - so the drift was invisible in the lock and would only
have surfaced as two versions of a hashing crate in one binary.
Every other crate has one and Cargo.toml already named it. Follows the
sibling shape: what it is, why it is a crate rather than more config, the
invariant that is easy to break (issuer_account must be absent, and
Token::new_user reintroduces it), and what the tests do NOT cover - the
introspection endpoint under test is a stub, so nothing here says anything
about the real authelia integration.
The discovery narrative belongs in the PR body, which carries it. What
stays at the line is what the code cannot say: that issuer_account must be
absent and why reaching for Token::new_user reintroduces the bug, the
BASE32HEX-vs-BASE32 distinction, and that nats-jwt is a test oracle rather
than a runtime dependency.
The responder answered granted=true and the server still refused every
client:
Error non operator mode account "AUTH": attempted to use issuer_account
nats_jwt::Token::new_user always sets issuer_account, which is an
operator-mode field naming the account when a signing key rather than the
account identity key signed the token. In server-config mode - what this
module renders - its mere presence is fatal, and the account is named by
the claims' aud instead.
nats-jwt can express neither aud nor the omission, so the user JWT is now
hand-built by the same signer as the response wrapper, and nats-jwt moves
to dev-dependencies as the encoder's test oracle.
Every unit test passed throughout: they assert fields that must be
present, and the defect was a field that must be absent.
Slice 1 shipped the NATS container with an auth_callout block and no
responder, which is the fail-closed state: the server answers
auth_required and admits nobody. This crate is what lets it say yes.
Connects as the callout-exempt user by nkey (never by name - the server
refuses to start if that entry carries a username), subscribes to
$SYS.REQ.USER.AUTH, validates the presented bearer token against
authelia's introspection endpoint, and replies with a signed NATS user
JWT. A denial is a signed response carrying an error, never silence: a
server that hears nothing cannot tell a refusing responder from a dead
one, so staying quiet would turn every rejection into a timeout and hide
an outage inside what looks like ordinary denials.
Everything that is not an explicit active:true denies - network error,
timeout, non-2xx, unparseable body, no token at all. Those are exactly
the conditions under which an attacker would most like this to fall
open. The introspection budget is held under the server's own 2s
auth_callout timeout by a test, since the two numbers live in different
languages in different files.
nats-jwt mints the user JWT. It cannot mint the authorization_response
wrapper - its claim enum is closed and its claims carry no aud, which
the response needs so a reply cannot be replayed at another server in
the cluster - so that half is hand-written, and a test builds a user
token both ways and requires the bytes to match. That is the only
honest basis for trusting the hand-written path on the shape the crate
does not model.
async-nats is taken with default-features off: the default set carries
jetstream, kv, object-store, websockets and service, none of which a
callout responder speaks.
mara: comments to code ratio too high. It was 16 comment lines for one
line of feature.
Kept only the fact that stops the feature being trimmed away again - it
is not a transport, it is in `default`, so `default-features = false`
drops it - and, in hive-metric, the one reason its declaration
deliberately differs. Why an HTTP 200 is the failure that matters, and
what a stderr-reporting CLI would cost, are the PR's and the follow-up
issue's job, not the manifest's.
`default-features = false` on opentelemetry-otlp was written to trim
transports and dropped `internal-logs` with them, so every otel_warn! and
otel_debug! inside that crate compiled to nothing. The failure that
matters is the one which returns HTTP 200: the collector accepts the
request and rejects the data points, and HttpMetricsClient.PartialSuccess
is the only place the rejection count and the collector's reason are ever
surfaced.
The feature is per-crate, not per-workspace: the macros are exported by
the opentelemetry API crate but their cfg and CARGO_PKG_NAME resolve in
the calling crate, so the API crate and the SDK had internal logs on
while the exporter did not.
hive-metric keeps the identical declaration on purpose - it installs no
tracing subscriber, so the feature would be inert there and would only
mislead. Both files now record which side of that they are on, and the
stale "same features as hive-metric" comment is corrected.
Not sufficient on its own: a hard export failure is logged by the SDK at
debug on the compiled timer path, so it stays below the info filter.
That needs a level rather than a feature and is left to review.
A bare `git pull --ff-only` merges whatever `branch.<current>.merge`
lists. A clone carrying more than one such entry aborts with "Cannot
fast-forward to multiple branches", which takes /knowledge out for every
agent on the hive - and the daemon neither writes that config nor can
see it, so the call worked only for as long as it happened to stay
clean.
Reproduced against two throwaway bare repos: duplicate merge entries
give that exact fatal, exit 128. A detached HEAD gives a different
error, and a missing tracking config does not reproduce at all - the
clone of an empty repo does write the tracking entry, so my first
explanation was wrong.
Naming origin and main makes the pull independent of local branch
config: a stray entry degrades to "the pull did not pick it up" instead
of breaking the shared mount. Does not explain how a second entry
appeared; nothing in this tree writes branch config.
Measured against a running server built from this module's own rendered
config, not a hand-written one -- which is the fix to the method as much
as to the file. Two shapes died in a row: a bare name authenticates
anyone who sends it, and a name plus an nkey refuses to start at all
(nkey users do not take usernames or passwords).
A malformed key is fail-closed too, so the only way to a live server
here is a real key whose seed nobody but the responder holds.
argus caught it and reproduced it: a users entry carrying only a name
has no credential, so CONNECT {"user":"auth"} is accepted with no
password. The name is a literal in this module, so on the shared netns
that identity was walk-in-able from every agent container -- the same
hole the module exists to close, moved rather than closed.
An nkey rather than a password: only the public half appears in config,
the seed reaches the responder alone, so a hive with no responder has
nobody who can authenticate as it. Asserted at eval, because the
rendered config is valid and the server starts either way.
A swarm has one message queue; this is the container that runs it, in
the same shape as every other swarm service. Off by default, and off
means absent.
The auth_callout block is the interim state and the final one at once.
Measured on the pinned nats-server rather than reasoned about: an empty
authorization block, and one with an empty users list, both accept an
anonymous client and answer PONG. auth_callout sets auth_required and
refuses everyone until a responder approves them, so a config whose
responder does not exist yet denies by default -- and the responder
lands beside it without any of this being swapped out.
That matters more here than it would elsewhere because the container
shares the host netns, so an unauthenticated interim state would be
reachable from every agent container on the hive.
No package option, deliberately: nixpkgs' nats module resolves
pkgs.nats-server itself and exposes none, so one here would name
something it cannot control.
No gateway vhost either: NATS speaks its own TCP protocol, nginx cannot
front it, and cross-hive reach is the wireguard mesh.
Per mara: setup should point at it. The link is placed as a precondition
rather than a see-also -- every step below assumes each credential is
generated where it is read, which is only true all-local.
Per argus: 'with a bound' was missing a word; it now names the actual
120s wait instead of gesturing at one.
Per mara on the CA question: outside all-local this is an ops problem,
and what is missing is documentation rather than machinery.
One row per secret, read against three topologies, because the same
credential is generated in place on one and handed over by an operator on
another. sso.md's table is replaced by a pointer -- two tables listing the
same secrets would drift, and its prose about why a secret is generated
in-container is the half worth keeping there.
States the constraint the whole thing rests on: a hive CA is name-
constrained to the hive domain and a swarm service name is a sibling of
it, so a hive CA cannot issue a swarm-service certificate at all. That is
why placing the swarm root is an operations decision.
The #2595 job-queue rework replaced hive-bash-mcp's direct broker send
with a todo push through the in-agent socket, so a bash-task-driven
turn now wakes with from == "todo", never "bash-task-<id>". This
branch has had nothing to match since, so it silently bucketed no
turns under the bash-task label; removed rather than left dead.
Closes#2256.
The registration half is shared; what each service does with the result
is not, so the differences get a table rather than a second page.
States why matrix reads its secret through LoadCredential and why
/_matrix/ is not a forward-auth surface -- both are conclusions a reader
would otherwise have to re-derive from the module.
Runs on the host: the two containers share a network namespace but not a
filesystem root, so this is the only place both trees are addressable.
A copy rather than a bindMounts entry. nixos-container refuses to start
when a bind source is missing, and the secret does not exist until
authelia's first boot has minted it. The registration token dodges that
with an activation script that pre-creates the file; that is unavailable
here, because tuwunel requires the secret to exist and be non-empty, so a
placeholder would satisfy the mount and then stop the homeserver.
Bounded wait then fail, never a silent skip: authelia's container can be
up while its generator is still minting.
Stage 1's actual login path. The provider entry is rendered only when
sso.enable is set, via optionalAttrs rather than an empty list, so a hive
that has not opted in renders byte-identical tuwunel settings.
The secret reaches tuwunel through LoadCredential rather than as a direct
path: upstream requires the file to survive systemd sandboxing and
suggests /etc/tuwunel/, which this container has no writable etc for. The
registration token two units below already solves it the same way, and
for the same reason -- DynamicUser and PrivateUsers stay intact.
Three assertions fail at eval instead of at boot. tuwunel reads identity
providers from its config file, so a half-configured one does not hide a
login button, it can stop the homeserver from starting.
Stage 1 of the staged rollout mara laid out on the issue: the options
and nothing that changes a running hive. `enable` defaults false, so
this commit is inert until someone opts in.
Three rules taken from hive-forge/default.nix rather than re-derived,
since it is the same pattern already merged and reviewed:
- `enable` ADDS a login path and leaves password login alone. An
identity provider that can take the homeserver offline when it
hiccups is worse than two ways in. Making authelia the only path is
tuwunel's `login_with_password`, a separate and reversible switch,
deliberately not folded in here.
- the client secret is a PATH, never a value: it has two holders in two
containers (authelia keeps a hash, the homeserver needs plaintext),
and a literal would be rendered into the world-readable store.
- required when enabled, no fallback. Worse here than for the forge:
tuwunel reads OIDC from its config file rather than a database row,
so a malformed block can stop the server outright instead of merely
hiding a button.
Also records the constraint that shapes the whole issue — matrix SSO
lives inside the homeserver, never behind a forward-auth proxy,
because the client-server API is spoken by non-browser clients holding
matrix access tokens plus federation.
Per mara's review: "describe what is, not what is not."
The header said "the swarm services' vhosts are NOT here", which
describes the file by its absences and dates immediately — the list of
what is missing grows every time a service is added, while the list of
what the file holds does not.
Reworded to say what it is: the hive's own gateway surface (the `_`
default server and the vhost named for the hive domain), with the scope
line stating where a swarm service's vhost lives rather than where it
doesn't.
The header still announced "the forge, matrix and authelia sub-domain
vhosts" and an "Accept-header SPA map". None of the three is true:
- the four swarm vhosts moved into their own service modules, so the
header named contents that are no longer in the file;
- `_` is described as serving the dashboard and per-agent routing, but
it is now a bare 444 catch-all — that surface moved to the vhost
named for the hive domain;
- the "Accept-header SPA map" is not merely gone, it is the mechanism
the file's own dashboard comment argues *against* ("routing is by
PATH, never content-type"), 140 lines below the header claiming it.
A file's opening comment is its orientation text — the first thing read
and the last thing checked — so a stale one costs more than a stale
line in the body. Rewritten to describe what the file holds now, and to
name the property the split established: a service owns its whole
gateway surface in its own module, and a service name appearing here is
the signal that the coupling is growing back.
mara asked whether these can live in Cargo.toml. They can, and they
should: [workspace.lints.rustdoc] sits alongside the existing clippy
table, every crate already opts in via `[lints] workspace = true`, and
the toolchain is well past the 1.74 that introduced lint tables.
The reason it is better than RUSTDOCFLAGS on the check: a plain local
`cargo doc` now fails exactly the way CI does. Setting the lints only in
the nix derivation would have made CI the one place the gate exists,
which is the same "you meet it too late" problem the gate was written to
solve.
The check keeps --workspace --no-deps --document-private-items and drops
the RUSTDOCFLAGS block entirely.
Verified by mutation rather than assumption, with no RUSTDOCFLAGS set
anywhere: clean tree exits 0; reintroducing one broken link makes
`cargo doc -p hive-jobq` exit 101 with `error: public documentation for
Outcome links to private item Scheduler::complete`. An error rather than
a warning is the proof the deny came from the lint table.
Nothing in the gate read doc-comments: clippy doesn't check intra-doc
links, cargo test doesn't, and no check built docs. So a [`Foo`] pointing
at a renamed, moved or deleted item rendered as plain text and had no
discoverer but a human happening to read the comment.
That matters here more than in most repos, because the convention is to
put a thing's authoritative description in one doc-comment and point at
it from everywhere else -- the design leans on the pointers being real,
and a dangling link is worse than no link since it names something and
sends the reader looking.
Adds `docs-rustdoc` to nix/checks.nix: craneLib.cargoDoc over
--workspace --no-deps --document-private-items, denying six rustdoc
lints. Listed explicitly rather than -D warnings so a new lint appearing
upstream cannot red the build on a class nobody has triaged.
--document-private-items is load-bearing rather than thoroughness for
its own sake: most of this workspace's doc-comments live on private
items and //! module headers, so without it rustdoc checks a small
fraction of the links and the gate sits green while the rot continues.
Then fixes every error it reports, 40 to 0 across nine crates. The
classes differ and so do the fixes:
- public item, wrong scope -> qualify. Node and Node::parent are both
public; the link failed only because scheduler.rs does not import
Node. Six sites become [`crate::Node::parent`].
- private item -> downgrade to backticks. Nothing was made public to
satisfy a lint; changing API surface to appease a doc check would be
the tail wagging the dog.
- genuinely dead -> [`JobBuilder::insert_into`] names a method that does
not exist. Insertion is Scheduler::insert_job.
- prose that looks like markup -> argv[0] parsed as a link, and
<args>/<hex>/<name> parsed as HTML tags.
Note for future fixes: pub(crate) resolves in an intra-doc link, a plain
private fn in a binary crate does not (wait_for_nodes resolved,
connect_hint did not, same crate, same shape).
The check does not ride the clippy/test artifact cache. It takes
cargoArtifacts, but rustdoc needs its own flavour of dependency
metadata, which cargo build does not produce, so a --no-deps docs build
still compiles dependencies it never documents. Measured at 6m47s cold;
that reasoning is recorded in the check's own comment so the next reader
does not re-derive it.
Verified by running the check's exact command against the pre-cleanup
tree first: 40 errors, build failed. A gate that cannot fail is not
evidence, and building it before the cleanup makes that proof free.
