vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

run the root check warn-only on hyperhive's own containers, per unit, not on the container@ template

Browse source
This commit is contained in:
atlas 2026-10-28 23:37:59 +01:00
commit 24afc89ab3
27 changed files with 707 additions and 172 deletions
Download patch file Download diff file Expand all files Collapse all files

48
docs/agent-lifecycle/persistence.md
View file

@ -385,32 +385,46 @@ never data.
**Checked before every start.** nixpkgs' start script runs as host root on
the root template, and its `mkdir -p`, `chmod`, `touch` and
`cp --remove-destination` follow a link the container planted at the paths
they touch. The host's nix config therefore adds a drop-in to the `container@`
template (`container@.service.d/`, from
`nix/host-modules/container-rootfs-check.nix`) with an `ExecStartPre=` that
runs `hive-priv check-rootfs %i`. It applies to every container instance, the
agents and the swarm's service containers alike, and runs on systemd's own
restarts too. systemd reads a drop-in after the unit file and runs the
`ExecStartPre=` lines in that order, all of them before `ExecStart=`, so the
check runs before the start script touches the root. The unit file names the
`hive-priv` store path, so the system closure keeps that build alive.
The check looks at each path with `lstat`, never following a link, and fails
the start with one line on stderr naming the path and its type when:
they touch. The container's limits drop-in
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
script, on systemd's own restarts too. It looks at each path with `lstat`,
never following a link, and fails the start with one line on stderr naming the
path and its type when:
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
exists and isn't a real directory;
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
(`etc/os-release` may also be a link, which the script skips and a booted
NixOS guest creates);
- the profile's `system` (`/nix/var/nix/profiles/per-container/<name>/system`)
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
exists and isn't a link that resolves into `/nix/store`, or the `system`
file inside the resolved directory isn't a regular file.
The name `%i` selects both paths, so the check refuses a name that isn't
lowercase ascii, digits and hyphens. Absent paths pass, so a first start works.
The check neither deletes nor repairs: the operator removes the offending
entry, and the next start goes through.
Absent paths pass, so a first start works. The check neither deletes nor
repairs: the operator removes the offending entry, and the next start goes
through.
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
points at a build without the sub-command. A drop-in written by an older
points at a build without the sub-command. That build's store path has a GC
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
drop-in and replaced on each write, so a garbage collection can't remove the
binary a restart would run. The directory sits beside `gcroots/per-container`,
which is the only gcroots path bound into a container, so only root writes it.
Destroying the container removes the drop-in and then its root; the build stays
until the next collection. A drop-in written by an older hive-priv has no
`ExecStartPre=` and no root, and the container starts as before.
**Hyperhive's platform containers: warn only.** The forge, the matrix server and
the swarm services (15 containers, declared by their own modules) run the same
check as `hive-priv check-rootfs --warn-only <name>`, from `ExecStartPre=` on
their own `container@<name>` unit. It evaluates every rule, logs each violation
as a warning, and exits 0, so the container starts either way. The line reads
`rootfs check (warn-only): <name>: <message>` with journald's warning priority.
Strict refusal needs the live roots of all 15 to pass first, and the warnings
show which ones would not. A container the operator runs that hyperhive doesn't
declare gets no check at all: the modules name their own containers, and
nothing enumerates the host's `containers`.
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and

4
docs/trust-boundary/security.md
View file

@ -263,8 +263,8 @@ known operations; there is no arbitrary command pass-through:
| `ReloadGatewayNginx` | `systemctl reload/start/reset-failed nginx` (host unit; the unit name is hard-coded, not a parameter) |
| `WriteNspawnFlags` | write `/etc/nixos-containers/<container>.conf` (bind-mount list, network isolation vars, and `--ephemeral` after the root's salvage). hive-priv first copies a stopped container's root to the agent's `work/`, then renames it aside (it deletes nothing), following no link the container could have planted |
| `EnsureAgentSocketDir` | `mkdirat` `/run/hive-agent/<name>` `0751 root:root` under an `O_NOFOLLOW` parent fd; leaves an existing directory alone, refuses anything else |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy |
| `WriteResourceLimits` | write `CPUQuota=`/`MemoryMax=`/`CPUWeight=`/`IOWeight=` systemd drop-in for agent container, with an `ExecStartPre=` that runs `hive-priv check-rootfs <container>` (no-follow check of the root paths the start script touches; see persistence.md), and a GC root on the build it names |
| `RemoveServiceDropin` | remove `container@<name>.service.d/` drop-in on destroy, and the GC root of the hive-priv build it named |
| `DaemonReload` | `systemctl daemon-reload` |
| `ControlInfraContainer` | `systemctl <action> nginx.service` for the gateway — the `InfraContainer` enum is the allowlist, and serde rejects unknown names at the wire boundary (`hive-c0re`, the forge and the matrix have no variant, so no request can name them) |
| `SyncAgentTmpfiles` | unlink `/etc/tmpfiles.d/hyperhive-agents.conf` and return `Ok` |

9
hive-priv-sock/src/lib.rs
View file

@ -369,7 +369,11 @@ pub enum PrivRequest {
/// Write `/run/systemd/system/container@<container>.service.d/hyperhive-limits.conf`
/// with `[Service]` carrying `MemoryMax=` / `CPUQuota=` (hard caps) and
/// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only).
/// `CPUWeight=` / `IOWeight=` (cgroup v2 relative shares, contention-only),
/// and an `ExecStartPre=` that runs `hive-priv check-rootfs <container>`
/// to refuse a start over a root with links where the start script
/// expects directories. The build that line names is rooted under
/// `/nix/var/nix/gcroots/hyperhive-check-rootfs/<container>`.
/// Written by `lifecycle::set_resource_limits`.
WriteResourceLimits {
container: String,
@ -388,7 +392,8 @@ pub enum PrivRequest {
io_weight: Option<u32>,
},
/// Remove `/run/systemd/system/container@<container>.service.d/` if present.
/// Remove `/run/systemd/system/container@<container>.service.d/` if present,
/// and the GC root of the hive-priv build its `ExecStartPre=` named.
/// Called by `lifecycle::destroy` to clean up the resource-limits drop-in.
RemoveServiceDropin { container: String },

117
hive-priv/src/container_config.rs
View file

@ -91,7 +91,7 @@ pub(crate) async fn handle_write_nspawn_flags_blocking(
/// the manager included) and return the logical agent name after it.
/// Anything else is refused, the swarm's forge and matrix containers
/// included: they belong to the host that runs them, not to a hive.
fn validate_container_system_name(name: &str) -> Result<&str> {
pub(crate) fn validate_container_system_name(name: &str) -> Result<&str> {
let Some(suffix) = name.strip_prefix(AGENT_PREFIX) else {
bail!("container name {name:?} is not managed by hive");
};
@ -100,13 +100,14 @@ fn validate_container_system_name(name: &str) -> Result<&str> {
}
/// `RemoveServiceDropin` — remove the container service's drop-in dir
/// if present (idempotent).
/// if present (idempotent), then the GC root of the build it named.
pub(crate) fn remove_service_dropin(container: &str) -> Result<(String, String)> {
validate_container_system_name(container)?;
let dir = format!("/run/systemd/system/container@{container}.service.d");
if Path::new(&dir).exists() {
std::fs::remove_dir_all(&dir).with_context(|| format!("remove {dir}"))?;
}
crate::prestart::unpin_binary(container)?;
Ok((String::new(), String::new()))
}
@ -114,6 +115,13 @@ pub(crate) fn remove_service_dropin(container: &str) -> Result<(String, String)>
/// container service's drop-in dir, together with a
/// `ConditionPathIsDirectory=` guard on the agent's MCP runtime dir.
///
/// The drop-in also carries the `ExecStartPre=` that runs
/// `hive-priv check-rootfs <container>` ([`crate::prestart`]) ahead of
/// nixpkgs' start script. It names this process's own binary, so the drop-in
/// and the binary that implements the check are always the same build. The
/// build is rooted against garbage collection for as long as the drop-in
/// names it ([`crate::prestart::pin_binary`]).
///
/// Two different kinds of setting land in the same file. `MemoryMax=` /
/// `CPUQuota=` are hard caps that throttle even on an idle host;
/// `CPUWeight=` / `IOWeight=` are cgroup v2 relative shares that only
@ -139,7 +147,19 @@ pub(crate) fn write_resource_limits(
let dir = format!("/run/systemd/system/container@{container}.service.d");
std::fs::create_dir_all(&dir).with_context(|| format!("create {dir}"))?;
let path = format!("{dir}/hyperhive-limits.conf");
let content = limits_dropin_body(&runtime_dir, memory_max, cpu_quota, cpu_weight, io_weight);
let exe = hive_priv_exe()?;
// Rooted before the drop-in names it, so the drop-in never names a build a
// collection can take.
crate::prestart::pin_binary(container, Path::new(&exe))?;
let rootfs_check = rootfs_check_command(&exe, container);
let content = limits_dropin_body(
&runtime_dir,
&rootfs_check,
memory_max,
cpu_quota,
cpu_weight,
io_weight,
);
// Published whole: a `daemon-reload` between truncate and write would
// otherwise load the unit with its limits and start guard missing.
publish_file(Path::new(&path), content.as_bytes(), 0o644, None)?;
@ -175,6 +195,10 @@ const RESTART_SEC: u32 = 5;
/// Render the body of `hyperhive-limits.conf`.
///
/// `rootfs_check` is the command line of the pre-start root check, the first
/// `[Service]` line: it runs before the nixpkgs start script and a non-zero
/// exit fails the start.
///
/// `[Unit]`: the condition is checked at start time — it skips (not fails)
/// the unit when the MCP socket dir is absent, avoiding restart loops —
/// plus the bounded start limit (see the constants above; `StartLimit*` are
@ -186,6 +210,7 @@ const RESTART_SEC: u32 = 5;
/// existed — or one whose nix option is `null` — renders no weight lines.
fn limits_dropin_body(
runtime_dir: &str,
rootfs_check: &str,
memory_max: &str,
cpu_quota: &str,
cpu_weight: Option<u32>,
@ -203,6 +228,7 @@ fn limits_dropin_body(
StartLimitBurst={START_LIMIT_BURST}\n\
\n\
[Service]\n\
ExecStartPre={rootfs_check}\n\
RestartSec={RESTART_SEC}\n\
MemoryMax={memory_max}\n\
CPUQuota={cpu_quota}\n\
@ -210,6 +236,36 @@ fn limits_dropin_body(
)
}
/// This binary's absolute path, for the drop-in's `ExecStartPre=`.
///
/// The path is the running binary's, not a fixed one, so a drop-in never names
/// a build without the sub-command. systemd expands `%` and `$` in command
/// lines and splits on spaces, so a path with any of them is refused rather
/// than escaped; store paths never have them.
fn hive_priv_exe() -> Result<String> {
let exe = std::env::current_exe().context("resolve the hive-priv binary path")?;
let exe = exe
.to_str()
.filter(|p| {
p.starts_with('/')
&& p.bytes()
.all(|b| b.is_ascii_alphanumeric() || b"/._+-".contains(&b))
})
.with_context(|| {
format!(
"hive-priv binary path {} is not safe in a unit file",
exe.display()
)
})?;
Ok(exe.to_owned())
}
/// The `ExecStartPre=` command of the pre-start root check for `container`:
/// the binary `exe`, then the sub-command and the name.
fn rootfs_check_command(exe: &str, container: &str) -> String {
format!("{exe} {} {container}", crate::prestart::SUBCOMMAND)
}
/// `DaemonReload` — `systemctl daemon-reload` on the host.
pub(crate) async fn daemon_reload() -> Result<(String, String)> {
let out = Command::new("systemctl")
@ -568,11 +624,15 @@ pub(crate) fn write_container_etc_file_in(rootfs: &Path, name: &str, contents: &
mod tests {
use super::{
BindMount, HiveFacts, NetworkIsolation, git_overlay_flags, handle_write_nspawn_flags,
limits_dropin_body, nspawn_flags, publish_file, remove_service_dropin, render_hive_facts,
validate_container_system_name, write_bridge_dns_marker_in, write_resource_limits,
hive_priv_exe, limits_dropin_body, nspawn_flags, publish_file, remove_service_dropin,
render_hive_facts, rootfs_check_command, validate_container_system_name,
write_bridge_dns_marker_in, write_resource_limits,
};
use crate::shared::test_support::{entries, scratch};
/// A stand-in for the pre-start check's command line.
const CHECK: &str = "/store/hive-priv/bin/hive-priv check-rootfs h-iris";
fn bind(container_path: &str) -> BindMount {
BindMount {
host_path: "/var/lib/hyperhive/whatever".to_owned(),
@ -630,13 +690,21 @@ mod tests {
#[test]
fn dropin_body_is_pinned_exactly() {
assert_eq!(
limits_dropin_body("/run/hyperhive/agents/iris", "4G", "200%", None, None),
limits_dropin_body(
"/run/hyperhive/agents/iris",
CHECK,
"4G",
"200%",
None,
None
),
"[Unit]\n\
ConditionPathIsDirectory=/run/hyperhive/agents/iris\n\
StartLimitIntervalSec=600\n\
StartLimitBurst=3\n\
\n\
[Service]\n\
ExecStartPre=/store/hive-priv/bin/hive-priv check-rootfs h-iris\n\
RestartSec=5\n\
MemoryMax=4G\n\
CPUQuota=200%\n"
@ -654,7 +722,7 @@ mod tests {
("512M", "50%", Some(10), Some(10)),
("infinity", "infinity", Some(10_000), None),
] {
let body = limits_dropin_body("/rt/x", mem, cpu, cw, iw);
let body = limits_dropin_body("/rt/x", CHECK, mem, cpu, cw, iw);
let unit = body
.split("[Service]")
.next()
@ -674,26 +742,55 @@ mod tests {
/// independently when `None`.
#[test]
fn weights_are_emitted_only_when_set() {
let both = limits_dropin_body("/rt/x", "4G", "200%", Some(80), Some(80));
let both = limits_dropin_body("/rt/x", CHECK, "4G", "200%", Some(80), Some(80));
assert!(
both.ends_with("CPUQuota=200%\nCPUWeight=80\nIOWeight=80\n"),
"{both}"
);
let cpu_only = limits_dropin_body("/rt/x", "4G", "200%", Some(80), None);
let cpu_only = limits_dropin_body("/rt/x", CHECK, "4G", "200%", Some(80), None);
assert!(
cpu_only.ends_with("CPUQuota=200%\nCPUWeight=80\n"),
"{cpu_only}"
);
assert!(!cpu_only.contains("IOWeight"), "{cpu_only}");
let io_only = limits_dropin_body("/rt/x", "4G", "200%", None, Some(80));
let io_only = limits_dropin_body("/rt/x", CHECK, "4G", "200%", None, Some(80));
assert!(
io_only.ends_with("CPUQuota=200%\nIOWeight=80\n"),
"{io_only}"
);
assert!(!io_only.contains("CPUWeight"), "{io_only}");
}
/// The pre-start check is the first `[Service]` line, whatever else the
/// caller sets, and a drop-in without it is what an old hive-priv wrote.
#[test]
fn the_rootfs_check_leads_the_service_section_whatever_the_caller_passes() {
for (cw, iw) in [(None, None), (Some(80), Some(80))] {
let body = limits_dropin_body("/rt/x", CHECK, "4G", "200%", cw, iw);
let service = body
.split("[Service]\n")
.nth(1)
.expect("the body always has a [Service] section");
assert_eq!(
service.lines().next(),
Some("ExecStartPre=/store/hive-priv/bin/hive-priv check-rootfs h-iris")
);
assert_eq!(body.matches("ExecStartPre=").count(), 1, "{body}");
}
}
/// The command names an absolute path to this very binary, the
/// sub-command that binary implements and the container.
#[test]
fn the_check_command_names_this_binary_the_subcommand_and_the_container() {
let exe = std::env::current_exe().unwrap();
let cmd = rootfs_check_command(&hive_priv_exe().unwrap(), "h-iris");
assert_eq!(cmd, format!("{} check-rootfs h-iris", exe.display()));
assert!(cmd.starts_with('/'), "{cmd}");
}
/// The mode lands exactly as asked, whatever the umask would have made
/// of it (0666 under the usual 022 would otherwise come out 0644), and
/// the requested owner is on the file.

440
hive-priv/src/prestart.rs
View file

@ -3,30 +3,44 @@
//! nixpkgs' `container@.service` start script runs as host root against the
//! container's own root, which is persistent and writable by the container.
//! `mkdir -p`, `chmod`, `touch` and `cp --remove-destination` there follow a
//! link the container planted at any of the names the script touches. A
//! drop-in of the `container@` template
//! (`nix/host-modules/container-rootfs-check.nix`) runs
//! `hive-priv check-rootfs %i` as an `ExecStartPre=` ahead of that script for
//! every instance; a non-zero exit fails the start. The container is stopped
//! link the container planted at any of the names the script touches. An agent
//! container's drop-in ([`crate::container_config`]) puts
//! `hive-priv check-rootfs <container>` ahead of that script as an
//! `ExecStartPre=`; a non-zero exit fails the start. The container is stopped
//! while this runs, so nothing changes under the walk.
//!
//! Hyperhive's own declarative containers (the forge, the matrix server, the
//! swarm services) run it as `check-rootfs --warn-only <container>` from their
//! own unit (`nix/host-modules/lib/container-rootfs-check.nix`): each violation
//! is logged at warning level and the start goes on.
//!
//! Every component is looked at with `symlink_metadata`: a link is reported,
//! never followed. Nothing is deleted or repaired, so whatever is there stays
//! for the operator.
//!
//! The drop-in names the running hive-priv binary, which nothing else roots
//! once the system has moved on to a newer build. [`pin_binary`] keeps a GC
//! root on it for as long as the drop-in exists, and [`unpin_binary`] drops
//! the root with the drop-in.
//!
//! The entry point is [`run`]; [`check`] holds the rules.
use std::fs::{FileType, Metadata};
use std::os::unix::fs::{DirBuilderExt as _, symlink};
use std::path::{Path, PathBuf};
use anyhow::{Context as _, Result, bail};
use crate::container_config::container_rootfs;
use crate::shared::validate_name_chars;
use crate::container_config::{container_rootfs, validate_container_system_name};
use crate::shared::{partial_name, validate_name_chars};
/// The sub-command the template drop-in's `ExecStartPre=` names.
/// The sub-command the drop-in's `ExecStartPre=` names.
pub(crate) const SUBCOMMAND: &str = "check-rootfs";
/// The flag that turns a violation into a warning: the check runs in full and
/// exits 0.
const WARN_ONLY: &str = "--warn-only";
/// The per-container nix profile directories on the host.
const PROFILES_ROOT: &str = "/nix/var/nix/profiles/per-container";
@ -59,47 +73,157 @@ const REGULAR_ONLY: [&str; 2] = ["etc/machine-id", "etc/resolv.conf"];
/// `/etc/static` once it has booted.
const OS_RELEASE: &str = "etc/os-release";
/// Where hive-priv roots the build named by each container's `ExecStartPre=`:
/// `<container>` is a symlink to that build's store path. The directory is
/// under the host's `/nix/var/nix/gcroots`, not `per-container`, which is the
/// only part of it a container is given (`--bind=.../gcroots/per-container/<c>`
/// in the nixpkgs start script), so a container can't reach it.
const GCROOTS: &str = "/nix/var/nix/gcroots/hyperhive-check-rootfs";
/// Root the store path of `exe` for `container`, replacing the root a previous
/// write left, so the build a drop-in names outlives a garbage collection for
/// as long as the drop-in does. One link per container, so they don't pile up.
pub(crate) fn pin_binary(container: &str, exe: &Path) -> Result<()> {
pin_binary_in(Path::new(GCROOTS), Path::new(STORE), container, exe)
}
/// Drop the root [`pin_binary`] left for `container`, once its drop-in is gone.
/// A name that holds anything but a link is left alone and reported.
pub(crate) fn unpin_binary(container: &str) -> Result<()> {
unpin_binary_in(Path::new(GCROOTS), container)
}
/// [`pin_binary`] with the roots directory and the store as parameters.
///
/// A binary outside `store` can't be collected, so it gets no root. The link
/// is made under a temporary name and renamed over `<container>`: `rename`
/// replaces whatever link is there without following it.
fn pin_binary_in(roots: &Path, store: &Path, container: &str, exe: &Path) -> Result<()> {
let Some(entry) = exe
.strip_prefix(store)
.ok()
.and_then(|rest| rest.components().next())
else {
return Ok(());
};
let target = store.join(entry);
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o755)
.create(roots)
.with_context(|| format!("create {}", roots.display()))?;
if !lstat(roots)?.is_some_and(|m| m.file_type().is_dir()) {
bail!("{} is not a real directory", roots.display());
}
let tmp = roots.join(partial_name(container));
symlink(&target, &tmp).with_context(|| format!("create {}", tmp.display()))?;
let root = roots.join(container);
std::fs::rename(&tmp, &root).map_err(|e| {
// Best effort: a leftover temp is a root too, but a harmless one.
let _ = std::fs::remove_file(&tmp);
anyhow::Error::new(e).context(format!("replace {}", root.display()))
})
}
/// [`unpin_binary`] with the roots directory as a parameter.
fn unpin_binary_in(roots: &Path, container: &str) -> Result<()> {
let root = roots.join(container);
match lstat(&root)? {
None => Ok(()),
Some(meta) if meta.file_type().is_symlink() => {
std::fs::remove_file(&root).with_context(|| format!("remove {}", root.display()))
}
Some(meta) => bail!(
"{} is {}, not a link: left alone",
root.display(),
kind(meta.file_type())
),
}
}
/// Handle `hive-priv <subcommand> ...` when the first argument names one.
/// `None` means it is not a sub-command and the daemon starts.
///
/// Prints one line to stderr and returns exit code 1 on a violation, so the
/// reason lands in the journal next to the failed unit.
/// `check-rootfs <container>` is strict: it prints one line to stderr and
/// returns exit code 1 on a violation, so the reason lands in the journal next
/// to the failed unit. `check-rootfs --warn-only <container>` prints every
/// violation as a warning and returns 0.
pub(crate) fn run(args: &[String]) -> Option<i32> {
if args.first().map(String::as_str) != Some(SUBCOMMAND) {
return None;
}
let result = match args {
[_, container] => check_container(container),
_ => Err(anyhow::anyhow!("usage: hive-priv {SUBCOMMAND} <container>")),
};
Some(match result {
Ok(()) => 0,
Err(e) => {
eprintln!("hive-priv {SUBCOMMAND}: refusing to start: {e:#}");
1
let (container, warn_only) = match args {
[_, container] => (container, false),
[_, flag, container] if flag == WARN_ONLY => (container, true),
_ => {
eprintln!("usage: hive-priv {SUBCOMMAND} [{WARN_ONLY}] <container>");
return Some(1);
}
})
};
let found = found_in_container(container, warn_only);
let (code, lines) = report(container, &found, warn_only);
for line in lines {
eprintln!("{line}");
}
Some(code)
}
/// [`check`] on the host paths of `container`.
fn check_container(container: &str) -> Result<()> {
let (root, system) = container_paths(container)?;
check(&root, &system, Path::new(STORE))
/// What a start would be refused for, on the host paths of `container`.
///
/// Strict mode stops at the first violation, as the start would. Warn-only
/// lists them all, and counts a name it can't use as one more: it never fails
/// the start over its own input.
fn found_in_container(container: &str, warn_only: bool) -> Vec<String> {
let paths = container_paths(container, warn_only);
match (paths, warn_only) {
(Ok((root, system)), true) => violations(&root, &system, Path::new(STORE)),
(Ok((root, system)), false) => check(&root, &system, Path::new(STORE))
.err()
.map(|e| format!("{e:#}"))
.into_iter()
.collect(),
(Err(e), _) => vec![format!("{e:#}")],
}
}
/// The container's root and its profile's `system` link on the host.
///
/// The unit instance supplies the name and it selects both paths, so anything
/// but a plain container name is refused: lowercase ascii, digits and
/// hyphens. That holds no `/` and is never `.` or `..`.
fn container_paths(container: &str) -> Result<(PathBuf, PathBuf)> {
validate_name_chars(container)?;
/// An agent container is named `h-<agent>`. A warn-only check also takes the
/// platform containers' names, which are lowercase ascii, digits and hyphens.
/// Either way the name holds no `/` and is never `.` or `..`.
fn container_paths(container: &str, warn_only: bool) -> Result<(PathBuf, PathBuf)> {
if warn_only {
validate_name_chars(container)?;
} else {
validate_container_system_name(container)?;
}
Ok((
container_rootfs(container),
Path::new(PROFILES_ROOT).join(container).join("system"),
))
}
/// The exit code and the stderr lines for the violations `found`.
///
/// Warn-only lines carry the `<4>` prefix, which journald reads as the
/// warning priority, so a log query can select them by severity.
fn report(container: &str, found: &[String], warn_only: bool) -> (i32, Vec<String>) {
if warn_only {
let lines = found
.iter()
.map(|v| format!("<4>rootfs check (warn-only): {container}: {v}"))
.collect();
return (0, lines);
}
match found.first() {
Some(v) => (
1,
vec![format!("hive-priv {SUBCOMMAND}: refusing to start: {v}")],
),
None => (0, Vec::new()),
}
}
/// Check the paths the start script touches as root, returning an error that
/// names the first path to hold something it must not.
///
@ -109,14 +233,36 @@ fn container_paths(container: &str) -> Result<(PathBuf, PathBuf)> {
/// A path that does not exist passes: a first start has nothing yet, and the
/// script creates it.
fn check(root: &Path, system: &Path, store: &Path) -> Result<()> {
match violations(root, system, store).into_iter().next() {
None => Ok(()),
Some(first) => bail!("{first}"),
}
}
/// Every path under the rules of [`check`] that holds something it must not,
/// in the order [`check`] looks at them, each message once.
fn violations(root: &Path, system: &Path, store: &Path) -> Vec<String> {
let mut found = Vec::new();
for dir in DIRS {
check_dir(root, dir)?;
note(&mut found, check_dir(root, dir));
}
for file in REGULAR_ONLY {
check_file(root, file, false)?;
note(&mut found, check_file(root, file, false));
}
note(&mut found, check_file(root, OS_RELEASE, true));
note(&mut found, check_system(system, store));
found
}
/// Add the message of an error to `found`, unless it is already there: a link
/// at `usr` is the finding for `usr/bin` too.
fn note(found: &mut Vec<String>, result: Result<()>) {
if let Err(e) = result {
let message = format!("{e:#}");
if !found.contains(&message) {
found.push(message);
}
}
check_file(root, OS_RELEASE, true)?;
check_system(system, store)
}
/// `symlink_metadata`, with `None` for a path that does not exist. The one
@ -221,8 +367,8 @@ mod tests {
use std::os::unix::fs::symlink;
use std::path::PathBuf;
use super::{check, container_paths, run};
use crate::shared::test_support::scratch;
use super::{check, container_paths, pin_binary_in, report, run, unpin_binary_in, violations};
use crate::shared::test_support::{entries, scratch};
/// A scratch container: root, the profile link's path and a store with one
/// toplevel in it. Nothing is linked yet.
@ -450,11 +596,133 @@ mod tests {
assert!(target.is_dir());
}
/// A scratch store holding one build with a binary in it, a roots dir and
/// the binary's path.
fn pin_fixture() -> (PathBuf, PathBuf, PathBuf, PathBuf) {
let base = scratch();
let store = base.join("store");
let build = store.join("abc-hive-priv");
std::fs::create_dir_all(build.join("bin")).unwrap();
let exe = build.join("bin/hive-priv");
std::fs::write(&exe, "").unwrap();
(base.join("roots"), store, build, exe)
}
#[test]
fn a_plain_container_name_selects_its_root_and_profile() {
// Agent and non-agent names alike.
for name in ["h-iris", "hive-forge", "swarm-review-classifier"] {
let (root, system) = container_paths(name).unwrap();
fn the_root_points_at_the_store_path_the_drop_in_names() {
let (roots, store, build, exe) = pin_fixture();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
let root = roots.join("h-iris");
assert!(std::fs::symlink_metadata(&root).unwrap().is_symlink());
// The build, not the binary inside it.
assert_eq!(std::fs::read_link(&root).unwrap(), build);
assert_eq!(std::fs::read_dir(&roots).unwrap().count(), 1);
}
#[test]
fn a_rewrite_replaces_the_root_and_leaves_no_temp() {
let (roots, store, build, exe) = pin_fixture();
let newer = store.join("def-hive-priv");
std::fs::create_dir_all(newer.join("bin")).unwrap();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
pin_binary_in(&roots, &store, "h-iris", &newer.join("bin/hive-priv")).unwrap();
pin_binary_in(&roots, &store, "h-bob", &exe).unwrap();
assert_eq!(std::fs::read_link(roots.join("h-iris")).unwrap(), newer);
assert_eq!(std::fs::read_link(roots.join("h-bob")).unwrap(), build);
assert_eq!(entries(&roots), ["h-bob", "h-iris"]);
}
#[test]
fn a_link_planted_at_the_root_name_is_replaced_not_followed() {
let (roots, store, build, exe) = pin_fixture();
let victim = victim();
std::fs::write(victim.join("keep"), "data").unwrap();
std::fs::create_dir_all(&roots).unwrap();
symlink(&victim, roots.join("h-iris")).unwrap();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
assert_eq!(std::fs::read_link(roots.join("h-iris")).unwrap(), build);
assert_eq!(
std::fs::read_to_string(victim.join("keep")).unwrap(),
"data"
);
assert_eq!(std::fs::read_dir(&victim).unwrap().count(), 1);
}
#[test]
fn a_roots_dir_that_is_a_link_is_refused() {
let (roots, store, _, exe) = pin_fixture();
let victim = victim();
symlink(&victim, &roots).unwrap();
let msg = format!(
"{:#}",
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap_err()
);
assert!(msg.contains("is not a real directory"), "{msg}");
assert_eq!(std::fs::read_dir(&victim).unwrap().count(), 0);
}
#[test]
fn a_binary_outside_the_store_gets_no_root() {
let (roots, store, _, _) = pin_fixture();
let elsewhere = scratch().join("hive-priv");
pin_binary_in(&roots, &store, "h-iris", &elsewhere).unwrap();
assert!(!roots.exists());
}
#[test]
fn unpinning_removes_the_container_root_and_nothing_else() {
let (roots, store, build, exe) = pin_fixture();
pin_binary_in(&roots, &store, "h-iris", &exe).unwrap();
pin_binary_in(&roots, &store, "h-bob", &exe).unwrap();
unpin_binary_in(&roots, "h-iris").unwrap();
assert_eq!(entries(&roots), ["h-bob"]);
// Already gone: fine. The build it named is untouched.
unpin_binary_in(&roots, "h-iris").unwrap();
assert!(build.join("bin/hive-priv").is_file());
}
#[test]
fn unpinning_leaves_a_name_that_is_not_a_link_alone() {
let (roots, _, _, _) = pin_fixture();
std::fs::create_dir_all(roots.join("h-iris")).unwrap();
std::fs::write(roots.join("h-iris/keep"), "").unwrap();
let msg = format!("{:#}", unpin_binary_in(&roots, "h-iris").unwrap_err());
assert!(msg.contains("left alone"), "{msg}");
assert!(roots.join("h-iris/keep").exists());
}
#[test]
fn only_the_check_rootfs_sub_command_is_taken() {
let args = |a: &[&str]| a.iter().map(|s| (*s).to_owned()).collect::<Vec<_>>();
assert_eq!(run(&[]), None);
assert_eq!(run(&args(&["serve"])), None);
// Wrong arity, an unknown flag and a name that is not an agent
// container are refused, whatever is on disk.
assert_eq!(run(&args(&["check-rootfs"])), Some(1));
assert_eq!(run(&args(&["check-rootfs", "not-hive"])), Some(1));
assert_eq!(run(&args(&["check-rootfs", "--strict", "h-iris"])), Some(1));
assert_eq!(run(&args(&["check-rootfs", "--warn-only"])), Some(1));
}
#[test]
fn a_warn_only_run_exits_zero_even_for_a_name_it_cannot_use() {
let args = |a: &[&str]| a.iter().map(|s| (*s).to_owned()).collect::<Vec<_>>();
assert_eq!(
run(&args(&["check-rootfs", "--warn-only", "no-such-container"])),
Some(0)
);
assert_eq!(
run(&args(&["check-rootfs", "--warn-only", "../etc"])),
Some(0)
);
}
#[test]
fn strict_takes_agent_names_and_warn_only_takes_plain_ones() {
assert!(container_paths("h-iris", false).is_ok());
assert!(container_paths("hive-forge", false).is_err());
for name in ["hive-forge", "swarm-review-classifier", "h-iris"] {
let (root, system) = container_paths(name, true).unwrap();
assert_eq!(
root,
PathBuf::from(format!("/var/lib/nixos-containers/{name}"))
@ -464,10 +732,6 @@ mod tests {
PathBuf::from(format!("/nix/var/nix/profiles/per-container/{name}/system"))
);
}
}
#[test]
fn a_name_that_is_not_a_plain_container_name_is_refused() {
for name in [
"",
".",
@ -475,28 +739,84 @@ mod tests {
"../h-iris",
"a/b",
"/etc",
"h-iris/",
"H-iris",
"a.b",
"a b",
] {
assert!(container_paths(name).is_err(), "{name:?} was accepted");
}
}
#[test]
fn only_the_check_rootfs_sub_command_is_taken() {
assert_eq!(run(&[]), None);
assert_eq!(run(&["serve".to_string()]), None);
// Wrong arity and a name that selects a path outside the containers
// directory are refused, whatever is on disk.
assert_eq!(run(&["check-rootfs".to_string()]), Some(1));
for name in ["", "..", "a/b"] {
assert_eq!(
run(&["check-rootfs".to_string(), name.to_string()]),
Some(1),
"{name:?}"
assert!(
container_paths(name, true).is_err(),
"{name:?} was accepted"
);
assert!(
container_paths(name, false).is_err(),
"{name:?} was accepted"
);
}
}
/// A clean root with a link at `root` and another at `etc/resolv.conf`.
fn two_violations() -> Fixture {
let f = Fixture::clean();
std::fs::remove_dir_all(f.root.join("root")).unwrap();
symlink(victim(), f.root.join("root")).unwrap();
std::fs::remove_file(f.root.join("etc/resolv.conf")).unwrap();
symlink("/etc/hosts", f.root.join("etc/resolv.conf")).unwrap();
f
}
fn found(f: &Fixture) -> Vec<String> {
violations(&f.root, &f.system, &f.store)
}
#[test]
fn warn_only_logs_every_violation_at_warning_level_and_exits_zero() {
let f = two_violations();
let found = found(&f);
assert_eq!(found.len(), 2, "{found:?}");
let (code, lines) = report("hive-forge", &found, true);
assert_eq!(code, 0);
assert_eq!(lines.len(), 2, "{lines:?}");
for line in &lines {
assert!(
line.starts_with("<4>rootfs check (warn-only): hive-forge: "),
"{line}"
);
}
assert!(lines[0].contains("root is a symlink"), "{lines:?}");
assert!(lines[1].contains("resolv.conf is a symlink"), "{lines:?}");
}
#[test]
fn strict_refuses_with_the_first_violation_only() {
let f = two_violations();
let found = found(&f);
let (code, lines) = report("h-iris", &found, false);
assert_eq!(code, 1);
assert_eq!(lines.len(), 1, "{lines:?}");
assert!(
lines[0].starts_with("hive-priv check-rootfs: refusing to start: "),
"{lines:?}"
);
assert!(lines[0].contains("root is a symlink"), "{lines:?}");
// The rules are the same: the strict refusal is `check`'s first error.
assert_eq!(f.refused(), found[0]);
}
#[test]
fn a_clean_root_prints_nothing_in_either_mode() {
let f = Fixture::clean();
assert!(found(&f).is_empty());
assert_eq!(report("hive-forge", &[], true), (0, Vec::new()));
assert_eq!(report("h-iris", &[], false), (0, Vec::new()));
}
#[test]
fn one_link_is_reported_once_though_two_paths_run_through_it() {
let f = Fixture::clean();
std::fs::remove_dir_all(f.root.join("usr")).unwrap();
symlink(victim(), f.root.join("usr")).unwrap();
let found = found(&f);
assert_eq!(found.len(), 1, "{found:?}");
assert!(found[0].contains("usr is a symlink"), "{found:?}");
}
}

2
hive-priv/src/shared.rs
View file

@ -75,7 +75,7 @@ pub(crate) fn ensure_plain_filename(who: &str, filename: &str) -> Result<()> {
/// The leading dot and the `.partial` suffix are load-bearing: a reader
/// globbing on a published name's prefix, or systemd reading `*.conf` out of
/// `tmpfiles.d` and drop-in dirs, cannot pick up a half-written temp.
fn partial_name(filename: &str) -> String {
pub(crate) fn partial_name(filename: &str) -> String {
use std::sync::atomic::{AtomicU64, Ordering};
static SEQ: AtomicU64 = AtomicU64::new(0);
let n = SEQ.fetch_add(1, Ordering::Relaxed);

44
nix/host-modules/container-rootfs-check.nix
View file

@ -1,44 +0,0 @@
# The pre-start check of a container's root, for every `container@` instance.
#
# nixpkgs' start script runs as host root against the container's own root,
# which is persistent and writable by the container, and its `mkdir`, `chmod`,
# `touch` and `cp` follow links the container planted there. A drop-in on the
# `container@` template runs `hive-priv check-rootfs %i` as an `ExecStartPre=`
# before the script (`hive-priv/src/prestart.rs` has the rules). It covers the
# agents' imperative instances and the declarative ones (hive-forge,
# hive-matrix, the swarm services), each of which has a unit file of its own:
# systemd reads the template's drop-in directory for every instance.
#
# Order: systemd parses a drop-in after the unit file, so this line follows the
# unit file's own `ExecStartPre=` (nixpkgs' `preStart`, which touches no path
# under the root), and `ExecStart=` runs only after every `ExecStartPre=`
# succeeded.
#
# The drop-in is a unit-file package rather than `systemd.services."container@"`
# because that attribute renders the template's own unit file, which a
# declarative instance's unit file replaces.
{
config,
lib,
pkgs,
...
}:
let
dropin = pkgs.writeTextFile {
name = "hyperhive-container-rootfs-check";
destination = "/lib/systemd/system/container@.service.d/10-hyperhive-rootfs-check.conf";
text = ''
[Service]
ExecStartPre=${config.services.hyperhive.c0re.package}/bin/hive-priv check-rootfs %i
'';
};
in
{
config = lib.mkIf config.boot.enableContainers {
systemd.packages = [ dropin ];
# The per-agent drop-ins this check replaced rooted a hive-priv build per
# agent here. Boot is the one time no such drop-in is loaded.
systemd.tmpfiles.rules = [ "R! /nix/var/nix/gcroots/hyperhive-check-rootfs - - - - -" ];
};
}

1
nix/host-modules/default.nix
View file

@ -22,7 +22,6 @@
./hive-matrix-service.nix
./hive-matrix.nix
./hive-network.nix
./container-rootfs-check.nix
./hive-priv.nix
./hive-priv-cache.nix
./hive-squid.nix

3
nix/host-modules/hive-forge/default.nix
View file

@ -78,6 +78,8 @@ let
deployCfg.bao.forgePublisherClientCertFile != null
&& deployCfg.bao.forgePublisherClientKeyFile != null;
rootfsCheck = import ../lib/container-rootfs-check.nix { inherit config; };
caTrust = import ../lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
# The container's journal forwarder pushes to its own receiver on the swarm
@ -471,6 +473,7 @@ in
# ../swarm-builder.nix's does.
systemd.services."container@hive-forge" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly "hive-forge")
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];

3
nix/host-modules/hive-matrix.nix
View file

@ -13,6 +13,8 @@ let
# the client is pointed at the server name instead.
hyperhiveDomain = config.services.hyperhive.domain;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# Same runtime→build-time bridge hive-forge and swarm-ci-runner cross:
# binds the hive trust bundle (which folds in the store's root) into the
# container and orders the container after `hive-tls-ca.service`. The
@ -1096,6 +1098,7 @@ in
systemd.services."container@hive-matrix" = lib.mkMerge [
{ after = [ "dnsmasq.service" ]; }
caTrust.containerOrdering
(rootfsCheck.warnOnly "hive-matrix")
];
};
}

32
nix/host-modules/lib/container-rootfs-check.nix Normal file
View file

@ -0,0 +1,32 @@
# The pre-start check of a container's root, for one of hyperhive's own
# declarative containers.
#
# nixpkgs' start script runs as host root against the container's own root,
# which is persistent and writable by the container, and its `mkdir`, `chmod`,
# `touch` and `cp` follow links the container planted there. `hive-priv
# check-rootfs` looks at the paths the script touches (rules in
# `hive-priv/src/prestart.rs`). With `--warn-only` it logs each violation at
# warning level and lets the start go on; the agent containers run it strict
# from their per-agent drop-in (hive-priv's `WriteResourceLimits`).
#
# Pure function, not a NixOS module. A container's module calls it with its
# own name and folds the result into its unit:
#
# rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# # systemd.services."container@${machine}" = lib.mkMerge [ (rootfsCheck.warnOnly machine) … ];
#
# The name list is the callers', never `config.containers`: that also holds
# containers the operator runs that hyperhive doesn't own, and those get
# nothing. nixpkgs renders each declarative container as its own
# `container@<name>` unit (`nixos-containers.nix`, the `mapAttrsToList` that
# builds `container@${name}`), so a setting on that unit reaches that
# container only.
{ config }:
{
# Attributes for `systemd.services."container@<name>"`.
warnOnly = name: {
serviceConfig.ExecStartPre = [
"${config.services.hyperhive.c0re.package}/bin/hive-priv check-rootfs --warn-only ${name}"
];
};
}

3
nix/host-modules/swarm-attic.nix
View file

@ -114,6 +114,8 @@ let
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# The forwarder reaches the swarm collector and authelia by name over TLS; in
# self-signed mode their leaves chain to the hive CA.
caTrust = import ./lib/hive-ca-trust.nix {
@ -226,6 +228,7 @@ in
# CA ordering keeps the trust bundle's bind source there first.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly machine)
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];

3
nix/host-modules/swarm-authelia.nix
View file

@ -183,6 +183,8 @@ let
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeSeededSecret = import ./lib/store-seeded-secret.nix { };
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
caTrust = import ./lib/hive-ca-trust.nix {
inherit lib gatewayCfg;
tlsCfg = deployCfg.hive-controller.tls;
@ -970,6 +972,7 @@ in
# not be up.
systemd.services."container@${cfg.machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly cfg.machine)
(lib.mkIf publishes {
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];

3
nix/host-modules/swarm-bao.nix
View file

@ -1689,6 +1689,8 @@ let
forwarderAuthName = "oauth2client/${swarmOtelCfg.storeProducerName}";
autheliaCfg = hyperhiveCfg.swarm.authelia;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# Outbound TLS for that collector: it dials the swarm collector's gateway
# name and authelia's token endpoint, both leaves under the store's runtime
# pki root, which cannot be baked into a derivation. Same helper, same two
@ -4091,6 +4093,7 @@ in
# cannot start is a store that is down.
systemd.services."container@${cfg.machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly cfg.machine)
{ after = [ "systemd-tmpfiles-resetup.service" ]; }
];

3
nix/host-modules/swarm-builder.nix
View file

@ -52,6 +52,8 @@ let
# containers' forwarders all the same.
forwarderTelemetryPort = 8892;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# The daemon reaches authelia, the store and the queue by name over TLS, and
# `git` reaches the forge; in self-signed mode their leaves chain to the hive
# CA.
@ -165,6 +167,7 @@ in
# the unit does not exist.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly machine)
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];

3
nix/host-modules/swarm-ci-runner.nix
View file

@ -43,6 +43,8 @@ let
# below compares the two, so both come from here.
forgeUrl = "https://${forgeCfg.domain}";
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# The forge's gateway serves a leaf from the store's `pki` mount in
# self-signed mode.
# The runner daemon is Go, whose `SSL_CERT_FILE` replaces the system store,
@ -329,6 +331,7 @@ in
# minting unit in and waits for it, as ./swarm-builder.nix's does.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly machine)
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];

3
nix/host-modules/swarm-claude-login.nix
View file

@ -54,6 +54,8 @@ let
# containers' forwarders all the same.
forwarderTelemetryPort = 8893;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# The daemon reaches authelia, the store and the queue by name over TLS, and
# in self-signed mode their leaves chain to the hive CA.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
@ -146,6 +148,7 @@ in
# missing, so where this host mints the leaf the container waits for it.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly machine)
{ after = [ "swarm-bao-pki.service" ]; }
];

3
nix/host-modules/swarm-grafana.nix
View file

@ -24,6 +24,8 @@ let
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
caTrust = import ./lib/hive-ca-trust.nix {
inherit lib gatewayCfg;
tlsCfg = deployCfg.hive-controller.tls;
@ -686,6 +688,7 @@ in
# ./swarm-builder.nix's does.
systemd.services."container@${cfg.machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly cfg.machine)
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];

7
nix/host-modules/swarm-nats.nix
View file

@ -33,6 +33,8 @@ let
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
gatewayCfg = config.services.hyperhive.gateway;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
# The responder introspects authelia over https BY NAME. Its HTTP client is
# reqwest/rustls, and `rustls-platform-verifier` resolves roots through
@ -961,7 +963,10 @@ in
# Order the container after the host CA generator, so the bind source
# exists before nspawn sets the mount up. Without it a late CA fails the
# container start outright rather than degrading.
systemd.services."container@${machine}" = caTrust.containerOrdering;
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly machine)
];
# Deliver the responder's store identity into the container before it
# starts. A copy rather than a `bindMounts` entry: nixos-container refuses

3
nix/host-modules/swarm-notify-router.nix
View file

@ -44,6 +44,8 @@ let
# containers' forwarders all the same.
forwarderTelemetryPort = 8902;
rootfsCheck = import ./lib/container-rootfs-check.nix { inherit config; };
# The daemon reaches authelia, the store and the queue by name over TLS, and
# in self-signed mode their leaves chain to the hive CA.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
@ -134,6 +136,7 @@ in
# missing, so where this host mints the leaf the container waits for it.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
(rootfsCheck.warnOnly machine)
{ after = [ "swarm-bao-pki.service" ]; }
];

Some files were not shown because too many files have changed in this diff Show more