From 5c658e9bcfe9395dcfb4e60eb0813de9715e3b75 Mon Sep 17 00:00:00 2001 From: atlas Date: Mon, 7 Sep 2026 14:07:07 +0200 Subject: [PATCH] docs/ci: stagger the GC we recommend, and say that temp roots stop at a store boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nightly GC this page tells operators to configure is what #4019 is about: a collection that lands on the same wall clock as CI. Two gaps in the guidance rather than one. `dates = "daily"` with no randomization puts the collection at the same time every night, so whatever else runs nightly meets it every night. `randomizedDelaySec` spreads it; the value is nixpkgs' own example for that option, and the option feeds `RandomizedDelaySec` on the timer. The overlap itself does not fail a build on a single host — the runner builds through the host nix-daemon (`hive-ci.nix` disables its own and bind-mounts the socket), so an in-flight build's temp roots are held by the daemon a host GC talks to. What it does cost is wall time: the first build after a collection rebuilds what the collection deleted. That is the every-run version of the same cause and it is what the comment now says. The Remote builders paragraph covered disk space only. Temp-root protection is per-daemon, so it does not span two stores: a collection on the builder is not aware of a coordinator build waiting on a path there. Stated as the reason to stagger the two hosts' windows rather than give them the same `dates`. Deliberately not claiming this fixes the failure in #4019. That one observed a missing build input with a remote builder in play, and distinguishing "the collector deleted it" from "it was never in that store" needs a correlated reading on both hosts during a live window, which has not been taken. This is the half that is knowable from the tree: our own recommendation is what schedules the collision. --- docs/scheduler/ci.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/scheduler/ci.md b/docs/scheduler/ci.md index 2640a155..bd3689bb 100644 --- a/docs/scheduler/ci.md +++ b/docs/scheduler/ci.md @@ -211,9 +211,17 @@ following to your host config: { # Daily GC: delete store paths not referenced by a live root and older # than a day. Keeps the store bounded between builds. + # + # `randomizedDelaySec` because a fixed daily time lands on the same wall + # clock as whatever else runs nightly. A collection that overlaps a CI + # job does not fail it — the runner builds through the host nix-daemon, + # so an in-flight build's temp roots are protected — but the first build + # after a collection rebuilds what the collection deleted, so a run that + # lands just behind one pays for it in wall time. nix.gc = { automatic = true; dates = "daily"; + randomizedDelaySec = "45min"; options = "--delete-older-than 1d"; }; @@ -231,6 +239,13 @@ following to your host config: store, so the same GC config should be applied wherever the builder runs — GC on the coordinator host won't reclaim space on the builder. +⚠️ Two stores also means the temp-root protection above **doesn't span +them**. Each daemon holds roots only for the builds it runs, so a +collection on the _builder_ knows nothing about a coordinator build +waiting on a path there. Stagger the two hosts' GC windows rather than +giving them the same `dates`, and prefer `min-free`/`max-free` on the +builder — disk-pressure collection at least runs while nothing is mid-copy. + ## References - `nix/host-modules/hive-ci.nix`: runner configuration, auto-registration script, container setup.