ci: coverage on manual dispatch, no threshold

Adds `cargo llvm-cov` to the devshell and a coverage workflow that runs
only when someone asks for it.

Manual dispatch rather than nightly or per-PR, per the discussion on the
issue: a coverage run needs its own instrumented build and cannot reuse
the normal test artifacts, so it roughly doubles a test job. "Do we have
glaring holes" is a question someone asks occasionally, not a gate every
PR pays for, and not a number worth spending farm time on every night
whether or not anyone reads it. Manual dispatch costs nothing until the
answer is wanted.

Its own workflow file rather than a job in ci.yml: that file already has
a `workflow_dispatch` trigger so `hive-forge ci-rerun` can retrigger
without an empty commit, and a trigger there fires EVERY job in the
file -- a coverage job added there would run on every pull request.

No threshold and no --fail-under-lines. A coverage gate mostly teaches
people to write assertion-free tests that execute lines; the report is
the deliverable and the number is for a human to read.

The devshell needs LLVM_COV / LLVM_PROFDATA set explicitly: cargo
llvm-cov expects rustup's `llvm-tools-preview` beside the toolchain and
nixpkgs has no such component, so without them it aborts with "failed to
find llvm-tools-preview" -- which reads like a missing install rather
than a path the shell has to name.

Verified by running it: `cargo llvm-cov --package hive-types` produces a
real report (3 tests, 85.44% regions). Both the variable names and the
package were wrong on the first attempt and only running it said so --
the names take no `_PATH` suffix, and the binaries are in
`llvmPackages.llvm`, not `llvmPackages.bintools`, which is the linker
wrapper and ships neither.
This commit is contained in:
atlas 2026-08-27 11:39:33 +02:00 committed by mara
commit 1e3347fb46
2 changed files with 61 additions and 0 deletions

View file

@ -0,0 +1,40 @@
name: coverage
# Manual dispatch ONLY — deliberately not on `pull_request`, and
# deliberately not nightly.
#
# A coverage run needs its own instrumented build: it cannot reuse the
# normal test artifacts, so it roughly doubles a test job. Coverage here
# answers "do we have glaring holes", which is a question someone asks
# occasionally — not a gate every PR pays for, and not a number worth
# spending farm time on every night whether or not anyone reads it.
# Manual dispatch costs nothing until someone wants the answer.
#
# Its own file rather than a job in ci.yml: that workflow already has a
# `workflow_dispatch` trigger (so `hive-forge ci-rerun` can retrigger
# without an empty commit), and a trigger there fires EVERY job in the
# file. A coverage job added to ci.yml would run on every pull request.
on:
workflow_dispatch:
jobs:
coverage:
name: cargo llvm-cov
runs-on: [hive-ci]
# Above the 30 min `nix flake check` allows, because instrumented
# builds are slower than the ordinary ones and this starts from a
# cache that the normal jobs never populate.
timeout-minutes: 60
steps:
- uses: actions/checkout@v3
- name: coverage
# No threshold and no `--fail-under-lines`: a coverage gate
# mostly teaches people to write assertion-free tests that
# execute lines. The report is the deliverable; the number is
# for a human to look at.
#
# `--workspace` because the interesting holes are in the crates
# nobody runs directly, and the per-crate default would hide
# exactly those behind a green summary for the one crate someone
# happened to name.
run: nix develop -c cargo llvm-cov --workspace --summary-only

View file

@ -9,10 +9,31 @@
rust.nativeBuildInputs
++ (with pkgs; [
cargo
cargo-llvm-cov
clippy
rust-analyzer
rustc
rustfmt
]);
# `cargo llvm-cov` shells out to `llvm-profdata` and `llvm-cov`,
# which it expects to find as rustup's `llvm-tools-preview` beside
# the rustc toolchain. Nixpkgs has no such component, so without
# these it aborts with "failed to find llvm-tools-preview" — a
# message that reads like a missing install rather than a path the
# shell simply has to name.
#
# ⚠️ Both the variable names and the package were wrong on the first
# attempt and only the tool itself said so: it reads `LLVM_COV` /
# `LLVM_PROFDATA` (no `_PATH` suffix), and the binaries live in
# `llvmPackages.llvm` — `llvmPackages.bintools` is the linker
# wrapper and ships neither.
#
# Coverage is deliberately NOT part of `nix flake check`: an
# instrumented build cannot reuse the normal test artifacts, so it
# roughly doubles a test job. It runs on manual dispatch — see
# .forgejo/workflows/coverage.yml for why that rather than nightly.
LLVM_PROFDATA = "${pkgs.llvmPackages.llvm}/bin/llvm-profdata";
LLVM_COV = "${pkgs.llvmPackages.llvm}/bin/llvm-cov";
};
}