From 1e3347fb4608e486a3bfc0acf00e2fbd5598b6e4 Mon Sep 17 00:00:00 2001 From: atlas Date: Thu, 27 Aug 2026 11:39:33 +0200 Subject: [PATCH] ci: coverage on manual dispatch, no threshold Adds `cargo llvm-cov` to the devshell and a coverage workflow that runs only when someone asks for it. Manual dispatch rather than nightly or per-PR, per the discussion on the issue: a coverage run needs its own instrumented build and cannot reuse the normal test artifacts, so it roughly doubles a test job. "Do we have glaring holes" is a question someone asks occasionally, not a gate every PR pays for, and not a number worth spending farm time on every night whether or not anyone reads it. Manual dispatch costs nothing until the answer is wanted. Its own workflow file rather than a job in ci.yml: that file already has a `workflow_dispatch` trigger so `hive-forge ci-rerun` can retrigger without an empty commit, and a trigger there fires EVERY job in the file -- a coverage job added there would run on every pull request. No threshold and no --fail-under-lines. A coverage gate mostly teaches people to write assertion-free tests that execute lines; the report is the deliverable and the number is for a human to read. The devshell needs LLVM_COV / LLVM_PROFDATA set explicitly: cargo llvm-cov expects rustup's `llvm-tools-preview` beside the toolchain and nixpkgs has no such component, so without them it aborts with "failed to find llvm-tools-preview" -- which reads like a missing install rather than a path the shell has to name. Verified by running it: `cargo llvm-cov --package hive-types` produces a real report (3 tests, 85.44% regions). Both the variable names and the package were wrong on the first attempt and only running it said so -- the names take no `_PATH` suffix, and the binaries are in `llvmPackages.llvm`, not `llvmPackages.bintools`, which is the linker wrapper and ships neither. --- .forgejo/workflows/coverage.yml | 40 +++++++++++++++++++++++++++++++++ nix/devshell.nix | 21 +++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 .forgejo/workflows/coverage.yml diff --git a/.forgejo/workflows/coverage.yml b/.forgejo/workflows/coverage.yml new file mode 100644 index 00000000..013285eb --- /dev/null +++ b/.forgejo/workflows/coverage.yml @@ -0,0 +1,40 @@ +name: coverage + +# Manual dispatch ONLY — deliberately not on `pull_request`, and +# deliberately not nightly. +# +# A coverage run needs its own instrumented build: it cannot reuse the +# normal test artifacts, so it roughly doubles a test job. Coverage here +# answers "do we have glaring holes", which is a question someone asks +# occasionally — not a gate every PR pays for, and not a number worth +# spending farm time on every night whether or not anyone reads it. +# Manual dispatch costs nothing until someone wants the answer. +# +# Its own file rather than a job in ci.yml: that workflow already has a +# `workflow_dispatch` trigger (so `hive-forge ci-rerun` can retrigger +# without an empty commit), and a trigger there fires EVERY job in the +# file. A coverage job added to ci.yml would run on every pull request. +on: + workflow_dispatch: + +jobs: + coverage: + name: cargo llvm-cov + runs-on: [hive-ci] + # Above the 30 min `nix flake check` allows, because instrumented + # builds are slower than the ordinary ones and this starts from a + # cache that the normal jobs never populate. + timeout-minutes: 60 + steps: + - uses: actions/checkout@v3 + - name: coverage + # No threshold and no `--fail-under-lines`: a coverage gate + # mostly teaches people to write assertion-free tests that + # execute lines. The report is the deliverable; the number is + # for a human to look at. + # + # `--workspace` because the interesting holes are in the crates + # nobody runs directly, and the per-crate default would hide + # exactly those behind a green summary for the one crate someone + # happened to name. + run: nix develop -c cargo llvm-cov --workspace --summary-only diff --git a/nix/devshell.nix b/nix/devshell.nix index 68885df1..36c07af8 100644 --- a/nix/devshell.nix +++ b/nix/devshell.nix @@ -9,10 +9,31 @@ rust.nativeBuildInputs ++ (with pkgs; [ cargo + cargo-llvm-cov clippy rust-analyzer rustc rustfmt ]); + + # `cargo llvm-cov` shells out to `llvm-profdata` and `llvm-cov`, + # which it expects to find as rustup's `llvm-tools-preview` beside + # the rustc toolchain. Nixpkgs has no such component, so without + # these it aborts with "failed to find llvm-tools-preview" — a + # message that reads like a missing install rather than a path the + # shell simply has to name. + # + # ⚠️ Both the variable names and the package were wrong on the first + # attempt and only the tool itself said so: it reads `LLVM_COV` / + # `LLVM_PROFDATA` (no `_PATH` suffix), and the binaries live in + # `llvmPackages.llvm` — `llvmPackages.bintools` is the linker + # wrapper and ships neither. + # + # Coverage is deliberately NOT part of `nix flake check`: an + # instrumented build cannot reuse the normal test artifacts, so it + # roughly doubles a test job. It runs on manual dispatch — see + # .forgejo/workflows/coverage.yml for why that rather than nightly. + LLVM_PROFDATA = "${pkgs.llvmPackages.llvm}/bin/llvm-profdata"; + LLVM_COV = "${pkgs.llvmPackages.llvm}/bin/llvm-cov"; }; }