vm-nightly: one job per test, with a lint that keeps the matrix list in step
A stuck test in the shared job was cut at the runner's one-hour limit and hid the other tests' results. The matrix list is static; check-vm-test-list.sh fails CI when it differs from nix/vm-tests.nix.
This commit is contained in:
parent
cead194b2a
commit
d3e15c7fbe
6 changed files with 96 additions and 26 deletions
|
|
@ -80,6 +80,15 @@ jobs:
|
||||||
- name: lint
|
- name: lint
|
||||||
run: sh scripts/check-dashboard-service-names.sh
|
run: sh scripts/check-dashboard-service-names.sh
|
||||||
|
|
||||||
|
vm-test-list:
|
||||||
|
name: vm-test-list lint
|
||||||
|
if: vars.PUBLIC_FORGE != 'true'
|
||||||
|
runs-on: ${{ vars.PUBLIC_FORGE == 'true' && 'nixos' || 'hive-ci' }}
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v3
|
||||||
|
- name: lint
|
||||||
|
run: sh scripts/check-vm-test-list.sh
|
||||||
|
|
||||||
shellcheck:
|
shellcheck:
|
||||||
name: shellcheck
|
name: shellcheck
|
||||||
if: vars.PUBLIC_FORGE != 'true'
|
if: vars.PUBLIC_FORGE != 'true'
|
||||||
|
|
|
||||||
|
|
@ -2,16 +2,26 @@ name: vm-nightly
|
||||||
|
|
||||||
# Builds the `vmTests` flake output: tests that boot NixOS VMs and need KVM.
|
# Builds the `vmTests` flake output: tests that boot NixOS VMs and need KVM.
|
||||||
# They are not `checks`, so the per-PR `nix flake check` job never runs them.
|
# They are not `checks`, so the per-PR `nix flake check` job never runs them.
|
||||||
# Nightly plus manual dispatch; a failing test fails the run, which the forge
|
# Nightly plus manual dispatch; a failing test fails its job, which the forge
|
||||||
# shows as a failed scheduled run.
|
# shows as a failed scheduled run.
|
||||||
#
|
#
|
||||||
|
# One job per test. The runner kills a job after its `jobTimeout` (one hour by
|
||||||
|
# default) and reports nothing for it, so a stuck test in a shared job would
|
||||||
|
# erase the others' results. Each test's `globalTimeout` stays below that hour.
|
||||||
|
#
|
||||||
|
# The matrix list is static: a matrix built from a `needs` output needs
|
||||||
|
# Forgejo 14 and a matching runner, and this runner is gitea-actions-runner.
|
||||||
|
# `scripts/check-vm-test-list.sh` fails CI when the list differs from
|
||||||
|
# `nix/vm-tests.nix`.
|
||||||
|
#
|
||||||
# Run one by hand from the forge UI (Actions, vm-nightly, "Run workflow",
|
# Run one by hand from the forge UI (Actions, vm-nightly, "Run workflow",
|
||||||
# `test` input) or locally with `nix build .#vmTests.x86_64-linux.<name>`.
|
# `test` input) or locally with `nix build .#vmTests.x86_64-linux.<name>`.
|
||||||
|
# With `test` set, every other matrix job skips its steps and ends green.
|
||||||
#
|
#
|
||||||
# ⚠️ The hour is deliberate: at the default job capacity of 1
|
# ⚠️ The hour is deliberate: at the default job capacity of 1
|
||||||
# (`services.hyperhive.deploy.ci-runner.concurrency`) this job holds the
|
# (`services.hyperhive.deploy.ci-runner.concurrency`) the matrix jobs run one
|
||||||
# runner for its whole run and everything else queues behind it. It sits
|
# after another and hold the runner until the last ends; everything else
|
||||||
# between coverage (02:00) and flake-update (03:00).
|
# queues behind them. It sits between coverage (02:00) and flake-update (03:00).
|
||||||
#
|
#
|
||||||
# Separate from ci.yml because a `workflow_dispatch` trigger there fires every
|
# Separate from ci.yml because a `workflow_dispatch` trigger there fires every
|
||||||
# job in that file, and this must not run per pull request.
|
# job in that file, and this must not run per pull request.
|
||||||
|
|
@ -27,24 +37,26 @@ on:
|
||||||
- cron: "30 2 * * *"
|
- cron: "30 2 * * *"
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
vm-tests:
|
vm-test:
|
||||||
name: nix build vmTests
|
name: vm test ${{ matrix.test }}
|
||||||
if: vars.PUBLIC_FORGE != 'true'
|
if: vars.PUBLIC_FORGE != 'true'
|
||||||
runs-on: ${{ vars.PUBLIC_FORGE == 'true' && 'nixos' || 'hive-ci' }}
|
runs-on: ${{ vars.PUBLIC_FORGE == 'true' && 'nixos' || 'hive-ci' }}
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
matrix:
|
||||||
|
test:
|
||||||
|
- agent-ephemeral-rootfs
|
||||||
|
- hive-squid
|
||||||
|
- hive-egress-antispoof
|
||||||
|
- swarm-archive
|
||||||
|
- swarm-builder-offline
|
||||||
|
- full-setup
|
||||||
|
- semi-spread
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v3
|
- uses: actions/checkout@v3
|
||||||
|
if: inputs.test == '' || inputs.test == matrix.test
|
||||||
- name: build
|
- name: build
|
||||||
|
if: inputs.test == '' || inputs.test == matrix.test
|
||||||
env:
|
env:
|
||||||
TEST: ${{ inputs.test }}
|
TEST: ${{ matrix.test }}
|
||||||
run: |
|
run: nix build --no-link --print-build-logs ".#vmTests.x86_64-linux.$TEST"
|
||||||
set -eu
|
|
||||||
system=x86_64-linux
|
|
||||||
tests="$TEST"
|
|
||||||
if [ -z "$tests" ]; then
|
|
||||||
tests="$(nix eval --raw ".#vmTests.$system" --apply 'ts: builtins.concatStringsSep " " (builtins.attrNames ts)')"
|
|
||||||
fi
|
|
||||||
set --
|
|
||||||
for t in $tests; do
|
|
||||||
set -- "$@" ".#vmTests.$system.$t"
|
|
||||||
done
|
|
||||||
nix build --no-link --keep-going --print-build-logs "$@"
|
|
||||||
|
|
|
||||||
|
|
@ -123,12 +123,16 @@ with stand-in values (`passthru.typecheckScript`), from the same function as the
|
||||||
one.
|
one.
|
||||||
`vm-test-typecheck-control` passes only when the checker rejects a known type error, which shows the
|
`vm-test-typecheck-control` passes only when the checker rejects a known type error, which shows the
|
||||||
checker resolves `test_driver`. [`.forgejo/workflows/vm-nightly.yml`](../../../.forgejo/workflows/vm-nightly.yml)
|
checker resolves `test_driver`. [`.forgejo/workflows/vm-nightly.yml`](../../../.forgejo/workflows/vm-nightly.yml)
|
||||||
builds all of them nightly at 02:30; a failing test fails that run. To run one
|
builds them nightly at 02:30, one job per test, so the runner's one-hour job
|
||||||
by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
|
limit applies to each test alone and a stuck test no longer hides the others'
|
||||||
workflow, then name one test in the `test` field), or run
|
results. The matrix in `vm-nightly.yml` is the list of tests;
|
||||||
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. An empty field builds all tests; together they can pass the runner's one-hour job limit, so name one test when you want one result. Current tests:
|
`scripts/check-vm-test-list.sh` fails CI when it differs from `nix/vm-tests.nix`,
|
||||||
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
|
so a new test goes into both.
|
||||||
root, through the real hive-priv unit) and `hive-squid`.
|
Each test sets `globalTimeout` below that hour. With the default job capacity of
|
||||||
|
1 the jobs run one after another. To run one by hand, start the workflow from
|
||||||
|
the forge UI (Actions, vm-nightly, Run workflow, then name one test in the
|
||||||
|
`test` field; the other jobs skip their steps), or run
|
||||||
|
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM.
|
||||||
|
|
||||||
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
|
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
|
||||||
Nix interpolation. Commands the script runs in the VM find their binaries on the
|
Nix interpolation. Commands the script runs in the VM find their binaries on the
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
# NixOS VM tests: one derivation per test, exposed as `vmTests.<system>.<name>`.
|
# NixOS VM tests: one derivation per test, exposed as `vmTests.<system>.<name>`.
|
||||||
# Kept out of `checks` because each boots VMs and needs KVM; the nightly
|
# Kept out of `checks` because each boots VMs and needs KVM; the nightly
|
||||||
# workflow `.forgejo/workflows/vm-nightly.yml` builds them.
|
# workflow `.forgejo/workflows/vm-nightly.yml` builds them, one job per test; a
|
||||||
|
# new test also needs an entry in that workflow's matrix.
|
||||||
{
|
{
|
||||||
pkgs,
|
pkgs,
|
||||||
hivePriv,
|
hivePriv,
|
||||||
|
|
|
||||||
38
scripts/check-vm-test-list.sh
Executable file
38
scripts/check-vm-test-list.sh
Executable file
|
|
@ -0,0 +1,38 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# CI lint: the `test` matrix in vm-nightly.yml must list exactly the
|
||||||
|
# attributes of `nix/vm-tests.nix`. The matrix is static, so a test missing
|
||||||
|
# from it never runs nightly and a stale entry fails its job.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
workflow=".forgejo/workflows/vm-nightly.yml"
|
||||||
|
tests_nix="nix/vm-tests.nix"
|
||||||
|
|
||||||
|
declared="$(sed -n 's/^ \([A-Za-z0-9_-][A-Za-z0-9_-]*\) = import .*/\1/p' "$tests_nix" | sort)"
|
||||||
|
listed="$(awk '
|
||||||
|
/^ test:$/ { inlist = 1; next }
|
||||||
|
inlist && /^ - / { sub(/^ - /, ""); print; next }
|
||||||
|
inlist { inlist = 0 }
|
||||||
|
' "$workflow" | sort)"
|
||||||
|
|
||||||
|
if [ -z "$declared" ]; then
|
||||||
|
echo "::error file=$tests_nix::no test attributes found; the extraction pattern no longer matches the file" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if [ -z "$listed" ]; then
|
||||||
|
echo "::error file=$workflow::no matrix entries found under 'test:'; the extraction pattern no longer matches the file" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$declared" != "$listed" ]; then
|
||||||
|
work="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$work"' EXIT
|
||||||
|
echo "$declared" >"$work/declared"
|
||||||
|
echo "$listed" >"$work/listed"
|
||||||
|
echo "::error file=$workflow::matrix test list differs from $tests_nix" >&2
|
||||||
|
echo "only in $tests_nix:" >&2
|
||||||
|
comm -23 "$work/declared" "$work/listed" >&2
|
||||||
|
echo "only in $workflow:" >&2
|
||||||
|
comm -13 "$work/declared" "$work/listed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "vm-nightly matrix lists all $(echo "$declared" | wc -l) tests of $tests_nix"
|
||||||
|
|
@ -45,5 +45,11 @@ if ! sh "$repo_root/scripts/check-dashboard-service-names.sh"; then
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
echo "pre-push: running vm-test-list lint..." >&2
|
||||||
|
if ! sh "$repo_root/scripts/check-vm-test-list.sh"; then
|
||||||
|
echo "pre-push: vm-test-list lint FAILED — fix before pushing" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
echo "pre-push: lints passed ✓" >&2
|
echo "pre-push: lints passed ✓" >&2
|
||||||
exit 0
|
exit 0
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue