Skip to content

Commit 6b9c47f

Browse files
committed
ci(matrix): nothing asserted that a host the table names was actually scanned
Every check in this workflow is per host, and no per-host check can notice a host that never ran. Deleting a host from the matrix silently retires every expectation the table holds for it: nothing measures those rows, nothing compares them, and the workflow is green. The new `coverage` job reads the expected table as a whole and requires the run to have covered it — both directions. A host with rows and no scan is named; a host that scanned and has no rows is named as a new build host nobody declared expectations for, which is a cause rather than 40 unexplained cells. ⭐ And it ties the axis to the release. The set of build hosts is DERIVED from `release.yml`'s asset names rather than restated, so adding a fifth published host fails this step until the matrix covers it. A host that gets a binary and no scan is a host whose target table is a claim nobody checked; a host scanned and never shipped is coverage spent on a machine no user has. ⚠️ It reds right now, by design: the release publishes four and expected.tsv holds one. The other three backfill from their first scan artifact.
1 parent 7fac046 commit 6b9c47f

1 file changed

Lines changed: 79 additions & 0 deletions

File tree

.github/workflows/ci-target-matrix.yml

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,3 +221,82 @@ jobs:
221221
bash tests/matrix/compare.sh measured-graph.tsv \
222222
tests/matrix/expected.tsv ${{ matrix.host }} graph || fail=1
223223
[ "$fail" = 0 ] || exit 1
224+
225+
coverage:
226+
# ⭐⭐ THE DENOMINATOR. Every check above is per host, and no per-host check
227+
# can notice a host that never ran.
228+
#
229+
# ⚠️ Each `scan` job compares the rows for ITS OWN host, so deleting a host
230+
# from the matrix above silently retires every expectation the table holds
231+
# for it: nothing measures those rows, nothing compares them, and the
232+
# workflow is green. This job is the one place that reads the expected table
233+
# as a whole and requires the run to have covered it.
234+
#
235+
# It runs even when a scan failed (`always()`), because "which hosts were
236+
# covered" is a different question from "did they pass" — and a run where a
237+
# host is missing entirely should say so in those words rather than leaving
238+
# a reader to infer it from a job list.
239+
name: coverage (every host the table names was scanned)
240+
needs: scan
241+
if: always()
242+
runs-on: ubuntu-24.04
243+
steps:
244+
- uses: actions/checkout@v4
245+
- uses: actions/download-artifact@v4
246+
with:
247+
pattern: matrix-*
248+
path: measured
249+
- name: Every host in expected.tsv produced rows
250+
run: |
251+
set -euo pipefail
252+
want=$(awk -F'\t' 'NF>=11 {print $2}' tests/matrix/expected.tsv | sort -u)
253+
[ -n "$want" ] || { echo "::error::expected.tsv names no host at all"; exit 1; }
254+
got=$(cat measured/*/measured.tsv 2>/dev/null \
255+
| awk -F'\t' 'NF>=11 {print $2}' | sort -u)
256+
echo "expected hosts: $(echo $want)"
257+
echo "scanned hosts: $(echo ${got:-<none>})"
258+
fail=0
259+
for h in $want; do
260+
printf '%s\n' "$got" | grep -qx "$h" || {
261+
echo "::error::expected.tsv holds rows for '$h', and no scan produced any"
262+
fail=1
263+
}
264+
done
265+
# ⭐ AND THE OTHER DIRECTION. A host that scanned but has no rows in the
266+
# table is a new build host nobody declared expectations for — the
267+
# per-host compare already reds on it, but saying it here names the
268+
# cause rather than listing 40 unexplained cells.
269+
for h in $got; do
270+
printf '%s\n' "$want" | grep -qx "$h" || {
271+
echo "::error::'$h' was scanned and the expected table does not mention it"
272+
echo " add its rows to tests/matrix/expected.tsv from this run's artifact"
273+
fail=1
274+
}
275+
done
276+
[ "$fail" = 0 ] || exit 1
277+
echo "OK: every build host the table names was scanned, and no other"
278+
279+
- name: The build hosts mcpp ships for are the ones scanned
280+
run: |
281+
set -euo pipefail
282+
# ⚠️⚠️ THE TABLE AND THE RELEASE MUST NAME THE SAME SET. A host that
283+
# gets a published binary and no scan is a host whose target table is
284+
# a claim nobody checked; a host that is scanned and never shipped is
285+
# coverage spent on a machine no user has.
286+
#
287+
# Derived from release.yml's asset names rather than restated here, so
288+
# adding a fifth host to the release fails this step until the matrix
289+
# covers it.
290+
ship=$(grep -oE 'mcpp-\$\{?[A-Za-z_{}. ]*\}?-(linux|macosx|windows)-(x86_64|aarch64|arm64)' \
291+
.github/workflows/release.yml \
292+
| sed -E 's/.*-(linux|macosx|windows)-/\1-/' \
293+
| sed 's/^macosx-/macos-/' | sort -u)
294+
scanned=$(awk -F'\t' 'NF>=11 {print $2}' tests/matrix/expected.tsv | sort -u)
295+
echo "release publishes: $(echo $ship)"
296+
echo "matrix declares: $(echo $scanned)"
297+
if [ "$ship" != "$scanned" ]; then
298+
echo "::error::the set of build hosts mcpp publishes and the set the target matrix declares differ"
299+
diff <(printf '%s\n' "$ship") <(printf '%s\n' "$scanned") | sed 's/^/ /' || true
300+
exit 1
301+
fi
302+
echo "OK: $(printf '%s\n' "$ship" | wc -l) build hosts, published and scanned"

0 commit comments

Comments
 (0)