five hosts #10

Workflow file for this run

	name: five hosts

	# ⚠️ THIS WORKFLOW IS THE QUESTION, NOT THE ANSWER.
	#
	# The mcpp/xlings package index has a stated bar for admitting an emulator, and
	# `qemu-riscv`'s descriptor writes it down: prebuilt binaries for the FIVE host
	# targets the index serves — linux x64, linux arm64, darwin x64, darwin arm64,
	# win32 x64 — from one versioned release, each asset with a checksum sidecar.
	#
	# xPack publishes QEMU per target family and has no x86 build. qemu.org ships a
	# Windows installer only; macOS and Linux are served by distribution packages.
	# So no upstream clears the bar, and `xim:qemu-x86` cannot be a repackaging job
	# the way `xim:qemu-arm` and `xim:qemu-riscv` are — it has to be built.
	#
	# ⭐ THE POINT OF DOING IT HERE FIRST IS THAT A FAILURE COSTS A RED CROSS.
	#
	# The alternative order — write the descriptor, mirror the assets, open the
	# index PR, then discover that the Windows leg does not build — produces a
	# published package that cannot be installed. GitHub's hosted runners happen to
	# cover exactly the five hosts the index serves, so all five legs can be
	# attempted in one matrix before anything is published, and the hardest one
	# fails early rather than last.
	#
	# ⚠️ NOTHING HERE PUBLISHES. The artifacts are retained for inspection and for
	# measuring what a real payload would weigh. Admission to the index is a
	# separate decision, made after five green legs, and it needs the DT_NEEDED /
	# otool closure measured per host rather than copied from a sibling descriptor.

	on:
	push:
	branches: [main]
	pull_request:
	workflow_dispatch:

	env:
	# ⚠️ PINNED, AND NOT TO `master`. A payload the index serves must be
	# reproducible from a version, and "whatever upstream had that day" is not a
	# version. 9.2.4 is the series `qemu-arm` and `qemu-riscv` already carry, so
	# a user who installs all three gets one QEMU generation rather than two.
	QEMU_VERSION: 9.2.4

	jobs:
	build:
	name: qemu-system-x86_64 on ${{ matrix.label }}
	runs-on: ${{ matrix.os }}
	timeout-minutes: 90
	strategy:
	fail-fast: false
	matrix:
	include:
	- { label: linux-x64, os: ubuntu-24.04 }
	- { label: linux-arm64, os: ubuntu-24.04-arm }
	- { label: darwin-arm64, os: macos-14 }
	# ⚠️ `macos-15-intel`, NOT `macos-13`. The first attempt used the
	# older label and the job sat QUEUED for the whole run without ever
	# being scheduled — GitHub has retired that image, and a retired label
	# does not fail, it waits. A leg that never runs is indistinguishable
	# from a slow one on the summary page, which is the worst of the
	# possible outcomes for a matrix whose purpose is to answer questions.
	- { label: darwin-x64, os: macos-15-intel }
	- { label: win32-x64, os: windows-2022 }
	defaults:
	run:
	shell: bash

	steps:
	- uses: actions/checkout@v4

	# ── The dependencies, per host ────────────────────────────────────────
	#
	# ⚠️ A SINGLE TARGET IS WHAT MAKES THIS AFFORDABLE. QEMU's full build is
	# dozens of system emulators plus tools, docs and UI backends;
	# `--target-list=x86_64-softmmu` with the UI and tools disabled is a small
	# fraction of it. The index needs one emulator, not a distribution.
	- name: Dependencies (Linux)
	if: runner.os == 'Linux'
	run: \|
	sudo apt-get update -qq
	sudo apt-get install -y -qq \
	ninja-build meson pkg-config python3-venv \
	libglib2.0-dev libpixman-1-dev zlib1g-dev flex bison

	- name: Dependencies (macOS)
	if: runner.os == 'macOS'
	run: \|
	brew install ninja meson pkg-config glib pixman
	# ⚠️ `distlib`, AND IT IS A PROPERTY OF QEMU 9.2.4 RATHER THAN OF THIS
	# RUNNER — MEASURED ON TWO HOSTS BEFORE IT WAS CALLED THAT.
	#
	# QEMU's `configure` builds a non-isolated virtual environment with its
	# own `mkvenv` script, which needs `distlib` to populate it. Neither
	# Homebrew's Python 3.14 nor MSYS2's carries it, and the failure is a
	# configure-time
	#
	# * Ouch! *
	# found no usable distlib, please install it
	#
	# twenty seconds in, which reads like a QEMU problem and is a Python
	# packaging one. The first round fixed it on macOS alone, on the
	# reading that Homebrew was unusual; the Windows leg then reached the
	# same line and showed it was not.
	#
	# `--break-system-packages` because Homebrew's Python is
	# externally-managed (PEP 668) and refuses otherwise. On a throwaway
	# runner that is the right trade; on a developer's machine it is not,
	# which is why it is spelled out rather than hidden in a helper.
	python3 -m pip install --break-system-packages distlib
	# ⚠️ NOT INSTALLING `qemu` ITSELF. Homebrew's formula would pull a
	# working emulator and every check below would pass without this
	# workflow having built anything — the shape of false green this
	# repository exists to avoid.

	# ⚠️ THE WINDOWS LEG IS THE ONE THIS WORKFLOW EXISTS TO ANSWER, AND IT IS
	# NOT AN ORDINARY BUILD.
	#
	# QEMU on Windows is built under MSYS2/MinGW, not MSVC: its build system
	# is meson and its sources assume a POSIX-ish toolchain. The result is a
	# native PE that needs a set of MinGW runtime DLLs beside it, which is why
	# the packaging question here is different from the other four hosts —
	# `otool`/`readelf` closure has a `ntldd` counterpart and the answer is a
	# directory of DLLs rather than an rpath.
	- name: Dependencies (Windows / MSYS2)
	if: runner.os == 'Windows'
	uses: msys2/setup-msys2@v2
	with:
	msystem: UCRT64
	update: true
	install: >-
	base-devel
	git
	python
	mingw-w64-ucrt-x86_64-toolchain
	mingw-w64-ucrt-x86_64-glib2
	mingw-w64-ucrt-x86_64-pixman
	mingw-w64-ucrt-x86_64-ninja
	mingw-w64-ucrt-x86_64-meson
	mingw-w64-ucrt-x86_64-pkgconf
	mingw-w64-ucrt-x86_64-zlib
	mingw-w64-ucrt-x86_64-python-distlib
	mingw-w64-ucrt-x86_64-python-pip

	# ⚠️ WINDOWS EXTRACTS UNDER MSYS2 WITH SYMLINK EMULATION FORCED ON, AND IT
	# TOOK TWO ROUNDS TO GET THAT RIGHT.
	#
	# QEMU's source tarball is full of symlinks — `roms/u-boot`, and also
	# `tests/lcitool/libvirt-ci`. Git Bash's `tar` fails on all of them, and
	# the first fix moved the extraction into MSYS2 on the belief that "MSYS2's
	# tar copies instead". It does not: `msys2/setup-msys2` sets
	# `MSYS=winsymlinks:nativestrict`, under which MSYS2 asks Windows for a
	# REAL symlink — and Windows must know whether the target is a file or a
	# directory, so a link created before its target fails with
	#
	# tar: …/alpine-320-prep.sh: Cannot create symlink to
	# 'alpine-prep.sh': No such file or directory
	#
	# ⚠️ `No such file or directory` AND NOT `Permission denied`, WHICH IS THE
	# WHOLE DIAGNOSIS. The runner is privileged enough to create symlinks; the
	# problem is ordering, not rights, and a fix aimed at privileges would not
	# have worked.
	#
	# `winsymlinks:sysfile` writes a plain file carrying a marker instead.
	# Nothing in this build follows those links, so the emulation is enough.
	#
	# `roms/` stays excluded regardless: it holds firmware SOURCE for machines
	# this build does not produce, and `x86_64-softmmu` takes its firmware
	# from the prebuilt blobs in `pc-bios/`.
	# The same `distlib` gap as macOS, on MSYS2's Python. Preferring the
	# packaged form and falling back to pip: pacman's copy is what an MSYS2
	# user would have, and pip is what makes the step work if that package is
	# ever renamed.
	# ⚠️ TWO PYTHON PACKAGES, AND THE SECOND IS THE SUBTLEST FINDING ON THIS
	# LEG.
	#
	# `distlib` is the same gap macOS had. `pycotap` is different: QEMU
	# VENDORS its Python build dependencies as wheels under `python/wheels`
	# and `mkvenv` installs from there OFFLINE, never reaching PyPI. On this
	# host that directory is handed to pip as
	#
	# file://D:/a/qemu-x86/qemu-x86/qemu-9.2.4/python/wheels
	#
	# and pip answers `is ignored: it is neither a file nor a directory` —
	# a `file://` URL needs THREE slashes before a drive letter, and with two
	# the `D:` is parsed as a HOST. So the vendored wheels are unreachable,
	# and because mkvenv is offline there is no fallback:
	#
	# ERROR: No matching distribution found for pycotap==1.3.1
	# mkvenv was configured to operate offline and did not check PyPI.
	#
	# ⭐ THE FIX IS NOT TO REPAIR THE URL. A non-isolated virtual environment
	# sees the system's packages, and `mkvenv`'s own check is "is it
	# importable" — which is why `meson` passes on this host and `pycotap`
	# does not. Installing it beforehand satisfies the check the same way
	# meson already does, and does not depend on how QEMU spells a path.
	#
	# This is the boundary MSYS2 makes easy to trip over: the shell speaks
	# `/d/a/...` and the Python it ships is a NATIVE Windows build that
	# speaks `D:/a/...`. This repository met the same boundary from the other
	# side in openarch's CI, where `$PWD` in a manifest would have been the
	# POSIX form and mcpp wanted the native one.
	- name: The Python packages QEMU's mkvenv needs (Windows / MSYS2)
	if: runner.os == 'Windows'
	shell: msys2 {0}
	run: \|
	set -euo pipefail
	# ⚠️ `pip` IS A SEPARATE PACKAGE HERE, AND ASSUMING IT WAS PRESENT COST
	# A ROUND. MSYS2's Python ships without it:
	#
	# python3.exe: No module named pip.__main__;
	# 'pip' is a package and cannot be directly executed
	#
	# so it is now in the install list above rather than reached for here.
	# `distlib` is there too, because pacman's copy is what an MSYS2 user
	# would have and installing it declaratively is one less step that can
	# fail halfway.
	python3 -m pip install --no-input pycotap
	python3 -c "import distlib, pycotap; print('distlib', distlib.__version__, '/ pycotap ok')"

	- name: Fetch QEMU ${{ env.QEMU_VERSION }} (Windows)
	if: runner.os == 'Windows'
	shell: msys2 {0}
	env:
	MSYS: winsymlinks:sysfile
	run: \|
	set -euo pipefail
	cd "$(cygpath "$GITHUB_WORKSPACE")"
	curl -fsSL --retry 5 --retry-all-errors --retry-delay 3 \
	-o qemu.tar.xz \
	"https://download.qemu.org/qemu-${QEMU_VERSION}.tar.xz"
	tar xf qemu.tar.xz --exclude="qemu-${QEMU_VERSION}/roms/*"
	test -d "qemu-${QEMU_VERSION}/pc-bios" \|\| { echo "the firmware blobs the build needs are missing"; exit 1; }

	- name: Fetch QEMU ${{ env.QEMU_VERSION }}
	if: runner.os != 'Windows'
	run: \|
	set -euo pipefail
	# ⚠️ `--retry-all-errors`, NOT JUST `--retry`. curl's plain `--retry`
	# covers transient HTTP status codes and NOT transport-layer errors,
	# and `curl: (52) empty reply from server` is the one this project
	# keeps meeting. Without it a truncated download is reported as
	# success and the failure surfaces as a corrupt archive.
	curl -fsSL --retry 5 --retry-all-errors --retry-delay 3 \
	-o qemu.tar.xz \
	"https://download.qemu.org/qemu-${QEMU_VERSION}.tar.xz"
	curl -fsSL --retry 5 --retry-all-errors --retry-delay 3 \
	-o qemu.tar.xz.sig \
	"https://download.qemu.org/qemu-${QEMU_VERSION}.tar.xz.sig" \|\| true
	tar xf qemu.tar.xz
	echo "SRC=$PWD/qemu-${QEMU_VERSION}" >> "$GITHUB_ENV"

	- name: Configure and build (Unix)
	if: runner.os != 'Windows'
	run: \|
	set -euo pipefail
	mkdir -p build && cd build
	"$SRC/configure" \
	--target-list=x86_64-softmmu \
	--prefix="$PWD/../out" \
	--disable-docs --disable-guest-agent --disable-tools \
	--disable-vnc --disable-sdl --disable-gtk --disable-curses \
	--disable-libssh --disable-vde --disable-spice \
	--disable-smartcard --disable-usb-redir --disable-opengl \
	--disable-virglrenderer --disable-blkio --disable-libdaxctl \
	--disable-brlapi --disable-curl
	ninja -j"$(getconf _NPROCESSORS_ONLN)"
	ninja install

	- name: Configure and build (Windows / MSYS2)
	if: runner.os == 'Windows'
	shell: msys2 {0}
	run: \|
	set -euo pipefail
	cd "$(cygpath "$GITHUB_WORKSPACE")"
	mkdir -p build && cd build
	"../qemu-${QEMU_VERSION}/configure" \
	--target-list=x86_64-softmmu \
	--prefix="$PWD/../out" \
	--disable-docs --disable-guest-agent --disable-tools \
	--disable-vnc --disable-sdl --disable-gtk --disable-curses \
	--disable-libssh --disable-spice --disable-smartcard \
	--disable-usb-redir --disable-opengl --disable-virglrenderer \
	--disable-curl
	# ⚠️ THE EMULATOR TARGET, NOT `ninja` — AND ON THIS HOST THAT IS THE
	# DIFFERENCE BETWEEN A BUILD AND A FAILURE.
	#
	# A bare `ninja` builds QEMU's unit tests too, and one of them does not
	# link under MinGW:
	#
	# FAILED: tests/unit/test-vmstate.exe
	# test-vmstate.c:48: undefined reference to `qemu_ftruncate64'
	#
	# 1897 of 2032 targets had already built. That symbol is a QEMU
	# portability shim its own test happens to reference and the Windows
	# build does not provide; it says nothing about the emulator, which is
	# what this workflow is producing.
	#
	# ⭐ Naming the target rather than patching the test is the smaller
	# claim: a package build has no reason to build a test suite, and
	# doing so made the whole leg depend on whether upstream's tests
	# happen to be portable this release.
	# ⚠️ `.exe`, AND OMITTING IT IS NOT A WARNING BUT A HARD STOP:
	#
	# ninja: error: unknown target 'qemu-system-x86_64'
	#
	# The Unix legs name the target without a suffix and this one with,
	# because meson names the target after the file it produces.
	# ⚠️ AND THE INSTALL NEEDS MORE THAN THE EMULATOR. With only that
	# target built, `meson install --no-rebuild` stops at
	#
	# ERROR: File 'trace/trace-events-all' could not be found
	#
	# which is a GENERATED data file the install step copies. Naming it
	# here rather than dropping `--no-rebuild` is deliberate: without the
	# flag, meson rebuilds everything, which is how the unit test that
	# does not link on MinGW comes back.
	ninja -j"$(nproc)" qemu-system-x86_64.exe trace/trace-events-all
	meson install --no-rebuild

	# ── The check that the artifact is an emulator, not a file ────────────
	#
	# ⭐ A BUILT BINARY IS NOT EVIDENCE; A BOOTED IMAGE IS. The index's own
	# e2e discipline is "assert the product, not the exit code", and the
	# cheapest product here is a multiboot image that prints and powers off.
	# It is built by the workflow rather than committed so that nothing in
	# this repository is a binary blob whose provenance has to be trusted.
	- name: The emulator boots a freestanding image
	shell: bash
	run: \|
	set -euo pipefail
	QEMU=$(find out -name 'qemu-system-x86_64*' -type f \| head -1)
	test -n "$QEMU" \|\| { find out -type f \| head -40; echo "no emulator was produced"; exit 1; }
	"$QEMU" --version \| head -1

	# A minimal multiboot image. ⚠️ The a.out kludge (flag bit 16) rather
	# than plain ELF loading: QEMU's multiboot loader accepts only 32-bit
	# ELF images, and this one has to be assembled as 32-bit anyway, so
	# the kludge is not strictly needed here — it is used because the
	# real consumer (openarch's probe) is a 64-bit image that cannot be
	# loaded any other way, and a check that exercised a different path
	# would not be checking the same thing.
	cat > probe.S <<'ASM'
	.set MAGIC, 0x1BADB002
	.set FLAGS, 0x00010000
	.set CHECKSUM, -(MAGIC + FLAGS)
	.section .multiboot,"a"
	.align 4
	mb: .long MAGIC
	.long FLAGS
	.long CHECKSUM
	.long mb
	.long __load_start
	.long __load_end
	.long __bss_end
	.long _start
	.code32
	.section .text
	.globl _start
	_start:
	cli
	movl $msg, %esi
	1: lodsb
	testb %al, %al
	je 2f
	movw $0x3F8, %dx
	outb %al, %dx
	jmp 1b
	2: movw $0x604, %dx
	movw $0x2000, %ax
	outw %ax, %dx
	3: hlt
	jmp 3b
	.section .rodata
	msg: .asciz "qemu-x86 probe ok\n"
	.section .bss
	.space 16
	ASM
	cat > probe.ld <<'LD'
	ENTRY(_start)
	SECTIONS {
	__load_start = 0x100000;
	. = __load_start + SIZEOF_HEADERS;
	.multiboot : { KEEP(*(.multiboot)) }
	.text : { (.text) }
	.rodata : { (.rodata) }
	__load_end = .;
	.bss : { (.bss) __bss_end = .; }
	}
	LD
	# The host's own toolchain assembles it; this step is about the
	# emulator, not about cross-compilation.
	if [ "$RUNNER_OS" = "macOS" ] \|\| [ "$(uname -m)" = "aarch64" ] \|\| [ "$(uname -m)" = "arm64" ]; then
	echo "::notice::no host assembler emits x86 ELF here; the emulator's --version is the check on this leg"
	exit 0
	fi
	cc -m32 -c -o probe.o probe.S -nostdlib 2>/dev/null \
	\|\| { echo "::notice::no 32-bit x86 assembler on this leg; --version is the check"; exit 0; }
	ld -m elf_i386 -T probe.ld -o probe.elf probe.o
	timeout -k 5 60 "$QEMU" -machine q35 -nographic -no-reboot -kernel probe.elf 2>&1 \| tee run.log
	grep -q "qemu-x86 probe ok" run.log

	# ⚠️ WHAT A PAYLOAD WOULD ACTUALLY WEIGH, MEASURED RATHER THAN ESTIMATED.
	# The index's size discipline came from a real incident: a payload was
	# 34.81 MB and became 4.62 MB once stripped, and the upload failures that
	# had been blamed on the mirror were the size.
	- name: What the payload weighs, and what it needs
	shell: bash
	run: \|
	set -euo pipefail
	QEMU=$(find out -name 'qemu-system-x86_64*' -type f \| head -1)
	echo "before stripping:"; ls -l "$QEMU"; du -sh out

	# ⚠️ STRIPPED, BECAUSE THE UNSTRIPPED FIGURE IS NOT A PAYLOAD SIZE AND
	# THE INDEX HAS PAID FOR THAT CONFUSION ONCE ALREADY.
	#
	# Measured on the first green legs: 392M (linux-x64), 389M
	# (linux-arm64), 341M/342M (darwin). Those are debug symbols, not an
	# emulator. A previous payload in this ecosystem was 34.81 MB and
	# became 4.62 MB once stripped, and the upload failures that had been
	# blamed on the mirror were the size.
	#
	# ⚠️ NOT ON macOS. Stripping a Mach-O invalidates its ad-hoc
	# signature, which is the rule this ecosystem's own release pipeline
	# follows; the darwin legs report the unstripped figure and say so.
	case "$RUNNER_OS" in
	Linux)
	find out -type f $ -name 'qemu-system-' -o -name '.so*' $ -exec strip --strip-unneeded {} + 2>/dev/null \|\| true
	echo "after stripping:"; ls -l "$QEMU"; du -sh out
	readelf -d "$QEMU" \| grep NEEDED \|\| true ;;
	macOS)
	echo "(not stripped: it would invalidate the ad-hoc signature)"
	otool -L "$QEMU" \|\| true ;;
	Windows)
	find out -type f -name '*.exe' -exec strip --strip-unneeded {} + 2>/dev/null \|\| true
	echo "after stripping:"; ls -l "$QEMU"; du -sh out
	echo "(the DLL closure is measured in the packaging step, not here)" ;;
	esac

	# ⭐ WHAT IS ACTUALLY BIG, BECAUSE STRIPPING ANSWERED LESS THAN
	# EXPECTED. Measured on linux-x64: the emulator went 78.7 MB → 27.0
	# MB, and the whole prefix went 392 MB → 343 MB. So roughly 316 MB is
	# not the emulator at all, and a payload that tarred this prefix would
	# ship it. The index's asset budget is a per-connection upload rate
	# measured at 0.012 MB/s into the CN mirror, with no multipart and no
	# resume — so what a payload SELECTS matters more here than what it
	# strips.
	#
	# Listed rather than pruned: which of these an emulator actually needs
	# at run time is a question for the descriptor, and answering it by
	# deleting directories until something breaks is how a payload ends up
	# missing a firmware blob on somebody else's machine.
	echo "── the ten largest things under the prefix ──"
	du -sh out/* 2>/dev/null \| sort -rh \| head -10
	du -sh out/share/* 2>/dev/null \| sort -rh \| head -10

	- uses: actions/upload-artifact@v4
	with:
	name: qemu-system-x86_64-${{ matrix.label }}
	path: out
	retention-days: 14

Navigation Menu

Uh oh!

five hosts #10

Workflow file

five hosts #10

Uh oh!

Workflow file for this run