macOS 那条腿静默退 1:一个没匹配到的 grep 被 pipefail 提升成了失败 #17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: five hosts | |
| # ⚠️ THIS WORKFLOW IS THE QUESTION, NOT THE ANSWER. | |
| # | |
| # The mcpp/xlings package index has a stated bar for admitting an emulator, and | |
| # `qemu-riscv`'s descriptor writes it down: prebuilt binaries for the FIVE host | |
| # targets the index serves — linux x64, linux arm64, darwin x64, darwin arm64, | |
| # win32 x64 — from one versioned release, each asset with a checksum sidecar. | |
| # | |
| # xPack publishes QEMU per target family and has no x86 build. qemu.org ships a | |
| # Windows installer only; macOS and Linux are served by distribution packages. | |
| # So no upstream clears the bar, and `xim:qemu-x86` cannot be a repackaging job | |
| # the way `xim:qemu-arm` and `xim:qemu-riscv` are — it has to be built. | |
| # | |
| # ⭐ THE POINT OF DOING IT HERE FIRST IS THAT A FAILURE COSTS A RED CROSS. | |
| # | |
| # The alternative order — write the descriptor, mirror the assets, open the | |
| # index PR, then discover that the Windows leg does not build — produces a | |
| # published package that cannot be installed. GitHub's hosted runners happen to | |
| # cover exactly the five hosts the index serves, so all five legs can be | |
| # attempted in one matrix before anything is published, and the hardest one | |
| # fails early rather than last. | |
| # | |
| # ⚠️ NOTHING HERE PUBLISHES. The artifacts are retained for inspection and for | |
| # measuring what a real payload would weigh. Admission to the index is a | |
| # separate decision, made after five green legs, and it needs the DT_NEEDED / | |
| # otool closure measured per host rather than copied from a sibling descriptor. | |
| on: | |
| push: | |
| branches: [main] | |
| # ⚠️ A TAG BUILDS THE SAME FIVE LEGS AND PUBLISHES THEM. The matrix is not | |
| # duplicated into a separate release workflow: a release built by different | |
| # steps from the ones that were tested is a release nobody tested. | |
| tags: ['[0-9]+.[0-9]+.[0-9]+*'] | |
| pull_request: | |
| workflow_dispatch: | |
| env: | |
| # ⚠️ PINNED, AND NOT TO `master`. A payload the index serves must be | |
| # reproducible from a version, and "whatever upstream had that day" is not a | |
| # version. 9.2.4 is the series `qemu-arm` and `qemu-riscv` already carry, so | |
| # a user who installs all three gets one QEMU generation rather than two. | |
| QEMU_VERSION: 9.2.4 | |
| jobs: | |
| build: | |
| name: qemu-system-x86_64 on ${{ matrix.label }} | |
| runs-on: ${{ matrix.os }} | |
| timeout-minutes: 90 | |
| permissions: | |
| contents: write # the tag path attaches archives to the release | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| include: | |
| - { label: linux-x64, os: ubuntu-24.04 } | |
| - { label: linux-arm64, os: ubuntu-24.04-arm } | |
| - { label: darwin-arm64, os: macos-14 } | |
| # ⚠️ `macos-15-intel`, NOT `macos-13`. The first attempt used the | |
| # older label and the job sat QUEUED for the whole run without ever | |
| # being scheduled — GitHub has retired that image, and a retired label | |
| # does not fail, it waits. A leg that never runs is indistinguishable | |
| # from a slow one on the summary page, which is the worst of the | |
| # possible outcomes for a matrix whose purpose is to answer questions. | |
| - { label: darwin-x64, os: macos-15-intel } | |
| - { label: win32-x64, os: windows-2022 } | |
| defaults: | |
| run: | |
| shell: bash | |
| steps: | |
| - uses: actions/checkout@v4 | |
| # ── The dependencies, per host ──────────────────────────────────────── | |
| # | |
| # ⚠️ A SINGLE TARGET IS WHAT MAKES THIS AFFORDABLE. QEMU's full build is | |
| # dozens of system emulators plus tools, docs and UI backends; | |
| # `--target-list=x86_64-softmmu` with the UI and tools disabled is a small | |
| # fraction of it. The index needs one emulator, not a distribution. | |
| - name: Dependencies (Linux) | |
| if: runner.os == 'Linux' | |
| run: | | |
| sudo apt-get update -qq | |
| sudo apt-get install -y -qq \ | |
| ninja-build meson pkg-config python3-venv \ | |
| libglib2.0-dev libpixman-1-dev zlib1g-dev flex bison | |
| - name: Dependencies (macOS) | |
| if: runner.os == 'macOS' | |
| run: | | |
| brew install ninja meson pkg-config glib pixman | |
| # ⚠️ `distlib`, AND IT IS A PROPERTY OF QEMU 9.2.4 RATHER THAN OF THIS | |
| # RUNNER — MEASURED ON TWO HOSTS BEFORE IT WAS CALLED THAT. | |
| # | |
| # QEMU's `configure` builds a non-isolated virtual environment with its | |
| # own `mkvenv` script, which needs `distlib` to populate it. Neither | |
| # Homebrew's Python 3.14 nor MSYS2's carries it, and the failure is a | |
| # configure-time | |
| # | |
| # *** Ouch! *** | |
| # found no usable distlib, please install it | |
| # | |
| # twenty seconds in, which reads like a QEMU problem and is a Python | |
| # packaging one. The first round fixed it on macOS alone, on the | |
| # reading that Homebrew was unusual; the Windows leg then reached the | |
| # same line and showed it was not. | |
| # | |
| # `--break-system-packages` because Homebrew's Python is | |
| # externally-managed (PEP 668) and refuses otherwise. On a throwaway | |
| # runner that is the right trade; on a developer's machine it is not, | |
| # which is why it is spelled out rather than hidden in a helper. | |
| python3 -m pip install --break-system-packages distlib | |
| # ⚠️ NOT INSTALLING `qemu` ITSELF. Homebrew's formula would pull a | |
| # working emulator and every check below would pass without this | |
| # workflow having built anything — the shape of false green this | |
| # repository exists to avoid. | |
| # ⚠️ THE WINDOWS LEG IS THE ONE THIS WORKFLOW EXISTS TO ANSWER, AND IT IS | |
| # NOT AN ORDINARY BUILD. | |
| # | |
| # QEMU on Windows is built under MSYS2/MinGW, not MSVC: its build system | |
| # is meson and its sources assume a POSIX-ish toolchain. The result is a | |
| # native PE that needs a set of MinGW runtime DLLs beside it, which is why | |
| # the packaging question here is different from the other four hosts — | |
| # `otool`/`readelf` closure has a `ntldd` counterpart and the answer is a | |
| # directory of DLLs rather than an rpath. | |
| - name: Dependencies (Windows / MSYS2) | |
| if: runner.os == 'Windows' | |
| uses: msys2/setup-msys2@v2 | |
| with: | |
| msystem: UCRT64 | |
| update: true | |
| install: >- | |
| base-devel | |
| git | |
| python | |
| mingw-w64-ucrt-x86_64-toolchain | |
| mingw-w64-ucrt-x86_64-glib2 | |
| mingw-w64-ucrt-x86_64-pixman | |
| mingw-w64-ucrt-x86_64-ninja | |
| mingw-w64-ucrt-x86_64-meson | |
| mingw-w64-ucrt-x86_64-pkgconf | |
| mingw-w64-ucrt-x86_64-zlib | |
| mingw-w64-ucrt-x86_64-python-distlib | |
| mingw-w64-ucrt-x86_64-python-pip | |
| # ⚠️ WINDOWS EXTRACTS UNDER MSYS2 WITH SYMLINK EMULATION FORCED ON, AND IT | |
| # TOOK TWO ROUNDS TO GET THAT RIGHT. | |
| # | |
| # QEMU's source tarball is full of symlinks — `roms/u-boot`, and also | |
| # `tests/lcitool/libvirt-ci`. Git Bash's `tar` fails on all of them, and | |
| # the first fix moved the extraction into MSYS2 on the belief that "MSYS2's | |
| # tar copies instead". It does not: `msys2/setup-msys2` sets | |
| # `MSYS=winsymlinks:nativestrict`, under which MSYS2 asks Windows for a | |
| # REAL symlink — and Windows must know whether the target is a file or a | |
| # directory, so a link created before its target fails with | |
| # | |
| # tar: …/alpine-320-prep.sh: Cannot create symlink to | |
| # 'alpine-prep.sh': No such file or directory | |
| # | |
| # ⚠️ `No such file or directory` AND NOT `Permission denied`, WHICH IS THE | |
| # WHOLE DIAGNOSIS. The runner is privileged enough to create symlinks; the | |
| # problem is ordering, not rights, and a fix aimed at privileges would not | |
| # have worked. | |
| # | |
| # `winsymlinks:sysfile` writes a plain file carrying a marker instead. | |
| # Nothing in this build follows those links, so the emulation is enough. | |
| # | |
| # `roms/` stays excluded regardless: it holds firmware SOURCE for machines | |
| # this build does not produce, and `x86_64-softmmu` takes its firmware | |
| # from the prebuilt blobs in `pc-bios/`. | |
| # The same `distlib` gap as macOS, on MSYS2's Python. Preferring the | |
| # packaged form and falling back to pip: pacman's copy is what an MSYS2 | |
| # user would have, and pip is what makes the step work if that package is | |
| # ever renamed. | |
| # ⚠️ TWO PYTHON PACKAGES, AND THE SECOND IS THE SUBTLEST FINDING ON THIS | |
| # LEG. | |
| # | |
| # `distlib` is the same gap macOS had. `pycotap` is different: QEMU | |
| # VENDORS its Python build dependencies as wheels under `python/wheels` | |
| # and `mkvenv` installs from there OFFLINE, never reaching PyPI. On this | |
| # host that directory is handed to pip as | |
| # | |
| # file://D:/a/qemu-x86/qemu-x86/qemu-9.2.4/python/wheels | |
| # | |
| # and pip answers `is ignored: it is neither a file nor a directory` — | |
| # a `file://` URL needs THREE slashes before a drive letter, and with two | |
| # the `D:` is parsed as a HOST. So the vendored wheels are unreachable, | |
| # and because mkvenv is offline there is no fallback: | |
| # | |
| # ERROR: No matching distribution found for pycotap==1.3.1 | |
| # mkvenv was configured to operate offline and did not check PyPI. | |
| # | |
| # ⭐ THE FIX IS NOT TO REPAIR THE URL. A non-isolated virtual environment | |
| # sees the system's packages, and `mkvenv`'s own check is "is it | |
| # importable" — which is why `meson` passes on this host and `pycotap` | |
| # does not. Installing it beforehand satisfies the check the same way | |
| # meson already does, and does not depend on how QEMU spells a path. | |
| # | |
| # This is the boundary MSYS2 makes easy to trip over: the shell speaks | |
| # `/d/a/...` and the Python it ships is a NATIVE Windows build that | |
| # speaks `D:/a/...`. This repository met the same boundary from the other | |
| # side in openarch's CI, where `$PWD` in a manifest would have been the | |
| # POSIX form and mcpp wanted the native one. | |
| - name: The Python packages QEMU's mkvenv needs (Windows / MSYS2) | |
| if: runner.os == 'Windows' | |
| shell: msys2 {0} | |
| run: | | |
| set -euo pipefail | |
| # ⚠️ `pip` IS A SEPARATE PACKAGE HERE, AND ASSUMING IT WAS PRESENT COST | |
| # A ROUND. MSYS2's Python ships without it: | |
| # | |
| # python3.exe: No module named pip.__main__; | |
| # 'pip' is a package and cannot be directly executed | |
| # | |
| # so it is now in the install list above rather than reached for here. | |
| # `distlib` is there too, because pacman's copy is what an MSYS2 user | |
| # would have and installing it declaratively is one less step that can | |
| # fail halfway. | |
| python3 -m pip install --no-input pycotap | |
| python3 -c "import distlib, pycotap; print('distlib', distlib.__version__, '/ pycotap ok')" | |
| - name: Fetch QEMU ${{ env.QEMU_VERSION }} (Windows) | |
| if: runner.os == 'Windows' | |
| shell: msys2 {0} | |
| env: | |
| MSYS: winsymlinks:sysfile | |
| run: | | |
| set -euo pipefail | |
| cd "$(cygpath "$GITHUB_WORKSPACE")" | |
| curl -fsSL --retry 5 --retry-all-errors --retry-delay 3 \ | |
| -o qemu.tar.xz \ | |
| "https://download.qemu.org/qemu-${QEMU_VERSION}.tar.xz" | |
| tar xf qemu.tar.xz --exclude="qemu-${QEMU_VERSION}/roms/*" | |
| test -d "qemu-${QEMU_VERSION}/pc-bios" || { echo "the firmware blobs the build needs are missing"; exit 1; } | |
| - name: Fetch QEMU ${{ env.QEMU_VERSION }} | |
| if: runner.os != 'Windows' | |
| run: | | |
| set -euo pipefail | |
| # ⚠️ `--retry-all-errors`, NOT JUST `--retry`. curl's plain `--retry` | |
| # covers transient HTTP status codes and NOT transport-layer errors, | |
| # and `curl: (52) empty reply from server` is the one this project | |
| # keeps meeting. Without it a truncated download is reported as | |
| # success and the failure surfaces as a corrupt archive. | |
| curl -fsSL --retry 5 --retry-all-errors --retry-delay 3 \ | |
| -o qemu.tar.xz \ | |
| "https://download.qemu.org/qemu-${QEMU_VERSION}.tar.xz" | |
| curl -fsSL --retry 5 --retry-all-errors --retry-delay 3 \ | |
| -o qemu.tar.xz.sig \ | |
| "https://download.qemu.org/qemu-${QEMU_VERSION}.tar.xz.sig" || true | |
| tar xf qemu.tar.xz | |
| echo "SRC=$PWD/qemu-${QEMU_VERSION}" >> "$GITHUB_ENV" | |
| - name: Configure and build (Unix) | |
| if: runner.os != 'Windows' | |
| run: | | |
| set -euo pipefail | |
| mkdir -p build && cd build | |
| "$SRC/configure" \ | |
| --target-list=x86_64-softmmu \ | |
| --prefix="$PWD/../out" \ | |
| --disable-docs --disable-guest-agent --disable-tools \ | |
| --disable-vnc --disable-sdl --disable-gtk --disable-curses \ | |
| --disable-libssh --disable-vde --disable-spice \ | |
| --disable-smartcard --disable-usb-redir --disable-opengl \ | |
| --disable-virglrenderer --disable-blkio --disable-libdaxctl \ | |
| --disable-brlapi --disable-curl | |
| ninja -j"$(getconf _NPROCESSORS_ONLN)" qemu-system-x86_64 | |
| # ⚠️ ASSEMBLED EXPLICITLY, THE SAME WAY THE WINDOWS LEG DOES, AND THE | |
| # NUMBERS ARE WHY. | |
| # | |
| # `ninja install` was used here first. Measured against the explicit | |
| # assembly on the same release: | |
| # | |
| # meson install (linux-x64) 392 MB → 343 MB stripped | |
| # explicit (win32-x64) 110 MB → 58 MB stripped | |
| # | |
| # So the install target ships roughly 285 MB that an emulator payload | |
| # has no use for. The index's asset budget is a per-connection upload | |
| # rate measured at 0.012 MB/s into the CN mirror, with no multipart | |
| # and no resume — a payload has to SELECT, and selecting here makes | |
| # the contents explicit rather than "whatever the install target | |
| # copied". | |
| # | |
| # ⚠️ NOTHING IS PRUNED FROM `pc-bios`. Which blobs a machine type | |
| # loads at run time is a question for the descriptor, and answering it | |
| # by deleting files until something breaks is how a payload ends up | |
| # missing one on somebody else's machine. | |
| mkdir -p ../out/bin ../out/share/qemu ../out/lib | |
| cp qemu-system-x86_64 ../out/bin/ | |
| cp -r "$SRC/pc-bios/." ../out/share/qemu/ | |
| # ⚠️ THE PAYLOAD MUST CARRY WHAT IT LINKS AGAINST, AND MEASURING THAT | |
| # IS WHAT CAUGHT THIS. | |
| # | |
| # An earlier build shipped only the emulator. Its measured closure: | |
| # | |
| # linux libpixman-1.so.0, libglib-2.0.so.0, libgio, libgobject, | |
| # libgmodule, libz.so.1, libzstd.so.1 -- none of them in | |
| # the payload, none of them core glibc | |
| # darwin /opt/homebrew/opt/{pixman,glib,zstd,gnutls,...}/lib/*.dylib | |
| # -- ABSOLUTE paths into the build machine's Homebrew | |
| # | |
| # So that payload installs and then fails on first run for anyone | |
| # without those libraries, and on macOS for anyone whose Homebrew is | |
| # not at the same prefix -- an Intel mac uses /usr/local. The index's | |
| # sibling packages avoid this by bundling: xPack's QEMU carries 51 | |
| # shared objects and reaches them through its own relative rpath. | |
| # | |
| # ⚠️ SYSTEM LIBRARIES ARE DELIBERATELY NOT BUNDLED. Core libc, libm | |
| # and the macOS frameworks come from the running system by design; | |
| # carrying a second libc into a process is the two-glibcs shape this | |
| # index's own contract text names as a SIGSEGV source. | |
| if [ "$RUNNER_OS" = "Linux" ]; then | |
| set +e # same reason as the macOS branch below | |
| ldd qemu-system-x86_64 \ | |
| | awk '/=> \// {print $3}' \ | |
| | grep -vE '/(libc|libm|libdl|librt|libpthread|libresolv|ld-linux[^ ]*)\.so' \ | |
| | while read -r so; do cp -Ln "$so" ../out/lib/ 2>/dev/null || true; done | |
| # ⚠️ `patchelf` on the EMULATOR, not on a loader. This project's own | |
| # notes forbid patching a loader's own paths; setting a relative | |
| # rpath on an ordinary executable is the opposite case, and it is | |
| # what makes `out/lib` reachable without an environment variable. | |
| sudo apt-get install -y -qq patchelf | |
| patchelf --set-rpath '$ORIGIN/../lib' ../out/bin/qemu-system-x86_64 | |
| for so in ../out/lib/*.so*; do patchelf --set-rpath '$ORIGIN' "$so" 2>/dev/null || true; done | |
| echo "bundled: $(ls -1 ../out/lib 2>/dev/null | wc -l) libraries" | |
| set -e | |
| elif [ "$RUNNER_OS" = "macOS" ]; then | |
| # ⚠️ `set +e` FOR THIS BLOCK, AND THE REASON IS A SILENT KILL. | |
| # | |
| # The first version ran under `set -euo pipefail` and died with exit | |
| # 1 and NO output, right after ninja reported 1564/1564. Every | |
| # command here is in a pipeline ending in `grep`, and a `grep` that | |
| # matches nothing returns 1 — which `pipefail` promotes to the | |
| # pipeline's status and `-e` turns into an exit. A bundling step | |
| # that finds nothing to bundle is not an error; a payload that still | |
| # points outside itself is, and the assertion below is what says so. | |
| set +e | |
| # Mach-O records each dependency by the install name the LIBRARY | |
| # carries, so copying is not enough: every reference has to be | |
| # rewritten, and so does each bundled library's own id. | |
| otool -L qemu-system-x86_64 | tail -n +2 | awk '{print $1}' \ | |
| | grep -E '^/(opt|usr/local)/' \ | |
| | while read -r dy; do | |
| cp -Ln "$dy" ../out/lib/ 2>/dev/null || true | |
| install_name_tool -change "$dy" "@loader_path/../lib/$(basename "$dy")" \ | |
| ../out/bin/qemu-system-x86_64 2>/dev/null || true | |
| done | |
| for dy in ../out/lib/*.dylib; do | |
| [ -e "$dy" ] || continue | |
| install_name_tool -id "@loader_path/$(basename "$dy")" "$dy" 2>/dev/null || true | |
| otool -L "$dy" | tail -n +2 | awk '{print $1}' | grep -E '^/(opt|usr/local)/' \ | |
| | while read -r dep; do | |
| cp -Ln "$dep" ../out/lib/ 2>/dev/null || true | |
| install_name_tool -change "$dep" "@loader_path/$(basename "$dep")" "$dy" 2>/dev/null || true | |
| done | |
| done | |
| # ⚠️ Rewriting invalidates the ad-hoc signature; re-sign, or the | |
| # binary is killed by the kernel rather than merely warned about. | |
| codesign --force -s - ../out/bin/qemu-system-x86_64 2>/dev/null || true | |
| for dy in ../out/lib/*.dylib; do codesign --force -s - "$dy" 2>/dev/null || true; done | |
| echo "bundled: $(ls -1 ../out/lib 2>/dev/null | wc -l) libraries" | |
| set -e | |
| fi | |
| rmdir ../out/lib 2>/dev/null || true | |
| - name: Configure and build (Windows / MSYS2) | |
| if: runner.os == 'Windows' | |
| shell: msys2 {0} | |
| run: | | |
| set -euo pipefail | |
| cd "$(cygpath "$GITHUB_WORKSPACE")" | |
| mkdir -p build && cd build | |
| "../qemu-${QEMU_VERSION}/configure" \ | |
| --target-list=x86_64-softmmu \ | |
| --prefix="$PWD/../out" \ | |
| --disable-docs --disable-guest-agent --disable-tools \ | |
| --disable-vnc --disable-sdl --disable-gtk --disable-curses \ | |
| --disable-libssh --disable-spice --disable-smartcard \ | |
| --disable-usb-redir --disable-opengl --disable-virglrenderer \ | |
| --disable-curl | |
| # ⚠️ THE EMULATOR TARGET, NOT `ninja` — AND ON THIS HOST THAT IS THE | |
| # DIFFERENCE BETWEEN A BUILD AND A FAILURE. | |
| # | |
| # A bare `ninja` builds QEMU's unit tests too, and one of them does not | |
| # link under MinGW: | |
| # | |
| # FAILED: tests/unit/test-vmstate.exe | |
| # test-vmstate.c:48: undefined reference to `qemu_ftruncate64' | |
| # | |
| # 1897 of 2032 targets had already built. That symbol is a QEMU | |
| # portability shim its own test happens to reference and the Windows | |
| # build does not provide; it says nothing about the emulator, which is | |
| # what this workflow is producing. | |
| # | |
| # ⭐ Naming the target rather than patching the test is the smaller | |
| # claim: a package build has no reason to build a test suite, and | |
| # doing so made the whole leg depend on whether upstream's tests | |
| # happen to be portable this release. | |
| # ⚠️ `.exe`, AND OMITTING IT IS NOT A WARNING BUT A HARD STOP: | |
| # | |
| # ninja: error: unknown target 'qemu-system-x86_64' | |
| # | |
| # The Unix legs name the target without a suffix and this one with, | |
| # because meson names the target after the file it produces. | |
| ninja -j"$(nproc)" qemu-system-x86_64.exe | |
| # ⚠️ ASSEMBLED BY HAND RATHER THAN BY `meson install`, AND THE REASON | |
| # GENERALISES BEYOND THIS HOST. | |
| # | |
| # `meson install --no-rebuild` stops at the first GENERATED file the | |
| # install list names but the emulator target did not produce: | |
| # | |
| # ERROR: File 'trace/trace-events-all' could not be found | |
| # | |
| # Naming that file makes it stop at the next one. Dropping | |
| # `--no-rebuild` instead makes meson rebuild everything, which brings | |
| # back the unit test that does not link on MinGW. Neither converges. | |
| # | |
| # ⭐ AND THE INSTALL TARGET IS NOT WHAT A PAYLOAD WANTS ANYWAY. | |
| # Measured on linux-x64: stripping took the emulator from 78.7 MB to | |
| # 27.0 MB while the whole installed prefix went only 392 MB to 343 MB, | |
| # so roughly 316 MB of `meson install`'s output is not the emulator. | |
| # A payload has to SELECT, and doing that here makes the contents | |
| # explicit rather than "whatever the install target happened to copy". | |
| # | |
| # What an x86_64 emulator needs: itself, and the firmware blobs in | |
| # `pc-bios` that its machine types load. Those are prebuilt files in | |
| # the tarball, not build outputs. | |
| mkdir -p ../out/bin ../out/share/qemu | |
| cp qemu-system-x86_64.exe ../out/bin/ | |
| cp -r "../qemu-${QEMU_VERSION}/pc-bios/." ../out/share/qemu/ | |
| # ⚠️ NOTHING IS DELETED HERE, AND RESISTING THAT IS THE POINT. The | |
| # tempting next step is to prune `keymaps` and the firmware | |
| # descriptors — but which of them a machine type loads at run time is | |
| # a question for the descriptor, and answering it by deleting | |
| # directories until something breaks is how a payload ends up missing | |
| # a blob on somebody else's machine. The whole `pc-bios` tree is | |
| # copied and its size is reported below. | |
| # ⚠️ THE DLL CLOSURE IS THE WINDOWS PACKAGING QUESTION, AND IT IS | |
| # ANSWERED BY MEASUREMENT RATHER THAN BY A LIST. The other four hosts | |
| # resolve their dependencies through an rpath inside the payload; a PE | |
| # has no such thing, so every MinGW runtime library the emulator loads | |
| # has to sit beside it. `ldd` under MSYS2 reports the closure; only | |
| # the ones from the UCRT64 prefix are ours to ship. | |
| ldd qemu-system-x86_64.exe \ | |
| | awk '/ucrt64/ {print $3}' \ | |
| | while read -r dll; do cp -n "$dll" ../out/bin/ 2>/dev/null || true; done | |
| echo "── the MinGW runtime the emulator needs ──" | |
| ls -1 ../out/bin/*.dll 2>/dev/null | wc -l | |
| # ── The check that the artifact is an emulator, not a file ──────────── | |
| # | |
| # ⭐ A BUILT BINARY IS NOT EVIDENCE; A BOOTED IMAGE IS. The index's own | |
| # e2e discipline is "assert the product, not the exit code", and the | |
| # cheapest product here is a multiboot image that prints and powers off. | |
| # It is built by the workflow rather than committed so that nothing in | |
| # this repository is a binary blob whose provenance has to be trusted. | |
| - name: The emulator boots a freestanding image | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| QEMU=$(find out -name 'qemu-system-x86_64*' -type f | head -1) | |
| test -n "$QEMU" || { find out -type f | head -40; echo "no emulator was produced"; exit 1; } | |
| "$QEMU" --version | head -1 | |
| # A minimal multiboot image. ⚠️ The a.out kludge (flag bit 16) rather | |
| # than plain ELF loading: QEMU's multiboot loader accepts only 32-bit | |
| # ELF images, and this one has to be assembled as 32-bit anyway, so | |
| # the kludge is not strictly needed here — it is used because the | |
| # real consumer (openarch's probe) is a 64-bit image that cannot be | |
| # loaded any other way, and a check that exercised a different path | |
| # would not be checking the same thing. | |
| cat > probe.S <<'ASM' | |
| .set MAGIC, 0x1BADB002 | |
| .set FLAGS, 0x00010000 | |
| .set CHECKSUM, -(MAGIC + FLAGS) | |
| .section .multiboot,"a" | |
| .align 4 | |
| mb: .long MAGIC | |
| .long FLAGS | |
| .long CHECKSUM | |
| .long mb | |
| .long __load_start | |
| .long __load_end | |
| .long __bss_end | |
| .long _start | |
| .code32 | |
| .section .text | |
| .globl _start | |
| _start: | |
| cli | |
| movl $msg, %esi | |
| 1: lodsb | |
| testb %al, %al | |
| je 2f | |
| movw $0x3F8, %dx | |
| outb %al, %dx | |
| jmp 1b | |
| 2: movw $0x604, %dx | |
| movw $0x2000, %ax | |
| outw %ax, %dx | |
| 3: hlt | |
| jmp 3b | |
| .section .rodata | |
| msg: .asciz "qemu-x86 probe ok\n" | |
| .section .bss | |
| .space 16 | |
| ASM | |
| cat > probe.ld <<'LD' | |
| ENTRY(_start) | |
| SECTIONS { | |
| __load_start = 0x100000; | |
| . = __load_start + SIZEOF_HEADERS; | |
| .multiboot : { KEEP(*(.multiboot)) } | |
| .text : { *(.text*) } | |
| .rodata : { *(.rodata*) } | |
| __load_end = .; | |
| .bss : { *(.bss*) __bss_end = .; } | |
| } | |
| LD | |
| # ⚠️ THE GATE IS "CAN THIS HOST LINK AN x86 ELF", AND MY FIRST VERSION | |
| # ASKED THE WRONG QUESTION. | |
| # | |
| # It checked for a 32-bit COMPILER and let the link run. On Windows | |
| # that passes — MinGW's `cc -m32` happily produces an object — and | |
| # then the link fails, because the linker beside it emits PE and | |
| # nothing else: | |
| # | |
| # ld.exe: unrecognised emulation mode: elf_i386 | |
| # Supported emulations: i386pep i386pe | |
| # | |
| # An arm64 or macOS runner fails earlier for a different reason. One | |
| # check covers all of them: ask the LINKER whether it knows the | |
| # emulation, which is the step that actually has to work. | |
| if ! ld -V 2>/dev/null | grep -q "elf_i386"; then | |
| echo "::notice::this host's linker emits no x86 ELF (supported: $(ld -V 2>/dev/null | tr '\n' ' ' | head -c 120)); the emulator's --version is the whole of the check on this leg" | |
| exit 0 | |
| fi | |
| cc -m32 -c -o probe.o probe.S -nostdlib 2>/dev/null \ | |
| || { echo "::notice::no 32-bit x86 assembler on this leg; --version is the check"; exit 0; } | |
| ld -m elf_i386 -T probe.ld -o probe.elf probe.o | |
| timeout -k 5 60 "$QEMU" -machine q35 -nographic -no-reboot -kernel probe.elf 2>&1 | tee run.log | |
| grep -q "qemu-x86 probe ok" run.log | |
| # ⚠️ WHAT A PAYLOAD WOULD ACTUALLY WEIGH, MEASURED RATHER THAN ESTIMATED. | |
| # The index's size discipline came from a real incident: a payload was | |
| # 34.81 MB and became 4.62 MB once stripped, and the upload failures that | |
| # had been blamed on the mirror were the size. | |
| - name: What the payload weighs, and what it needs | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| QEMU=$(find out -name 'qemu-system-x86_64*' -type f | head -1) | |
| echo "before stripping:"; ls -l "$QEMU"; du -sh out | |
| # ⚠️ STRIPPED, BECAUSE THE UNSTRIPPED FIGURE IS NOT A PAYLOAD SIZE AND | |
| # THE INDEX HAS PAID FOR THAT CONFUSION ONCE ALREADY. | |
| # | |
| # Measured on the first green legs: 392M (linux-x64), 389M | |
| # (linux-arm64), 341M/342M (darwin). Those are debug symbols, not an | |
| # emulator. A previous payload in this ecosystem was 34.81 MB and | |
| # became 4.62 MB once stripped, and the upload failures that had been | |
| # blamed on the mirror were the size. | |
| # | |
| # ⚠️ NOT ON macOS. Stripping a Mach-O invalidates its ad-hoc | |
| # signature, which is the rule this ecosystem's own release pipeline | |
| # follows; the darwin legs report the unstripped figure and say so. | |
| case "$RUNNER_OS" in | |
| Linux) | |
| find out -type f \( -name 'qemu-system-*' -o -name '*.so*' \) -exec strip --strip-unneeded {} + 2>/dev/null || true | |
| echo "after stripping:"; ls -l "$QEMU"; du -sh out | |
| readelf -d "$QEMU" | grep NEEDED || true ;; | |
| macOS) | |
| echo "(not stripped: it would invalidate the ad-hoc signature)" | |
| otool -L "$QEMU" || true ;; | |
| Windows) | |
| find out -type f -name '*.exe' -exec strip --strip-unneeded {} + 2>/dev/null || true | |
| echo "after stripping:"; ls -l "$QEMU"; du -sh out | |
| echo "(the DLL closure is measured in the packaging step, not here)" ;; | |
| esac | |
| # ⭐ WHAT IS ACTUALLY BIG, BECAUSE STRIPPING ANSWERED LESS THAN | |
| # EXPECTED. Measured on linux-x64: the emulator went 78.7 MB → 27.0 | |
| # MB, and the whole prefix went 392 MB → 343 MB. So roughly 316 MB is | |
| # not the emulator at all, and a payload that tarred this prefix would | |
| # ship it. The index's asset budget is a per-connection upload rate | |
| # measured at 0.012 MB/s into the CN mirror, with no multipart and no | |
| # resume — so what a payload SELECTS matters more here than what it | |
| # strips. | |
| # | |
| # Listed rather than pruned: which of these an emulator actually needs | |
| # at run time is a question for the descriptor, and answering it by | |
| # deleting directories until something breaks is how a payload ends up | |
| # missing a firmware blob on somebody else's machine. | |
| # ⭐ THE PAYLOAD MUST NOT REACH OUTSIDE ITSELF, ASSERTED RATHER THAN | |
| # ASSUMED. This is the check that would have failed on the first | |
| # build, and it fails loudly instead of at somebody's first run. | |
| echo "── every library the emulator names ──" | |
| case "$RUNNER_OS" in | |
| Linux) | |
| readelf -d out/bin/qemu-system-x86_64 | grep -E 'NEEDED|RUNPATH|RPATH' || true | |
| # Resolve the closure the way the loader will, from the payload. | |
| bad=$(ldd out/bin/qemu-system-x86_64 2>/dev/null \ | |
| | awk '/=> \// {print $3}' \ | |
| | grep -vE "^$PWD/out/" \ | |
| | grep -vE '/(libc|libm|libdl|librt|libpthread|libresolv|ld-linux[^ ]*)\.so' || true) | |
| if [ -n "$bad" ]; then | |
| echo "$bad" | |
| echo "::error::the payload links against libraries it does not carry; installing it would succeed and running it would not" | |
| exit 1 | |
| fi | |
| echo "resolved entirely inside the payload, plus core glibc" ;; | |
| macOS) | |
| otool -L out/bin/qemu-system-x86_64 | tail -n +2 | awk '{print $1}' | |
| bad=$(otool -L out/bin/qemu-system-x86_64 | tail -n +2 | awk '{print $1}' \ | |
| | grep -E '^/(opt|usr/local)/' || true) | |
| if [ -n "$bad" ]; then | |
| echo "$bad" | |
| echo "::error::the payload names absolute paths into this build machine's package manager; it would fail on any host without the same prefix" | |
| exit 1 | |
| fi | |
| echo "no build-machine paths remain" ;; | |
| Windows) | |
| echo "(the DLL closure sits beside the exe; counted above)" ;; | |
| esac | |
| echo "── the ten largest things under the prefix ──" | |
| du -sh out/* 2>/dev/null | sort -rh | head -10 | |
| du -sh out/share/* 2>/dev/null | sort -rh | head -10 | |
| # ── The publishable archive ──────────────────────────────────────── | |
| # | |
| # ⚠️ NAMED FOR THE INDEX, NOT FOR THIS REPOSITORY. `xim`'s descriptor | |
| # builds the URL from `${version}` and a platform/arch pair, so the file | |
| # name has to be derivable from those alone — `qemu-x86-<ver>-<host>.<ext>` | |
| # with the same `linux`/`darwin`/`win32` and `x64`/`arm64` spellings | |
| # `qemu-riscv` uses. A name only this workflow understands would force the | |
| # descriptor to special-case it. | |
| # | |
| # The sidecar carries the same hash the descriptor will quote, computed | |
| # here from the archive that is actually uploaded rather than recomputed | |
| # later from a download. | |
| - name: Pack the archive | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| V="${GITHUB_REF_NAME}" | |
| case "$V" in refs/*|main) V="${QEMU_VERSION}" ;; esac | |
| NAME="qemu-x86-${V}-${{ matrix.label }}" | |
| mkdir -p dist | |
| if [ "$RUNNER_OS" = "Windows" ]; then | |
| ( cd out && 7z a -tzip "../dist/${NAME}.zip" . > /dev/null ) | |
| F="dist/${NAME}.zip" | |
| else | |
| tar czf "dist/${NAME}.tar.gz" -C out . | |
| F="dist/${NAME}.tar.gz" | |
| fi | |
| # ⚠️ `shasum -a 256` rather than `sha256sum`: macOS has the former and | |
| # not the latter, and a fallback chain here is one line against a leg | |
| # that would otherwise fail after a 40-minute build. | |
| if command -v sha256sum > /dev/null; then sha256sum "$F" > "$F.sha256"; | |
| else shasum -a 256 "$F" > "$F.sha256"; fi | |
| ls -l dist/ | |
| cat "$F.sha256" | |
| - uses: actions/upload-artifact@v4 | |
| with: | |
| name: qemu-x86-${{ matrix.label }} | |
| path: dist | |
| retention-days: 14 | |
| # ⚠️ ONLY ON A TAG, AND ONLY AFTER THE CHECKS ABOVE. A leg that built but | |
| # did not produce a runnable emulator must not reach a release. | |
| - name: Publish to the release | |
| if: startsWith(github.ref, 'refs/tags/') | |
| uses: softprops/action-gh-release@v2 | |
| with: | |
| files: dist/* | |
| fail_on_unmatched_files: true |