2026-09-04. The batch described in
2026-09-04-named-runners-and-the-universal-command-surface.md delivered four of
its six numbered batches in full. This note covers what is left: two items that
were in scope and were not finished, and two defects the batch surfaced and did
not close. Section 1 is the root-cause work for the fourth, because the plan for
it depends on the analysis.
Each section states what is ESTABLISHED by measurement separately from what is INFERRED, because one of these four has a signature and no stack trace.
Observed three times, in three separate fresh sandboxes, always on the mcpp run
that follows a mcpp build in the same project, on riscv64-none-elf and
aarch64-none-elf:
| observation | reading |
|---|---|
| CPU | 99.9% of one core, sustained (utime 165168 ticks over 1629s elapsed) |
| child processes | zero |
| system time | zero |
/proc/<pid>/syscall |
empty — the process is not in a syscall |
| the graph | 8 edges, every declared output present, no missing input, no future mtime |
| the same project on the host | builds and runs in about two seconds |
A ninja that is neither running commands nor making system calls, on a graph whose outputs all exist, is not working. It is looping in its own scheduler.
ninja's build loop is, in shape:
while (plan.more_to_do()) {
edge = plan.FindWork();
if (edge) { StartEdge(edge); continue; }
if (commands_running) WaitForCommand(); // a syscall
else break;
}
Zero children and zero system time exclude both branches that leave user space.
What remains is a loop in which more_to_do() stays true and FindWork() keeps
returning an edge whose completion never decrements the outstanding count. For
that to burn CPU without spawning anything, the edge must be one ninja finishes
WITHOUT running a command.
In this graph exactly one edge finishes without a command: the phony.
build _mcpp_staged_cache : phony obj/…/riscv_virt_rt.m.o pcm.cache/….pcm
build obj/main.cpp.ddi : cxx_scan …/src/main.cpp || _mcpp_staged_cache
build obj/main.o : cxx_object …/src/main.cpp | obj/main.cpp.ddi.dd \
|| _mcpp_staged_cache
Its two inputs are stage_file edges, and stage_file is the one rule in mcpp's
generated manifest whose command is REGULARLY SKIPPED while succeeding:
mcpp stage does not write when the destination is already equivalent, which is
the case the rule's restat = 1 exists to exploit — a skipped stage must not
dirty every importer of the staged BMI.
So the shape present here is: restat edges whose outputs do not change, feeding a phony, consumed as an order-only dependency. That is the narrowest description of the suspect, and every part of it is mcpp's own construction rather than something a user wrote.
That this shape CAUSES the loop is inference, not measurement. Three attempts to obtain a stack all failed, and the failure of each is itself informative:
gdb -pis refused:ptrace_scope = 1, and the spinning ninja is not a descendant of any shell that can attach to it.perf record -pis refused:perf_event_paranoid = 4.- Wrapping ninja so that gdb is its PARENT — which is legal under yama level 1 — makes the spin stop happening. Under gdb the same invocation ran seven subprocesses and exited normally.
The last of those is the important one. It says the trigger is timing- or environment-sensitive rather than a pure function of the graph, which is consistent with a scheduler bookkeeping race and inconsistent with "this manifest always loops".
Two further measurements bound the suspect from the other side:
- Both ninja binaries, run 60 times each over the completed graph by hand, never hung and never took longer than 500ms. A no-op run of this manifest is not sufficient to trigger it.
- The same project, built and then run on the host, completes in two seconds,
and its
build.ninjacarries the same twostage_fileedges and the same phony. The graph shape alone is not sufficient either.
mcpp does not kill the ninja it spawned. Every timeout-terminated mcpp run left an orphan spinning at 100% of a core. One of them outlived the removal
of the entire sandbox it belonged to; its working directory read (deleted) and
it was still burning a core half an hour later.
This is deterministic, has nothing to do with the spin's cause, and makes the
spin far more expensive than it would otherwise be: any CI that wraps mcpp in
timeout leaks a busy core per timeout.
xim:ninja@1.12.1 does not name one artefact. The descriptor promises
xlings-res's glibc-static build (ninja-1.12.1-linux-x86_64.tar.gz); this host
has that one, at 2202320 bytes, statically linked. Several sandboxes have a
273768-byte dynamically linked binary built with GCC 4.8.5 on Red Hat — the
shape and build date (2024-05-11, ninja 1.12.1's release day) of upstream's
official ninja-linux.zip.
Both answer --version with 1.12.1, and their SHA-256 differ. Neither hung in
the 120-run comparison above, so this is not offered as the cause. It is offered
as a fact that has to be false before any conclusion about "ninja 1.12.1
behaves like X" can be trusted.
The plan expected the first 32-bit backend to surface a width assumption:
arch_pte_make_leaf returns arch_u64, and nothing had ever asked whether that
is right on a machine whose page-table entry is 32 bits wide.
Cortex-M arrived and did not settle it. M-profile has an MPU and no MMU, so the
backend's pte_impl.cpp implements the group as functions that exist and refuse,
and the package withholds the openarch:address-space capability so that a
consumer needing an address space is refused at resolution rather than at run
time. That is the correct design for that machine, and it means the width
assumption was never executed.
The machine that settles it is 32-bit AND has paging: ARMv7-A.
Add backends/armv7a implementing all four groups, with the pte group backed by
the ARMv7-A short-descriptor format, whose second-level entries are 32 bits.
The deliverable is not the backend. The deliverable is the answer to one
question: can a 32-bit page-table entry be expressed through an interface that
types it as arch_u64?
- If yes — the value fits, the extra bits are ignored, and no caller stores the return into a machine-width slot — then record that the interface is width-independent, with the ARMv7-A implementation as the evidence, and the question is closed rather than open.
- If no — some caller or some table needs the machine's own width — then the interface changes now, while openarch is at 0.x and four backends exist to change together, rather than after a fifth consumer has depended on it.
Either answer is worth the work; only the second changes the interface.
backends/armv7a/withcpu_impl,context,trap_impl,pte_impl.- mcpp already has the target rows (
armv7a-none-eabi,armv7a-none-eabihf), the matrix cells and e2e336_armv7a_builds_and_boots.sh, so the toolchain side needs nothing. - qemu
-M virt -cpu cortex-a15runs it; the existing e2e already boots an ARMv7-A image. - Release openarch 0.9.0. The
arch_trap_switchsignature is already frozen by four implementations, so this backend implements a settled interface.
The one assertion that must exist: a test that builds a leaf entry, reads it back, and compares against a hand-written 32-bit descriptor. Without it the backend can be written to whatever the interface says and the width question stays unasked a second time.
Independent of everything else here; blocks nothing.
riscv-virt-rt and aarch64-virt-rt declare
[xlings.workspace]
"xim:qemu-riscv" = "9.2.4-1"which is the untiered form, meaning ToolWhen::Always: provisioned by every verb
that builds. A CI job that compiles firmware and never runs it downloads a
33 MB emulator. cortex-m-rt declares the tiered form and does not.
The engine side of batch 2 shipped all four values (build, run, dev, and
the implicit always); only the adoption is incomplete, and it is incomplete in
the two packages that most obviously want run.
[xlings.workspace]
"xim:qemu-riscv" = { version = "9.2.4-1", when = "run" }Release riscv-virt-rt 0.7.1 and aarch64-virt-rt 0.2.1, one index entry each.
The assertion goes in each package's CI and follows the criterion batch 2 already
settled (§14.4 of the plan): test what would be installed, not what was
installed. With MCPP_NO_AUTO_INSTALL set, a build must report that it needs
no emulator and a run must report that it needs exactly one. That runs in seconds
and needs no download, so it can sit in the ordinary build job rather than in a
job that has an emulator.
Both halves are load-bearing. "The build does not install qemu" is also true when the package is broken and installs nothing at all; the pair distinguishes them.
Low risk, small, and it removes an inconsistency inside one batch.
Both spawn sites in execute.cppm end in return rc == 0 ? 0 : 1, and the
comment states the intent: 2 means "could not start", 1 means "ran and failed",
distinct from each other on purpose.
Measured consequences:
- Hosted: a program whose
mainreturns 3 makesmcpp runexit 1. - Freestanding: the same image under qemu directly exits 3; through
mcpp runit exits 1.
So a program's own status is not observable through the command this ecosystem
tells people to type. That matters more after this batch than before it, because
the batch's claim is that running on a device is like running hosted, and a
hosted run that cannot report a status is not that.
A. Pass the child's status through; move mcpp's own failures to 125–127.
Borrows the convention env, timeout and nice already use and that shells
document: 125 = the tool itself failed, 126 = found but not executable, 127 = not
found. "Could not start" then lands on 126/127 by meaning rather than by
allocation, and every value below 125 belongs to the program.
Cost: the current 2 stops appearing from these two sites, which is a
compatibility change for anything that tests for it. Scripts testing != 0 are
unaffected.
B. Pass the child's status through; keep 2 for mcpp's own failures. Smaller change, but a program that exits 2 becomes indistinguishable from a launcher failure — reintroducing exactly the ambiguity the current code was written to avoid.
C. Keep the current contract; expose the real status in the machine
interface. mcpp run --format json (or the existing NDJSON path) carries
exit_code, and the human exit stays 0/1/2.
Cost: the common case still cannot be scripted with $?, which is what people
actually do.
Recommendation: A. It is the only one of the three in which the obvious thing a user types produces the obvious thing they expect, and the convention it borrows is old enough that 126/127 will read correctly to anyone who meets them.
- Both sites in
execute.cppm, plusmcpp test's aggregation, which reports per-test status already and would report the true code. docs/11-machine-output.mdand its Chinese counterpart: the exit-status table is part of the machine interface contract and must state the new allocation. A published contract page that describes the old one is worse than none.- One e2e asserting a hosted program returning 3 gives 3, and one asserting a missing runner still gives 127 rather than 3.
This is the one item in this note that needs a decision before work starts. The other three have a right answer; this one has a trade-off.
Split into two, because one half is established and the other is not.
mcpp must terminate the ninja it spawned. On POSIX, put the child in its own
process group and signal the group; on Windows, a job object with
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE (the console-control-event route is already
known to be wrong here — it reaches the whole console, which once took down an
entire e2e runner with no FAIL: line to explain it).
The criterion is not "the code calls kill". It is: after mcpp is terminated mid-build, no ninja remains. An e2e that starts a slow build, kills mcpp, waits, and then asserts that no ninja process survives — asserting on process state, not on a log line.
This is worth doing whatever causes the spin, and it converts the spin from "leaks a core forever" into "one build hangs until its timeout".
The one instrument that works on a process nobody can attach to is ninja's own
-d explain, which prints its dirtiness reasoning as it goes. A spinning ninja
that is re-deciding the same edge will emit a repeating line naming that edge,
which either confirms §1.2's suspect or names a different one.
Concretely: an MCPP_NINJA_DEBUG escape hatch that appends -d explain and
captures the output, used to re-run the verification until it hangs again. The
sandbox reproduces it about once per run, so a handful of runs suffices.
Second instrument, if the machine's owner is willing: ptrace_scope = 0 on the
development box makes gdb -p work on the live spinner, which answers it in one
attempt.
The phony exists to give consumers one order-only token to depend on instead of listing every staged output. If it is implicated, the direct alternative is to drop it and attach the staged outputs to each consumer's order-only list directly. That costs a longer edge line per consumer and removes from the plan the only edge that completes without a command.
Removing restat = 1 from stage_file is the other obvious candidate and is
the WORSE one: the rule's comment states what restat buys — a skipped stage must
not dirty every importer of the staged BMI — and that benefit is real and
measured. Do not trade it away for a defect whose cause is not yet established.
Make xim:ninja@1.12.1 name one artefact. Whichever build is intended, the
descriptor and the payload should agree, and a machine that already has the other
one should be able to tell. A single-line assertion — the installed binary's
SHA-256 against the descriptor's — would have made §1.5 a report rather than a
discovery.
| # | item | depends on | needs a decision |
|---|---|---|---|
| 5.1 | mcpp kills its ninja | nothing | no |
| 3 | two boards adopt when = "run" |
nothing | no |
| 5.4 | one artefact per ninja version | nothing | no |
| 2 | openarch ARMv7-A backend | nothing | no |
| 4 | mcpp run exit status |
— | yes, before work starts |
| 5.2 | instrument the spin | 5.1 landing first is convenient, not required | no |
| 5.3 | change the staging graph | 5.2 | not until 5.2 answers |
Everything except item 4 can start without further input. Item 4 is a compatibility decision about a published contract, and the recommendation in §4.2 is a recommendation rather than a conclusion.
All four items were implemented, one pull request per repository. What follows records what was measured, including two places where a first attempt passed without testing anything.
| gap | delivered | where |
|---|---|---|
| 1 | openarch 0.9.0: ARMv7-A backend, arch_pte_entry_bytes() |
openarch #9 |
| 2 | when = "run" on both boards, with the pair asserted |
riscv-virt-rt #5, aarch64-virt-rt #3 |
| 3 | mcpp run reports the program's status |
mcpp #555 |
| 4 | mcpp kills its child; MCPP_NINJA_DEBUG; ninja pinned by checksum |
mcpp #555, xim-pkgindex #756 |
The width question is settled. ARMv7-A's short-descriptor entry is 32 bits and
FITS in the arch_u64 the interface carries, so the carrier did not change —
but nothing could ask how wide the storage is, and a kernel sizing a table from
sizeof(arch::pte) builds one twice as large as the hardware walks. Nothing
diagnoses that: the table is well formed, the entries are correct, and the
machine reads the gaps.
arch_pte_entry_bytes() is the addition, implemented by all five backends. The
header's own claim — "a page-table entry is 64 bits on every machine here" — was
corrected rather than left standing.
Asserted twice, because one place cannot cover both halves: tests/pte_encoding
compares the encoder against a descriptor written out from the architecture
manual, and a CI job boots a program on qemu -M virt -cpu cortex-a15 that
reads the width back THROUGH THE ABI and exits 0 only if it is 4. Measured:
armv7a pte width 4, entry fits 32 bits, exit 0.
openarch:preemption is withheld on this machine. The resumption address lives
on the SVC stack rather than in a register, so a trap-time switch is well
defined only if the resumed context was suspended the same way — a real design,
and not one this backend has measured. Withholding it refuses a consumer by
name at resolution, which is the mechanism Cortex-M already uses for
openarch:address-space.
The tier itself was two lines. The assertion was not: on a machine where the
emulator is already installed, MCPP_NO_AUTO_INSTALL has nothing to report, so
BOTH halves read empty and the step passes without testing anything. Measured in
exactly that state before the step was moved to an isolated MCPP_HOME.
With one, the pair is decisive and needs no download: a build names the emulator
zero times, and a run refuses with declared but not provisioned naming
xim:qemu-riscv@9.2.4-1. CI reports ok build asks for no emulator, run asks for xim:qemu-riscv on both boards.
mcpp run passes the child's status through. Spawn refusals move to 125-127,
the band env, timeout and nice already use. mcpp's own refusals BEFORE a
spawn — no binary target, no runner declared, a runner not on PATH — keep exit
2, which is what every other mcpp command uses, so no existing script that
tests for it changes meaning. Only the case where the program never ran moved.
mcpp test is unchanged at 0/1: it aggregates many programs and has no single
status to pass through.
Fixing run_exec alone left the orphan in place, because a full build spawns
ninja through capture_exec. The A/B caught it; reading the code had not.
THE TEST SIGNALS mcpp's PID ALONE. timeout and Ctrl-C both signal the process
group and reached ninja even before the fix, so a test built on timeout passes
either way — the first version of this test did exactly that. The second version
used 200 parallel translation units and passed against a defective build because
a 32-core host finished the batch inside the ten-second window. --jobs 1 makes
the window deterministic by construction. The final test FAILS against the
released 2026.9.4.2, naming the orphan and its (deleted) working directory.
The spin's cause remains unestablished and §1.3 stands. MCPP_NINJA_DEBUG
appends -d explain to both ninja launches so the next occurrence can be read
rather than guessed at.
xim:ninja@1.12.1 now carries a checksum. While computing it, a stronger fact
than §1.5's appeared: this machine holds ninja binaries at THREE distinct sizes
under that one version, and the twelve copies at 290152 bytes have twelve
DIFFERENT SHA-256 values.
Same size and different content is the signature of in-place patching, which xlings does to installed binaries. So an installed ninja cannot be compared against a published checksum at all — only the downloaded archive can, which is what the checksum added here covers. A clean-home install was verified to fetch, verify and produce the 2202320-byte static binary the descriptor describes.