Skip to content

Commit c3b2b6e

Browse files
committed
docs: the ecosystem side — payloads, adapters and board-support packages
The assessment of the finished book found a fourth reader the design had missed, and the measurement that found them is the sharpest in this batch: `xim:` payloads are named 84 times across 12 chapters. How to make one is explained nowhere. `xim-pkgindex` appears 17 times, all in the chapter about releasing mcpp. Every toolchain, device toolkit, shader compiler and emulator in this ecosystem is a payload. The documentation taught all of consuming them and none of producing them, and the omission was hidden by how often they are mentioned. Two more of the same shape: a `compat:` runtime adapter is named in four chapters and authored in none, and a board-support package is described from the consumer's side in 40 and 41 with no chapter on writing one. Three chapters, and `3x` becomes "extending mcpp and its ecosystem": 32 Authoring a Payload what an xim: descriptor must say, the two mirrors, `latest` as a reference, the exports that write a consumer's RPATH, and the `when` tier 33 Authoring a Runtime Adapter why a dlopen fails against a library that is plainly installed, and what a symlink farm plus runtime.library_dirs fixes 34 Authoring a Board-Support Package one package and two environments, the zero-libc tier, the machine table that is not a default, and run_exclusive Every fact is read out of a published package -- glslang.lua, compat.vulkan-runtime.lua, cortex-m-rt -- rather than out of the mechanism. The adapter chapter records a decision that is not a packaging choice: a proprietary driver's userspace is in ABI lockstep with a kernel module and its licence forbids redistribution, so it is a host capability and the adapter is how an artifact reaches it. An open driver is a payload and needs no adapter. Which answer applies is decided by the licence and the ABI, not by preference. Scenario 10 and a role-index row carry the new reader; the lookup index gains `xim:` and `compat:`.
1 parent b07c06c commit c3b2b6e

11 files changed

Lines changed: 808 additions & 2 deletions

.agents/docs/2026-09-08-the-documentation-as-a-book.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -395,3 +395,57 @@ belongs to `9x` or to SPEC-004. Both were named in §6 and neither is moved yet.
395395
reference chapter's own section list, so a key added to `04` and not indexed is
396396
invisible. The check is the same shape as rule 4 (every specification appears in
397397
every index) and is one loop.
398+
399+
---
400+
401+
## 12. The ecosystem side, which the first design missed
402+
403+
§5 designed the book for three readers — someone using mcpp, someone extending
404+
one project's build, and someone changing mcpp. It missed a fourth, and the
405+
measurement that found it is the sharpest in this record:
406+
407+
> `xim:` payloads are named **84 times across 12 chapters**. How to make one is
408+
> explained **nowhere**. `xim-pkgindex` appears 17 times, all of them in the
409+
> chapter about releasing mcpp itself.
410+
411+
Every toolchain, every device toolkit, every shader compiler and every emulator
412+
in this ecosystem is a payload. The documentation taught the whole of consuming
413+
them and none of producing them, and the omission was hidden by how often they
414+
are mentioned.
415+
416+
Two more of the same shape: a `compat:` runtime adapter — the layer that makes a
417+
host library reachable from an artifact on Linux — is named in four chapters and
418+
authored in none; and a board-support package is described from the consumer's
419+
side in `40` and `41` with no chapter on writing one.
420+
421+
### 12.1 Three chapters, and the band they complete
422+
423+
`3x` was "extending the build graph" and is now **extending mcpp and its
424+
ecosystem**: the two scales of one project's own step, then the three kinds of
425+
package that serve everyone else.
426+
427+
| # | chapter | reader | the one question | excludes |
428+
|---|---|---|---|---|
429+
| 32 | Authoring a Payload | someone packaging a tool or a prebuilt library | what is an `xim:` payload made of, and what must its descriptor say | source packages (11), host libraries (33) |
430+
| 33 | Authoring a Runtime Adapter | someone making a host-supplied library reachable | why an artifact cannot see a library that is installed, and what fixes it | anything redistributable, which is a payload (32) |
431+
| 34 | Authoring a Board-Support Package | someone bringing up a board | what does a BSP supply, and how does one package serve an emulator and hardware | using a BSP (40), the runner a consumer sees (41) |
432+
433+
Each is written from a real published package rather than from the mechanism:
434+
`xim-pkgindex/pkgs/g/glslang.lua`, `mcpp-index/pkgs/c/compat.vulkan-runtime.lua`,
435+
and `mcpplibs/cortex-m-rt`. Every fact in them was read out of those files.
436+
437+
### 12.2 The decision the adapter chapter records, which is not a packaging choice
438+
439+
A proprietary driver's userspace is in ABI lockstep with a kernel module and its
440+
licence forbids redistribution. Neither is solved by effort, so it is modelled
441+
as a **host capability** and the adapter is how an artifact reaches it. An open
442+
driver takes the other answer — it is a payload, and a machine using one needs
443+
no adapter. **Which answer applies is decided by the licence and the ABI, not by
444+
preference**, and that is the sentence a reader of scenario 10 leaves with.
445+
446+
### 12.3 What this changes about §5's claim
447+
448+
§5 said 24 chapters for a tool that is three tools is not too many. The count is
449+
31 now, and the reason is that the tool is four things rather than three: a
450+
build system, a package manager, a toolchain provisioner, **and an ecosystem
451+
other people publish into**. The fourth had no chapters at all.

docs/02-scenarios.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ people. Read only the scenario that matches; they do not build on each other.
2828
| [7](#7-compute-on-a-gpu-or-an-accelerator) | compute on a GPU or an accelerator | `examples/09-heterogeneous` |
2929
| [8](#8-graphics-rendering) | graphics rendering | `examples/10-graphics/offscreen` |
3030
| [9](#9-a-build-step-the-project-needs) | a build step the project needs, and sharing it | `examples/08-build-rules`, `12-a-new-device-language` |
31+
| [10](#10-packaging-a-tool-a-driver-or-a-board) | packaging a tool, a driver or a board for others | the descriptors in `xim-pkgindex` and `mcpp-index` |
3132

3233
## 1. A command-line tool or service
3334

@@ -253,6 +254,36 @@ step. Without that, editing the generator leaves every edge clean and the
253254
artifact keeps the bytes the previous generator produced — a green build over a
254255
stale result.
255256

257+
## 10. Packaging a tool, a driver or a board
258+
259+
**The situation.** Something other projects should be able to declare and get:
260+
a compiler, a shader compiler, an emulator, a host graphics driver, a board.
261+
262+
**What mcpp contributes.** A consumer declares it by name and gets a working
263+
program — the payload is installed, its libraries are on the artifact's search
264+
path, and the tier decides whether a build that never runs pays for it at all.
265+
266+
**The path.**
267+
268+
1. [32 — Authoring a Payload](32-authoring-a-payload.md) — a tool or a prebuilt
269+
library mcpp installs.
270+
2. [33 — Authoring a Runtime Adapter](33-authoring-an-adapter.md) — when the
271+
library belongs to the host and cannot be redistributed.
272+
3. [34 — Authoring a Board-Support Package](34-authoring-a-bsp.md) — a board,
273+
its memory map, and the way in.
274+
4. [31 — Authoring a Rule Package](31-authoring-a-rule-package.md) — if a build
275+
step drives the tool.
276+
277+
**Run.** The descriptors themselves: `xim-pkgindex/pkgs/g/glslang.lua` for a
278+
payload, `mcpp-index/pkgs/c/compat.vulkan-runtime.lua` for an adapter,
279+
`mcpplibs/cortex-m-rt` for a board.
280+
281+
**What surprises people.** Which answer applies is decided by the licence and
282+
the ABI, not by preference. A driver whose userspace is in lockstep with a
283+
kernel module and whose licence forbids redistribution **cannot** be a payload;
284+
it is a host capability, and the adapter is how an artifact reaches it. An open
285+
driver is a payload, and a machine using one needs no adapter at all.
286+
256287
## Current limitations
257288

258289
- The scenarios here are the ones with a runnable project or a published

docs/32-authoring-a-payload.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# 32 — Authoring a Payload
2+
3+
**Reader:** someone packaging a tool or a prebuilt library so that mcpp projects
4+
can declare it and mcpp installs it.
5+
6+
**The question this chapter answers:** what is an `xim:` payload made of, and
7+
what must its descriptor say for a consumer to name it and get a working
8+
program.
9+
10+
**Not here:** publishing a **source package** others `import`, which is
11+
[11 — Publishing a Library](11-publishing-a-library.md); making a **host**
12+
library reachable from an artifact, which is
13+
[33 — Authoring a Runtime Adapter](33-authoring-an-adapter.md); and consuming a
14+
payload, which is [23 — The Project Environment](23-the-project-environment.md).
15+
16+
Before: [31 — Authoring a Rule Package](31-authoring-a-rule-package.md), whose
17+
rules declare the payloads they drive. After:
18+
[33 — Authoring a Runtime Adapter](33-authoring-an-adapter.md).
19+
20+
## What a payload is
21+
22+
Everything mcpp installs and does not compile: a compiler, a shader compiler, a
23+
device toolkit, an emulator, a probe driver, a prebuilt C library. A project
24+
names one in `[xlings.workspace]`, or a rule package names it in
25+
`[feature-xlings.<f>]`, and mcpp provisions it before the build runs.
26+
27+
A payload lives in `xim-pkgindex` as one Lua file: a `package` table that
28+
describes it, and two functions that place and register it.
29+
30+
The shortest one that works:
31+
32+
```lua
33+
package = {
34+
spec = "2",
35+
name = "glslang",
36+
description = "Khronos reference GLSL/ESSL front end and validator",
37+
licenses = {"BSD-3-Clause", "Apache-2.0", "MIT"},
38+
type = "package",
39+
archs = {"x86_64"},
40+
41+
xpm = {
42+
linux = {
43+
["latest"] = { ref = "15.1.0" },
44+
["15.1.0"] = {
45+
url = {
46+
GLOBAL = "https://github.com/…/glslang-15.1.0-linux-x86_64.tar.gz",
47+
CN = "https://gitcode.com/…/glslang-15.1.0-linux-x86_64.tar.gz",
48+
},
49+
sha256 = "87167c9cb32f258addbedb607639b2c1f484c029ba91542a92f19ead21d65d13",
50+
},
51+
},
52+
},
53+
}
54+
55+
function install()
56+
local dir = pkginfo.install_dir()
57+
os.tryrm(dir)
58+
os.mv("glslang-15.1.0", dir)
59+
return true
60+
end
61+
62+
function config()
63+
xvm.add(package.name)
64+
return true
65+
end
66+
```
67+
68+
`install()` places the extracted tree where mcpp will look for it; `config()`
69+
registers what the payload offers. Everything else in this chapter is one of
70+
those two doing more.
71+
72+
## The four things a descriptor must get right
73+
74+
**One version, two URLs.** Every version carries a `GLOBAL` and a `CN` URL and
75+
one `sha256`. The two mirrors serve the same bytes; a consumer behind either
76+
mirror resolves the same hash, and a descriptor with one URL is unusable for
77+
half the ecosystem.
78+
79+
**`latest` is a reference, not a version.** `["latest"] = { ref = "15.1.0" }`.
80+
It is what a consumer gets when it names no version, and moving it is a
81+
deliberate act — a consumer pinned to `15.1.0` is unaffected.
82+
83+
**`archs` and the platform table are what a resolution reads.** A payload
84+
published only for `linux` and `x86_64` says so, and a consumer on another
85+
platform is refused by name rather than handed something that will not run.
86+
87+
**Dependencies are `xim:` addresses with floors.**
88+
89+
```lua
90+
deps = { "xim:gcc-runtime@>=15", "xim:glibc@>=2.38" },
91+
```
92+
93+
## Making the payload reachable
94+
95+
A payload that only unpacks is not usable. Three declarations turn a directory
96+
into something a build can consume.
97+
98+
**A program on the path.** `xvm.add(package.name)` registers the payload's
99+
`bin/` so mcpp can find the program by its bare name. A rule package should
100+
**name the program, not a path** — mcpp searches the `bin/` of every declared
101+
payload and then `PATH`, and can report exactly which directories it searched.
102+
103+
**Libraries a consumer will link or load.**
104+
105+
```lua
106+
exports = {
107+
runtime = { libdirs = { "lib" } },
108+
},
109+
```
110+
111+
`elfpatch` reads this from each dependency and writes the consumer's `RPATH`,
112+
which is what makes a stack of payloads resolve without anyone setting
113+
`LD_LIBRARY_PATH`.
114+
115+
**Headers, so a compiler in this environment can build against it.**
116+
`sysroot.declare_libs(...)` and the header declaration place the payload into
117+
the SubOS sysroot view. **Declared rather than copied** — xlings removes them
118+
with the package, and a copy would outlive its owner.
119+
120+
## The tier: when a payload is needed
121+
122+
```toml
123+
"xim:qemu-arm" = { version = "9.2.4-1", when = "run" }
124+
```
125+
126+
`when` is a second, independent gate beside the feature that selects the
127+
payload. The feature says **who** needs the tool; the tier says **when**. An
128+
emulator is needed to run and not to compile, so a CI job that builds firmware
129+
and never flashes it downloads nothing.
130+
131+
## Current limitations
132+
133+
- A payload is published to `xim-pkgindex`, which is a separate repository with
134+
its own review; nothing in mcpp publishes one.
135+
- `latest` and "the highest version in the table" are two different questions,
136+
and a consumer that wants the newest published version names `latest`.
137+
- A payload's own CI cannot verify that a consumer resolves it: that is what a
138+
sandbox check against the published descriptor is for, and it is the only
139+
thing that verifies the published bytes rather than the working tree.

docs/33-authoring-an-adapter.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# 33 — Authoring a Runtime Adapter
2+
3+
**Reader:** someone making a library the **host** supplies reachable from an
4+
mcpp-built artifact on Linux — a graphics driver, a Vulkan ICD, a proprietary
5+
runtime.
6+
7+
**The question this chapter answers:** why an artifact cannot see a library that
8+
is plainly installed on the machine, and what a package has to declare to fix
9+
it.
10+
11+
**Not here:** packaging a library mcpp installs, which is
12+
[32 — Authoring a Payload](32-authoring-a-payload.md) and is the right answer
13+
whenever the library *can* be redistributed; and the runtime contract's fields,
14+
which are [04 — The mcpp.toml Manifest](04-mcpp-toml.md) §`[runtime]`.
15+
16+
Before: [32 — Authoring a Payload](32-authoring-a-payload.md). After:
17+
[34 — Authoring a Board-Support Package](34-authoring-a-bsp.md).
18+
19+
## The failure this exists for
20+
21+
The library is installed. The loader finds its manifest. The `dlopen` fails
22+
anyway:
23+
24+
```
25+
DRIVER: Found the following files: /usr/share/vulkan/icd.d/lvp_icd.json …
26+
ERROR: libvulkan_lvp.so: cannot open shared object file
27+
```
28+
29+
The libraries are in `/usr/lib/x86_64-linux-gnu`. What cannot reach them is the
30+
**process**: an mcpp-built binary runs under mcpp's own glibc, with its own
31+
search path,
32+
33+
```
34+
interp: …/xpkgs/xim-x-glibc/2.39/lib64/ld-linux-x86-64.so.2
35+
rpath : …/xim-x-glibc/2.39/lib64:…/xim-x-gcc/…/lib64:$ORIGIN
36+
```
37+
38+
so a bare-soname `dlopen` from inside that process does not search the host's
39+
library path at all. Nothing is broken; the artifact is simply not looking
40+
there, which is the property that makes an mcpp build reproducible in the first
41+
place.
42+
43+
## What an adapter is
44+
45+
**A symlink farm plus the metadata that makes it reachable.** Nothing is
46+
vendored, nothing is redistributed, and the package carries no upstream bytes.
47+
`runtime.library_dirs` puts a package-owned directory of symlinks on the
48+
artifact's runtime search path, and the chain resolves.
49+
50+
A project declares the adapter as an ordinary dependency and does nothing else.
51+
52+
## Why the driver itself is not a package
53+
54+
A proprietary driver's userspace is in **ABI lockstep with a kernel module**,
55+
and its licence forbids redistribution. Neither is a packaging problem that
56+
effort solves, so such a driver is modelled as a **host capability** — something
57+
the machine either has or does not — and the adapter is how an artifact reaches
58+
it.
59+
60+
An **open** driver is a different case and takes the other answer: it is a
61+
payload (`xim:mesa-lavapipe` for the CPU, `xim:mesa` for AMD hardware), and a
62+
machine using one needs nothing from the farm. Since 2026.09.05 an adapter also
63+
prefers the payload when one is published and its symbol set covers the host
64+
copy, so what the farm actually records is proprietary userspace and packaging
65+
backlog.
66+
67+
## The three things an adapter gets wrong
68+
69+
**The pattern list must cover transitive dependencies.** The whole chain has to
70+
resolve through the same directory. Mesa's software rasteriser pulls in LLVM;
71+
an NVIDIA driver pulls its own family. Listing the ICD alone produces the same
72+
`cannot open shared object file` one level down.
73+
74+
**`libstdc++` belongs in the list, and it is not an oversight.** mcpp links
75+
libstdc++ **statically** — it is absent from a built binary's `NEEDED` — so a
76+
`dlopen`ed C++ driver has nothing to resolve against unless the host copy is
77+
provided here.
78+
79+
**Nothing may be required.** A machine with no such driver at all is a
80+
legitimate configuration, and every CI runner in this ecosystem is one. The farm
81+
is then empty and the program reports what it actually found. An adapter that
82+
errors on a missing host library turns a supported configuration into a build
83+
failure.
84+
85+
## Current limitations
86+
87+
- **Linux only, by construction.** macOS's dyld and the Windows PE loader have
88+
no equivalent layer, so a project targeting them declares no adapter.
89+
- An adapter cannot make a driver work that the machine does not have. It
90+
removes one obstacle — reachability — and reports the rest as absence.
91+
- The farm's contents are decided when the adapter is installed. A driver
92+
installed afterwards is not picked up until the adapter is reinstalled.

0 commit comments

Comments
 (0)