ci-macos xcode-27: ld64.lld cannot parse arm64e.x1 in either available SDK (upstream, tracked) #669

Summary

Two macOS CI jobs are red on every run against the xcode-27 runner image, and will stay red until upstream LLVM ships a release carrying a specific ld64.lld fix. This is understood and tracked here; it is not a regression in this repository, and it is not the SDK-selection bug xim-pkgindex#858 fixed (see "What this is NOT" below).

Affected jobs (both only on the xcode-27 matrix leg; macos-15 is unaffected):

• ci-macos.yml → "macOS ARM64 — xlings LLVM end-to-end (xcode-27)"
• ci-macos-e2e.yml → "e2e suite (macOS ARM64, self-host, xcode-27)"

Exact failure signature (identical in both jobs):

ld64.lld: error: could not load TAPI file at
  <SDK>/usr/lib/libSystem.tbd: malformed file
<SDK>/usr/lib/libSystem.tbd:4:20: error: unknown architecture
                   arm64e.x1-macos, arm64e.x1-maccatalyst ]
                                     ^~~~~~~~~~~~~~~

(libc++.tbd shows the same error too.) ld64.lld (bundled with the xim:llvm@22.1.8 package) cannot parse a .tbd file that lists the arm64e.x1 architecture. Followed by a long list of undefined symbol errors, which are a consequence of the .tbd files failing to load, not a separate problem.

The surprising part: it is not just the Command Line Tools SDK

The original diagnosis (recorded elsewhere, and the reason xim-pkgindex#858 exists) was that only /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk was affected, and that selecting the Xcode.app SDK instead (via xcrun --show-sdk-path) would avoid it — mirroring mcpp's own fix in #665. That fix is correct and necessary, but on the xcode-27-arm64/20260912.0186 image (the "Release Candidate" build, still being served under the xcode-27 label alongside an older image) it is not sufficient, because both SDKs available on that image carry the same broken .tbd format.

Evidence — correction (2026-09-18): this is not "the runner image moved between yesterday and today." GitHub was running a gradual rollout, serving two xcode-27-arm64 images simultaneously on 9-17: 20260907.0173 (bundling Xcode_27_beta_6.app) and 20260912.0186 (bundling Xcode_27_Release_Candidate.app). Which image a given job landed on was runner-assignment luck, not a function of when it ran — one of the three runs below is older in image terms than a run that started earlier the same day. This is measured directly from each job's own "Runner Image" log group, not inferred:

(ci-macos-e2e's own last-green data point, run 35213851695 / job 105177519935, 2026-09-17 11:05, is the ci-macos-e2e companion of the same push as the failed ci-macos run directly above — same commit, same minute, and it landed on the other image and passed. Two jobs from one push, on two different images.)

So the true picture: image 20260912.0186 (Release Candidate) is the one with the broken SDKs, and it has coexisted with the older, working 20260907.0173 (beta 6) image on the xcode-27 label since at least 2026-09-17 11:05 — which is why the failure looked intermittent rather than a clean before/after split.

The earlier attribution — that mcpp#665 fixed the original red — was wrong, and has been corrected in our own 0.13 record as well. Here's why it doesn't hold up, mechanically, independent of the image mix-up:

clang++.cfg's --sysroot (present the whole time before xim-pkgindex#858, always CommandLineTools) wins over a command-line -isysroot for what the linker opens — this is the same mechanism xim-pkgindex#858's own fix relies on. mcpp#665 added -isysroot $(xcrun --show-sdk-path) to the raw-compile CI steps, which changes header search at compile time, but never the linker's sysroot while a --sysroot-writing cfg is in play. So for every run before xim-pkgindex#858 merged (2026-09-18), including run 35219991759 (the green run right after #665 merged), the actual link always went through whatever clang++.cfg said — Command Line Tools — regardless of what -isysroot on the command line said. Run 35219991759 went green because it happened to land on the 20260907.0173 image, where Command Line Tools' SDK was still fine; #665's -isysroot flag was mechanically inert for the link outcome. Its own PR title even names the image it was responding to ("xcode-27 image 20260912") — the fix was aimed at the Release Candidate image, but a compile-time flag can't fix a link-time SDK choice.

mcpp#665 remains correct and worth keeping on its own merits — using the SDK xcrun reports instead of assuming one is the right general strategy, and it does control header search correctly — but it solved a different, narrower problem than the record credited it with, and it did not and could not have fixed this link failure by itself.

Does the choice of SDK matter at all on the Release Candidate image?

No. This was checked directly, not assumed: on image 20260912.0186, both available SDKs were exercised and both fail with the identical arm64e.x1 signature —

• Command Line Tools' SDK: runs 35288886139 and 35213851816 (both pre-xim-pkgindex#858, clang++.cfg pointing at CLT), fails.
• The Release Candidate's own bundled Xcode SDK: runs 35291934706 / 35291934667 attempt 3 (post-xim-pkgindex#858, clang++.cfg and xcrun both pointing at the Xcode SDK), fails identically.

So on this image, no --sysroot/-isysroot choice avoids the failure — there is no third SDK to try. Please don't spend time on another sysroot variant; the constraint is upstream ld64.lld, not which path is selected.

This shows up in two different code paths in the same CI run, not just one:

• The raw xlings-installed clang++ (ci-macos, using the now-fixed clang++.cfg) fails this way.
• mcpp's own self-hosted build (mcpp build, in the ci-macos-e2e job) — which computes its own --sysroot from mcpp::platform::macos::sdk_path() (modules/platform/src/macos/macos.cppm, wired into the link model in modules/toolchain-model/src/linkmodel.cppm) — fails identically, same file, same architecture list. Both paths currently resolve to the same SDK (the Xcode Release Candidate one), so this is corroboration that that SDK is the broken one, not proof that the two paths are mechanically independent of each other.

What this is NOT

xim-pkgindex#858 (openxlings/xim-pkgindex#858) fixed a real, separate bug and is not what is failing here. Before that PR, the xlings llvm package's pkgs/l/llvm.lua hardcoded /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk as its first candidate whenever SDKROOT was unset, regardless of what xcode-select actually pointed at. That PR changed it to ask xcrun --show-sdk-path first (the same canonical probe mcpp's own sdk_path() uses), falling back to the fixed candidate list only when xcrun can't answer. It is merged (openxlings/xim-pkgindex commit a96ab1b881f43de0aa43d5969581ad05aabef224), verified safe (a Lua harness compared its output against the unmodified function across xcrun-absent / empty / non-zero / stale-path / SDKROOT-set cases — all byte-identical to the old behavior; only the case where xcrun succeeds changes, which is the fix doing its job), and confirmed live on real macOS CI. The mechanism of that fix is directly visible working in the failing job's own log — clang++.cfg now reads --sysroot=/Applications/Xcode_27_Release_Candidate.app/..., not the Command Line Tools path. It just isn't enough on the 20260912.0186 (Release Candidate) image, because its own bundled Xcode SDK carries the identical arm64e.x1 defect that Command Line Tools does — see "Does the choice of SDK matter at all on the Release Candidate image?" below. Please don't re-diagnose the SDK-selection question; it's closed, and the remaining failure is upstream.

Upstream state (checked directly against llvm/llvm-project, with dates)

• The fix: llvm/llvm-project#222721, "[ld64.lld, llvm-otool] Minimal arm64e.x1 support", merged to main at commit b8007a8e4020b8bca2b12e941660e10bf5bf6716 on 2026-09-11.
• Not in 22.1.8 (what xlings/mcpp use today): tagged 2026-06-16, three months before the fix existed. release/22.x's HEAD commit is dated 2026-06-15 — the branch has taken nothing since 22.1.8 was cut, and there is no backport PR targeting it for this fix. That series is effectively closed.
• Not in any released 23.x point release either. 23.1.0 tagged 2026-08-25, 23.1.1 tagged 2026-09-08 — both before the fix merged to main.
• A backport exists: llvm/llvm-project#224185, "release/23.x: [ld64.lld, llvm-otool, lldb] Minimal arm64e.x1 support", opened 2026-09-17, approved by a reviewer the same day, milestone "LLVM 23.x Release" (no specific point-release number attached). As of now it is still open and unmerged. No 23.1.2 tag exists, and no published schedule for one was found.

What closing this requires, on our side

Once (if) a release carrying the backport ships, this is not a one-line version bump in the xlings llvm package:

1. It's a major-version jump, 22.x → 23.x, not a same-series point bump — worth its own scoping rather than a routine bump. One reason to take that seriously: the backport author noted in the PR thread that "the abi change is true" — the fix is not a pure bugfix, it carries an ABI-relevant change.
2. The macOS rows in pkgs/l/llvm.lua are carved sub-packages, not raw upstream downloads. They're built by .agents/tools/build-llvm-subpkg.sh from upstream's official release tarball, then re-uploaded to xlings-res/llvm GitHub releases (mirrored to gitcode) before a version+URL+sha256 entry can be added to the index. So even once 23.1.2 (or whatever the release turns out to be) exists upstream, there's a packaging and re-hosting step in between, not just editing a table.

Condition for closing this issue

Both of:

1. xlings ships (packages and indexes) an LLVM release that carries the arm64e.x1 backport, for macOS arm64.
2. Both jobs above go green on a rerun of the xcode-27 leg using that LLVM.

Decision on the interim (for the record)

Considered and rejected: continue-on-error: true / marking the job as a soft "expected failure" (turns a real signal into a warning — the day this failure stops being the arm64e.x1 issue and becomes an actual regression, nobody would notice), and pinning the job to an older runner image (stops testing against the newest Xcode entirely, hiding a real incompatibility rather than surfacing one). Decision: leave both jobs red, with this issue as the one-click answer for anyone who lands on the failing PR.