translator: expand multi-architecture coverage from ecosystem discovery

[cpunion]

Summary

This draft expands typed Go assembler coverage across 386, amd64, ARM, ARM64, and wasm, and imports an independently discovered third-party assembly corpus. LLVM 22 is required; missing tools, translation failures, object-compilation failures, incomplete reports, and resource failures remain hard failures. The PR stays Draft until current-head external verification, CI, review, and coverage are green.

The rebased source also adds ARM64 signed-lane SMOV/SMOVW and BF16 BFDOT lowering, and fixes scalar SQSHLU code generation so LLVM 22 does not exhaust memory on one-lane intrinsic legalization. No fallback to another LLVM version is used.

Committed discovery funnel

The repository stores one bidirectional manifest and 256 module-hashed, semver-sorted JSONL record files. Negative scans remain recorded; retry failures are not marked complete. The separate assembly ledger records pending, passed, failed, and evidence-backed not_applicable states per matched exact version. cgo results stay outside this repository.

Stage	Count	Meaning
Go index records	12,665,430	154 continuous recorded ranges
Inspected exact module versions	802,019	Includes no-assembly results
Unique module paths	811,704	Includes paths with retryable outcomes
Assembly-bearing exact versions	4,783	Generic discovery, not a curated list
Assembly file paths	61,029	Retained for future architecture queries
Retryable discovery failures	13,631	Not counted as successful scans

The committed history cursor is 2026-03-18T00:17:13.411942Z (history_before), and the incremental cursor is 2026-09-22T22:43:08Z (incremental_since). The manifest derives these endpoints from the earliest/latest recorded ranges and validates continuity. Historical backfill is still incomplete.

Assembly-ledger state at this commit	Count
Matched candidates	4,783
Pending candidates	4,783
Passed	0
Failed	0
Evidence-backed not applicable	0
Reports included in the pushed ledger snapshot	0/32

Thus the pushed ledger snapshot does not claim that all discovered assembly compiles. Current CI reports are audited separately below without changing or interrupting the running source.

The last separately recorded remote-scanner snapshot (not refreshed during this CI repair) reports latest_records=729,533, cgo_records=791,846, cgo_modules=16,127, cgo_failures=1,729, and cgo_pending=6,228. These values are operational progress only and are not mixed into the repository ledger.

The generic scanner independently contains the reported issue libraries, including github.com/coder/websocket@v1.8.15 (llgo#2464), github.com/klauspost/compress@v1.20.0 (llgo#2552), github.com/tmthrgd/go-hex@v0.0.0-20190904060850-447a3041c3bc (llgo#2576), and github.com/tailscale/wireguard-go@v0.0.0-20260911194433-e3222a3340cd. Their presence in the ledger is discovery evidence, not a current-head pass result.

Official Go assembler coverage (Go 1.27.1)

Architecture	Official forms	Supported	Context-only	Unsupported	Parse errors
386	57	50	7	0	0
amd64	6,740	6,734	6	0	0
arm	499	455	44	0	0
arm64	2,048	1,980	68	0	0
wasm	120	0	120	0	0

The observed-form gate is green for all five architectures. The arm64 check also confirms all 98 toolchain-available required opcode families are encoder-defined, observed, and lowerable. Context-only forms require their source/function environment; they are not standalone lowering or runtime claims.

Verification

• Pushed head b93666f5 includes main@7cc8c0f and remains Draft. The contribution is pushed only to the cpunion fork. Current-head CI started on 2026-09-27 at 10:00 UTC. The latest checkpoint has 73 successful jobs, eight failures, six running and five queued. Let this run finish once: no push, cancellation or rerun while it is active. Verified development is being committed locally meanwhile.
• The preceding run's 11 failed jobs are repaired: unresolved ARM64 displacement macros now require header context instead of silently using zero; proved runtime-sized syscall frames receive actual dynamic backing with overflow checks and relocation of live stack aliases; Windows fixtures now handle executable suffixes, CRLF and native fake toolchain binaries.
• Preceding CI at 0d96d26: the last pre-push snapshot had 63 successful jobs and one failure. All 11 earlier failed jobs passed, including all 10 standard-library lanes and Windows. The Linux/Windows root suites, race, required cross-runtime, benchmark and coverage jobs passed. External shards 2, 6 and 7 passed; shard 3 exposed a single-character stack-slot parsing bug in gopkg.in/agiledragon/gomonkey.v2@v2.14.3, now fixed in the pushed head. These historical results are not current-head completion.
• Pushed repair a7e98cc accepts Go-valid single-character stack annotations such as n-8(SP). Retained red/green tests cover identifier/offset forms; both affected-library assembly files now translate and compile. The full root suite passed in 631 seconds, both CLI suites passed, and Go 1.20/1.27 focused checks, three-OS LLVM 22 objects, Darwin runtime and Linux/QEMU runtime pass.
• The same batch includes bounded scalable pool aliases, all 16 architectural vector lengths, complete typed vector-effect families, whole-vector/predicate pool loads, and NZCV preservation through explicitly modeled SVE families. Flags are never inferred from absence of GP writes. All focused pool tests, three-OS objects, Go 1.20 and required QEMU execution pass, including 18,864 guarded-read comparisons for the final flag-provenance change. The full root regression at the exact pushed source passed in 627.880 seconds; all root-module subpackages passed too.
• Windows root tests passed in 1,566.780 seconds with 89.6% statement coverage; both Windows CLI suites passed. Codecov patch coverage is green. These are distinct from complete external-corpus coverage.
• Local full root regression for the dynamic-stack repair passed in 665 seconds; both CLI suites, vet, Go 1.20 focused checks, Darwin execution, required Linux/QEMU, three-OS LLVM 22 objects and the five-architecture official gate passed. The strict benchmark compiled 184/184 files with zero N/A in 28 target-seconds plus one build-second.
• The pushed assembly-ledger snapshot still has 4,783 pending candidates. Current CI evidence is separately audited at the identical source below; it has not been pushed or relabeled as evidence for the newer local source. The current-head 32-shard replay, aggregate verification and review must pass before this PR is marked Ready.
• Local unpushed ARM64 work adds complete 512-bit indexed ZDUP forms, broadcast/CPY effects, constant CMP/CMN and masked-affine relations, and proved single-iteration/strided loops. The sequential-loop repair (7c07c82b), scalable count/address NZCV effects (0186a071) and aligned ORR/EOR relations (c433e78b) have focused full pool regressions, Go 1.20, three-OS LLVM 22 compilation and required Linux/QEMU evidence; scalar fixtures also execute on Darwin. The scalable fixture checks 132 cases at all 16 architectural vector lengths. At frozen 2265cb14, the full root suite and all root subpackages pass (1,059.348 seconds); both CLI full suites pass too. The ARM64 standard-library matrix compiles 950 IR files across 21 configurations. Complete shards 15/17 now both pass. The full required Linux/QEMU runtime selection also passes in 636.849 seconds. Latest strict benchmark: 184/184 files, zero N/A, 23 target-seconds plus two build-seconds. Local verification does not imply current CI completion.
• The SIMD repair checkpoint has two complete passing shard reports, preserved locally at 46ca52a7: 218 passed versions, 56 source N/A, 4,509 pending, zero failures and 5,154 successful target translations. Both canonical and mirrored simd@v1.21.1 pass all 120 applicable files on six targets (360 translations each, zero N/A). Overall complete/verified remain false because only 2/32 shards are included.
• A newer local batch (fe8f8e41, ffa77183, 72789337) prevents mixed network/resource/checksum failures from becoming source N/A, rejects such N/A evidence before ledger publication, and adds bounded retries for ordinary HTTP EOF. Real proxy-disconnection tests reproduce the old one-attempt failure and verify recovery or failure after at most three attempts. Production checksum verification remains enabled. The full corpus suite, race, Go 1.20, vet and Windows test-object compilation pass; all five diagnostic functions have 100% statement coverage. A fresh complete shard 21 replay and full root regression are running. This changed source has a newly bound ledger; the SIMD checkpoint and old CI evidence are not relabeled as current-source passes.

Current CI artifacts are validated at identical b93666f5 source content in a separate evidence worktree (local evidence commit 1965527d). Thirteen completed shards pass (1/2/3/4/6/7/8/10/12/13/16/18/19), and eight fail (0/5/9/11/14/15/17/21). All indices are zero-based. These are translation plus LLVM object-compilation results, not execution of every external library's tests:

Current b93666f5 CI assembly evidence	Count
Complete shard reports	21/32
Passing / failing shards	13 / 8
Passed exact versions	2,582
Source not applicable	550
Pinned invalid-source skips	4
Failed exact versions	9
Pending exact versions	1,638
Successful target translations	51,697

The 4,783-version accounting is complete as a progress snapshot, but both complete and verified remain false.

The nine current-CI failed candidates remain failures in this exact-source ledger: GoJIT searches raw function bytes for an embedded marker and enters native code mid-function; GopherJRE takes an out-of-directive-group code address and exchanges a custom register/stack contract with generated code; Sharkie switches native stacks and fabricates a code-relative return address; both case-distinct ContainerFS paths encounter 503/429 responses alongside obsolete dependency 404s after retries; both SIMD paths fail the same SVE byte-parsing file after 46/47 Darwin/ARM64 files compile; shard 21 also loses the exact Klaytn and lagpixellol/xray-core downloads to plain HTTP EOF at the checksum-proxy endpoint. The local unpushed fix now compiles the entire SVE byte file on Linux, Darwin and Windows, and its actual SVE2 parsing/formatting functions pass 3,120 runtime scenarios (130 parse and 65 format cases at each of 16 VLs) under Linux/QEMU. Both complete SIMD shards now pass locally as reported above; the newer proxy repair's shard replay is still running. Old CI evidence is not promoted to a new-source pass. Neither native-layout dependence nor infrastructure errors are silently marked passed or source N/A. Earlier 0d96d26 evidence remains preserved separately at b434d36e, never mixed into the current-source accounting.

[codecov]

Codecov Report

❌ Patch coverage is 94.29382% with 181 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
amd64_raw_broadcast_literal.go	97.86%	19 Missing ⚠️
amd64_lower_convert_fp16.go	88.65%	11 Missing ⚠️
amd64_raw_legacy_scalar_rip.go	69.69%	10 Missing ⚠️
amd64_lower_bf16_convert.go	87.32%	9 Missing ⚠️
amd64_raw_clmul.go	89.41%	9 Missing ⚠️
amd64_lower_vec.go	65.21%	8 Missing ⚠️
amd64_raw_packed_widen_conversion.go	88.05%	8 Missing ⚠️
amd64_raw_scaled_round.go	88.70%	7 Missing ⚠️
amd64_lower_convert_fp16_scalar.go	79.31%	6 Missing ⚠️
amd64_continuation.go	95.32%	5 Missing ⚠️
... and 29 more

📢 Thoughts on this report? Let us know!

[fennoai]

Review summary

Large PR (~175k insertions across 518 files). The bulk is generated discovery testdata (testdata/discovery/ledger/**) and highly repetitive AMD64/ARM instruction-lowering tables, both of which I skimmed rather than reviewed line-by-line. I focused on the high-signal changes: the new discovery/scan tooling (cmd/plan9asmcorpus/discovery.go, cmd/plan9asmdiscover, cmd/plan9asmll, cmd/plan9asmscan), internal/discoverymeta, and the core translate/parser changes.

Overall this is solid, defensively-written code: file handles are consistently closed on error paths, zip extraction blocks zip-slip (proxy-prefix + path.Clean + filepath.Rel + O_EXCL), external commands use fixed argv with escaped module paths, and remote fetches are size-bounded. Doc/AGENTS.md claims (flags, scripts, committed manifest counts, test names) were verified accurate.

One inline correctness/consistency finding below, plus a few non-blocking notes.

Non-blocking notes:

• Exponential build-tag search — cmd/plan9asmcorpus/discovery.go (findDiscoveryBuildTags). The search iterates 1 << len(customTags) masks, each doing filesystem-backed ctx.MatchFile calls, and the outer wantedCount loop re-walks the mask space ~len(customTags)+1 times. With the 16-tag cap this is bounded but can reach millions of MatchFile calls for a package near the limit. Usually fine (tags are typically 0–2); consider iterating masks once (prioritizing by popcount) if large-tag modules become common.

• Symlink escape in local include expansion — cmd/plan9asmll/main.go (pathWithinRoot / expandAsmIncludes). Containment is checked lexically via filepath.Rel; a #include target that is a symlink pointing outside the source root would pass the check and be read (no filepath.EvalSymlinks). Blast radius is contained (read-only, inlined into a discarded IR artifact, ephemeral CI against public modules; the zip extractor already rejects non-regular files), so this is defense-in-depth only. Resolving symlinks before the containment check would close the gap.

• Readability — cmd/plan9asmll/main.go:988 (and :1023). op == "JMP" || op == "B" || op == "RET" && len(ins.Args) == 1 relies on && binding tighter than ||. The current behavior is correct, but explicit parentheses around (op == "RET" && len(ins.Args) == 1) would guard against a future || term silently changing the grouping.