Repository navigation
compiler: WASM DWARF misses ~40% of code bytes with opts=1 #5736
Description
Build instructions are for the same binary as #5657 (comment)
$ tinygo version
tinygo version 0.43.0-dev-38222384 linux/arm64 (using go version go1.26.8 and LLVM version 22.1.4)
Built with flags -target=wasm -opt=1 -gc=conservative
result of running bindiff (was validated against llvm-dwarfdump by llm)
$ ls -lah ./local/netbird-debug.wasm
-rw-r--r-- 1 pato pato 68M sep 24 08:36 ./local/netbird-debug.wasm
$ bindiff -kind package profile ./local/netbird-debug.wasm | head -n 16
KIND NAME SIZE
package [unattributed] 44760644
package runtime 3677665
package [c] 2823351
package [type] 1790110
package github.com/netbirdio/netbird/shared/management/proto 1263801
package google.golang.org/protobuf/internal/impl 974844
package github.com/miekg/dns 939426
package github.com/google/gopacket/layers 703523
package github.com/netbirdio/netbird/client/internal 408850
package golang.zx2c4.com/wireguard/device 353801
package github.com/netbirdio/netbird/client/internal/peer 341440
package golang.org/x/crypto/ssh 338807
package github.com/pion/ice/v4 322116
package encoding/gob 306661
package google.golang.org/grpc 294868
LLM specifications:
Details
1. 9.25 MB of 23.2 MB of code (40%) has no line-table rows. Coverage was measured with llvm-dwarfdump --debug-line (13,970,898 of 23,225,047 bytes covered).
runtime.initAll(3.58 MB, the largest function) has no rows at all. My guess is that initializer code TinyGo re-emits intoinitAllcarries no debug locations. I haven't confirmed it.- 7,653 functions (5.9 MB) have no rows. These are mostly compiler-made wrappers (
$bound,$typeassert,interface:{…}.$invoke) plus 5,410 asyncify-generated functions (*.asyncifysuspendcatch.N.target, ~1 MB). - 3.4 MB are gaps inside functions that do have rows. Example: the DWARF entry for
crypto/internal/fips140/mlkem.kemDecapsspans the whole body (low_pc0x1375178 tohigh_pc0x138169a), but its line sequence covers only the middle. About 11 KB at entry and 7.7 KB at exit have no rows. - Likely cause: the wasm post-processing step.
builder/build.go:930runswasm-opt --asyncify -O1 -g. Asyncify adds unwind/rewind code at function entry and exit, and Binaryen keeps the existing DWARF but gives that new code no source location.
Impact: size and profiling tools can't attribute 40% of code to source. Stack traces and debuggers also lose locations in asyncify entry and exit paths.
2. .debug_pubnames is 12.4 MB, bigger than .debug_info (8.5 MB). It is a lookup index that current debuggers don't use (DWARF 5 replaced it), so dropping it would save about 18% of the module.
Suggested checks
- Rebuild with
-scheduler=noneto separate asyncify's share from the rest. - Check whether the code
interpre-emits intoinitAllgets a debug location. - Consider not emitting
.debug_pubnames/.debug_pubtypesfor wasm.
Activity
deadprogram commentedon Sep 25, 2026
Thanks for the detailed report @soypat. Here is what I found using a series of automated analyses.
The following is edited from the results:
runtime.initAllhad no debug info at all. It is created inbuilder/build.gowithout a subprogram, and interp emits leftover init code into it with no location. When it is small it gets inlined intoruntime.runand uses that call site. When it is too big to inline, as in your build, it has no line table rows. builder: add debug info to runtime.initAll #5746 fixes this. On a test program with a large non-inlinedinitAll, line coverage went from 9.2% to 99.9% before wasm-opt.Asyncify is the next biggest problem. On a small test program:
-scheduler=none-scheduler=asyncifyBinaryen adds the asyncify code without source locations, and
-O1alone also loses some. This part is in Binaryen, not TinyGo. Apparently this updatedid not improve that: binaryen: update to version 133 #5732
$boundwrapper functions have no line rows even before wasm-opt. This is a small separate gap in the compiler..debug_pubnamesis emitted because LLVM defaults to GDB tuning with DWARF 4 (DwarfCompileUnit.cpp,hasDwarfPubSections). Turning it off needsNameTableKind: Noneon the compile unit, which go-llvm does not expose yet. I intend to look it that as well after this.builder: add debug info to runtime.initAll (#5746)
builder: add debug info to runtime.initAll (tinygo-org#5746)
compiler: add debug info to synthetic wrapper functions
compiler: add debug info to synthetic wrapper functions
deadprogram commentedon Sep 28, 2026
A follow up on point 2 of my earlier comment. I redid the asyncify measurement using the current
devbranch with wasm-opt rebuilt from Binaryen 133. It turned out my earlier numbers were measured using 116. The following is edited from the results of another round of automated analyses:Binaryen 133 gives the same results as 116. On a small program using goroutines,
encoding/json, andfmt, built with-target=wasip1 -opt=1 -gc=conservative, every result was within 0.2 points.-O1--asyncify -O1These are code bytes covered by line table rows, counting only functions that have DWARF. wasi-libc and compiler-rt have none.
-O1by itself does not lose coverage. It drops about 2% of line rows, but the code stays covered by nearby rows. This corrects what I said before.The asyncify loss is all at function entry and exit, where it adds code to restore and save locals. Uncovered bytes at entry go from 3.2% to 9.3%, and at exit from 2.7% to 7.0%. Everything in the middle of functions is covered as it is expected to be.
Binaryen drops every line 0 row, even when no passes run (wasm-debug.cpp,
needToEmit). The test input has 21,756 line 0 rows and the output has none. The previous row then covers that code, so compiler generated code, including new asyncify code in the middle of a function, gets attributed to the wrong source line. :/This is a known limit of how Binaryen handles DWARF. It maps old instruction addresses to new ones instead of tracking source locations through passes, so any code a pass creates or rewrites loses its row. Asyncify also runs
flattenandcoalesce-locals, which the normal pipeline skips when DWARF is present (Asyncify.cpp). The Binaryen README says the DWARF mode is "not suitable for a fully optimized release build".deadprogram commentedon Sep 28, 2026
I opened WebAssembly/binaryen#9164 upstream in Binaryen for the line 0 problem from point 4 of my previous comment. It has a small C repro that does not involve TinyGo.
compiler: add debug info to synthetic wrapper functions
deadprogram commentedon Oct 1, 2026
Recent work in Binaryen looks interesting WebAssembly/binaryen#8964
soypat commentedon Oct 2, 2026
Let me know if there are concrete steps I can do to workaround the binaryen issue within TinyGo's flags :)
deadprogram commentedon Oct 8, 2026
@soypat I think we need to await the next Binaryen release with those changes, and then it might "just work better"?
deadprogram commentedon Oct 8, 2026
Actually @soypat I am mistaken. More is still required, please see WebAssembly/binaryen#9194 (comment)