winpin: Pinball Soccer '98

[avanturist888]

First of all, thank you for this project. I found it through your blog post and it is easily the most fun thing I have read about in a long time. Watching an exe turn into Rust that just runs is quite something. After spending a while in the code I decided the best way to say thanks was to actually pitch in rather than watch from the sidelines, so I picked a game and worked until it ran.

This adds a new target: Pinball Soccer '98, a 1998 Windows pinball game. It gets to actual gameplay, both natively and in the browser.

Per your comment, the generated code is not checked in. out/winpin is just the crate scaffolding plus a translate.sh recipe, so it builds after translate.sh winpin if you own a copy of the game, and the diff here is about 6k lines of hand-written code.

tc

Most of it already worked. The gaps were all around finding code:

• MSVC switch tables indexed backwards from the displacement (sub ecx, 4; jb; jmp [ecx*4 + table])
• Taking the table length from a preceding and mask, so a table whose first slot is padding does not cut the scan short
• Scanning for function prologues in the gaps between known blocks
• A feedback loop: the runtime appends addresses it could not resolve to a file (THESEUS_MISSING_ADDRS), which you feed back in with --entry-points-file

Together those take .text coverage on this exe from 6% to 93.9% (24114 blocks).

Generated output is now split into part_NN.rs files of roughly 1MB, which took a cold build of this target from 60s to 24s.

winapi

New files: dinput (keyboard and mouse), the mmio family in winmm, ole32 and msacm32 stubs, and a shared input state in user32 that both dinput and the message pump read from.

Filled in: a software mixer for dsound (resampling, volume, pan), colorkey blits and palette handling in ddraw, LoadLibrary/GetProcAddress backed by a module registry, and a fair amount of kernel32's file and NLS surface.

runtime

indirect gets a 64-entry direct-mapped cache in front of your binary search, which measured 99.7% hits on this game. Numbers are in the comment thread below.

Two pieces worth separating

runtime/src/ops/misc.rs fixes a bug that predates this branch: setge tested ZF == OF instead of SF == OF. It is buried in b23a8d9 along with a lot of unrelated work, so happy to lift it into its own commit.

tc: avoid unstable push_mut is a two-line change that makes tc build on stable. Independent of everything else here.

Things I am unsure you will want

• catch_unwind around each instruction in codegen turns an unhandled instruction into a todo!() in the output instead of stopping the build. The aggressive scanning turns up junk blocks that would otherwise be fatal, but the cost is that a genuine mistranslation gets logged rather than failing loudly. See the thread below, I think the honest fix is upstream of codegen.
• static-server.go grew /log and /frame endpoints so a script can watch a page it cannot see. Reasonable to want those out, or behind a flag.
• translate.sh has my local path to the game.

Known gaps

I went over this before sending and there is a list of things I know are wrong but have not fixed, nearly all in code this branch adds. The main ones: wsprintfA does not cap output at 1024 the way the real one does, wildcard_match in kernel32 backtracks exponentially, the wasm filesystem leaks a string per path component, and refcounts on ddraw and dsound objects are fake, so a balanced AddRef/Release frees a live object. I am happy to work through them. I just did not want to keep sitting on the branch without knowing whether you want it at all.

State of the game

Intro, language select, table select, and gameplay with working flippers. Sound works in the browser. Under WSL it cuts out after a few seconds, but that is a WSLg bug (microsoft/wslg#1392) rather than anything here.

[evmar]

Wowwwww this is incredible! It will take me a bit to get to this but at a skim it is great and I want it all. I am fine with code that partially works as long is it is in the right direction, so the missing things you mentioned sound fine.

I am extra impressed you navigated through all of the random places I did something quite hacky, sorry for those!

A few first-glance comments:

• Maybe we shouldn't check in the generated code for this? It seems pretty large. In general I probably shouldn't be checking in any of this generated code, it's just been very convenient for diffing purposes to be able to see what changes when I change the generator. I briefly experimented with putting the generated code in a separate repo but it quickly got kind of annoying to manage. Having all of these programs checked in is also annoying because if I change any API then my editor re-type-checks all of the programs and spews a bunch of errors...

• Is BlockMap a hash table? Why not a Rust HashMap? (I only skimmed.) The binary search is definitely bad! I did some experiments in retrowin32 which had a similar thing, and found that a trivial cache https://github.com/evmar/retrowin32/blob/7c9243072755da54054bcaa9e1351371af638ded/x86/src/icache.rs#L142-L143 had a surprisingly great hit rate, like 90+%, because generally you have small hot loops. My long-term idea here is we could use a perfect hash table.

• Regarding splitting up the generated code, what you have seems fine, though it's disappointing it takes so long to compile! The main idea I have here is that I could build a block dominator graph to find isolated subcomponents that then could maybe get compiled in parallel in their own crates, but it might be a stretch. The other thing that will help is not splitting into so many tiny functions if it can translate loops/ifs/etc to Rust-level loops/ifs/etc. I've been toying with such tiny toy programs I haven't thought about it too much yet.

• I think I only depended on nightly for the wasm bits, because I need -Z build-std to enable Rust's support for JS workers.

[evmar]

Oh and re the catch_unwind thing, one trick I've been doing you can replace most todo!() (which is a crash at compile time) into a self.todo() (which puts a todo!() in the output, which is a crash at runtime only if the code is hit). There's also a bit of support for the translation returning errors to drop a block entirely, like see the "suspicious block of 0" bit of code.

[avanturist888]

Sorry for the slow reply, I was away on holiday. Thanks for such a generous response. I have rebased onto main and acted on your points; the PR is now about 6k lines instead of 385k.

Generated code. Dropped, along with the game's data/*.raw. out/winpin is now just Cargo.toml, lib.rs, main.rs and a .gitignore saying the rest comes from translate.sh winpin. Worth noting separately from size: those .raw files were the game's own code and data segments, so checking them in meant shipping a commercial game's binary in your repo. Both problems go away together.

I left out/winpin in members rather than adding default-members, since cargo build already fails on win32-extract and sbaitso-sbtalker without files you have locally. Winpin without its generated source is the same situation, so it seemed wrong to invent a new convention for it.

BlockMap. Gone. Your instinct was right and by more than I expected, so I replaced it with a 64-entry direct-mapped cache in front of your binary search. I instrumented indirect() and ran the game for 70 seconds:

105M indirect calls (~1.5M/s), 935 distinct targets

cache entries    1      4      16     64     256    1024
hit rate       96.8%  98.5%  99.3%  99.7%  99.9%   100%

A one-entry cache gets 96.8%, so your retrowin32 number reproduces here and whatever sits behind the cache barely matters. The other figure that stands out is 935: that is every indirect target this program ever reaches, out of 24114 blocks, so a perfect hash looks very reachable when you want it.

Caveat: that is the intro and menus, since I was not driving input, so real table play would widen the working set. Two runs agreed to within a percent. I can measure during a game if it would be useful.

To answer the original question anyway: it was an open-addressed table of u32 indices, and there was no good reason it was not a HashMap beyond it being 30 lines and my not wanting to pick a hasher. Please do not read it as a considered choice. The cache is better on every axis, including that it lives in the Context rather than in a global, so threads no longer share it.

Since you now take &self in indirect, the cache slots are Cells. Happy to shape that differently if you would rather.

Compile time. Agreed, and your second idea is the one I would bet on. 24114 blocks become 24114 functions and I think per-function overhead is most of the cost. Sharding took a cold build from 60s to 24s purely by handing LLVM parallel units, which is the same lever your dominator graph idea pulls harder.

nightly. It was not only the wasm bits: gather.rs called instrs.push_mut(...), which is still unstable (rust#135974), so cargo +stable check -p tc failed on your tree with none of my changes. I have pushed a two-line commit replacing it with push plus last_mut, and tc now builds on stable. rust-toolchain.toml is dropped from the PR, so nightly is genuinely wasm-only again.

catch_unwind. Thanks, I am already using self.todo() wherever the instruction reaches a match arm I wrote. What catch_unwind catches beyond that is asserts firing deep in operand handling on blocks that are not code at all. The honest fix is upstream of codegen: my looks_like_code accepts anything that decodes as a single instruction, which is loose, and the junk it lets through is what makes codegen panic. I would rather tighten that and use your bail path to drop the block, the way the zero-filled check does, than keep a blanket catch. That is the next thing I will do unless you would rather I did something else first.

[evmar]

Sorry for the delay, my son had a week off from summer camp so I also didn't have time to look at this! I read through and merged a bunch of the first commit so if you rebase you shouldn't (I think?) conflict. I left comments on some of the other bits.

Big picture, this would be easier to review if it were smaller separate commits. Is this LLM-generated? If it's easy for you to split it I would appreciate it. Otherwise I can try to continue to split as I've done already.

[evmar]

I failed at writing a better note to myself, but my recollection is one of my test programs failed when this function returned any data due to other missing functions. It's plausible you implemented all the missing functions in this branch though so I will just figure it out and remember to write a better comment next time. :)

[evmar]

This isn't true, but I guess the warning here will help me track it down if it matters.

[evmar]

I would rather define structures than poke at offsets in a buffer like this.

[evmar]

Is this legal? I think better to fail here

[evmar]

I think this should instead be a Mutex and then no mutex within State. I'll merge for now though.

[evmar]

I worry this approach will mean it's easy to get into a state where we miss the code pointers from an unlikely branch. Like if there's a vtable with 5 entries but when you run in this mode and you only hit three methods, things will seem ok but then we'll crash when one of the other methods gets hit.

What I've been doing so far is when I hit one of these, I disassemble the source program and try to understand where the code pointer came from, so I can proactively collect all the relevant addresses. This approach doesn't scale well though. :(

I guess I'm trying to say I'm not sure how I feel about this approach, maybe it's fine to merge for now.

[evmar]

These should be methods on RECT I think

[evmar]

Can this be an enum using derive(ABIEnum) macro?

[evmar]

Can these be an enum?

[evmar]

Can these use win32flags! macro?

[evmar]

Also, push_mut is now in stable Rust: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.push_mut

[avanturist888]

No apology needed, and thank you for splitting and merging as much as you did. Rebased onto main; the merged parts dropped out cleanly.

Yes, it is LLM-assisted. I worked through this with Claude. I read and tested everything that went in, and I am on the hook for it, but you should know that when you weigh how much to trust the parts you have not read. It also explains the volume and probably the comment style. If that changes how you want to take this, or if you would rather I mark which parts got the least human scrutiny, say so.

Split. The three big commits are now 18, one per subsystem, largest 681 lines:

tc: avoid unstable push_mut
runtime: report code addresses the scan missed
runtime: cache indirect jump targets
tc: scan memory, immediates and jump tables for code
tc: stub unknown blocks, split output into parts
host: keyboard and mouse button events
host: in-memory filesystem for wasm
kernel32: file api
kernel32: module registry for LoadLibrary/GetProcAddress
kernel32: nls and lstr* functions
user32: shared input state, keyboard messages
dinput: keyboard and mouse
winmm: mmio
dsound: software mixer
ddraw: colorkey, palettes, pixel formats
winapi: ole32, msacm32, advapi32
start on winpin, pinball soccer '98
web: run winpin in the browser

They are split by file, so where one file holds two ideas they stayed together: tc/src/gather.rs has all the scanning work in one commit, and codegen/mod.rs has both the unknown-block stubs and the output sharding. Happy to break those down further if it helps.

Also removed a web/game symlink pointing into my home directory that I had committed by accident.

Your comments. Fixed:

• CreateFileA and SetFilePointer now take ABIEnum enums (CreationDisposition, MoveMethod), so the match arms are exhaustive and an unknown value fails loudly like the rest of the codebase.
• GetCPInfo writes a real CPINFO struct. It needs an explicit trailing _pad: [u8; 2] because the Windows struct has that padding and zerocopy will not write a type with implicit padding.
• C1 classification bits are a win32flags! struct now. The macro is fixed to u32, so the value is narrowed at the two write sites.
• lstrlenA(NULL) no longer returns 0. You were right that it is not legal; it faults on Windows, and now the null page guard catches it here.
• dinput is Mutex<Option<State>> plus LockedState, matching dsound, with no mutex inside State.
• full_rect/clip_rect are now RECT::from_size and RECT::clip_to_size.
• The ExitThread comment was simply wrong, as you said. It now says what actually happens, which is that a CreateThread worker calling it kills the process.

On push_mut being stable now: good to know, thank you. I left the commit in since it is two lines and lets tc build on an older stable, but drop it if you would rather wait for the toolchain to catch up.

On THESEUS_MISSING_ADDRS. Your worry is right and I do not think it can be argued away: it only finds pointers on paths that actually run, so a vtable entry never exercised stays missing until it crashes. It is a net to catch what the static scanning misses, not a substitute for understanding where the pointer came from, and it works because the crash is loud and the fix is one line in a file. If you would rather not carry it, the scanning commits stand on their own without it. I would rather drop it than have it paper over gaps you would otherwise fix properly.

catch_unwind is still on my list, along with the wsprintfA and wildcard_match problems I mentioned. Let me know if you would rather I did something else first.

[avanturist888]

Went through the whole list of things I had flagged as known-bad, so this is one message rather than a trickle. 30 commits now, still one subsystem each.

catch_unwind is gone. Worth reporting the measurement: it fired zero times on this game. Coverage, block count and generated output are identical without it, and 4 blocks still get dropped by your bail! path, which is doing the real work. It was scaffolding from when the scanning was cruder, and I left it in after the thing it protected against stopped happening. Your objection was right and it was cheaper than either of us expected.

wsprintfA, in the version you merged, had three problems:

• No 1024 cap, so a long %s writes past the caller's buffer into whatever follows.
• The field width is parsed unbounded into a usize that then sizes an allocation, so %2000000000d asks for 2GB, and width * 10 overflows on a long enough digit run.
• An unhandled specifier printed %f but did not consume its argument, so every argument after it read the wrong stack slot. That one is easy to hit: wsprintfA(buf, "%f fps in %s", flt, name) makes %s treat the float's low dword as a pointer.

All three fixed.

wildcard_match was recursive with backtracking, so FindFirstFileA("*a*a*a*a*a*a*a*b") against a 40-character name hangs. It is iterative now, backing up only to the most recent *, with a test including that case.

The rest, one commit each:

• GetAsyncKeyState never set the low bit, so the standard & 1 edge test never fired and that key looked dead. It now tracks presses since the last call.
• normalize_module_name stripped .dll before folding case, so GetModuleHandleA("DDraw.Dll") returned null.
• The no-audio-device path returned u32::MAX from queued_bytes, which strands winmm/wave.rs: its while queued_bytes() < 8<<10 loop never runs, buffers are never returned, and the program waits forever. It reports 0 now, and dsound skips mixing when there is no device rather than burning cpu.
• The wasm filesystem leaked a String per path component, and resolve_path calls exists() once per component, so a session leaked steadily in a heap that cannot be reclaimed.
• GetDeviceState allocated cbData bytes straight from the app; it validates against the device now. GetDeviceData with a null array was counting events, but that form is documented to discard them, so a game flushing stale input after Acquire replayed everything instead.
• dsound cursors could land outside the buffer: the write cursor was not wrapped, and SetCurrentPosition was not clamped.
• A clipped blit moved the destination rect but not the source, so a sprite leaving the left edge jumped right instead of being cropped. Colorkey blits also hit unreachable! at 16bpp, which is reachable since 16bpp surfaces are creatable.
• Surfaces and sound buffers are reference counted for real. The old comment claimed a balanced AddRef/Release was safe, which was exactly backwards: the first Release freed the object.
• EnumDisplayModes advertised 24bpp that to_rgba cannot convert, so a game preferring the deepest mode got a black window. Dropped it.
• Removed the DDERR_ALREADYINITIALIZED alias bound to DD::OK; a name meaning "error" with a success value reads as a bug. Also dropped a redundant present after Unlock, which unlock already does.
• mmio chunk arithmetic saturates, so a truncated RIFF file returns an error instead of overflowing.
• The web host called preventDefault on every key, swallowing F5 and Ctrl+R, and registered document key listeners per window, so a second window would double every press. Comments are // rather than /// now, which is what the rest of the file does.

What I have not done: ExitThread still exits the process, since doing it properly means unwinding a translated thread and I would rather not guess at that; the comment says so plainly now. The wasm filesystem still loses writes if the same file is open twice concurrently.

Verified on the game after all of this: same coverage, no panics, and the screen renders the same as before the blit change. cargo check is clean on stable.