Add a search benchmark, on desktop and on device - #22
Open
lepht wants to merge 1 commit into
Open
Conversation
lepht
force-pushed
the
fix/search-index-cap
branch
from
August 12, 2026 00:41
c7033f6 to
d44a140
Compare
lepht
force-pushed
the
feat/search-bench
branch
from
August 12, 2026 00:41
c43c084 to
64c646d
Compare
lepht
force-pushed
the
fix/search-index-cap
branch
from
August 12, 2026 01:25
d44a140 to
a0f2fcf
Compare
lepht
force-pushed
the
feat/search-bench
branch
from
August 12, 2026 01:25
64c646d to
32bd15b
Compare
lepht
force-pushed
the
fix/search-index-cap
branch
from
August 12, 2026 03:42
a0f2fcf to
131616b
Compare
lepht
force-pushed
the
feat/search-bench
branch
from
August 12, 2026 03:42
32bd15b to
69899a8
Compare
Every performance claim about search so far has been a number I read off my own terminal once. This makes them reproducible. The binary prints nothing unless asked. NEXTUI_BENCH=1, or a bench file on the card, logs one key=value line per measured operation - the file matters because a device has no convenient way to set an env var for nextui.elf, which the launcher runs in a loop. Off, the cost is one getenv and one stat at startup and a predictable branch at each measurement site. NEXTUI_BENCH=run measures and exits, for the desktop harness. It is fenced to desktop builds: on a device, exiting immediately would leave the launcher respawning us forever. It also leaves without the normal teardown, because PLAT_quitVideo() destroys textures and flushes the renderer assuming frames have been drawn, and this mode exits before the first one - an ordinary exit after a rendered frame is clean, going through it from here segfaults. bench/run_desktop.sh compares two commits on one machine. It generates a deterministic library so both see byte-identical input, and adds it as its own Bench System * folders with BENCH* pak tags that cannot collide with a real emulator, recording every path it creates so cleanup removes exactly those and nothing else. bench/device_bench.sh measures a real card on real hardware, which is the only place the walk cost can be honestly established - it is I/O bound, and the same code moves 3x on one desktop between a warm and cold page cache. No CI gate. Wall-clock thresholds on shared runners would measure server-class ARM and a page cache instead of an A53 and a microSD, and would need to be so loose to avoid flaking that they would only catch what is already obvious. The regressions worth catching are structural and countable, but nextui.c is one translation unit with everything static, so there is nothing to assert against yet. Reasoning written up in bench/README.md rather than lost in a thread.
lepht
force-pushed
the
fix/search-index-cap
branch
from
August 12, 2026 05:04
131616b to
1332aa4
Compare
lepht
force-pushed
the
feat/search-bench
branch
from
August 12, 2026 05:04
69899a8 to
98a7d40
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every performance claim I've made about search so far has been a number I read off my own terminal once. This makes them reproducible. Based on #21.
Measurement is opt-in
The binary prints nothing unless asked:
NEXTUI_BENCH=1— measure the normal flow, log the numbers..userdata/shared/.minui/benchon the card — same thing. The file exists because a device has no convenient way to set an env var fornextui.elf, whichMinUI.pak/launch.shruns in a loop.NEXTUI_BENCH=run— measure, print, exit. Desktop builds only.Output is one
key=valueline per operation, so a script can diff two runs without parsing prose:With it off, the cost is one
getenvand onestatat startup, then a predictable branch at each measurement site. Nothing is timed, nothing is logged, no behaviour changes.Two safety fences worth review
runmode is desktop-only. On a device, a mode that exits as soon as it has its numbers would leave the launcher respawning nextui forever. The guard is aPLATFORMstring check, and the card marker file only ever selects log mode — so there is no path by which a device reaches the exiting mode.runmode skips the normal teardown, and that is deliberate.PLAT_quitVideo()destroys textures and flushes the renderer on the assumption that frames have been drawn; this mode exits before the first one. I found this the direct way — the first version fell through to the normal teardown and segfaulted inPLAT_quitVideo. An ordinary exit after a rendered frame is clean (verified: baseline game-launch exit returns 0), so this looks like a latent fragility in the video teardown rather than anything about search. Worth an issue on its own; not fixed here.bench/run_desktop.shAnswers "did this commit make search slower than the last one". Generates a deterministic library so both commits see byte-identical input.
SDCARD_PATHis compiled into the binary, so the benchmark has to use the card the desktop build was built for. Rather than take it over, the generator adds its ownBench System *folders withBENCH*pak tags that cannot collide with a real emulator, records every path it creates in a manifest, and removes exactly those on cleanup. It refuses to start if a generated library is already present, and refuses to touch a folder or pak it did not make. Verified the dev card comes back byte-identical: 65 ROMs and 7 paks before and after.The harness immediately showed something I'd stated too simply in #21: bytes/entry is not constant. It climbs from 190 to 231 as the library grows, because more names collide and the dedupe pass allocates a
uniquestring for more of them. The 32768 ceiling is therefore closer to ~7.5 MB than the ~6.5 MB I quoted. Same conclusion, but the number was worth getting right, and it is exactly the kind of thing a harness is for.bench/device_bench.shAnswers "how long does this take on a Brick with a real card" —
on/report/offover SSH. This is the only place the walk cost can be honestly established: it is I/O bound, and the same code moves 3× on one desktop between a warm and cold page cache. It closes out the number I've been flagging as unmeasured in the last two PRs.Deliberately not a CI gate
Wall-clock thresholds on shared runners would measure server-class ARM and a page cache instead of an A53 and a microSD, and would need thresholds so loose to avoid flaking that they'd only catch what's already obvious. A check that cries wolf gets ignored, then disabled — worse than no check.
The regressions worth catching here are structural and countable (
entries,heap_bytes,heap_per_entryare already deterministic): the index rebuilt per frame instead of once per launch, a linear scan turning quadratic, astat()per file appearing in the walk. Asserting on those is blocked onnextui.cbeing one 4000-line translation unit with everythingstatic, so there's nothing to link a test against. If search ever moves into its own unit this gets cheap; until then it'd be a refactor wearing a testing costume.That reasoning is in
bench/README.mdrather than left in a thread, so the next person to want a perf gate finds the argument instead of relitigating it.