Skip to content

Add a search benchmark, on desktop and on device - #22

Open
lepht wants to merge 1 commit into
fix/search-index-capfrom
feat/search-bench
Open

Add a search benchmark, on desktop and on device#22
lepht wants to merge 1 commit into
fix/search-index-capfrom
feat/search-bench

Conversation

@lepht

@lepht lepht commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Every performance claim I've made about search so far has been a number I read off my own terminal once. This makes them reproducible. Based on #21.

Measurement is opt-in

The binary prints nothing unless asked:

  • NEXTUI_BENCH=1 — measure the normal flow, log the numbers.
  • .userdata/shared/.minui/bench on the card — same thing. The file exists because a device has no convenient way to set an env var for nextui.elf, which MinUI.pak/launch.sh runs in a loop.
  • NEXTUI_BENCH=run — measure, print, exit. Desktop builds only.

Output is one key=value line per operation, so a script can diff two runs without parsing prose:

BENCH search.index entries=20018 truncated=0 walk_ms=19.59 sort_ms=8.47 dedupe_ms=5.17 total_ms=33.23 heap_bytes=4625664 heap_per_entry=231.1
BENCH search.filter query=zel query_len=3 entries=20018 hits=832 ms=1.737

With it off, the cost is one getenv and one stat at startup, then a predictable branch at each measurement site. Nothing is timed, nothing is logged, no behaviour changes.

Two safety fences worth review

run mode is desktop-only. On a device, a mode that exits as soon as it has its numbers would leave the launcher respawning nextui forever. The guard is a PLATFORM string check, and the card marker file only ever selects log mode — so there is no path by which a device reaches the exiting mode.

run mode skips the normal teardown, and that is deliberate. PLAT_quitVideo() destroys textures and flushes the renderer on the assumption that frames have been drawn; this mode exits before the first one. I found this the direct way — the first version fell through to the normal teardown and segfaulted in PLAT_quitVideo. An ordinary exit after a rendered frame is clean (verified: baseline game-launch exit returns 0), so this looks like a latent fragility in the video teardown rather than anything about search. Worth an issue on its own; not fixed here.

bench/run_desktop.sh

Answers "did this commit make search slower than the last one". Generates a deterministic library so both commits see byte-identical input.

SDCARD_PATH is compiled into the binary, so the benchmark has to use the card the desktop build was built for. Rather than take it over, the generator adds its own Bench System * folders with BENCH* pak tags that cannot collide with a real emulator, records every path it creates in a manifest, and removes exactly those on cleanup. It refuses to start if a generated library is already present, and refuses to touch a folder or pak it did not make. Verified the dev card comes back byte-identical: 65 ROMs and 7 paks before and after.

roms     entries    walk_ms   sort_ms  dedup_ms   total_ms   heap_bytes   b/entry
1000     1020          2.09      0.27      0.03       2.39       194304     190.5
5000     5024          5.85      1.70      0.47       8.02      1045424     208.1
20000    20018        19.59      8.47      5.17      33.23      4625664     231.1

The harness immediately showed something I'd stated too simply in #21: bytes/entry is not constant. It climbs from 190 to 231 as the library grows, because more names collide and the dedupe pass allocates a unique string for more of them. The 32768 ceiling is therefore closer to ~7.5 MB than the ~6.5 MB I quoted. Same conclusion, but the number was worth getting right, and it is exactly the kind of thing a harness is for.

bench/device_bench.sh

Answers "how long does this take on a Brick with a real card" — on / report / off over SSH. This is the only place the walk cost can be honestly established: it is I/O bound, and the same code moves 3× on one desktop between a warm and cold page cache. It closes out the number I've been flagging as unmeasured in the last two PRs.

Deliberately not a CI gate

Wall-clock thresholds on shared runners would measure server-class ARM and a page cache instead of an A53 and a microSD, and would need thresholds so loose to avoid flaking that they'd only catch what's already obvious. A check that cries wolf gets ignored, then disabled — worse than no check.

The regressions worth catching here are structural and countable (entries, heap_bytes, heap_per_entry are already deterministic): the index rebuilt per frame instead of once per launch, a linear scan turning quadratic, a stat() per file appearing in the walk. Asserting on those is blocked on nextui.c being one 4000-line translation unit with everything static, so there's nothing to link a test against. If search ever moves into its own unit this gets cheap; until then it'd be a refactor wearing a testing costume.

That reasoning is in bench/README.md rather than left in a thread, so the next person to want a perf gate finds the argument instead of relitigating it.

Every performance claim about search so far has been a number I read off my
own terminal once. This makes them reproducible.

The binary prints nothing unless asked. NEXTUI_BENCH=1, or a bench file on the
card, logs one key=value line per measured operation - the file matters because
a device has no convenient way to set an env var for nextui.elf, which the
launcher runs in a loop. Off, the cost is one getenv and one stat at startup
and a predictable branch at each measurement site.

NEXTUI_BENCH=run measures and exits, for the desktop harness. It is fenced to
desktop builds: on a device, exiting immediately would leave the launcher
respawning us forever. It also leaves without the normal teardown, because
PLAT_quitVideo() destroys textures and flushes the renderer assuming frames
have been drawn, and this mode exits before the first one - an ordinary exit
after a rendered frame is clean, going through it from here segfaults.

bench/run_desktop.sh compares two commits on one machine. It generates a
deterministic library so both see byte-identical input, and adds it as its own
Bench System * folders with BENCH* pak tags that cannot collide with a real
emulator, recording every path it creates so cleanup removes exactly those and
nothing else.

bench/device_bench.sh measures a real card on real hardware, which is the only
place the walk cost can be honestly established - it is I/O bound, and the same
code moves 3x on one desktop between a warm and cold page cache.

No CI gate. Wall-clock thresholds on shared runners would measure server-class
ARM and a page cache instead of an A53 and a microSD, and would need to be so
loose to avoid flaking that they would only catch what is already obvious. The
regressions worth catching are structural and countable, but nextui.c is one
translation unit with everything static, so there is nothing to assert against
yet. Reasoning written up in bench/README.md rather than lost in a thread.
@lepht
lepht force-pushed the fix/search-index-cap branch from 131616b to 1332aa4 Compare August 12, 2026 05:04
@lepht
lepht force-pushed the feat/search-bench branch from 69899a8 to 98a7d40 Compare August 12, 2026 05:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant