Rework mmap index into standalone readable/readonly/writable implementations and evaluate end-to-end impact#313
Rework mmap index into standalone readable/readonly/writable implementations and evaluate end-to-end impact#313Copilot wants to merge 14 commits into
Conversation
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/a349b1b9-d1b5-483c-bf52-45c8299f7fc4 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/48db503a-b711-4c37-a5c9-a7db9b33521f Co-authored-by: albe <4259532+albe@users.noreply.github.com>
|
@copilot turn this into a fully compatible index implementation, without a mappedLength constraint. Then replace all usage of index with the mmap version and run the full benchmark set as well as the stress tests. I want to see how much index performance impacts overall performance of the store and if the mmap solution leads to more data loss, because the write isn't synced |
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/9e067eb3-42d4-4a54-9444-9718d90e69e6 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/9e067eb3-42d4-4a54-9444-9718d90e69e6 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Implemented in 9f7920b and polished in 6530e5a. Added a fully compatible mmap-backed index implementation (dynamic growth, no mapped-length cap), wired storage to accept pluggable index classes, and switched latest benchmark/stress paths to mmap indexes when available. Full benchmark suite rerun (bench-index, bench-storage, bench-eventstore, bench-read-scenarios) plus stress tests rerun. Latest Node 20 results showed index: stable ~1244 ops/sec, latest ~1333 ops/sec, mmap ~954 ops/sec; storage/eventstore/read-scenarios latest remained slower than stable with mmap. Crash stress test passed with observed data loss within bound (e.g. 5 events lost vs 28 allowed), and unit tests remain 432 passing. |
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/618ca70d-1321-47fc-8f9e-8db11674eda7 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/3e31aa06-d0fe-40d8-a820-951420b2d7ca Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Implemented in 5bf0500. The mmap index remains fully compatible without a mapped-length cap, and mmap usage is wired through latest paths. I reran full benchmarks and stress tests on Node 20 with mmap enabled: index mmap ~906 ops/sec (stable ~1200), storage latest ~118 vs stable ~109 ops/sec, eventstore latest ~79 vs stable ~85 ops/sec, read scenarios latest below stable; crash stress and memory stress both passed with no bounded-loss violation.
Done in 5bf0500. I copied/aligned index method docblocks from the base readable/writable/readonly implementations and only adjusted wording where mmap behavior differs (direct mapped reads/writes and remap strategy). |
…ose timing Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/f1c014cc-a21d-4923-bae7-7361d621ee18 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/f1c014cc-a21d-4923-bae7-7361d621ee18 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/ddccc562-a9fe-457a-9d86-c8e2b147c848 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/ddccc562-a9fe-457a-9d86-c8e2b147c848 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/6dab9728-78ed-4cdf-b0f4-07d3490b1e60 Co-authored-by: albe <4259532+albe@users.noreply.github.com>
|
Closing in favor of #315 |
MmapReadableIndex.range()to return an array (populatethis.datathen slice forward, build result array for reverse)MmapReadableIndex.all()to return an array (delegate torange())iterateEntries/iterateEmptyEntriesgenerator methods