Add mmap-io-backed append-only file, mmap-backed Index, and mmap-backed Partition with direct mapped-buffer reads

[Copilot]

• Review the existing storage benchmark and storage read/watch APIs to mirror current benchmark coverage
• Run the repository tests to capture the current baseline before changes
• Add a new storage benchmark that forks async child readers, excludes process start/stop time, and reports wall-clock completion plus per-reader completion averages
• Measure write-to-reader visibility latency for each storage variant and reader count, and include those metrics in the benchmark output
• Run the repository tests again and execute the new benchmark script to verify it works end to end

[albe]

@copilot You forgot the "mmap" part of the thing. Use mmap-io (@riaskov/mmap-io version) to implement actual OS based memory mapping of a file, with the same instructions I gave earlier

[Copilot]

@copilot You forgot the "mmap" part of the thing. Use mmap-io (@riaskov/mmap-io version) to implement actual OS based memory mapping of a file, with the same instructions I gave earlier

Implemented with real OS-backed file mapping via @riaskov/mmap-io in src/File/AppendOnlyMmapedFile.js (map/remap/grow now use mmap.map, flush uses mmap.sync), and updated dependency metadata/lockfile in commit e3aeef7.

[albe]

@copilot address the comments

[Copilot]

@copilot address the comments

All review comments addressed in commits 531cc20 and a99e26f:

• writeFileSizeMarker, flush, and truncate moved to WritableAppendOnlyMmapedFile
• pageSize uses mmap.PAGESIZE directly and lives in the writable subclass (only consumer)
• writeBufferSize is clamped to Math.max(this.pageSize, options.writeBufferSize || DEFAULT_WRITE_BUFFER_SIZE)
• Dirty-range tracking (markDirty, clearDirtyRange, dirtyFrom, dirtyTo) removed entirely

[albe]

@copilot add an example implementation of an append only mmap file backed Index and place it besides the existing Index implementation.

[Copilot]

@copilot add an example implementation of an append only mmap file backed Index and place it besides the existing Index implementation.

Added in commit 8f87e5e (cleanup in cec0c39):

• src/Index/MmapReadableIndex.js — base class; reads entries directly from the mmap buffer via ReadableAppendOnlyMmapedFile, same header format as the existing index (nesidx01 magic + JSON metadata). No extra copy or syscall per read.
• src/Index/MmapWritableIndex.js — appends fixed-size entry records into the mmap file using WritableAppendOnlyMmapedFile. The metadata header is passed as initialData so it is written atomically on first creation. Delayed flush with per-position callbacks mirrors the existing WritableIndex API.
• src/Index/MmapReadOnlyIndex.js — wraps ReadOnlyAppendOnlyMmapedFile and translates the file-level append/truncate byte-size events into entry-count events (emit('append', prevLen, newLen)).
• src/MmapIndex.js — facade mirroring src/Index.js; exported from the package as MmapIndex.
• test/MmapIndex.spec.js — 18 tests covering open/close, add, get, range, find, truncate, metadata persistence, and live reader events.

[albe]

@copilot apply the comment change, then run the index benchmark with the mmap'd index implementation against stable and latest basic version. If mmap is slower for the read benchmark, implement a caching of the read Entry items that fills up and returns a slice of the cached array

[albe]

Add a file API that allows entry.toBuffer() to receive the buffer location of current write position of the file, with a given length (or until the end of the mapped range by default). e.g. entry.toBuffer(this.file.write(this.EntryClass.size), 0)

[Copilot]

Done in commit f9516f6. Added a dedicated reserve(size) method to WritableAppendOnlyMmapedFile that reserves bytes at the current write position and returns the mapped subarray for in-place serialisation. MmapWritableIndex.add() now uses entry.toBuffer(this.file.reserve(this.EntryClass.size), 0) — no intermediate Buffer.allocUnsafe per add. The original write(data) keeps its Buffer/string signature unchanged.

[albe]

Bench result with 1000 Writes being consumed across N readers. 1 OP = 1000 reads

Readers	Stable ops/s	Latest ops/s	Mmap ops/s
1	172	161	190
2	145	148	170
4	126	139	153
8	115	134	138
16	104	126	129

[albe]

@copilot also implement your earlier suggestion:

Keep logical size in memory during hot write loop and write marker once per flush.

After doing so, rerun the benchmark again.

[albe]

Where is the copied reference to fd needed? Only file should hold it

[Copilot]

@copilot also implement your earlier suggestion:

Keep logical size in memory during hot write loop and write marker once per flush.
...

Implemented in e4ab266: WritableAppendOnlyMmapedFile.grow() no longer writes the file-size marker (or marks pending) during growth, so marker updates stay flush-driven while logical size remains in memory during the write loop.

Reran the storage benchmark (bench/bench-storage.js) after the change:

• stable: 175 / 148 / 128 / 116 / 105 ops/s (readers 1/2/4/8/16)
• latest: 164 / 145 / 138 / 130 / 122 ops/s
• mmap: 183 / 164 / 152 / 143 / 130 ops/s