perf: instrument cursor collect/refill timing, reuse scan buffers#241
Closed
Keykyrios wants to merge 1 commit into
Closed
perf: instrument cursor collect/refill timing, reuse scan buffers#241Keykyrios wants to merge 1 commit into
Keykyrios wants to merge 1 commit into
Conversation
Add per-collect timing to ScanCursor and per-refill timing to DBScan so that cursor resume latency is visible in dbScanPerf logs. This makes it possible to distinguish cold seeks (page faults after B-tree mutation) from warm sequential iteration. Move refillBuffer/mergeBuffer from stack locals to DBScan members to eliminate repeated deque allocation during scan refills. New log fields (emitted when relay.logging.dbScanPerf = true): refills, avgRefillUs, maxRefillUs, cursorSeeks, avgSeekUs, maxSeekUs Add scripts/cursor_resume_bench.py for measuring REQ latency under concurrent write load, and scripts/scan_perf_report.py for parsing the enhanced log output. Ref: hoytech#195 Signed-off-by: Keykyrios <mitrajitghorui@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ref: #195
Adds timing to
ScanCursor::collectandDBScanrefill so cursor resume cost shows up indbScanPerflogs. Also movesmoreEvents/newEventQueuefrom stack locals to persistent members (same idea as #198).New log fields (only when
relay.logging.dbScanPerf = true):refills, avgRefillUs, maxRefillUs, collects, avgCollectUs, maxCollectUs
Also adds:
scripts/cursor_resume_bench.py— REQ latency sweep under concurrent write loadscripts/scan_perf_report.py— parses the enhanced log linesHow to test:
Requires
pip install websockets.