fix: scope pool totals to --dax-path and serialize RpcClient requests#62
Open
seohui-XCENA wants to merge 2 commits into
Open
fix: scope pool totals to --dax-path and serialize RpcClient requests#62seohui-XCENA wants to merge 2 commits into
seohui-XCENA wants to merge 2 commits into
Conversation
request_alloc only claims regions from the configured pools, but get_stats/get_usage summed capacity across every resource-manager pool. On hosts with foreign DAX devices this inflates cxl_pool.free_size (observed: a 3.5 TiB dax0.0 pushing an eviction-watermark total to ~4 TiB against a 466 GiB usable pair), deferring client-side eviction until allocation fails. Filter the summed pools by the server's dax_path scope; unrestricted servers keep the all-pools behavior.
The shared ZMQ REQ socket is not thread-safe and enforces strict send->recv alternation, but concurrent callers can hit it at once (e.g. the eviction loop and a metrics scrape both issue get_stats). Overlapping exchanges corrupt the REQ state and both calls degrade to spurious timeouts plus socket resets. Guard connect/close/_send_request with one lock; _reset_socket stays lock-free (only called under it).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two issues surfaced while wiring the LMCache MP L1 fullness gauges:
get_stats/get_usagesummed CXL capacity across every resource-manager pool (foreign DAX devices inflate the eviction-watermark total), and the shared ZMQ REQ socket inRpcClientcorrupts when two threads issue RPCs concurrently (eviction loop + metrics scrape both callingget_stats).Key Changes
maru_server — scope
_pool_totalsto the server's--dax-pathpoolsrequest_alloconly claims regions from the configured pools, but the totals summed every RM pool. On a host with a foreign device this inflatescxl_pool.free_size, which clients use as the eviction-watermark total./dev/dax0.0pushed the watermark total to ~4 TiB against a 466 GiB usable pair (dax1.0+dax2.0) — usage ratio stuck at ~5%, so client-side LRU eviction would never engage and stores would run into allocation failure instead._pool_totals()now filters by the server'sdax_pathscope; unrestricted servers keep the all-pools behavior. Covers bothget_stats.cxl_pool(watermark input) andget_usage.pool_total/pool_free(marutop).maru_handler — serialize
RpcClientrequests with a per-call lockconnect/close/_send_request(including the reset paths) now run under one per-instance lock; concurrent callers queue._reset_socketis only called with the lock held.RpcClient(separate lock), so it never contends with the control path.AsyncRpcClientis untouched.Test Plan
test_pool_totals_scoped_to_configured_dax_paths: foreign pool excluded fromget_usageandget_stats.cxl_poolTestRpcClientThreadSafety::test_concurrent_send_requests_are_serialized: 8 threads, no overlapping send→recv exchangepytest -v) — all consumer suites green (test_maru_server,test_rpc_client,test_server,test_usage_monitor,test_rpc_handler_mixin,test_allocation_manager,test_rpc_async_client,test_shm_client,test_shm_get_dax_path; 217 tests).test_maru_handler.py/test_thread_safety.pybus-error on the dev host identically on cleanmain(live-RM + DAX mmap environment issue, unrelated to this change).--dax-path dax1.0,dax2.0server now reportscxl_pool466 GiB vs 4042 GiB unrestricted.Related Issues
Related: #60 (scopes the
cxl_pooltotals introduced there)