Skip to content

fix: scope pool totals to --dax-path and serialize RpcClient requests#62

Open
seohui-XCENA wants to merge 2 commits into
xcena-dev:mainfrom
seohui-XCENA:fix/stats-dax-scope-and-rpc-lock
Open

fix: scope pool totals to --dax-path and serialize RpcClient requests#62
seohui-XCENA wants to merge 2 commits into
xcena-dev:mainfrom
seohui-XCENA:fix/stats-dax-scope-and-rpc-lock

Conversation

@seohui-XCENA

Copy link
Copy Markdown
Collaborator

Summary

Fixes two issues surfaced while wiring the LMCache MP L1 fullness gauges: get_stats/get_usage summed CXL capacity across every resource-manager pool (foreign DAX devices inflate the eviction-watermark total), and the shared ZMQ REQ socket in RpcClient corrupts when two threads issue RPCs concurrently (eviction loop + metrics scrape both calling get_stats).

Key Changes

maru_server — scope _pool_totals to the server's --dax-path pools

  • request_alloc only claims regions from the configured pools, but the totals summed every RM pool. On a host with a foreign device this inflates cxl_pool.free_size, which clients use as the eviction-watermark total.
  • Observed on a 3-device host: a 3.5 TiB /dev/dax0.0 pushed the watermark total to ~4 TiB against a 466 GiB usable pair (dax1.0+dax2.0) — usage ratio stuck at ~5%, so client-side LRU eviction would never engage and stores would run into allocation failure instead.
  • _pool_totals() now filters by the server's dax_path scope; unrestricted servers keep the all-pools behavior. Covers both get_stats.cxl_pool (watermark input) and get_usage.pool_total/pool_free (marutop).

maru_handler — serialize RpcClient requests with a per-call lock

  • ZMQ sockets are not thread-safe and REQ enforces strict send→recv alternation; overlapping exchanges degrade both calls to spurious timeouts plus socket resets.
  • connect/close/_send_request (including the reset paths) now run under one per-instance lock; concurrent callers queue. _reset_socket is only called with the lock held.
  • No measurable overhead: A/B against a local REP echo server shows sequential mean 59.3µs (main) vs 59.1µs (this branch); an uncontended acquire+release is ~113ns (~0.2% of a roundtrip). Under 2-thread contention, main fails 30/100 calls while this branch fails 0/100 — the lock only ever blocks a caller in the case where main returned garbage.
  • The handler's stats flusher uses its own dedicated RpcClient (separate lock), so it never contends with the control path. AsyncRpcClient is untouched.

Test Plan

  • Unit tests added/updated
    • test_pool_totals_scoped_to_configured_dax_paths: foreign pool excluded from get_usage and get_stats.cxl_pool
    • TestRpcClientThreadSafety::test_concurrent_send_requests_are_serialized: 8 threads, no overlapping send→recv exchange
  • Existing tests pass (pytest -v) — all consumer suites green (test_maru_server, test_rpc_client, test_server, test_usage_monitor, test_rpc_handler_mixin, test_allocation_manager, test_rpc_async_client, test_shm_client, test_shm_get_dax_path; 217 tests). test_maru_handler.py/test_thread_safety.py bus-error on the dev host identically on clean main (live-RM + DAX mmap environment issue, unrelated to this change).
  • E2E: against the live resource manager on the 3-device host, a --dax-path dax1.0,dax2.0 server now reports cxl_pool 466 GiB vs 4042 GiB unrestricted.

Related Issues

Related: #60 (scopes the cxl_pool totals introduced there)

request_alloc only claims regions from the configured pools, but
get_stats/get_usage summed capacity across every resource-manager pool.
On hosts with foreign DAX devices this inflates cxl_pool.free_size
(observed: a 3.5 TiB dax0.0 pushing an eviction-watermark total to
~4 TiB against a 466 GiB usable pair), deferring client-side eviction
until allocation fails. Filter the summed pools by the server's
dax_path scope; unrestricted servers keep the all-pools behavior.
The shared ZMQ REQ socket is not thread-safe and enforces strict
send->recv alternation, but concurrent callers can hit it at once
(e.g. the eviction loop and a metrics scrape both issue get_stats).
Overlapping exchanges corrupt the REQ state and both calls degrade to
spurious timeouts plus socket resets. Guard connect/close/_send_request
with one lock; _reset_socket stays lock-free (only called under it).
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant