feat: unify maru monitoring into a single marutop CLI#61
Conversation
Add a new `maru_tools` package and a `marutop` console script, folding the four standalone tools/*.py monitors into
one umbrella command. The old scripts become thin back-compat shims.
- marutop (default): htop-style curses TUI fusing three sections —
DEVICES (DAX pools from the Resource Manager), INSTANCES (per-instance
allocated/used/slack from MaruServer), and STATS (compact per-op summary).
* background poller thread keeps the UI smooth / keys responsive even
when a server is slow or unreachable (per-section "(unavailable)")
* maru-server auto-discovery by scanning /proc for their --port, so
multiple servers are covered without knowing ports (-p/--host pins one)
* per-instance per-DAX-device gauges (which device each instance landed
on and how much of it it holds)
* colored gauges + [s]ort/[i]nterval/[q]uit keys; --once for plain text
- subcommands: marutop pool | usage | stats | device (init|show|clear)
- RM stays a fixed well-known address (:9850, --address to override) since
it is a singleton daemon; only MaruServers vary and are discovered.
protocol: InstanceUsage gains a `devices` map (dax_path -> bytes), resolved
server-side via MaruShmClient.get_dax_path; backward compatible (defaults {}).
pyproject: register `marutop = maru_tools.cli:main` and package maru_tools.
docs: rewrite tools/README + README around marutop. tests updated (import
from maru_tools.usage; MockShmClient.get_dax_path; devices-breakdown test).
Split the live TUI into two screens instead of stacking STATS under the overview (which overflowed short terminals and collided with the footer): - Overview: DEVICES + INSTANCES, now with up/down instance selection (▶ row highlight). - STATS view: `enter` on the selected instance opens a full-screen per-op table (count / avg / min / max latency / hit% / bytes, busiest op first); `esc`/`left` returns. Server stats are keyed by client_id == the instance's owner_instance_id, so the table is exactly that instance's ops. Needs MARU_STAT=1 on the client (otherwise a hint is shown). The background poller now stores the raw stats_manager dict and summarizes per instance_id at render time. Section drawing never touches the reserved footer row.
Promote the drill-in STATS screen from a static op table to the full `marutop stats` dashboard, scoped to the selected instance: - op table with count / delta / avg / min / max latency + an activity sparkline per op - ↑↓ selects an op; a detail box shows count / hit / miss / bytes, a hit-rate bar, and throughput (MB/s) - a min/avg/max latency line graph over time for the selected op The background poller now accumulates bounded per-(server, client_id) op history (counts, latency, sparkline) each tick; the UI renders from that, so graphs stay smooth and network latency never blocks input. Reuses the stats dashboard's sparkline primitive; the latency graph is drawn with the live palette. `marutop stats` stays as the port-pinned standalone dashboard.
hyunyul-XCENA
force-pushed
the
feat/marutop
branch
from
fb9317c to
1fd4516
Compare
youngrok-XCENA
left a comment
youngrok-XCENA
left a comment
There was a problem hiding this comment.
리뷰 요약 (상세는 인라인 코멘트 참조)
🔴 Critical 1건 (머지 차단): MaruShmClient.get_dax_path() 미구현 → 실서버 GET_USAGE 전체 실패. 최신 PR head + 최신 main + 클래스 상속까지 확인했고, 정의는 테스트 목에만 있습니다. 유닛 테스트는 목 치환 때문에 초록입니다.
🟡 Medium 2건: (1) 신규 maru_tools/* 8파일 SPDX 헤더 누락, (2) _Poller 히스토리 dict 무한 증가(사라진 서버 키 미제거).
통합 방향(단일 진입점 · 백그라운드 폴러 · 하위호환 shim)과 TUI 설계는 좋습니다. InstanceUsage.devices 와이어 변경도 하위호환 확인했습니다. Critical 해결 + 실클라이언트 테스트 추가 후 재요청 부탁드립니다.
| ] | ||
| out: dict[str, dict[str, int]] = {} | ||
| for region_id, instance_id, length in snapshot: | ||
| dax_path = self._client.get_dax_path(region_id) or "(unknown)" |
There was a problem hiding this comment.
🔴 Critical — 실서버에서 GET_USAGE 전체가 깨집니다 (머지 차단).
여기서 호출하는 self._client.get_dax_path(region_id) 가 프로덕션에서는 실제 MaruShmClient 인데, 이 클래스에는 get_dax_path() 가 없습니다.
확인한 것 (최신 PR head 1fd4516 = GitHub head SHA, 최신 main, 상속까지):
- 트리 전체에서
def get_dax_path정의는tests/unit/conftest.py의 목 1곳뿐 MaruShmClient공개 메서드 =is_running / stats / alloc / free / mmap / munmap / close(base class 없음,__getattr__없음)- 최신 main 에도 없음
server.get_usage() 가 devices_by_instance() 를 무조건 호출하고, rpc_handler_mixin._handle_message 는 핸들러 예외를 감싸지 않으므로 AttributeError 가 그대로 전파되어 GET_USAGE RPC 자체가 실패합니다. → 기존 GET_USAGE 소비자 + marutop usage + marutop 라이브 INSTANCES 섹션이 모두 영향받습니다.
수정 방향: MaruShmClient 에 공개 get_dax_path(region_id) 추가. 이미 _path_cache: region_id → path 가 alloc/mmap 시 채워지므로 최소 구현은 return self._path_cache.get(region_id), 캐시 미스 시 GET_ACCESS/RM 라운드트립으로 보강하면 docstring("RM-backed, cached")과 일치합니다.
| def free(self, handle): | ||
| pass | ||
|
|
||
| def get_dax_path(self, region_id): |
There was a problem hiding this comment.
위 Critical 과 연결 — 유닛 테스트가 초록인 이유가 여기입니다.
autouse fixture 가 maru_server.allocation_manager.MaruShmClient 를 이 MockShmClient 로 치환하고, 이 목에만 get_dax_path 가 stub 으로 존재합니다. 실제 클라이언트엔 없는 메서드라 프로덕션 경로는 테스트가 잡지 못합니다.
권장: 목이 아닌 실제 MaruShmClient 를 쓰는 통합/smoke 테스트(또는 real-server GET_USAGE 검증)를 하나 추가하면 이 사각지대가 재발하지 않습니다.
| @@ -0,0 +1,11 @@ | |||
| """Maru monitoring and admin tooling. | |||
There was a problem hiding this comment.
🟡 Medium — SPDX 헤더 누락.
신규 maru_tools/ 8개 파일(__init__.py, cli.py, _common.py, live.py, pool.py, stats.py, usage.py, device.py)이 전부 docstring 으로 바로 시작합니다. 기존 패키지 모듈(maru_server/*, maru_common/*, maru_handler/*)은 전부 첫 줄이 # SPDX-License-Identifier: Apache-2.0 입니다.
maru pre-commit 은 ruff 만 돌려 하드 CI 게이트는 아니지만, 패키지 일관성 차원에서 8개 파일 모두 SPDX 헤더 추가를 권장합니다. (참고: tools/*.py shim 은 shebang 스타일이라 별개 규약)
| targets = self._targets(self_pid) | ||
| live_keys = {(h, p) for h, p, _ in targets} | ||
| # Drop clients for servers that disappeared. | ||
| for key in list(clients): |
There was a problem hiding this comment.
🟡 Medium — _Poller 히스토리 dict 무한 증가.
이 루프는 사라진 서버의 clients 는 pruning 하지만, _h_count / _h_lat / _h_spark / _h_accum (키 = (label, client_id)) 은 절대 제거하지 않습니다.
자동 발견 모드로 포트/인스턴스를 계속 갈아끼우는 벤치를 장시간 지켜보면 키 집합이 무한 증가합니다 (각 value 리스트는 bounded 지만 키 개수는 unbounded).
수정: 이 pruning 루프에서 live_keys 에 없는 label 의 히스토리 키도 함께 drop 하면 됩니다.
kihwan-XCENA
left a comment
kihwan-XCENA
left a comment
There was a problem hiding this comment.
코드 감사합니다.
non-MP에서 잘 동작해요!
근데, 동작시키다 보니 marutop 프로그램 실행시킬 때 아래 같은 에러 로그가 뜨더라고요.
get_dax_path 구현이 MaruShmClient에 없다고 하는데 현율님은 이런 에러 안뜨셨나요?
[2026-07-08 16:01:18,535] maru ERROR: Error handling message (rpc_server.py:78:maru_server.rpc_server)
Traceback (most recent call last):
File "/home/kihwan/maru/maru_server/rpc_server.py", line 66, in start
response = self._handle_message(header.msg_type, request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/rpc_handler_mixin.py", line 59, in _handle_message
return handler(request)
^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/rpc_handler_mixin.py", line 246, in _handle_get_usage
usage = self._server.get_usage()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/server.py", line 310, in get_usage
devices_by_instance = self._allocation_manager.devices_by_instance()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/allocation_manager.py", line 229, in devices_by_instance
dax_path = self._client.get_dax_path(region_id) or "(unknown)"
^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MaruShmClient' object has no attribute 'get_dax_path'
[2026-07-08 16:01:19,559] maru ERROR: Error handling message (rpc_server.py:78:maru_server.rpc_server)
Traceback (most recent call last):
File "/home/kihwan/maru/maru_server/rpc_server.py", line 66, in start
response = self._handle_message(header.msg_type, request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/rpc_handler_mixin.py", line 59, in _handle_message
return handler(request)
^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/rpc_handler_mixin.py", line 246, in _handle_get_usage
usage = self._server.get_usage()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/server.py", line 310, in get_usage
devices_by_instance = self._allocation_manager.devices_by_instance()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kihwan/maru/maru_server/allocation_manager.py", line 229, in devices_by_instance
dax_path = self._client.get_dax_path(region_id) or "(unknown)"
^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MaruShmClient' object has no attribute 'get_dax_path'
…ening Critical: `MaruShmClient` had no `get_dax_path()` on main (it lived only on the still-open plugin branch and the test mock), so `server.get_usage()` -> `AllocationManager.devices_by_instance()` raised AttributeError on a real server, failing the whole GET_USAGE RPC (marutop usage / live INSTANCES + any GET_USAGE consumer). - add public `MaruShmClient.get_dax_path(region_id)` — served from the `_path_cache` populated at alloc()/mmap() (every region the server allocated is present); cache miss returns None. - harden `devices_by_instance()`: resolve device best-effort (getattr + guard), falling back to "(unknown)" so GET_USAGE never fails on device breakdown. - add tests/unit/test_shm_get_dax_path.py: binds the *real* MaruShmClient (before the autouse mock patch) to catch the mock-vs-real gap. - SPDX headers on the 8 new maru_tools/*.py files. - prune _Poller op history for disappeared servers (keys were unbounded under auto-discovery churn).
|
리뷰 감사합니다 🙏 🔴 Critical —
사각지대(목이 초록으로 가림)
🟡 Medium 1 — SPDX 헤더: 신규 🟡 Medium 2 — @kihwan-XCENA 실행 시 뜨던 재리뷰 부탁드립니다. |
youngrok-XCENA
left a comment
youngrok-XCENA
left a comment
There was a problem hiding this comment.
리뷰 반영 확인 — Approve ✅
b0e49e7 에서 직전 리뷰 3건 모두 반영된 것 확인했습니다.
- 🔴 Critical: 실제
MaruShmClient.get_dax_path()구현 +devices_by_instance()방어(getattr+try/except, fallback"(unknown)") → 실서버GET_USAGE정상 동작. 목-실제 갭을 겨냥해 autouse 패치 이전에 실제 클라이언트를 바인딩하는tests/unit/test_shm_get_dax_path.py회귀 테스트까지 추가된 점 좋습니다. - 🟡 Medium:
maru_tools/*.py8파일 SPDX 헤더 추가. - 🟡 Medium:
_Poller히스토리 dict 를 사라진 서버(label) 기준으로 pruning — auto-discovery churn 시 키 누수 해소 (락 하에서, transient 에러 서버는 보존).
통합 방향(단일 진입점 · 백그라운드 폴러) · TUI 설계 · 하위호환(shim, InstanceUsage.devices) 모두 견고합니다. LGTM 👍
3 participants
🤔 Background & Motivation (Why)
maru shipped four separate
tools/*.pymonitors (pool / usage / stats / device-admin), each invoked ad-hoc viapython -m tools.Xorpython tools/X.py. There was no single entry point, invocation styles were inconsistent, andpool_monitor/usage_monitorduplicated their formatting helpers. Answering "how full is each device / who holds what / how fast are the ops" meant juggling three tools.🏗️ Design Changes
maru_toolspackage + amarutopconsole script ([project.scripts]). The oldtools/*.pybecome thin back-compat shims (python -m tools.pool_monitorstill works).InstanceUsage(GET_USAGE) gains adevices: dict[dax_path -> bytes]field. Backward compatible — defaults to{}, so older servers/clients are unaffected.:9850,--addressto override) since it is a singleton daemon; only MaruServers vary and are auto-discovered.📝 Implementation Details
marutop(default) is anhtop-style curses TUI with two screens:maru-serverprocesses are auto-discovered by scanning/procfor their--port, so multiple servers are covered without knowing ports;-p/--hostpins a single (possibly remote) server.enteron a selected instance) — a full per-instance dashboard: op table (count / delta / avg / min / max latency + an activity sparkline), an↑↓-selected op detail box (hit-rate bar, throughput), and a min/avg/max latency graph over time.esc/←returns.marutop pool | usage | stats | device (init|show|clear).(unavailable)).MaruShmClient.get_dax_path(RM-backed, cached) to populateInstanceUsage.devices.MARU_STAT=1; the STATS view shows a hint otherwise.✅ Tests
maru_tools.usage;MockShmClientgainsget_dax_pathmaru-server; curses views verified via fake-window render checks + pty smoke🔗 Related Issues (optional)
🌿 Related PRs (optional)
📦 Release Note (for auto-generation / write in English)
NEW
marutop— a single monitoring/admin CLI. The defaulthtop-style TUI fuses DAX pool usage, per-instance allocation (with per-device breakdown), and a per-instance op-stats dashboard; subcommandspool/usage/stats/device. Oldtools/*.pyinvocations still work via shims.CHANGED
GET_USAGEresponses now include a per-instancedevicesmap (dax_path → bytes); backward compatible (empty from older servers).FIXED
IMPORTANT NOTES
marutoprequire clients to run withMARU_STAT=1.🤖 Generated with Claude Code