Skip to content

feature(monitor): post-capture filters + JSON/CSV export#174

Open
jamby77 wants to merge 2 commits into
feature/monitor-web-live-tailfrom
feature/monitor-web-filters-export
Open

feature(monitor): post-capture filters + JSON/CSV export#174
jamby77 wants to merge 2 commits into
feature/monitor-web-live-tailfrom
feature/monitor-web-filters-export

Conversation

@jamby77
Copy link
Copy Markdown
Collaborator

@jamby77 jamby77 commented May 11, 2026

Summary

PR 11 of 25 in docs/plans/specs/monitor-command/plan-implementation.md. Stacked on top of #173 (PR 10). Closes Phase 2 frontend MVP. The session-detail page gains a Filters & export panel that filters the captured stream by command, client, key glob, and time window — then downloads the filtered slice as JSON or CSV via a new server endpoint.

Backend

  • monitor-line.parser.ts — pure module
    • parseMonitorLine(text){ts, tsRaw, db, addr, cmd, args, key, raw}. Handles backslash-escaped quotes, IPv6 bracket addresses, keyless commands (PING etc.).
    • matchesFilters(line, {command, client, key, afterTs, beforeTs}): case-insensitive command exact match, addr substring match, glob match on the first arg / key (* and ?), inclusive timestamp window.
    • lineToCsvRow + CSV_HEADER: RFC 4180 escaping for commas, quotes, newlines.
  • GET /monitor/sessions/:id/export?format=json|csv (+ filter query params) streams persisted chunks, parses each line, applies filters, emits either {sessionId, count, lines[]} JSON or a CSV with a header row. Content-Disposition: attachment; filename=monitor-session-<id>.<fmt>. 404 on unknown session. Default format is json when an unknown value is supplied.

Frontend

  • New FiltersAndExport component on the session-detail page (below the live tail).
    • Five inputs: Command, Client, Key glob, After (datetime-local), Before (datetime-local).
    • In-page preview count derived from the live-tail buffer (parser inlined to avoid a server round-trip on every keystroke). Export buttons hit the server endpoint, which sees the full session.
    • Two <a download> buttons styled as outline Buttons — clean browser-native download with the URL carrying the current filters.
  • useMonitorTail lifted from TailView up to MonitorSession so the Filters panel and the TailView share one buffer.

Test plan

  • SKIP_DOCKER_SETUP=true pnpm --filter api test -- --testPathPatterns "monitor|capture-sessions|health-gate|provider-detector|acl-checker|preflight|capture-writer|demo-mode|tail|monitor-line" → 184 tests across 14 suites, all pass
  • pnpm --filter web exec tsc --noEmit → exit 0
  • Run dev: MONITOR_DEV_PREVIEW=true pnpm dev:api and VITE_MONITOR_DEV_PREVIEW=true pnpm dev:web
  • Start a session, send mixed traffic, let it complete
  • curl '/monitor/sessions/:id/export?format=json'{count, lines[]} with parsed structure
  • curl '/monitor/sessions/:id/export?format=json&command=GET' → filtered subset
  • curl '/monitor/sessions/:id/export?format=json&key=user:*' → globbed subset
  • curl '/monitor/sessions/:id/export?format=csv' → starts with ts,ts_raw,db,addr,cmd,args,key
  • Browser: navigate to a completed session, type GET in Command field → preview "Buffer match: N of M lines" updates
  • Add user:* in Key glob → preview narrows further; Export-link URL carries command=GET&key=user%3A*
  • Click Export JSON → file downloads with monitor-session-<id>.json; opens in browser and matches the preview count
  • Click Export CSV → file downloads with monitor-session-<id>.csv; first line is the header

Notes for reviewers

  • IPv6 bracket addresses caught my parser early — the initial header regex used a greedy [^\]]+ that stopped at the inner ] of [::1]:port. Fixed with a non-greedy (.+?)\]\s+" anchor. Covered by a unit test.
  • The frontend parser is duplicated. The in-page preview count needs to filter the live-tail buffer client-side; I inlined a small parser in filters-and-export.tsx to avoid round-tripping on every keystroke. The parser is intentionally simpler than the backend's (no args array, no escape decoding) since it only feeds filter predicates. If we ever surface the parsed fields in the UI, the right move is a shared @betterdb/shared parser — flagged.
  • AUTH redaction in the CSV output above is a server-side feature in iovalkey's MONITOR stream (visible in the docker dev run — Valkey itself redacts AUTH args to (redacted)), not something this PR adds. We get it for free from the underlying protocol.
  • Net diff: 707 lines. Above the 400 target. Composition: parser 142 + parser tests 159 + filters-and-export panel 191 + controller endpoint 83 + controller tests 92 + small wiring. Tight enough to review as one slice but a parser-only split would have left an unverifiable PR.
  • Screenshot at docs/assets/pr11-filters-export.png.

Stacked PR

Base branch is feature/monitor-web-live-tail (#173), so the diff shown is ONLY PR 11 changes.

Phase 2 MVP done

This closes Phase 2 (frontend MVP). Next is Phase 3: cross-reference engine + UI panel — the actual differentiator the spec calls out.

@jamby77 jamby77 force-pushed the feature/monitor-web-live-tail branch from 0f03629 to e8e7644 Compare May 12, 2026 11:10
@jamby77 jamby77 force-pushed the feature/monitor-web-filters-export branch from 0502b31 to c2e8cb8 Compare May 12, 2026 11:12
@jamby77 jamby77 force-pushed the feature/monitor-web-live-tail branch from e8e7644 to b90faee Compare May 12, 2026 12:39
@jamby77 jamby77 closed this May 12, 2026
@jamby77 jamby77 force-pushed the feature/monitor-web-filters-export branch from c2e8cb8 to b90faee Compare May 12, 2026 12:39
@github-actions github-actions Bot locked and limited conversation to collaborators May 12, 2026
@jamby77 jamby77 reopened this May 13, 2026
@jamby77 jamby77 force-pushed the feature/monitor-web-live-tail branch from 2d42d85 to 997fbc8 Compare May 13, 2026 12:35
jamby77 added 2 commits May 13, 2026 15:35
Closes Phase 2 frontend MVP. The session-detail page gains a Filters &
export panel that filters the captured stream by command, client,
key glob, and time window — then downloads the filtered slice as JSON
or CSV via a new server endpoint.

Backend:
- monitor-line.parser.ts: pure module
  - parseMonitorLine(text) → {ts, tsRaw, db, addr, cmd, args, key, raw}
    Handles backslash-escaped quotes, IPv6 bracket addresses, keyless
    commands (PING etc.).
  - matchesFilters(line, {command, client, key, afterTs, beforeTs}):
    case-insensitive command exact match, addr substring match, glob
    match on the first arg / key (* and ?), inclusive timestamp window.
  - lineToCsvRow + CSV_HEADER: RFC 4180 escaping for commas, quotes,
    newlines.
- GET /monitor/sessions/:id/export?format=json|csv (+ filter query
  params) streams persisted chunks, parses each line, applies filters,
  and emits either {sessionId, count, lines[]} JSON or a CSV with a
  header row. Content-Disposition: attachment; filename=
  monitor-session-<id>.<fmt>. 404 on unknown session. Default format
  is json when an unknown value is supplied.

Frontend:
- New FiltersAndExport component on the session-detail page (below
  the live tail).
  - 5 inputs: Command, Client, Key glob, After (datetime-local),
    Before (datetime-local).
  - In-page preview count derived from the live-tail buffer (parser
    inlined to avoid a server round-trip on every keystroke). Export
    buttons hit the server endpoint which sees the full session.
  - Two <a download> buttons styled as outline Buttons — clean
    browser-native download with the URL carrying current filters.
- useMonitorTail lifted from TailView up to MonitorSession so the
  Filters panel and the TailView share one buffer.

Tests:
- monitor-line.parser.spec.ts: 22 cases covering all parser edge cases
  (escaped quotes, backslashes, IPv6, keyless commands, malformed
  input), all filter axes including glob wildcards and AND semantics,
  and CSV escaping.
- monitor.controller.spec.ts extended with 5 export cases (404,
  unfiltered JSON, command-filtered, CSV header, format fallback).

Total backend suite: 184 tests across 14 suites, all green (1 caught
during testing: IPv6 bracket address parsing — the initial header
regex was too greedy and stopped at the inner `]`; fixed with a
non-greedy capture anchored on `\]\s+"`).

Verification (Playwright, live):
- Started a 5s session, sent 30 SETs against foo + 30 GETs against
  user:* → session captured 70 lines (mix of AUTH/INFO/PING + SETs
  + GETs).
- Direct curl to /export?format=json → count: 70, cmds:
  ['AUTH', 'GET', 'INFO', 'PING', 'SET'].
- Filter ?command=GET → count: 30.
- Filter ?key=user:* → count: 30, all GET user:N.
- CSV export starts with header `ts,ts_raw,db,addr,cmd,args,key`.
- UI: typed `GET` and `user:*` into the form → preview text
  "Buffer match: 30 of 70 lines. Export uses the full session,
  server-side." matched the API. Export-link URLs updated with the
  filter query string in real time.

Screenshot at docs/assets/pr11-filters-export.png.

Part of PR 11 of 25 in
docs/plans/specs/monitor-command/plan-implementation.md (closes
Phase 2 frontend MVP).
Self-review fixes:
- Filter inputs (command/client/key) now go through trimmedOrUndefined
  so the buffer-preview client behaviour (which trims) and the export
  endpoint behaviour stay in sync. Without this, 'GET ' could match in
  the preview but return 0 rows in the export.
- Key glob filters are now bounded to 128 characters via cappedKeyFilter
  to defuse catastrophic backtracking via patterns like '*a*a*a*a*a*' —
  globToRegex compiles to a non-anchored .* chain that is ReDoS-prone
  against equally long captured keys.
@jamby77 jamby77 force-pushed the feature/monitor-web-filters-export branch from d307ec0 to 606edcb Compare May 13, 2026 12:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant