Skip to content

Add LingTai agent event index baseline#394

Draft
feichai0017 wants to merge 2 commits into
mainfrom
feature/lingtai-agent-index-baseline
Draft

Add LingTai agent event index baseline#394
feichai0017 wants to merge 2 commits into
mainfrom
feature/lingtai-agent-index-baseline

Conversation

@feichai0017

@feichai0017 feichai0017 commented Jul 3, 2026

Copy link
Copy Markdown
Member

Summary

This PR adds a LingTai-facing agent event-index surface to nokv-agent and keeps it separate from the NoKV DFS namespace tool surface. LingTai logs/events.jsonl remains the source of truth; the Holt-backed NoKV index is a rebuildable derived sidecar intended to replace tui/internal/sqlitelog query paths, not to expose a SQL compatibility layer.

Changes include:

  • split the existing NoKV namespace tool surface into crates/nokv-agent/src/namespace.rs while preserving the latest main/workbench P0 tool behavior;
  • add nokv_agent::event with typed records, projection, codec, key layout, ingest, store trait, and Holt implementation;
  • add nokv-agent lingtai ... CLI commands for ingest, coverage, session rows, errors, molt/recent probes, TUI clear completion, notification navigation, parsed notification blocks/snapshots, lifecycle filters, facets, and tool traces;
  • preserve full fields_json for SQLite replacement semantics by externalizing oversized payloads into a Holt payloads tree and transparently rehydrating records on read;
  • add a LingTai workload extractor and lingtai-index-bench benchmark;
  • add docs for the crate boundary, LingTai adapter mapping, benchmark arms, and measured tradeoffs;
  • upgrade workspace holt to 0.8.1.

LingTai replacement scope

The intended adapter mapping is function-level, not raw SQL-level:

LingTai sqlitelog call NoKV-agent replacement
QueryEventsIndexCoverage nokv-agent lingtai coverage --source-file logs/events.jsonl
StreamSessionEvents nokv-agent lingtai session-rows
QueryErrorEvents nokv-agent lingtai errors
HasTUIClearCompletionEvent nokv-agent lingtai clear-completion --source-offset <offset>
QueryMoltSessionWindows nokv-agent lingtai molt-windows
QueryRecentMoltTimes nokv-agent lingtai recent --type psyche_molt
QueryRecentRefreshCompleteTimes nokv-agent lingtai recent --type refresh_complete
QueryNotificationBlocks nokv-agent lingtai notification-blocks
QueryNotificationBlockSnapshots nokv-agent lingtai notification-block-snapshots
QueryNotifications nokv-agent lingtai notification-events or lifecycle-filtered notifications
QueryNotificationByID nokv-agent lingtai notification-by-id --event-id <id>
QueryNotificationBefore nokv-agent lingtai notification-before --event-id <id>
QueryNotificationAfter nokv-agent lingtai notification-after --event-id <id>

Current completeness details:

  • notification-blocks parses notification_pair_injected into the same summary/meta/source shape used by LingTai QueryNotificationBlocks, and still returns raw events for diagnostics.
  • notification-block-snapshots parses both modern _meta envelopes and legacy payload/meta rows used by QueryNotificationBlockSnapshots, and still returns raw events.
  • errors mirrors LingTai's current default types: aed_attempt, aed_exhausted, and refresh_init_error; rows without a non-empty error field are skipped.
  • notification-by-id follows LingTai's current SQL and returns the row by id without requiring the type to contain notification.
  • Full fields_json is preserved. Event records larger than Holt's single-value limit store fields_json in chunked payload records and rehydrate it before returning any public EventRecord or session row.

Adapter responsibilities that intentionally remain outside NoKV-agent:

  • keep the existing JSONL fallback when ingest fails, coverage is stale, or coverage does not reach the JSONL tail;
  • do not replace LingTai's live JSONL tail path in this PR;
  • if the adapter shells out to nokv-agent, keep LingTai's worker-local timeout, freshness cache, and stale-on-error fallback around molt-windows.

Benchmark evidence

The benchmark compares four arms:

  • sqlite_current_v1: exported LingTai logs/log.sqlite;
  • sqlite_projected_v1: current SQLite plus query-shape indexes;
  • sqlite_rebuilt_from_jsonl_v1: SQLite rebuilt from the same JSONL source used by NoKV-agent;
  • nokv_agent_index_v1: Holt-backed NoKV-agent index.

On the filtered 2026-06-30 LingTai export, exported SQLite and JSONL coverage differed, so equality claims use the rebuilt SQLite arm. Rows/fingerprints matched for the public sqlitelog-shaped queries. Tool facet fingerprints intentionally differ because NoKV-agent parses stringified tool_args into semantic projections while SQLite JSON expressions do not.

Pre-completeness benchmark highlights, before the full-fields_json chunking change:

Query SQLite current p50 SQLite projected p50 JSONL-rebuilt SQLite p50 NoKV-agent p50
coverage 120.339 ms 109.028 ms 109.913 ms 0.001 ms
tool/action grouping 117.273 ms 1.418 ms 1.788 ms 0.028 ms
tool-name grouping 109.305 ms 0.845 ms 1.063 ms 0.028 ms
TUI clear completion 0.187 ms 0.224 ms 0.225 ms 0.005 ms
notification before/after 0.011 / 0.011 ms 0.010 / 0.010 ms 0.012 / 0.010 ms 0.004 / 0.006 ms
notification by id 0.008 ms 0.007 ms 0.008 ms 0.014 ms
notification events 50 0.072 ms 0.047 ms 0.049 ms 0.495 ms
latest notification block/pair 0.188 / 0.074 ms 0.024 / 0.020 ms 0.022 / 0.017 ms 0.178 / 0.095 ms
recent molt/refresh times 0.012 / 0.012 ms 0.017 / 0.015 ms 0.016 / 0.015 ms 0.051 / 0.060 ms
session row replay 568.077 ms 578.335 ms 628.612 ms 783.415 ms

The same pre-completeness run measured NoKV-agent ingest at 7,173.482 ms and the Holt index at 5,610,024,663 bytes. Because this PR now preserves oversized fields_json instead of compacting it, index size must be remeasured before making a final storage-footprint claim. The current evidence still supports the optimization direction for coverage, semantic facets, TUI clear completion, and direct notification neighbor traversal; SQLite remains faster for several small point/page queries and full session replay.

Boundary notes

  • nokv-agent remains transport-free and does not depend on nokv-client, nokv-protocol, or nokv-control.
  • The event index opens an independent agent-index Holt directory and does not open or mutate a NoKV metadata directory.
  • Session index values store event-id pointers instead of duplicating fields_json; rows are reconstructed from the canonical event record.
  • Product APIs do not hide benchmark-only answer files or task-specific shortcuts.

Validation

cargo fmt --all -- --check
cargo test -p nokv-agent
cargo clippy -p nokv-agent --all-targets -- -D warnings
cargo check -p nokv-bench --bin lingtai-index-bench
cargo clippy -p nokv-bench --bin lingtai-index-bench -- -D warnings
git diff --check
cargo tree -p nokv-agent -e normal | rg "nokv-(client|protocol|control)" || true

Draft status

This should remain draft until the LingTai-side adapter is implemented and the benchmark is rerun after full-fields_json chunking. The main open product decision is whether LingTai should call the CLI initially or link the Rust API directly.

@feichai0017 feichai0017 moved this to In Progress in NoKV Delivery Jul 3, 2026
@feichai0017
feichai0017 marked this pull request as ready for review July 6, 2026 10:13
Copilot AI review requested due to automatic review settings July 6, 2026 10:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Signed-off-by: eric_song <songguocheng348@gmail.com>
Signed-off-by: eric_song <songguocheng348@gmail.com>
@feichai0017
feichai0017 force-pushed the feature/lingtai-agent-index-baseline branch from 9e16bea to e09b6e2 Compare July 8, 2026 00:07
@feichai0017
feichai0017 marked this pull request as draft July 8, 2026 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants