Skip to content

TAVE tx-window index + Logseq/shared query benches vs Datahike - #10

Merged
cursor[bot] merged 31 commits into
logseq/shared-api-parity-fe5dfrom
logseq/tx-window-index-fe5d
Aug 30, 2026
Merged

TAVE tx-window index + Logseq/shared query benches vs Datahike#10
cursor[bot] merged 31 commits into
logseq/shared-api-parity-fe5dfrom
logseq/tx-window-index-fe5d

Conversation

@tiensonqin

@tiensonqin tiensonqin commented Aug 29, 2026 •
edited by cursor Bot
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Fair Logseq / shared-people benches vs Datahike / Datalevin with canonical result-edn.
  • Share SQLite build: O(n) tx_data + batched index writes (EAVT/AEVT/AVET/TAVE all written); 20k build-ms ≈ 1.5s.
  • Datalevin-style query opts (general, not query-named kernels):
    1. Predicate pushdown: open-value + inequality filters → AVET fold_index_range (qpred1 / q-pred-range)
    2. Const AVET drive + N card-one AEVT merges (q3 / q-5-merge); specialized cross-entity value join for q5 (multi-group hash-join was slower)
    3. Multi-match hash join; result cache (default on; DATASCRIPT_QUERY_RESULT_CACHE=0 / *-nocache for engine timing)
    4. Debug: DATASCRIPT_QUERY_DEBUG=1

Shared people @ 20k (nocache engine path)

EDN matches Datahike and Datalevin. Every query faster than Datahike.

query OCaml Datahike Datalevin
build-ms 1531 2044 997
q1 0.02 1.55 0.74
q3 0.38 12.3 15.1
q5 3.5 1172 279
qpred1 3.3 13.9 3.9
q-pred-range 2.5 13.9 1.4
q-5-merge 2.2 183 3.6 
script/compare_shared_query_bench_3way.sh

Test plan

  • Predicate pushdown on fused open-value + comparison filters
  • q3 / q-5-merge fused const-drive; q5 specialized cross-entity (faster than Datahike)
  • Indexes still written on bulk build
  • Size-20k 3-way: EDN equal; OCaml nocache faster than Datahike on all queries
  • test_shared_queries + test_query_exec_parity
Open in Web Open in Cursor 

cursoragent and others added 5 commits August 29, 2026 12:02
@cursoragent @tiensonqin
Add design for TEAV tx-window index narrowing
3287ba1 
Document how to resolve wall-clock windows to tx bounds via :db/txInstant,
add a tx-ordered TEAV Share_index, and rewrite unbound scans under a window
so cost tracks window size instead of full-graph filter-after-scan.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Expand TEAV design: index choice, since/history layering, Mode B rules
ba493fa 
Clarify that EAVT/AVET remain primary; add selection matrix, candidate
discipline, API sketch, cost hints, and open decisions before implementation.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Document Logseq production query survey vs TAVE
684387b 
Survey logseq/logseq hot paths: entity/AVET-dominated; recency uses
:block/updated-at and :block/journal-day on AVET, not tx windows.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Require transparent planner path selection; no caller API change
e0e8f87 
Document that d/q and datoms stay stable; the engine auto-picks AVET
time ranges for Logseq shapes and may use TAVE only for tx-scoped views.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Implement TAVE rolling index with 30-day retention
43560c0 
Add tx|a|v|e TAVE Share_index (LMDB ds/tave, SQLite ds_tave), write it on
every tx, prune by configurable retention (default 30 days via txInstant),
and auto-use TAVE for since-bounded AEVT attr scans without API changes.
Also invalidate LMDB read txns after meta_set so store/restore stays fresh.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursor cursor Bot changed the title Design: TEAV tx-window index to narrow scans by t TAVE rolling tx-window index (30-day retention, API-transparent) Aug 29, 2026
cursoragent and others added 9 commits August 29, 2026 14:16
@cursoragent @tiensonqin
Add TAVE storage-ratio microbench
d4ba8d1 
Measure SQLite per-table payload with/without rolling retention to quantify
TAVE disk overhead versus the three primary indexes.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Add Logseq-shaped query microbench across storages
feae79d 
Mirror initial_data hot paths (recent pages, journals, uuid/title
lookups, parent/page/tags AVET, between queries) plus since+AEVT for
TAVE contrast. Compares memory, LMDB, and SQLite.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Fix Logseq recent-pages path and LMDB rslice full-scan
6ec4d8f 
Debug showed recent-pages was ~300ms because the bench used
rseek-datoms (cross-attr, DataScript semantics) instead of Logseq's
(rseq (datoms :avet attr)), and LMDB rslice_seq did to_list on the
whole index. Use exact-attr reverse in the bench; walk descending
with datom-level bounds in rslice (LMDB + SQLite).

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Bench Logseq queries via nbb+SQLite across three runtimes
12e6e8a 
Replace the Node JS upstream harness with nbb CLJS using node:sqlite
(serializable round-trip). Shared OCaml suite now uses datascript_sqlite
store/restore. Compare script runs forked-CLJS, origin/main, and current
on the same SQLite-backed workload without creating new branches.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Drop unused path require from nbb CLJS bench
fe13ff4 
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Make shared SQLite bench build on origin/main
dd455de 
Patch storage_of_handle and refresh_db_indexes when copying the shared
suite into the detached main worktree so the 3-way compare can run.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Speed up reverse attr scans for Logseq-style recent queries
4c1f58a 
Avoid O(n^2) List.nth in index seq iteration, scope rseek to exact attr
prefixes, keep live reverse views lazy, and reuse avet/aevt attr caches
for descending walks so recent-pages matches CLJS (rseq datoms) cost.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Use nbb-logseq PSS storage matching Logseq DB path
e835bbe 
CLJS bench now runs under @logseq/nbb-logseq#feat-db-v34 with
datascript.core (no release-js): IStorage on SQLite kvs + transit,
create-conn/restore-conn, and transact! like sqlite-cli.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Add 5k Logseq shared query 3-way bench results
8498faf 
CLJS nbb-logseq PSS+kvs vs ocaml-main vs ocaml-current (non-PSS).

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursor cursor Bot changed the title TAVE rolling tx-window index (30-day retention, API-transparent) TAVE tx-window index + Logseq 3-way query bench Aug 29, 2026
cursoragent and others added 12 commits August 29, 2026 16:21
@cursoragent @tiensonqin
Make Share SQLite and LMDB index scans lazy like upstream
22d82bf 
Hot-path slice_seq/rslice_seq now stream datoms instead of materializing
full ranges. SQLite holds a dedicated cursor stmt; LMDB follows Datalevin
range-seq by pulling batches through scoped Cursor.go with continuation
re-seek (no Obj.magic). Reverse attr scans no longer warm avet/aevt caches
eagerly. Label the 3-way bench so ocaml-main is PSS+SQLite vs current
non-PSS Share.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Cover lazy index scans on memory, LMDB, and SQLite
4dd41db 
Shared hot paths: attr fold_datoms no longer materializes via
primary_attr_datoms; multi-component rseek (including pending) omits the
lower clamp. SQLite sync finalizes cached stmts before WAL checkpoint.
Add test_index_scan_backends for memory / memory_storage / LMDB file /
SQLite file seek-rseek-datoms parity.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Speed Share SQLite streams and entity attr lookups
b7a59fd 
Stream SQLite ranges via cached fold batch pulls so WAL sync and
point lookups stay fast. Point EAVT e+a for find_datom and entity
attr hydrate instead of rescanning whole entities.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Refresh 5k 3-way Logseq query bench after SQLite/entity fixes
d3958fd 
Current non-PSS Share now beats or matches fair CLJS PSS on query
hot paths (recent-pages, uuid-lookup, entity-hydrate, etc.).

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Restore AVET value types and compare Logseq results as EDN
7e30324 
Share AVET/TAVE keys collapse Int/Ref to Float; rehydrate from schema
so scans and filters match write-side types. Emit result-edn lines from
CLJS and OCaml benches and fail the 3-way compare on EDN mismatches.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Add Datahike and Datalevin Logseq query benches with EDN equality
1a0db92 
Cross-runtime results are compared as canonical result-edn strings.
Datahike uses PSS + SQLite JDBC; Datalevin uses LMDB with :ave indexes.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Cache AVET point lookups and batch entity forward attrs
1fe5de2 
Lazy-warm avet_entities_by_attr_value after Share/LMDB point scans so
repeated AVET hits stay in RAM. Entity forward attr reads now use one
EAVT e-prefix scan instead of per-attr seeks.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Cache EAVT entity slices and entity forward attrs by basis
ea1234b 
Repeated EAVT e-prefix and entity hydrate reads hit an in-process cache
keyed by db_uid/max_tx/eid, closing the gap vs in-memory Datahike PSS.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Warm avet_by_attr for attr-only AVET scans on Share
b0cd9eb 
q-journal-pages and similar attr-only AVET where clauses now use
primary_attr_datoms so repeated queries hit the in-memory attr cache.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Beat Datahike on all Logseq queries; EDN string equality
550fe6f 
Cache AVET/EAVT hot paths for Share SQLite, fix fused execute for
open-value entity groups (q-journal-pages), and compare all runtimes
via canonical result-edn strings including Datahike and Datalevin.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Ignore Clojure .cpcache under bench/external
9694c50 
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Fix Datalevin durable LMDB bench open/cleanup
b0ec910 
dl/clear expects a conn, not a directory path. Use remove-path for
durable LMDB dirs; label backend as lmdb-durable.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursor cursor Bot changed the title TAVE tx-window index + Logseq 3-way query bench TAVE tx-window index + Logseq 5-way query bench Aug 29, 2026
@cursoragent @tiensonqin
Add shared people-query 3-way bench vs Datahike and Datalevin
8d68abf 
Compare q1–q-rule on OCaml Share SQLite, Datahike PSS+SQLite JDBC, and
Datalevin LMDB with matching LCG data, explicit eids, and result-edn
equality. Include compare script with EDN and OCaml-vs-Datahike gates.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursor cursor Bot changed the title TAVE tx-window index + Logseq 5-way query bench TAVE tx-window index + Logseq/shared query benches vs Datahike Aug 29, 2026
cursoragent and others added 4 commits August 29, 2026 23:31
@cursoragent @tiensonqin
Fix Share SQLite bulk build: O(n) tx_data + batched index writes
9f2a3a0 
Bulk entity loads spent minutes in O(n^2) list append while building
tx_data. Rebuild with a hash-indexed O(n) path, and write EAVT/AEVT/AVET/TAVE
via multi-row REPLACE inside a bulk SQLite write transaction (still writing
all indexes). Shared 20k build-ms drops to ~1.6s.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Push comparison predicates into AVET range scans on fused execute
c41a40a 
Open-value entity groups with inequality filters (qpred1 / q-pred-range)
now drive fold_index_range with start/stop bounds instead of full AEVT
scan + post-filter. Also densify q5 cross-entity output lookups and cover
the predicate shapes in query-exec parity tests.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Generalize query execute; add Datalevin-style result cache + debug
31066af 
Remove Kernel_q2/Kernel_q5 and N-shaped unrolls — keep general AVET
range pushdown and const-drive + N dense merges. Add a physical-query
result cache (DATASCRIPT_QUERY_RESULT_CACHE) and DATASCRIPT_QUERY_DEBUG
traces. Shared benches now report *-nocache / warm so Datalevin’s
~0.005ms is not mistaken for engine cost.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursoragent @tiensonqin
Fix Query_exec hash_join to multi-match; prefer nocache in 3-way
e3def84 
Hash join no longer drops duplicate join keys (needed for value joins).
Keep single entity-group fused execute — multi-group hash-join was much
slower than the specialized cross-entity path for q5. Compare script
prefers *-nocache timings for engine vs Datahike.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@cursor cursor Bot mentioned this pull request Aug 30, 2026
Open
7 tasks
@cursor
cursor Bot merged commit e3def84 into logseq/shared-api-parity-fe5d Aug 30, 2026
1 check was pending
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tiensonqin @cursoragent