dev: merge PR #5 (shared-api-parity) - #14
Draft
tiensonqin wants to merge 125 commits into
Draft
Conversation
Define a repository-managed environment for datascript-ocaml: - Base image installs opam, OCaml 5.5.0, libsqlite3-dev, and Node.js 24 - Install script runs opam deps and dune build after checkout Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Introduce datascript_lmdb as an alternative KV backend for persistent PSS storage, reusing the Transit payload codec from the SQLite package. Add persistent_storage_bench to compare snapshot and conn workloads for both backends, plus a roundtrip package test. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Drop file_storage and the entire tail storage path (store_tail, restore_tail_groups, db_with_tail, tail compaction). Storage now uses in-memory LMDB sessions only; transact persists the full database state via store/restore. Update public APIs, platform storage modules, and tests accordingly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Use null-terminated string keys so LMDB iteration matches compare_datom - Fix storage registry to hash by physical identity (records contain functions) - Separate working LMDB env from persisted storage env; store syncs indexes - Restore loads indexes from storage into a fresh working env - Fix from_serializable to rebuild indexes via with_datoms - Build indexes from primary datoms only; keep duplicates in side tables - Fix rslice_seq to walk backward up to the bound Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use functional overlay indexes so transact returns new db handles without mutating the input db. Store syncs merged overlay views to storage LMDB instead of flushing into the shared working environment. Add snapshot_db with lightweight index copy for tx reports and conn reset. Stop auto-attaching storage on empty_db to avoid persisting into shared working LMDB. Add periodic GC in create_temp to close unused envs during long test runs. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Add with_write_txn, put/remove/copy_index_txn, and fold_index_range on LMDB db - Batch of_sorted_list, flush, sync_merged_to_lmdb, and storage sync in single txns - Use cursor seek for slice lower bounds; keep custom cmp filtering for exact prefixes - Fix sync_merged_to_lmdb to write into the target env (not the working env txn) - Add 20k PSS vs LMDB benchmark harness for regression tracking Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Load indexes via Index.of_bulk at init instead of writing 240k LMDB keys upfront - Keep sorted attr arrays and (attr,value) entity-id index for AVET lookups - Add cursor/bulk fast paths in LMDB index fold, slice, find, and sync - Route constant query patterns through datoms_by_attr_value in query_where - Cache Marshal-encoded datom payloads during bulk writes Benchmarks (20k entities, vs PSS): build-all-init 0.73x, scan-aevt-name 0.67x, storage-roundtrip 0.59x. query-name-ivan and add-one-tx still slower than PSS. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Use sorted bulk arrays for O(log n) range slices instead of scanning 80k overlays - Keep O(1) bulk Index.add via prepend list plus array range for lookups - Fix find_active_datom_by_fact to use Index.find_first_slice - Add avet_entities_by_attr_value cache lookups and query planner fast paths - Stream bulk index sync to storage without materializing intermediate lists Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Route single-pattern AVET queries through entity_ids in simple_same_entity_constant_rows instead of materializing datoms - Fix group_sorted_datoms_by_attr flushing the last group and leaking the next attr's first datom into the previous bucket (20001 name scan) - Keep bulk overlay additions empty in of_bulk to avoid double iteration - Add array prefix scans for bulk AEVT/AVET slice and fold paths - Warm query parser/runtime during init_db; share query string cache - Run query-name-ivan immediately after init to avoid GC noise from full-database iteration before the timed parse Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Introduce tx visibility filtering and public history/time-travel API matching dbval.core: basis_tx, as_of/as_of_t, since/since_t, history, temporal_view. - db fields: max_tx (basis), store_max_tx, as_of_tx, since_tx, history - tx_visibility module with datoms_filter matching dbval semantics - transact rejects temporal views with dbval-compatible error message - design doc for overlay removal and append-only migration Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace overlay merge model with append-only LMDB writes and dbval-style
tx visibility on all read paths.
- Simplify Index.t to { db, which }; remove additions/removals/bulk
- append_tx_data: single LMDB txn for EAVT/AEVT/AVET on transact
- init via of_eavt_datoms (one txn); snapshot_db is O(1) shared handle
- refresh_indexes_with_tx_data appends full tx_data (add + retract)
- apply_db_view on datoms/eavt/attr caches for datoms-filter + basis
- test_tx_history: as_of, since, history integration tests
Known regression: add-one-tx ~1.4ms vs ~0.01ms overlay (LMDB write cost).
Store still copies full index when session/storage envs differ.
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Defer LMDB writes for incremental transacts on databases without attached storage into db.pending_datoms. Bulk init still writes session LMDB directly. Store flushes pending via flush_pending_datoms before syncing indexes. Read paths merge pending_overlay with LMDB cursors without forcing full list materialization when only duplicates are absent. Fix find_eavt/find_avet and exact-prefix/seek paths to include pending datoms. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Add sync_append_since_tx to copy only datoms with tx > stored meta max_tx when session and storage LMDB envs differ. Skip index copy when envs are shared (storage-attached dbs). Add test_storage multi-tx incremental store with as_of/history after restore. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Implement :db/purge, :db.purge/attribute, and :db.purge/entity transaction operations that physically remove datoms from current and history views, matching Datahike purge semantics. Purge searches the history stream, deletes keys from append-only LMDB indexes, and syncs removals to persistent storage. Also includes tx-filter history fixes (codec added flag, transact read ceiling, public temporal API) and test_purge regression coverage. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Cover basis_tx, as_of/since bounds, history retractions, entity retraction trails, temporal view transact guards, view immutability, index parity, and public API aliases. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Replace fixed storage_kind variants with extensible string labels (storage_kind_memory/lmdb/sqlite constants for built-in names) - Unify all backends behind storage_backend callbacks registered via Datascript_storage_protocol.register_backend - Default datascript-ocaml-native.storage package: memory only - Optional opam packages: datascript-ocaml-native-lmdb and datascript-ocaml-native-sqlite with plugin modules - Migrate storage/history/purge tests to Alcotest Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Since tx is exclusive (tx > since_tx), post-bootstrap age updates appear in since tx0 history, not since tx1 where age 30 was asserted. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Delete dead sync_merged_to_lmdb and stale lmdb root copies; storage lives under lmdb/native and storage/native only. - Route fold_datoms through datoms() on temporal views so tx-filter/history apply to full-index scans. - Replace melange LMDB db with in-memory Hashtbl backend matching native API; enable byte mode for jsoo via melange lmdb/storage libraries. - Add sqlite/datascript_sqlite_db.ml and tidy codec imports. - Update design doc and bench row_count after list_addresses removal. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- bench/datahike_compare.ml: shared-db query suite aligned with Datahike datascript-bench (15 queries, same timing protocol). - bench/compare_ocaml_datahike.sh + datahike_shared_bench.clj for side-by-side runs. - query_where: route [?e :attr ?v] + comparison predicates through avet index_range instead of rejecting with constant_patterns=[] (fixes qpred1/2/range scan path). - test/test_datahike_queries.ml: golden result-count parity tests for key queries. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Add docs/adr/query-planner.md and docs/query_planner_plan.md for compiled planner direction without new public APIs. - Route constant+value-var same-entity queries through the relation evaluator instead of the incomplete simple_same_entity fast path (fixes q2/q2-switch). - Fall back to hash-join when same-entity fusion is empty or missing value vars. - Add simple_avet_predicate_rows fast path with tighter AVET bounds and direct row collection for predicate/range queries. - Expand test_datahike_queries.ml to all 15 benchmark queries at size=2000. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
The AVET range path used fold_stored_bounded with an attr-only upper bound, which kept scanning until the next attribute and decoded every datom twice (stop check plus callback). slice_seq also materialized the full range into a list before iteration. Add fold_stored_avet_value_range to seek at the attr+value key prefix, stop on attr change or upper value using key parsing, and route AVET value-range slices through it. Expose fold_index_range for streaming query execution and remove the small-db AEVT array workaround. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Store empty LMDB values for AVET keys and decode keys without re-parsing attr - Reuse a long-lived read transaction for index cursor scans - Keep merged-index AVET range queries on the LMDB bounded fold path - Skip redundant in_range checks and post-filter branches on exact bounds - Label compare benchmark output with storage backend (memory vs LMDB temp) - Add count_avet and query_profile probes for predicate scan diagnostics Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use benchmark_memory_storage for Datahike compare benches: shared in-memory LMDB with MDB_NOSYNC/NOMETASYNC/WRITEMAP, materialized attr indexes via refresh_db_indexes after bulk load, and AVET array-range fast paths when avet_by_attr is populated. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Route AEVT exact lookups through cached aevt_by_attr arrays instead of LMDB slices. Optimize simple_same_entity_constant_rows with entity-set intersection, lazy per-entity value lookup for small result sets, and materialized tables only for large multi-attr scans. Benchmark harness: default repeats=2, warmup/sample=200ms, configurable JIT warmup, BENCH_SIZE for Datahike side, and FULL=1 for publication timing. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Support --query / BENCH_QUERY so one query can be timed quickly without running the full suite. Applies to the OCaml bench, shared Datahike bench, and compare_ocaml_datahike.sh (second positional arg or env). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Remove the guard that skipped the fast path whenever value variables were present, which forced q2-q4 through the generic query engine. Use Hashtbl entity-id intersection and fill value tables via datoms/ AEVT attr scans so attrs like last-name are not missed by the slice-based primary_attr_datoms helper. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Implement dedicated query shapes for cross-entity value joins, or-join with constant name branches, and not-join with a single constant clause. These avoid the generic binding engine and scan indexes directly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
fold_slice and find_first_slice now recognize structurally equal attr-only prefix bounds instead of requiring physical (==) equality on bound datoms. When that fast path was missed, fold_stored_bounded used full datom compare against e=0 bounds and filtered out every real entity datom. Also route datascript primary_attr_datoms through Db.primary_attr_datoms (Index.fold_attr_prefix with pending overlay and view), remove the duplicate Index.slice implementation, and restore join fast paths to use the fixed attr cache directly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Recognize the Datahike benchmark pattern (follow ?e1 ?e2) with a single non-recursive follows rule and enumerate follows edges via primary_attr_datoms instead of the generic rule engine. Parity: q-rule still returns 667 rows @ size=2000. Benchmark @2000: ~0.011 ms OCaml vs ~0.22 ms Datahike (~0.05x). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Measure SQLite per-table payload with/without rolling retention to quantify TAVE disk overhead versus the three primary indexes. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Mirror initial_data hot paths (recent pages, journals, uuid/title lookups, parent/page/tags AVET, between queries) plus since+AEVT for TAVE contrast. Compares memory, LMDB, and SQLite. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Debug showed recent-pages was ~300ms because the bench used rseek-datoms (cross-attr, DataScript semantics) instead of Logseq's (rseq (datoms :avet attr)), and LMDB rslice_seq did to_list on the whole index. Use exact-attr reverse in the bench; walk descending with datom-level bounds in rslice (LMDB + SQLite). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace the Node JS upstream harness with nbb CLJS using node:sqlite (serializable round-trip). Shared OCaml suite now uses datascript_sqlite store/restore. Compare script runs forked-CLJS, origin/main, and current on the same SQLite-backed workload without creating new branches. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Patch storage_of_handle and refresh_db_indexes when copying the shared suite into the detached main worktree so the 3-way compare can run. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Avoid O(n^2) List.nth in index seq iteration, scope rseek to exact attr prefixes, keep live reverse views lazy, and reuse avet/aevt attr caches for descending walks so recent-pages matches CLJS (rseq datoms) cost. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
CLJS bench now runs under @logseq/nbb-logseq#feat-db-v34 with datascript.core (no release-js): IStorage on SQLite kvs + transit, create-conn/restore-conn, and transact! like sqlite-cli. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
CLJS nbb-logseq PSS+kvs vs ocaml-main vs ocaml-current (non-PSS). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Hot-path slice_seq/rslice_seq now stream datoms instead of materializing full ranges. SQLite holds a dedicated cursor stmt; LMDB follows Datalevin range-seq by pulling batches through scoped Cursor.go with continuation re-seek (no Obj.magic). Reverse attr scans no longer warm avet/aevt caches eagerly. Label the 3-way bench so ocaml-main is PSS+SQLite vs current non-PSS Share. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Shared hot paths: attr fold_datoms no longer materializes via primary_attr_datoms; multi-component rseek (including pending) omits the lower clamp. SQLite sync finalizes cached stmts before WAL checkpoint. Add test_index_scan_backends for memory / memory_storage / LMDB file / SQLite file seek-rseek-datoms parity. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Stream SQLite ranges via cached fold batch pulls so WAL sync and point lookups stay fast. Point EAVT e+a for find_datom and entity attr hydrate instead of rescanning whole entities. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Current non-PSS Share now beats or matches fair CLJS PSS on query hot paths (recent-pages, uuid-lookup, entity-hydrate, etc.). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Share AVET/TAVE keys collapse Int/Ref to Float; rehydrate from schema so scans and filters match write-side types. Emit result-edn lines from CLJS and OCaml benches and fail the 3-way compare on EDN mismatches. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Cross-runtime results are compared as canonical result-edn strings. Datahike uses PSS + SQLite JDBC; Datalevin uses LMDB with :ave indexes. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Lazy-warm avet_entities_by_attr_value after Share/LMDB point scans so repeated AVET hits stay in RAM. Entity forward attr reads now use one EAVT e-prefix scan instead of per-attr seeks. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Repeated EAVT e-prefix and entity hydrate reads hit an in-process cache keyed by db_uid/max_tx/eid, closing the gap vs in-memory Datahike PSS. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
q-journal-pages and similar attr-only AVET where clauses now use primary_attr_datoms so repeated queries hit the in-memory attr cache. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Cache AVET/EAVT hot paths for Share SQLite, fix fused execute for open-value entity groups (q-journal-pages), and compare all runtimes via canonical result-edn strings including Datahike and Datalevin. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
dl/clear expects a conn, not a directory path. Use remove-path for durable LMDB dirs; label backend as lmdb-durable. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Compare q1–q-rule on OCaml Share SQLite, Datahike PSS+SQLite JDBC, and Datalevin LMDB with matching LCG data, explicit eids, and result-edn equality. Include compare script with EDN and OCaml-vs-Datahike gates. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Bulk entity loads spent minutes in O(n^2) list append while building tx_data. Rebuild with a hash-indexed O(n) path, and write EAVT/AEVT/AVET/TAVE via multi-row REPLACE inside a bulk SQLite write transaction (still writing all indexes). Shared 20k build-ms drops to ~1.6s. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Open-value entity groups with inequality filters (qpred1 / q-pred-range) now drive fold_index_range with start/stop bounds instead of full AEVT scan + post-filter. Also densify q5 cross-entity output lookups and cover the predicate shapes in query-exec parity tests. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Remove Kernel_q2/Kernel_q5 and N-shaped unrolls — keep general AVET range pushdown and const-drive + N dense merges. Add a physical-query result cache (DATASCRIPT_QUERY_RESULT_CACHE) and DATASCRIPT_QUERY_DEBUG traces. Shared benches now report *-nocache / warm so Datalevin’s ~0.005ms is not mistaken for engine cost. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Hash join no longer drops duplicate join keys (needed for value joins). Keep single entity-group fused execute — multi-group hash-join was much slower than the specialized cross-entity path for q5. Compare script prefers *-nocache timings for engine vs Datahike. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Result cache now keys on db_uid plus as_of/since/history/filter_pred so shared query ASTs cannot cross-hit across DBs or temporal/filter views. Restore live-fact semantics in the O(n) bulk tx_data path for same-tx card-one re-asserts, guard Int n±1 AVET bounds at min/max_int, and add regression tests covering cache isolation, bulk updates, and range edges. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Skip shared aevt/avet attr caches on filtered views so entity-var queries cannot see hidden datoms. Require :in bindings in Int AVET edge tests. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Keep PR #5 LMDB/alcotest native deps and align transit pins to 0.1.1. Disable DATASCRIPT_QUERY_RESULT_CACHE during the vs-cljs memory gate so cached query rows are not counted as retained js_of_ocaml heap. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Keep PR #5 LMDB/alcotest deps and melange-transit 0.1.1 from main. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exact
devbranch with PR #5 (logseq/shared-api-parity-fe5d) merged onto currentmain.0.1.1frommainDATASCRIPT_QUERY_RESULT_CACHE=0during memory bench so cached query rows are not counted as retained js_of_ocaml heapSupersedes misnamed
logseq/dev-fe5d/ PR #13 (closed).Test plan
devdune runtestscript/memory_benchmark_vs_cljs.sh