Skip to content

feat(datagen): add single-log checkpoint store#161

Merged
beinan merged 1 commit into
mainfrom
design/datagen-checkpoint-schema
Jul 21, 2026
Merged

feat(datagen): add single-log checkpoint store#161
beinan merged 1 commit into
mainfrom
design/datagen-checkpoint-schema

Conversation

@beinan

@beinan beinan commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a first-class DatagenStore backed by one append-only Lance log, avoiding cross-dataset consistency hazards
  • model item lifecycle, field deltas, failures, terminals, no-op step completion, deterministic retry ids, fold, and trajectory reads
  • use multi-shard MemWAL ingest with a resident writer, retry deduplication, per-shard cleanup, and row-id blob point reads
  • document the single-log schema and re-export the public API from lance-context

Testing

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test -p lance-context-core datagen --no-fail-fast (8 passed)
  • cargo test -p lance-context-core --no-fail-fast: 148 unit tests passed; the existing rollout wal_merge_generation_cleanup integration test fails on current main after perf(rollout): reuse a resident MemWAL ShardWriter across appends #159 with 32 rows observed for 30 appends
  • uv run ruff format --check python/
  • uv run ruff check python/
  • uv run pyright
  • uv run pytest: 181 passed, 5 skipped, 1 xfailed; 10 unrelated current-main failures in stale DummyInner signatures and moto S3 bucket visibility

@beinan
beinan merged commit 9d074f7 into main Jul 21, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant