feat(archive): add local-save archive with subscription UI#1442
Draft
wpfleger96 wants to merge 7 commits into
Draft
feat(archive): add local-save archive with subscription UI#1442wpfleger96 wants to merge 7 commits into
wpfleger96 wants to merge 7 commits into
Conversation
SQLite store in the Buzz nest (~/.buzz/archive/archive.db) with three tables: - archived_events — one raw event row per (identity, relay, id) - archived_event_scopes — many-to-many scope membership rows - save_subscriptions — which scopes the user has subscribed to save Four Tauri commands: - archive_events — batch-first; two proof paths (persistent vs ephemeral) - create_save_subscription — per-scope access probe before persisting - list_save_subscriptions — scoped to current identity + relay - delete_save_subscription — subscription row only; retention decoupled (v1) Persistent scopes (channel_h, referenced_e) use a batched authed /query to the relay as the access proof. Ephemeral scope (owner_p, kind 24200) uses local validation: valid sig/id, kind 24200, #p == identity, agent tag present, frame == telemetry, author == agent tag. Both paths are fail-closed. relay_url is canonicalized to WS-form via relay_ws_url_with_override everywhere so the same relay never splits rows across wss:// and https:// variants. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Three IMPORTANT defects from Thufir pass 1:
1. Kinds not enforced at archive time: add get_subscription_kinds() to
store.rs (returns Option<String> via OptionalExtension). Both paths
now parse the kinds JSON and reject events whose kind is not in the
subscription's allowed list.
2. Persistent proof path used unscoped relay filter: replace bare
{ids}-only /query with {ids, #h/#e: scope_value, kinds: allowed_kinds}.
The relay returning an event for this scoped filter IS proof of scope
membership — no local tag re-derivation needed, which also correctly
admits h-less events matched via StoredEvent.channel_id fallback.
3. Missing end-to-end tests: add plan_archive / commit_archive split
(extracted from archive_events for Send-safety and testability),
plus 14 new tests covering: persistent persists/drops, h-less event
via scoped filter proof, kind mismatch drops (both paths), no-sub
drops, mixed batch exact counts, ephemeral persists/drops,
ephemeral kind-not-in-subscription drops.
Send-safety: rusqlite::Connection is !Send. archive_events now follows
the managed_agents/persona_events.rs pattern — DB work in scoped blocks
that drop the connection before any .await — so the Tauri command future
is Send.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
archive/mod.rs exceeded the 1000-line gate (1201 lines per the check). Move plan_archive / query_buckets / commit_archive and their private types (Parsed, Bucket, ArchivePlan, BucketWithResult) to a new archive/pipeline.rs submodule. mod.rs: 1200 → 915 lines (gate counts 916, well under 1000). pipeline.rs: 312 lines. Zero logic changes — pure structural move. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Add the frontend layer for local-save archive (P3): - tauriArchive.ts: typed wrappers for all 4 Tauri commands. Decodes the SaveSubscription.kinds column (stored as a JSON text string in SQLite) into number[] once at the boundary. Emits a module-level subscription-change notification after create/delete so the sync manager reloads without manager instances threaded through props. Wire-shape: ArchiveCandidate fields use verbatim Rust names (raw_event_json, matched_scope, scope_type, scope_value) — Tauri 2 only camelCases top-level command arg names, not nested struct fields. - archiveSyncManager.ts: always-on manager started at AppShell mount via useArchiveSync(). Opens one live relay subscription per saved config (channel_h → #h, owner_p → #p, referenced_e → #e; all with limit: 0 for live-only). Buffers matched events and flushes to archive_events in batches (≥25 events or 2s idle). Resubscribes automatically on subscription-change signal; tears down cleanly on workspace switch, flushing any buffered events. - archiveSyncManager.test.mjs: 20 tests covering kinds decoding (valid, empty, non-array, wrong types, malformed JSON), preset kind arrays (messages/aux/all derived from shared constants — exact saved arrays asserted), subscription-change notifier contract, manager lifecycle (filter shapes, flush on batch size, resubscribe on create/delete, flush on destroy). - LocalArchiveSettingsCard.tsx: settings UI wired into SettingsPanels (new 'local-archive' section, Archive icon, no feature gate). Lists active subscriptions with delete; add flow for channel_h (channel picker from joined channels + kind presets) and owner_p (fixed kinds [24200]). Kind presets derived from CHANNEL_MESSAGE_EVENT_KINDS + KIND_STREAM_MESSAGE_DIFF, CHANNEL_AUX_EVENT_KINDS, and CHANNEL_EVENT_KINDS — never raw numeric literals. - SettingsPanels.tsx + SettingsView.tsx: local-archive section wired into the sidebar nav under Personal alongside custom-emoji. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Record<string, unknown> was not assignable to RelaySubscriptionFilter, which requires kinds: number[] and limit: number. Importing the type and narrowing the return type catches this at tsc rather than at runtime. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Two fixes from Thufir pass 1 review: 1. Kinds-aware resubscribe (IMPORTANT): the active-sub key now encodes scope+kinds (scopeType:scopeValue:sorted-kinds). When the same scope is upserted via create_save_subscription with a different preset, the stale key is absent from the wanted set and gets torn down; the open loop immediately starts a new subscription with the updated filter. No second bookkeeping map — the single wanted-set diff handles both removals and kind changes uniformly. 2. Constructor DI (MINOR): ArchiveSyncManager now accepts optional deps (relayClient, listSaveSubscriptions, archiveEvents, onSubscriptionChange, flushBatchSize, flushIdleMs) with production defaults. The TestableArchiveSyncManager copy in the test file is gone — all 8 manager tests now exercise the real class via makeManager(relay, archive). Adds test: manager_resubscribes_with_new_filter_when_kinds_upserted. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add local SQLite archive for relay messages (P1–P3).
What this does
Adds a complete local-save archive feature: users can subscribe to specific channels or scopes, the app opens live relay subscriptions for each saved config, and matched events are persisted to a per-identity SQLite database in the Buzz nest (
~/.buzz/archive/archive.db).Backend (P1–P2)
Four Tauri commands:
create_save_subscription,list_save_subscriptions,delete_save_subscription,archive_events.Access control at every stage:
create_save_subscriptionruns a per-scope access probe against the relay (NIP-98 authed/query). Channel memberships are verified via kind 39002 + open-channel fallback. Event readability viaidsfilter.owner_prestricted to current identity's own pubkey.archive_eventsre-verifies every candidate via scoped relay query ({ids, #h/#e, kinds}). The relay returning the event IS the proof of scope membership — no local tag re-derivation, so h-less channel events can persist when relay confirms them. Only events the relay returns are written.owner_p, kind 24200 observer frames): relay never stores these, so local validation applies instead (sig/id + kind + #p + agent tag + frame=telemetry + author == agent).Subscription kinds enforcement: both paths filter by the subscription's allowed kinds list before insert.
SQLite schema (
archive/store.rs): three tables —archived_events(raw JSON + metadata),archived_event_scopes(many-to-many),save_subscriptions. WAL mode. GC: raw event rows are removed when their last scope row is deleted. Idempotent upserts.Send-safety:
rusqlite::Connectionis!Send. All DB work is bracketed in syncplan_archive/commit_archivefunctions that drop the connection before any.await, matching the pattern inpersona_events.rs. Pipeline internals live inarchive/pipeline.rsto keepmod.rsunder the 1000-line file-size gate.Frontend (P3)
tauriArchive.ts— typed wrappers for all 4 commands. TheSaveSubscription.kindscolumn is stored as a JSON text string in SQLite; the wrapper decodes it tonumber[]once at the boundary, failing closed on malformed JSON.ArchiveCandidatestruct fields use verbatim Rust snake_case names (raw_event_json,matched_scope,scope_type,scope_value) — Tauri 2 only camelCases top-level command arg names, not nested struct fields. Emits a module-level subscription-change notification after create/delete.archiveSyncManager.ts— always-on manager started at AppShell mount viauseArchiveSync(). Opens one live relay subscription per saved config (channel_h→#h,owner_p→#p,referenced_e→#e; alllimit: 0for live-only). Buffers matched events and flushes toarchive_eventsin batches (≥25 events or 2s idle). Resubscribes automatically on subscription-change signal. Tears down cleanly on workspace switch, flushing buffered events.LocalArchiveSettingsCard.tsx— settings UI at Settings → Local Archive (Archive icon, no feature gate, opt-in by construction). Lists active subscriptions with delete. Add flow for:channel_h: channel picker from joined channels + kind preset checkboxes. Presets derived fromCHANNEL_MESSAGE_EVENT_KINDS + KIND_STREAM_MESSAGE_DIFF(Messages & posts),CHANNEL_AUX_EVENT_KINDS(Reactions, edits & deletions), andCHANNEL_EVENT_KINDS(All channel activity). No raw numeric literals.owner_p: fixed kinds[24200](agent observer frames).referenced_e: backend-only in v1 (no sane UI affordance for "paste an event id").Tests
Rust (39 tests,
src-tauri): persistent path (channel_h + referenced_e persist/drop, h-less events via scoped relay proof), kind mismatch drops on both paths, no-subscription drops, mixed batch exact counts, ephemeral path (valid/invalid frames, kind enforcement, missing tags, tampered events), store schema (idempotent upserts, GC, list scoping).JS (20 new tests,
archiveSyncManager.test.mjs): kinds decoding (valid, empty, non-array, wrong types, malformed JSON), preset kind arrays (exact saved arrays asserted, verified against constants — edits not misclassified as messages, forum kinds included), subscription-change notifier contract, manager lifecycle (filter shapes, flush on batch size, resubscribe on create/delete, flush on destroy).E2E (click-path, local)
With
just devfrom the worktree:sqlite3 ~/.buzz-dev/archive/archive.db 'SELECT id,kind FROM archived_events; SELECT scope_type,scope_value FROM archived_event_scopes;'Files changed