diff --git a/docs/design/dedicated-demo-tables.md b/docs/design/dedicated-demo-tables.md new file mode 100644 index 00000000..274e8db5 --- /dev/null +++ b/docs/design/dedicated-demo-tables.md @@ -0,0 +1,250 @@ +# Dedicated demo corpus tables + +Status: implemented and deployed; recovery snapshot recorded on 2026-10-05. +This document records the selected design and the deployed implementation. The +production recovery completed with 27/27 demo sources READY, 2,562 published +chunks, and shared generation 27. The operator evidence is retained locally in +the ignored `.demo-originals/production-restoration-20261005/` directory. + +## Decision + +Keep one PostgreSQL database. Persist the public demo corpus in dedicated tables. +The existing upload API accepts a reserved demo namespace from an authorized +maintainer, and the existing Worker parses and publishes the result into these +tables. Any authenticated user can query the reserved namespace. Publication +occurs once per demo version, with no per-user content materialization. + +The delivered system supports direct queries against the public demo namespace. +Private namespace queries keep their existing corpus. The previous proposal to +create user-owned document references is superseded; clients query private and +demo namespaces separately and merge results when they need both. + +## End-to-end flow + +```mermaid +flowchart TD + Upload[Authenticated upload API] --> Resolve[Resolve namespace and write permission] + Resolve --> Job[Existing job ledger / original file storage] + Job --> Worker[Celery Worker / actual parser / chunk conversion] + Worker --> Publish[Shared publication algorithms / selected storage adapter] + Publish --> Private[Ordinary namespace: existing document tables] + Publish --> Demo[Reserved namespace: dedicated demo tables] + Demo --> Ready[Complete serving index / atomic revision activation] + Query[Authenticated retrieval API] --> ReadScope[Resolve caller and corpus] + ReadScope --> PrivateRead[Private corpus adapter] + ReadScope --> DemoRead[Public demo corpus adapter] + Private --> PrivateRead + Ready --> DemoRead + PrivateRead --> Engine[Existing scoring / agent tools / hydration] + DemoRead --> Engine + Engine --> Result[Citations / page assets / caller-scoped analytics] +``` + +The API decides the corpus before enqueuing a task. It records a trusted +publication target with the job. Worker retries use that persisted decision; +they cannot reclassify a job using caller-controlled parser metadata. + +## Routing and permissions + +The reserved namespace is fixed as `__knowhere_demo__`. Confirm it +does not collide with an existing private namespace before activation. + +| Request | Authorization | Storage | +| --- | --- | --- | +| Ordinary upload/query | Existing caller ownership | Existing tables | +| Demo upload/update/archive | Explicit maintainer permission | Demo tables | +| Demo query/list/read/assets | Authenticated catalog read access | READY demo revisions | + +All users can read available demo documents. Publication permission remains +restricted to maintainers, avoiding arbitrary uploads into the shared corpus. +Validate the namespace and privilege at API ingestion and Worker publication. +Do not accept a request field such as `is_admin` as authorization. + +Keep authenticated caller identity separate from corpus identity. Billing, +rate limits, retrieval runs, traces, cache entries, and personal hit statistics +belong to the caller even when content comes from demo tables. Never replace +the caller with the uploader or a system owner throughout the retrieval context. + +API and Worker use non-superuser runtime roles without BYPASSRLS. Authorized +demo writes set a server-controlled, transaction-local maintainer context; +FORCE RLS denies content writes without that context. Schema migrations use a +separate management connection. No separate demo-only writer credentials are +required. Validate policies using the actual runtime roles. + +## Implemented table families + +Use separate physical tables for content and its derived serving structures. +Retain the current shapes where feasible to reuse algorithms and contracts. + +| Tables | Responsibility | +| --- | --- | +| `demo_documents` | Stable source ID, display title, status, current published revision | +| `demo_document_sections`, `demo_document_chunks` | Versioned hierarchy, body, search fields, metadata, asset references | +| `demo_document_map_units`, `demo_document_map_unit_tokens` | Exact path/content units and token frequencies | +| `demo_document_map_unit_indexes` | Completeness marker and existing BM25 revision statistics | +| `demo_retrieval_serving_revision_manifests` | Immutable navigation/chunk manifest per revision | +| `demo_retrieval_namespace_generations`, `demo_retrieval_namespace_map_snapshots` | Coherent shared corpus generation and navigation snapshot | +| `demo_graph_nodes`, `demo_graph_edges` | Shared document graph consumed by agent tools | +| `demo_retrieval_hit_stats` | Caller-scoped hit records with a demo document foreign key | + +Jobs, job state history, job results, retrieval runs, and retrieval steps remain +the permanent processing/analytics ledger in their existing tables. Assets remain +in existing object storage, versioned per demo revision and shared by all readers. + +The existing `job_results.document_id` foreign key targets ordinary `documents`. +Add a nullable `demo_document_id` foreign key to `demo_documents` and a constraint +preventing both document references from being set. A result can initially have +neither reference during processing. Publication binds the appropriate reference; +replay/idempotency checks must recognize both. Never put a demo ID into the +existing ordinary-document foreign key or silently drop referential integrity. + +The existing hit-stat foreign key has the same ordinary-document restriction, +hence the dedicated demo hit table above. Retrieval-run final IDs are stored with +explicit corpus provenance. Document IDs must be distinguishable and resolve only +within the server-authorized corpus, including saved citations and asset requests. + +Demo token indexes serve the same query predicates as the existing reader. +Start with the required primary/uniqueness, token-leading lookup, and unit lookup +indexes. Do not copy historical redundant indexes. New migrations are additive +and idempotent; existing migrations remain unchanged. + +## Publication module and storage adapters + +Introduce a typed corpus target resolved by the server: PRIVATE or DEMO. Its +interface describes authorization, publication/read operations, revision identity, +and completeness. Two storage adapters select the correct models/tables. + +Reuse parser routing, chunk conversion, section construction, lexical preparation, +score-unit generation, token frequencies, statistics, manifest construction, and +graph algorithms. Hide table selection in repositories/adapters. Existing raw SQL +must be compiled from allowlisted table definitions; never substitute a user +namespace into SQL identifiers or apply global string replacements. + +The actual integration points include the API ingestion target, shared job +lifecycle publication, publication content/map-unit/manifest writers, retrieval +context, and every corpus reader. This is a persistence and authorization change, +not a fork of the parsing or scoring implementation. + +## Complete lifecycle + +1. A maintainer uploads a demo with the reserved namespace through the real API. +2. The API creates the ordinary processing job, stores the original, and enqueues + the same Celery task with its authorized corpus target. +3. Worker executes the selected parser and converts the real parse output. +4. Demo publication writes sections/chunks/map units/tokens/statistics/manifest + for a new immutable revision. Failed publication rolls back and follows the + existing job retry policy. +5. Activate the revision, update graph/snapshot/generation, and commit atomically. + Readers never observe a document with an incomplete token index. +6. Listing/catalog readiness comes from the published database state. Queries + use the completed demo revision without copying it into a user's namespace. + +Use a stable source key for update/re-upload idempotency. A new version replaces +the active pointer only after successful publication. Existing requests capture +revision pins/generation; they retry under the current consistency policy rather +than mixing old and new content. Saved citations identify their source revision, +which remains available under the read policy. + +Maintainer archive removes a demo from future public discovery and advances the +shared generation. User archive requests cannot archive shared demo documents. +Define retention for old revisions before adding cleanup; retain them initially. +Deleting a processing job/result must not cascade-delete published demo content +that readers or saved citations still reference; protect published ledger rows or +detach archival ledger references through a controlled lifecycle operation. + +## Retrieval compatibility + +Corpus resolution occurs once at the request entrance. Carry it through classic +BM25 discovery, map lighting, small-corpus reads, hydration, connected chunks, +document/section filters, and revision pins. + +Default agent retrieval must use the same target for list, recall, grep, read, +outline, neighbors, and final reference resolution. Document/chunk/page-citation +and asset endpoints also resolve it; supporting only the ranking query would +leave results that cannot be opened. + +Keep token strings/frequencies, channel lengths, average-IDF combination, query +document frequencies, type filtering, RRF, ordering, and public result shapes. +Statistics are calculated over the selected demo query corpus. Cache keys include +caller, corpus target, shared generation, query, and filters; private/demo entries +cannot collide. No background indexing after a successful publication response. + +## Demo recovery and migration + +The production recovery re-uploaded all 27 original demo files through the v2 +jobs API and the normal Worker path, including SpaceX. Stable catalog source IDs +and canonical document IDs were preserved. Parsing can change chunk boundaries; +old static chunk IDs are not promised after a reparse. Historical revisions and +assets remain available for revision-pinned reads. + +SpaceX has 407 physical pages and 388 indexed pages. The missing introductory +pages are a page-memory hierarchy/TOC scope limitation in the parser and are +tracked separately from the dedicated-table publication work. + +For publication parity checks, publish the same captured ParseOutput through both +adapters. This distinguishes persistence/scoring drift from model nondeterminism +between two fresh parses. Separately exercise actual parsing end to end. + +The production catalog is now published and usable from the dedicated tables. +Parsing time remains separate from user query and read latency; no ten-second +full-parsing promise is made. + +## Rollout and rollback + +The shared namespace is enabled directly; there is no demo feature flag and no +second publication strategy. Authorized maintainers restore sources through the +existing v2 jobs API and the production parser/Worker. The old materialization +endpoint returns 410, while existing personal copies continue to use ordinary +private tables. + +Rollback restores the previous compatible API and Worker images together after +stopping new demo imports. Keep the additive tables, original backups, and +revision assets in place; do not run a destructive down migration. Demo reads +may remain unavailable during rollback. Re-enable imports only after the +compatible pair is running. + +## Performance and historical implementation estimate + +Shared demo content is published once. User access writes no copied chunks or +tokens, eliminating the previous per-user materialization workload entirely. +Separate B-trees avoid maintaining the personal token indexes when publishing +new demos. Both families still share PostgreSQL CPU, memory, WAL, and storage I/O; +table separation is not resource isolation. + +The earlier SpaceX materialization had 227 chunks/units and 10,778 token rows per +user. These no longer grow with the number of demo readers. Query load, personal +hit statistics, and the number/size of maintained demo versions still grow. + +The ten-second publication target no longer applies to every user viewing a demo: +that synchronous materialization operation is removed. Actual query latency still +depends on corpus size and agent/model calls. Initial PDF parsing/publication may +take much longer than ten seconds; this design does not promise otherwise. + +| Work | Estimate | +| --- | ---: | +| Additive schema, foreign keys, corpus adapters and permissions | 2-3 engineering days | +| API/Worker ingestion and complete demo publication | 2-3 days | +| All retrieval tools, document/asset reads, catalog and cache routing | 2-3 days | +| Focused verification, reimport pilot and rollout/rollback handling | 1-3 days | + +Total estimate: **7-12 engineering days**, plus full catalog parsing/deployment +elapsed time. Main risk is missing a read/write/foreign-key path, not inventing a +new ranking algorithm. Stage the implementation around a real API -> Worker -> +parser -> demo publication -> retrieval vertical slice first. + +## Verification and follow-up + +The deployed backend was checked with focused contract coverage for private/demo +routing, unauthorized public writes, foreign keys, retries, publication atomicity, +readiness, filters, revision pins, classic/agent tools, graph, assets, cache +isolation, and rollback behavior. Runtime checks used the non-bypassing database +role rather than only a privileged test connection. + +Real HTTP validation covered SpaceX and the other restored sources through API, +Celery, parser, publication, query, document/chunk reads, originals, assets, and +page-citation sources. Multiple readers did not create new demo jobs, chunks, +tokens, or asset uploads. + +Notebook/client migration is still required before the complete user-facing Add +Demo workflow is accepted. Until that migration lands, clients must use the +shared catalog/document/revision contract directly. diff --git a/docs/design/shared-demo-namespace.md b/docs/design/shared-demo-namespace.md new file mode 100644 index 00000000..d6fd1209 --- /dev/null +++ b/docs/design/shared-demo-namespace.md @@ -0,0 +1,423 @@ +# Shared read-only demo namespace + +> **Historical proposal — superseded.** The current implementation is documented +> in [Dedicated demo tables](dedicated-demo-tables.md) and +> [Shared demo rollout](../shared-demo-rollout.md). It uses dedicated demo +> tables, direct reads from `__knowhere_demo__`, and the real API/Worker +> ingestion path. It does not create per-user demo materializations. + +Status: superseded design, 2026-10-03. The sections below are retained as design +history and are not an implementation or operations contract. In particular, +their materialization tables, fallback behavior, shared-namespace feature flag, +and rollback conversion flow are not part of the current system. The +production recovery snapshot on 2026-10-05 recorded 27/27 READY sources in the +dedicated demo tables. + +## 1. Decision and scope + +Store each published demo version once under an internal system owner in a +reserved namespace, provisionally `__knowhere_demo__`. An operator script +publishes new sources and versions. Authenticated users can retrieve catalog +sources from this namespace but cannot upload, replace, archive, or publish +content into it. + +For the existing Add Demo workflow, adding a demo to a personal namespace creates +a small user-owned document reference. Its chunks, sections, map units, token +frequencies, serving manifest, and assets are read from the shared version. +Ordinary uploaded documents retain their existing publication path. + +This requires an explicit shared-corpus read policy. A namespace name by itself +does not grant cross-user access: current queries also restrict `user_id`. +Namespaces for ordinary documents remain caller-owned isolation labels. + +## 2. Evidence and expected benefit + +The recorded October 1 production SpaceX request took 25.599 seconds over HTTP. +Its sections/chunks/map-index stage took 22.506 seconds and bundle reuse took +1.649 seconds. It wrote 227 chunks, 227 map units, and 10,778 token-frequency +rows. These are historical observations, not a fresh deployment inspection. + +The database mechanics experiment reproduced substantial token-index page reads +on a large clone. It did not isolate every part of the production request. +The historical database-mechanics notes remain an unpublished local draft. + +The shared namespace removes these per-user writes entirely, rather than making +the same COPY operations faster. Publication still happens once when an operator +prepares a source. That operation may take more than ten seconds and is outside +the user Add Demo request. + +| Operation | Current materialization | Proposed materialization | +| --- | --- | --- | +| Result ZIP/raw assets | Existing canonical reuse | Reuse prepared version | +| Sections/chunks/search text | Copy for each user | Read shared rows | +| Map units/token index | Build for each user | Read shared rows | +| Serving manifest | Build for each user | Read shared manifest | +| User state | Job, result, document, claim | Lightweight job, result, document, claim, binding | +| Namespace update | Publish full document manifest | Record reference and advance generation | + +With 10,000 references to SpaceX, the existing pattern would add approximately +107.78 million token rows and 2.27 million chunk rows. The shared version keeps +10,778 token rows and 227 chunk rows, plus small reference records. Personal +uploads, reference listings, request concurrency, and retrieval output still grow +with use; this does not make the whole application independent of data volume. + +## 3. Architecture + +```mermaid +flowchart TD + Script[Operator preparation script] --> Publish[Existing publication pipeline] + Publish --> Shared[System owner / reserved demo namespace] + Shared --> Rows[Chunks / sections / map units / token frequencies] + Shared --> Manifest[Complete immutable serving manifest] + Shared --> Assets[Canonical ZIP and raw assets] + Add[User adds a catalog demo] --> Reference[User document reference / pinned shared version] + Query[Authenticated retrieval request] --> Resolve[Resolve authorized logical document views] + Reference --> Resolve + Public[Explicit public demo namespace query] --> Resolve + Own[User's ordinary documents] --> Resolve + Resolve --> Reader[Shared corpus read module] + Rows --> Reader + Manifest --> Reader + Reader --> Score[Existing BM25 / agent tools / hydration] + Score --> Output[User-visible citations and assets] +``` + +The same scoring and token storage are used for personal and shared physical +documents. The new module resolves which physical revision backs each authorized +logical document and projects reads into that logical document's identity. +No external search service or new tokenizer is needed. + +## 4. Data model + +Reuse existing document/publication tables for the shared physical corpus. Add +two small tables rather than creating a parallel token-index schema. + +| Record | Proposed fields and responsibility | +| --- | --- | +| `demo_source_versions` | `version_id`, `demo_source_id`, source digest, lexical/compiler fingerprint, physical document ID and job-result ID, manifest digest, counts, preparation state, timestamps | +| `document_demo_bindings` | User document ID and user job-result ID, pinned demo version ID, view-format version; unique user document revision | +| `documents` | Existing user-owned shell for a mounted demo; ordinary ownership, namespace, status, title, and current job-result fields | +| `demo_materializations` | Existing per-user/namespace/source claim and idempotency record; points to the user shell | +| Existing shared publication rows | System-owned job/result/document, sections, chunks, map-unit marker and tokens, graph, manifest, canonical asset pointers | + +Use foreign keys to enforce document/revision relationships and `RESTRICT` deletion +of a referenced shared version. A binding must refer to the shell's document +revision, not merely to an arbitrary job-result ID. Add composite uniqueness/FKs +where necessary for that invariant. Index bindings by user document revision and +shared version for reads and rollback inventory. + +The shared corpus owner is an internal principal that cannot authenticate using +normal user credentials. The reserved namespace is a configured internal scope, +not a new public namespace-management resource. + +Use one physical document per source version. The catalog chooses the version +available for new references; existing references remain pinned to their version. +Retained old versions do not enter public searches unless explicitly selected by +the server's catalog policy. Never discover the shared corpus by selecting all +documents belonging to the system owner. + +Keep lightweight user jobs/results initially to preserve the processing ledger +and existing response fields. Their result pointers reference the canonical +artifacts; they contain no duplicate parsed content. Their publication kind is +explicitly a shared reference, so index readiness does not require a nonexistent +local token index. Client-supplied metadata cannot select this kind. + +## 5. Access and authorization + +### Direct shared namespace queries + +An authenticated query for the reserved namespace resolves only catalog-approved +READY source versions. By default it can search all currently available demos; +an explicit source/document selection can narrow that set. This satisfies access +for all users without requiring them to copy or mount the corpus first. + +The authenticated caller remains the identity used for rate limits, billing, +retrieval analytics, caching, and permission checks. A separate resolved read +scope contains the system-owned physical rows. Do not impersonate the system +owner by replacing the request's `user_id` throughout execution. + +### Personal namespace queries + +Resolve the union of ordinary user-owned documents and active demo references in +the requested personal namespace. Include only demos that the user added there; +do not silently append the whole public catalog to every private query. + +A reference is authorized by its user-owned shell. Exclusions, document filters, +revision pins, section filters, and archive state operate on that logical identity. +Two users referencing the same demo share content, not personal documents, +retrieval history, or archive state. + +### Write protection + +Reject the reserved namespace at all user write entry points, including uploads, +remote ingestion, retry/reprocess operations, demo materialization targets, and +any update/move operations. Enforce the same rule again in shared publication +before writes, so a queued Worker task cannot bypass an API guard. + +Only an explicit operator preparation context may publish into the system corpus. +Do not grant this privilege from job metadata, request fields, or model tool +arguments. Operator preparation uses a separate database write role; runtime +roles cannot modify READY shared source data. Apply database privileges/RLS to +the system rows as part of delivery. Verify policies using the runtime role; +table-owner bypass must not undermine protection. + +Original files, chunks, page citations, and assets use the same catalog read +policy. A guessed document/asset ID must not unlock another system-owned corpus. + +## 6. Operator preparation and versioning + +Provide an idempotent operator command with dry-run, prepare, inspect, and activate +operations. Exact CLI names are to be chosen during implementation. Preserve the +existing bundle preparation command and reuse its versioned storage outputs. + +1. Load and validate a catalog source; calculate source and preparation digests. +2. Acquire a build lock for that exact source/version, not the whole namespace. +3. Prepare/reuse canonical ZIP and raw assets outside the publication transaction. +4. Publish through the existing complete publication pipeline under the system + owner: sections, chunks, map index, graph, serving manifest, and statistics. +5. Validate counts, integrity, asset references, exact frequencies, and manifest. +6. Commit the complete source version as READY. Failed publication rolls back; + retry can reuse already prepared immutable storage objects. +7. Activate the READY version in the catalog in a short transaction, advancing + the shared serving generation. Report version, digest, counts, and outcome. + +Readiness is a database fact; a Redis entry or an S3 ZIP alone is insufficient. +Preparation states are BUILDING, READY, and FAILED. Catalog availability is a +separate pointer/policy, so withdrawing a source from new selection does not +mutate READY content or invalidate existing references unexpectedly. + +The fingerprint includes projected title/path, ordered chunk content and relevant +metadata, section summaries, normalization/deduplication, provider/root-asset +ownership, score-unit construction, tokenizer implementation/dictionary/config, +and index format. Use deterministic structured serialization. Random user IDs, +timestamps, and job IDs do not define semantic identity. Asset delivery version +must also be pinned; never point an old index at a newer overwritten bundle. + +READY content is immutable. A change creates a new version; never rebuild its +tokens in place. First release retains every READY version, including those +needed for rollback. Catalog size is small enough to avoid runtime reference +counters and garbage-collection contention. + +## 7. Add Demo transaction (superseded reference design) + +The historical proposal assumed that the existing materialization endpoint and +response could remain compatible; the current endpoint instead returns 410. + +1. Authenticate the caller and reject a reserved target namespace. +2. Resolve selected catalog versions and verify READY/compatible storage. +3. Use existing per-user/namespace/source claims to serialize concurrent adds. +4. Create lightweight job, job result, user document shell, and version binding. +5. Mark the claim ready; patch namespace reference metadata and advance its + generation in the same transaction; commit and invalidate affected caches. +6. Return the user-owned document ID. Immediate retrieval can resolve the + complete shared revision. No post-response indexing window is introduced. + +No source parsing, archive creation, asset upload, chunk COPY, token COPY, graph +recalculation, or per-user full-manifest serialization runs on this path. +Namespace snapshot storage records a reference; it must not copy the full +canonical manifest into every user's snapshot. Resolve it from the immutable +manifest when reading and cache by its version/digest. + +If a shared version is absent or incompatible before publication, use the existing +full local materialization path while rolling out, recording the fallback reason. +Do not build a missing shared corpus synchronously in a user's request. A failure +after the reference transaction starts rolls it back; do not silently run a second +publication in that partially failed transaction. Broken bindings at retrieval +time raise an explicit integrity error, rather than returning empty results. + +## 8. Unified retrieval and document reads + +Introduce one corpus read module whose interface accepts an authenticated caller, +requested namespace/document filters, and captured revision scope. It returns +authorized logical document views and exposes the existing chunk, section, +manifest, map-unit, and frequency reads through those views. + +Each view contains a logical document/revision identity, physical document/revision +identity, owner/read policy, and immutable version. Callers do not add ad hoc +`OR namespace = ...` clauses. Parameters are typed, bounded, and server-resolved; +the interface does not accept arbitrary SQL or arbitrary physical IDs from users. + +| Existing consumer | Required change | +| --- | --- | +| Retrieval context and serving generation | Resolve local/reference/public views; capture logical pins and source versions | +| Namespace snapshot / manifest cache | Compose referenced manifests through views; include binding/view version in cache identity | +| Classic `map_unit_discovery` | Read physical units/tokens, project logical IDs, validate reference-aware completeness | +| `map_lighting` / outline filters | Use the same views for units, frequencies, and statistics | +| Agent list/recall/grep/read/outline/neighbors and reference resolution | Apply identical authorized scope and logical identity mapping | +| Small-corpus route / hydration / connected targets | Read shared chunks through the view, including linked assets | +| Document list/detail/chunks/page-citation endpoints | Return user shell metadata plus content projected from its pinned version | +| Archive and statistics | Remove the user's reference from scope; record hits against caller/logical document | +| Graph discovery | Resolve physical demo graph through the selected views; prevent system-wide graph leakage | + +Mounted views return the user document ID in citations. Section/chunk/map-unit +view IDs are deterministic opaque IDs derived from the binding and canonical row, +with bounded lengths compatible with existing contracts. Resolve them through +the authorized version's manifest; do not treat an ID prefix as authorization. +Content hashes retain their existing meaning. Direct public queries can return +catalog canonical identities, resolved to READY physical versions by the same +module. System job/revision IDs are not presented as another user's job history. + +For a namespace containing both uploaded content and demo references, collect +the same logical corpus that local materialization would have produced. Keep +exact path/content token frequencies, channel lengths, per-revision average-IDF +combination, document-frequency denominators, type filtering, stable ordering, +and RRF behavior. Do not search the two corpora independently and concatenate +their top-K results: that changes scoring and recall. + +Physical reads of the same shared version can be deduplicated, but logical copies +must be expanded before computing statistics and ranking. If two authorized +logical documents reference one physical version, they count as two documents +under existing semantics. Sharing storage is not content deduplication at query +time. A global shared BM25 score/IDF cache would violate mixed-corpus parity. + +Completeness checks accept either a complete local publication or a complete +reference to a READY shared publication. Existing checks that count local token +rows must resolve the backing physical revision. Zero owner token rows are valid +for a reference and do not justify bypassing canonical integrity checks. + +## 9. Generations, updates, and archive + +Private queries capture the personal namespace generation and all logical revision +pins, including pinned source versions. Direct public queries capture the shared +catalog generation and version selection. Existing READY versions do not change +when the catalog activates a new version, so personal references remain stable. +Generation mismatch follows the existing retry/consistency policy; no fallback +may skip shared-corpus authorization or reinterpret a reference as a local index. + +Cache keys include caller, namespace, filters, generation, selected versions, and +view format. Immutable physical manifests may be shared in cache; assembled +responses and hit statistics retain caller scope. + +User archive changes only the user shell, reference visibility, graph view, and +personal namespace generation. It does not archive/delete a shared document or +change another user's retrieval. Re-adding can pin the latest activated version +according to the existing claim/archive behavior. + +A normal re-upload to a mounted document produces a new ordinary local revision +through the Worker path. Never write user content into its backing shared version. +Title/path edits that change lexical semantics require a local publication or a +new operator-created shared version. A future namespace-only move can preserve +the binding if lexical fields are unchanged and both namespace generations advance. + +Cross-document demo-to-personal graph edges are not represented by the canonical +graph. `corpus.neighbors` is a live consumer, so graph handling is required. +Retain existing local-to-local edges, and project shared-to-shared edges only +between selected views. Derive reference-to-local relationships on a neighbors +read using the existing typed-entity/keyword overlap algorithm and authorized +revision metadata. Cache them by caller scope, generation, and source versions. +Do not scan chunks, call models, or rebuild graphs during Add Demo. Query-time +metadata work is a separate scaling risk to measure. A fixture comparing a locally +published demo with its reference must establish neighbor parity before activation; +if the current edge algorithm depends on publication-time state, resolve that +semantic difference explicitly rather than accepting silent drift. Never expose +the system owner's full graph. + +## 10. Rollout and rollback (superseded reference design) + +The historical proposal used a new, off-by-default reference-writer flag, +provisionally `DEMO_SHARED_NAMESPACE_ENABLED`, plus a source allowlist. The +current implementation has no shared-namespace feature flag. + +1. Deploy additive schema, write protection, read module, and conversion tooling + to both API and Worker, with reference creation disabled. +2. Prepare SpaceX under the operator role; validate READY data and public reads. +3. Exercise one mounted SpaceX through the real API and both retrieval modes; + verify a normal parsed Worker upload in the same personal namespace. +4. Enable SpaceX references for the rollout cohort, then expand by source. +5. Retain existing locally materialized demos; migration is optional and separate. + No existing token/index deletion is required for this release. + +Writer rollback: disable the flag. New adds use local materialization; existing +references keep working through the deployed reader. Read back the deployed +configuration; ensure deployment rendering preserves the flag and allowlist. + +Binary rollback to a version without shared readers requires conversion first. +Provide a bounded, resumable tool that expands each reference into local sections, +chunks, map index, graph, and manifest using its pinned version. Preserve the +logical view IDs used in saved citations, validate parity, then switch the binding +to local atomically under namespace generation control. Do not upgrade existing +users to a newer source during conversion. Retain ledger rows and canonical data. +Report remaining references; downgrade only when they reach zero. Recreating +token rows can be slow and must not be described as instant rollback. + +Cloud deployment, production preparation, and flag changes remain separate +authorized operator actions. This document does not perform them. + +## 11. Delivery plan and estimates + +| Phase | Deliverable | Engineering estimate | +| --- | --- | ---: | +| A | Shared source/version records, reserved-scope write guards, idempotent preparation using existing publication | 1-2 days | +| B | Authorized document-view reads; direct public queries, mixed private queries, tools, hydration and document endpoints | 3-4 days | +| C | Lightweight Add Demo references, namespace snapshots, archive, cache/generation integration | 1-2 days | +| D | Reference-to-local rollback conversion and deployment flag handling | 1-2 days | +| E | Focused contracts, real API/Worker paths, narrow large-clone measurement, rollout checks | 1-2 days | + +Full compatibility delivery: approximately **7-12 engineering days**, excluding +external deployment/access delays. This includes work on readers beyond BM25; +token-only sharing would have a smaller scope but retain per-user chunk writes. +Direct public-namespace retrieval can be delivered earlier in approximately +3-5 days, but alone does not replace the existing Add Demo workflow or guarantee +mixed private/public retrieval and citation compatibility. + +Performance budget for one preprepared SpaceX add: reference resolution and +claims below 1 second, reference transaction/generation below 2 seconds, and +remaining HTTP/cache work below 2 seconds. The initial target is **under 5 seconds +in normal conditions, under 10 seconds at the agreed concurrency**. These are +engineering budgets, not measured predictions or an SLA. Lock contention and +pool/Redis waits still require measurement. Batch-add latency must be measured +separately and cannot inherit a single-source guarantee. + +Expected deterministic gain: no new chunk, section, map-unit, or token rows per +reference; no archive upload or token-index maintenance on that request. The +observed 25.599-second request cannot establish a numerical improvement until +the real path measures the residual work. Query latency should be comparable +with additional bounded reference resolution, not assumed faster automatically. + +## 12. Focused verification and completion criteria + +Use contract tests against the corpus-read and publication interfaces. Keep the +fixture set small; use the existing real SpaceX corpus for performance evidence. + +- Same source locally materialized and shared: exact frequencies/statistics, + equivalent ranked evidence, tie ordering, citations, and asset references. +- Personal upload plus demo, two references to one version, include/exclude + filters, section targets, revision pins, images/tables, and no-match readiness. +- Classic retrieval and default agent list/recall/grep/read/outline/neighbors, small-corpus + behavior, and linked target hydration. Agent prose need not match word for word. +- Two users: read public data successfully; archive one reference; verify the + other remains readable and personal content is never exposed. +- User writes rejected at API and Worker publication; guessed IDs, inactive + catalog versions, and foreign bindings cannot broaden authorized reads. +- Inject failure before commit, replay concurrent adds, activate a new source + version, disable the writer flag, and convert a reference back to local. +- Real isolated API SpaceX add/retrieval/archive plus API upload -> Celery -> + actual parser -> Worker publication -> mixed retrieval. Model-backed parser + and agent verification require working provider credentials; a fallback or + mock is reported as such, not as successful provider validation. + +For performance, use a few paired real HTTP observations on the disposable Mac +large clone: local materialization and shared-reference addition, same source, +namespace conditions, and explicit warm/cold labels. Capture total latency, +claim/lock/pool waits, stage spans, and inserted-row counts. A small sample verifies +the mechanism and target observations, not a reliable p95. Expand sampling only +for a concrete unresolved problem or an agreed deployment gate. + +Completion requires compatible reads across every listed consumer, zero shared +payload writes per user add, immediate complete retrieval after commit, tested +conversion rollback, and observed Add Demo latency within the target at the +specified load. Until those are met, this remains a design rather than a claim +that production SpaceX now completes within ten seconds. + +## Source pointers + +- Historical demo materializer: `apps/api/app/services/demo/source_materializer.py` + (removed from the current implementation) +- [Demo path projection](../../apps/api/app/services/demo/source_projection.py) +- [Publication content](../../packages/shared-python/shared/services/retrieval/publication_content.py) +- [Map-unit constructor](../../packages/shared-python/shared/services/retrieval/map_unit_index.py) +- [Classic discovery](../../packages/shared-python/shared/services/retrieval/search/map_unit_discovery.py) +- [Map lighting](../../packages/shared-python/shared/services/retrieval/scoring/map_lighting.py) +- [Document lifecycle reads](../../apps/api/app/services/documents/lifecycle_service.py) +- [Atomic publication ADR](../adr/0006-atomically-publish-retrieval-serving-index.md) +- [Serving generation ADR](../adr/0007-use-coherent-retrieval-serving-generations.md) diff --git a/docs/shared-demo-rollout.md b/docs/shared-demo-rollout.md index 6985ddca..9bb076c8 100644 --- a/docs/shared-demo-rollout.md +++ b/docs/shared-demo-rollout.md @@ -6,6 +6,24 @@ Remove `KNOWHERE_PUBLICATION_STRATEGY` from deployment settings; no switch enabl the new demo path. Historical benchmark baseline/candidate labels are retained. This document defines the rollout contract for the shared demo backend. +## Deployment and recovery snapshot — 2026-10-05 + +The shared demo backend was deployed in staging and production. The production +recovery completed on 2026-10-05 with all 27 catalog sources READY, 2,562 demo +chunks, generation 27, and 25.26 credits charged for parsing. The runtime +database role is neither a superuser nor a `BYPASSRLS` role, and all twelve demo +tables use FORCE RLS. HTTP checks covered catalog, classic retrieval, agent +retrieval for Tesla/SpaceX/Microsoft, document and chunk reads, originals, +assets, and page-citation sources. Evidence is retained in the ignored local +directory `.demo-originals/production-restoration-20261005/`; it is not a source +artifact and must not be committed. + +The restored sources use `/api/v2/jobs`: PDF and PPTX inputs follow the +`page_memory` track, while DOCX inputs follow the `chunk` track. `/api/v1/demo/catalog` +is the read-only catalog endpoint. The old materialization endpoint returns 410. +Notebook/client migration remains a separate follow-up before the complete +end-user Add Demo workflow is considered finished. + ## Storage and access The existing PostgreSQL database gains twelve dedicated `demo_*` tables. @@ -57,15 +75,14 @@ invalid. Resume reruns reuse only objects whose size and source ETag still match Do not switch with an incomplete inventory. These S3 originals have no versioned recovery copy, so preserve an independent persistent backup. -The 2026-10-03 backup contains 27 verified originals totaling 225,749,274 bytes. -The prior backup run copied them to -`/Users/ontosai/knowhere-demo-originals/20261003` and independently verified remote -SHA-256 values. The local validation copy is -`/tmp/knowhere-demo-originals/20261003`; this temporary copy is not the persistent -backup. The current local run also copied and independently rehashed all originals -into `.demo-originals/20261003` in this checkout. That directory is Git-ignored -and survives removal of temporary validation files. Preserve it before deleting -the checkout. Mac availability was not reverified after the prior backup run. +The original backup inventory contains 27 verified originals totaling +225,749,274 bytes. Use a persistent operator backup directory; a temporary +validation copy is not a recovery backup. The production restoration used the +verified inventory and retained its validation report under +`.demo-originals/production-restoration-20261005/`. +That evidence directory is Git-ignored; preserve the original-file backup and +evidence independently before deleting the checkout. The helpers can run on any +host with the required credentials and persistent storage. SpaceX is 153,808,384 bytes / 407 pages. Ensure **both** API and Worker configure `MAX_FILE_SIZE=314572800`; older local configurations may still have a 100 MiB limit. @@ -78,7 +95,7 @@ Stop if this preflight returns rows: SELECT document_id, user_id FROM documents WHERE namespace = '__knowhere_demo__'; SELECT current_user, rolsuper, rolbypassrls FROM pg_roles -WHERE rolname = current_user; +WHERE rolname = current_user AND (rolsuper OR rolbypassrls); ``` Migration `4d5e6f7a8b9c` repeats the reserved-namespace conflict check. It is @@ -137,7 +154,8 @@ that password has already been changed. 5. Stop old materialization admission and drain its tasks before updating API and Worker together. Old tasks must not execute new publication code. 6. Verify private upload/retrieval, catalog preparing state and materialization - 410 before manually restoring sources. Demo downtime is expected. + 410 before manually restoring sources. Demo downtime is expected during the + initial cutover; this step is complete for the current deployment. 7. Restore Tesla first as a smoke test, then SpaceX, then remaining inventory. No service startup automatically parses demos. A failed source does not block @@ -267,4 +285,9 @@ pair is running. Production deployment and Notebook changes are separate from this backend implementation. See [the local verification report](shared-demo-validation-20261003.md) for -measured results, the SpaceX hierarchy/agent fixes and validation scope. +the historical local measurements and SpaceX hierarchy/agent fixes. That local +run covered all 407 SpaceX pages; the production recovery snapshot indexed +388/407 pages. Production READY means a complete publication of the parser's +output, not full physical-page coverage. The page-memory hierarchy/TOC scope +limitation remains a separate parser follow-up. The production snapshot above +comes from the retained restoration evidence, not the local validation report.