[suguanYang]

Summary

Progressive ingestion publishes a readable baseline, then runs durable automatic enhancement under the same business Job. Keep that baseline searchable until all scopes, required vectors, and the immutable final archive are ready; publish the final content/index, snapshot pointer, and completion event together.

• Add immutable baseline/final snapshot delivery, ordered webhook milestones, failed-enhancement retry, durable stage recovery, and claim-generation fencing without repeating ingestion charges.
• Preserve source-page citation coordinates and fail retrieval when a required query embedding fails. Fence concurrent retrieval against final publication and document replacement.
• Integrate current main's native M0 page images while deferring their vectors into durable staged work. Required vector failures withhold final completion; successful retries preserve existing vectors.
• Fix a recovery/manual-retry race that could restore an older failed attempt, and a simultaneous first-publication primary-key collision in namespace generation creation. Worker concurrency and scheduling configuration remain unchanged.

The implementation and acceptance references are in docs/design/progressive-ingestion-development-plan.md, its hardening plan, and ADRs 0011–0022.

Verification

• make check passed.
• Full Worker suite: 569 passed, 1 skipped. Full API suite: 528 passed; final publication/concurrency contract module: 13 passed.
• Includes final-publish/concurrent-retrieval, snapshot staging, vector repair, source pages, manual retry, and simultaneous first namespace publication coverage.
• A real PostgreSQL two-thread regression reproduced the stale recovery overwrite before the fix and passes after it.
• Three-repository standards/spec review findings were fixed and re-reviewed.
• Prior mixed-workload capacity measurements remain historical; capacity was not rerun after main's new native M0 rendering behavior. Live Hipo product acceptance remains a release prerequisite.

Deployment notes

Two append-only migrations add scope claim generations and the pending-enhancement recovery index. Existing API defaults remain ingestion_mode=full; progressive uses the existing mode selector. Deploy compatible server snapshot/callback contracts before publishing the SDK and enabling Hipo's matching dependency. Hipo must retain the baseline until its final local copy commits.

Related SDK: Ontos-AI/knowhere-node-sdk#139

Checklist

• Regression and contract coverage updated
• API and product contracts documented
• Migrations are append-only and idempotent
• Concurrency configuration unchanged
• Release prerequisites and verification limits recorded

Additional baseline checks: the shared suite reports 277 passed and 4 failures; all four failures reproduce against an independent main (eca92766) source copy and are pre-existing. API and Worker suites must run sequentially locally because the existing harness shares filesystem object storage and Redis state.

Related Hipo integration: https://github.com/Ontos-AI/hipo/pull/408 (Draft until compatible SDK publication/pin).

CodeQL's assertion-side-effect finding was fixed by moving the database call out of the assertion; its focused contract passes. Alembic's revision metadata and a Protocol ellipsis are informational scan notes.