Add bounded read bundles and covering indexes

CHANGELOG.md

@@ -6,6 +6,23 @@ The format follows Keep a Changelog and the package uses semantic versioning.
 
 ## Unreleased
 
+## 3.1.0 - 2026-09-25
+
+- Added opt-in bounded point-read bundles that reduce cold trie and snapshot
+  reads to one browser request when the authority advertises the endpoint.
+- Kept automatic fallback to the existing encrypted-object path for older
+  authorities, legacy metadata, and bundles above the four-object or 4 MiB
+  decoded limits.
+- Added explicit covering fields to declared secondary indexes.
+- Bounded each covering projection to 64 KiB and every immutable index page to
+  4 MiB before changed document objects are written.
+- Added typed query projections through `.select(...)`; covered queries can
+  return declared fields from the index without loading full documents.
+- Kept full-document reads for ordinary queries and whenever a predicate,
+  ordering field, or selected field is not covered.
+- Documented embedded and separately deployed authority topologies and their
+  operational tradeoffs.
+
 ## 3.0.0 - 2026-09-25
 
 - Added the optional ThimbleDB Studio management frontend to the npm package.

README.md

@@ -89,6 +89,8 @@ Only bindings, credentials, and browser read authorisation differ.
 - retained deletion, restoration, and quiescent physical collection
 - immutable snapshot and content-addressed trie collection layouts
 - typed collections, bounded predicates, and declared secondary indexes
+- bounded cold point-read bundles with object-path fallback
+- explicit covering index fields and typed projections
 - versioned logical archives and explicit migration adapters
 - local application scaffolding and diagnostics
 - optional package-owned Studio management frontend
@@ -102,7 +104,7 @@ Only bindings, credentials, and browser read authorisation differ.
 - reusable Node and Cloudflare authority endpoint exports
 - Docker, Wrangler, Bicep, and CloudFormation deployment paths
 
-The reference browser build is about 58.6 KB uncompressed and 16.3 KB gzip.
+The reference browser build is about 68.3 KB uncompressed and 18.7 KB gzip.
 It ships no database runtime or WASM module.
 
 ## Quick start
@@ -161,6 +163,10 @@ Use the browser/core API from `thimbledb`, external identity primitives from
 `thimbledb/authority/node` or `thimbledb/authority/cloudflare`. Consumers
 supply their own domain, storage, OIDC application, and secrets.
 
+The authority can share the application deployment or run as a separate
+service behind the same public browser origin. See
+[Authority deployment modes](docs/AUTHORITY-DEPLOYMENT.md).
+
 After the authority session exists:
 
 ```ts
@@ -249,6 +255,7 @@ layout decision thresholds.
 | [npm publishing](docs/NPM-PUBLISHING.md) | OIDC trusted publisher setup and release process |
 | [Use cases](docs/USE-CASES.md) | Fit criteria and application-specific guides |
 | [Architecture](docs/ARCHITECTURE.md) | Components, data flow, and scope model |
+| [Authority deployment modes](docs/AUTHORITY-DEPLOYMENT.md) | Embedded and separate authority topologies and decision criteria |
 | [System diagrams](docs/DIAGRAMS.md) | Trust boundaries, sequences, keys, and providers |
 | [Storage providers](docs/STORAGE-PROVIDERS.md) | Provider abstraction and conformance requirements |
 | [Security](docs/SECURITY.md) | Threat model, encryption, keys, and revocation |

deploy/cloudflare/wrangler.example.jsonc

@@ -15,6 +15,7 @@
     "THIMBLE_KEY_VERSION": "1",
     "THIMBLE_READ_KEY_VERSIONS": "",
     "THIMBLE_HEAD_TTL_MS": "1000",
+    "THIMBLE_READ_BUNDLES": "false",
     "THIMBLE_COLLECTION_LAYOUTS": "",
     "THIMBLE_COLLECTIONS": "",
     "THIMBLE_RETIRED_COLLECTION_LAYOUTS": "",

docs/ARCHITECTURE.md

@@ -7,6 +7,8 @@ storage abstraction.
 
 See [System diagrams](DIAGRAMS.md) for trust boundaries, request sequences,
 scope separation, and provider layouts.
+See [Authority deployment modes](AUTHORITY-DEPLOYMENT.md) for embedded and
+separately deployed authority choices.
 See [Storage providers](STORAGE-PROVIDERS.md) for the provider contract and
 conformance requirements.
 
@@ -22,6 +24,7 @@ Browser
 Private read broker
   session and scope validation
   encrypted binary envelopes from R2
+  optional bounded HTTPS cache-value bundles for cold point reads
 
 Browser mutations
           |
@@ -67,29 +70,42 @@ encryption. Private scopes use a versioned AES-256-GCM data key.
 
 ## Read path
 
-1. The browser reads `HEAD.json` from memory or IndexedDB.
-2. If its TTL expired, the browser revalidates HEAD with `If-None-Match`.
-3. A 304 response keeps the current layout generation.
-4. ID equality resolves directly to one document path.
-5. A matching declared index resolves a bounded set of candidate IDs.
-6. Queries without a usable index use an explicitly bounded scan.
-7. Trie HEAD points to an immutable root, branch, and leaf path.
-8. Snapshot HEAD points to one immutable collection snapshot.
-9. The browser coalesces concurrent reads of the same immutable object.
-10. It re-evaluates the complete predicate, orders, limits, and returns the
+1. The browser first checks the authority-and-scope cache namespace.
+2. On a cold point read, a version 3.1 authority can return a bounded HEAD and
+   immutable-object bundle in one browser request.
+3. Older authorities, legacy metadata, oversized bundles, and cache hits use
+   the individual encrypted-object path.
+4. If a cached HEAD TTL expired, the browser revalidates it with
+   `If-None-Match`.
+5. A 304 response keeps the current layout generation.
+6. ID equality resolves directly to one document path.
+7. A matching declared index resolves a bounded set of candidate IDs.
+8. An explicit `.select(...)` can use declared covering fields without
+   loading full documents.
+9. Queries without a usable index use an explicitly bounded scan.
+10. Trie HEAD points to an immutable root, branch, and leaf path.
+11. Snapshot HEAD points to one immutable collection snapshot.
+12. The browser coalesces concurrent reads of the same immutable object.
+13. It re-evaluates the complete predicate, orders, limits, and returns the
     query plan with the documents.
 
 Immutable pages do not need revalidation. Their object key identifies their
 content within the scope and key version.
 
+Individual object reads remain TDB1 envelopes that the browser decrypts.
+Bounded read bundles are assembled by the trusted authority and carry decoded
+cache values over HTTPS with `no-store`; object storage remains encrypted and
+private.
+
 ## Write path
 
 1. The browser sends a mutation to the authority.
 2. The authority authenticates the session and resolves allowed scopes.
 3. Application validation runs before storage work.
 4. Changed trie pages or the next immutable snapshot are serialised,
    gzip-compressed when useful, and encrypted.
-5. Every configured secondary index is updated or rebuilt.
+5. Every configured secondary index and declared covering projection is
+   updated or rebuilt.
 6. New immutable document and index objects are created.
 7. HEAD publishes the document root and all active index references with one
    ETag compare-and-swap.

docs/AUTHENTICATION.md

@@ -106,6 +106,10 @@ account security version, provider, expiry, CSRF token, and issued scope
 grants. Cookies use HttpOnly, SameSite=Strict, and Secure outside local
 development.
 
+Embedded and separately deployed authorities should normally remain behind one
+public browser origin. See
+[Authority deployment modes](AUTHORITY-DEPLOYMENT.md).
+
 On every authenticated request the authority reloads the current internal user
 record and recalculates grants. Role or tenant removal observed during a later
 OIDC exchange therefore also affects existing sessions.
@@ -235,6 +239,7 @@ grants. `thimble.admin` does not make ungranted data scopes visible.
 | `POST /api/auth/identities/unlink` | Recent session + CSRF | Remove a non-final identity and revoke sessions |
 | `GET /api/config` | Session | Current user, scope, CSRF, and cache config |
 | `GET /api/keys/:scope` | Session + read grant | Scope key grant |
+| `GET /api/read-bundles/:scope/:collection/:id` | Session + read grant | Optional bounded cold point-read bundle |
 | `GET /api/admin/users` | `thimble.admin` | List identity mappings |
 | `POST /api/admin/users/:id` | `thimble.admin` + CSRF | Change status, application roles, or tenants |
 | `POST /api/admin/users/:id/revoke-sessions` | `thimble.admin` + CSRF | Revoke every user session |

docs/AUTHORITY-DEPLOYMENT.md

@@ -0,0 +1,201 @@
+# Authority deployment modes
+
+ThimbleDB separates the browser client from the authenticated authority. The
+authority can run in the application's deployment or as a separately operated
+Worker, container, or function.
+
+This is a process and deployment choice. In both modes, expose application and
+authority routes through one public browser origin unless an independently
+reviewed cross-origin session design replaces the default Strict cookie
+contract.
+
+## Decision summary
+
+| Consideration | Embedded authority | Separate authority service |
+| --- | --- | --- |
+| Deployment units | One application deployment | Application plus authority deployment |
+| Browser origin | Naturally the same | Use a gateway or path route to preserve one public origin |
+| Secrets | Application runtime also holds authority and storage secrets | Storage credentials and master key remain outside the application runtime |
+| Release cadence | Application and authority change together | Authority can be upgraded independently |
+| Scaling | Application and authority scale together | Reads, writes, and application rendering can scale separately |
+| Failure boundary | One runtime can affect both application and data API | Application and authority failures are isolated |
+| Operations | Simplest setup and observability | More routing, monitoring, version coordination, and incident paths |
+| Latency | No service-to-service hop inside the deployment | Gateway and service routing may add latency |
+| Best fit | One small application and one operations team | Shared platform boundary, stricter secret isolation, or independently scaled authority |
+
+The storage provider does not determine the topology. A Node authority can use
+local files, Azure Blob Storage, S3, or R2 in either deployment model. A
+Cloudflare authority can share one Worker deployment with application assets
+or run as a dedicated routed Worker.
+
+## Embedded authority
+
+The authority starts as part of the application deployment. The application
+and data API normally share one release, hostname, logs, and scaling policy.
+
+Cloudflare example:
+
+```ts
+import {
+  createCloudflareAuthority,
+} from "thimbledb/authority/cloudflare";
+import {
+  collectionIndexes,
+  collectionLayouts,
+} from "./collections";
+
+export default createCloudflareAuthority({
+  studio: true,
+  readBundles: true,
+  collections: ["notes"],
+  collectionIndexes,
+  collectionLayouts,
+});
+```
+
+The same Worker can serve static assets through an `ASSETS` binding and run
+the authority for `/api/*`.
+
+Node example:
+
+```ts
+import {
+  startNodeAuthority,
+} from "thimbledb/authority/node";
+
+await startNodeAuthority({
+  studio: true,
+  readBundles: true,
+  collections: ["notes"],
+  collectionIndexes,
+  collectionLayouts,
+});
+```
+
+The application deployment owns the authority process. A gateway can expose
+the application frontend and authority listener through one public origin.
+
+Choose this mode when:
+
+- one application owns the data model
+- one deployment lifecycle is acceptable
+- the smallest operational surface is more important than secret isolation
+- application and authority traffic have similar scaling needs
+- a same-origin browser path should require no additional routing layer
+
+Avoid it when a compromise of the application runtime must not expose the
+storage credential or deployment master key.
+
+## Separate authority service
+
+The authority runs in its own Worker, container, Lambda function, Container
+App, or Node service. The browser application remains a normal ThimbleDB
+client.
+
+Typical public routing:
+
+```text
+https://app.example.com/           -> application assets or application server
+https://app.example.com/api/*      -> separate ThimbleDB authority
+https://app.example.com/studio/*   -> authority or version-matched Studio assets
+```
+
+The authority service owns:
+
+- OIDC token exchange and opaque sessions
+- CSRF and exact Origin enforcement
+- scope grants and key grants
+- read bundles and encrypted-object broker routes
+- validation, writes, retained deletion, and maintenance
+- storage credentials and the deployment master key
+
+The application runtime needs none of those storage secrets.
+
+Choose this mode when:
+
+- application and authority releases need independent approval or rollback
+- storage credentials require a smaller runtime trust boundary
+- several application processes use one authority contract
+- write and broker traffic need independent scaling or observability
+- platform routing already supports path-based service isolation
+
+The additional cost is real: another deployment, route, health check, log
+stream, alert set, version boundary, and incident path must be operated.
+
+## Same-origin browser boundary
+
+The default ThimbleDB session cookie is `HttpOnly` and `SameSite=Strict`.
+Browser caches, BroadcastChannel logout, localStorage cache registries, and
+IndexedDB are origin-scoped. For that reason, a separate process should not
+automatically imply a separate browser hostname.
+
+`THIMBLE_ALLOWED_ORIGIN` validates state-changing requests. It does not by
+itself turn the default browser client into a cross-origin cookie system.
+
+Prefer a reverse proxy, Worker route, Function URL gateway, Front Door route,
+or application gateway that preserves one public origin. Review all of the
+following before intentionally introducing a separate authority origin:
+
+- cookie `SameSite`, `Secure`, and domain attributes
+- credentialed CORS responses and preflight behaviour
+- CSRF and exact Origin validation
+- cache namespace and logout coordination across origins
+- Studio hosting and session behaviour
+- redirect and callback URLs at the OIDC provider
+
+## Performance implications
+
+Read bundles work in both deployment modes because the browser discovers the
+optional endpoint from `/api/config`.
+
+Enable them explicitly with `readBundles: true` or
+`THIMBLE_READ_BUNDLES=true`. Existing deployments retain the individual TDB1
+object path until that capability is enabled.
+
+The trusted authority assembles bundle cache values after decrypting storage
+objects and sends them over HTTPS with `no-store`. Leave the capability
+disabled if the deployment requires every read response above TLS to remain a
+TDB1 envelope.
+
+An embedded authority removes one internal routing boundary. A separate
+authority can instead be placed near object storage and scaled independently.
+Neither choice changes the number of browser requests once the same public
+route reaches the authority.
+
+Measure:
+
+- browser-to-authority latency
+- authority-to-object-storage latency
+- cold read-bundle duration and fallback count
+- session and key-grant duration
+- conditional-write conflicts
+- application and authority CPU independently
+
+Do not claim one topology is faster without testing the actual gateway,
+runtime, and storage region.
+
+## Security implications
+
+Both modes enforce the same sessions, scope grants, encryption, deletion, and
+conditional-write rules.
+
+Embedded mode has a larger runtime blast radius because application server
+code and authority secrets coexist. Separate mode narrows that secret boundary
+but adds routing and service-to-service configuration that can itself be
+misconfigured.
+
+In either mode:
+
+- keep object storage private
+- expose reads only through the authenticated broker or bounded bundle route
+- keep provider credentials and the master key out of browser code
+- use exact allowed origins
+- keep application and authority package versions compatible
+- use logical exports and provider backups independently of deployment shape
+
+## Recommendation
+
+Start embedded for one small application unless a concrete security,
+operations, or scaling requirement justifies a separate authority. Move the
+authority into a separate service without changing application collection
+code, storage layout, or browser query semantics.

docs/BENCHMARKS.md

@@ -62,6 +62,13 @@ with a ten-second HEAD TTL.
 - The result supports adaptive layout selection for this workload. It does not
   establish superiority over another database.
 
+Version 3.1 adds an optional bounded point-read bundle. Automated browser tests
+verify that an eligible cold trie point read uses one browser request instead
+of four and that unsupported or oversized bundles fall back to the original
+object path. No new live multi-region latency result has yet been published,
+so the request reduction is verified but its production p95 effect remains to
+be measured.
+
 ## Measurements not covered
 
 The published runs do not cover: