From 911733a8a2dfe00c024f6c636da18e1a19ca345e Mon Sep 17 00:00:00 2001
From: Jason Doyle <46789294+Jason-Doyle@users.noreply.github.com>
Date: Fri, 25 Sep 2026 21:56:04 -0700
Subject: [PATCH] Correct system diagrams
---
README.md | 34 +++--
docs/ARCHITECTURE.md | 25 ++--
docs/DIAGRAMS.md | 118 +++++++++++-------
site/public/diagrams/0c8435b9462cd3a63ff5.svg | 1 +
site/public/diagrams/79ad5efd6e4ed3f2a894.svg | 1 +
site/public/diagrams/897ea0c26deb6b3cb03a.svg | 1 -
site/public/diagrams/bdfadb8fe6895bdd60be.svg | 1 +
site/public/diagrams/c5bc1a796ce8e93f2840.svg | 1 -
...2f929f172.svg => e792e1d4c044189f52e3.svg} | 2 +-
site/public/diagrams/f9b79a62f37a16efb8ec.svg | 1 -
10 files changed, 111 insertions(+), 74 deletions(-)
create mode 100644 site/public/diagrams/0c8435b9462cd3a63ff5.svg
create mode 100644 site/public/diagrams/79ad5efd6e4ed3f2a894.svg
delete mode 100644 site/public/diagrams/897ea0c26deb6b3cb03a.svg
create mode 100644 site/public/diagrams/bdfadb8fe6895bdd60be.svg
delete mode 100644 site/public/diagrams/c5bc1a796ce8e93f2840.svg
rename site/public/diagrams/{5628899b5012f929f172.svg => e792e1d4c044189f52e3.svg} (54%)
delete mode 100644 site/public/diagrams/f9b79a62f37a16efb8ec.svg
diff --git a/README.md b/README.md
index be5e78e..86209ad 100644
--- a/README.md
+++ b/README.md
@@ -72,21 +72,27 @@ flowchart TD
HeadFresh -- No --> Revalidate["Authenticated HEAD revalidation with If-None-Match"]
Revalidate --> HeadResult{"HEAD result"}
HeadResult -- "304" --> Resolve
- HeadResult -- "Changed" --> Objects
+ HeadResult -- "Changed" --> DecodeHead
HeadResult -- "Network unavailable and cached HEAD usable" --> Resolve
HeadResult -- "Network unavailable and no usable cache" --> ReadError["Return explicit read error"]
Resolve --> ValuesCached{"Required immutable values cached?"}
ValuesCached -- Yes --> Plan["Validate document or query plan, predicate, ordering, projection, and limit"]
- ValuesCached -- No --> Objects
+ ValuesCached -- No --> Objects["Revalidate read grant and return missing immutable TDB1 objects"]
BundleCheck -- Yes --> Bundle["Authority revalidates read grant, reads encrypted HEAD and required objects"]
Bundle --> BundleLimit{"At most 4 objects and 4 MiB with authenticated size metadata?"}
BundleLimit -- Yes --> Decoded["Return decoded cache values over HTTPS with no-store"]
- BundleLimit -- No --> Objects
- BundleCheck -- No --> Objects["Revalidate read grant and return authenticated TDB1 objects"]
- Objects --> BrowserDecrypt["Browser decrypts and validates HEAD, index, snapshot, or trie objects"]
+ BundleLimit -- No --> FetchHead
+ BundleCheck -- No --> FetchHead["Revalidate read grant and return authenticated TDB1 HEAD"]
+ FetchHead --> DecodeHead["Browser decodes, decrypts when required, and validates HEAD within the safety limit"]
+ DecodeHead --> CacheHead["Encrypt decoded HEAD with device key and update the scoped cache"]
+ CacheHead --> Resolve
+ Objects --> BrowserDecrypt["Browser decodes, decrypts when required, and validates index, snapshot, or trie objects"]
+ BrowserDecrypt --> DecodedLimit{"Decoded object within configured safety limit?"}
+ DecodedLimit -- No --> LimitError["Return explicit decoded-size error"]
+ DecodedLimit -- Yes --> DeviceCache
Decoded --> DeviceCache["Encrypt decoded values with device key and update the scoped cache"]
- BrowserDecrypt --> DeviceCache --> Plan
+ DeviceCache --> Plan
Plan --> ReadResult["Document, or bounded query result with point, index, or scan plan"]
```
@@ -96,14 +102,15 @@ flowchart TD
2. Cold point reads can use one
bounded decoded bundle when explicitly enabled; every ineligible or failed
bundle falls back to authenticated TDB1 object reads.
-3. Queries remain bounded and report whether they used a point, declared
- index, covering projection, or collection scan plan.
+3. Queries remain bounded and report a point, index, or scan plan. Explicit
+ selected fields can be served from a covering index without full-document
+ reads.
### Mutation and cache-synchronisation flow
```mermaid
flowchart TD
- Mutation["Create, replace, delete, restore, purge, scope erase, or index rebuild"]
+ Mutation["Create, replace, delete, or restore one document"]
Mutation --> Request["Session + CSRF + exact Origin scope + layout generation"]
Request --> Guards{"Operation allowed by maintenance state and generation current?"}
Guards -- No --> Reject["Return explicit maintenance or layout-changed error"]
@@ -111,14 +118,14 @@ flowchart TD
Grant --> Authorised{"Authorised?"}
Authorised -- No --> Deny["Return explicit forbidden response"]
Authorised -- Yes --> Load["Read current HEAD and affected immutable objects"]
- Load --> Validate["Validate route, ID, document, limits, layout, and complete index configuration"]
- Validate --> Immutable["Create immutable document, root, and index objects"]
+ Load --> Validate["Validate route, ID, document, decoded-object limit, layout, and complete index configuration"]
+ Validate --> Immutable["Create immutable layout and index objects"]
Immutable --> Publish["Publish one HEAD with If-Match"]
Publish --> Conflict{"ETag conflict?"}
Conflict -- Yes --> Retry{"Bounded retry remains?"}
Retry -- Yes --> Load
Retry -- No --> ConflictError["Return explicit conflict"]
- Conflict -- No --> Commit["Return committed values and new ETag"]
+ Conflict -- No --> Commit["Return committed cache-value bundle"]
Commit --> Cache["Update the current scoped cache"]
Cache --> Tabs["Notify other tabs through BroadcastChannel"]
Tabs --> Result["Committed mutation result"]
@@ -128,6 +135,9 @@ flowchart TD
index objects, and publishes their references through one conditional HEAD.
2. Successful writes update the current cache and notify other tabs. Logout
revokes the session and clears the affected browser cache namespace.
+3. Administrative purge, scope erase, index rebuild, and layout migration use
+ separate guarded endpoints. They do not return the ordinary document
+ mutation bundle shown here.
### Deployment, scaling, and storage flow
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 0a5513f..e9ea515 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -75,18 +75,20 @@ encryption. Private scopes use a versioned AES-256-GCM data key.
immutable-object bundle in one browser request.
3. Older authorities, legacy metadata, oversized bundles, and cache hits use
the individual encrypted-object path.
-4. If a cached HEAD TTL expired, the browser revalidates it with
+4. Each individual object is decoded with a 16 MiB default safety limit before
+ JSON parsing or cache insertion.
+5. If a cached HEAD TTL expired, the browser revalidates it with
`If-None-Match`.
-5. A 304 response keeps the current layout generation.
-6. ID equality resolves directly to one document path.
-7. A matching declared index resolves a bounded set of candidate IDs.
-8. An explicit `.select(...)` can use declared covering fields without
+6. A 304 response keeps the current layout generation.
+7. ID equality resolves directly to one document path.
+8. A matching declared index resolves a bounded set of candidate IDs.
+9. An explicit `.select(...)` can use declared covering fields without
loading full documents.
-9. Queries without a usable index use an explicitly bounded scan.
-10. Trie HEAD points to an immutable root, branch, and leaf path.
-11. Snapshot HEAD points to one immutable collection snapshot.
-12. The browser coalesces concurrent reads of the same immutable object.
-13. It re-evaluates the complete predicate, orders, limits, and returns the
+10. Queries without a usable index use an explicitly bounded scan.
+11. Trie HEAD points to an immutable root, branch, and leaf path.
+12. Snapshot HEAD points to one immutable collection snapshot.
+13. The browser coalesces concurrent reads of the same immutable object.
+14. It re-evaluates the complete predicate, orders, limits, and returns the
query plan with the documents.
Immutable pages do not need revalidation. Their object key identifies their
@@ -101,7 +103,8 @@ private.
1. The browser sends a mutation to the authority.
2. The authority authenticates the session and resolves allowed scopes.
-3. Application validation runs before storage work.
+3. Application validation and the decoded-object limit run before storage
+ publication.
4. Changed trie pages or the next immutable snapshot are serialised,
gzip-compressed when useful, and encrypted.
5. Every configured secondary index and declared covering projection is
diff --git a/docs/DIAGRAMS.md b/docs/DIAGRAMS.md
index 832d3fc..8424ac3 100644
--- a/docs/DIAGRAMS.md
+++ b/docs/DIAGRAMS.md
@@ -6,61 +6,60 @@ index, deletion, migration, and deployment paths.
## Trust boundaries
```mermaid
-flowchart LR
+flowchart TB
subgraph Browser["Browser trust boundary"]
+ direction LR
UI["Application UI"]
Client["ThimbleDB client"]
- Memory["Scope-namespaced memory cache"]
- IDB["Scope-namespaced, device-key-encrypted IndexedDB"]
+ Cache["Scope-namespaced memory and encrypted IndexedDB"]
ReadKeys["Non-extractable decrypt-only scope keys"]
UI --> Client
- Client --> Memory
- Client --> IDB
+ Client --> Cache
Client --> ReadKeys
end
subgraph Authority["Application authority boundary"]
- Exchange["OIDC token exchange"]
- Session["Opaque revocable session"]
- Authorizer["Current scope-grant calculation"]
+ direction LR
+ Identity["OIDC exchange and opaque session"]
+ Access["Current scope grants and key derivation"]
Broker["Authenticated ciphertext-object and decoded-bundle broker"]
Mutation["Mutation validation and execution"]
- Exchange --> Session
- Session --> Authorizer
- Authorizer --> Broker
- Authorizer --> Mutation
+ Identity --> Access
+ Access --> Broker
+ Access --> Mutation
end
subgraph DataStore["Private data object storage"]
+ direction LR
Heads["Mutable collection HEAD records"]
Pages["Immutable encrypted document and index pages"]
end
subgraph AuthStore["Separate private authentication storage"]
- Identities["Identity mappings and users"]
- Sessions["Session digests and CSRF state"]
- Limits["Authentication rate-limit state"]
+ direction LR
+ AuthRecords["Identity mappings, session digests, CSRF, and rate-limit state"]
end
subgraph Platform["Platform secret boundary"]
+ direction LR
Master["Deployment master key"]
Provider["Storage credential or platform binding"]
end
- UI -- "Session cookie + CSRF + requested scope" --> Authority
- Client -- "Ciphertext objects or opt-in decoded bundles" --> Broker
+ UI -- "OIDC token" --> Identity
+ Client -- "Mutation + CSRF" --> Mutation
+ Client -- "Read + scope" --> Broker
+ Broker -- "Ciphertext or bundle" --> Client
+ Access -- "Scope keys" --> ReadKeys
Broker --> Heads
Broker --> Pages
- Mutation -- "Create immutable objects" --> Pages
- Mutation -- "Conditional HEAD publication" --> Heads
- Exchange --> Identities
- Session --> Sessions
- Exchange --> Limits
- Master --> Mutation
- Master --> ReadKeys
- Master --> AuthStore
+ Mutation -- "Immutable writes" --> Pages
+ Mutation -- "CAS HEAD" --> Heads
+ Identity --> AuthRecords
+ Master -- "Derived keys" --> Access
+ Access --> AuthRecords
Provider --> Broker
Provider --> Mutation
```
@@ -175,6 +174,7 @@ flowchart TD
Bundle{"Cold cache and bundle endpoint available?"}
BundleRead["One bounded authority read bundle"]
BundleValidate["Validate bundled HEAD and immutable values"]
+ BundleCache["Apply bundled values to the scoped cache"]
Index{"Matching declared index with indexable values?"}
PointHead["Read collection HEAD"]
IndexHead["Read collection HEAD"]
@@ -190,8 +190,11 @@ flowchart TD
Query --> Validate --> Point
Point -- Yes --> Bundle
- Bundle -- Yes --> BundleRead --> BundleValidate --> Result
- Bundle -- No or fallback --> PointHead --> ReadDocs
+ Bundle -- Yes --> BundleRead --> BundleAccepted{"Bundle accepted?"}
+ BundleAccepted -- Yes --> BundleValidate --> BundleCache --> Result
+ BundleAccepted -- "Unavailable, legacy, or over limit" --> PointHead
+ Bundle -- No --> PointHead
+ PointHead --> ReadDocs
Point -- No --> Index
Index -- Yes --> IndexHead --> IndexPage --> Candidates --> Covered
Covered -- Yes --> Projection --> Predicate
@@ -218,35 +221,51 @@ sequenceDiagram
Client->>Client: Validate query and choose point, index, or scan plan
Client->>Cache: Read collection HEAD
- alt Cold point read and bundle endpoint advertised
+ alt Eligible cold point read and bundle endpoint advertised
Client->>Broker: GET bounded point-read bundle
Broker->>Broker: Require current read grant and enforce object/byte limits
- Broker-->>Client: Decoded HEAD and immutable cache values over HTTPS
- Client->>Cache: Store returned values with device-key encryption
+ alt Bundle accepted
+ Broker-->>Client: Decoded HEAD and immutable cache values over HTTPS
+ Client->>Cache: Store returned values with device-key encryption
+ else Bundle unavailable, legacy, or oversized
+ Broker-->>Client: Explicit fallback status
+ Client->>Broker: GET encrypted HEAD with session
+ Broker->>Store: Read object with ETag condition
+ Store-->>Broker: Encrypted HEAD
+ Broker-->>Client: Encrypted HEAD
+ Client->>Client: Decode TDB1 within the per-object limit
+ Client->>Cache: Store decoded value under device-key encryption
+ end
else HEAD missing or stale
Client->>Broker: GET encrypted HEAD with session
Broker->>Store: Read object with ETag condition
Store-->>Broker: Encrypted HEAD
Broker-->>Client: Encrypted HEAD
- Client->>Cache: Store decrypted value under device-key encryption
+ Client->>Client: Decode TDB1 within the per-object limit
+ Client->>Cache: Store decoded value under device-key encryption
+ else Fresh cached HEAD
+ Client->>Cache: Use cached HEAD
end
- alt Index plan
- Client->>Cache: Read referenced immutable index page
- Client->>Broker: Fetch index page on cache miss
- Client->>Client: Resolve candidate IDs within maxScan
- alt Explicit projection is covered
- Client->>Client: Validate predicate and order from declared projections
- Client->>Client: Build projected documents without full-document reads
- else Full documents required
- Client->>Cache: Resolve snapshot once or shared trie nodes
- Client->>Broker: Fetch only missing immutable objects
+ opt Non-point query
+ alt Index plan
+ Client->>Cache: Read referenced immutable index page
+ Client->>Broker: Fetch index page on cache miss
+ Client->>Client: Resolve candidate IDs within maxScan
+ alt Explicit projection is covered
+ Client->>Client: Validate predicate and order from declared projections
+ Client->>Client: Build projected documents without full-document reads
+ else Full documents required
+ Client->>Cache: Resolve snapshot once or shared trie nodes
+ Client->>Broker: Fetch only missing immutable objects
+ end
+ else Scan plan
+ Client->>Client: Enforce bounded collection scan
end
- else Scan plan
- Client->>Client: Enforce bounded collection scan
end
- Client->>Client: Decrypt when required, validate candidates, order, and limit
+ Client->>Client: Decode fetched TDB1 objects within the per-object limit
+ Client->>Client: Validate candidates, order, and limit
Client-->>App: Documents plus point/index/scan plan
```
@@ -265,14 +284,19 @@ sequenceDiagram
Authority->>Auth: Reload user and recalculate grants
Auth-->>Authority: Current write grant or denial
Authority->>Store: Read HEAD and affected immutable objects
- Authority->>Authority: Validate document and update configured indexes
- Authority->>Store: Create immutable document/root objects
+ Authority->>Authority: Validate document, decoded-object limit, and configured indexes
+ Authority->>Store: Create immutable layout objects
Authority->>Store: Create immutable index pages
Authority->>Store: Publish one HEAD with all document and index references using If-Match
alt ETag conflict
Store-->>Authority: Precondition failed
- Authority->>Store: Reload current HEAD and retry
+ Authority->>Authority: Increment bounded retry count
+ alt Retry remains
+ Authority->>Store: Reload current HEAD and retry
+ else Retry exhausted
+ Authority-->>App: Explicit conflict
+ end
else Commit
Store-->>Authority: New HEAD ETag
Authority-->>App: Committed object bundle
diff --git a/site/public/diagrams/0c8435b9462cd3a63ff5.svg b/site/public/diagrams/0c8435b9462cd3a63ff5.svg
new file mode 100644
index 0000000..316e641
--- /dev/null
+++ b/site/public/diagrams/0c8435b9462cd3a63ff5.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/site/public/diagrams/79ad5efd6e4ed3f2a894.svg b/site/public/diagrams/79ad5efd6e4ed3f2a894.svg
new file mode 100644
index 0000000..3d67013
--- /dev/null
+++ b/site/public/diagrams/79ad5efd6e4ed3f2a894.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/site/public/diagrams/897ea0c26deb6b3cb03a.svg b/site/public/diagrams/897ea0c26deb6b3cb03a.svg
deleted file mode 100644
index c0fa702..0000000
--- a/site/public/diagrams/897ea0c26deb6b3cb03a.svg
+++ /dev/null
@@ -1 +0,0 @@
-
\ No newline at end of file
diff --git a/site/public/diagrams/bdfadb8fe6895bdd60be.svg b/site/public/diagrams/bdfadb8fe6895bdd60be.svg
new file mode 100644
index 0000000..017b76d
--- /dev/null
+++ b/site/public/diagrams/bdfadb8fe6895bdd60be.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/site/public/diagrams/c5bc1a796ce8e93f2840.svg b/site/public/diagrams/c5bc1a796ce8e93f2840.svg
deleted file mode 100644
index 9b06136..0000000
--- a/site/public/diagrams/c5bc1a796ce8e93f2840.svg
+++ /dev/null
@@ -1 +0,0 @@
-
\ No newline at end of file
diff --git a/site/public/diagrams/5628899b5012f929f172.svg b/site/public/diagrams/e792e1d4c044189f52e3.svg
similarity index 54%
rename from site/public/diagrams/5628899b5012f929f172.svg
rename to site/public/diagrams/e792e1d4c044189f52e3.svg
index ce0ad3a..1344b81 100644
--- a/site/public/diagrams/5628899b5012f929f172.svg
+++ b/site/public/diagrams/e792e1d4c044189f52e3.svg
@@ -1 +1 @@
-
\ No newline at end of file
+
\ No newline at end of file
diff --git a/site/public/diagrams/f9b79a62f37a16efb8ec.svg b/site/public/diagrams/f9b79a62f37a16efb8ec.svg
deleted file mode 100644
index 5fc7c63..0000000
--- a/site/public/diagrams/f9b79a62f37a16efb8ec.svg
+++ /dev/null
@@ -1 +0,0 @@
-
\ No newline at end of file