Repository navigation
docs(plans): design json_tree tokens derivable from the path alone - #1361
Conversation
An object member's token mixes in its ordinal, a keyed rank among its siblings that the client cannot know for a row it has not read. So the client can derive a token only for a top-level field and the array positions after it, and walks the rest through gates. A read of $.profile.address.city therefore sends $.profile's token, a capability over that whole field in every row, and the fix proposed for #1356 widens that further by sending an array's token to read one element. Record the agreed change: member tokens drop the ordinal, every entry gains an eight-byte self tag so a read can find a node from its own token and return it at depth 0, and the client expands wildcards into concrete paths with a bound and a probe, so gates and json_tree_subtree_at go away. The section gives the derivations, the SQL read, the adapted client check, the wire change, the leakage, the implementation and measurement plan, and the inconsistencies found while checking the design against the code, as open questions for the owner. Short dated pointers mark where sections 2, 4 and 6 and the Phase 3b notes state what the change reverses. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The leakage page told customers that a subtree read sends the token of the top-level field the path starts with. Under the design change in the json_tree design record, a read sends the token of exactly the node asked for, a wildcard sends a bounded list of concrete tokens and one probe per level, and every entry carries a self tag. Revise the subtree read bullet, add the self tag to the at-rest table and the unauthenticated fields, adapt the walk check, the comparison row and the sealed_root option, so the page describes the format the implementation stacked on this branch will build. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
…e-derivable-tokens Brings in the review fixes on #1358: a walk cut short is JsonWrongEntry, a walk with no gates refuses an array that does not start at element 0, and the leakage page says what that walk still does not check. The leakage page conflicted. This branch's "What is authenticated" describes the derivable-token design, in which every read returns the node it names first, so the ungated-walk paragraph #1358 added does not apply here; this branch's text is kept. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…rivable-tokens Brings in the review fixes on #1358 through #1361. Conflicts in `json/derive.rs`, `json/query.rs` and `tests/json_read_checks.rs`, all where this branch replaced the gated and ungated walks with token reads. This branch's `place` and `JsonSubtreeKey::open` are kept: the leading-element check #1358 added belongs to the walk with no gates, which no longer exists here, and every read now returns the node it names with the child count that bounds its children. The truncation fix is carried over by hand: `rebuild_walked` here still refused rows that ran out before a child count was met as `OpenedType`, while the leakage page lists a truncated row among the `JsonWrongEntry` refusals. It is now `JsonWrongEntry`, like a gap. `a_truncated_read_is_refused` now expects that. #1358's ungated-walk tests are replaced by one that pins what this design closes: an array read missing its first or its last element is refused. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…quality
Three statements on the json_tree leakage page were wrong or missing, and
one was new with the derivable-token design:
- An object's members are stored in the byte order of their keys' UTF-8
encodings (serde_json's default BTreeMap), not alphabetical order, so `Z`
sorts before `a`. A build that unifies serde_json's `preserve_order`
feature stores insertion order instead, which the design doc's Phase 4
notes already record and the page did not.
- The `op` row did not say how long a term is. A CLLW OPE term is one byte
per bit of the order encoding plus a carry byte: 65 bytes for every
number, 8 x (n + 1) + 1 for a string whose folded UTF-8 form is n bytes,
so it shows a string's length. Strings are case- and accent-folded first
("Abc", "abc" and "ábc" get one term), so the equality it shows is of
folded values. Both measured under the deterministic key source.
- The `op` row says leaves at two ordered paths compare, and the term is
deterministic under one key per column, so two ordered paths holding
equal values carry equal terms. The `q` row and the closing "not
revealed at rest" line said the opposite without the exception.
- A wildcard read sends a probe for every array it expands, not one per
wildcard level: two levels send N + 1 probes (`probes()` in the key,
`nested_wildcards_multiply` in derive.rs). The bullet and the comparison
table now say so, and that for `$.items[*].price` the tokens show which
elements hold `price`, not the array's length.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…t of its error
Two statements in the derivable-token design disagreed with the rest of
the section and with the code that implements it:
- Section 3 said a truncated wildcard read fails with an error naming the
bound and the path. The error names the bound only, deliberately: a
path's key names can be customer data, and no error message in the crate
repeats one (`Error::JsonWildcardTruncated { bound }`).
- Section 6 said a wildcard read sends N + 1 tokens per level. It sends
N + 1 for each array it expands, which section 3 and the code count as
N^2 + N + 1 for two levels; and for a path that continues below the
element, the tokens show which elements hold the rest of the path, which
is an array's length only where the path ends at the element.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Review of the design write-up (as of 790d827)I checked the "Design change (2026-10-24)" section and the leakage page against #1363's code and SQL, and against the owner's decisions. Findings, most important first. Items 3 to 6 are on the leakage page; 4 to 6 are statements about the stored format that this PR's revision of the page carries forward, and are the three that #1362's agent recorded against an earlier version of the page. All six are confirmed against the code.
Checked and consistent: the token, self-tag and parent-tag formulas in §1, §2 and the decisions match On #1363 only, where the decisions replaced the open questions: §3 still pointed to "the open questions" and gave the walk's limit as the entry count plus one, against decision 9's entries × tokens. I fixed both there, since this branch's text is right about its own open questions. Fixes for 1 to 6 follow in commits on this branch. |
Fixes pushed (head now f2331ab)All six findings from the review comment above are fixed, in two commits:
None is left open. This branch was merged into #1363 at e0191df with no conflicts; on that branch I also fixed the two stale pointers to the open questions (d1025be). The doc-anchor gate passes. The base is unchanged ( |
…e-derivable-tokens Brings in next after #1345, through #1358's merge of it. No conflict: the only line of these two documents that #1345 touched is the design record's "Where it is" paragraph (the generated oracle and the proptest properties), which this branch does not change. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…rivable-tokens Brings in next after #1345 (through #1358 and #1361). Nothing in this branch's walk, derivations, SQL, EQL types or leakage page changes: the merge touches only tests, the proptest regressions and the oracle. Conflicts: - packages/stack-encrypt/Cargo.toml: both sides added the same proptest dev-dependency line with different comments. One line, one comment naming both uses. - packages/stack-encrypt/tests/json_index.rs: #1345 rewrote it into given_<scenario> modules. I took that file (with #1358's changes already in it) as the base and re-applied this branch's changes to the tests they now live in: - helpers: gated_walk becomes read_rows (the rows json_tree_subtree returns for key.tokens()); subtree_ordinals takes the token list; child_token and subtree_nodes are gone, with nothing left to use them. - given_the_fixture_document: stores_the_wire_form_the_sql_domain_checks expects the self tag `x`; places_the_chain_fields_where_the_design_says checks one distinct self tag per entry; stores_no_token follows the token list and checks self tags as well as parent tags. - given_a_subtree_key: #1345 split this branch's old token test three ways, and each part is ported to the token list. unlocks_exactly_its_subtree_from_the_path_alone is renamed finds_exactly_its_node_from_the_path_alone (one token, no probe, the node itself at depth 0). gates_each_key_step_below_its_head is renamed derives_every_token_below_its_head (the second element's own token, the server deriving the token the client sends, find). Its wildcard gates assertion is dropped, because a wildcard key carries no gates; the wildcard's tokens and probe are pinned in given_wildcard_rows. has_no_token_at_a_sealed_root_and_derives_from_an_open_one is renamed reads_the_root_under_either_option: `$` is now a read under the sealed root, so its TermError::JsonSealedRoot refusals (`$`, `$[*].x`) and the open-root gate count are gone, as on this branch. - given_a_position: reaches_no_object_member takes this branch's form, over `$.a[0]` and `$.h.a[0]`. That covers gates_on_the_elements_selector_below_a_key_step (a gates() assertion), which is removed. - given_an_extracted_node: the walk reads by token, and the $.nested read opens as a list to the one node at the path. - given_gated_walk_rows::open_to_the_reached_nodes_value becomes given_wildcard_rows::open_to_one_value_per_element_or_fail_with_the_bound, this branch's test. With it goes #1358's "ungated rows under a gated key" refusal: no walk is gated now. - given_a_derived_record: rows from read_rows. - given_a_fallback_root::opens_to_what_the_walk_opens_to takes this branch's form (paths compared against JsonPath::select, as a list and as an optional value). - given_rows_the_server_chose: the repeated row is JsonWrongEntry, as febc4b2 decided. #1345's generated oracle (json_tree/generated.rs) already opens through JsonTreeExtracted::from_selector and builds against this branch's types unchanged. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f3a384eab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
cipherstash-agent
left a comment
There was a problem hiding this comment.
Review attestation
Source: automated Claude Code review (/code-review medium), cross-checked with company-skills:cross-check-review-findings, against head 2f3a384.
Critical or blocking issues: none.
New findings: 1 minor, inline: the open-question 1 example does not show what dropping the consecutive-position check would lose.
Already covered: the six findings in coderdan's earlier review comment (probe counts, op length, equal values at two paths, key order, truncation error text, per-array token counts). All six are fixed at this head.
Checked and consistent: the token input lengths (33 and 37 bytes), the N² + N + 1 = 4,161 count, the x size estimate, the op term lengths, and the two-doc cross-references. Docs only, so no changeset or skill update is needed.
Reminder from the PR itself: do not merge ahead of the implementation.
Generated by Claude Code
…xception in the token Answers the two review threads on #1361. The probe thread (open question 3): the question said a separate probes argument, answered with existence only, would remove the leakage exception for a probe that hits. It would not. A probe is the element's token, and whoever holds a token can find that element in any row by its self tag, probe its children's parent tags and derive every descendant's token from the public derivations of sections 1 and 2, whatever the SQL function returns, in rows written later too. The capability is in the token, so the exception holds for every variant that sends it. Only a different derivation, such as a one-way existence value sent in place of the token and a tag stored from it, would be existence-only; that is a format change, recorded as a future option. Section 6 and the leakage page now say the same. The question also said the server cannot tell a probe from any other token, which #1363 corrected in the SQL comment: it can. The consecutive-position thread (open question 1 and section 4): the example given for keeping the check was a swap the check does not see in the form described. The new example is one it does refuse and nothing else does: a read of $.items[0] answered with the element's own entry and then $.items[1].price, its self tag rewritten, which has the right selector, label and child count and the wrong position. The question now also says what the rule does not reach (a depth-0 row, such as a read of $.items[1].price itself). Section 4 no longer says the self-tag check replaces the rule: x is a consistency check, and the bound fields with the consecutive-position rule are the security check, as the leakage page has it. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…rivable-tokens Brings in 0edead7, the answers to the two review threads on #1361: the probe exception is in the token, not in what the SQL does with it, so it holds for every variant that sends the element's token; and the consecutive-position check is part of the security check, which the self-tag check does not replace. The design doc conflicted in section 4 and where #1361's open questions became this branch's decisions. This branch's text is kept in both, and the corrections are carried into it: section 4 says nothing replaces the consecutive-position rule; decision 3 says a separate existence-only argument would not remove the probe exception and records an existence-only derivation as a future option; decision 8 gains the example the rule alone refuses (a read of $.items[0] answered with $.items[1].price) and says the allowed swap is of a depth-0 row; and section 6 points at decision 3. The leakage page merged cleanly. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…catches Design decision 8 keeps the consecutive-position check, and the design record's example for it (corrected on #1361) is a read of one array element answered with the element itself and then the other element's child in place of its own, self tag rewritten: selector, label, depth and child count all agree, and only the row's authenticated position refuses it. No committed test isolated that case; the closest ones are also caught by the child counts or the selectors. This one opens to {"x": 2} for {"x": 1} with the j comparison in rebuild_walked removed. Refs #1363 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Summary
A write-up only: no code changes. EQL v4's
json_treecolumn stores an encrypted JSON document as one entry per JSON node. To read part of one, the client sends the database a token, a secret derived from the path that lets the database map the structure under that node in every row. Today an object member's token mixes in its place among its siblings, which the client cannot know. So a read of$.profile.address.cityhas to send the token of the whole$.profilefield. The same cause is behind #1356, and the fix proposed in #1359 widens the leak further. This PR records the agreed change that removes it (tracked in #1360):It also revises the leakage page to say what the database learns after the change. It lists eight open questions, found while checking the design against the code, for the owner to answer before implementation starts.
Changes
docs/plans/2026-10-17-json-tree-index.md: a new dated section, "Design change (2026-10-24): derivable tokens, the self tag, and wildcard expansion on the client". It covers why, then the token rules, the self tag, wildcard expansion and the new SQL read, the client's checks, the wire change, the leakage, the plan with its test list and benchmark step, and the open questions. One-line dated pointers sit where sections 2, 4 and 6, the Phase 1/2 deviations and the Phase 3b notes state what the change reverses. Nothing earlier is rewritten.packages/eql/docs/reference/json-tree-leakage.md: the subtree-read bullet (exact tokens, wildcard expansion and its probe, the root), a new row for the self tagxin the at-rest table,xin the list of fields that are not authenticated, the walk check, the comparison row, and thesealed_rootoption.Do not merge this ahead of the implementation. The leakage page will then describe a format that is not built yet. The plan is to stack the implementation on this branch so the two land together.
No changeset: this changes documentation only, in the repository's design record and the EQL reference page, and no published package's surface.
Verification
bash tasks/test/doc-anchors.shinpackages/eql(withEQL_ROOTset to the subtree):doc-anchor gate OK — 123 intra-document link(s) resolve.docs:validate:sourcewas not run: it checks the--!comments in the SQL sources only, and this PR changes no SQL.grep eql_v2on the leakage page: no match (whattest:docs_v3_grepchecks).child_token,parent_tag,rank/shuffled_ordinals,entry_aad, the walk's head derivation, andsubtree_key_forinpackages/stack-encrypt/src/json/derive.rs;selector_bytes/token_bytesinpath.rs; andjson_tree_child_token,json_tree_subtree_atand the walk's "bound" comment inpackages/eql/src/v3/json_tree/functions.sql.Related
Refs #1360 (the issue this designs).
Refs #1356 (solved as a side effect), #1359 (superseded in mechanism; its behaviour and oracle test carry over), #1358 (base branch; its checks are adapted), #1357 (unchanged by this), #1345 (should merge before #1358).
https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu