feat(eql, stack-encrypt): chain json_tree tokens by ordinal and probe the walk to the first miss

.changeset/eql-json-tree-ordinal-chain.md

@@ -0,0 +1,11 @@
+---
+"@cipherstash/eql": minor
+---
+
+EQL v4 `json_tree` (unreleased): the token chain is derived by ordinal and the walk probes to the first miss.
+
+A stored entry now carries four fields, `s`, `c`, `p` and `op`. The value selector `vs` lives only in the document's projection `q`, which the engine now reads back as the projection instead of rebuilding it from the entries; the group tags `n` and `g` are gone with it, since nothing read them once `q` was stored. The wrapped token `w` and the ordinal `o` are gone too: a child's token is `H("t" ‖ t(parent) ‖ LE32(o))`, which the server derives as it walks, and the ordinals under a node are dense from zero, so the walk probes them in turn and stops at the first miss. A subtree of `N` nodes costs about `2N` probes (a SHA-256 and a lookup each) and `N` token hashes whatever the document's width, where it cost a probe for every ordinal up to the document's widest node at every node visited. The entry validator accepts the four fields only, so a document written by the earlier build of this unreleased family is refused.
+
+The client can derive a token only for a chain head and the array positions that directly follow it, since an object member's ordinal is a rank among its siblings' keys. `JsonSubtreeKey::gates()` gives the rest of a path as one gate per step, a selector or `#n` for an array position, and `eql_v4.json_tree_subtree_at(doc, token, gates text[], budget DEFAULT 64)` walks them, replacing the earlier `json_tree_subtree_at(doc, token, selector, budget)`. `json_tree_children` and `json_tree_child_count` take the same `gates`, optional and empty by default. A gated walk returns the node the gates reach with `d = 0` and its descendants beneath it, so a wildcard reaches several nodes and opens as a list; `json_tree_subtree(doc, token, budget)` is now `json_tree_subtree_at` with no gates and still returns the descendants of the token's node from `d = 1`. Past the budget a walk still returns the sealed root alone, with `j = 0` and `d = 0`. `eql_v4_internal.json_tree_unwrap`, `json_tree_xor16` and `json_tree_fanout` are removed; `eql_v4_internal.json_tree_child_token` is added.
+
+Measured on 10,000 and 100,000 rows of a nested corpus (about 213 entries a document), query-only medians: subtree walks 30 to 55% faster (a 122-entry subtree walked out, 23 ms to 11 ms), `json_tree_children` and `json_tree_child_count` 6 to 9%, a stored nested document 25% smaller (heap 810 MB to 621 MB at 10,000 rows), containment and field queries unchanged within the run's noise.

.changeset/eql-json-tree-walk-and-projection.md

@@ -11,3 +11,5 @@
 Also: `eql_v4.json_tree_path_query_first`, the function `->` and the ordering comparisons extract through, is costed at 1000 so a prepared statement's generic plan keeps the functional btree on `eql_v4.ord_term(col -> sel)` instead of filtering a sequential scan (17 ms down to 0.1 ms after the fifth execution). `eql_v4_internal.json_tree_group`, the server-side projection builder, is removed.
 
 The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct with `rows` and an optional `path`; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtree::at`, `path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged.
+
+**The leaf ordering term `op` is opt-in per path.** A stored leaf carried the order-preserving term on every number and string, about 80 bytes each and the largest field of a nested document, whether or not anything ordered by it. Now the JSON index's options name the paths that order (`{"json": {"ordered": ["$.age", "$.orders[*].total"]}}` on the data plan; `JsonOptions::ordering([...])` in Rust), a wildcard covering every element of an array and a position one element, and no path is ordered by default. An ordering query at a path the index does not order is refused before any key loads rather than silently matching no row. The SQL is unchanged: `eql_v4.ord_term` was already NULL for a leaf without a term. `eql-bindings` gains `v4::JsonTree`, the `json_tree` layout under options, and `stack-encrypt` gains `target::Layout` and `Plan::value().encrypt_as(layout)`, a plan laid out by a value whose queries run under its options.

packages/eql/crates/eql-bindings/src/v4/json_tree.rs

@@ -23,9 +23,9 @@ use schemars::{schema_for, JsonSchema, Schema};
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
 use stack_encrypt::json::{
-    JsonDocument, JsonLeafQuery, JsonNode, JsonPath, JsonQuery, JsonSubtree,
+    JsonDocument, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSubtree,
 };
-use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec};
+use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec, Json, Layout};
 use stack_encrypt::{
     DecryptField, DecryptInto, Decryptable, Decryption, EncryptFrom, Encryption, Error, NonEmpty,
 };
@@ -35,19 +35,20 @@ use crate::v3::terms::{EntryCiphertext, KeyHeader, OpeCllw, Selector};
 use crate::v4::{DomainType, SchemaVersion};
 use crate::Identifier;
 
-/// A sixteen-byte chain tag, hex: a parent tag `p`, a wrapped token `w`,
-/// a group tag `n` or `g`. Per document, salted by the document's `iv`, so
-/// unlinkable across rows at rest.
+/// A sixteen-byte chain tag, hex: a parent tag `p`. Per document, salted
+/// by the document's `iv`, so unlinkable across rows at rest.
 #[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, TS, JsonSchema)]
 pub struct JsonTreeTag(pub String);
 
 /// `public.eql_v4_json_tree`: a JSON tree document, `{v, i, h, sv:
 /// [entry], q: group}`. Strict. `h` is the document's key header, stored
 /// once (every entry is sealed under the document's one data key); `sv`
 /// holds one entry per node in depth-first order, the root first; `q` is
-/// the document's containment projection, its `{s, vs}` pairs nested by
-/// array element, which the client writes beside the entries so the
-/// server's `eql_v4.json_tree_query` reads it rather than rebuilding it.
+/// the document's containment projection, one `{s, vs}` pair per node
+/// nested by array element, which the client writes beside the entries
+/// and which is the only place the value selectors live: the server's
+/// `eql_v4.json_tree_query` reads it, and the engine takes it as the
+/// projection when it reads a document back.
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
 #[serde(deny_unknown_fields)]
 pub struct JsonTreeDocument {
@@ -58,35 +59,21 @@ pub struct JsonTreeDocument {
     pub q: JsonTreeQueryGroup,
 }
 
-/// `public.eql_v4_json_tree_entry`: one node. `s` and `vs` are the path
-/// and value selectors, `c` the node's own ciphertext; `o`, `p` and `w`
-/// the chain fields below the heads; `n` and `g` the group tags of
-/// container array elements and what sits inside them; `op` the leaf
-/// ordering term. `h`, `j`, `d`, `v` and `i` are what the extractors graft
-/// onto an entry they return, so it opens on its own.
+/// `public.eql_v4_json_tree_entry`: one node. `s` is the path selector,
+/// `c` the node's own ciphertext; `p` the parent tag below the chain's
+/// heads, which the walk probes by ordinal; `op` the leaf ordering term.
+/// `h`, `j`, `d`, `v` and `i` are what the extractors graft onto an entry
+/// they return, so it opens on its own.
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
 #[serde(deny_unknown_fields)]
 pub struct JsonTreeEntry {
     pub s: Selector,
-    pub vs: Selector,
     pub c: EntryCiphertext,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     #[ts(optional)]
-    pub o: Option<u32>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    #[ts(optional)]
     pub p: Option<JsonTreeTag>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     #[ts(optional)]
-    pub w: Option<JsonTreeTag>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    #[ts(optional)]
-    pub n: Option<JsonTreeTag>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    #[ts(optional)]
-    pub g: Option<JsonTreeTag>,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    #[ts(optional)]
     pub op: Option<OpeCllw>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     #[ts(optional)]
@@ -237,9 +224,11 @@ impl JsonTreeDocument {
         serde_json::from_value(value).map_err(codec)
     }
 
-    /// The engine's document: the header and entries without the envelope.
+    /// The engine's document: the header, entries and projection without
+    /// the envelope.
     fn document(&self) -> Result<JsonDocument, Error> {
-        serde_json::from_value(serde_json::json!({ "h": self.h, "sv": self.sv })).map_err(codec)
+        serde_json::from_value(serde_json::json!({ "h": self.h, "sv": self.sv, "q": self.q }))
+            .map_err(codec)
     }
 }
 
@@ -281,19 +270,62 @@ impl JsonTreeEntry {
 // Stack Encrypt targets
 // =============================================================================
 
-/// A document seals as the JSON index lays it out, under the column it is
-/// stored in: the [`Identifier`] is the context the data key is minted
-/// under and every selector derives under, and is written to `i`.
+/// The `json_tree` layout under the JSON index's options: what seals a
+/// value into a [`JsonTreeDocument`] under options other than the
+/// defaults, such as the paths that carry an ordering term
+/// ([`JsonOptions::ordered`]). A [`Layout`], so a Rust program runs it as
+/// `keyset.run(JsonTree::new(options).encryption(), &value, column)`, and
+/// its index is the engine's [`Json`] under the same options, which a
+/// query derives through. [`JsonTreeDocument`]'s own [`EncryptFrom`] is
+/// this layout under the defaults.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct JsonTree {
+    index: Json,
+}
+
+impl JsonTree {
+    /// The layout under `options`.
+    pub fn new(options: JsonOptions) -> Self {
+        Self {
+            index: Json::new(options),
+        }
+    }
+
+    /// The index the layout seals under, with its options.
+    pub fn index(&self) -> &Json {
+        &self.index
+    }
+}
+
+/// A document seals as the JSON index lays it out under the layout's
+/// options, under the column it is stored in: the [`Identifier`] is the
+/// context the data key is minted under and every selector derives
+/// under, and is written to `i`.
+impl Layout<Value> for JsonTree {
+    type Target = JsonTreeDocument;
+    type Index = Json;
+    fn encryption<'s, K: 'static>(
+        &self,
+    ) -> Encryption<'s, Value, JsonTreeDocument, K, NonEmpty<Identifier>> {
+        Layout::<Value>::encryption::<K>(&self.index)
+            .accepting::<NonEmpty<Identifier>>()
+            .map_with_context(|document, i: NonEmpty<Identifier>| (i, document))
+            .try_map(|(i, document)| JsonTreeDocument::envelope(i.into_inner(), &document))
+    }
+    fn index(&self) -> &Json {
+        &self.index
+    }
+}
+
+/// A document seals as the JSON index lays it out under the default
+/// options ([`JsonTree::default`]): no path ordered, the root sealed.
 impl EncryptFrom<Value> for JsonTreeDocument {
     type Context = NonEmpty<Identifier>;
     fn encryption<'s, K: 'static>() -> Encryption<'s, Value, Self, K, Self::Context> {
-        <JsonDocument as EncryptFrom<Value>>::encryption()
-            .accepting::<NonEmpty<Identifier>>()
-            .map_with_context(|document, i: NonEmpty<Identifier>| (i, document))
-            .try_map(|(i, document)| Self::envelope(i.into_inner(), &document))
+        JsonTree::default().encryption()
     }
     fn indexes() -> Vec<IndexSpec> {
-        <JsonDocument as EncryptFrom<Value>>::indexes()
+        JsonTree::default().specs()
     }
 }
 
@@ -353,7 +385,7 @@ impl DecryptInto<Value> for JsonTreeSubtree {
 }
 
 /// The rows of a walk open to the complete subtrees they hold, in order:
-/// one per node whose selector `json_tree_subtree_at` matched.
+/// one per node the gates of `json_tree_subtree_at` reached.
 impl DecryptInto<Vec<Value>> for JsonTreeSubtree {
     type Context = NonEmpty<Identifier>;
     fn decryption<K: 'static>(self, context: Self::Context) -> Decryption<Vec<Value>, K> {
@@ -421,7 +453,7 @@ mod tests {
     fn a_document_carries_its_projection() {
         let document = serde_json::json!({
             "v": 4, "i": {"t": "users", "c": "profile"}, "h": "AA==",
-            "sv": [{"s": "aa", "vs": "bb", "c": "AA=="}],
+            "sv": [{"s": "aa", "c": "AA=="}],
             "q": {"sv": [{"s": "aa", "vs": "bb"}], "el": []},
         });
         let stored: JsonTreeDocument = serde_json::from_value(document.clone()).unwrap();
@@ -451,7 +483,7 @@ mod tests {
     #[test]
     fn an_entry_without_the_grafted_fields_does_not_open() {
         let entry: JsonTreeEntry = serde_json::from_value(serde_json::json!({
-            "s": "00", "vs": "00", "c": "AA=="
+            "s": "00", "c": "AA=="
         }))
         .unwrap();
         assert!(entry.node().is_err());