Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/eql-json-array-root-flag.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@cipherstash/eql': patch
---

Encrypted JSON arrays now agree between the Rust/TypeScript bindings and the SQL. `SteVecDocument` models the root `a` array flag that the SQL root validator accepts and the array accessors require, so an array document the database accepts also parses in the bindings. The `a` flag on documents and entries is now optional but not nullable in the generated TypeScript and JSON Schema (`a?: boolean`), matching the SQL CHECK; a `null` is rejected instead of failing later with a cast error. `eql_v3.jsonb_array_length` returns 0 for an empty array (it returned NULL), and `eql_v3.jsonb_array_elements` and `eql_v3.jsonb_document_contains(jsonb[], jsonb)` no longer raise "upper bound of FOR loop cannot be null" on one.
13 changes: 11 additions & 2 deletions packages/eql/crates/eql-bindings/bindings/v3/SteVecDocument.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,20 @@ import type { SteVecForm } from "./SteVecForm";

/**
* `public.eql_v3_json_search` — a SteVec encrypted-JSON document
* (`{v, k, i, h, sv:[entry]}`, no root ciphertext — the root document
* (`{v, k, i, h, sv:[entry]}` plus an optional array-root flag `a`, no root ciphertext — the root document
* ciphertext lives on the root sv entry). Strict. `k` is the `"sv"` form
* discriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],
* stored once for the whole document (every entry encrypts under the
* document's single data key, with per-entry nonces derived from the
* entries' selectors).
*/
export type SteVecDocument = { v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>, };
export type SteVecDocument = { v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>,
/**
* Array-root marker. The SQL root validator allows it and the array
* accessors (`jsonb_array_length`, `jsonb_array_elements`, `->(doc, int)`)
* REQUIRE it (`eql_v3_internal.is_ste_vec_array`), so a document the
* database accepts must also parse here. Optional and non-nullable: SQL
* rejects `"a": null` (`jsonb_typeof(...) = 'boolean'`), so the schema
* and deserializer reject it too.
*/
a?: boolean, };
4 changes: 2 additions & 2 deletions packages/eql/crates/eql-bindings/bindings/v3/SteVecEntry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ import type { Selector } from "./Selector";
* selector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the
* decryption unit is `h` + `s` + `c`, with the document `h` grafted onto
* extracted entries by `->`), an optional array-membership marker `a`
* (emitted only when true), and — for ordered (number/string) path entries
* (a boolean; producers emit it only when true), and — for ordered (number/string) path entries
* only — the `op` ordering term. Value entries (value-inclusive selectors)
* and non-orderable path entries are term-less: exact matching is selector
* presence, so no per-entry equality term exists (`hm` is retired). The
* optional `i`/`v`/`h` fields model metadata grafted onto an entry returned by
* `->`; naming them explicitly keeps unknown keys rejectable.
*/
export type SteVecEntry = { s: Selector, c: EntryCiphertext, a?: boolean | null, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, };
export type SteVecEntry = { s: Selector, c: EntryCiphertext, a?: boolean, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, };
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,10 @@
"$id": "https://schemas.cipherstash.com/eql/v3/eql_v3_json_entry.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(emitted only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(a boolean; producers emit it only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
"properties": {
"a": {
"type": [
"boolean",
"null"
]
"type": "boolean"
},
"c": {
"$ref": "#/$defs/EntryCiphertext"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,10 @@
},
"SteVecEntry": {
"additionalProperties": false,
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(emitted only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(a boolean; producers emit it only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
"properties": {
"a": {
"type": [
"boolean",
"null"
]
"type": "boolean"
},
"c": {
"$ref": "#/$defs/EntryCiphertext"
Expand Down Expand Up @@ -112,8 +109,12 @@
"$id": "https://schemas.cipherstash.com/eql/v3/eql_v3_json_search.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"description": "`public.eql_v3_json_search` — a SteVec encrypted-JSON document\n(`{v, k, i, h, sv:[entry]}`, no root ciphertext — the root document\nciphertext lives on the root sv entry). Strict. `k` is the `\"sv\"` form\ndiscriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],\nstored once for the whole document (every entry encrypts under the\ndocument's single data key, with per-entry nonces derived from the\nentries' selectors).",
"description": "`public.eql_v3_json_search` — a SteVec encrypted-JSON document\n(`{v, k, i, h, sv:[entry]}` plus an optional array-root flag `a`, no root ciphertext — the root document\nciphertext lives on the root sv entry). Strict. `k` is the `\"sv\"` form\ndiscriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],\nstored once for the whole document (every entry encrypts under the\ndocument's single data key, with per-entry nonces derived from the\nentries' selectors).",
"properties": {
"a": {
"description": "Array-root marker. The SQL root validator allows it and the array\naccessors (`jsonb_array_length`, `jsonb_array_elements`, `->(doc, int)`)\nREQUIRE it (`eql_v3_internal.is_ste_vec_array`), so a document the\ndatabase accepts must also parse here. Optional and non-nullable: SQL\nrejects `\"a\": null` (`jsonb_typeof(...) = 'boolean'`), so the schema\nand deserializer reject it too.",
"type": "boolean"
},
"h": {
"$ref": "#/$defs/KeyHeader"
},
Expand Down
47 changes: 39 additions & 8 deletions packages/eql/crates/eql-bindings/src/v3/json.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ impl JsonSchema for SteVecForm {
}

/// `public.eql_v3_json_search` — a SteVec encrypted-JSON document
/// (`{v, k, i, h, sv:[entry]}`, no root ciphertext — the root document
/// (`{v, k, i, h, sv:[entry]}` plus an optional array-root flag `a`, no root ciphertext — the root document
/// ciphertext lives on the root sv entry). Strict. `k` is the `"sv"` form
/// discriminator (see [`SteVecForm`]); `h` is the document [`KeyHeader`],
/// stored once for the whole document (every entry encrypts under the
Expand All @@ -100,13 +100,37 @@ pub struct SteVecDocument {
pub i: Identifier,
pub h: KeyHeader,
pub sv: Vec<SteVecEntry>,
/// Array-root marker. The SQL root validator allows it and the array
/// accessors (`jsonb_array_length`, `jsonb_array_elements`, `->(doc, int)`)
/// REQUIRE it (`eql_v3_internal.is_ste_vec_array`), so a document the
/// database accepts must also parse here. Optional and non-nullable: SQL
/// rejects `"a": null` (`jsonb_typeof(...) = 'boolean'`), so the schema
/// and deserializer reject it too.
#[serde(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

major The root validator (eql_v3_is_valid_ste_vec_document_payload, src/v3/json/types.sql:106) only allow-lists a; it has no jsonb_typeof(val->'a') = 'boolean' check, so SQL accepts "a": null or "a": "true" at root while this deserializer now rejects it. Add the root CHECK in SQL, or stop claiming and enforcing it at root here.

Source: code-review · claude-sonnet-5-5


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 9f756f7. The root validator now requires a to be a JSON boolean when present, matching the entry validator and the bindings. is_ste_vec_array also checks the type before casting, so a raw jsonb value with "a": "maybe" gets "not an array" instead of a cast error. New SQLx test v3_jsonb_root_array_flag_must_be_boolean covers both, and I ran it against Postgres 16.

The SteVecEntry "emitted only when true" doc is corrected too. The tracked install bundles regenerate at release, so I left them alone.


Generated by Claude Code

default,
skip_serializing_if = "Option::is_none",
deserialize_with = "deserialize_present_bool"
)]
#[schemars(with = "bool")]
#[ts(optional, type = "boolean")]
pub a: Option<bool>,
}

/// Deserialize an optional boolean that, when present, must be a real boolean.
/// `Option<bool>`'s default deserializer maps JSON `null` to `None`; the SQL
/// CHECKs reject `null` for `a`, so the wire types must too.
fn deserialize_present_bool<'de, D>(deserializer: D) -> Result<Option<bool>, D::Error>
where
D: serde::Deserializer<'de>,
{
bool::deserialize(deserializer).map(Some)
}

/// `public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a
/// selector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the
/// decryption unit is `h` + `s` + `c`, with the document `h` grafted onto
/// extracted entries by `->`), an optional array-membership marker `a`
/// (emitted only when true), and — for ordered (number/string) path entries
/// (a boolean; producers emit it only when true), and — for ordered (number/string) path entries
/// only — the `op` ordering term. Value entries (value-inclusive selectors)
/// and non-orderable path entries are term-less: exact matching is selector
/// presence, so no per-entry equality term exists (`hm` is retired). The
Expand All @@ -123,13 +147,20 @@ pub struct SteVecDocument {
pub struct SteVecEntry {
pub s: Selector,
pub c: EntryCiphertext,
// `#[ts(optional = nullable)]` emits `a?: boolean | null` — ts-rs does NOT
// `#[ts(optional, type = "boolean")]` emits `a?: boolean` — ts-rs does NOT
// infer optionality from serde `default`/`skip_serializing_if` (10.1: only an
// explicit `#[ts(optional)]`/`optional = nullable` does), so without this the
// TS binding would render `a` REQUIRED and drift from the JSON Schema, which
// excludes `a` from `required`. `= nullable` keeps the `| null`.
#[serde(default, skip_serializing_if = "Option::is_none")]
#[ts(optional = nullable)]
// explicit `#[ts(optional)]` does), so without this the TS binding would
// render `a` REQUIRED and drift from the JSON Schema, which excludes `a`
// from `required`. Non-nullable: the SQL entry CHECK requires
// `jsonb_typeof(VALUE->'a') = 'boolean'`, so `"a": null` is rejected there
// and must not be advertised (or accepted) here.
#[serde(
default,
skip_serializing_if = "Option::is_none",
deserialize_with = "deserialize_present_bool"
)]
#[schemars(with = "bool")]
#[ts(optional, type = "boolean")]
pub a: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
#[ts(optional)]
Expand Down
70 changes: 63 additions & 7 deletions packages/eql/crates/eql-bindings/tests/v3_conformance.rs
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,41 @@ fn stevec_document_round_trips_and_enforces_envelope() {
assert!(serde_json::from_value::<SteVecDocument>(extra).is_err());
}

#[test]
fn stevec_array_root_flag_matches_sql_contract() {
use eql_bindings::v3::json::{SteVecDocument, SteVecEntry};
// The SQL root validator allows root `a`, and the array accessors require
// it, so an array document the database accepts must parse and round-trip
// here — including an empty one.
let wire = json!({
"v": 3, "k": "sv", "i": { "t": "users", "c": "tags" },
"h": "kh", "a": true, "sv": []
});
let parsed: SteVecDocument = serde_json::from_value(wire.clone()).unwrap();
assert_eq!(parsed.a, Some(true));
assert_eq!(serde_json::to_value(&parsed).unwrap(), wire);
// Absent `a` stays absent on re-serialization.
let mut no_a = wire.clone();
no_a.as_object_mut().unwrap().remove("a");
let parsed: SteVecDocument = serde_json::from_value(no_a.clone()).unwrap();
assert_eq!(parsed.a, None);
assert_eq!(serde_json::to_value(&parsed).unwrap(), no_a);
// SQL rejects a non-boolean `a` (null included) at root and entry level.
for bad in [json!(null), json!("true"), json!(1)] {
let mut w = wire.clone();
w["a"] = bad.clone();
assert!(
serde_json::from_value::<SteVecDocument>(w).is_err(),
"root a = {bad} must fail"
);
assert!(
serde_json::from_value::<SteVecEntry>(json!({ "s": "sel", "c": "ct", "a": bad }))
.is_err(),
"entry a = {bad} must fail"
);
}
}

#[test]
fn stevec_entry_term_is_optional_and_op_only() {
use eql_bindings::v3::json::SteVecEntry;
Expand Down Expand Up @@ -524,6 +559,7 @@ fn stevec_document_and_query_schemas_are_strict() {
},
h: "kh".to_string().into(),
sv: vec![],
a: None,
};
let sdoc = serde_json::to_value(doc.schema()).unwrap();
assert_eq!(sdoc.pointer("/additionalProperties"), Some(&json!(false)));
Expand All @@ -541,6 +577,26 @@ fn stevec_document_and_query_schemas_are_strict() {
Some(&json!("#/$defs/SteVecForm"))
);
assert_eq!(sdoc.pointer("/$defs/SteVecForm/const"), Some(&json!("sv")));
// Root `a` is optional (not required) and non-nullable, matching the SQL
// CHECKs (`jsonb_typeof(val->'a') = 'boolean'`); same for the entry's `a`.
assert_eq!(
sdoc.pointer("/properties/a/type"),
Some(&json!("boolean")),
"root a must be a plain boolean, not nullable"
);
assert!(
!sdoc["required"].as_array().unwrap().contains(&json!("a")),
"root a must not be required"
);
let entry: eql_bindings::v3::json::SteVecEntry =
serde_json::from_value(json!({ "s": "sel", "c": "ct" })).unwrap();
let sentry = serde_json::to_value(entry.schema()).unwrap();
assert_eq!(
sentry.pointer("/properties/a/type"),
Some(&json!("boolean")),
"entry a must be a plain boolean, not nullable"
);
assert!(!sentry["required"].as_array().unwrap().contains(&json!("a")));
let q = SteVecQuery { sv: vec![] };
let sq = serde_json::to_value(q.schema()).unwrap();
assert_eq!(sq.pointer("/additionalProperties"), Some(&json!(false)));
Expand Down Expand Up @@ -592,15 +648,15 @@ fn stevec_ts_exports_have_expected_shape() {
};

// SteVecEntry: direct fields s/c, optional op and extracted metadata, and
// the OPTIONAL nullable array marker `a`. `a?: boolean | null` (not `a: boolean |
// null`) pins ts-rs optionality so the TS binding agrees with the JSON Schema,
// the OPTIONAL, non-nullable array marker `a`. `a?: boolean` (not `a: boolean`)
// pins ts-rs optionality so the TS binding agrees with the JSON Schema,
// which excludes `a` from `required` — the drift a bare `Option<bool>` without
// `#[ts(optional = nullable)]` silently reintroduces.
// `#[ts(optional)]` silently reintroduces.
let entry = export_body("SteVecEntry.ts", "SteVecEntry");
for needle in [
"s: Selector",
"c: EntryCiphertext",
"a?: boolean | null",
"a?: boolean",
"op?: OpeCllw",
"i?: Identifier",
"v?: SchemaVersion",
Expand All @@ -617,14 +673,14 @@ fn stevec_ts_exports_have_expected_shape() {
// in `json.rs` (which changes the wire/consumer contract) fails here rather
// than escaping to a manual diff.
assert!(
entry.contains("{ s: Selector, c: EntryCiphertext, a?: boolean | null, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, }"),
entry.contains("{ s: Selector, c: EntryCiphertext, a?: boolean, op?: OpeCllw, i?: Identifier, v?: SchemaVersion, h?: KeyHeader, }"),
"SteVecEntry.ts field order must be s, c, a, op, i, v, h; got: {entry}"
);
let document = export_body("SteVecDocument.ts", "SteVecDocument");
assert!(
document.contains(
"{ v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>, }"
"{ v: SchemaVersion, k: SteVecForm, i: Identifier, h: KeyHeader, sv: Array<SteVecEntry>, a?: boolean, }"
),
"SteVecDocument.ts field order must be v, k, i, h, sv, got: {document}"
"SteVecDocument.ts field order must be v, k, i, h, sv, a, got: {document}"
);
}
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,10 @@
"$id": "https://schemas.cipherstash.com/eql/v3/eql_v3_json_entry.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"additionalProperties": false,
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(emitted only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
"description": "`public.eql_v3_json_entry` — one sv element (returned by `->`). Carries a\nselector `s`, raw AEAD ciphertext `c` (see [`EntryCiphertext`] — the\ndecryption unit is `h` + `s` + `c`, with the document `h` grafted onto\nextracted entries by `->`), an optional array-membership marker `a`\n(a boolean; producers emit it only when true), and — for ordered (number/string) path entries\nonly — the `op` ordering term. Value entries (value-inclusive selectors)\nand non-orderable path entries are term-less: exact matching is selector\npresence, so no per-entry equality term exists (`hm` is retired). The\noptional `i`/`v`/`h` fields model metadata grafted onto an entry returned by\n`->`; naming them explicitly keeps unknown keys rejectable.",
"properties": {
"a": {
"type": [
"boolean",
"null"
]
"type": "boolean"
},
"c": {
"$ref": "#/$defs/EntryCiphertext"
Expand Down
Loading
Loading