diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md new file mode 100644 index 000000000..efa040690 --- /dev/null +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -0,0 +1,495 @@ +--- +status: accepted +date: 2026-10-08 +revised: 2026-10-12 +relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 +--- + +# One value model, three encodings in vitaminc, and EQL v4 separated by producer + +> **Revised 2026-10-12.** This ADR now carries the decisions from +> [`docs/plans/2026-10-11-order-and-equality-terms-layering.md`](../plans/2026-10-11-order-and-equality-terms-layering.md): +> vitaminc sits at the bottom of the stack, equality and order terms take +> per-domain transforms over a shared semantic baseline, `vitaminc-prf` no +> longer depends on order encodings, and `orderable-bytes` is frozen. The plan +> keeps the steps; this document is the decision. One question is still open, +> under "Open decision". + +Every value Stack Encrypt handles is encoded three times: as a **ciphertext** +(reversible and self-describing), as an **equality term** (a keyed hash that +must be unambiguous) and as an **order term** (bytes whose order is the +value's order). This ADR decides where each encoding lives, which kinds of +value exist, how each kind is normalised before a term is derived from it, +and how EQL keeps columns written by Stack Encrypt apart from columns written +by cipherstash-client. + +## The problem + +In October 2026 Stack Encrypt could produce one of EQL's 51 types, `TextEq`. +The number, date, timestamp and decimal families were blocked on encoding, +not code. (`Bool` was not: it already had a kind, and EQL's only boolean +type, `eql_v3_boolean`, stores a ciphertext and nothing else.) + +- **The three encodings disagreed about which types exist.** `i16` had an + equality domain in `vitaminc-prf` and an order encoding, but no ciphertext + tag, so it could not be a kind. Date, timestamp and decimal had order + encodings only. +- **The mapping from a kind to a Rust type lived in Stack Encrypt.** + `dynamic::term::Scalar` mirrored `FfiValue`'s variants one for one to + dispatch to the term crates, and `dynamic::Value` wrapped `FfiValue` only to + add `Clone`. Both restated vitaminc's vocabulary in a downstream crate. +- **Ordering was implemented twice.** `orderable-bytes` defines a canonical, + order-preserving encoding per type. cllw-ore uses it for chrono and decimal + but hand-rolls integers and floats, and differs on `-0.0`. Block ORE was not + derivable at all, so `TextOrdOre` and `TextSearchOre` were refused. +- **Order encodings were spread over three repositories.** Encodings lived in + ore.rs, schemes in ore.rs and cipherstash-suite, and term derivation in + vitaminc and Stack Encrypt. Every change needed releases in all of them, + and the scheme crates accepted raw bytes, so a hand-encoded integer + (little-endian, sign bit not flipped) encrypted without error and sorted + wrong. +- **The existing writer's encodings are inconsistent.** cipherstash-client + hashes a timestamp's milliseconds but orders by its nanoseconds, and hashes + a decimal with its scale while ordering ignores it. Its block ORE text is + ASCII-only, lowercased, maps every digit to one symbol and truncates to six + blocks. cllw-ore's text order form (`orderize_string`) decomposes to NFD and + then drops every character that is not alphanumeric, whitespace or ASCII + punctuation, which removes symbols, emoji and non-ASCII punctuation as well + as accents, and does not fold case. +- **Two producers wrote the same EQL domains.** A Stack Encrypt query term + never matches a cipherstash-client term for the same value, and the + `eql_v3_*` domains could not tell them apart. A query from one against a + column written by the other returned no rows, silently. + +## Options considered + +**Where the value model lives.** + +1. **Move `vitaminc-aead-value` into Stack Encrypt.** It looks like FFI + plumbing. But it is the value model for vitaminc's own `Cipher` traits: + `aead-napi` and vitaminc's Go binding encrypt with `Aes256Cipher` through + it, without Stack Encrypt. Moving it makes vitaminc Rust-only, makes + vitaminc depend on Stack Encrypt (which depends on six vitaminc crates), + or forks the frozen tag table into two copies. +2. **Keep it in vitaminc, and move term derivation there too.** All three + encodings then live in one repository, next to the value type they + encode. Chosen. How "a kind exists in every layer or in none" is enforced + is the open decision below; until it is settled, the conformance test + described under "Which kinds get which terms" is what enforces it. + +**Where order schemes sit.** + +1. **`vitaminc-ore` adapts the scheme crates.** It depends on cllw-ore and + ore-rs and wraps each in its own trait, with `orderable-bytes` underneath. + This is the spread over three repositories described above. +2. **vitaminc at the bottom.** `vitaminc-ore` owns the order encodings and an + `OrderScheme` trait and depends on no scheme crate; cllw-ore and ore-rs + implement the trait. A scheme's input is a sealed type only `vitaminc-ore` + can build, so raw bytes cannot reach a scheme. Chosen. + +**How EQL separates the two producers.** The SQL for both is identical; only +the producer of the terms differs. + +1. **Name only.** Stack Encrypt payloads go in the `eql_v3_*` domains, + distinguished by the `stack-encrypt:1:` ciphertext prefix. Nothing in the + database stops a cross-producer comparison. +2. **A producer tag in every payload and term, checked by every operator.** + A runtime check on the hottest SQL paths, across about 24k lines of + hand-written SQL. +3. **One SQL source, emitted under two names.** The build writes the same + source out as `eql_v3` (cipherstash-client terms) and `eql_v4` (Stack + Encrypt terms). A column's domain names its producer: a payload written + to the other producer's domain fails its check on insert, and a typed + client sees two different types. Names alone do not stop a cross-producer + `=` in SQL; see Consequences. Chosen. + +**How v3 and v4 ship.** Bumping `main` to v4 and patching v3 from a branch +needs a maintenance release path for five lockstep artefacts that does not +exist, and the EQL publish script puts every stable release on `latest`, so a +3.x patch would move `latest` backwards. A second package doubles the +trusted-publishing and release surface. One package carrying both bundles +avoids both, at the cost of teaching the release tooling about a second +bundle (see Consequences). Chosen. + +## Decision + +### Kinds and the ciphertext + +- vitaminc 0.6.0 adds the kinds `Int8`, `UInt8`, `Int16`, `UInt16`, `Int128`, + `UInt128`, `Date`, `Timestamp` and `Decimal`, each with a `ValueKind` name, + a `Value` variant and a ciphertext tag, in one breaking release. Rust `i128` + and `u128` get `Encrypt` and `Decrypt` impls. +- `FfiValue` is renamed `Value`, with `#[deprecated] pub type FfiValue = Value` + for one release. Whether `Value` and `ValueKind` are `#[non_exhaustive]` is + the open decision. +- Every variant exists in every build. `Date`, `Timestamp` and `Decimal` use + vitaminc's own representation (the payloads in the table below), with + conversions to and from chrono and rust_decimal behind those features. A + variant that appeared only with a feature would break any exhaustive + handling outside `vitaminc-aead-value`, because features unify across a + build and a crate can check only its own. +- `Value` implements `Clone` as a deep copy that rebuilds every leaf into a + fresh `Protected`. A clone is under the same custody as its original. +- The sealed leaf tag table (`tags.rs`) stays frozen and contiguous. The + transport codec's framing tags move from `0x10`–`0x12` to `0xF0`–`0xF2`; + transport carries no compatibility commitment and every user of it updates + together. A leaf is `[tag] ++ payload`. The new leaves, which every binding + must produce byte for byte: + + | Tag | Kind | Payload | + |---|---|---| + | `0x0C`, `0x0D` | `Int8`, `UInt8` | 1 byte; two's complement for `Int8` | + | `0x0E`, `0x0F` | `Int16`, `UInt16` | 2 bytes, little-endian; two's complement for `Int16` | + | `0x10`, `0x11` | `Int128`, `UInt128` | 16 bytes, little-endian; two's complement for `Int128` | + | `0x12` | `Date` | `i32`, little-endian: the day number in the proleptic Gregorian calendar with **0001-01-01 as day 1** (chrono's `num_days_from_ce`). 0001-01-01 is `01 00 00 00`, not zero | + | `0x13` | `Timestamp` | `i64` Unix seconds, then `u32` nanoseconds, both little-endian, UTC. Nanoseconds are below 1,000,000,000, except at a leap second (seconds modulo 60 is 59), where they may go up to 1,999,999,999 (chrono's representation). Anything else is refused | + | `0x14` | `Decimal` | 16 bytes: a little-endian `u32` of flags (sign in bit 31, scale in bits 16–23 and at most 28, every other bit zero), then the low, middle and high little-endian `u32` words of the 96-bit mantissa. This is rust_decimal's `serialize()` and keeps the scale. There is no NaN or infinity | + +- `transport` stays a module of `vitaminc-aead-value`. + +### Semantic baseline and per-domain transforms + +The ciphertext keeps the value exactly as given: `1.50` decrypts as `1.50`, +and a timestamp keeps its nanoseconds. Terms are derived in two steps. + +**The semantic baseline** makes values that are equal under `Value`'s own +semantics encode the same. Every term applies it; it loses nothing a user +would call a different value. + +| Kind | Baseline | +|---|---| +| `Decimal` | scale normalised: `1`, `1.0` and `1.00` are equal | +| `Float32`, `Float64` | `-0.0` folded into `+0.0`; every NaN replaced by one positive quiet NaN, which sorts above +Infinity. This is Postgres's own `float8` semantics: `'NaN' = 'NaN'`, `'-0' = '0'` and `'NaN' > 'Infinity'` are all true | +| text | NFC | +| every other kind | none | + +**Lossy transforms** sit on top of the baseline, chosen by each term's +domain: timestamp resolution, text folding and prefix truncation. Each +transform and its version is named in the domain label, for example +`…/timestamp/day/v1`. The ones defined now: + +- **Timestamp resolution**: exact, microsecond or day. Microsecond rounds to + the nearest microsecond, ties to even, on the signed microsecond count + since the epoch. That is what Postgres does when it parses a timestamp with + more digits than it keeps (`.1234565` becomes `.123456` and `.1234575` + becomes `.123458`, before and after 1970; checked on Postgres 17.5). A + value that already has microsecond precision is unchanged. Day takes the + UTC calendar day. +- **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt` + statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of + every code point with General_Category `Mn`, then NFC. The order of the + steps is part of the transform, because folding can produce text that is + no longer normalised and the steps do not commute. +- **Prefix truncation**: a number of code points kept after any other + transform. Alphabet packing, if ever built, is a transform of the same + kind. + +Text transforms are pinned to a Unicode version, which is part of the label +(for example `text-fold/unicode-16/v1`), and the `unicode-normalization` and +case-folding tables are pinned to it. Code points unassigned in that version +pass through unchanged (canonical combining class 0, no decomposition, no +case mapping), so the transform is a fixed function and nothing is refused on +write. The cost is that two strings using characters added after the pin, +which a later Unicode would treat as equivalent, may not match. Moving the +pin is a new label: existing columns keep the label they were written under +and stay queryable, and a column moves only by re-encryption. + +Order terms fold case and accents because code-point order puts `é` after +`z`; a fixed fold approximates the first level of Unicode collation without +depending on ICU, whose sort keys change between versions. Equality on text +is not folded by default: a folded order term only produces ties, while a +folded equality term produces false matches. A case-insensitive equality is +its own domain. + +**Equality and order may take different transforms**, for example an exact +timestamp for equality and the day for order, or the same one ("locked"). +Stack's index spec carries the choice per column. Two rules bound it: + +- **A non-injective order transform needs an equality term beside it.** In + the EQL SQL, the numeric, date and timestamp `_ord`, `_ord_ope` and + `_ord_ore` domains store no `hm` and answer `=` with the order term + (`timestamp_ord_functions.sql`: `eq` is `ord_term(a) = ord_term(b)`). A + day-resolution order term there would make `=` match every row from that + day. So a lossy order transform is allowed only on a domain that also + stores an equality term and answers `=` through it, the shape `text_ord` + already has (`hm` and `op`). On the other domains the order transform is + injective: exact for numbers and dates, and for timestamps whatever + resolution the equality term would have used. +- **Labels separate domains; they do not make terms incomparable.** Terms + built with different transforms are unrelated bytes, so a probe built with + one matches nothing built with the other. Nothing in a stored term records + its label. Keeping a query on the same transforms as the column is the + index spec's job, and the EQL domain is what ties a column to its spec. + +A column's leakage is the combined leakage of all its terms. Truncating the +order term to the day hides the order of events within a day, but an exact +equality term on the same column still reveals which rows share a timestamp. + +### Which kinds get which terms + +- **Every scalar kind has a ciphertext.** Containers and null have no terms. +- **Equality:** every scalar kind except `Bool`, including floats over their + baseline bits. Stack Encrypt refused float equality until now, on the + grounds that equality on IEEE-754 values is a modelling error. The baseline + removes the two cases where bitwise equality and numeric equality disagree + (signed zero and NaN payloads), and what is left is exactly Postgres's + `float8 =`, which EQL's `real_eq` and `double_eq` domains already expose. + Whether a computed float is the value the caller meant is the caller's + problem, as it is for a plaintext float column. +- **Order:** every scalar kind except `Bool`, under all three schemes, with + one exception: block ORE for text, which waits for ore-rs's variable-length + block ORE (see Deferred). +- **`Bool` has a ciphertext only.** A keyed hash or an order term over a + domain of two values hides nothing: it splits the rows into two groups, and + an order term also says which group is `true`. Stack Encrypt stops + producing CLLW order terms for `Bool`, and `vitaminc-ore` has no `Bool` + encoding. cllw-ore keeps its `Bool` impl, so cipherstash-client and its + stored v3 terms are unaffected. + +These exceptions are an explicit, documented list kept next to vitaminc's +conformance test. The test fails for any kind that is missing from +`vitaminc-prf` or `vitaminc-ore` without an entry, and checks that values +equal under `Value` semantics produce equal terms in both. + +### Where each encoding lives + +- **Ciphertext:** `vitaminc-aead-value`. It does no term normalisation: its + `canonical` module and its dependency on `orderable-bytes` are removed. +- **Equality:** `vitaminc-prf`, which depends only on `vitaminc-aead-value`, + gains domains for every new kind, floats and `Decimal`, and applies the + baseline and its own transforms. It has no ordering dependency. The + value-derived PRF domains landed after vitaminc 0.5.1 and are unreleased, + so their bytes are still free to choose. +- **Order:** a new `vitaminc-ore` crate, released in lockstep with the other + vitaminc crates. + - It owns the order encodings (ported from `orderable-bytes`, kept in + `Protected`, pinned by golden vectors), the baseline, the order + transforms, and derivation of order terms from `&Value`. + - Its `OrderScheme` trait takes a sealed `OrderPlaintext`, fixed-length or + variable-length, that only `vitaminc-ore` can build, and returns a + scheme-defined term, roughly `type Term: Ord + AsRef<[u8]>`. Each scheme + keeps its own comparison and stored bytes. The trait also covers the key + type, domain separation (the label goes in as the salt) and typed errors. + - It depends on no scheme crate; its tests use a mock scheme. +- **Schemes:** cllw-ore (CLLW ORE and OPE) and ore-rs (block ORE) implement + `OrderScheme` behind an optional `vitaminc` feature, so cipherstash-client + takes no vitaminc dependency. Their typed APIs are unchanged and not + feature-gated. Block ORE through the trait uses each kind's natural width; + the 8-byte padding stays a detail of the typed API cipherstash-client uses. +- **`orderable-bytes` is frozen.** It stays as it is for cipherstash-client's + stored terms and is not part of the target design. +- **Term derivation** takes a `&Value` in `vitaminc-prf` and `vitaminc-ore` + and keeps leaves inside `Protected`. A pairing a layer does not support + returns a typed error from vitaminc. + +### Stack Encrypt + +- `dynamic::Value` and `dynamic::term::Scalar` are deleted. `dynamic::term` + takes a `&vitaminc_aead_value::Value`. `admits` keeps only the rules that + belong to Stack Encrypt, such as `Match` taking text only. +- Order terms go through `vitaminc-ore`. The index for an ORE or OPE column + picks the concrete scheme (cllw-ore or ore-rs, through `OrderScheme`) and + its transforms; Stack Encrypt no longer calls cllw-ore's typed API. A block + ORE term kind joins CLLW ORE and OPE. +- `TextEq` normalises to NFC and moves to EQL v4. The Stack Encrypt targets on + `eql_v3_*` domains are deleted. The `stack-encrypt:1:` ciphertext prefix + stays: decryption does not pass through Postgres types, and the prefix is + what lets it reject the other producer's payload. + +### Host languages + +A binding picks the kind from the field's **declared EQL target**, never +from the host type. This keeps the rule `eql-bindings` already enforces: a +target accepts exactly one kind, and nothing below the binding converts. The +binding converts a host value into the target's kind when it can do so +without loss, and refuses it otherwise. + +| Kind | EQL family | | +|---|---|---| +| `Int16` | `smallint` | | +| `Int32` | `integer` | | +| `Int64` | `bigint` | | +| `Float32` | `real` | | +| `Float64` | `double` | | +| `Decimal` | `numeric` | | +| `Date` | `date` | | +| `Timestamp` | `timestamp` | | +| `String` | `text` | | +| `Bool` | `boolean` | | +| `Int8`, `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Int128`, `UInt128`, `Bytes` | none | no EQL domain and no target name yet; a kind is selectable only once a target names it | + +- **Go.** Any Go integer type whose every value fits the target's kind is + accepted (`int8` into `smallint`, `int32` into `bigint`); a wider one is + range-checked and refused if the value does not fit. `int8`, `int16`, + `uint8` and `uint16` get their own kinds when a target names them; no + target does yet, so today they reach `Int16`, `Int32` or `Int64` through + the fits-losslessly rule. + `Int128` and `Uint128` are SDK value types. `time.Time` means `Timestamp`, + and `encrypt.Date{Year, Month, Day}` is a date. A field whose target is a + date family also accepts `time.Time` and takes its calendar day **in the + value's own location** (`t.Date()`), which is how Go code builds a date; + taking the UTC day would store the previous day for a local midnight east + of UTC. +- **JavaScript.** A `BigInt` takes the target's kind, and is refused if it + does not fit. It decodes as a `BigInt`. `Date` means `Timestamp`; a date + has its own wrapper. +- **Decimal.** A host decimal (Go's `shopspring/decimal`, a JavaScript + string) is refused if it is NaN or infinite, has more than 28 digits after + the point, or does not fit a 96-bit mantissa at its scale. Postgres + `numeric` holds values that `Decimal` cannot, and a binding refuses them + rather than rounding. +- **Every binding** runs vitaminc's shared test vectors: a host value, the + target, the kind it must map to, and the exact `[tag] ++ payload` leaf + bytes, including normalisation and refusal cases. The leaf is + deterministic even though the AEAD is not. + +### EQL v4 + +- **v4 is the v3 SQL with Stack Encrypt terms.** One SQL source is emitted + under two names. The hand-written SQL takes the schema as a build-time + placeholder, as eql-codegen's templates already do. Consistent with + ADR-0001, the data-bearing domains (`eql_v3_*`, `eql_v4_*`) live in + `public` and survive reinstall, and the implementation schemas + (`eql_v3`, `eql_v4` and their `_internal` schemas) stay disposable. The + second name separates producers; it is not a versioned upgrade mechanism of + the kind ADR-0001 rejects. +- **A v4 payload's envelope carries `"v": 4`**, and the `eql_v4_*` domains' + check constraints test it, as the `eql_v3_*` ones test `3`. The version is + one more build-time placeholder. A payload written to the other producer's + domain then fails on insert, rather than only matching nothing when it is + queried. Queries are a different matter; see Consequences. +- **New EQL targets are v4-only.** In v3 they stay refused, with a reason + that points to v4. A domain that a lossy order transform needs, with an + `hm` beside its `op`, is one of these, and is the first point where the v4 + SQL differs from the v3 SQL. +- **`@cipherstash/eql` 4.x ships both bundles** from `main`. A SQL fix lands in + both names in one release. +- **`--eql-version` comes back, on purpose.** It was removed when v3 became + the only installable generation, and the CLI now refuses it. It returns as + `--eql-version 3|4|all` on both `stash eql install` and + `stash eql migration`, so the migration-first paths (Drizzle, and Supabase + with a local `supabase/` directory, where a direct install does not survive + `supabase db reset`) can install v4 too. The migration they write carries + the chosen bundles, and `--supabase` grants on each bundle's schemas. The + default is 3, which is today's behaviour, and changes to 4, announced in + advance, when the TypeScript stack has moved to Stack Encrypt. `2` keeps + its current message pointing at the EQL 2.3.1 release, rather than + becoming an ordinary invalid value. +- **During the overlap**, expected to last a quarter or more, cipherstash-client + takes fixes only. New kinds and domains are produced through Stack Encrypt + and v4. An exception is a decision written down in its issue. + +### Review + +A domain that locks equality and order together derives both from the same +transformed bytes, and that choice is frozen once data is stored under it. +Dan Draper signs off each such domain in writing before it ships. Domains +whose equality and order transforms are independent need no such sign-off. + +## Open decision: exhaustive handling of `Value` + +`vitaminc-prf`, `vitaminc-ore` and every binding must handle every kind of +`Value`, and adding a kind should fail to build until each one does. Two +approaches remain: + +- **A: drop `#[non_exhaustive]`.** Every crate that matches `Value` gets + exhaustiveness, and adding a kind is a breaking change for all of them. +- **B: a `ValueVisitor` trait with one required method per kind, keeping + `#[non_exhaustive]`.** Crates that must handle every kind implement it, and + adding a kind is breaking only for them; other crates keep a `_ =>` arm. + +The leaning is B. The layering plan sets out the comparison. Either way, +the conformance test is the backstop: a `#[non_exhaustive]` enum lets a +downstream match compile with a wildcard, so the compiler alone does not +guarantee coverage under B. + +## Consequences + +- **The release order is fixed.** vitaminc 0.6.0 first, then cllw-ore and + ore-rs (additive releases implementing `OrderScheme`), then the Stack + Encrypt breaking release. Stack Encrypt must reach crates.io before + `eql-bindings` uses its new API, since `cargo publish` builds + `eql-bindings` against the registry. The EQL v4 bundle and the Go SDK + changes follow. +- **Adding a kind later** takes a tag, PRF and order encodings, a codec in + every binding, test vectors and a row in the kind-to-family table, or an + entry in the exceptions list. Whether it is a breaking change depends on + the open decision. +- **Stack Encrypt's terms change** for `-0.0` and negative-sign NaN floats, + for non-NFC text equality, for `Bool` order terms, which are removed, and + for every text order term: `orderize_string` drops symbols and keeps case, + while the fold keeps symbols and folds case. None is deployed, so the + breaking release carries them without migration. +- **cipherstash-client's terms do not change.** `orderable-bytes`, cllw-ore's + typed impls (`Bool` included) and its block ORE padding are all kept, so + every stored v3 payload stays queryable. +- **Separate names do not stop a cross-producer comparison in SQL.** EQL's + domains are over `jsonb`, and when no operator is defined for an exact pair + of types, Postgres treats a domain as its base type. `v3col = v4term` then + runs as the built-in `jsonb = jsonb` and returns no rows, silently, as + `eql_v3_text_eq = eql_v3.query_integer_eq` already does within v3. A join + between a v3 and a v4 column does the same, and + `eql_v3.eq_term(v3col) = eql_v4.eq_term(v4term)` compares the two terms as + text. Three things do fail loudly: an insert of the wrong version (the + domain check), the function form `eql_v3.eq(v3col, v4term)` ("not unique"), + and `OPERATOR(public.=)` between a v3 and a v4 column ("not unique"). + Verified on Postgres 17 with both bundles installed in one database. +- **Tooling that recognises encrypted columns by the `eql_v3_` prefix must + learn `eql_v4_` before any v4 column can exist.** Today that is + `stash init`'s introspection (`cli/src/commands/init/lib/introspect.ts`), + the wizard's tools (`wizard/src/tools/wizard-tools.ts`) and + `classifyEqlDomain` (`migrate/src/version.ts`), which `stash status` and + `stash encrypt status` use. Getting this wrong shows an encrypted column as + plaintext, which invites encrypting it a second time. The installer's + ADR-0001 capture, rebuild and verify logic hard-codes the disposable + schemas as `('eql_v3', 'eql_v3_internal')` + (`installer/derived-search-index-restoration.ts`, + `installer/installation-state.ts`); without `eql_v4` and + `eql_v4_internal` there, a v4 reinstall would capture and restore none of + its own indexes. Both land with the first released v4 bundle. +- **The release tooling assumes one bundle.** The lockstep stamp is read from + `COMMENT ON SCHEMA eql_v3` (`scripts/sync-lockstep-versions.mjs`, + `packages/eql/tasks/release/prepare-bindings-assets.sh`); there is one + `release-manifest.json` per artefact, which the CLI's `bundle-digest.ts` + and `eql-sql-asset-freshness.test.mjs` check; `eql_bindings::sql` embeds + only the v3 files; and `_build-eql-sql.yml` uploads only them. The v4 + bundle needs its own stamp, manifest entries, freshness check, crate + constants and release upload before it ships, or v4 drift goes unnoticed + with every digest still verifying. The `postgres-eql` image states which + bundles it installs. +- **Moving a column from v3 to v4 means re-encrypting it.** The ciphertext and + every term change. How that migration runs belongs to the decision that + moves the TypeScript stack onto Stack Encrypt. +- **The CLI surface changes** (`--eql-version` on `stash eql install` and + `stash eql migration`), so `skills/stash-cli`, `skills/stash-indexing` and + `skills/stash-postgres` change in the same PR. +- **This replaces a definition in a plan.** `docs/plans/2026-10-04-plan-builder.md` + called "EQL v4" a name for a Stack Encrypt payload in the v3 envelope and + the `eql_v3_*` domains. That plan is updated to this ADR. + +## Deferred + +- **Block ORE for text** waits for ore-rs's chained, variable-length block ORE + to be reviewed and released, then arrives in a minor release of + `vitaminc-ore`. Until then it is an entry in the exceptions list. +- **Block ORE for `Bool`.** Block ORE stored as right ciphertexts only is + fully randomised and semantically secure, so a two-value domain leaks + nothing through it. `Bool` could support that scheme alone, enforced by a + marker trait on schemes. Not built until needed. +- **Cross-version blocker operators.** Blockers for every v3/v4 pair of the + same type family and capability would turn the silent cross-producer `=` + into an error. About 2,300 operators, generated from the catalog like the + existing blockers. Not built yet; until then, a cross-producer query is + caught only by typed clients, the function form and `OPERATOR(public.=)`; + the plain `=` returns no rows. When they are built, + **the v4 bundle owns them**: it creates them in `eql_v4_internal`, against + whichever `public.eql_v3_*` domains exist, on every install. A v4 reinstall + drops them with its schema and creates them again. A v3 reinstall leaves + them alone, because they depend only on the `public` domains, which + survive, and not on anything in `eql_v3`. Installing v3 after v4 into a + database that had no v3 domains leaves them absent until v4 is reinstalled, + so `--eql-version all` installs v3 first. +- **Locale-aware collation**, as its own domain with the collation version in + its name. +- **ASCII-packed text domains**, until a customer's column sizes make the + case. diff --git a/docs/plans/2026-10-04-plan-builder.md b/docs/plans/2026-10-04-plan-builder.md index 88dd628fb..5037c67ad 100644 --- a/docs/plans/2026-10-04-plan-builder.md +++ b/docs/plans/2026-10-04-plan-builder.md @@ -662,9 +662,11 @@ In the `Text` family, the three `Ord` suffixes carry equality too. Each type has a query type, with `Query` after its name: `TextEqQuery`. The value of `encrypt_into` is the Go type name. -An EQL value has the EQL v3 envelope: its version field is `3`, and Postgres stores it in an `eql_v3` domain. +Once EQL v4 lands, an EQL value from the engine is an EQL v4 value: its version field is `4`, and Postgres stores it in an `eql_v4_*` domain. The ciphertext inside it is a Stack Encrypt ciphertext, which starts with `stack-encrypt:1:`. -"EQL v4" in this plan is the name of that form, and not a new envelope. +EQL v4 is the v3 SQL, emitted from the same source under a second name, so that a column written by Stack Encrypt and one written by cipherstash-client are different Postgres types ([ADR-0002](../adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md)). +This replaces this plan's first definition, under which "EQL v4" named a Stack Encrypt payload in the v3 envelope and an `eql_v3` domain; Postgres could not tell the two producers apart, and a query from one against a column written by the other matched nothing. +The `TextEq` described below is still produced into an `eql_v3` domain until that move lands. The engine produces one EQL type today: `TextEq`. Status (2026-10-06, #1062): `TextEq` is producible through the data plan's target form. @@ -1157,9 +1159,10 @@ model rather than a strain: it puts key material in the database. ## EQL v4 types as field targets -Naming: **EQL v4** is the EQL form of a stack-encrypt payload. **EQL v3** is -the existing SQL bundle and its `eql_v3_*` domains, which this section does -not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt +Naming: **EQL v4** is the EQL form of a stack-encrypt payload: the v3 SQL +emitted under a second name, with `eql_v4_*` domains and `"v": 4` envelopes +(ADR-0002). **EQL v3** is the same SQL holding cipherstash-client payloads in +its `eql_v3_*` domains, which this section does not change. Depends on #971 (`TextEq` / `TextEqQuery` through stack-encrypt in `eql-bindings`, and `Identifier` as a two-segment `Label`). **The engine returns an EQL type only when a plan names it as a target.** diff --git a/docs/plans/2026-10-11-order-and-equality-terms-layering.md b/docs/plans/2026-10-11-order-and-equality-terms-layering.md new file mode 100644 index 000000000..aacd27a2d --- /dev/null +++ b/docs/plans/2026-10-11-order-and-equality-terms-layering.md @@ -0,0 +1,343 @@ +# Order and equality terms: vitaminc at the bottom + +> **Plan, not specification.** This document is an indicative sketch of the +> steps required, written before the work was done. It is not kept in step +> with the implementation and must not be used as a formal specification or +> as a reference for reviewing what was actually built: the code, its rustdoc +> and the tests are the source of truth. Where the two disagree, the code wins +> and this document is simply out of date. + +**Status:** planned. Decisions settled 2026-10-11, except the one listed +under "Open decision". +**Date:** 2026-10-11 +**Amends:** ADR-0002 (`docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md`). +The decisions below were folded into the ADR on 2026-10-12, and the ADR is +now the reference for them. This plan keeps the steps. +**Issues:** cipherstash/vitaminc#374 (`vitaminc-ore`), #1063 (index crates) + +## Why + +Starting on cipherstash/vitaminc#374 under ADR-0002 turned up four problems. + +- **Too many dependencies across repositories.** `vitaminc-ore` was to depend + on cllw-ore and ore-rs and adapt each to its `Scheme` trait, with + `orderable-bytes` underneath both. Order encodings, scheme traits and term + derivation ended up spread over three repositories, and every change needed + releases in all of them. +- **Raw bytes reach the schemes.** cllw-ore and ore-rs accept bytes that the + caller has already encoded. Bytes encoded by hand (little-endian integers, + an unflipped sign bit, `f64::to_bits`) encrypt without error and sort + wrong. Fixing that inside cllw-ore needed wrapper types in `orderable-bytes` + (cipherstash/ore.rs#103) and a feature split in cllw-ore + (cipherstash/cipherstash-suite#2308). Cargo features unify across a build, + so the feature split could not hold whenever anything else in the build + turned the feature on. +- **PRF depends on ordering.** `vitaminc-prf` reaches `orderable-bytes` + through `vitaminc-aead-value`'s `canonical` module, which reuses order + encodings as equality input. Equality needs a canonical form, not an order. +- **One canonical form for both term kinds is too rigid.** ADR-0002 builds + equality and order terms from the same canonical form. A leakage-tuned + domain needs them to differ, for example an exact timestamp for equality + and only the day for order. + +## Decisions + +### vitaminc is the bottom of the stack + +- `vitaminc-ore` owns: + - the order encodings; + - an `OrderScheme` trait; + - the derivation of order terms from `&Value`, generic over the scheme. + + It depends on no scheme crate. +- A scheme's input is a sealed type, `OrderPlaintext`, that only + `vitaminc-ore` can construct. It comes in fixed-length and variable-length + forms. A scheme can't be given bytes that skipped the encoding, so no + wrapper types or feature flags are needed to keep raw input out. +- `OrderScheme` returns a scheme-defined term, roughly + `type Term: Ord + AsRef<[u8]>`, whose length is only known at runtime. Each + scheme keeps its own comparison (CLLW ORE and block ORE don't compare byte + by byte) and its own stored bytes. Fixed-width types such as + `OreCllw8V1` stay as inherent APIs on the scheme crates, because stable + Rust can't name their width generically. +- The trait also covers the key type, domain separation (the term's domain + label goes in as the salt) and typed errors. +- cllw-ore and ore-rs depend on `vitaminc-ore` and implement `OrderScheme` + behind an optional `vitaminc` feature, so cipherstash-client doesn't take a + vitaminc dependency. Their existing typed APIs stay as they are, so + cipherstash-client's stored terms don't change. +- `vitaminc-ore` releases in lockstep with the other vitaminc crates. On each + vitaminc minor release, cllw-ore and ore-rs bump their optional + `vitaminc-ore` dependency before stack can upgrade. That is acceptable + while stack is the only user of the trait path. +- `vitaminc-ore`'s tests use a mock scheme defined inside the crate. A + dev-dependency on cllw-ore would form a cycle across repositories and pull + a second `vitaminc-ore` in from crates.io. + +### Equality and order are tuned per domain + +- There is no shared canonical form. Each kind has a **semantic baseline**, + which makes values that are equal under `Value`'s own semantics encode the + same: decimal scale (`1.0` = `1.00`), signed zero, NaN, NFC. +- On top of the baseline, each term's domain chooses its own **lossy + transforms**, such as timestamp resolution (exact, microsecond, day), + text folding (removing combining marks, case folding, with the Unicode + version pinned) and prefix truncation. +- Each transform is named in its term's domain label, for example + `…/timestamp/day/v1`, so terms built with different transforms are + domain-separated: a probe built with one matches nothing built with the + other. A stored term does not record its label, so they are not made + incomparable. Keeping a query on the column's transforms is the index + spec's job. +- Equality and order can be **independent** (an exact timestamp for + equality, the day for order) or **locked** (the same transform value + passed to both). Stack's index spec (#1063) carries the choice per column. +- A **lossy order transform needs an equality term beside it.** EQL's + numeric, date and timestamp `_ord` domains store no `hm` and answer `=` + with the order term, so a day-resolution order term there would make `=` + match every row from that day. A non-injective order transform is allowed + only on a domain that stores an equality term and answers `=` through it, + as `text_ord` does. Those domains are new, v4-only EQL targets. +- The semantic baseline lives in each term crate: `vitaminc-prf` and + `vitaminc-ore` each apply it. A conformance test checks that values equal + under `Value` semantics produce equal terms in both. There is no shared + canonicalisation crate. +- A column's leakage is the combined leakage of all its terms. Truncating the + order term to the day hides the order of events within a day, but an exact + equality term on the same column still reveals which rows share a + timestamp. Domain design must account for the combination. + +### PRF has no ordering dependency + +- `vitaminc-prf` depends only on `vitaminc-aead-value` and applies its own + baseline and transforms. +- `vitaminc-aead-value` does no term normalisation. Its `canonical` module and + its dependency on `orderable-bytes` are removed. +- The value-derived PRF domains (cipherstash/vitaminc#380) haven't been + released: they landed on 2026-10-08, after vitaminc 0.5.1. So `vitaminc-prf` + can choose its own canonical bytes for dates, timestamps and decimals + without affecting any stored term. + +### `orderable-bytes` is frozen + +- `orderable-bytes` stays as it is for cipherstash-client's stored terms. It + is not part of the target design. +- There is no 0.3. cipherstash/ore.rs#103 (binary wrapper types) is closed. +- 0.2.0 is not yanked: ore-rs 0.8.4 depends on it and nothing new needs it + gone. + +## Open decision: exhaustive matching over `Value` + +Once `canonical.rs` leaves `vitaminc-aead-value`, `vitaminc-prf` and +`vitaminc-ore` must each handle every kind of `Value`, and adding a kind must +fail to compile until each one does. Two approaches are on the table. + +### Prerequisite for either approach: variants that exist in every build + +`Value::Date` and `Value::Timestamp` exist only with `vitaminc-aead-value`'s +`chrono` feature, and `Value::Decimal` only with its `rust_decimal` feature. +Because features unify across a build, an exhaustive match outside +`vitaminc-aead-value` breaks: + +1. `vitaminc-prf` matches `Value` exhaustively, with its `Date` arm behind its + own `chrono` feature. +2. In a build where another crate turns on `vitaminc-aead-value/chrono` and + `vitaminc-prf`'s `chrono` is off, `Value::Date` exists and `vitaminc-prf` + has no arm for it. `vitaminc-prf` fails to compile in that build. + +A visitor fails the same way: a required `visit_date` that exists only with +`chrono` is a method an implementor can't know whether to write. A crate can +check its own features, never a dependency's. Today's design avoids this only +because the match sits inside `vitaminc-aead-value`, which checks its own +features. + +The fix is to make every variant exist in every build, using vitaminc's own +representation, with conversions to and from chrono and rust_decimal behind +the features: + +| Variant | Representation | +|---|---| +| `Date` | days from the start of the common era, `i32` | +| `Timestamp` | Unix seconds `i64`, nanoseconds `u32` | +| `Decimal` | mantissa and scale | + +The ciphertext wire format already uses these payloads (tags `0x12`–`0x14` +are independent of chrono), so stored ciphertexts don't change. The Rust +types do, which is a breaking change to `Value`. + +### The two approaches + +**A: drop `#[non_exhaustive]`.** `vitaminc-prf`, `vitaminc-ore` and the +bindings match `Value` exhaustively in their own crates. + +**B: a visitor trait, keeping `#[non_exhaustive]`.** `vitaminc-aead-value` +defines a `ValueVisitor` with one required method per kind and does the +dispatch. Crates that must handle every kind implement it. Other crates keep +matching with a `_ =>` arm. + +| | A: drop `#[non_exhaustive]` | B: visitor, keep `#[non_exhaustive]` | +|---|---|---| +| Who gets exhaustiveness | Every crate that matches `Value` | Crates that implement the visitor | +| Adding a kind | Breaking for every crate that matches | Breaking for visitor implementors, additive for others | +| Code to maintain | Nothing new | One trait, one dispatch function, a method per kind in each implementor | +| Grouping | Each match writes out every integer width | The trait can offer grouped hooks, such as one integer method that the per-width methods default to | +| Changing `Value`'s representation | Breaks every match | Hidden behind the method signatures | +| Main risk | Low | A default on a per-kind method lets a new kind compile without being handled. Per-kind methods need no defaults; only grouping hooks may have them. | + +**Evidence:** +- Outside `vitaminc-aead-value`, `Value` is matched in `vitaminc-prf` (through + `canonical`), in `aead-napi`'s `convert.rs` and `scalar.rs`, and in the hmac + tests. +- `aead-napi` already has `_ => Err("unsupported value kind")` + (`convert.rs`). That is the silent refusal `#[non_exhaustive]` invites. +- ADR-0002 requires every binding to handle every kind, so the bindings want + exhaustiveness as well. +- Under A they get it automatically. Under B each binding opts in by + implementing the visitor. + +**Leaning:** B. The decision waits on a closer look at the `Value` enum's +design, which may change the representation in ways that favour one +approach. + +## Current state + +Arrows point from a crate to what it depends on. cipherstash-client and +`vitaminc-protected` are left out. + +```mermaid +flowchart TB + subgraph stack + SE[stack-encrypt] + end + subgraph suite["cipherstash-suite"] + CLLW["cllw-ore 0.5
typed impls, own encodings"] + end + subgraph vitaminc + PRF[vitaminc-prf] + AV["vitaminc-aead-value
Value, canonical module"] + end + subgraph orers["ore.rs"] + ORERS["ore-rs 0.8"] + OB["orderable-bytes"] + end + SE --> PRF + SE --> AV + SE --> CLLW + PRF -- "canonical feature" --> AV + AV -- "git pin, pre-0.2" --> OB + CLLW -- "0.1, chrono and decimal" --> OB + ORERS -- "0.2" --> OB +``` + +- stack-encrypt calls cllw-ore directly for order terms. There is no block + ORE path. +- Equality bytes for dates, timestamps and decimals come from order + encodings, through `vitaminc-aead-value`. +- `orderable-bytes` is reached three ways, at three different versions. + +## Target state + +```mermaid +flowchart TB + subgraph stack + EQI["Equality index
picks transforms"] + ORI["Ore / Ope index
picks scheme and transforms"] + end + subgraph schemes["scheme crates"] + ORERS["ore-rs
block ORE"] + CLLW["cllw-ore
CLLW ORE and OPE"] + end + subgraph vitaminc + PRF["vitaminc-prf
equality terms, own baseline and transforms"] + VORE["vitaminc-ore
order encodings, transforms,
OrderScheme, sealed OrderPlaintext"] + AV["vitaminc-aead-value
Value"] + end + EQI --> PRF + ORI --> VORE + ORI --> ORERS + ORI --> CLLW + ORERS -- "impl OrderScheme" --> VORE + CLLW -- "impl OrderScheme" --> VORE + PRF --> AV + VORE --> AV +``` + +- vitaminc depends on nothing in the other repositories. +- The scheme crates implement `vitaminc-ore`'s trait. The index crates choose + a concrete scheme and the transforms for each column. +- `orderable-bytes` doesn't appear. It stays frozen for cipherstash-client. + +## Changes to ADR-0002 + +These have been made in the ADR itself. + +| ADR-0002 says | Now | +|---|---| +| "Equality and order terms are computed from one canonical form per kind, and both layers use the same one" | Each kind has a semantic baseline; lossy transforms are chosen per domain, independently or locked | +| Canonical forms table: timestamp truncated to microseconds, text order folded | These become per-domain transforms, named in each domain's label | +| Order: "The plaintext layer is `orderable-bytes`, which … gains a variable-length encoding" | `vitaminc-ore` owns the order encodings; `orderable-bytes` is frozen | +| "A `Scheme` trait, with block ORE (over ore-rs), CLLW ORE and CLLW OPE (over cllw-ore)" | `OrderScheme` lives in `vitaminc-ore`; ore-rs and cllw-ore implement it | +| "cllw-ore keeps its typed impls … behind a cargo feature that cipherstash-client enables, and gains a bytes-level entry point" | cllw-ore's typed API is unchanged and not feature-gated; it gains an `OrderScheme` impl behind an optional `vitaminc` feature | +| Review: "Using the canonical order bytes as the PRF input … Dan Draper signs it off" | PRF no longer uses order bytes; the review item applies only to domains that lock equality and order together | +| `Value` and `ValueKind` become `#[non_exhaustive]` | Open: see "Open decision" | +| Release order: "ore.rs (`orderable-bytes`) and cllw-ore first, then vitaminc 0.6.0" | vitaminc first, then cllw-ore and ore-rs, then stack | + +## Steps + +### Phase 1: tidy up work in flight + +1. Close cipherstash/ore.rs#103. Don't yank `orderable-bytes` 0.2.0. +2. Cut cipherstash/cipherstash-suite#2308 down to its first two commits: + known-answer tests that pin every cllw-ore encrypt path, and wiping the + plaintext bit buffer in `encrypt_ope_bits`. It becomes a cllw-ore patch + release. The known-answer tests then protect cipherstash-client's terms + through the rest of this work. + +### Phase 2: vitaminc + +3. Settle the open decision. If the `Value` changes are accepted, make every + variant exist in every build. +4. `vitaminc-aead-value` and `vitaminc-prf`: + - add the chosen exhaustiveness mechanism; + - remove `canonical.rs` and the `orderable-bytes` dependency from + `vitaminc-aead-value`; + - move the equality baseline and transforms into `vitaminc-prf`, with each + transform named in its domain label. +5. New crate `vitaminc-ore` (rewrite cipherstash/vitaminc#374 to match): + - `OrderScheme` and the sealed `OrderPlaintext`; + - order encodings ported from `orderable-bytes`, kept in `Protected`, + pinned by golden vectors, and checked against `orderable-bytes` where + they match byte for byte; + - per-domain transforms: timestamp resolution, text folding with a pinned + Unicode version, room for prefix truncation; + - order-term derivation from `&Value`, generic over the scheme; + - a mock scheme for tests; + - docs that state the combined-leakage rule. +6. A conformance test across `vitaminc-prf` and `vitaminc-ore`: every kind is + handled or listed as an exception, and values equal under `Value` + semantics produce equal terms. +7. Release vitaminc 0.6.0. + +### Phase 3: scheme crates + +8. cllw-ore: implement `OrderScheme` behind an optional `vitaminc` feature, + with known-answer tests for the trait path. This is an additive release. +9. ore-rs: the same. Block ORE uses each kind's natural width; + cipherstash-client keeps its 8-byte padding through the existing API. + Block ORE for text waits for cipherstash/ore.rs#95 and goes on the + exceptions list. +10. Release both. + +### Phase 4: stack + +11. Index crates (#1063): each index spec carries its scheme and transforms. + The Equality index calls `vitaminc-prf`; the Ore and Ope index builds the + concrete scheme and calls `vitaminc-ore`. stack-encrypt drops its direct + use of cllw-ore. +12. EQL v4 domain labels name their transforms. + +### Later + +- Block ORE for text, once cipherstash/ore.rs#95 is released. +- Retire cllw-ore's typed impls and `orderable-bytes` once cipherstash-client + is retired.